# SLU11 - Tree-based models: Learning notebook

## Imports

```python
from math import log2

import pandas as pd
import numpy as np

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from IPython.display import Image

from utils.utils import (
    make_data,
    separate_target_variable,
    process_categorical_features,
    visualize_tree
)
```

# Table of Contents

1. [Decision trees](#decision-trees)
2. [Tree-based ensembles](#tree-based-ensembles)
3. [Bagging with random forests](#bagging-with-random-forests)
4. [Boosting with gradient boosting](#boosting-with-gradient-boosting)

# Decision trees

A **decision tree** is a decision support tool in a form of a tree-like structure, it can be used for classification or regression, being more used for classification, where each node is associated with a test on a feature, each branch represents an outcome of the test, and each leaf the final decision.

In order to explain you better how **decision trees** work lets use an example, where according with the weather we want to know if we should go hiking. A simple **decision tree** can be represented as:

*Fig. 1: A simple decision-tree used for classification (Quinlan, 1986).*

This **decision tree**, displays a flow of conditional statements (i.e., `if Condition then Outcome`) where:

* Each **node** represents a test on a feature (E.g. is it windy?)
* Each **branch** represents the outcome of the test (E.g. as true or false)
* Each **leaf** represents an outcome or decision (represented with N and P).

The paths from root to leaf represent the rules, e.g., `if Outlook is Sunny and Humidity is Normal then Go Hiking`.

## Learning sets of rules as decision trees

Typically, a decision tree is developed from the top-down (known as top-down induction), and there are several algorithms that can be used to build them, including:
* **Iterative Dichotomiser 3 (ID3)**, for classification using categorical features (use of entropy and information gain as metric.)
* **C4.5**, successor to ID3 to support non-categorical features
* **Classification and Regression Trees (CART)**, generalizes C4.5 to support regression (use of Gini impurity as the metric).

We follow ID3 ([Quinlan, 1986][1]), for binary classification using categorical features with a small set of possible values (i.e. low cardinality).

[1]: http://hunch.net/~coms-4771/quinlan.pdf "Quinlan, J. 1986. Induction of Decision Trees. Machine Learning 1: 81-106."

```python
data = make_data()
```

### Algorithm overview

Take an arbitrary training set $C = \{\textbf{x}_i, y_i\}_{i=1}^m$, a collection of labeled observations, with $y_i \in \{L_1, \dots, L_v\}$.

The features vector is represented as $\textbf{x}_i = \{x_i^{(1)}, \dots, x_i^{(n)}\}$, where $x_i^{(j)}$ is the value of the $j$-th feature for the $i$-th observation.

In ID3, $x_i^{(j)}$ can take on one of a fixed number of possible values $x_i^{(j)} \in \{A_1^{(j)}, \dots, A_w^{(j)}\}$.

$$X = \begin{bmatrix}
  x_1^{(1)} & x_1^{(2)} & \dots & x_1^{(n)} \\
  x_2^{(1)} & x_2^{(2)} & \dots & x_2^{(n)} \\
  \dots     & \dots     & \dots & \dots     \\
  x_m^{(1)} & x_m^{(2)} & \dots & x_m^{(n)}
 \end{bmatrix}$$

```python
X, y = separate_target_variable(data)

X
```

      Outlook
      Temperature
      Humidity
      Windy

      0
      sunny
      hot
      high
      false

      1
      sunny
      hot
      high
      true

      2
      overcast
      hot
      high
      false

      3
      rain
      mild
      high
      false

      4
      rain
      cool
      normal
      false

      5
      rain
      cool
      normal
      false

      6
      overcast
      cool
      normal
      false

      7
      sunny
      mild
      high
      false

      8
      sunny
      cool
      normal
      false

      9
      rain
      mild
      normal
      false

      10
      sunny
      mild
      normal
      true

      11
      overcast
      mild
      high
      true

      12
      overcast
      hot
      normal
      false

      13
      rain
      mild
      high
      true

This is a binary classification problem, where $y_i \in \{0, 1\}.$

$$y = \begin{bmatrix}
  y_1   \\
  y_2   \\
  \dots \\
  y_3
 \end{bmatrix}$$

```python
y
```

    0     0
    1     0
    2     1
    3     1
    4     1
    5     0
    6     1
    7     0
    8     1
    9     1
    10    1
    11    1
    12    1
    13    0
    Name: Class, dtype: int64

Now, imagine a test, $T$, on $x_i$, with possible outcomes $O_1, O_2, \dots, O_w$.

It can be a condition on the features, for example.

$T$ produces a partition $\{C_1, C_2, \dots, C_w\}$ of $C$, where $C_k$ contains those observations having outcome $O_k$.

*Fig. 2: Partitioning of the objects in $C$ with a test $T$ (Quinlan, 1986).*

We can recursively apply different tests to each of these partitions, creating smaller and smaller sub-partitions, until we get homogenous (i.e., single-class) leaves. The result of this process is a decision tree for all of $C$.

This tree-building algorithm can be described in the following way:
```
ID3 (Data, Target, Attributes)
    If all examples are positive, Return the single-node tree Root, with label = 1.
    If all examples are negative, Return the single-node tree Root, with label = 0.
    Otherwise Begin
        A <- Pick the Attribute that best classifies examples.
        Decision tree for Root = A.
        For each possible value, O_i, of A,
            Add a new tree branch, corresponding to the test A = O_i.
            Let Data(v_i) be the subset of examples that have the value O_i for A.
            Below this new branch add the subtree ID3 (Data(O_i), Target, Attributes – {A}).
    End
    Return Root
```

But what is the **attribute that best classifies examples** at each step? As we will soon see, this is related with the concepts of **information gain** and **entropy**.

## Attribute selection

A test will have a high **information gain** if it generates partitions with low [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)). In general: the greater the homogeneity of a partition, the lower its entropy.

In this case, a partition is said to be very homogeneous if the majority of its elements belong to a single class.

Consider the problem of binary classification (it can be extended to support multi-class settings). Take $p$ as the probability of the positive class, i.e., the proportion of positive cases in the set. In this case, the entropy is given by:

$$I(p) = - p \log_2 p - (1 - p) \log_2 (1-p)$$

It ranges between 0 and 1:

* 0, when $p=1$ or $p=0$. We are in a situation of low entropy/high purity, because there is no "surprise" in the outcome. We say the information is minimum, because observing the class of an instance will not give us any new information - we already know what it will be.
* 1, when $p=\frac{1}{2}$. We are in a situation of high entropy/low purity, because the "uncertainty" of the outcome is maximum. This is equivalent to saying that the information is maximum.

Out of curiosity, entropy is measured in **bits** (hence the $log_2$ in the formula, since it tells you the number of digits you would need to represent a number in base 2).

```python
def entropy(p):
    if 0 < p < 1:
        return -p * log2(p) - (1 - p) * log2(1 - p)
    else:
        return 0
```

Let's compute the entropy for the entire data set:

```python
def compute_probability(data):
    n = data.shape[0]
    f = (data['Class'] == 1).sum()

    return f / n

p = compute_probability(data)
```

```python
entropy(p)
```

    0.9402859586706311

We have a fairly high entropy for the entire data set.

This means that substantial proportions of the data set belong to each of the classes, as confirmed by the value of $p$ - the data is not homogenous.

```python
p
```

    0.6428571428571429

Starting from a set with high entropy, we would like to select a test that divides it into partitions that are as homogeneous as possible. In the ideal case, each partition would be a leaf node, containing only instances from a single class.

A test is restricted to branching on an attribute $A$ with values $\{A_1, A_2, \dots, A_v\}$, thus partitioning $C$ into $\{C_1, C_2, \dots, C_v\}$.

The expected entropy of the test is obtained as a weighted average of the entropy of the resulting groupings:

$$E(A) = \sum_{i=1}^v \frac{\|C_i\|}{\|C\|}I(p_i)$$

In short, and as expected, we are measuring an attribute's ability to generate homogeneous groupings.

Let's calculate the expected information of the `Outlook` feature:

```python
def mean_entropy(data, attribute):
    c_norm = data.shape[0]
    information = 0
    values = data[attribute].unique()
    for value in values:
        group = data[data[attribute] == value]

        c_i_norm = group.shape[0]

        w = c_i_norm / c_norm

        p = compute_probability(group)
        e = entropy(p)

        information += w * e

    return information

mean_entropy(data, 'Outlook')
```

    0.6935361388961918

Given this result, the `Outlook` feature yields partitions that have, on average, a much lower entropy than the entire data set. It is a good candidate for the first test (first node that is the root!)

We call this loss of entropy the **information gain** of branching on an attribute $A$, and it is given by:

$$IG(A) = I(p) - E(A)$$

where $I(p)$ is the entropy on the node over which the test is being performed, and $E(A)$ is the expected entropy of the test.

Let's compute it.

```python
def information_gain(data, attribute):
    p = compute_probability(data)
    i = entropy(p)

    e = mean_entropy(data, attribute)

    return i - e

information_gain(data, 'Outlook')
```

    0.24674981977443933

To generate the simplest possible tree, it is a good choice to branch, at each step, on the attribute with the highest information gain. Let's find out which:

```python
def examine_candidate_attributes(data, attributes):
    return {
        attribute:information_gain(data, attribute)
        for attribute
        in attributes
    }

attributes = [c for c in data.columns if c is not 'Class']
examine_candidate_attributes(data, attributes)
```

    {'Outlook': 0.24674981977443933,
     'Temperature': 0.02922256565895487,
     'Humidity': 0.15183550136234159,
     'Windy': 0.02507817350585062}

So, we would start by branching on the values of the `Outlook` attribute.

```python
outlook_values = data['Outlook'].unique()
outlook_values
```

    array(['sunny', 'overcast', 'rain'], dtype=object)

We repeat the process in each partition generated by the `Outlook` attribute:

```python
for i, value in enumerate(outlook_values):
    partition = data[data['Outlook'] == value]
    results = examine_candidate_attributes(partition, attributes)
    print("\n---\n")
    print("Partition: Outlook == {}. \nResults:\n{}.".format(value, results))
```

    ---

    Partition: Outlook == sunny.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.5709505944546686, 'Humidity': 0.9709505944546686, 'Windy': 0.01997309402197489}.

    ---

    Partition: Outlook == overcast.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.0, 'Humidity': 0.0, 'Windy': 0.0}.

    ---

    Partition: Outlook == rain.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.01997309402197489, 'Humidity': 0.01997309402197489, 'Windy': 0.3219280948873623}.

Interestingly enough, none of the attributes provides any information gain in the `Outlook == overcast`  case. Let's try to understand why:

```python
data[data['Outlook'] == 'overcast']
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      2
      overcast
      hot
      high
      false
      1

      6
      overcast
      cool
      normal
      false
      1

      11
      overcast
      mild
      high
      true
      1

      12
      overcast
      hot
      normal
      false
      1

As it turns out, this grouping is already homogeneous (all instances are positive) - it is already a leaf. We figured out our first rule:

```
if Outlook is Overcast then Go Hiking
```

Finally, we would continue partitioning recursively until all the resulting groupings are homogeneous (we obtain pure leaves), or until we have exhausted our features (in which case, we assign to the final leaves the majority class in their partition).

## Using decision trees

The `sklearn` implementation uses an optimized version of the CART algorithm. This means that the resulting decision tree might be different than if you had used ID3.

We won't go into this in detail, but don't worry - the principles behind it are very similar to what you've learned.

```python
X_ = process_categorical_features(X)

clf = DecisionTreeClassifier(criterion='entropy', random_state=101)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', random_state=101)

Above, we do a trick to convert categorical variables into something more `sklearn`-friendly.

(ignore it for now - you will learn all about it in due time.)

Then, we train a `DecisionTreeClassifier` (refer to the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) for more information).

Let's visualize the resulting tree:

```python
class_names = ["Don't go hiking!", "Go hiking!"]
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

On each node, we can see:

* two different paths, indicated by each result of the test `(feature_value <= 0.5)`, applied to the current node. `True` means **all instances in this node for which `feature` IS NOT equal to `value`**. The left path is always True, and the right path is always False.
* the entropy in that node.
* the `value` array, which indicates how many samples reaching that node belong to each class (the classes are in ascending numerical order)
* the name of the majority class in the node (in case of a tie, the first class in numerical order is taken)

The CART algorithm can also be used for regression problems. In that case, you should use the `DecisionTreeRegressor` (refer to the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html)). To visualize a simple example, check out [this page](https://scikit-learn.org/stable/auto_examples/tree/plot_tree_regression.html).

## Feature importance

A great characteristic of decision trees is that they allow us to compute **feature importance**: a score for each feature, based on how useful they are at predicting the target variable.

The importance of a feature is given by the decrease in node impurity (according to the criterion being used - in this case, the entropy), weighted by the proportion of samples reaching the node.

We can calculate the feature importance in the following way:

```python
feature_importances = pd.Series(data=clf.feature_importances_, index=X_.columns)
feature_importances.sort_values(ascending=False)
```

    Outlook_overcast    0.283411
    Humidity_high       0.249079
    Outlook_sunny       0.211799
    Windy_true          0.179147
    Temperature_mild    0.076563
    Outlook_rain        0.000000
    Temperature_cool    0.000000
    Temperature_hot     0.000000
    Humidity_normal     0.000000
    Windy_false         0.000000
    dtype: float64

By inspecting the visual representation of the tree, we can see that the `Windy_true` attribute has the highest information gain (from 1 to 0 on both leaves it generate), and therefore the highest decrease in node impurity.

However, since not many samples reach that node (only 2 in the training set), this is not the most important feature, since it is not as relevant to predicting the class in the overall dataset. Compare it with the feature with the highest importance, `Outlook_overcast`, placed right at the root of the tree: despite not having as high of an information gain, it immediately allows us to classify 4 samples as "Go Hiking".

## Pros and cons

### Pros

Decision trees are a straightforward way to represent rules that:

* are simple to understand, interpret and visualize
* require little to no data preparation (for example, don't require data scaling and normalization)
* are able to handle numerical and categorical variables
* missing values in the data doesn't affect (to a larger extent) the process of building the tree
* are a white-box model, all decisions are replicable and easily explainable.

### Cons

Decision trees are extremely flexible and prone to creating over-complex trees that:
* small changes in the data can cause a large change in the structure of the decision tree causing instability
* usually involves higher time to train (leading to an increase in price as the complexity and time increase)
* Overfitting
* Overfitting
* Overfitting.

(Repetition makes perfect.)

Mechanisms such as pruning (removing sections of the tree) and setting the maximum depth of the tree help with overfitting. Let's see how we can configure the maximum tree depth with `sklearn`:

```python
clf = DecisionTreeClassifier(
    criterion='entropy',
    max_depth=2,
    random_state=101
)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', max_depth=2, random_state=101)

```python
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

We can also set the minimum number of samples required to split a node, in order to avoid fully-grown trees.

```python
clf = DecisionTreeClassifier(
    criterion='entropy',
    min_samples_split=5,
    random_state=101
)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', min_samples_split=5,
                           random_state=101)

```python
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

These all help with controlling overfitting, but they might not be enough; and in fact, sometimes this approach is too heavy-handed, and might lead to underfitting. We will need to find a smarter way to control overfitting, while still allowing our trees to represent complex rules.

Some final notes:

* if the attributes are adequate, it is always possible to construct a decision-tree that correctly classifies every training instance.
* attributes are **inadequate** if the data contains two objects that have identical values for each attribute and yet belong to different classes.
* the key takeaway: **decision trees overfit like hell**.

## Tree-based ensembles

**Ensemble methods** combine the predictions of several models, known as **base learners** or **base estimators**. Each individual estimator in the ensemble is often a "weak learner" (i.e. has slightly better accuracy than random). However, combining the predictions of the entire ensemble, we often get better results that are less prone to overfitting, when compared to using a single, larger model.

*Fig. 3: A simple ensemble model, using many trained models to generate a single prediction.*

There are homogeneous and heterogenous ensembles, based on whether the base learners are all of the same type or not.

We focus on homogenous ensembles of decision-trees, particularly:

* Building several independent trees and then average their predictions, so that variance is reduced (i.e., **bagging**)
* Building trees sequentially as to reduce the bias of the combined estimator (i.e., **boosting**).

# Bagging with random forests

Bootstrap aggregating, also known as bagging, consists in:

1. Creating several independent data sets
2. Training a model in each data set
3. Aggregating individual predictions.

Bagging can be seen as training several independent models in parallel and averaging the predictions.

Imagine the following example: you are experiencing a strange feeling in your nose, and want to get a diagnosis. You have the option of going to a single, extremely-specialized podiatrist (that's a foot doctor, in case you didn't know, and in this analogy, it represents a single large model), or you could get your diagnosis from an association of 500 3rd year university students of medicine (which in this analogy represent a bag of simpler, shallower models).

Now, the podiatrist is very specialized: one might even say that he might have **overfit** to foot diseases. He might be able to extend his knowledge to your particular affliction and give a good diagnosis, but odds are he will diagnose you with something totally unrelated.
The group of 3rd year medicine students, on the other hand, might lack individual expertise; but if each of them has at least attended *some* classes, and *if they haven't all attended the same classes*, it's very likely that their collective knowledge surpasses the podiatrist's, and that their majority vote will be the right diagnosis.

This example illustrates the main strength of bagging, which is that several weak models can cancel each other's weaknesses, provided that they are highly independent and were exposed to different information.

But how does this relate to decision trees, you might ask? Well, decision-trees don't simply overfit, they are also highly unstable: small variations in the data result in wildly different trees.

This makes them particularly suited for bagging.

## Bagging

Suppose you have a sequence of data sets $\{D_1, D_2, \dots, D_k\}$, with observations from the same underlying distribution $\mathcal{D}$.

We can obtain an **ensemble of models**, $\{h_1, h_2, \dots, h_k\}$, by training a model in each data set.

*Fig. 4: In bagging different data sets are generated from the main one, models are trained in parallel and predictions averaged.*

By running the models in parallel, we get $\{\hat{y}_1\, \hat{y}_2\, \dots, \hat{y}_k\}$, a list of predictions from our ensemble.

To obtain a single prediction we can then evaluate all models and aggregate the results by:
* Averaging the $k$ predictions (regression)
* Using majority voting to predict a class (classification).

More often than not, however, we don't have multiple data sets. In this situation, we can do bootstrapping.

## Bootstrapping

We can take repeated random samples with replacement $\{C^1, C^2, \dots, C^b\}$ from $C$, our full dataset:

```python
def make_bootstrap_data(data, b):
    n = data.shape[0]
    return [data.sample(n=n, replace=True) for i in range(b)]

bootstrap_data = make_bootstrap_data(data, 2)
```

Typically, the bootstrapped data sets are the same size as the original data set.

```python
bootstrap_data[0]
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      7
      sunny
      mild
      high
      false
      0

      8
      sunny
      cool
      normal
      false
      1

      4
      rain
      cool
      normal
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      10
      sunny
      mild
      normal
      true
      1

      12
      overcast
      hot
      normal
      false
      1

      0
      sunny
      hot
      high
      false
      0

      6
      overcast
      cool
      normal
      false
      1

      4
      rain
      cool
      normal
      false
      1

      0
      sunny
      hot
      high
      false
      0

      2
      overcast
      hot
      high
      false
      1

      5
      rain
      cool
      normal
      false
      0

```python
bootstrap_data[1]
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      2
      overcast
      hot
      high
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      3
      rain
      mild
      high
      false
      1

      9
      rain
      mild
      normal
      false
      1

      9
      rain
      mild
      normal
      false
      1

      2
      overcast
      hot
      high
      false
      1

      4
      rain
      cool
      normal
      false
      1

      8
      sunny
      cool
      normal
      false
      1

      11
      overcast
      mild
      high
      true
      1

      1
      sunny
      hot
      high
      true
      0

      4
      rain
      cool
      normal
      false
      1

      3
      rain
      mild
      high
      false
      1

      10
      sunny
      mild
      normal
      true
      1

      7
      sunny
      mild
      high
      false
      0

We would now train a different decision tree in each data set and use voting to predict whether or not to go hiking.

Bagging works very well in controlling overfitting and the generalization error in unstable models. If one of the models in the bag overfits to a certain noisy observation, it's very likely that its vote will be drowned out by the rest of the models in the ensemble.

## Random forests

A **random forest** is an ensemble learning method created by bagging multiple decision trees.
In random forests, bagging is used in tandem with random feature selection:

* Datasets are generated from the original data, using bootstrapping (row sampling)
* Then a tree is grown on each bootstrapped data set using random feature selection (column sampling).

Random feature selection means that only a random subset of the features is available at each split.

Randomizing features acts as a kind of regularization, further mitigating overfitting, because it forces each individual classifier to be as good as possible, while having access to limited information. This increases diversity inside the ensemble, which is often very beneficial.

Fortunately, `sklearn` implements all this intricate logic for us.

```python
rf = RandomForestClassifier(
    n_estimators=10,
    criterion='entropy',
    max_features=2,
    bootstrap=True
)

rf.fit(X_, y)
```

    RandomForestClassifier(criterion='entropy', max_features=2, n_estimators=10)

For detailed information about the models, check the documentation:
* RandomForestClassifier ([docs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html))
* RandomForstsRegressor ([docs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)).

Bagging is a great technique to reduce overfitting, and as such, to reduce the variance of our final predictions. This comes at the cost of interpretability - it is much easier to interpret the rules contained within a single decision tree, than it is to analyze the driving forces in a majority vote of hundreds or even thousands of models.

# Boosting with gradient boosting

**Boosting** is a different ensemble learning technique. In this case, instead of training several base learners in parallel and averaging their predictions, we train them sequentially; and the input of each model is the residual error of the previous model.

The general idea is to build strong ensembles by combining base learners sequentially, each of them correcting previous errors (and thus reducing bias).

**Note:** this section can be pretty intimidating at first, but don't despair! It will be worth it in the end.

## Boosting

Trees are grown sequentially, and each tree is created using information derived from the previous tree. In each iteration:

1. Errors and misclassifications of the past model are given increased weight in a new training data set
2. The current model is trained on the new training set, fitting the residual errors of the previous model.

As such, each model specializes in correcting past mistakes and misclassifications.

Intuitively, this produces an ensemble of models that are good in different "parts" of the training data, as they sequentially correct each other's mistakes. The final prediction is obtained by summing the predictions of each model in the ensemble.

Boosting is less robust than bagging against overfitting, and as such, it is recommended to control the number of estimators used, the strength of each individual estimator, the learning rate, and use other regularization techniques.

In this example we will use the [boston](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) house-prices data set (regression).

```python
def prepare_boston():
    boston = load_boston()
    X = pd.DataFrame(data=boston.data, columns=boston.feature_names)
    y = pd.Series(data=boston.target, name='price')

    return X, y

X_boston, y_boston = prepare_boston()
```

## What do we mean by "gradient" boosting

To understand the meaning of **gradient boosting**, we need to review a couple of concepts.

### Loss function

A loss function quantifies how bad our predictions are. An example is the squared error, seen below:

$$L_i = (y_i - h(x_i))^2$$

As we can see, the square error increases with the square of the difference between each of our predictions, and the real value. The worse this error across all data samples, the worse will the overall model performance be.

The training of an individual machine learning model is driven by the minimization the total loss, over all observations in the training set, with respect to the model's parameters $\theta$:

$$\min_{\theta} J = \sum_{i=1}^N L(y_i, h(x_i, \theta))$$

$h(x_i, \theta)$ represents our model's predictions, or $\hat{y}$.

### Gradient descent

The gradient of a loss function ($\nabla$), at each point, is its inclination or slope.

It's the fancy term for the multi-variable generalization of the derivative, i.e., the partial derivatives of function $f$ at a given point.

When training a model by gradient descent, we calculate, at each training step, the derivative of the loss $L$ with respect to the model's parameters, and then update the parameters by giving a small step in the opposite direction of this gradient - the direction in which the loss is decreasing:

$$\theta^{n+1} = \theta^{n} - \eta \nabla_{\theta} L(y, \hat{y})$$

As such, at each step, we are trying to gradually move our model to a location in parameter space where the loss function has a lower value.

## Gradient boosting in detail

We now have all pieces required to understand gradient boosting. Let's consider the case of a numerical target variable.

We have our first model in the boosting ensemble, $F^{(1)}(x, \theta)$. This model achieves a certain loss, given by $L(\textbf{y}, F^{(1)}(x, \theta))$.

We now want to add a second model, $h$, to improve our first model in a way that

$$
F^{(1)}(x_1) + h(x_1) = y_1 \\
F^{(1)}(x_2) + h(x_2) = y_2 \\
...
$$

or equivalently,

$$
h(x_1) = y_1 - F^{(1)}(x_1) \\
h(x_2) = y_2 - F^{(1)}(x_2) \\
...
$$

Maybe there isn't a single model that can achieve this. But perhaps some regression tree can do it approximately. We can try to fit it to the following data points:

$$
(x_1, y_1 - F^{(1)}(x_1)) \\
(x_2, y_2 - F^{(1)}(x_2)) \\
...
$$

$y_i - F^{(1)}(x_i)$ are called **residuals**. The role of $h$ is to be strong where the initial model was weak. And if the new model $F^{(2)} = F^{(1)} + h$ still isn't good enough, we can fit an additional regression tree to its residuals, and so on and so forth.

Taking a step back to our first model, imagine we are using the squared error loss function:

$$
L(\textbf{y},F_1(x)) = (\textbf{y} - F^{(1)}(x))^2
$$

We want to minimize $J = \sum_i{L(y_i, F^{(1)}(x_i))}$ by adjusting $F^{(1)}(x_1)$, $F^{(1)}(x_2)$, ...

Now, what we are truly adjusting during learning are the parameters $\theta$ of $F^{(1)}$. But if we take a functional perspective, we can observe that $F^{(1)}(x_i)$ are simply numbers - we can treat them as parameters and take derivatives:

$$
\begin{align}
\frac{\partial J}{\partial F^{(1)}(x_i)} & = \frac{\partial \sum_{j=0}^{n}{L(y_j, F^{(1)}(x_j))}}{\partial F^{(1)}(x_i)} \\
& = \frac{\partial L(y_i, F^{(1)}(x_i))}{\partial F^{(1)}(x_i)} \\
& = -2 (y_i - F^{(1)}(x_i)) \\
& = -2*residuals
\end{align}
$$

As you may have noticed, when the loss function is the squared error, the residuals are proportional to the negative gradients of the loss function!

Thus arises the similarity with gradient descent:

$$
\begin{align}
F^{(2)}(x_i) & = F^{(1)}(x_i) + h(x_i) \\
& = F^{(1)}(x_i) + y_i - F^{(1)}(x_i) \\
& = F^{(1)}(x_i) - \frac{1}{2}\frac{\partial J}{\partial F^{(1)}(x_i)} \\
\end{align}
$$

and

$$
\theta^{n+1} = \theta^{n} - \eta \frac{\partial J}{\partial F(\theta_i)}
$$

To summarize: for regression with **square loss**,

* the residual <=> negative gradient
* fitting h to the residual <=> fitting h to the negative gradient
* adding a new estimator based on the residual <=> adding a new estimator based on the negative gradient

When adding a new estimator to our boosting ensemble, we are actually minimizing a global loss function by using gradient descent, in function space!

The final thing to understand here is that the concept of gradients is more useful than the concept of residuals, because it allows us to generalize to loss functions other than the squared error. As long as we fit each additional model to the negative gradients of the current global model, we will be minimizing our desired global loss function.

## In practice

Let's implement one step of a boosting algorithm, for when the loss function is the mean squared error, based on what we have discussed so far!

### Initialization

We initialize all $F^{(1)}(x_i)$ to a sensible constant, $F^{(1)}(x_i) = \gamma$.

Since our goal is to minimize the mean squared error, we will use the mean value of the target variable in the training set.

```python
y_boston_h0_pred = np.repeat(y_boston.mean(), y_boston.size)
```

We compute the mean squared error of this initial set of predictions.

```python
mean_squared_error(y_boston, y_boston_h0_pred)
```

    84.41955615616556

### Iterations

#### Generating a new data set

We build a new training set at the start of each iteration, by recomputing the target variable to be the gradient of the previous model.

More concretely, at the $j$-th iteration we compute a new target variable $r^{(j)}_i$, corresponding to the gradient of the past iteration:

$$r^{(j)}_{i} = \frac{\partial L(y_i, F^{(j-1)}(x_i))}{\partial F^{(j-1)}(x_i)} = -2(r^{(j-1)}_{i} - F^{(j-1)}(x_i))$$

For the first iteration:

* $r^{(j-1)}_i = y_i$, the original targets
* $F^{(j-1)}(x_i)) = F^{(1)}(x_i) = \gamma$

```python
def compute_gradient(y, y_pred):
    return -2 * (y - y_pred)

y_boston_h0_residual = compute_gradient(y_boston, y_boston_h0_pred)
```

#### Training a decision-tree

We now fit a new decision-tree to the negative gradient that we just calculated, $r^{(j)}_i$:

```python
dt = DecisionTreeRegressor(max_depth=1)
dt.fit(X_boston, y_boston_h0_residual)

y_boston_h0_residual_pred = dt.predict(X_boston)
```

Typically, we will want to train the simplest, shallowest decision-trees as base learners, to avoid overfitting (which is why we are setting the maximum depth of the tree to one.)

#### Updating the global model

We now want to add the predictions made by this model (over the residuals) to the predictions made by the base model. The update rule goes as follows, where $\eta$ is a **learning rate** that controls the magnitude of the update and acts as a regularizer:

$$F^{(j)}(x_i)) = F^{(j-1)}(x_i)) - \eta \cdot \frac{\partial L(y_i, F^{(j-1)}(x_i))}{\partial F^{(j-1)}(x_i)}$$

The learning rate is also known as the *shrinkage factor*, as it shrinks the impact of the corrections if $\eta$ between 0 and 1.

Since we don't know the true value of $y_i$ at prediction time, we plug-in the prediction of the decision-tree we  fit in the previous step:

$$F^{(j)}(x_i)) \approx F^{(j-1)}(x_i)) - \eta \cdot \hat{r}^{(j)}_i(x_i) $$

If everything goes well, with each step we are moving closer and closer to the function that minimizes the squared error.

```python
def update_predictions(previous_prediction, residual_prediction, lr):
    return previous_prediction - lr * residual_prediction

y_boston_h1_pred = update_predictions(y_boston_h0_pred, y_boston_h0_residual_pred, lr=0.1)
```

Now, we compute the mean squared error for the updated predictions.

```python
mean_squared_error(y_boston, y_boston_h1_pred)
```

    70.660188943705

Hurrah! The error decreased. We would continue the process until the error stopped decreasing, or until we detected signs of overfitting.

### Putting it all together

Suppose we have $m$ individual decision-trees (boosting stages)

The final prediction will be given by the sum of the initial constant and all the gradient-based corrections:

$$F^m_i(x_i) = \gamma + \sum_{j=1}^m \eta \cdot \hat{r}^{(j)}_i(x_i)$$

Where $\hat{r}^{(j)}_i(x_i)$ is the output of a decision-tree that predicts the negative gradient of the previous iteration.

A final note: don't worry if you didn't understand all the details of previous section - it was quite heavy on the mathematical side. Just make sure you understand the basics of boosting and the idea behind it, and `sklearn` has got you covered!

## With sklearn

Enough with theory: `sklearn` to the rescue! Let's create a `GradientBoostingClassifier`, which can be used for classification problems:

```python
gb = GradientBoostingClassifier(
    learning_rate=.1,
    n_estimators=10
)

gb.fit(X_, y)
```

    GradientBoostingClassifier(n_estimators=10)

#### Include sampling

Some implementations provide the ability to sample observations and features at each iteration, similarly to what happens with Random Forests. Unsurprisingly, this reduces overfitting at the expense of increased bias.

```python
gb = GradientBoostingClassifier(
    learning_rate=.1,
    n_estimators=10,
    subsample=.5
)
gb.fit(X_, y)
```

    GradientBoostingClassifier(n_estimators=10, subsample=0.5)

# Conclusion

You are now armed with the power of bagging and boosting, two techniques you can easily use to obtain models with a lot of expressive power that also generalize better! Hopefully, you also got some insight into the world of more advanced machine learning techniques.

Make sure to review this notebook well, and when you're ready, go solve the exercises. Good luck!

S01 - Bootcamp and Binary Classification/SLU11 - Tree-Based Models/Learning notebook.ipynb


```python
#6. Formulacion cinetica de lagrange
```

```python
from sympy import symbols, sin, cos
from sympy.physics.mechanics import ReferenceFrame, inertia, Point, dynamicsymbols
from sympy.physics.mechanics import RigidBody, LagrangesMethod, Lagrangian
import numpy as np
```

```python
# Para el ejemplo del movimiento de un cilindro sobre una rampa
t,m,g,I,R,theta,=symbols('t,m,g,I,R,theta')
q1,q2=dynamicsymbols('q1,q2')

a=ReferenceFrame('A')
d=a.orientnew('D','Axis',(theta,a.z))
c=d.orientnew('C','Axis',(q2,d.z))

#Defina puntos de interés y velocidades
o=Point('O')
c_com=o.locatenew('C_com',d.x*q1)
o.set_vel(a,0)
c_com.set_vel(c,0)

Ic=inertia(c,0,0,I)

#Defina un cuerpo rígido para el cilindro
body_cilindro=RigidBody('cilindro',c_com,c,m,(Ic,c_com))

V=m*g*q1*sin(theta)
T=body_cilindro.kinetic_energy(a)
```

```python
#Calcule el Lagrangiano
L=T-V
```

```python
# Utilice el método de Lagrange para obtener las equaciones de movimiento
lm=LagrangesMethod(L,[q1,q2],nonhol_coneqs=[q1.diff(t)-q2.diff(t)*R])
#Observe que se define lambda1 (multiplicador de Lagrange)
lm.form_lagranges_equations()
```

$\displaystyle \left[\begin{matrix}g m \sin{\left(\theta \right)} + m \frac{d^{2}}{d t^{2}} \operatorname{q_{1}}{\left(t \right)} + \operatorname{lam_{1}}{\left(t \right)}\\I \frac{d^{2}}{d t^{2}} \operatorname{q_{2}}{\left(t \right)} - R \operatorname{lam_{1}}{\left(t \right)}\end{matrix}\right]$

```python
# Observe las equaciones de movimiento descritas en la forma MM(q,q')[q' q'' lambda]=[F(q,q') constraints]
MM=lm.mass_matrix_full
F=lm.forcing_full
# Defina el vector xdot como [q' q'' lambda] (expandiendo con lambda)
xdot=lm.q.col_join(lm.u).diff(t)
xdot=xdot.col_join(lm.lam_vec)
MM*(xdot)-F # ==0
```

$\displaystyle \left[\begin{matrix}0\\0\\g m \sin{\left(\theta \right)} + m \frac{d^{2}}{d t^{2}} \operatorname{q_{1}}{\left(t \right)} - \operatorname{lam_{1}}{\left(t \right)}\\I \frac{d^{2}}{d t^{2}} \operatorname{q_{2}}{\left(t \right)} + R \operatorname{lam_{1}}{\left(t \right)}\\- R \frac{d^{2}}{d t^{2}} \operatorname{q_{2}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{q_{1}}{\left(t \right)}\end{matrix}\right]$

```python
# Para el ejemplo del modelo de un tren de aterrizaje

# Defina las variables de movimiento
# y parametros del modelo
q1,q2=dynamicsymbols('q1,q2')
# Parámetros del modelo
h,d,mb,mc,Ia,Ir,g=symbols('h,d,mb,mc,Ia,Ir,g')

#Marcos de referencia
a=ReferenceFrame('A')
b=a.orientnew('B','Axis',(q1,a.z))
c=b.orientnew('C','Axis',(q2,b.x))

# Puntos de interes
o=Point('O')
b_com=o.locatenew('Bcom',-h*b.y/2)
b_end=o.locatenew('Bend',-h*b.y)
c_com=b_end.locatenew('Ccom',d*b.x)

# Defina la velocidad de cada punto
o.set_vel(a,0)
b_com.set_vel(b,0)
c_com.set_vel(c,0)
b_com.v2pt_theory(o,a,b)
c_com.v2pt_theory(b_end,b,c)

#Propiedades inerciales
ic=inertia(c, Ia, Ir, Ir)

# Cuerpos rígidos
body_llanta=RigidBody('llanta',c_com,c,mc,(ic,c_com))

# En el ejemplo anterior calculo el Lagrangiano definiendo explícitamente L=T-V
# en este caso utilice el método Lagrangian
body_llanta.potential_energy=mc*g*(h-h*cos(q1)+d*sin(q1))
L=Lagrangian(a,body_llanta).simplify()
L
```

$\displaystyle \frac{Ia \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2}}{2} + \frac{Ir \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2}}{2} - g mc \left(d \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} - h \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + h\right) + \frac{mc \left(d^{2} + h^{2}\right) \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2}}{2}$

```python
# Defina el escalar M3 como el momento externo sobre el marco B
M3=symbols('M3')
lm=LagrangesMethod(L,[q1,q2],forcelist=[(b,M3*b.z)],frame=a)
lm.form_lagranges_equations()
```

$\displaystyle \left[\begin{matrix}Ir \frac{d^{2}}{d t^{2}} \operatorname{q_{1}}{\left(t \right)} - M_{3} + g mc \left(d \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + h \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)}\right) + mc \left(d^{2} + h^{2}\right) \frac{d^{2}}{d t^{2}} \operatorname{q_{1}}{\left(t \right)}\\Ia \frac{d^{2}}{d t^{2}} \operatorname{q_{2}}{\left(t \right)}\end{matrix}\right]$


# Low rank approximation of BEM matrices with adaptive cross approximation (ACA).

In the last section, I demonstrated how to avoid constructing the full dense BEM matrix by regenerating matrix entries whenever they are needed. This can be helpful for reducing memory costs and, in some situations, actually results in a faster solver too. But, it still suffers from the fundamental problem of having to work with a dense matrix where each matrix-vector product is an $O(n^2)$ operation. Ultimately, for boundary integral methods to be a useful technology, we need to avoid working with dense matrices entirely.

So, here, I'll be demonstrating how the off-diagonal blocks of a BEM matrix can be dramatically compressed via a low-rank approximation. The result will be $O(n\log n)$ or $O(n)$ solution methods that can scale up to millions of elements. Hierarchical matrices (aka H-matrices,{cite:p}`bebendorfHierarchicalMatricesMeans2008`), tree-codes {cite:p}`barnesHierarchicalLogForcecalculation1986`, fast multipole methods (FMM, {cite:p}`beatsonShortCourseFast1997`) and several other techniques all make use of this basic concept. I'd argue these linear or log-linear methods are the largest distinguishing factor between "basic" BEM approaches from "advanced" or "modern" BEM approaches. So, let's dive into a tour of low rank matrices! The tools we build here will form the core of the hierarchical matrix (H-matrix) implementation in the next section.

To start out, let's generate, yet again, a simple self-interaction matrix for a free surface. For simplicity, I just used a triangulated planar rectangle rather than anything realistic. I hid these cells since the code is nothing new.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%config InlineBackend.figure_format='retina'

import cutde
```

```python
surf_L = 4000
n_els_per_dim = 50

mesh_xs = np.linspace(-surf_L, surf_L, n_els_per_dim + 1)
mesh_ys = np.linspace(-surf_L, surf_L, n_els_per_dim + 1)
mesh_xg, mesh_yg = np.meshgrid(mesh_xs, mesh_ys)
surf_pts = np.array([mesh_xg, mesh_yg, 0 * mesh_yg]).reshape((3, -1)).T.copy()

surf_tris = []
nx = ny = n_els_per_dim + 1
idx = lambda i, j: i * ny + j
for i in range(n_els_per_dim):
    for j in range(n_els_per_dim):
        x1, x2 = mesh_xs[i : i + 2]
        y1, y2 = mesh_ys[j : j + 2]
        surf_tris.append([idx(i, j), idx(i + 1, j), idx(i + 1, j + 1)])
        surf_tris.append([idx(i, j), idx(i + 1, j + 1), idx(i, j + 1)])

surf_tris = np.array(surf_tris, dtype=np.int64)
surf_tri_pts = surf_pts[surf_tris]
surf_centroids = np.mean(surf_tri_pts, axis=1)

surf_surf_mat = cutde.disp_matrix(
    surf_centroids + np.array([0, 0, 0.01]), surf_pts[surf_tris], 0.25
)

lhs_reordered = np.empty_like(surf_surf_mat)
lhs_reordered[:, :, :, 0] = surf_surf_mat[:, :, :, 1]
lhs_reordered[:, :, :, 1] = surf_surf_mat[:, :, :, 0]
lhs_reordered[:, :, :, 2] = surf_surf_mat[:, :, :, 2]
lhs_reordered = lhs_reordered.reshape((surf_tris.shape[0] * 3, surf_tris.shape[0] * 3))
lhs_reordered += np.eye(lhs_reordered.shape[0])

A = lhs_reordered
```

## Near-field vs far-field

Let's dig in and investigate the matrix itself! In particular, let's start by looking at two blocks of the matrix.
1. A "near-field" block will contain the diagonal of the matrix. Remember that the diagonal of the matrix consists of entries representing the displacement at the center of the same element on which the slip occurred. You can see the bright yellow diagonal in the figure below. The matrix entries decay rapidly away from the diagonal.
2. The other, a "far-field" block will consist of matrix entries coming from interactions between observation points and source elements that are very far from each other. In the figure below, there's no intense variation in entries.

The thing to notice here is that there is, in some sense, just a lot more going on in the near-field matrix.

```python
nrows = 150
near_field_block = A[:nrows, :nrows]
far_field_block = A[-nrows:, :nrows]

log_near = np.log10(near_field_block)
log_near[np.isnan(log_near)] = -10
log_far = np.log10(far_field_block)
log_far[np.isnan(log_far)] = -10

fig = plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.imshow(log_near, vmin=-10, vmax=1)
plt.title("Near-field")
plt.subplot(1, 2, 2)
ims = plt.imshow(log_far, vmin=-10, vmax=1)
plt.title("Far-field")
cbar_ax = fig.add_axes([0.935, 0.125, 0.015, 0.75])
cb = fig.colorbar(ims, cax=cbar_ax)
plt.show()
```

Another way of visualizing this same "a lot going on" property would be to look at the action of the matrix. So, we'll apply both the nearfield and far-field blocks to a vector with random elements. The difference is striking: the output from the near-field matrix preserves the "randomness" of the input whereas the far-field matrix actually smooths out the input dramatically.

```python
x = np.random.rand(nrows)
y1 = near_field_block.dot(x)
y2 = far_field_block.dot(x)

plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.plot(np.log10(np.abs(y1[2::3])))
plt.title("Near-field")
plt.subplot(1, 2, 2)
plt.plot(np.log10(np.abs(y2[2::3])))
plt.title("Far-field")
plt.show()
```

A third and more rigorous way of discussing this same property is to look at the singular values of the two matrix blocks.

```python
U_near, S_near, V_near = np.linalg.svd(near_field_block)
U_far, S_far, V_far = np.linalg.svd(far_field_block)
```

The plot below shows the $\log_{10}$ ratio of each singular value to the first singular value. Things to note here:
* The near-field singular values do not decay. They are almost all between 1.58 and 1.41. This reflects the behavior above where the randomness of the `x` vector was mostly preserved.
* The far-field singular values decay very quickly. The majority of the singular values are smaller than 1e-6. In other words, the matrix has a "1e-6 approximate rank" of 14 despite technically have a rank of 150. That means that if we only care about accuracy up to about 6 digits, then we can compress this matrix by about a factor of 6.

```python
plt.plot(np.log10(S_near / S_near[0]), label="nearfield")
plt.plot(np.log10(S_far / S_far[0]), label="farfield")
plt.legend()
plt.show()
```

The 14th far-field singular value is below 1e-6.

```python
S_far[14] / S_far[0]
```

    1.8980941903866048e-07

## Approximation with the singular value decomposition (SVD)

So, if these off-diagonal blocks of the matrix have singular values that decay quite quickly, can we approximate these blocks with just a few singular values? The answer is an emphatic yes and this idea is the basis of low-rank approximation methods. Let's explore this idea a bit more with a larger 3000x3000 block coming the lower left corner of the matrix. Because it's also off-diagonal, this block shows the same singular value magnitude decay that we saw before.

```python
block = A[-3000:, :3000]
U, S, V = np.linalg.svd(block)

plt.plot(np.log10(S / S[0]))
plt.show()
```

To continue, it's useful to define what the relevant metric for the matrix approximation that we're going to do. A useful norm is the Frobenius norm because it depends only on the matrix entries and for a matrix $M \in \mathbb{R}^{n x m}$ is defined as:
\begin{equation}
\|M\|_F = \sqrt{\sum_{i,j}^{m,n} M_{ij}^2}
\end{equation}
An important fact is that the Frobenius norm of a matrix is the root of the sum of the square of the singular values. That is:
\begin{equation}
\|M\|_F = \sqrt{\sum_{k}^{min(m,n)} \sigma_k^2}
\end{equation}

So, what if we want to approximate a matrix with a particular tolerance, $\epsilon$ in terms of the Frobenius norm? One way to do it will be to compute the SVD and then find the lowest index $K$ for which the constraint is true:
\begin{equation}
\sqrt{\sum_{k=1}^K \sigma_k^2} - \|M\|_F = \sqrt{\sum_{k=K}^{min(m,n)} \sigma_k^2} < \epsilon
\end{equation}

The middle sum from $K$ to the number of rows/columns is essentially a measure of how important the dropped singular values are. So we want that sum to be smaller than our tolerance! The actual implementation below is surprisingly simple.

Here, I'll choose $\epsilon = 10^{-8}$. As demonstrated below, we only need the first 40 singular values to compute such an approximation, meaning that we can store the left and right singular vectors in matrices of shape `(3000, 40)`. This will compress the matrix by a factor of 37.5x!

```python
eps = 1e-8
eps
```

    1e-08

```python
# Reverse the list of singular values and sum them to compute the
# error from each level of truncation.
frob_K = np.sqrt(np.cumsum(S[::-1] ** 2))[::-1]

appx_rank = np.argmax(frob_K < eps)
appx_rank
```

    40

```python
frob_K
```

    array([1.03496590e-02, 7.73331069e-03, 3.53462239e-03, ..., 2.62094754e-20, 1.47210395e-20, 4.74891493e-21])

And let's take a look at the performance and error resulting from applying this low rank approximation to a random vector. First, I'll create an random input vector `x`. Then, I'll form two matrices `Uappx` and `Vappx`. You can sort of think of these as the entrance and exit to our low-dimensional space. This low-dimensional space allows for a very computationally efficient representation of the TDE interactions. Mutliplying `Vappx` by the 3000-dimensional `x` returns a 40-dimensional vector and then we expand back to 3000 dimensional by multiplying by `Uappx`.

```python
Uappx = U[:, :appx_rank]
Vappx = S[:appx_rank, None] * V[:appx_rank]
```

Next, I'll calculate the correct matrix vector product, `y_true = block.dot(x)` and record the runtime.

```python
x = np.random.rand(block.shape[1])
```

```python
full_time = %timeit -o block.dot(x)
y_true = block.dot(x)
```

    15.8 ms ± 355 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

And then, calculate the low-rank matrix vector product, `y_appx` and record the runtime.

```python
lowrank_time = %timeit -o Uappx.dot(Vappx.dot(x))
y_appx = Uappx.dot(Vappx.dot(x))
```

    114 µs ± 3.8 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

A cursory comparison of the output suggests that the approximation is very very accurate.

```python
print(y_true[:5], y_appx[:5])
```

    [ 0.00156452 -0.00307875  0.00146438  0.00158299 -0.00314753] [ 0.00156452 -0.00307875  0.00146438  0.00158299 -0.00314753]

And the calculation is far, far faster!!

```python
speedup = full_time.best / lowrank_time.best
memory_reduction = block.nbytes / (Uappx.nbytes + Vappx.nbytes)
print(f"speed up         = {speedup}")
print(f"memory reduction = {memory_reduction}")
```

    speed up         = 141.24925308518618
    memory reduction = 37.5

Let's do a bit more detailed investigation of the error and look at the $L^1$, $L^2$ and $L^{\infty}$ relative error in the matrix vector product. We'll also compare the Frobenius norm of the approximated matrix with the Frobenius norm of the original matrix.

```python
abs_true = np.abs(y_true)
l2_true = np.mean(abs_true)
l1_true = np.mean(abs_true)
linf_true = np.max(abs_true)

abs_diff = np.abs(y_appx - y_true)
l2_err = np.sqrt(np.mean(abs_diff ** 2)) / l2_true
l1_err = np.mean(abs_diff) / l1_true
linf_err = np.max(abs_diff) / linf_true
true_frob = np.sqrt(np.sum(block ** 2))
frob_err = np.sqrt(np.sum((Uappx.dot(Vappx) - block) ** 2))
print(f"L1(UVx-y)        = {l1_err}")
print(f"L2(UVx-y)        = {l2_err}")
print(f"Linf(UVx-y)      = {linf_err}")
print(f"Frobenius(M-UV)  = {frob_err}")
```

    L1(UVx-y)        = 1.1535850312628712e-08
    L2(UVx-y)        = 1.5729499549438905e-08
    Linf(UVx-y)      = 4.840348614253369e-08
    Frobenius(M-UV)  = 8.902174079271246e-09

It looks like this SVD approximation worked out really well here! We're getting very low error matrix-vector products. These errors are all somewhat similar to threshold we used for the singular value cutoff. And, based on the Frobenius error, the approximate matrix itself is extremely similar to the original matrix. The Frobenius error is just barely better than our tolerance condition, as expected.

Before we move on, I'll record these error values in a dataframe so that it's easy to compare with the fancier methods in the next two sections.

```python
err_df = pd.DataFrame(
    index=["Rank", "L2(UVx-y)", "L1(UVx-y)", "Linf(UVx-y)", "Frobenius(M-UV)"],
    data=dict(true=[block.shape[0], 0, 0, 0, 0]),
)
err_df["SVD"] = [appx_rank, l2_err, l1_err, linf_err, frob_err]
```

```python
err_df.T
```

      Rank
      L2(UVx-y)
      L1(UVx-y)
      Linf(UVx-y)
      Frobenius(M-UV)

      true
      3000.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00

      SVD
      40.0
      1.572950e-08
      1.153585e-08
      4.840349e-08
      8.902174e-09

## Adaptive cross approximation (ACA)

So, we've managed to create an extremely efficient approximation of our off-diagonal matrix block by using the SVD. That's definitely useful for computing fast matrix-vector products or even for computing a LU decomposition. But, it still suffers from the need to compute the entire matrix block in the first place, an $O(n^2)$ operation! If we're going to be throwing all that information away immediately after computing the SVD, is there a way to avoid computing the dense matrix block in the first place? There are several solutions to this problem like randomized SVDs {cite:p}`halkoFindingStructureRandomness2010`, but the most useful solution for our setting is the adaptive cross approximation (ACA) method {cite:p}`bebendorfApproximationBoundaryElement2000, bebendorfAdaptiveLowRankApproximation2003`. These algorithms depending on the ability to compute arbitrary individual matrix entries without computing the entire matrix. By making certain assumptions about the structure of a matrix, we can be confident that an accurate approximation can be constructed from just a few rows and columns.

The basic idea of ACA is to approximate a matrix with a rank 1 outer product of one row and one column of that same matrix. And then iteratively use this process to construct an approximation of arbitrary precision. Ideally, at each step, we will choose the best fitting row and column. After the first iteration, we are no longer trying to approximate the original matrix, but instead we approximate the residual matrix formed by the difference between the original matrix and the current approximation matrix. Eventually, given certain matrix properties that have been proven true for BEM problem  {cite:p}`bebendorfAdaptiveLowRankApproximation2003`, the procedure will converge.

For the sake of real-world usage, the description above will be sufficient for understanding what's going on. But, if you want to dive into a detailed description and implementation of the method, see below! Otherwise, skip over the next two sub-sections.

### ACA and ACA+, the details
The simple version, runs like (following {cite:t}`grasedyckAdaptiveRecompressionMatrices2005`):

**ACA with full pivoting**: Given a matrix $M \in \mathbb{R}^{n x m}$, we'll construct an approximation like $\sum_{k}^{r} u_k v_k^T$. The task will be to construct the row and column vectors $u_k$ and $v_k$ on each iteration such that the Frobenius norm error eventually converges. To do that, the key will be to iteratively form rank-1 approximations to the residual matrix. The residual matrix is the matrix forming the difference between $M$ and the current approximation and can be written as $R_{ij} = M_{ij} - \sum_{k}^{r} u_{kj} v_{ki}$ representing the entry-wise difference between the target matrix and the current approximation. The goal will be to have $R$ satisfy $\|R\|_2 < \epsilon$ where $\epsilon$ is a user-specified accuracy parameter. To do this, during each iteration:
1. Determine a "pivot" $(i^*, j^*)$ as the indices that maximize $R_{ij}$. Intuitively, the largest rows and columns are going to be most "important".
2. Assign a new rank-1 component of the approximation:

    \begin{align}
    u_{kj} &= R_{i^*,j} / R_{i^*,j^*} \\
    v_{ki} &= R_{i,j^*}
    \end{align}
3. Update $R$ to account for the new rank-1 update to the approximation.
4. If the magnitude of the $u_k v_k$ update is small enough, stop. Otherwise, return to step 1.

The reason the algorithm is called "ACA with full pivoting" is because we're allowing the algorithm to choose an arbitrary $(i^*, j^*)$ in the first step. However, that is impossible in our real-world setting because it requires full knowledge of the residual matrix. But, we don't have all the entries of $M$ so we can't have all the entries of $R_{ij}$. Computing all the entries of $M$ is exactly what we're trying to avoid! But, this basic "full information" algorithm is instructive and can be modified slightly so that we don't need the whole matrix.

**ACA+**: Most real-world application use either *ACA with partial pivoting* or the *ACA+* ("ACA plus") algorithm. Here, I'll introduce the modifications necessary for ACA+. The main distinction is that instead of searching over all matrix indices in step 1, we will search over a subset specified by a random row and a random row. This row and column will be our window into the matrix. Instead of finding the largest entry across the full pivoting algorithm in step 1, we will just find the largest entry in either the row window or the column window. While the ACA+ approximation will be less efficient and will converge more slowly than ACA with full pivoting, the end result will still be very good. Note that this approach may completely fail for general low rank matrices, but is very successful for the low rank matrices specifically coming from BEM problems.

The algorithm: Before starting the iteration, we choose a random row, $i_{\mathrm{ref}}$ and random $j_{\mathrm{ref}}$. And we will maintain the corresponding row and column of the residual matrix, $R$. At the start of the algorithm $R_{i_{\mathrm{ref}}, j} = M_{i_{\mathrm{ref}}, j}$ and $R_{i, j_{\mathrm{ref}}} = M_{i, _{\mathrm{ref}}j}$

Then, the iteration proceeds like:
1. Find the index $j^*$ that maximizes $R_{i_{\mathrm{ref}}, j}$ and the index $i^*$ that maximizes $R_{i, j_{\mathrm{ref}}}$. Essentially, we are finding the largest entries in each of these vectors.
2. If $R_{i_{\mathrm{ref}}, j^*} > R_{i^*, j_{\mathrm{ref}}}$ then, we compute the column corresponding to $j^*$ or if the opposite is true, we compute the row corresponding to $i^*$. Essentially, we are determining here whether we should pivot first based on the row or the column.
3. If we pivoted based on column, we should now have computed a new residual column $R_{i,j^*} = M_{i,j^*} - \sum_{k}^{r} u_{kj^*} v_{ki}$. Find the missing pivot index now by maximizing $R_{i,j^*}$ to get $i^*$. Or if we pivoted on the row, we will have a new residual row $R_{i^*,j} = M_{i^*,j} - \sum_{k}^{r} u_{kj} v_{ki^*}$ and we maximize $R_{i^*,j}$ to get $j^*$. The idea here is to finish the pivot operation from step 2 by pivoting in the dimension that we have not considered yet. At each step, we are essentially trying to find the largest residual matrix element out of all the entries we have seen so far with the goal of getting as close as possible to the full pivoting algorithm without actually needing to calculate all the residual matrix elements. Why am I referring to the identification of the largest entries as "pivoting"? This is by analogy to various matrix operations like LU decomposition where the numerical stability is best when the largest entries are handled first. ACA can also be reframed as an iterative triangular decomposition of a matrix. Before moving on, note how we calculate the residual row (column) here by computing the original matrix row and then subtracting the row (column) of the current approximation. This is critical since it means that we're only compunting a single row or column of the original matrix.
4. Next, just like in the full pivoting algorithm, assign:

    \begin{align}
    u_{kj} &= R_{i^*,j} / R_{i^*,j^*} \\
    v_{ki} &= R_{i,j^*}
    \end{align}
5. And update the $R_{i_{\mathrm{ref}}, j}$ row and $R_{i, j_{\mathrm{ref}}}$ by subtracting the new terms of the approximation.
6. Finally, if the magnitude of the $u_k v_k$ update is small enough, stop. Otherwise, return to step 1.

I hope the key difference with the full pivoting algorithm is now clear -- we only need to compute a single row and a single column of the matrix for each iteration. If the algorithm converges in a number of iterations less than the rank of the matrix $M$, then we will have computed only a small subset of the entries. I've deliberately left some of the details vague in order to make the salient features of the algorithm more prominent. But, below, I'm going to go through a full implementation of the algorithm so hopefully that will clear up any of the details.

### Implementing ACA+

An implementation of ACA+ is below. I've put lots of comments throughout to help explain the details. But, if you don't want to dive into the details here, just skip over this section.

```python
def ACA_plus(n_rows, n_cols, calc_rows, calc_cols, eps, max_iter=None, verbose=False):
    """
    Run the ACA+ plus algorithm on a matrix implicitly defined by the
    row and column computation functions passed as arguments.

    :param n_rows:
    :param n_cols:
    :param calc_rows: A function that accepts two parameters (Istart, Iend)
        specifying the first and last row desired and returns a numpy array
        with shape (Iend-Istart, N_col) with the corresponding rows of the
        input matrix
    :param calc_cols: A function that accepts two parameters (Jstart, Jend)
        specifying the first and last column desired and returns a numpy array
        with shape (N_rows, Jend-Jstart) with the corresponding columns of the
        input matrix
    :param eps: The tolerance of the approximation. The convergence condition is
        in terms of the difference in Frobenius norm between the target matrix
        and the approximation
    :param max_iter:
    :param verbose: Should we print information at each iteration. Just included
        for demonstration here.

    :return U_ACA: The left-hand approximation matrix.
    :return V_ACA: The right-hand approximation matrix.
    """

    us = []  # The left vectors of the approximation
    vs = []  # The right vectors of the approximation
    prevIstar = []  # Previously used i^* pivots
    prevJstar = []  # Previously used j^* pivots

    # a quick helper function that will help find the largest entry in
    # an array while excluding some list of `disallowed`  entries.
    def argmax_not_in_list(arr, disallowed):
        arg_sorted = arr.argsort()
        max_idx = arg_sorted.shape[0] - 1
        while True:
            if arg_sorted[max_idx] in disallowed:
                max_idx -= 1
            else:
                break
        return arg_sorted[max_idx]

    # A function that will return a contiguous block of rows of the
    # residual matrix
    def calc_residual_rows(Istart, Iend):
        # First calculate the rows of the original matrix.
        out = calc_rows(Istart, Iend).copy()
        # Then subtract the current terms of the approximation
        for i in range(len(us)):
            out -= us[i][Istart:Iend][:, None] * vs[i][None, :]
        return out

    # See above, except this function calculates a block of columns.
    def calc_residual_cols(Jstart, Jend):
        out = calc_cols(Jstart, Jend).copy()
        for i in range(len(us)):
            out -= vs[i][Jstart:Jend][None, :] * us[i][:, None]
        return out

    # A function for finding a reference row and updating
    # it with respect to the already constructed approximation.
    def reset_reference_row(Iref):
        # When a row gets used in the approximation, we will need to
        # reset to use a different reference row. Just, increment!
        while True:
            Iref = (Iref + 3) % n_rows
            Iref -= Iref % 3
            if Iref not in prevIstar:
                break

        # Grab the "row" (actually three rows corresponding to the
        # x, y, and z components for a single observation point)
        return calc_residual_rows(Iref, Iref + 3), Iref

    # Same function as above but for the reference column
    def reset_reference_col(Jref):
        while True:
            Jref = (Jref + 3) % n_cols
            Jref -= Jref % 3
            if Jref not in prevJstar:
                break

        return calc_residual_cols(Jref, Jref + 3), Jref

    # If we haven't converged before running for max_iter, we'll stop anyway.
    if max_iter is None:
        max_iter = np.min([n_rows, n_cols])
    else:
        max_iter = np.min([n_rows, n_cols, max_iter])

    # Create a buffer for storing the R_{i^*,j} and R_{i, j^*}
    RIstar = np.zeros(n_cols)
    RJstar = np.zeros(n_rows)

    # Choose our starting random reference row and column.
    # These will get incremented by 3 inside reset_reference_row
    # so pre-subtract that.
    Iref = np.random.randint(n_rows) - 3
    Jref = np.random.randint(n_cols) - 3
    # And collect the corresponding blocks of rows/columns
    RIref, Iref = reset_reference_row(Iref)
    RJref, Jref = reset_reference_col(Jref)

    for k in range(max_iter):
        # These two lines find the column in RIref with the largest entry (step 1 above).
        maxabsRIref = np.max(np.abs(RIref), axis=0)
        Jstar = argmax_not_in_list(maxabsRIref, prevJstar)

        # And these two find the row in RJref with the largest entry (step 1 above).
        maxabsRJref = np.max(np.abs(RJref), axis=1)
        Istar = argmax_not_in_list(maxabsRJref, prevIstar)

        # Check if we should pivot first based on row or based on column (step 2 above)
        Jstar_val = maxabsRIref[Jstar]
        Istar_val = maxabsRJref[Istar]
        if Istar_val > Jstar_val:
            # If we pivot first on the row, then calculate the corresponding row
            # of the residual matrix.
            RIstar[:] = calc_residual_rows(Istar, Istar + 1)[0]

            # Then find the largest entry in that row vector to identify which
            # column to pivot on. (See step 3 above)
            Jstar = argmax_not_in_list(np.abs(RIstar), prevJstar)

            # Calculate the corresponding residual column!
            RJstar[:] = calc_residual_cols(Jstar, Jstar + 1)[:, 0]
        else:
            # If we pivot first on the column, then calculate the corresponding column
            # of the residual matrix.
            RJstar[:] = calc_residual_cols(Jstar, Jstar + 1)[:, 0]

            # Then find the largest entry in that row vector to identify which
            # column to pivot on.  (See step 3 above)
            Istar = argmax_not_in_list(np.abs(RJstar), prevIstar)

            # Calculate the corresponding residual row!
            RIstar[:] = calc_residual_rows(Istar, Istar + 1)[0]

        # Record the pivot row and column so that we don't re-use them.
        prevIstar.append(Istar)
        prevJstar.append(Jstar)

        # Add the new rank-1 outer product to the approximation (see step 4 above)
        vs.append(RIstar / RIstar[Jstar])
        us.append(RJstar.copy())

        # How "large" was this update to the approximation?
        step_size = np.sqrt(np.sum(us[-1] ** 2) * np.sum(vs[-1] ** 2))
        if verbose:
            print(
                f"pivot row={Istar:4d}, pivot col={Jstar:4d}, "
                f"step size={step_size:1.3e}, "
                f"tolerance={eps:1.3e}"
            )

        # The convergence criteria will simply be whether the Frobenius norm of the
        # step is smaller than the user provided tolerance.
        if step_size < eps:
            break

        # We also break here if this is the last iteration to avoid wasting effort
        # updating the reference row/column
        if k == max_iter - 1:
            break

        # If we didn't converge, let's prep the reference residual row and
        # column for the next iteration:

        # If we pivoted on the reference row, then choose a new reference row.
        # Remember that we are using a x,y,z vector "row" or
        # set of 3 rows in an algebraic sense.
        if Iref <= Istar < Iref + 3:
            RIref, Iref = reset_reference_row(Iref)
        else:
            # If we didn't change the reference row of the residual matrix "R",
            # update the row to account for the new components of the approximation.
            RIref -= us[-1][Iref : Iref + 3][:, None] * vs[-1][None, :]

        # If we pivoted on the reference column, then choose a new reference column.
        # Remember that we are using a x,y,z vector "column" or
        # set of 3 columns in an algebraic sense.
        if Jref <= Jstar < Jref + 3:
            RJref, Jref = reset_reference_col(Jref)
        else:
            # If we didn't change the reference column of the residual matrix "R",
            # update the column to account for the new components of the approximation.
            RJref -= vs[-1][Jref : Jref + 3][None, :] * us[-1][:, None]

    # Return the left and right approximation matrices.
    # The approximate is such that:
    # M ~ U_ACA.dot(V_ACA)
    U_ACA = np.array(us).T
    V_ACA = np.array(vs)

    return U_ACA, V_ACA
```

## ACA+ is accurate and efficient

And let's demonstrate the algorithm on the same block we used to demonstrate the SVD compression method. One note here: I set the error tolerance to `eps / 50.0` because the error tolerance from the ACA algorithm is imprecise. The factor of 50 is large enough to be very confident that the error will be less than or equal to `eps`. We'll see in the next section how to get a tighter tolerance.

```python
U_ACA, V_ACA = ACA_plus(
    block.shape[0],
    block.shape[1],
    lambda Istart, Iend: block[Istart:Iend, :],
    lambda Jstart, Jend: block[:, Jstart:Jend],
    eps / 50.0,
    verbose=True,
)
```

    pivot row= 221, pivot col=2917, step size=5.593e-03, tolerance=2.000e-10
    pivot row= 220, pivot col=2918, step size=5.590e-03, tolerance=2.000e-10
    pivot row=  29, pivot col=2719, step size=3.061e-03, tolerance=2.000e-10
    pivot row=  28, pivot col=2720, step size=3.072e-03, tolerance=2.000e-10
    pivot row= 298, pivot col=2996, step size=6.632e-04, tolerance=2.000e-10
    pivot row= 118, pivot col=2828, step size=6.273e-04, tolerance=2.000e-10
    pivot row= 107, pivot col=2815, step size=7.560e-04, tolerance=2.000e-10
    pivot row= 599, pivot col=2995, step size=5.317e-04, tolerance=2.000e-10
    pivot row=2702, pivot col=2700, step size=1.364e-04, tolerance=2.000e-10
    pivot row=   3, pivot col=   2, step size=1.370e-04, tolerance=2.000e-10
    pivot row= 297, pivot col= 299, step size=9.059e-05, tolerance=2.000e-10
    pivot row=2701, pivot col= 101, step size=1.407e-04, tolerance=2.000e-10
    pivot row=2810, pivot col= 142, step size=1.168e-04, tolerance=2.000e-10
    pivot row= 219, pivot col=2960, step size=3.189e-05, tolerance=2.000e-10
    pivot row=   2, pivot col=2808, step size=7.423e-05, tolerance=2.000e-10
    pivot row= 161, pivot col=2869, step size=2.313e-05, tolerance=2.000e-10
    pivot row= 111, pivot col=2774, step size=4.236e-05, tolerance=2.000e-10
    pivot row=2999, pivot col=2997, step size=4.235e-05, tolerance=2.000e-10
    pivot row= 263, pivot col=2928, step size=1.205e-05, tolerance=2.000e-10
    pivot row=  45, pivot col=2702, step size=7.899e-06, tolerance=2.000e-10
    pivot row=   1, pivot col= 803, step size=1.101e-05, tolerance=2.000e-10
    pivot row=  71, pivot col=2742, step size=2.492e-06, tolerance=2.000e-10
    pivot row=2627, pivot col=2868, step size=6.883e-06, tolerance=2.000e-10
    pivot row= 159, pivot col=1235, step size=1.700e-06, tolerance=2.000e-10
    pivot row=1232, pivot col=1801, step size=1.790e-06, tolerance=2.000e-10
    pivot row=2995, pivot col=2864, step size=3.872e-06, tolerance=2.000e-10
    pivot row= 166, pivot col=1784, step size=2.127e-06, tolerance=2.000e-10
    pivot row=1354, pivot col=1949, step size=7.805e-07, tolerance=2.000e-10
    pivot row=1043, pivot col=2455, step size=6.554e-07, tolerance=2.000e-10
    pivot row=1499, pivot col=2350, step size=6.247e-07, tolerance=2.000e-10
    pivot row= 797, pivot col= 297, step size=4.305e-07, tolerance=2.000e-10
    pivot row=2997, pivot col=1502, step size=3.025e-07, tolerance=2.000e-10
    pivot row= 256, pivot col=2099, step size=1.669e-07, tolerance=2.000e-10
    pivot row=1202, pivot col=   1, step size=1.220e-07, tolerance=2.000e-10
    pivot row= 980, pivot col=2773, step size=1.585e-07, tolerance=2.000e-10
    pivot row= 287, pivot col=2958, step size=1.192e-07, tolerance=2.000e-10
    pivot row=1151, pivot col=2965, step size=5.752e-08, tolerance=2.000e-10
    pivot row=2700, pivot col=2315, step size=1.643e-07, tolerance=2.000e-10
    pivot row=  70, pivot col=1877, step size=1.287e-07, tolerance=2.000e-10
    pivot row= 261, pivot col=2888, step size=1.342e-07, tolerance=2.000e-10
    pivot row= 137, pivot col= 298, step size=6.915e-08, tolerance=2.000e-10
    pivot row=  75, pivot col=2744, step size=3.963e-08, tolerance=2.000e-10
    pivot row=2972, pivot col=2772, step size=2.674e-08, tolerance=2.000e-10
    pivot row=1174, pivot col=2804, step size=1.857e-08, tolerance=2.000e-10
    pivot row=1228, pivot col= 269, step size=1.423e-08, tolerance=2.000e-10
    pivot row=2735, pivot col=   0, step size=9.794e-09, tolerance=2.000e-10
    pivot row=  11, pivot col=2236, step size=1.040e-08, tolerance=2.000e-10
    pivot row= 189, pivot col=1199, step size=8.859e-09, tolerance=2.000e-10
    pivot row=2802, pivot col=  99, step size=6.024e-09, tolerance=2.000e-10
    pivot row= 190, pivot col=1442, step size=7.604e-09, tolerance=2.000e-10
    pivot row= 814, pivot col=2126, step size=4.106e-09, tolerance=2.000e-10
    pivot row= 299, pivot col=2701, step size=6.409e-09, tolerance=2.000e-10
    pivot row= 191, pivot col=2838, step size=4.836e-09, tolerance=2.000e-10
    pivot row=2177, pivot col=2098, step size=4.466e-09, tolerance=2.000e-10
    pivot row=1613, pivot col=2904, step size=2.597e-09, tolerance=2.000e-10
    pivot row=1973, pivot col=2718, step size=1.892e-09, tolerance=2.000e-10
    pivot row=2911, pivot col=  29, step size=3.342e-09, tolerance=2.000e-10
    pivot row=2730, pivot col= 249, step size=5.779e-09, tolerance=2.000e-10
    pivot row=2934, pivot col= 167, step size=3.386e-09, tolerance=2.000e-10
    pivot row= 314, pivot col=1009, step size=1.504e-09, tolerance=2.000e-10
    pivot row=  46, pivot col=2978, step size=2.503e-09, tolerance=2.000e-10
    pivot row= 280, pivot col=2213, step size=2.066e-09, tolerance=2.000e-10
    pivot row=  53, pivot col=2743, step size=4.349e-10, tolerance=2.000e-10
    pivot row=2785, pivot col=  39, step size=1.016e-09, tolerance=2.000e-10
    pivot row= 403, pivot col=1019, step size=1.478e-09, tolerance=2.000e-10
    pivot row=1181, pivot col=2839, step size=4.810e-10, tolerance=2.000e-10
    pivot row= 245, pivot col=2982, step size=5.693e-10, tolerance=2.000e-10
    pivot row= 647, pivot col=2097, step size=6.857e-10, tolerance=2.000e-10
    pivot row=2102, pivot col= 817, step size=2.573e-10, tolerance=2.000e-10
    pivot row=  21, pivot col=2708, step size=2.621e-10, tolerance=2.000e-10
    pivot row=1121, pivot col=2893, step size=1.041e-10, tolerance=2.000e-10

```python
U_ACA.shape
```

    (3000, 71)

Clearly, the process is converging. We've produced an accurate approximation with a rank of 72. You can see the step sizes are converging at almost the rate we would expect from the graph of singular value magnitudes above. As we'll see below, the ACA+ algorithm is achieving similar levels of accuracy to the SVD with only about 50% more rows and columns.

First, let's check the accuracy of this matrix approximation by looking at the error from a matrix-vector product.

```python
y_aca = U_ACA.dot(V_ACA.dot(x))
```

```python
abs_diff = np.abs(y_aca - y_true)
l2_err = np.sqrt(np.mean(abs_diff ** 2)) / l2_true
l1_err = np.mean(abs_diff) / l1_true
linf_err = np.max(abs_diff) / linf_true
frob_err = np.sqrt(np.sum((U_ACA.dot(V_ACA) - block) ** 2))
print(f"L1(UVx-y)        = {l1_err}")
print(f"L2(UVx-y)        = {l2_err}")
print(f"Linf(UVx-y)      = {linf_err}")
print(f"frob error(M-UV) = {frob_err}")

err_df["ACA"] = [U_ACA.shape[1], l2_err, l1_err, linf_err, frob_err]
```

    L1(UVx-y)        = 8.219936808032213e-09
    L2(UVx-y)        = 1.3115374322498997e-08
    Linf(UVx-y)      = 2.3520182731878858e-08
    frob error(M-UV) = 4.3695129771496895e-10

Looking great! We specified the tolerance as `eps` for both ACA+ and for the SVD. ACA overshot a little and is giving accuracy substantially better than the SVD. This is because of the lower tolerance we passed. Due to the inherent randomness in the algorithm, sometimes ACA+ won't overshoot quite as much. Either way, for a confident tolerance with ACA+ we need to overshoot a little bit because ACA+ is using only local information from the current iteration to decide when to stop whereas the SVD is able to be absolutely certain about the singular values.

```python
terms = []
l1_mvp_errs = []
frob_errs = []
for i in range(500):
    U, V = ACA_plus(
        block.shape[0],
        block.shape[1],
        lambda Istart, Iend: block[Istart:Iend, :],
        lambda Jstart, Jend: block[:, Jstart:Jend],
        eps / 50.0,
        verbose=False,
    )
    terms.append(U.shape[1])
    y_aca = U.dot(V.dot(x))
    l1_mvp_errs.append(np.mean(np.abs(y_aca - y_true)) / l1_true)
    frob_errs.append(np.sqrt(np.sum((U.dot(V) - block) ** 2)))
```

Before moving on, I wanted to demonstrate that the randomness inherent in the reference row/column selection in the ACA+ algorithm is not affecting the end result. See below for histograms showing the distribution of the number of approximation terms and the $L^1$ matrix-vector error of the approximate matrix. As you can see, ACA+ is producing consistent results within a small range of rank. The matrix-vector error is also consistently and acceptably good. The Frobenius error is also 5-50x below the tolerance specified.

```python
plt.figure(figsize=(13, 4))
plt.subplot(1, 3, 1)
plt.title("Approximation rank")
plt.hist(terms, bins=np.arange(70, 85), color="k")
plt.xlabel("rank of approximation")
plt.subplot(1, 3, 2)
plt.title("$L^1$ matrix-vector error")
plt.hist(np.log10(l1_mvp_errs), bins=15, color="k")
plt.xlabel("$\log_{10}(\|E\|^1)$")
plt.subplot(1, 3, 3)
plt.title("Frobenius Error")
plt.hist(np.log10(frob_errs), bins=15, color="k")
plt.gca().axvline(np.log10(eps), color="r")
plt.xlabel("$\|E\|_{F})$")
plt.show()
```

## SVD Recompression

In some sense, the approximation we built with ACA is too good. Most of the time, we're getting Frobenius errors much lower than we asked for. But there's some inconsistency in the output error. This inconsistency is why we asked for `eps/50.0` as the convergence criterion. However, we would ideally want an algorithm that *juuust barely* meets the tolerance condition. Unfortunately that's quite difficult when the algorithm is stuck with limited information about the matrix under consideration. So, in this final section I'll demonstrate a simple post-processing method that will reduce the size/cost of the approximation to more precisely fit the requested tolerance.

The basic idea is quite simple: just do another compression of the ACA approximation matrices with the singular value decomposition. We wanted to avoid doing an SVD on the original full-size matrix block, but now that we have constructed a reduced expression, it can be worthwhile to do the extra effort to reduce it even further with an exact SVD.

How do we do that? I've already built an approximation that looks like:
\begin{equation}
M \approx UV^T
\end{equation}
Basically $V^T$ maps from a high-dimensional space to a low-dimensional space and then $U$ maps back from the low-dimensional space to the high-dimensional space. So, let's first extract the low-dimensional operation. To do that, perform a QR decomposition on both $U$ and $V$. Then we have:
\begin{equation}
UV^T = Q_UR_UR_V^TQ_V^T
\end{equation}
One way of thinking about this is that $Q_V^T$ and $Q_U$ are just rotations now and $R_UR_V^T$ is a square matrix in the low-dimensional subspace that is performing the non-rotational action of $M$.

Now, take the SVD of that small action matrix:
\begin{equation}
R_UR_V^T = W \Sigma Z^T
\end{equation}

And we'll filter on the singular values in $\Sigma$ to reduce the rank of the matrix just like we did before with the full SVD. That's it! To reconstruct the approximation, we just need:
\begin{align}
\overline{U} &= Q_UW\Sigma \\
\overline{V} &= ZQ_V \\
M &\approx \overline{U}\overline{V}^T
\end{align}

The transformation into code is super straightforward!

```python
QU, RU = np.linalg.qr(U_ACA)
QV, RV = np.linalg.qr(V_ACA.T)
W, SIG, Z = np.linalg.svd(RU.dot(RV.T))
```

```python
# Reverse the list of singular values and sum them to compute the
# error from each level of truncation.
frob_K = np.sqrt(np.cumsum(SIG[::-1] ** 2))[::-1]
r = np.argmax(frob_K < eps)
r
print(f"Recompressing from rank {SIG.shape[0]} to rank {r}")
```

    Recompressing from rank 71 to rank 40

```python
U_ACA2 = QU.dot(W[:, :r] * SIG[:r])
V_ACA2 = Z[:r, :].dot(QV.T)
```

Just seven lines of code to recompress by almost a factor of 2! This will make later matrix-vector products almost twice as fast.

```python
y_aca_2 = U_ACA2.dot(V_ACA2.dot(x))
```

```python
abs_diff = np.abs(y_aca_2 - y_true)
l2_err = np.sqrt(np.mean(abs_diff ** 2)) / l2_true
l1_err = np.mean(abs_diff) / l1_true
linf_err = np.max(abs_diff) / linf_true
frob_err = np.sqrt(np.sum((U_ACA2.dot(V_ACA2) - block) ** 2))
print(f"L1(UVx-y)        = {l1_err}")
print(f"L2(UVx-y)        = {l2_err}")
print(f"Linf(UVx-y)      = {linf_err}")
print(f"frob error(M-UV) = {frob_err}")

err_df["Recompress"] = [U_ACA2.shape[1], l2_err, l1_err, linf_err, frob_err]
```

    L1(UVx-y)        = 1.5730624490743533e-08
    L2(UVx-y)        = 2.058148695561588e-08
    Linf(UVx-y)      = 5.0074610366361435e-08
    frob error(M-UV) = 8.910902539999634e-09

As you can see in the table below, the amazing thing here is that ACA and then SVD recompression can recover a matrix approximation that is almost exactly as good as the compression that is computed using a direct SVD. The error in the table between the "SVD" and "Recompress" is shockingly close.

```python
err_df.T
```

      Rank
      L2(UVx-y)
      L1(UVx-y)
      Linf(UVx-y)
      Frobenius(M-UV)

      true
      3000.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00

      SVD
      40.0
      1.572950e-08
      1.153585e-08
      4.840349e-08
      8.902174e-09

      ACA
      71.0
      1.311537e-08
      8.219937e-09
      2.352018e-08
      4.369513e-10

      Recompress
      40.0
      2.058149e-08
      1.573062e-08
      5.007461e-08
      8.910903e-09

And to finish this section, I'll repeat this process many times to demonstrate that the randomness of the ACA+ does not affect the ability of the SVD recompression to produce almost exactly the same approximation  every time!

```python
def SVD_recompress(U_ACA, V_ACA, eps):
    """
    Recompress an ACA matrix approximation via SVD.

    :param U_ACA: The left-hand approximation matrix.
    :param V_ACA: The right-hand approximation matrix.
    :param eps: The tolerance of the approximation. The convergence condition is
        in terms of the difference in Frobenius norm between the target matrix
        and the approximation.

    :return U_SVD: The SVD recompressed left-hand approximation matrix.
    :return V_SVD: The SVD recompressed right-hand approximation matrix.
    """
    UQ, UR = np.linalg.qr(U_ACA)
    VQ, VR = np.linalg.qr(V_ACA.T)
    W, SIG, Z = np.linalg.svd(UR.dot(VR.T))

    frob_K = np.sqrt(np.cumsum(SIG[::-1] ** 2))[::-1]
    r = np.argmax(frob_K < eps)

    U = UQ.dot(W[:, :r] * SIG[:r])
    V = Z[:r, :].dot(VQ.T)
    return U, V
```

```python
terms = []
l1_mvp_errs = []
frob_errs = []
for i in range(50):
    U, V = ACA_plus(
        block.shape[0],
        block.shape[1],
        lambda Istart, Iend: block[Istart:Iend, :],
        lambda Jstart, Jend: block[:, Jstart:Jend],
        eps / 50.0,
        verbose=False,
    )
    U, V = SVD_recompress(U, V, eps)
    terms.append(U.shape[1])
    y_aca = U.dot(V.dot(x))
    l1_mvp_errs.append(np.mean(np.abs(y_aca - y_true)) / l1_true)
    frob_errs.append(np.sqrt(np.sum((U.dot(V) - block) ** 2)))
```

```python
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.title("Approximation rank")
plt.hist(terms, bins=np.arange(35, 45), color="k")
plt.xlabel("rank of approximation")
plt.subplot(1, 2, 2)
plt.title("Frobenius Error")
plt.hist(
    np.log10(frob_errs),
    bins=np.linspace(np.log10(eps * 0.5), np.log10(eps * 1.5), 11),
    color="k",
)
plt.gca().axvline(np.log10(eps), color="r")
plt.xlabel("$\|E\|_{F})$")
plt.show()
```

I think it's worth stopping for a moment to appreciate how cool math can be. We started from an algorithm that involves both some randomness and a severe lack of information on the matrix being approximated. And from that algorithm, we've built an almost perfectly consistent algorithm for producing a precise approximate matrix. In the SVD recompressed results, there's exactly zero variation in the rank and almost zero variation in the error. And the error is just barely on the correct side of the tolerance!


# 상미분 방정식의 엄밀해
An Exact Solution of an Ordinary Differential Equation

미분 방정식의 한 간단한 예는 다음과 같다.
A simple example of a differential equation is as follows.

$$
\begin{cases}
    \begin{align}
        a_0 \frac{d}{dt}x(t)+a_1 x(t)&=0 \\
        x(0)&=x_0 \\
    \end{align}
\end{cases}
$$

위 미분방정식의 해는 함수 $x(t)$ 이다.

A solution for the differential equation above is the function $x(t)$.

어떤 미분방정식을 푼다는 것은, 해당 미분방정식을 만족하는 함수 $x(t)$를 찾는 것을 뜻한다.

Solving for a differential equation means searching for a function $x(t)$ satisfying the differential equation.

처음에는 어떤 형태의 함수가 해당 미분 방정식을 만족시킬 것인지 잘 모를 수도 있겠지만 몇 가지 함수를 생각해 보자.

At first, we may not be certain about which type of function may satisfy the equation above, but let's consider a few possible candiates.

## 다항식?
A polynomial?

다음과 같은 다항식이 만족시킬 수 있을까?
Is a polynomial as follows suitable?

$$
    x(t) = A_0 t^2 + A_1 t + A_2
$$

미분해 보자
Let's differentiate.

$$
    \frac{d}{dt}x(t) = 2 A_0 t + A_1
$$

각각 계수로 곱해보자.
Let's multiply with respective coefficients.

$$
\begin{cases}
    \begin{align}
        a_0 \frac{d}{dt}x(t) &= a_0 \left(2 A_0 t + A_1 \right)\\
        a_1 x(t)&=a_1 \left( A_0 t^2 + A_1 t + A_2 \right) \\
    \end{align}
\end{cases}
$$

등호의 오른쪽은 오른쪽 끼리, 등호의 왼쪽은 왼쪽 끼리 더해 보자.
Let's add each side respectively.

$$
\begin{align}
    a_1 x(t) + a_0\frac{d}{dt}x(t) &= a_1 \left( A_0 t^2 + A_1 t + A_2 \right) + a_0 \left(2 A_0 t + A_1 \right) \\
    & = a_1 A_0 t^2 + \left(2 a_0 A_0 + a_1 A_1\right) t + \left(a_0 A_1 + a_1 A_2\right)
\end{align}
$$

위에서
Remember

$$
a_0 \frac{d}{dt}x(t)+a_1 x(t)=0
$$

이었으므로 $t$가 어떤 값을 가지든
 thus for all $t$

$$
a_1 A_0 t^2 + \left(2 a_0 A_0 + a_1 A_1\right) t + \left(a_0 A_1 + a_1 A_2\right) = 0
$$

이어야 한다.
 should be satisfied.

이러한 조건이 전혀 만족될 수 없는 것은 아닐 것이나, 일단은 좀 더 일반적으로 유용할 수 있어 보이는 경우를 살펴보도록 하자.

Although this could be satisfied in some way or the other, however, for now let's take a look at a different case that seems more generally useful.

## 지수 함수?
Exponential Function?

아래와 같은 지수함수를 생각해 보자.

Let's think about an exponential function as follows.

$$
    x(t) = A_0 e^{\lambda t}
$$

전과 마찬가지로, 미분하고 계수를 곱해 더해보자.

As before, let's differentiate and add after multiplying coefficients.

$$
    \frac{d}{dt}x(t) = A_0 \lambda e^{\lambda t}
$$

$$
\begin{cases}
    \begin{align}
        a_0 \frac{d}{dt}x(t) &= a_0 \left(A_0 \lambda e^{\lambda t} \right)\\
        a_1 x(t)&=a_1 \left( A_0 e^{\lambda t} \right) \\
    \end{align}
\end{cases}
$$

$$
    a_0 \frac{d}{dt}x(t) + a_1 x(t) = a_0 \left(A_0 \lambda e^{\lambda t} \right) + a_1 \left( A_0 e^{\lambda t} \right) =0
$$

$t$가 어떤 값을 가지든 상관 없이 위 관계는 만족되어야한다.
For all $t$, the relationship above must be satisfied.

$A_0 e^{\lambda t}$ 에 관해 묶어 보자.
Let's factor out $A_0 e^{\lambda t}$.

$$
\begin{align}
a_0 \left(A_0 \lambda e^{\lambda t} \right) + a_1 \left( A_0 e^{\lambda t} \right) &= A_0 e^{\lambda t} a_0 \lambda  + A_0 e^{\lambda t} a_1 \\
&= A_0 e^{\lambda t} \left(a_0 \lambda  + a_1 \right)=0
\end{align}
$$

모든 $t$ 에 대해 위 식이 $0$이 되려면, $A_0=0$ 이거나 $a_0 \lambda + a_1 = 0$ 이어야 한다.

To make the equation above to be $0$ for all $t$, either $A_0=0$ or $a_0 \lambda + a_1 = 0$.

$A_0=0$ 인 경우:
If $A_0=0$:

$$
    x(t)=0 \cdot e^{\lambda t} = 0
$$

$a_0 \lambda + a_1 = 0$ 인 경우:
If $a_0 \lambda + a_1 = 0$:

$$
\begin{align}
    a_0 \lambda + a_1 &= 0 \\
    \lambda &= -\frac{a_1}{a_0} \\
    x(t)&=A_0 e^{\lambda t}=A_0 e^{-\frac{a_1}{a_0} t}=A_0 exp\left(-\frac{a_1}{a_0} t\right)
\end{align}
$$

다항식의 경우와 비교해 보면 지수함수쪽이 조금 더 흥미있어 보일 수 있다.
Comparing with the polynomial, the exponential functions may seem somewhat more interesting.

## 초기 조건
Initial condition

그런데 $A_0$는 어떻게 정하는 것이 좋을까? 처음의 식으로 돌아가 보자.
However, what could be a good way to decide $A_0$? Let's go back to the first equation.

$$
\begin{cases}
    \begin{align}
        a_0 \frac{d}{dt}x(t)+a_1 x(t)&=0 \\
        x(0)&=x_0 \\
    \end{align}
\end{cases}
$$

따라서 $x(0)=x_0$ 였음을 이용해 보자. 여기서 $x_0$는 어떤 알려진 상수이다.
Thus let's use $x(0)=x_0$. Here, $x_0$ is a known constant.

$$
\begin{align}
    x(t)&=A_0 e^{-\frac{a_1}{a_0} t} \\
    x(0)&=A_0 e^{-\frac{a_1}{a_0}\cdot 0}=A_0 e^0=A_0 \cdot 1 = A_0=x_0 \\
    x(t)&=x_0 e^{-\frac{a_1}{a_0} t}
\end{align}
$$

이렇게 어떤 미분방정식의 엄밀해를 구할 수 있었다.
This way, we could find an exact solution of a differential equation.

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```