<center>

## [mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course

Author: [Alexey Natekin](https://www.linkedin.com/in/natekin/), OpenDataScience founder, Machine Learning Evangelist. Translated and edited by [Olga Daykhovskaya](https://www.linkedin.com/in/odaykhovskaya/), [Anastasia Manokhina](https://www.linkedin.com/in/anastasiamanokhina/), [Egor Polusmak](https://www.linkedin.com/in/egor-polusmak/), and [Yuanyuan Pao](https://www.linkedin.com/in/yuanyuanpao/). This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. Free use is permitted for any non-commercial purpose.

# <center> Topic 10. Gradient Boosting

So far, we've covered 9 topics from Exploratory Data Analysis to Time Series Analysis in Python:
<spoiler title="List of the series' articles">
1. [Exploratory Data Analysis with Pandas](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-1-exploratory-data-analysis-with-pandas-de57880f1a68)
2. [Visual data analysis with Python](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-2-visual-data-analysis-in-python-846b989675cd)
3. [Classification, Decision Trees and k Nearest Neighbors](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-3-classification-decision-trees-and-k-nearest-neighbors-8613c6b6d2cd)
4. [Linear Classification and Regression](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-4-linear-classification-and-regression-44a41b9b5220)
5. [Bagging and Random Forest](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-5-ensembles-of-algorithms-and-random-forest-8e05246cbba7)
6. [Feature Engineering and Feature Selection](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-6-feature-engineering-and-feature-selection-8b94f870706a)
7. [Unsupervised Learning: Principal Component Analysis and Clustering](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-7-unsupervised-learning-pca-and-clustering-db7879568417)
8. [Vowpal Wabbit: Learning with Gigabytes of Data](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-8-vowpal-wabbit-fast-learning-with-gigabytes-of-data-60f750086237)
9. [Time Series Analysis in Python](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-time-series-analysis-in-python-a270cb05e0b3)

Today we are going to have a look at one of the most popular and practical machine learning algorithms: gradient boosting.

###  Article Outline
We recommend going over this article in the order described below, but feel free to jump around between sections.

1. [Introduction and history of boosting](#1.--Introduction-and-history-of-boosting)
   - [History of Gradient Boosting Machine](#History-of-GBM)
1. [GBM algorithm](#2.-GBM-algorithm)
   - [ML Problem statement](#ML-problem-statement)
   - [Functional gradient descent](#Functional-gradient-descent)
   - [Friedman's classic GBM algorithm](#Friedman's-classic-GBM-algorithm)
   - [Step-by-step example of the GBM algorithm](#Step-By-Step-example:-How-GBM-Works)
1. [Loss functions](#3.-Loss-functions)
   - [Regression loss functions](#Regression-loss-functions)
   - [Classification loss functions](#Classification-loss-functions)
   - [Weights](#Weights)
1. [Conclusion](#4.-Conclusion)
1. [Demo assignment](#5.-Demo-assignment)
1. [Useful resources](#6-Useful-resources)

# 1.  Introduction and history of boosting
Almost everyone in machine learning has heard about gradient boosting. Many data scientists include this algorithm in their data scientist's toolbox because of the good results it yields on any given (unknown) problem.

Furthermore, XGBoost is often the standard recipe for [winning](https://github.com/dmlc/xgboost/blob/master/demo/README.md#usecases) [ML competitions](http://blog.kaggle.com/tag/xgboost/). It is so popular that the idea of stacking XGBoosts has become a meme. Moreover, boosting is an important component in [many recommender systems](https://en.wikipedia.org/wiki/Learning_to_rank#Practical_usage_by_search_engines); sometimes, it is even considered a [brand](https://yandex.com/company/technologies/matrixnet/).
Let's look at the history and development of boosting.

Boosting was born out of [the question:](http://www.cis.upenn.edu/~mkearns/papers/boostnote.pdf) is it possible to get one strong model from a large amount of relatively weak and simple models? By saying "weak models", we do not mean simple basic models like decision trees but models with poor accuracy performance, where poor is a little bit better than random.

[A positive mathematical answer](http://www.cs.princeton.edu/~schapire/papers/strengthofweak.pdf) to this question was identified, but it took a few years to develop fully functioning algorithms based on this solution e.g. AdaBoost. These algoritms take a greedy approach: first, they build a linear combination of simple models (basic algorithms) by re-weighing the input data. Then, the model (usually a decision tree) is built on earlier incorrectly predicted objects, which are now given larger weights.

<spoiler title="More about AdaBoost">
Many machine learning courses study AdaBoost - the ancestor of GBM (Gradient Boosting Machine). However, since AdaBoost merged with GBM, it has become apparent that AdaBoost is just a particular variation of GBM.

The algorithm itself has a very clear visual interpretation and intuition for defining weights. Let's have a look at the following toy classification problem where we are going to split the data between the trees of depth 1 (also known as 'stumps') on each iteration of AdaBoost. For the first two iterations, we have the following picture:

The size of point corresponds to its weight, which was assigned for an incorrect prediction.  On each iteration, we can see that these weights are growing -- the stumps cannot cope with this problem. Although, if we take a weighted vote for the stumps, we will get the correct classifications:

Pseudocode:
- Initialize sample weights $\Large w_i^{(0)} = \frac{1}{l}, i = 1, \dots, l$.
- For all $t = 1, \dots, T$
    * Train base algo $\Large b_t$, let $\epsilon_t$ be it's training error.
    * $\Large \alpha_t = \frac{1}{2}ln\frac{1 - \epsilon_t}{\epsilon_t}$.
    * Update sample weights: $\Large w_i^{(t)} = w_i^{(t-1)} e^{-\alpha_t y_i b_t(x_i)}, i = 1, \dots, l$.
    * Normalize sample weights: $\Large w_0^{(t)} = \sum_{j = 1}^l w_j^{(t)}, w_i^{(t)} = \frac{w_i^{(t)}}{w_0^{(t)}}, i = 1, \dots, l$.
- Return $\sum_t^{T}\alpha_tb_t$

[Here](https://www.youtube.com/watch?v=k4G2VCuOMMg) is more detailed example of AdaBoost where, as we iterate, we can see the weights increase, especially on the border between classes.

AdaBoost works well, but [the lack](https://www.cs.princeton.edu/courses/archive/spring07/cos424/papers/boosting-survey.pdf) of explanation for why the algorithm is successful sewed the seeds of doubt. Some considered it a super-algorithm, a silver bullet, but others were skeptical and believed AdaBoost was just overfitting.

The overfitting problem did indeed exist, especially when data had strong outliers. Therefore, in those types of problems, AdaBoost was unstable. Fortunately, a few professors in the statistics department at Stanford, who had created Lasso, Elastic Net, and Random Forest, started researching the algorithm. In 1999, Jerome Friedman came up with the generalization of boosting algorithms development - Gradient Boosting (Machine), also known as GBM. With this work, Friedman set up the statistical foundation for many algorithms providing the general approach of boosting for optimization in the functional space.

CART, bootstrap, and many other algorithms have originated from Stanford's statistics department. In doing so, the department has solidified their names in future textbooks. These algorithms are very practical, and some recent works have yet to be widely adopted. For example, check out [glinternet](https://arxiv.org/abs/1308.2719).

Not many video recordings of Friedman are available. Although, there is a very interesting [interview](https://www.youtube.com/watch?v=8hupHmBVvb0) with him about the creation of CART and how they solved statistics problems (which is similar to data analysis and data science today) more than 40 years ago.

There is also a great [lecture](https://www.youtube.com/watch?v=zBk3PK3g-Fc) from Hastie, a retrospective on data analysis from one of the creators of methods that we use everyday.

In general, there has been a transition from engineering and algorithmic research to a full-fledged approach to building and studying algorithms. From a mathematical perspective, this is not a big change - we are still adding (or boosting) weak algorithms and enlarging our ensemble with gradual improvements for parts of the data where the model was inaccurate. But, this time, the next simple model is not just built on re-weighted objects but improves its approximation of the gradient of overall objective function. This concept greatly opens up our algorithms for imagination and extensions.

## History of GBM

It took more than 10 years after the introduction of GBM for it to become an essential part of the data science toolbox.
GBM was extended to apply to different statistics problems: GLMboost and GAMboost for strengthening already existing GAM models, CoxBoost for survival curves, and RankBoost and LambdaMART for ranking.
Many realizations of GBM also appeared under different names and on different platforms: Stochastic GBM, GBDT (Gradient Boosted Decision Trees), GBRT (Gradient Boosted Regression Trees), MART (Multiple Additive Regression Trees), and more. In addition, the ML community was very segmented and dissociated, which made it hard to track just how widespread boosting had become.

At the same time, boosting had been actively used in search ranking. This problem was rewritten in terms of a loss function that penalizes errors in the output order, so it became convenient to simply insert it into GBM. AltaVista was one of the first companies who introduced boosting to ranking. Soon, the ideas spread to Yahoo, Yandex, Bing, etc. Once this happened, boosting became one of the main algorithms that was used not only in research but also in core technologies in industry.

 ML competitions, especially Kaggle, played a major role in boosting's popularization. Now, researchers had a common platform where they could compete in different data science problems with large number of participants from around the world. With Kaggle, one could test new algorithms on the real data, giving algoritms oppurtunity to "shine", and provide full information in sharing model performance results across competition data sets. This is exactly what happened to boosting when it was used at [Kaggle](http://blog.kaggle.com/2011/12/21/score-xavier-conort-on-coming-second-in-give-me-some-credit/) (check interviews with Kaggle winners starting from 2011 who mostly used boosting). The [XGBoost](https://github.com/dmlc/xgboost) library quickly gained popularity after its appearance. XGBoost is not a new, unique algorithm; it is just an extremely effective realization of classic GBM with additional heuristics.

This algorithm has gone through very typical path for ML algorithms today: mathematical problem and algorithmic crafts to successful practical applications and mass adoption years after its first appearance.

# 2. GBM algorithm
### ML problem statement

We are going to solve the problem of function approximation in a general supervised learning setting. We have a set of features $ \large x $ and target variables $\large y, \large \left\{ (x_i, y_i) \right\}_{i=1, \ldots,n}$ which we use to restore the dependence $\large y = f(x) $. We restore the dependence by approximating $ \large \hat{f}(x) $ and by understanding which approximation is better when we use the loss function $ \large L(y,f) $, which we want to minimize: $ \large y \approx \hat{f}(x), \large \hat{f}(x) = \underset{f(x)}{\arg\min} \ L(y,f(x)) $.

At this moment, we do not make any assumptions regarding the type of dependence $ \large f(x) $, the model of our approximation $ \large \hat{f}(x) $, or the distribution of the target variable $ \large y $. We only expect the function $ \large L(y,f) $ to be differentiable. Our approach is very general: we define $ \large \hat {f}(x) $ by minimizing the loss:
$$ \large  \hat{f}(x) = \underset{f(x)}{\arg\min} \ \mathbb {E} _{x,y}[L(y,f(x))]  $$

Unfortunately, the number of functions $ \large f(x) $ is not just large, but its functional space is infinite-dimensional. That is why it is acceptable for us to limit the search space by some family of functions $ \large f(x, \theta), \theta \in \mathbb{R}^d $. This simplifies the objective a lot because now we have a solvable optimization of parameter values:
$ \large \hat{f}(x) = f(x, \hat{\theta}),$
$$\large \hat{\theta} = \underset{\theta}{\arg\min} \ \mathbb {E} _{x,y}[L(y,f(x,\theta))] $$

Simple analytical solutions for finding the optimal parameters $ \large \hat{\theta} $ often do not exist, so the parameters are usually approximated iteratively. To start, we write down the empirical loss function $ \large L_{\theta}(\hat{\theta}) $ that will allow us to evaluate our parameters using our data. Additionally, let's write out our approximation $ \large \hat{\theta} $ for a number of $ \large M $ iterations as a sum:
$ \large \hat{\theta} = \sum_{i = 1}^M \hat{\theta_i}, \\
\large L_{\theta}(\hat{\theta}) =  \sum_{i = 1}^N L(y_i,f(x_i, \hat{\theta}))$

Then, the only thing left is to find a suitable, iterative algorithm to minimize $\large L_{\theta}(\hat{\theta})$. Gradient descent is the simplest and most frequently used option. We define the gradient as $\large \nabla L_{\theta}(\hat{\theta})$ and add our iterative evaluations $\large \hat{\theta_i}$ to it (since we are minimizing the loss, we add the minus sign). Our last step is to initialize our first approximation $\large \hat{\theta_0}$ and choose the number of iterations $\large M$. Let's review the steps for this inefficient and naive algorithm for approximating $\large \hat{\theta}$:

1. Define the initial approximation of the parameters $\large \hat{\theta} = \hat{\theta_0}$
2. For every iteration $\large t = 1, \dots, M$ repeat steps 3-7:
1. Calculate the gradient of the loss function $\large \nabla L_{\theta}(\hat{\theta})$ for the current approximation $\large \hat{\theta}$
$\large \nabla L_{\theta}(\hat{\theta}) = \left[\frac{\partial L(y, f(x, \theta))}{\partial \theta}\right]_{\theta = \hat{\theta}}$
2. Set the current iterative approximation $\large \hat{\theta_t}$ based on the calculated gradient
$\large \hat{\theta_t} \leftarrow −\nabla L_{\theta}(\hat{\theta})$
3. Update the approximation of the parameters $\large \hat{\theta}$:
$\large \hat{\theta} \leftarrow \hat{\theta} + \hat{\theta_t} = \sum_{i = 0}^t \hat{\theta_i} $
3. Save the result of approximation $\large \hat{\theta}$:
$\large \hat{\theta} = \sum_{i = 0}^M \hat{\theta_i} $
4. Use the function that was found $\large \hat{f}(x) = f(x, \hat{\theta})$

### Functional gradient descent

Let's imagine for a second that we can perform optimization in the function space and iteratively search for the approximations $\large \hat{f}(x)$ as functions themselves. We will express our approximation as a sum of incremental improvements, each being a function. For convenience, we will immediately start with the sum from the initial approximation $\large \hat{f_0}(x)$:
$$\large \hat{f}(x) = \sum_{i = 0}^M \hat{f_i}(x)$$

Nothing has happened yet; we have only decided that we will search for our approximation $\large \hat{f}(x)$ not as a big model with plenty of parameters (as an example, neural network), but as a sum of functions, pretending we move in functional space.

In order to accomplish this task, we need to limit our search by some function family $\large \hat{f}(x) = h(x, \theta)$. There are a few issues here -- first of all, the sum of models can be more complicated than any model from this family; secondly, the general objective is still in functional space. Let's note that, on every step, we will need to select an optimal coefficient $\large \rho \in \mathbb{R}$. For step $\large t$, the problem is the following:
$$\large \hat{f}(x) = \sum_{i = 0}^{t-1} \hat{f_i}(x), \\
\large (\rho_t,\theta_t) = \underset{\rho,\theta}{\arg\min} \ \mathbb {E} _{x,y}[L(y,\hat{f}(x) +  \rho \cdot h(x, \theta))], \\
\large \hat{f_t}(x) = \rho_t \cdot h(x, \theta_t)$$

Here is where the magic happens. We have defined all of our objectives in general terms, as if we could have trained any kind of model $\large h(x, \theta)$ for any type of loss functions $\large L(y, f(x, \theta))$. In practice, this is extremely difficult, but, fortunately, there is a simple way to solve this task.

Knowing the expression of loss function's gradient, we can calculate its value on our data. So, let's train the models such that our predictions will be more correlated with this gradient (with a minus sign). In other words, we will use least squares to correct the predictions with these residuals. For classification, regression, and ranking tasks, we will minimize the squared difference between pseudo-residuals $\large r$ and our predictions. For step $\large t$, the final problem looks like the following:
$$ \large \hat{f}(x) = \sum_{i = 0}^{t-1} \hat{f_i}(x), \\
\large r_{it} = -\left[\frac{\partial L(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x)=\hat{f}(x)}, \quad \mbox{for } i=1,\ldots,n ,\\
\large \theta_t = \underset{\theta}{\arg\min} \ \sum_{i = 1}^{n} (r_{it} - h(x_i, \theta))^2, \\
\large \rho_t = \underset{\rho}{\arg\min} \ \sum_{i = 1}^{n} L(y_i, \hat{f}(x_i) + \rho \cdot h(x_i, \theta_t))$$

### Friedman's classic GBM algorithm

We can now define the classic GBM algorithm suggested by Jerome Friedman in 1999. It is a supervised algorithm that has the following components:

- dataset $\large \left\{ (x_i, y_i) \right\}_{i=1, \ldots,n}$;
- number of iterations $\large M$;
- choice of loss function $\large L(y, f)$ with a defined gradient;
- choice of function family of base algorithms $\large h(x, \theta)$ with the training procedure;
- additional hyperparameters $\large h(x, \theta)$ (for example, in decision trees, the tree depth);

The only thing left is the initial approximation $\large f_0(x)$. For simplicity, for an initial approximation, a constant value $\large \gamma$ is used. The constant value, as well as the optimal coefficient $\large \rho $, are identified via binary search or another line search algorithm over the initial loss function (not a gradient). So, we have our GBM algorithm described as follows:

1. Initialize GBM with constant value $\large \hat{f}(x) = \hat{f}_0, \hat{f}_0 = \gamma,  \gamma \in \mathbb{R}$
$\large \hat{f}_0 = \underset{\gamma}{\arg\min} \ \sum_{i = 1}^{n} L(y_i, \gamma)$
2. For each iteration $\large t = 1, \dots, M$, repeat:
1. Calculate pseudo-residuals $\large r_t$
$\large r_{it} = -\left[\frac{\partial L(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x)=\hat{f}(x)}, \quad \mbox{for } i=1,\ldots,n$
2. Build new base algorithm $\large h_t(x)$ as regression on pseudo-residuals $\large \left\{ (x_i, r_{it}) \right\}_{i=1, \ldots,n}$
3. Find optimal coefficient $\large \rho_t $ at $\large h_t(x)$ regarding initial loss function
$\large \rho_t = \underset{\rho}{\arg\min} \ \sum_{i = 1}^{n} L(y_i, \hat{f}(x_i) +  \rho \cdot h(x_i, \theta))$
4. Save $\large \hat{f_t}(x) = \rho_t \cdot h_t(x)$
5. Update current approximation $\large \hat{f}(x)$
$\large \hat{f}(x) \leftarrow \hat{f}(x) + \hat{f_t}(x) = \sum_{i = 0}^{t} \hat{f_i}(x)$
3. Compose final GBM model $\large \hat{f}(x)$
$\large \hat{f}(x) = \sum_{i = 0}^M \hat{f_i}(x) $
4. Conquer Kaggle and the rest of the world

### Step-By-Step example: How GBM Works

Let's see an example of how GBM works. In this toy example, we will restore a noisy function $\large y = cos(x) + \epsilon, \epsilon \sim \mathcal{N}(0, \frac{1}{5}), x \in [-5,5]$.

This is a regression problem with a real-valued target, so we will choose to use the mean squared error loss function. We will generate 300 pairs of observations and approximate them with decision trees of depth 2. Let's put together everything we need to use GBM:
- Toy data $\large \left\{ (x_i, y_i) \right\}_{i=1, \ldots,300}$ ✓
- Number of iterations $\large M = 3$ ✓;
- The mean squared error loss function $\large L(y, f) = (y-f)^2$ ✓
- Gradient of $\large L(y, f) = L_2$ loss is just residuals $\large r = (y - f)$ ✓;
- Decision trees as base algorithms $\large h(x)$ ✓;
- Hyperparameters of the decision trees: trees depth is equal to 2 ✓;

For the mean squared error, both initialization $\large \gamma$ and coefficients $\large \rho_t$ are simple. We will initialize GBM with the average value $\large \gamma = \frac{1}{n} \cdot \sum_{i = 1}^n y_i$, and set all coefficients $\large \rho_t$ to 1.

We will run GBM and draw two types of graphs: the current approximation $\large \hat{f}(x)$ (blue graph) and every tree $\large \hat{f_t}(x)$ built on its pseudo-residuals (green graph). The graph's number corresponds to the iteration number:

By the second iteration, our trees have recovered the basic form of the function. However, at the first iteration, we see that the algorithm has built only the "left branch" of the function ($\large x \in [-5, -4]$). This was due to the fact that our trees simply did not have enough depth to build a symmetrical branch at once, and it focused on the left branch with the larger error. Therefore, the right branch appeared only after the second iteration.

The rest of the process goes as expected -- on every step, our pseudo-residuals decreased, and GBM approximated the original function better and better with each iteration. However, by construction, trees cannot approximate a continuous function, which means that GBM is not ideal in this example. To play with GBM function approximations, you can use the awesome interactive demo in this blog called [Brilliantly wrong](http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html):

# 3. Loss functions

If we want to solve a classification problem instead of regression, what would change? We only need to choose a suitable loss function $\large L(y, f)$. This is the most important, high-level moment that determines exactly how we will optimize and what characteristics we can expect in the final model.

As a rule, we do not need to invent this ourselves – researchers have already done it for us. Today, we will explore loss functions for the two most common objectives: regression $\large y \in \mathbb{R}$ and binary classification $\large y \in \left\{-1, 1\right\}$.

### Regression loss functions

Let's start with a regression problem for $\large y \in \mathbb{R}$. In order to choose the appropriate loss function, we need to consider which of the properties of the conditional distribution $\large (y|x)$ we want to restore. The most common options are:

- $\large L(y, f) = (y - f)^2$ a.k.a. $\large L_2$ loss or Gaussian loss. It is the classical conditional mean, which is the simplest and most common case. If we do not have any additional information or requirements for a model to be robust, we can use the Gaussian loss.
- $\large L(y, f) = |y - f|$ a.k.a. $\large L_1$ loss or Laplacian loss. At the first glance, this function does not seem to be differentiable, but it actually defines the conditional median. Median, as we know, is robust to outliers, which is why this loss function is better in some cases. The penalty for big variations is not as heavy as it is in $\large L_2$.
- $ \large \begin{equation}  L(y, f) =\left\{   \begin{array}{@{}ll@{}}     (1 - \alpha) \cdot |y - f|, & \text{if}\ y-f \leq 0 \\     \alpha \cdot |y - f|, & \text{if}\ y-f >0  \end{array}\right. \end{equation}, \alpha \in (0,1)
$ a.k.a. $\large L_q$ loss or Quantile loss.  Instead of median, it uses quantiles. For example, $\large \alpha = 0.75$ corresponds to the 75%-quantile. We can see that this function is asymmetric and penalizes the observations which are on the right side of the defined quantile.

Let's use loss function $\large L_q$ on our data. The goal is to restore the conditional 75%-quantile of cosine. Let us put everyting together for GBM:
- Toy data $\large \left\{ (x_i, y_i) \right\}_{i=1, \ldots,300}$ ✓
- A number of iterations $\large M = 3$ ✓;
- Loss function for quantiles $ \large \begin{equation}   L_{0.75}(y, f) =\left\{
\begin{array}{@{}ll@{}}    0.25 \cdot |y - f|, & \text{if}\ y-f \leq 0 \\     0.75 \cdot |y - f|, & \text{if}\ y-f >0   \end{array}\right. \end{equation} $ ✓;
- Gradient $\large L_{0.75}(y, f)$ - function weighted by $\large \alpha = 0.75$. We are going to train tree-based model for classification:
$\large r_{i} = -\left[\frac{\partial L(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x)=\hat{f}(x)} = $
$\large = \alpha I(y_i > \hat{f}(x_i) ) - (1 - \alpha)I(y_i \leq \hat{f}(x_i) ), \quad \mbox{for } i=1,\ldots,300$ ✓;
- Decision tree as a basic algorithm $\large h(x)$ ✓;
- Hyperparameter of trees: depth =  2 ✓;

For our initial approximation, we will take the needed quantile of $\large y$. However, we do not know anything about optimal coefficients $\large \rho_t$, so we'll use standard line search. The results are the following:

We can observe that, on each iteration, $\large r_{i} $ take only 2 possible values, but GBM is still able to restore our initial function.

The overall results of GBM with quantile loss function are the same as the results with quadratic loss function offset by $\large \approx 0.135$. But if we were to use the 90%-quantile, we would not have enough data due to the fact that classes would become unbalanced. We need to remember this when we deal with non-standard problems.

*"A few words on regression loss functions"*

For regression tasks, many loss functions have been developed, some of them with extra properties. For example, they can be robust like in the [Huber loss function](https://en.wikipedia.org/wiki/Huber_loss). For a small number of outliers, the loss function works as $\large L_2$, but after a defined threshold, the function changes to $\large L_1$. This allows for decreasing the effect of outliers and focusing on the overall picture.

We can illustrate this with the following example. Data is generated from the function  $\large y = \frac{sin(x)}{x}$ with added noise, a mixture from normal and Bernulli distributions. We show the functions on graphs A-D and the relevant GBM on F-H (graph E represents the initial function):

 [Original size](https://habrastorage.org/web/130/05b/222/13005b222e8a4eb68c3936216c05e276.jpg).

In this example, we used splines as the base algorithm. See, it does not always have to be trees for boosting?

We can clearly see the difference between the functions $\large L_2$, $\large L_1$, and Huber loss. If we choose optimal parameters for the Huber loss, we can get the best possible approximation among all our options. The difference can be seen as well in the 10%, 50%, and 90%-quantiles.

Unfortunately, Huber loss function is supported only by very few popular libraries/packages; h2o supports it, but XGBoost does not. It is relevant to other things that are more exotic like [conditional expectiles](https://www.slideshare.net/charthur/quantile-and-expectile-regression), but it may still be interesting knowledge.

### Classification loss functions

Now, let's look at the binary classification problem $\large y \in \left\{-1, 1\right\}$. We saw that GBM can even optimize non-differentiable loss functions. Technically, it is possible to solve this problem with a regression $\large L_2$ loss, but it wouldn't be correct.

The distribution of the target variable requires us to use log-likehood, so we need to have different loss functions for targets multiplied by their predictions:  $\large y \cdot f$. The most common choices would be the following:

- $\large L(y, f) = log(1 + exp(-2yf))$ a.k.a. Logistic loss or Bernoulli loss. This has an interesting property that penalizes even correctly predicted classes, which helps not only helps to optimize loss but also to move the classes apart further, even if all classes are predicted correctly.
- $\large L(y, f) = exp(-yf)$ a.k.a. AdaBoost loss. The classic AdaBoost is equivalent to GBM with this loss function. Conceptually, this function is very similar to logistic loss, but it has a bigger exponential penalization if the prediction is wrong.

Let's generate some new toy data for our classification problem. As a basis, we will take our noisy cosine, and we will use the sign function for classes of the target variable. Our toy data looks like the following (jitter-noise is added for clarity):

We will use logistic loss to look for what we actually boost. So, again, we put together what we will use for GBM:
- Toy data $\large \left\{ (x_i, y_i) \right\}_{i=1, \ldots,300}, y_i \in \left\{-1, 1\right\}$ ✓
- Number of iterations $\large M = 3$ ✓;
- Logistic loss as the loss function, its gradient is computed the following way:
$\large r_{i} = \frac{2 \cdot y_i}{1 + exp(2 \cdot y_i \cdot \hat{f}(x_i)) }, \quad \mbox{for } i=1,\ldots,300$ ✓;
- Decision trees as base algorithms $\large h(x)$ ✓;
- Hyperparameters of the decision trees: tree's depth is equal to 2 ✓;

This time, the initialization of the algorithm is a little bit harder. First, our classes are imbalanced  (63% versus 37%). Second, there is no known analytical formula for the initialization of our loss function, so we have to look for $\large \hat{f_0} = \gamma$ via search:

Our optimal initial approximation is around -0.273. You could have guessed that it was negative because it is more profitable to predict everything as the most popular class, but there is no formula for the exact value. Now let's finally start GBM, and look what actually happens under the hood:

The algorithm successfully restored the separation between our classes. You can see how the "lower" areas are separating because the trees are more confident in the correct prediction of the negative class and how the two steps of mixed classes are forming. It is clear that we have a lot of correctly classified observations and some amount of observations with large errors that appeared due to the noise in the data.

### Weights

Sometimes, there is a situation where we want a more specific loss function for our problem. For example, in financial time series, we may want to give bigger weight to large movements in the time series; for churn prediction, it is more useful to predict the churn of clients with high LTV (or lifetime value: how much money a client will bring in the future).

The statistical warrior would invent their own loss function, write out the gradient for it (for more effective training, include the Hessian), and carefully check whether this function satisfies the required properties. However, there is a high probability of making a mistake somewhere, running up against computational difficulties, and spending an inordinate amount of time on research.

In lieu of this, a very simple instrument was invented (which is rarely remembered in practice): weighing observations and assigning weight functions. The simplest example of such weighting is the setting of weights for class balance. In general, if we know that some subset of data, both in the input variables $\large x$ and in the target variable $\large y$, has greater importance for our model, then we just assign them a larger weight $\large w(x,y)$. The main goal is to fulfill the general requirements for weights:
$$ \large w_i \in \mathbb{R}, \\
\large w_i \geq 0 \quad \mbox{for } i=1,\ldots,n, \\
\large \sum_{i = 1}^n w_i > 0 $$

Weights can significantly reduce the time spent adjusting the loss function for the task we are solving and also encourages experiments with the target models' properties. Assigning these weights is entirely a function of creativity. We simply add scalar weights:
$$ \large L_{w}(y,f) = w \cdot L(y,f), \\
\large r_{it} =   - w_i \cdot \left[\frac{\partial L(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x)=\hat{f}(x)}, \quad \mbox{for } i=1,\ldots,n$$

It is clear that, for arbitrary weights, we do not know the statistical properties of our model. Often, linking the weights to the values $\large y$ can be too complicated. For example, the usage of weights proportional to $\large |y|$ in $\large L_1$ loss function is not equivalent to $\large L_2$ loss because the gradient will not take into account the values of the predictions themselves: $\large \hat{f}(x)$.

We mention all of this so that we can understand our possibilities better. Let's create some very exotic weights for our toy data. We will define a strongly asymmetric weight function as follows:
$$ \large \begin{equation} w(x) =\left\{   \begin{array}{@{}ll@{}}     0.1, & \text{if}\ x \leq 0 \\     0.1 + |cos(x)|, & \text{if}\ x >0 \end{array}\right. \end{equation} $$

With these weights, we expect to get two properties: less detailing for negative values of $\large x$ and the form of the function, similar to the initial cosine. We take the other GBM's tunings from our previous example with classification including the line search for optimal coefficients. Let's look what we've got:

We achieved the result that we expected. First, we can see how strongly the pseudo-residuals differ; on the initial iteration, they look almost like the original cosine. Second, the left part of the function's graph was often ignored in favor of the right one, which had larger weights. Third, the function that we got on the third iteration received enough attention and started looking similar to the original cosine (also started to slightly overfit).

Weights are a powerful but risky tool that we can use to control the properties of our model. If you want to optimize your loss function, it is worth trying to solve a more simple problem first but add weights to the observations at your discretion.

# 4. Conclusion

Today, we learned the theory behind gradient boosting. GBM is not just some specific algorithm but a common methodology for building ensembles of models. In addition, this methodology is sufficiently flexible and expandable -- it is possible to train a large number of models, taking into consideration different loss-functions with a variety of weighting functions.

Practice and ML competitions show that, in standard problems (except for image, audio, and very sparse data), GBM is often the most effective algorithm (not to mention stacking and high-level ensembles, where GBM is almost always a part of them).  Also, there are many adaptations of GBM [for Reinforcement Learning](https://arxiv.org/abs/1603.04119) (Minecraft, ICML 2016). By the way, the Viola-Jones algorithm, which is still used in computer vision, [is based on AdaBoost](https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework#Learning_algorithm).

In this article, we intentionally omitted questions concerning GBM’s regularization, stochasticity, and hyper-parameters. It was not accidental that we used a small number of iterations $\large M = 3$ throughout. If we used 30 trees instead of 3 and trained the GBM as described, the result would not be that predictable:

[Interactive demo](http://arogozhnikov.github.io/2016/07/05/gradient_boosting_playground.html)

# 5. Demo assignment
Your task is to beat benchmarks in the "Flight delays" Kaggle Inclass [competition](https://www.kaggle.com/c/flight-delays-fall-2018). You are given a [CatBoost starter](https://www.kaggle.com/kashnitsky/mlcourse-ai-fall-2019-catboost-starter), the trick is to come up with some good features.

# 6. Useful resources
- Main course [site](https://mlcourse.ai), [course repo](https://github.com/Yorko/mlcourse.ai), and YouTube [channel](https://www.youtube.com/watch?v=QKTuw4PNOsU&list=PLVlY_7IJCMJeRfZ68eVfEcu-UcN9BbwiX)
- Course materials as a [Kaggle Dataset](https://www.kaggle.com/kashnitsky/mlcourse)
- mlcourse.ai lectures on gradient boosting: [theory](https://youtu.be/g0ZOtzZqdqk) and [practice](https://youtu.be/V5158Oug4W8)
- [Original article](https://statweb.stanford.edu/~jhf/ftp/trebst.pdf) about GBM from Jerome Friedman
- “Gradient boosting machines, a tutorial”, [paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/) by Alexey Natekin, and Alois Knoll
- [Chapter in Elements of Statistical Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf) from Hastie, Tibshirani, Friedman (page 337)
- [Wiki](https://en.wikipedia.org/wiki/Gradient_boosting) article about Gradient Boosting
- [Introduction to boosted trees (Xgboost docs)](https://xgboost.readthedocs.io/en/latest/tutorials/model.html)
- [Video-lecture by Hastie](https://www.youtube.com/watch?v=wPqtzj5VZus) about GBM at h2o.ai conference
- [CatBoost vs. Light GBM vs. XGBoost](https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db) on "Towards Data Science"
- [Benchmarking and Optimization of
Gradient Boosting Decision Tree Algorithms](https://arxiv.org/abs/1809.04559), [XGBoost: Scalable GPU Accelerated Learning](https://arxiv.org/abs/1806.11248) - benchmarking CatBoost, Light GBM, and XGBoost (no 100% winner)


# Lead-Acid Models

We compare a standard porous-electrode model for lead-acid batteries with two asymptotic reductions. For a more in-depth introduction to PyBaMM models, see the [SPM notebook](./SPM.ipynb). Further details on the models can be found in [[1]](#ref)

[1] Sulzer, V., Chapman, S. J., Please, C. P., Howey, D. A., & Monroe, C. W. (2019). “Faster Lead-Acid Battery Simulations from Porous Electrode Theory:  II. Asymptotic Analysis”. Journal of The Electrochemical Society, 166(12), A2372-A2382.

```python
%pip install pybamm -q    # install PyBaMM if it is not installed
import pybamm
import numpy as np
import os
import matplotlib.pyplot as plt
os.chdir(pybamm.__path__[0]+'/..')
```

    Note: you may need to restart the kernel to use updated packages.

### "Full" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Electrolyte Current

$$
    \frac{\partial i_{\text{e}}}{\partial x} = j, \\
    \mathcal{C}_\text{e} i_{\text{e}} = \epsilon_k^b \kappa(c)
        \left( \chi \frac{\partial}{\partial x}\log(c)
        - \frac{\partial\phi_{\text{e}}}{\partial x}\right)\\
    i_{\text{e}}\big|_{x=0}= i_{\text{e}}\big|_{x=1}=0,
$$

#### Electrode Current

$$
    \frac{\partial i_{\text{s}}}{\partial x} = -j,\\
    i_{\text{s}} = -\sigma\frac{\partial\phi_{\text{s}}}{\partial x},\\
    \phi_{\text{s}}\big|_{x=0}
        = i_{\text{s}}\big|_{x=l_\text{n}}
        = i_{\text{s}}\big|_{x=1-l_\text{p}} = 0, \\
    i_{\text{s}}\big|_{x=1}=\mathcal{I},
$$

#### interfacial current density

$$
j = \begin{cases}
    2j_0(c) \sinh\left(\eta\right), \quad &0 < x < l_\text{n} \\
    0, \quad &l_\text{n} < x < 1-l_\text{p} \\
    2j_0(c) \sinh\left(\eta\right), \quad &1-l_\text{p} < x < 1
\end{cases}
\\
\eta = \phi_{\text{s}} - \phi_\text{e} - U(c),
$$

This model is implemented in PyBaMM as the `Full` model

```python
full = pybamm.lead_acid.Full()
```

### "Leading-order" model

$$
    \frac{\mathrm{d} }{\mathrm{d} t}\left(\epsilon c\right) = (s_\text{n} - s_\text{p})\mathrm{I}, \\
    \frac{\mathrm{d} \epsilon}{\mathrm{d} t} = -\beta^\text{surf}j, \\
    j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
    \phi_\text{e} = -U_\text{n}(c) + \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{n}j_{0\text{n}}(c)}\right) \\
    V = -\phi_\text{e} + U_\text{p}(c) - \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{p}j_{0\text{p}}(c)}\right) \\
$$

This model is implemented in PyBaMM as `LOQS` (leading-order quasi-static)

```python
loqs = pybamm.lead_acid.LOQS()
```

### "Composite" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Interfacial current density

$$
j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
$$

#### Voltage

$$
V = U_{\text{eq}} + \eta_r + \eta_c + \Delta\Phi_{\text{Elec}} + \Delta\Phi_{\text{Solid}}
$$
where
\begin{align}
    & U_{\text{eq}} = U_p(\bar{c}_p) - U_n(\bar{c}_n), \\
     &\eta_{r} = -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0n} l_n}\right)
     -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0p} l_p}\right), \\
     &\eta_c =  \mathcal{C}_e\chi\left(\log(\bar{c}_{p}) - \log(\bar{c}_{n})\right), \\
    &\Delta \Phi_{\text{Elec}}= -\frac{\mathcal{C}_e \mathcal{I}}{\kappa(\bar{c})}\left(\frac{l_n}{3\epsilon_n^b} + \frac{l_s}{\epsilon_{s}^b} + \frac{l_p}{3\epsilon_p^b} \right),
     \label{eqn:SPMe:electrolyte_ohmic_losses} \\
     &\Delta \Phi_{\text{Solid}} =  -\frac{\mathcal{I}}{3}\left(\frac{l_p}{\sigma_p} + \frac{l_n}{\sigma_n} \right),
\end{align}
with
\begin{equation}
    \bar{\cdot}_{\text{n}} =  \frac{1}{l_n}\int_0^{l_n} \cdot_{\text{n}} \, \text{d}x, \quad
    \bar{\cdot}_{\text{p}} =  \frac{1}{l_p}\int_{1-l_p}^{1} \cdot_{\text{p}} \, \text{d}x,
\end{equation}

This model is implemented in PyBaMM as `Composite`

```python
composite = pybamm.lead_acid.Composite()
```

## Solving the models

We load process parameters for each model, using the same set of (default) parameters for all. In anticipation of changing the current later, we make current an input parameter

```python
# load models
models = [loqs, composite, full]

# process parameters
param = models[0].default_parameter_values
param["Current function [A]"] = "[input]"
for model in models:
    param.process_model(model)
```

Then, we discretise the models, using the default settings

```python
for model in models:
    # load and process default geometry
    geometry = model.default_geometry
    param.process_geometry(geometry)

    # discretise using default settings
    mesh = pybamm.Mesh(geometry, model.default_submesh_types, model.default_var_pts)
    disc = pybamm.Discretisation(mesh, model.default_spatial_methods)
    disc.process_model(model)
```

Finally, we solve each model using CasADi's solver and a current of 1A

```python
timer = pybamm.Timer()
solutions = {}
t_eval = np.linspace(0, 3600 * 17, 100) # time in seconds
solver = pybamm.CasadiSolver()
for model in models:
    timer.reset()
    solution = solver.solve(model, t_eval, inputs={"Current function [A]": 1})
    print("Solved the {} in {}".format(model.name, timer.time()))
    solutions[model] = solution
```

    Solved the LOQS model in 162.928 ms
    Solved the Composite model in 1.264 s
    Solved the Full model in 1.400 s

## Results

To plot the results, the variables are extracted from the solutions dictionary. For example, we can compare the voltages:

```python
for model in models:
    time = solutions[model]["Time [h]"].entries
    voltage = solutions[model]["Terminal voltage [V]"].entries
    plt.plot(time, voltage, lw=2, label=model.name)
plt.xlabel("Time [h]", fontsize=15)
plt.ylabel("Terminal voltage [V]", fontsize=15)
plt.legend(fontsize=15)
plt.show()
```

Alternatively, using `QuickPlot`, we can compare the values of some variables

```python
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(solution_values)
quick_plot.dynamic_plot();
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=17.0, step=0.17), Output()), _dom_classes=('…

If we update the current, setting it to be 20 A, we observe a greater discrepancy between the full model and the reduced-order models.

```python
t_eval = np.linspace(0, 3600, 100)
for model in models:
    solutions[model] = solver.solve(model, t_eval, inputs={"Current function [A]": 20})

# Plot
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(solution_values)
quick_plot.dynamic_plot();
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=3563.636363636364, step=35.63636363636364), …

```python

```

```python

```


# Linear Models

- [Problem Definition](#problem-definition)
    - [Linear Model](#linear-model)
    - [Simplified CAPM](#simplified-capm)
- [Target Function](#target-function)
- [Raw Data](#raw-data)
- [Preprocessing](#preprocessing)
- [Machine Learning](#machine-learning)
    - [Training](#training)
    - [Model](#model)
    - [Error Function](#error-function)
- [Stochasic Gradient Descent](#stochastic-gradient-descent)
    - [`NumPy`](#sgd-numpy)
    - [`scikit-learn`](#sgd-sklearn)
    - [`TensorFlow`](#sgd-tf)
    - [`Keras`](#sgd-keras)
- [Maximum Likelihood Estimator](#maximum-likelihood-estimator)
    - [`NumPy`](#mle-numpy)
    - [`scikit-learn`](#mle-sklearn)
- [Challenge](#challenge)

## Problem Definition

Provided a set $\mathcal{S}$ of $(x_{i}, y_{i})$ pairs:
    $$\mathcal{S} = \{ (x_{1}, y_{1}), (x_{2}, y_{2}), \text{...}, (x_{k}, y_{k}) \}$$
Find a function $f$ that maps $x_{i} \rightarrow y_{i}$:
    $$f: \mathcal{X} \rightarrow \mathcal{Y}$$

$f$ is a:
* **(Discriminate) Model**
* **Map Function**
* **Function Approximation**
for $\mathcal{S}$, such that:
    $$f(x_{i}) = \hat{y_{i}} \approx y_{i}, \quad \forall i$$

## Linear Model

$f$ is a **Linear Model**, iff:
\begin{equation}
    f(x_{i}) = w * x_{i}, \quad x_{i} \in \mathbb{R}^{n}; y_{i} \in \mathbb{R}^{m}; w \in \mathbb{R}^{m \times n}
\end{equation}

An example of the $k^{th}$ datapoint for $n=2$ and $m=1$ is:
$$ x_{k} = \begin{bmatrix}
               x_{k}^{(1)} & x_{k}^{(2)}
           \end{bmatrix}^{T}
         = \begin{bmatrix}
               1 & 2
           \end{bmatrix}^{T}
\text{,} \quad
   y_{k} = \begin{bmatrix}
               y_{k}^{(1)}
           \end{bmatrix}
         = \begin{bmatrix}
               3
           \end{bmatrix}
$$

For a finite $\mathcal{S}$ where $|\mathcal{S}| = k$, we express the above equation in matrix format as:
    $$f(\mathbf{X}) = \mathbf{\hat{y}} = \mathbf{X} * \mathbf{w},
    \quad \mathbf{X} \in \mathbb{R}^{k \times n}; \mathbf{y} \in \mathbb{R}^{k \times m}; \mathbf{w} \in \mathbb{R}^{n \times m}$$

Matrices $\mathbf{X}$ and $\mathbf{y}$ are constructed by stacking the $k$ pairs as row vectors.

### Simplified CAPM (Capital Asset Pricing Model )

Factor models are a way of explaining the returns of one asset via a linear combination of the returns of other assets.
The general form of a factor model is:
\begin{equation}
    y = \alpha + \beta_{1} x_{1} + \beta_{2} x_{2} + \text{...} + \beta_{n} x_{n}
    \label{eq::beta}
\end{equation}

An asset's $\beta$ to another is just the $\beta$ factor from the equation above.

Look for:
    $$y_{stock} = \beta x_{benchmark} + \alpha =
        \begin{bmatrix}
           x_{benchmark} & 1
       \end{bmatrix}
       \begin{bmatrix}
           \beta \\
           \alpha
       \end{bmatrix}$$

A sample of the dataset $S$ is visualized below:

| $x_{benchmark}$  | $y_{stock}$ |
| ---------------- | ----------- |
| 5.0              | 1.6         |
| 1.0              | 0.6         |
| 3.7              | 1.3         |
| 9.8              | 2.3         |
| 2.8              | 0.8         |

```python
# scientific computing library
import numpy as np
# machine learning library
import sklearn
# deep learning framework
import tensorflow as tf

# visualization tools
import matplotlib.pyplot as plt
import seaborn as sns

# show plots without need of calling `.show()`
%matplotlib inline

# prettify plots
plt.rcParams['figure.figsize'] = [20.0, 5.0]
sns.set_palette(sns.color_palette("muted"))
sns.set_style("ticks")

# supress warnings
import warnings
warnings.filterwarnings('ignore')
```

## Helper Functions

```python
def visualize(x, y=None, y_noise=None, y_hat=None):
    """Visualization helper function.

    Parameters
    ----------
    x: array-like
        x-axis values
    y: array-like
        y-axis target values
    y_noise: array-like
        y-axis observations
    y_hat: array-like
        y-axis model predictions
    """
    if y is not None:
        plt.plot(x, y, label='Target Function')
    if y_noise is not None:
        plt.plot(x, y_noise, 'o', label='Noisy Observations')
    if y_hat is not None:
        plt.plot(x, y_hat, '--o', label='Model Predictions')
    plt.legend();
```

## Target Function

```python
N = 10
np.random.seed(0)

x = np.linspace(0, N, N)
y = 0.5 + 0.2 * x
visualize(x, y);
```

## Raw Data

```python
y_noise = y + np.random.normal(0,0.1, N)

visualize(x, y, y_noise)
```

## Preprocessing

```python
X = np.ones((N, 2))
X[:, 1 ] = x
```

## Machine Learning

Machine learning is the science of getting computers to act without being explicitly programmed.
Their course of action highly depends on the data that has been used in order to **train** them, while the generated program (a.k.a **model**) is a function of the data and the algorithm used to consume this data.

### Training

Training is nothing more than the process of optimizating:
$$\mathcal{J}(w) = \min_{w}\bigg[ \mathbb{E} \big[ \mathcal{L}(\mathbf{y}, f(\mathbf{X} | w)) \big] + \lambda \mathcal{R}(w) \bigg]$$

where,
* $\mathcal{J}$: Objective Function
* $\mathcal{L}$: Loss Function
* $\mathcal{R}$: Regularization Term
* $f$: Mapping Function (`model`)
* $w$: `model` Parameters
* $S = \{ (x_{i}, y_{i}): i \in \mathcal{D} \}$

Assumming that $\mathcal{J}$ has a (global) minimum, such that:

$\DeclareMathOperator*{\argmin}{arg\,min}$
$$w^{*} = \argmin_{w}\bigg[ \mathbb{E} \big[ \mathcal{L}(\mathbf{y}, f(\mathbf{X} | w)) \big] + \lambda \mathcal{R}(w) \bigg]$$

where,
* $w^{*}$: Objective Function (Global) Minimizer

### Model

Having found $w^{*}$ the `model` $f$ can be deployed and predictions can be made according to:

$$\mathbf{\hat{y}} = f(\mathbf{X} | w^{*}) \approx \mathbf{y}$$

In the case of Linear Models, $f$ gets the form of a linear function, with $w^{*}$ coefficients:

$$\mathbf{\hat{y}} = f(\mathbf{X} | w^{*}) = \mathbf{X} * w^{*} \approx \mathbf{y}$$

### Error Function

Use **Mean Squared Error (MSE)** as the metric function in order to evaluate how good/bad the model $f$ is, such that:
    $$MSE_{f} = \frac{1}{2k} \sum_{i=1}^{k} (y_{i} - \hat{y}_{i})^{2} = \frac{1}{2k} \sum_{i=1}^{k} (y_{i} - w_{i} * x_{i})^{2}$$
In matrix form:
    $$MSE_{f} = \frac{1}{2k} (\mathbf{y} - \mathbf{X} * \mathbf{w})^{T}(\mathbf{y} - \mathbf{X} * \mathbf{w})$$

We can address the optimization problem of $MSE_{f}$:
* computationally (Stochastic Gradient Descent)
* analytically (Maximum Likelihood Estimator)

## Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is the simplest and most used optimizer.
It is based on making incremental updates, towards the correctsolution,
using the derivative of the error at each step.

$$w_{t+1} = w_{t} - \eta (y_{i} - \hat{y}) x_{i}$$

### `NumPy`

```python
# hyperparameters
n_epochs = 50
eta = 0.01

# parameters initialization
w = np.random.random(X.shape[1])

# training error over time
loss = []

# model fitting
for _ in range(n_epochs):
    y_hat = np.dot(X, w)
    l = 0.5 * np.sum((y_noise - y_hat)**2)
    i = np.random.randint(len(y_noise))
    error = y_hat[i] - y_noise[i]
    w = w - eta * error * X[i]
    loss.append(l)

# plot results
visualize(x, y, y_noise, y_hat)

# plot training error
plt.figure()
plt.plot(loss)
plt.title('Learning Curve')
plt.legend(['Training Error']);
```

### `scikit-learn`

```python
from sklearn.linear_model import SGDRegressor

# hyperparameters
n_epochs = 100
eta = 0.01

# model initialization
model = SGDRegressor(loss='squared_loss', penalty='none', max_iter=n_epochs, learning_rate='constant', eta0=eta)

# model fitting & predictions
model.fit(X, y_noise)
y_hat = model.predict(X).ravel()

# plot results
visualize(x, y, y_noise, y_hat)
```

### `TensorFlow`

```python
# hyperparameters
n_epochs = 100
eta = 0.01

# placeholders
X_tf = tf.placeholder(tf.float64, shape=[None, X.shape[1]])
Y_tf = tf.placeholder(tf.float64, shape=[None])

# variables
W_tf = tf.Variable(np.random.random(X.shape[1]).reshape(X.shape[1], 1), name='parameters')

# model prediction
y_hat = tf.matmul(X_tf, W_tf)

# optimization
loss = tf.reduce_mean(tf.square(tf.squeeze(y_hat)-Y_tf)) # L2 loss
optimizer = tf.train.GradientDescentOptimizer(eta).minimize(loss) # Gradient descent

# initializing the variables
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    # model fitting
    for _ in range(n_epochs):
        sess.run(optimizer, feed_dict={X_tf: X, Y_tf: y_noise})

    # model predictions
    y_hat = sess.run(y_hat, feed_dict={X_tf: X, Y_tf: y_noise})

    # plot results
    visualize(x, y, y_noise, y_hat)
```

### `Keras`

```python
from tensorflow.contrib.keras.api.keras.models import Sequential
from tensorflow.contrib.keras.api.keras.layers import Dense

# hyperparameters
n_epochs = 100
eta = 0.01

# model initialization
model = Sequential()
model.add(Dense(1, input_shape=(1, 1), activation='linear'))
model.compile(loss='mean_squared_error', optimizer='sgd')

# keras batch training interface
X_ = x.reshape(len(x), 1, 1)
y_ = y_noise.reshape(len(y_noise), 1, 1)

# model fitting & predictions
model.fit(X_, y_, batch_size=1, epochs=n_epochs, verbose=0)
y_hat = model.predict(X_).ravel()

# plot results
visualize(x, y, y_noise, y_hat)
```

## Maximum Likelihood Estimator

Differentiate $MSE_{f}(w)$ and find its minimum. There is a global minimiser, since $MSE_{f}$ is convex (quadratic).

$$\mathbf{w_{MLE}} = (\mathbf{X}^{T} \mathbf{X})^{-1} * \mathbf{X}^{T} * \mathbf{y}$$

### `NumPy`

```python
w = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y_noise)
y_hat = np.dot(X, w)

visualize(x, y, y_noise, y_hat)
```

### `scikit-learn`

```python
from sklearn.linear_model import LinearRegression

# model initialization
model = LinearRegression()

# model fitting & predictions
model.fit(X, y_noise)
y_hat = model.predict(X).ravel()

# plot results
visualize(x, y, y_noise, y_hat)
```

## Challenges

### Beta Hedging

Repeat the above experiment using the implementation you prefer most, on real world market data.

Using the data provided in `{proj_dir}/notebooks/data/`:
* **S&P500 (SPY)**: as the market index (benchmark)
* **Apple (AAPL)**: as stock A
* **Google (GOOG)** as stocks B

come up with a Beta Hedging Strategy to minimize risk.

Load the data to a numpy.ndarray using the commands below:

```python
# helper loader function
loader = lambda asset: np.loadtxt('./data/%s.txt' % asset, delimiter=',')

# raw data
spy = loader('SPY')
aapl = loader('AAPL')
goog = loader('GOOG')

# normalised data
spy_ = spy / spy[0] - 1
aapl_ = aapl / aapl[0] - 1
goog_ = goog / goog[0] - 1

# visualization
plt.plot(spy_, label='S&P500')
plt.plot(aapl_, label='Apple Inc.')
plt.plot(goog_, label='Alphabet Inc.')
plt.title('Normalised Prices')
plt.ylabel('Normalised Change')
plt.xlabel('Time Index')
plt.legend();
```

```python
# regression plots
sns.regplot(spy_, aapl_, label='APPL')
sns.regplot(spy_, goog_, label='GOOG')
plt.title('Beta Exposure to S&P500')
plt.xlabel('SPY Percentage Change')
plt.ylabel('Stock Price Percentage Change')
plt.legend();
```

```python
# helper percentage return
pct_change = lambda series: np.diff(series) / (series[:-1] + 1e-6) # WARNING: prone to division by zero

aapl_pct = pct_change(aapl)
goog_pct = pct_change(goog)
spy_pct = pct_change(spy)

sns.distplot(aapl_pct, label='AAPL')
sns.distplot(goog_pct, label='GOOG')
sns.distplot(spy_pct, label='SPY')
plt.legend();
```


*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Univariate data
- 2-class problem
- different variances
- Gaussian model (2 parameters)
- No Risk function

# Sections

• Given information

• Deriving the decision boundary

• Plotting the class conditional densities, posterior probabilities, and decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate normal (Gaussian) model for the class-conditional densities

$ p(x | \omega_j) \sim N(\mu|\sigma^2) $

$ p(x | \omega_j) \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu}{\sigma}\bigg)^2 \bigg] } $

####Prior probabilities:

$ P(\omega_1) = P(\omega_2) = 0.5 $

#### Variances of the sample distributions

$ \sigma_1^2 = 4, \quad \sigma_2^2 = 1 $

#### Means of the sample distributions

$ \mu_1 = 4, \quad \mu_2 = 10 $

## Deriving the decision boundary
[back to top]

### Bayes' Rule:

$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} $

###Bayes' Decision Rule:

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

\begin{equation}
\begin{aligned}
&\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{aligned}
\end{equation}

###Bayes' Decision Rule:

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

$ \Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)} $

We can drop $ p(x) $ since it is just a scale factor.

$ \Rightarrow P(x|\omega_1) * P(\omega_1) > p(x|\omega_2) * P(\omega_2) $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{P(\omega_2)}{P(\omega_1)} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{0.5}{0.5} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > 1 $

$ \Rightarrow \frac{1}{\sqrt{2\pi\sigma_1^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } > \frac{1}{\sqrt{2\pi\sigma_2^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } \quad \bigg| \quad ln $

$ \Rightarrow ln(1) - ln\bigg({\sqrt{2\pi\sigma_1^2}}\bigg) -\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2  > ln(1) - ln\bigg({{\sqrt{2\pi\sigma_2^2}}}\bigg) -\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \quad \bigg| \quad \sigma_1^2 = 4, \quad \sigma_2^2 = 1,\quad \mu_1 = 4, \quad \mu_2 = 10 $

$ \Rightarrow -ln({\sqrt{2\pi4}}) -\frac{1}{2}\bigg( \frac{x-4}{2}\bigg)^2  > -ln({{\sqrt{2\pi}}}) -\frac{1}{2}(x-10)^2 $

$ \Rightarrow -\frac{1}{2} ln({2\pi}) - ln(2) -\frac{1}{8} (x-4)^2  > -\frac{1}{2}ln(2\pi) -\frac{1}{2}(x-10)^2 \quad \bigg| \; \times\; 2 $

$ \Rightarrow -ln({2\pi}) - 2ln(2) - \frac{1}{4}(x-4)^2  > -ln(2\pi) - (x-10)^2 \quad \bigg| \; + ln(2\pi) $

$ \Rightarrow -4ln(4) - (x-4)^2  >- 4(x-10)^2 $

$ \Rightarrow -ln(4) - \frac{1}{4}(x-4)^2  > -(x-10)^2 \quad \big| \; \times \; 4 $

$ \Rightarrow -8ln(2) - x^2 + 8x - 16  > - 4x^2 + 80x - 400 $

$ \Rightarrow 3x^2 - 72x + 384 -8ln(2) > 0 $

$ \Rightarrow x < 7.775 \quad and \quad x > 16.225 $

## Plotting the class conditional densities, posterior probabilities, and decision boundary

[back to top]

```python
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def pdf(x, mu, sigma):
    """
    Calculates the normal distribution's probability density
    function (PDF).

    """
    term1 = 1.0 / ( math.sqrt(2*np.pi) * sigma )
    term2 = np.exp( -0.5 * ( (x-mu)/sigma )**2 )
    return term1 * term2

# generating some sample data
x = np.arange(-100, 100, 0.05)

# probability density functions
pdf1 = pdf(x, mu=4, sigma=4)
pdf2 = pdf(x, mu=10, sigma=1)

# Class conditional densities (likelihoods)
plt.plot(x, pdf1)
plt.plot(x, pdf2)
plt.title('Class conditional densities (likelihoods)')
plt.ylabel('p(x)')
plt.xlabel('random variable x')
plt.legend(['p(x|w_1) ~ N(4,4)', 'p(x|w_2) ~ N(10,1)'], loc='upper left')
plt.ylim([0,0.5])
plt.xlim([-15,20])
plt.show()
```

```python
def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).

    """
    return likelihood * prior

# probability density functions
posterior1 = posterior(pdf(x, mu=4, sigma=4), 0.5)
posterior2 = posterior(pdf(x, mu=10, sigma=1), 0.5)

# Class conditional densities (likelihoods)
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundaries')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'p(w_2|X)'], loc='upper left')
plt.ylim([0,0.25])
plt.xlim([-15,20])
plt.axvline(7.775, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.axvline(16.225, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R2', xy=(10, 0.2), xytext=(10, 0.22))
plt.annotate('R1', xy=(4, 0.2), xytext=(4, 0.22))
plt.annotate('R1', xy=(17, 0.2), xytext=(17.5, 0.22))
plt.show()
```

## Classifying some random example data

[back to top]

```python
# Parameters
mu_1 = 4
mu_2 = 10
sigma_1_sqr = 4
sigma_2_sqr = 1

# Generating 10 random samples drawn from a Normal Distribution for class 1 & 2
x1_samples = sigma_1_sqr**0.5 * np.random.randn(20) + mu_1
x2_samples = sigma_1_sqr**0.5 * np.random.randn(20) + mu_2
y = [0 for i in range(20)]

# Plotting sample data with a decision boundary

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.title('Classifying random example data from 2 classes')
plt.ylabel('P(x)')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')
plt.ylim([-0.1,0.1])
plt.xlim([0,20])
plt.axvline(7.775, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.axvline(16.225, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R2', xy=(10, 0.03), xytext=(10, 0.03))
plt.annotate('R1', xy=(4, 0.03), xytext=(4, 0.03))
plt.annotate('R1', xy=(17, 0.03), xytext=(17.5, 0.03))
plt.show()
```

## Calculating the empirical error rate

[back to top]

```python
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 > 7.775 and x1 < 16.225:
        w1_as_w2 += 1
    if x2 <= 7.775 and x2 >= 16.225:
        w2_as_w1 += 1

emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))
```

    Empirical Error: 0.0%

```python

```

```python

```

```python
test complete; Gopal
```


# BIAS-VARIANCE AND MODEL SELECTION

---

## No free lunch theorems

---

Let's start by considering one of the most famous and important theorem in machine learning. Focus now the attention on a binary classification problem and define:

$$
\begin{align}
Acc_G(L) &= \textrm{Generalization accuracy of learner L} \\
&= \textrm{Accuracy of learner L on non-training samples}
\end{align}
$$

$$
\begin{align}
\mathcal F = \textrm{set of all possible concepts  } y = f(\textbf x)
\end{align}$$

We can view at $\mathcal F$ as the space of all the possible hypothesis as for example the set of all possible functions with binary output.

**Theorem**

*For any learner L, $\frac{1}{|\mathcal F|}\sum_{\mathcal F}Acc_G(L) = \frac{1}{2}$, given any distribution $\mathcal P$ over $\textbf x$ and training set size $N$.*

**Proof**

For every concept $f$ where $Acc_G(L) = 0.5 + \delta$, exists a concept $f'$ where $Acc_G(L)=0.5 - \delta:$$ \forall \textbf x \in \mathcal D, f'(\textbf x) = f(\textbf x); \forall \mathcal \notin \mathcal D, f'(\textbf x) \neq f(\textbf x)$

**Corollary**

*For any two learners $L_1$ and $L_2$, if $\exists$ learning problem s.t. $Acc_G(L_1)> Acc_G(L_2)$, then $\exists$ learning problem s.t. $Acc_G(L_2)> Acc_G(L_1)$*

Accuracy $0.5$ means that the learner is guessing randomly. It is important to notice that this value is the same no matter what is the training size considered. Any function has the same probability to be the true function and the dataset is useless. This can seems something weird and imposes some precautions. ML is based on the hypothesis that not all the functions have the same probability and we assume also some regularity on functions which allow us to generalize. No algorithm is a priori better than the others so there is no a single approach that can dominate. The best algorithm is problem dependent and depends also on the assumptions made.

> So what? You can't fall in love with a single technique

Since there is no a best learner for all the situations you have to try different approaches.

## Bias-Variance tradeoff

---

Our goal is to find a model $y(\textbf x)$ that approximates $t$ as well as possible. In order to evaluate the goodness of the prediction we use a loss function, which expected value is given by

$$
\begin{align}
\mathbb E[\mathcal L] = \int \int \mathcal L(t, y(\textbf x)) p(\textbf x, t) d\textbf x dt
\end{align}
$$

Which in the specific case of squared loss becomes

$$
\begin{align}
\mathbb E[\mathcal L] = \int \int (y(\textbf x) - t)^2 p(\textbf x, t) d\textbf x dt
\end{align}
$$

There is immediately a problem, however, since we don't know the probability $p(\textbf x, t)$. The bias-variance decomposition is a framework to evaluate the performance of models. Let's assume we have a data set $\mathcal D$ with $N$ samples generated from the model

$$
\begin{align}
t_i = f(\textbf x_i) + \epsilon
\end{align}
$$

where $\epsilon$ has expected value $\mathbb E[\epsilon] = 0 $ and variance $Var[\epsilon] = \sigma^2 $ (notice that this in not the same as Gaussian assumption).  Let's consider the expected square error on an unseen sample

$$
\begin{align}
\mathbb E[(t - y(\textbf x))^2] &= \mathbb E[t^2 + y(\textbf x)^2 - 2ty(\textbf x)] \quad \textrm{compute the square} \\
&= \underbrace{\mathbb E[t^2]}_\text{$2^{nd}$ moment of $t$} + \underbrace{\mathbb E[y(\textbf x)^2]}_\text{$2^{nd}$ moment of $y$}  - \mathbb E[2ty(\textbf x)] \quad \textrm{thanks to the linearity of $\mathbb E[\cdot]$} \\
&= \mathbb E[t^2] \pm \mathbb E[t]^2 + \mathbb E[y(\textbf x)^2] \pm \mathbb E[y(\textbf x)]^2 - 2f(\textbf x)\mathbb E[y(\textbf x)] \quad \textrm{since $\mathbb E[t] = f(\textbf x)$} \\
&= Var[t] + \mathbb E[t]^2 + Var[y(\textbf x)] + \mathbb E[y(\textbf x)]^2 - 2f(\textbf x)\mathbb E[y(\textbf x)] \quad \textrm{since $\mathbb E[x^2] - \mathbb E[x]^2 = Var[x]$} \\
&= Var[t] + Var[y(\textbf x)] + (f(\textbf x) - \mathbb E[y(\textbf x)])^2 \\
&= \underbrace{Var[t]}_{\sigma^2} + \underbrace{Var[y(\textbf x)]}_{variance} + \underbrace{\mathbb E[f(\textbf x) - y(\textbf x)]^2}_{(bias)^2} \quad \textrm{since $\mathbb E[f(\textbf x)] = f(\textbf x)$}
\end{align}$$

$$ \textrm{expected loss = (bias)$^2$ + variance + noise} $$

The first thing that we can observe is that all the three terms are always positive and to the second power so, they have the same additive importance. The first term derive from the intrinsic variability of the target data and since it is independent of $y(\textbf x)$, it represents the irreducible minimum value of the loss function. To better understand this term just think at the procedure to collect the points inside the data set. We can perform for example an experiment which, also if perfectly controlled, it is inevitably subjected to randomness which means that given the same combination of test parameters we will obtain results that are not exactly equal. The last two terms are those on which we can work on.

* **Bias**: represents the extent to which the average of our estimate differs from the desired regression function. It is high when we have few samples and many parameters.

* **Variance**: measures the extent to which the solution vary around the average. It is high when we have a high hypothesis space.

It is important to notice that we have to minimize the sum of these two term not just one since their effect is additive and no one can be negative. When we have a good algorithm in supervised learning means that it is able to correctly balance the bias-variance trade-off.

We can generalize the previous consideration assuming that we have a larger number of data sets each of size $N$ and each drawn from the same distribution. For any given dat set $\mathcal D$ we can run our learning algorithm and obtain a prediction $y(\textbf x, \mathcal D)$. Since each data set is different, we will obtain differen value of the loss function running the same algorithm on each of them. The performance of each particular learning algorithm can be addressed by averaging over the ensamble of data sets. For each ensamble we have to integrate over all the input space to compute the desired quantities

$$
\begin{align}
\textrm{bias}^2 = \int(f(\textbf x) - \mathbb E[y(\textbf x)])^2 p(\textbf x) d\textbf x
\end{align}
$$

$$
\begin{align}
\textrm{variance} = \int \mathbb E[(y(\textbf x) - \mathbb E[y(\textbf x)])^2]p(\textbf x) d\textbf x
\end{align}
$$

The picture below helps to give an intuitive overview of the possible cases in which a machine learning algorithm can falls.

On the top right we have the **overfitting** problem since the solution is really sensitive to the dataset, as witnessed by the high variance of the points. On the bottom left, we have the opposite case called **underfitting** in which the variance is low since the model is not so complex but, it is completely unable to capture the true behavior of the system and the bias is high. These two are the most usual situations that one can encounter during the design of a machine learning problem. At this point one can think why we can't try to move the biased underfitted model toward the center of the target. The answer is in the bias-variance equations showed before. If we try to move the points in the bottom left target towards the center they will start to spread as in the top right picture. You still don't have understand what is overfitting? It's easy..

We have that the only way to decrease the bias is with more complex model so, increasing the size of the hypothesis space. In order to decrease the variance we have to move in the opposite direction decreasing the hypothesis space since simple models have low variance. Differently from bias, though, we can reduce the variance also by increasing the number of samples. The perfect solution is to use many many samples and a very complex model but this causes computational issues.

The picture below shows the shape of the bias and variance curves as the complexity of the model is varied. Any supervised learning algorithm has some parameters that allow to obtain the required balance between the two terms. The regularization techniques tend to move the model toward the left increasing the bias.

Let's consider now that we have a data set $\mathcal D$ with $N$ samples and we chose for example the RSS loss function. The loss on the training set is so computed as

$$
\begin{align}
L_{train} &= \frac{1}{N} \sum_{n=1}^N(t_n - y(\textbf x_n))^2 \quad \textrm{for regression} \\
&= \frac{1}{N} \sum_{n=1}^N(I(t_n \neq y(\textbf x_n)) \quad \textrm{for classification}
\end{align}
$$

As the model complexity increases the loss on the training set tends to decrease as tends to decrease also the bias from the true model. The problem is that as we have already said, in machine learning we are trying to minimize a surrogate of the real objective function. Our goal is not to obtain the minimum of the loss on the training points but to generalize our model to unseen samples. So, we try to minimize the loss on the training set in the hope to minimize the loss also on unseen points. As we can see from the picture below, the training loss is a good approximation of the real loss just when we have simple models. When the complexity of the model increases to much the two curves tend to diverge.

> The training loss is an **optimistically biased** estimate of the prediction error

This consideration should give an idea of why we need another set, the **test** set. What is done in practice is to "randomly" split the data set into train and test, and then used the training set to optimize the parameters and the test set to evaluate the prediction error. This strategy allows us to obtain an unbiased estimate of the model goodness. The loss on the test set is computed as

$$
\begin{align}
L_{test} = \frac{1}{N_{test}} \sum_{n=1}^{N_{test}}(t_n - y(\textbf x_n))^2
\end{align}
$$

It is important to remark that the test set is unbiased only if it is not used also for the training. In the pictures below the red curve is the prediction error while the black one is its estimates obtained with the test set, which is noisy since we are working with finite data.

The bad news is that there is still no solution on the number of points that must make up the test and the number that must make up the training set. In the picture below we fix the complexity of the model and look at the curves as the number of points in the training set is varied. As the number of points increases the training error increases but decreases the prediction error. When we have a lot of samples the training error becomes a good estimate of the prediction error.

The relative trend of the train and test losses can gives important informations on the goodness of the model. If the test error and the training error are distant it means that the variance is high and so the hypothesis space is too large. If the two curves tend to converge fast it means that we have a too high bias.

The bias-variance trade-off can be managed using different techniques:

* **Model selection**: features selection, regularization, dimension reduction

* **Model ensamble**: bagging, boosting

Bagging and boosting are metatechniques that allow to reduce one error without modifying to much the other.

## Curse of dimensionality

---

In real world applications we usually have to deal with input vectors with a lot of features, not just two or three as we are used to visualize. The high dimensionality of the input space poses serious challenges and has a high impact on the design of pattern recognition techniques. The difficulties related to design of techniques able to work with inputs of high dimensionality are described by a phenomenon called **curse of dimensionality**. To better understand this phenomenon just imagine a problem in which we want to classify a point based on its neighbors. An easy way to calculate the class of a point relying on its neighbors is to divide the space into cells and then calculate the class with maximum probability inside each cell. This example is really useful to understand the problem related to high dimensional inputs because it is easy to see that the number of cells grows exponentially with the number of features.

If the number of cells grows exponential also the number of inputs need to follow the same growth in order to assure that all the cells could contain at least one point. Another geometric example is the calculation of the volume fraction of a $D$-dimensional sphere that lies between $r = 1 - \epsilon$ and $r = 1$ where $r$ is the radius of the sphere. After some basic math calculations we will see that, for large $D$, this fraction tends to $1$. In spaces of high dimensionality most of the volume of the sphere is concentrated in a thin shell near the surface.

> Not all the intuitions developed in spaces of low dimensionality will generalize to spaces of many dimensions

We can summarize the problems related to high dimensional input space as

* Large variance

* Many samples required

* High computational costs

These problems can be tackled in different ways

* **Feature selection:** usually real data is often confined into a region of the space having lower effective dimensionality. This consideration can be exploited by identifying a subset of input features that are most related to the output.

* **Dimension reduction:** usually real data have particular directions along which import variations in the target variables occur. This consideration brings to the intuition that the input variables could be projected into a lower-dimensional subspace able to describe just the directions of higher variability.

* **Shrinkage:** real data will typically exhibit some smoothness properties so we can think to use all the input features and then shrink some coefficient to zero as in regularization, reducing the variance.

## Feature selection

---

The most simple and intuitive way of calculating the best set of features is to evaluate all the possible subset of the original features. This approach is called **Best Subset Selection** and is based on the following steps:

1. Let $\mathcal M_0$ denote the null model, which contains no input feature so it only predicts the sample mean of each observation.

2. For $k=1, ...,M$:

   * Fit all $\binom{M}{k}$ models that contain exactly $k$ features

   * Pick the best among these $\binom{M}{k}$ models and call it $\mathcal M_k$. With best model we mean the one that has the smallest $RSS$ o largest $R^2$.

3. Select the single best model among $\mathcal M_0, ..., \mathcal M_M$ (will be clear later how to choose the best model)

Below the envelope of the RSS and of R$^2$ of the best subset with the number of features is reported. As we can see we can't just pick the model with the best score since both of the performance metrics are monotonic.

With this simple procedure we can definitely get the best subset of features but, it has serious computational problem when $M$ is large as the possible number of subset grow as $2^M$. Moreover, if we don't choose the final best subset carefully we will end up in overfitting. There exists three metaheuristics to tackle the first problem.

* **Filter:** we rank the features and select the best ones

* **Embedded:** the learning algorithm exploits its own variable selection technique as in *Lasso*, *Decision trees*, *Auto-encoding*, etc.. In this case Lasso is used just to wee which features are associated to zero parameters and then we will use another model for the regression as Artificial Neural Network or Gaussian Processes. In Decision trees we can remove the features that are not use by the tree

* **Wrapper:** instead of considering all the possible subset we can follow a sequential procedure. Two possible methods are present:

    * **Forward Step-wise Selection:** starts from an empty model and then adds features one-at-a-time until all features are in the model. At each time is added the feature that gives the greatest additional improvement

    * **Backward Step-wise Selection:** starts with all the features and removes the last useful features, on-at-a-time

Both the wrapper methods here presented do not guarantee to get the best model.

As visible from the picture above, the model containing all the features is for sure the one that has the smallest training error. However, as we have already said, the train error is not always a good approximation of the prediction error. Remember that our goal is to perform well on unseen data not on the training set. Instead of looking at the training error we have to look at the error on the test set and there are two approaches to estimate it:

* *Direct* estimation of the prediction error using a validation approach

* *Indirect* estimation by making an *adjustment* of the training error to account for model complexity

### Cross-Validation

Let's consider now the direct approach. We already know that we can't use the train error to evaluate the goodness of a model and of curse we can't use the test error . The test error cannot be used to influence our decision during the building of the model, it must be independent of the whole learning process. A simple possible solution is to split the data set not in two parts but in three parts:

* **Training data:** used to learn the parameters of the model

* **Validation data:** used for model selection or model assessment

* **Test data:** used for the very final evaluation of the model performance

So we have introduced a new set that will be used for the fine tuning of the parameters of the learning algorithm. The problem is that also this solution is not completely safe. The problem is related to the fact that using the validation for model selection we can overfit also this set obtaining bad results on the test points. Another problem is related to the fact that in real case applications the data is limited and removing data from the training set may compromise the performance of the model. All the time we wish to use the maximum number of points to train the model. A solution could be to use just a small part of the training set to build the validation one but in this case we will obtain a noisy and not representative estimate of predictive performance. One solution to this dilemma is to use **cross-validation**. Let's consider the whole training set composed by $D$ samples and to pick only one sample from this set to build the validation set. In this case we will have $D-1$ samples in the training set and $1$ sample in the validation one. Now we learn the model using the dataset given by $\mathcal D \backslash \{n\}$, where $\mathcal D$ contains all the points, and then we estimate the prediction error using just the single point $\{n\}$. We will repeat this process using all the time a different sample for the validation until all the samples are used for approximating the error.

This technique is called **leave-one-out cross validation** (LOOCV) and the final performance are obtained averaging over all the model obtained.

$$
\begin{align}
L_{LOO} = \frac{1}{N} \sum_{n=1}^N (t_n - y_{\mathcal D \backslash \{ n \} }(\textbf x_N))^2
\end{align}
$$

After this process the model can be learned used all the data points inside the training. LOO is almost unbiased  and slightly pessimistic since we use $N-1$ point inside the training set instead of $N$. Obviously, also in this case there is a problem :). Suppose we have $100,000$ data points and our learning algorithm requires just $1s$ to complete. The entire procedure of LOO will require $1$ entire day and if we have also to repeat the process in order to tune the hyperparameters as $\lambda$ it will take forever!

Another better solution is to use **k-fold cross validation**. In this case, instead of using just one sample inside the validation set we divide the training data into $k$ equal parts $\mathcal D_1, ..., \mathcal D_k$. We repeat the same procedure as in leave-one-out but in this case every time one of the $k$ part is used for validation and all the other for the training as depicted below.

For every run the estimate of the error will be

$$
\begin{align}
L_{\mathcal D_i} = \frac{k}{N} \sum_{(\textbf x_n, t_n) \in \mathcal D_i} (t_n + y_{\mathcal D \backslash \mathcal D_i}(\textbf x_n))^2
\end{align}
$$

And the $k$-fold cross validation error is the average over all the data splits

$$
\begin{align}
L_{k-fold} = \frac{1}{k} \sum_{i=1}^k L_{\mathcal D_i}
\end{align}
$$

It is easy to see that LOOCV is just a special case of the more general $k$-fold CV.The $k$-fold cross validation is much faster than LOO and more pessimistically biased since in this case there are less points inside the training set at each run. Usually $k$ is taken equal to $10$ but actually depends on the specific problem.

Differently from direct methods, adjustment techniques involve just the training data and then correct the accuracy and error estimate inserting terms related to the model complexity. Since these methods are not iterative they are faster than direct methods. Just to mention some techniques we can use $C_P$, *Akaike information criterion* (AIC), *Bayesian information criterion* (BIC) or *Adjusted* $R^2$.

## Shrinkage

---

Let's now consider the effect of shrinkage methods. We have already seen that regularization approaches such Ridge regression or Lasso shrink the parameters towards zero and this shrinkage has the effect of reducing the variance. We will consider an example involving simulated data containing $45$ features and $50$ observations.

In the picture below we see a comparison between Ridge regression and OLS. The black curve is the squared bias, the green (or light blue) is the variance, the purple is the MSE and the dashed line is the minimum achievable MSE. We see on the left plot that as $\lambda$ increases the flexibility of the Ridge decreases, witnessed by the decrease of the variance. We see that when $\lambda=0$, which corresponds to the OLS estimate, the variance is high but the bias is zero. We can notice how, until the purple cross, the reduction in variance is much higher than the increase in bias.

The well know disadvantage of Ridge regression is that it shrinks all the coefficient toward zero but actually, no coefficient will be exactly null. This may cause a problem not for the prediction but for the model interpretation when the number of features is high. Below we compare the Ridge regression and the Lasso one. In the figure on left the curves are associated to the Lasso while on the right we have solid curves for Lasso and dashed curves for Ridge. We can see that in this case the performance of the two methods are quite similar, with Ridge that has slightly lower variance. The data used to fit the models in the figures below are generated in such a way that all the features were related to the response.

But, what happens if just $2$ of the $45$ features are important?

We see that in this case the Lasso outperform the accuracy obtained with Ridge. These two examples illustrate that neither Ridge regression not the Lasso will universally dominate the other.

In both cases, the major problem is related to the choice of the best value of the regularization coefficient. One possibility to find the best possible value of $\lambda$, the one corresponding to the purple cross in the picture above is to use cross-validation using a grid of $\lambda$ values. We find with cross-validation the best model associated to the best regularization value and then we retrain the model with the optimal $\lambda$ and all the points inside the training set.

## Dimensionality reduction

---

In dimensionality reduction methods we operate a transformation of the original features and then the model is learned on the transformed variables. This concept differs from feature selection in two main aspects:

1. it uses all the features

2. it is an unsupervised approach

These techniques fall into **unsupervised learning** and some examples of them are:

* Principal Component Analysis (PCA)

* Independent Component Analysis (ICA)

* Self-organizing Maps

* Autoencoders

* t-SNE

### Principal component analysis

PCA is also known as *Karhunen-Loève* transform and is widely used for dimensionality reduction, lossy data compression, feature extraction and data visualization. The idea behind PCA is to perform an orthogonal projection of the data onto a lower dimensional linear space, known as the *principal subspace* which accounts for most of the variance in the data. It can be also viewed as the **linear projection** that minimizes the average projection loss, defined as the squared distance between the original points and their projections. An important hypothesis that we make before applying PCA is that we assume regularity in the function. If this assumption is not meet can happens that a function vary a lot in the directions of minimum variance of the inputs and in this case PCA becomes useless. In the picture below we see that the points are distributed in the plane and vary both along the horizontal and the vertical axes. After applying PCA we discover the two directions that maximize the variance of the data. The first principal direction is the one that explain most of the variation of the input.

The conceptual algorithm of PCA is to start finding the direction such that when the data is projected onto that line, it has the maximum variance. After that we search for the second line, orthogonal to the first one, that has maximum projected variance. This process is repeated until the desired number of dimensions is achieved. Our goal is to project the data, onto a space having dimensionality $M<D$ while maximizing the variance of the projected data. The first step is to compute the data in order to center them

$$
\begin{align}
\bar{\textbf x} = \frac{1}{N} \sum_{n=1}^N \textbf x_n
\end{align}
$$

Now we define the variance matrix

$$
\begin{align}
\textbf S = \frac{1}{N-1} \sum_{n=1}^N (\textbf x_n - \bar{\textbf x}_n)(\textbf x_n - \bar{\textbf x}_n)^T
\end{align}
$$

where $(N-1)$ in the denominator is for obtaining an unbiased estimate of the variance. The direction of the first principal component will be the first eigenvector $\textbf e_1$ of $\textbf S$ associated to the first eigenvalue $\lambda_1$ which represents the variance expressed along the direction $\textbf e_1$. We can decide the dimension of the final projection space by taking the first $k$ eigenvectors which represent a new orthogonal basis for the feature space whose axes are aligned with the maximum variances of the original data. We can rewrite the new input space as

$$
\begin{align}
\textbf X' = \textbf X \textbf E_k
\end{align}
$$

The new input data $\textbf X'$ will not contain all the original information unless all the eigenvectors are used for the transformation. Since our goal is to reduce the input dimension we have to accept the loss of some information at the benefit of a smaller problem. We can calculate the amount of variance captured by looking at the eigenvalues

$$
\begin{align}
\textrm{Variance captured} = \frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^N \lambda_j}
\end{align}
$$

An historical application of PCA is to face recognition. A typical image size is of $256 \times 128$, so it is associated to a $32768$ - dimensional space. Each face image lies somewhere in this high dimensional space. An important consideration behind this application is that all the faces have some similar feature, they cannot be randomly distributed in this space. According to this idea PCA can help finding the appropriate subspace able to capture and correctly describe a face.  The picture below shows the mean and the first $15$ eigenvectors (shown as images). These eigenvectors are called **eigenfaces**.

Now we can construct a face by randomly taking a point inside this new space.

We previously mentioned also autoencoders, they are similar to PCA. The main difference is that PCA can operate just a linear projection, is fast and can be computed in closed form. Autoencoders are more powerful since allow also nonlinear projections but require long learning time.

## Bagging and boosting

---

The methods seen so far allow to reduce the variance by increasing the bias and vice versa. Someone may wonder if there is the possibility to reduce the variance without increasing the bias or decrease the bias without increasing the variance. The answer is yes, the first goal can be achieved with **bagging** and the second with **boosting**. These two algorithms are called meta-algorithms since they can be applied to any existing algorithm. The basic idea is that the performance into a specific task can be increased by combining multiple models together instead of using just a single model for the solution. Let assume that we have $N$ independent datasets and to learn from them $N$ different models, *bagging* compute the prediction of the output as

$$
\begin{align}
y_{COM} = \frac{1}{N} \sum_{n=1}^Ny_i
\end{align}
$$

The subscript COM stands for **commettees** since this is the name given to these combination of models.

If the datasets are independent the variance of the committee will be $1/N$ of the variance of the single model $y_i$. This can be shown by considering a random variable $\bar{x} = \sum_N x$ and calculating its variance

$$
\begin{align}
Var(\bar{x}) = \frac{1}{N^2}\sum_N Var(x)=  \frac{1}{N}Var(x)
\end{align}
$$

Unfortunately this result is based on the assumption that the $N$ datasets are all independent. Obviously, this is not actually possible to obtain since the dataset is just one and it is impossible to correctly satisfy the assumption of $N$ independent errors of the models. What usually happen is that we can obtain $N$ correlated data sets and the reduction in variance will be always lower than the one in the equation above. One possibility to obtain these $N$ data sets is to use **bootstrap**. Suppose our training set $\textbf X$ consists of $n$ data points, we can create a new data set by drawing $n$ points with replacement from $\textbf X$. Repeating this procedure $N$ times allows to obtain $N$ different data sets since in each one can be repetitions of same points of $\textbf X$ and the absence of others. So, in bagging we create $N$ data sets, we train a model from each data set generated and then we compute the prediction for new samples by evaluating all the trained model and combining the outputs with majority voting for classifications or averaging for regression. Bagging is useful when it is applied to *unstable learner*, i.e., learners that change significantly with even small changes in the data. This learner are characterized by low bias and high variance that is what usually happens when the hypothesis space is really large. When we apply this method we want that our single learners overfit the data so it is used with really deep neural network, linear models with many features or with decision trees. Usually all the learners are made by the same model but this is not strictly required, we can obtain an ensamble by combining also different algorithms. It is important not notice that bagging does not help when learners are robust models (high bias low variance). Another important aspect of bagging is that all the models can be trained in parallel on a parallel architecture.

Boosting is another techniques to generate ensamble model with small bias by combining weak learners. With weak learners we mean models that are just slightly better than random if we consider for example a classification application. In other words in boosting the single models underfit the data, in this way we can have low variance on the single model and combining them we reduce also the bias. These models are trained sequentially as follows:

* weight all training samples equally

* Train model on train set

* Compute error

* Increase weights on point where model gets wrong

* Train new model on re-weighted train set

* Repeat until tired

* Final performance given by weighted prediction of each model

So, starting from the first model trained with all the points in the training set equally important, we increase the focus of subsequent learners on points where the previous model is wrong by increasing their weights somehow. Notice that all learner are global learner not local but is local each loss function since it is weighted unevenly in the design space. The most widely used form of boosting is called *Adaboost*.

The log term $\log 1/\beta_r$ is always positive since $\beta_r \in [0, 1]$. It is important for boosting to be applied to weak learner since it is not able to take the variance under control.

In conclusion:

* Bagging:

    * reduces variance

    * doesn't work well with stable model

    * can be applied to noisy data

    * almost always help also if sometimes just few

    * parallel

* Boosting:

    * reduces bias

    * works with stable learners

    * might have problems with noisy data

    * on average helps more than bagging but can also hurt performance sometimes

    * serial

References:

1. *Restelli M., Course slides*

2. *Bishop C.M., "Pattern Recognition and Machine Learning", chapters: 3.2, 12.1, 14.2, 14.3*

3. *Hastie T. et al, "Elements of Statistical Learning", chapters: 3.3, 7.3*

4. *James G. et al, "Introduction to Statistical Learning", chapters: 5.1, 6.1.3*