## CMA Diagram

Clemmow-Mullaly-Allis (CMA) Diagram

**Warning**: This notebook would store data (png images) under your jupyter working directory. To be accurate, that is `/the-path-to-your-jupyter-working-directroy/sinupy_data/dispersion/*.png`. Of course you can modify it (`data_path`) in the following block.

```python
from sympy import sqrt, pi, init_printing; init_printing()
from scipy.constants import e, m_p, m_e, c
import sinupy.mediums.plasma as pms

import matplotlib.pyplot as plt
from pathlib import Path
data_path = Path('./sinupy_data/dispersion'); data_path.mkdir(parents=True, exist_ok=True)
from sinupy.draw import draw_discontinuable_expr, add_line_with_slope

import sinupy.algebra.utility as fualguti
```

```python

```

```python
from sinupy import mediums, waves
from sinupy.waves import EM
plasma = mediums.ColdMagnetizedPlasma(species='e+i')
wave_eq = waves.EM.WaveEq(plasma)
wave = wave_eq.wave
```

```python
m_i_N = m_p
m_e_N = m_e
omega_ce = pms.omega_ce(plasma=plasma)
omega_pe = pms.omega_pe(plasma=plasma)
# Even if your plasma.species is 'e', the ion-relevant symbols would not interrupt ...
# our calculation procedure, because `expr.subs(a_specific_symbol, a_numeric_value)` ...
# also would not interrupt our procedure (i.e. throw an exception) when it finds there ...
# does not exist such `a_specific_symbol` in the formula.
omega_ci = pms.omega_cj(plasma=plasma, varidx='i')
omega_pi = pms.omega_pj(plasma=plasma, varidx='i')

# Substitute symbol parameters with accurate numerical values.
# Note the function will capture the variables B, n_0, m_i from the working scope.

w2N = lambda expr: expr\
    .subs(omega_ce, pms.omega_ce(B=B))\
    .subs(omega_pe, pms.omega_pe(n_0=n_0))\
    .subs(omega_ci, pms.omega_cj(q_e=1, m=m_i_N, B=B))\
    .subs(omega_pi, pms.omega_pj(n_0=n_0, q_e=1, m=m_i_N))
```

### $N^2(\omega, \theta=0)$ and $\omega$ Singularies
Express $N^2$ with $\omega$, $\omega_{ce}$, $\omega_{pe}$, rather than $\kappa_\perp$, $\kappa_\times$, $\kappa_\parallel$.

There exist omega $\omega$ singularites. At these points, $\omega$ would cause an infinite $N^2$, *i.e.* induce resonance.

The number of numerical result may be less than analytic symbol results, because sympy knows $\omega \geq 0$ and removes some obviously wrong answers.

```python
# Substitute kappa components with omega.
N2_in_omega = [
    pms.kappa2omega(sol, wave, plasma) for sol in
    EM.solve_N2(wave_eq, theta=0)] # <-- Set theta here

# Symbol results of omega singularities
[fualguti.find_singularities(sol, wave.w) for sol in N2_in_omega]

# Substitute constant parameters with accurate numerical values.
B, n_0 = 5, 1e20
N2_in_omega = [w2N(sol) for sol in N2_in_omega]

# Numerical result of omega singularities
omega_singularites = \
    [fualguti.find_singularities(sol, wave.w) for sol in N2_in_omega]

```

```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all" # display all expression in one cell instead of the last one
```

```python
from sympy import symbols, solve, Eq, sqrt, pi
from sinupy.algebra.draw import draw_discontinuable_expr, add_line_with_slope
e, epsilon = symbols('e, epsilon', positive=True)
m_e, m_i = plasma.m['e'], plasma.m['i']
n_e, n_i = plasma.n['e'], plasma.n['i']
B = plasma.B_amp()

w2Bn = lambda expr: expr\
    .subs(omega_ce, e * B / m_e)\
    .subs(omega_ci, e * B / m_i)\
    .subs(omega_pe**2, e**2 * n_e /(epsilon * m_e))\
    .subs(omega_pi**2, e**2 * n_e /(epsilon * m_i))
```

### Wave Resonance
Wave resonance happens when the relative refracion, $N$ blow up to infinity. As we already know, $N^2$ is a function of the wave angular frequency $\omega$, the angle $\angle(\vec{k}, \vec{B})$ between the wave $\vec{k}$ vector, the external magnetic field $\vec{B}$, and other characteristic frequency in plasma, *i.e.* $\omega_{pe}$ ,$\omega_{ce}$ and so on. In the following blocks, we fix the angle and find the $\omega^2$ that would make $N$ blow up.

```python
N2_in_omega = [
    pms.kappa2omega(sol, wave, plasma) for sol in
    EM.solve_N2(wave_eq, theta=pi/2)] # <-- Set theta here
N2_in_omega
```

```python
resonance_omega_points = [fualguti.find_singularities(sol, wave.w) for sol in N2_in_omega]
resonance_omega_square_points = [
        list(set(map(lambda x:pow(x,2), branch_omega_points)))
    for branch_omega_points in resonance_omega_points]
resonance_omega_square_points
# The above expressions contain $omega_{pe}$, $\omega_{ce}$ and so on.
# We transform them to basic plasma parameters like $\vec{B}$, $n_e$ as follows.
resonance_omega_square_points = [[
    w2Bn(omega_square) for omega_square in branch
] for branch in resonance_omega_square_points]
resonance_omega_square_points
```

```python
cutoff_omega_square_points = [
    w2Bn(omega_square) for omega_square in
    [((omega_ci-omega_ce + sqrt((omega_ce+omega_ci)**2 + 4 * omega_pe**2))/2)**2,
     ((omega_ci-omega_ce - sqrt((omega_ce+omega_ci)**2 + 4 * omega_pe**2))/2)**2]
#     [((pms.omega_ce + sqrt(pms.omega_ce**2 + 4 * pms.omega_pe**2))/2)**2,
#      ((-pms.omega_ce + sqrt(pms.omega_ce**2 + 4 * pms.omega_pe**2))/2)**2, ]
]

cutoff_omega_square_points
```

```python
X, Y = symbols('X, Y', real=True, negative=False)
Bn2XY = lambda expr: expr\
    .subs(B**2, Y * (m_e * m_i * wave.w**2) /(e**2))\
    .subs(B, sqrt(Y * (m_e * m_i)) * wave.w / e)\
    .subs(n_e, X * (epsilon * m_e * wave.w**2) / e**2)

```

```python
resonance_omega_square_points = [[
    Bn2XY(omega_square) for omega_square in branch
] for branch in resonance_omega_square_points]
resonance_omega_square_points
```

```python
cutoff_omega_square_points = [
    Bn2XY(omega_square.expand()) for omega_square in cutoff_omega_square_points
]
cutoff_omega_square_points
```

```python
resonance_points_as_Eq_1 = [
        [omega_square.subs(wave.w, 1) for omega_square in branch]
    for branch in resonance_omega_square_points]
resonance_points_as_Eq_1
```

```python
cutoff_points_as_Eq_1 = [
        omega_square.subs(wave.w, 1)
    for omega_square in cutoff_omega_square_points]
cutoff_points_as_Eq_1
```

```python
from sympy import solve, Eq
solve(Eq(resonance_points_as_Eq_1[1][0], 1), Y)
solve(Eq(resonance_points_as_Eq_1[1][1], 1), Y)
solve(Eq(resonance_points_as_Eq_1[1][2], 1), Y)
solve(Eq(resonance_points_as_Eq_1[1][3], 1), Y)

```

```python
solve(Eq(cutoff_points_as_Eq_1[0], 1), X)
solve(Eq(cutoff_points_as_Eq_1[1], 1), X)
```

```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "last" # display all expression in one cell instead of the last one
```

```python
import matplotlib.pyplot as plt
fig_CMA, ax_CMA = plt.subplots(figsize=(20, 30))
ax_CMA.set_xscale('log')
ax_CMA.set_yscale('log')
ax_CMA.set_xlabel('$\omega_p^2/\omega^2$', loc='right', fontdict={'size': 32})
ax_CMA.set_ylabel('$\\frac{\omega_{ce}\omega_{ci}}{\omega^2}$ ', loc='top', fontdict={'size': 32}, rotation=0)
# ax_CMA.set_xticks([1.0])
# ax_CMA.set_xticklabels(size=20)
ax_CMA.set_yticks([1./m_i_N, 1.0, m_i_N/1])
ax_CMA.set_yticklabels(['$m_e/m_i$', '$1.0$', '$m_i/m_e$'], size=20)
# change the fontsize
ax_CMA.tick_params(axis='x', labelsize=20)

ax_CMA.axhline(
    y=solve(Eq(resonance_points_as_Eq_1[1][0], 1), Y)[0].subs(m_i, m_i_N).subs(m_e, m_e_N),
    color='blue', linestyle=':', label='$u_L=0$, $\omega=\omega_{ce}$')
ax_CMA.axhline(
    y=solve(Eq(resonance_points_as_Eq_1[1][2], 1), Y)[0].subs(m_i, m_i_N).subs(m_e, m_e_N),
    color='purple', linestyle=':', label='$u_R=0$, $\omega=\omega_{ci}$')
ax_CMA.axvline(
    x=1,
    color='darkcyan', linestyle=':', label='$u_O=\infty$, $\omega=\omega_{pe}$')
draw_discontinuable_expr(
    [sol.subs(m_i, m_i_N).subs(m_e, m_e_N)
     for sol in solve(Eq(resonance_points_as_Eq_1[1][1], 1), Y)], X, # [1][3] is also okay
    varlim = (1e-3, 1e7), exprlim=(1e-5, None), num=500,
    var_sample_scale='log', fig=fig_CMA, ax=ax_CMA, labels=['$u_X=0$, $\omega=\omega_{UH}$', '$u_X=0$, $\omega=\omega_{LH}$']
)
draw_discontinuable_expr(
    [sol.subs(m_i, m_i_N).subs(m_e, m_e_N)
     for sol in solve(Eq(cutoff_points_as_Eq_1[0], 1), Y)], X,
    varlim = (1e-3, 1e7), exprlim=(1e-5, None), num=500,
    var_sample_scale='log', fig=fig_CMA, ax=ax_CMA, labels=['$u_R=\infty$, $\omega=\omega_{R}$', '$u_L=\infty$, $\omega=\omega_{L}$']
)

ax_CMA.legend(prop={'size': 20})
plt.close(fig_CMA)
```

    <string>:2: RuntimeWarning: invalid value encountered in sqrt
    <string>:2: RuntimeWarning: invalid value encountered in sqrt

```python
from matplotlib.patches import Circle
from matplotlib.offsetbox import (TextArea, DrawingArea, OffsetImage,
                                  AnnotationBbox)
from matplotlib.cbook import get_sample_data

for i, (B, n_0, omega) in enumerate(plasma_B_n_0_omega):

    with get_sample_data((data_path / f"v_ph_{i}.png").absolute()) as file:
        arr_img = plt.imread(file, format='png')

    imagebox = OffsetImage(arr_img, zoom=0.28)
    imagebox.image.axes = ax_CMA
    imagebox
    x_CMA = w2N((omega_pe**2 + omega_pi**2) / omega**2)
    y_CMA = w2N(omega_ce * omega_ci / omega**2)
    xy_CMA = (x_CMA, y_CMA)
    print(xy_CMA)
    ab = AnnotationBbox(imagebox, xy_CMA,
                        xybox=(150., -200.),
                        xycoords='data',
                        boxcoords="offset points",
                        pad=0.5,
                        arrowprops=dict(
                            arrowstyle="->",
                            connectionstyle="angle,angleA=0,angleB=90,rad=3")
                        )

    ax_CMA.add_artist(ab)
    print(f"The {i}-th phase speed polar plot.")
fig_CMA
```

### References:

- For better color impression, [matplotlib official color gallery](https://matplotlib.org/3.1.0/gallery/color/named_colors.html) can ben refered.


# Practical Session 3: Ensemble Learning Techniques

*Notebook by Ekaterina Kochmar*

This practical will address the use of ensemble-based learning techniques. You will be working with the otherwise familiar settings of classification and regression tasks. In this practical, you will use the familiar datasets and will also learn how to generate artificial datasets.

## Learning objectives

In this practical you will learn about:
- simple voting classifiers using hard and soft voting strategies
- bagging and pasting ensembles
- boosting and early stopping
- popular ensemble-based learning algorithms including `RandomForests` and `AdaBoost`

In addition, you will get more practice with `sklearn's` machine learning routines, and you will learn how to implement ensembles both from scratch and using `sklearn's` implementation.

**Bibliography**: Aurelien Geron, *Hands-On Machine Learning with Scikit-Learn and TensorFlow*.

## Setup

First of all, let's import a few common modules, ensure `matplotlib` plots are inline and define the parameters for the figures. Feel free to add your own settings for the notebook here:

```python
# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
np.random.seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
```

## Simple voting classifiers: hard and soft voting strategies

[The wisdom of the crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd) is the collective opinion of a group of individuals rather than that of a single expert. It has been shown that if you ask a complex question to thousands of random people and then aggregate their answers, in many cases you are likely to get an answer that is more accurate than an expert's prediction. This effect is nowadays widely used in practice and can be said to provide the basis for crowdsourcing approaches.

By analogy, if you combine decisions of multiple diverse classifiers, the resulting vote will often be better than a vote of any single predictor. Such combination of classifiers is called _ensemble_. Perhaps somewhat counterintuitively, the aggregated vote of an ensemble often achieves a higher accuracy than the best classifier in the ensemble. This works even if each predictor in the ensemble is a _weak learner_ (meaning that it works only slightly better than random guessing). The ensemble can still be a _strong learner_ achieving high accuracy, provided that there are:
- a sufficient number of weak learners, and
- their decisions are sufficiently diverse.

How is this possible? Here is a statistical analogy: suppose you have a slightly biased coin that has a $51\%$ chance of coming up heads and a $49\%$ chance of coming up tails. If you toss it $1000$ times, you will get around $510$ heads and $490$ tails, and hence a majority of heads. If you do the math or run simulations, you will find out that the probability of obtaining the majority of heads after $1000$ tosses of this coin approaches $73\%$. After $10K$ tosses, the probability of having a majority of heads is about $97\%$. This is explained by the [_law of large numbers_](https://en.wikipedia.org/wiki/Law_of_large_numbers): over a large number of tosses the ratio of heads gets closer to the probability of heads ($51\%$).

```python
import scipy, math
import scipy.special

def probability(p, n, x):
    binom = scipy.special.comb(n, x, exact=True) * math.pow(p, x) * math.pow((1-p), n-x)
    return binom

prob_maj = 0.0
for x in range(501, 1001):
    prob_maj += probability(0.51, 1000, x)

print(prob_maj)
```

The code below simulates $10$ independent series of coin tosses, and then plots the results. You can see in the figure below that as the number of tosses increases, all $10$ series end up close to $51\%$ and thus are consistently above $50\%$.

```python
np.random.seed(42)
heads_proba = 0.51
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)

plt.figure(figsize=(8,3.5))
plt.plot(cumulative_heads_ratio)
plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
plt.xlabel("Number of coin tosses")
plt.ylabel("Heads ratio")
plt.legend(loc="lower right")
plt.axis([0, 10000, 0.42, 0.58])
plt.show()
```

Similarly, if you combine $1000$ classifiers, each of which is individually correct only $51\%$ of the time, and then predict the majority voted class, you can get up to $73\%$ accuracy.

For this to work, the classifiers need to be independent and sufficiently diverse, i.e. they need to make independent, uncorrelated errors. In practice, this is hard to achieve because the classifiers are usually all trained on the same data.

Let's apply voting strategy to some data: this time, we'll be working with a synthetic (i.e., artificially generated) [`moons` dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html), that includes two interleaving half circles and provides a good 'toy' example for testing out classification strategies.

Let's first generate the data points and visualise them on a plot:

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)

def plot_dataset(X, y, axes):
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
    plt.axis(axes)
    plt.grid(True, which='both')
    plt.xlabel(r"$x_1$", fontsize=20)
    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)

plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.show()
```

Next, let's split the dataset into training and test sets and train some classifiers on the training data:

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

The code below shows how to apply several classifiers to this data, with a couple of classifiers that we haven't used before in this course: [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and [`Support Vector Machines`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) (`SVC`) classifier. For the moment, don't worry about the particular details of the classifiers' implementation (if you are interested in learning more about them, check out `sklearn's` documentation; we'll discuss `RandomForestClassifier` in this practical; `Support Vector Machines` is covered in detail in the Part II [Machine Learning and Bayesian Inference](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/) course). The point here is that we are combining *diverse* classifiers in an ensemble: in essence, `RandomForestClassifier` runs recursive partitioning on the data, while `Support Vector Machines` try to construct a linear boundary between the classes (as discussed in Lecture 3). Feel free to experiment with your own selection of classifiers from the [`sklearn's`](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) suite.

```python
from sklearn.ensemble import VotingClassifier

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf.fit(X_train, y_train)
```

Let's look into how correlated the predictions of the classifiers are: first, get the predictions on the test data, next store them as a `pandas` DataFrame, and finally apply `.corr()` function:

```python
import pandas as pd

def get_predictions(clf):
    clf.fit(X_train, y_train)
    return clf.predict(X_test)

preds = {'lr': get_predictions(log_clf),
        'rf': get_predictions(rnd_clf),
        'svc': get_predictions(svm_clf)}
df = pd.DataFrame(data=preds)
df[:100]
```

```python
df.corr()
```

Will this correlation in the individual classifiers' predictions be sufficient for an ensemble? Let's check this out by combining the votes with a *hard voting strategy*: the ensemble classifier will simply choose the majority class predicted by the three classifiers.

The code below prints out individual classifiers' accuracy scores along with the ensemble's accuracy score:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

As you can see, the voting classifier slightly outperforms each of the three individual classifiers, including the most accurate one(s). In addition, if individual classifiers can estimate class probabilities (i.e., if they have `predict_proba()` method), you can use _soft voting_ strategy, that is, estimate highest class probability averaged over individual classifiers. In comparison to the hard voting strategy, soft voting gives more weight to highly confident votes.

`SVC` classifier doesn't estimate class probabilities by default, so you need to set the `probability` hyperparameter to `True`.

```python
log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)
```

Not let's estimate the accuracy of the voting classifier in this mode:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

## Bagging ensembles: bagging and pasting

One way to ensure that the classifiers are sufficiently diverse in their predicitons is to select different training algorithms. As we said before, often it is hard to ensure that the classifiers' errors are not highly correlated because the classifiers are ultimately trained on the same data.

So, another way to make sure you end up with diverse classifiers is to train them on different random subsets of the training set. There are two ways to do that:
- sampling _with replacement_ is called [_bagging_](https://link.springer.com/article/10.1023/A:1018054314350) – short for _bootstrap aggregating_
- sampling _without replacement_ is called [_pasting_](https://link.springer.com/article/10.1023/A:1007563306331)

Both bagging and pasting allow training instances to be sampled several times across multiple predictors, but only bagging allows training instances to be sampled several times for the same predictor.

When applied to a new, test instance the ensemble classifier aggregates the predictions of all predictors and estimates the statistical mode (i.e., most frequent prediction).

**Question**: Is this similar to the hard or the soft voting strategy?

Let's use `sklearn's BaggingClassifier` to train an ensemble of $500$ [`Decision Trees`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html), each trained on $100$ training instances randomly selected from the training set with replacement (for bagging, but you can change the strategy to pasting by setting `bootstrap` to `False`). In addition, both bagging and pasting strategies are good for parallelisation: predictors can all be trained in parallel on different CPU cores or even different servers. Setting `n_jobs` parameter to $-1$ tells `sklearn` to use all available CPU cores:

```python
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

#use bootstrap=False for pasting
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
#n_jobs = use all of the available CPU cores
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

**Question**: [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?

Let's calculate the accuracy score for the bagging classifier that combines $500$ Decision Tree estimators with a prediction of a single Decision Tree trained on the same data:

```python
print(accuracy_score(y_test, y_pred))
```

```python
tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))
```

To get more insight into the results, let's also plot the decision boundary for the two classifiers:

```python
from matplotlib.colors import ListedColormap

def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

plt.figure(figsize=(11,4))
plt.subplot(121)
plot_decision_boundary(tree_clf, X, y)
plt.title("Decision Tree", fontsize=14)
plt.subplot(122)
plot_decision_boundary(bag_clf, X, y)
plt.title("Decision Trees with Bagging", fontsize=14)
plt.show()
```

Even though the ensemble makes a comparable number of errors on the training set as a single Decision Tree, its decision boundary is less irregular. This suggests that ensemble's predictions will likely generalise better than the single `Decision Tree`'s predictions
when applied to the new data.

### Out-of-Bag evaluation

With bagging, some of the instances get sampled several times for any given predictor, while others may never be selected at all. By default, `BaggingClassifier` samples $m$ training instances with replacement, where $m$ is the size of the training set. As $m$ grows, the ratio of the training instances that are sampled on average for each predictor approaches $1 - exp(-1)$, i.e. around $63\%$. The remaining $37\%$ of the training instances that are never used by the predictors are called *out-of-bag* (*oob*) instances.

Mathematical proof of the above: With $m$ training instances in the set, the probability of not picking an instance in a random draw is $\frac{m-1}{m}$. For sampling with replacement, the probability of not picking an instance in $m$ random draws is $(\frac{m-1}{m})^m$. In the limit of a large number of $m$, this becomes:

\begin{equation}
\lim_{m\to\infty} (1 - \frac{1}{m})^m = e^{-1} = 0.368 \approx 37\%
\end{equation}

Since a predictor never sees these instances during training, they can be used for evaluation without the need for an additional validation set or cross-validation experiments. You can evaluate the ensemble itself by averaging each predictor's performance on the oob instances.

The code below shows you how to do that: if you set `oob_score` to `True`, the bagging classifier will be evaluated on the oob instances after training, and you can output the score as `oob_score_`.

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_
```

This means that the bagging classifier is likely to achieve around $90\%$ accuracy on the test data, too. Let's check the results:

```python
from sklearn.metrics import accuracy_score
y_pred = bag_clf.predict(X_test)
accuracy_score(y_test, y_pred)
```

Close enough! You can also take a look into the decision process by printing out the predictions of the classifier on the training instances. Since the base estimator has a `predict_proba()` method, what you see here are the class probabilities assigned by the classifier to negative and positive classes for each instance:

```python
bag_clf.oob_decision_function_
```

## Random Forests

[`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) is an ensemble of Decision Trees typically trained via the bagging method. The number of training instances (`max_samples`) is usually set to the total size of the training set. So, in fact, `RandomForestClassifier` will be roughly equivalent to the `BaggingClassifier` that takes Decision Trees as base estimators with the following parameters:

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
    n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

The key difference is that, rather than using `BaggingClassifier` and passing it a `DecisionTreeClassifier`, you can rely on the `sklearn's RandomForestClassifier` implementation, which is more convenient and optimised for Decision Trees:

```python
from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,
                                 n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)

y_pred_rf = rnd_clf.predict(X_test)
```

Let's estimate the difference between the two classifiers' predictions:

```python
np.sum(y_pred == y_pred_rf) / len(y_pred)  # see to what extent predictions are identical
```

I.e., the two classifiers are almost identical in their predictions. The Random Forest algorithm introduces extra randomness when growing trees: instead of searching for the very best feature when splitting a node, it searches for the best feature among a random subset of features. As a result, the model is more diverse and in general yields better results.

Let's visualise the decision boundary for a random set of $15$ Decision Trees to get a better idea of where the diversity comes from:

```python
plt.figure(figsize=(6, 4))

for i in range(15):
    tree_clf = DecisionTreeClassifier(max_leaf_nodes=16, random_state=42 + i)
    indices_with_replacement = np.random.randint(0, len(X_train), len(X_train))
    tree_clf.fit(X[indices_with_replacement], y[indices_with_replacement])
    plot_decision_boundary(tree_clf, X, y, axes=[-1.5, 2.5, -1, 1.5],
                           alpha=0.02, contour=False)

plt.show()
```

**Extra material**: One further way to introduce randomness into the training process is to not only select a random subset of features when splitting a node, but also use a random threshold for each feature rather than searching for the best possible thresholds as traditional Decision Trees do. Since estimating best possible threshold for each feature at every node is also one of the most time-consuming tasks, this modification to the algorithm makes it more efficient. The algorithm that exploits this idea is called *Extremely Randomised Trees ensemble*, or *Extra-Trees*, and you can use it via [`sklearn's ExtraTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.ExtraTreeClassifier.html) implementation.

### Feature importance

Another useful property of the Random Forests classifier is that it can help you measure the relative importance of each feature by looking at how much the tree nodes that use this particular feature reduce impurity of the nodes on average (i.e., across all trees in the forest). The results are scaled so that the sum of all feature importances is equal to $1$.

Because features used in the *iris* and *digits* datasets are highly interpretable, let's estimate feature importances on these two datasets:

```python
from sklearn.datasets import load_iris
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=42)
rnd_clf.fit(iris["data"], iris["target"])
for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
    print(name, score)
```

```python
rnd_clf.feature_importances_
```

**Question**: What do these results suggest? Are they similar to your observations from the previous experiments on this dataset?

*Digits* dataset provides further interpretability of the results if we plot them. What does the figure below suggest?

```python
from sklearn import datasets
digits = datasets.load_digits()

rnd_clf = RandomForestClassifier(random_state=42)
rnd_clf.fit(digits["data"], digits["target"])
```

```python
def plot_digit(data):
    image = data.reshape(8, 8)
    plt.imshow(image, cmap = matplotlib.cm.hot,
               interpolation="nearest")
    plt.axis("off")

plot_digit(rnd_clf.feature_importances_)

cbar = plt.colorbar(ticks=[rnd_clf.feature_importances_.min(), rnd_clf.feature_importances_.max()])
cbar.ax.set_yticklabels(['Not important', 'Very important'])

plt.show()
```

## AdaBoost

*Boosting*, or *hypothesis boosting* is an ensemble method that can combine several weak learners into a strong learner. The general idea is to train predictors sequentially in such a way that each next predictor tries to correct its predecessor. The most popular among boosting methods are *AdaBoost* (which stands for *Adaptive Boosting*) and *Gradient Boosting*. Let's apply *AdaBoost* first.

The idea behind *AdaBoost* is as follows: in order to correct the predecessor's mistakes, the algorithm assigns higher weights to the training instances that the predecessor underfitted. I.e., the next step pays more attention to such instances and the algorithm focuses more on the hard cases.

Here is the step-by-step strategy:
- start with the first classifier (i.e., this can be a Decision Trees classifier)
- train it and use it to make predictions on the training set
- increase the relative weight of misclassified training instances
- train a second classifier with these new updated weights and make new predictions
- update weights using the new predictions
- continue until stopping criteria are satisfied.

For instance, suppose each instance's original weight $w^{(i)}$ is set to $\frac{1}{m}$, where $m$ is the number of instances. The first classifier is applied, and its error rate $r_1$ is computed on the training set using the equation below:

\begin{equation}
r_j = \frac{\sum_{\hat{y}^{(i)}_j \neq y^{(i)}} w^{(i)}}{\sum_{i=1}^m w^{(i)}}
\end{equation}

where $\hat{y}^{(i)}_j$ is the prediction of the $j$-th classifier on the $i$-th instance. The predictor's weight $\alpha_j$ is then estimated using:

\begin{equation}
\alpha_j = \eta log \frac{1-r_j}{r_j}
\end{equation}

where $\eta$ is the learning rate, a hyperparameter that defaults to $1$. The more accurate the predictor is, the higher its weight will be; if a predictor is guessing randomly, its weight will be close to $0$; and if it performs worse than random guessing it will get a high negative weight.

Next, the weights are updated and the misclassified instances are boosted as follows:

\begin{equation}
  w^{(i)}=\begin{cases}
    w^{(i)}, & \text{if $\hat y_j^{(i)} = y_j^{(i)}$}\\
    w^{(i)} exp(\alpha_j), & \text{if $\hat y_j^{(i)} \neq y_j^{(i)}$}
  \end{cases}
\end{equation}

Then all the instances weights are normalised (i.e., divided by $\sum_{i=1}^m w^{(i)}$). The new predictor is trained on the updated training instances, applied to the training set, its weight is computed, weights are updated again, and so on. The algorithm stops when either the predefined number of predictors is reached or a perfect predictor is found.

(This process should remind you about other sequential learning techniques we've discussed, e.g. Gradient Descent. The difference is, instead of tweaking a single predictor's parameters to minimise a cost function, AdaBoost adds predictors to the ensemble gradually making the prediction better.)

**Question**: We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?

At prediction time, AdaBoost computes the predictions of all the predictors and weighs them according to $\alpha_j$. The predicted class is then the one that receives the majority of weighted votes, i.e.:

\begin{equation}
 \hat y(x) = argmax_{k} \sum_{j=1; \hat y_j(x)=k} ^{N} \alpha_j
\end{equation}

with $N$ being the number of predictors.

`sklearn's` implementation of AdaBoost uses *Stagewise Additive Modeling using a Multiclass Exponential loss function* (or *SAMME* for short) multiclass version of the algorithm, which in a binary case simply defaults to AdaBoost. In addition, if the base estimators can estimate class probabilities (have `predict_proba()` method), `sklearn's` algorithm relies on class probabilities rather than predictions, which generally performs better. This version is called *SAMME.R*, where *R* stands for "Real".

Let's train [`sklearn's AdaBoostClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html), for example using $200$ *Decision Stumps* (trees composed of a single decision node and two leaf nodes, `max_depth=1`):

```python
from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)
```

And let's plot the decision boundary of the algorithm:

```python
plot_decision_boundary(ada_clf, X, y)
```

To get a bit more insight into how the decision boundaries change from one step to another and with a different learning rate, let's plot decision boundaries for $5$ consecutive predictors, in this case using `SVC` as the estimator. Note how the first classifiers usually get many instances wrong, while the following predictors are gradually getting better. The plot on the right presents the very same $5$ consecutive classifiers, but assigns half the learning rate (i.e., the misclassified instances weights are only boosted half as much at every iteration):

```python
m = len(X_train)

plt.figure(figsize=(11, 4))
for subplot, learning_rate in ((121, 1), (122, 0.5)):
    sample_weights = np.ones(m)
    plt.subplot(subplot)
    for i in range(5):
        svm_clf = SVC(kernel="rbf", C=0.05, gamma="auto", random_state=42)
        svm_clf.fit(X_train, y_train, sample_weight=sample_weights)
        y_pred = svm_clf.predict(X_train)
        sample_weights[y_pred != y_train] *= (1 + learning_rate)
        plot_decision_boundary(svm_clf, X, y, alpha=0.2)
        plt.title("learning_rate = {}".format(learning_rate), fontsize=16)
    if subplot == 121:
        plt.text(1.70,  -0.90, "1", fontsize=14)
        plt.text(-0.40, -0.35, "2", fontsize=14)
        plt.text(-0.55, -0.05, "3", fontsize=14)
        plt.text(-0.70,  0.20, "4", fontsize=14)
        plt.text(-0.85,  0.45, "5", fontsize=14)

plt.show()
```

In addition, here is the full list of parameters and attributes of the algorithm:

```python
list(m for m in dir(ada_clf) if not m.startswith("_") and m.endswith("_"))
```

E.g., you can check classification errors for each estimator in the boosted ensemble as follows:

```python
ada_clf.estimator_errors_
```

## Gradient Boosting

Gradient Boosting is another popular boosting algorithm. It also works sequentially adding new predictors to the ensemble, each one correcting the errors from its predecessor. Unlike AdaBoost, it's trying to fit each new predictor to the *residual errors* made by the previous predictor.

Let's this time apply the boosting algorithm to a regression task. First, generate a synthetic noisy quadratic training set:

```python
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
```

Next, let's fit a single [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html):

```python
from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
```

Now train a second `DecisionTreeRegressor` on the residual errors made by the first predictor:

```python
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
```

Finally, let's train a third regressor on the residual errors made by the second predictor:

```python
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
```

That's it – you've just built an ensemble containing three trees. It can now be applied to new instances by making predictions based on adding up the predictions from all three trees:

```python
X_new = np.array([[0.8]])

y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))

y_pred
```

Let's visualise the results by plotting the predictions from each tree on the left and the ensemble on the right. E.g., in the first row the ensemble contains only one tree, so its predictions are exactly the same as the first tree's predictions. In the second row, the second tree is trained on the residual errors from the first tree, and the ensemble's prediction is based on the sum of the predictions of the two trees. You can see how the predictions of the ensemble are gradually getting better:

```python
def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
    x1 = np.linspace(axes[0], axes[1], 500)
    y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
    plt.plot(X[:, 0], y, data_style, label=data_label)
    plt.plot(x1, y_pred, style, linewidth=2, label=label)
    if label or data_label:
        plt.legend(loc="upper center", fontsize=16)
    plt.axis(axes)

plt.figure(figsize=(11,11))

plt.subplot(321)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h_1(x_1)$", style="g-", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Residuals and tree predictions", fontsize=16)

plt.subplot(322)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1)$", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Ensemble predictions", fontsize=16)

plt.subplot(323)
plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
plt.ylabel("$y - h_1(x_1)$", fontsize=16)

plt.subplot(324)
plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.subplot(325)
plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_3(x_1)$", style="g-", data_style="k+")
plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
plt.xlabel("$x_1$", fontsize=16)

plt.subplot(326)
plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
plt.xlabel("$x_1$", fontsize=16)
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.show()
```

A simpler way to train ensembles of *Gradient Boosted Regression Trees* is to use [`sklearn's GradientBoostingRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) implementation. In addition to the typical hyperparameters that control the growth of the trees, it also includes hyperparameters that control the ensemble training, e.g. `n_estimators`:

```python
from sklearn.ensemble import GradientBoostingRegressor

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
gbrt.fit(X, y)
```

**Question**: Another hyperparameter is `learning_rate`. What is it responsible for and what effect on the resulting ensemble does it have?

Feel free to check `sklearn's` documentation, as well as look into the comparison of two ensembles: the one with a lower number of estimators and a higher learning rate, and a 'slow learner' with a lower learning rate but a higher number of estimators:

```python
gbrt_slow = GradientBoostingRegressor(max_depth=2, n_estimators=200, learning_rate=0.1, random_state=42)
gbrt_slow.fit(X, y)
```

```python
plt.figure(figsize=(11,4))

plt.subplot(121)
plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="Ensemble predictions")
plt.title("learning_rate={}, n_estimators={}".format(gbrt.learning_rate, gbrt.n_estimators), fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_slow], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("learning_rate={}, n_estimators={}".format(gbrt_slow.learning_rate, gbrt_slow.n_estimators), fontsize=14)

plt.show()
```

**Question**: What generalisation behaviour can you expect from each of these ensembles?

### Gradient Boosting with Early Stopping

The examples above demonstrate ensembles with a different number of estimators. Which one is better? How can you quantitatively estimate the optimal number of predictors?

One technique that you can use is called *early stopping* – it allows your algorithm to stop as soon as the validation error reaches a minimum. An easy way to implement this with `sklearn` is to use `staged_predict()` method as the code below demonstrates:

```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=100)

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
gbrt.fit(X_train, y_train)

errors = [mean_squared_error(y_val, y_pred)
          for y_pred in gbrt.staged_predict(X_val)]
bst_n_estimators = np.argmin(errors)

gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators, random_state=42)
gbrt_best.fit(X_train, y_train)
```

Let's visualise the validaton errors and the optimal number of estimators:

```python
min_error = np.min(errors)

plt.figure(figsize=(11, 4))

plt.subplot(121)
plt.plot(errors, "b.-")
plt.plot([bst_n_estimators, bst_n_estimators], [0, min_error], "k--")
plt.plot([0, 120], [min_error, min_error], "k--")
plt.plot(bst_n_estimators, min_error, "ko")
plt.text(bst_n_estimators, min_error*1.2, "Minimum", ha="center", fontsize=14)
plt.axis([0, 120, 0, 0.01])
plt.xlabel("Number of trees")
plt.title("Validation error", fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("Best model (%d trees)" % bst_n_estimators, fontsize=14)

plt.show()
```

Finally, an alternative to training a large number of estimators and then looking back in order to find the optimal number of those is to allow the algorithm to stop training when the validation error does not improve over a number of consecutive iterations (e.g., $5$ in the code below). For that to work, set the `warm_start` to `True`:

```python
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)

min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping

print(gbrt.n_estimators)
```

# Your tasks

1. Run the code in the notebook. During the practical session, be prepared to discuss the methods and answer the questions from this notebook:

    * When applied to a new, test instance bagging classifier aggregates the predictions of all predictors and estimates the statistical mode. Is this similar to the hard or the soft voting strategy?
    * [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?
    * What do the feature importance weights obtained on the *iris* dataset suggest? Are they similar to your observations from the previous experiments on this dataset?
    * We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?
    * One of the hyperparameters of the boosting algorithms is learning_rate. What is it responsible for and what effect on the resulting ensemble does it have?
    * What generalisation behaviour can you expect from each of the tree regressor ensembles (using different learning rates and different number of estimators – see above)?

2. Apply ensemble techniques of your choice to one of the datasets you've worked on during the previous practicals and report your findings.

3. **Optional**: If you want more practice with these techniques, try implementing a [*stacking algorithm*](https://en.wikipedia.org/wiki/Ensemble_learning#Stacking). There is no available implementation of this approach in `sklearn` so it needs to be implemented from scratch. The idea is as follows:
    * Split a dataset of your choice into three subsets – training, validation and test.
    * Train a number of predictors on the training data and apply them to the validation data.
    * Treat the predictions of those predictors on the validation data to generate new training set: you can use the predictions of the estimators as new features (i.e., each training instance will have as many features as the number of predictors you originally used), and the validation data targets as the new training instances targets.
    * Now train a *blender* – a classifier of your choice – on the new training data created this way. Together with the original classifiers, the blender forms a stacking ensemble.
    * Finally, apply your original classifiers to the test set, feed their predictions to the blender, and use blender's output as the prediction on the test set.

```python

```


# Descriptive Statistics.

Working with dataset:
- https://archive.ics.uci.edu/ml/datasets/Vertebral+Column

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

Vamos a ver aquí algunas medidas iniciales para el análisis de datos. El análisis inicial de los datos es muy importante para hacer estimaciones y sacar conclusiones.

```python
names=["pelvic_incidence","pelvic_tilt","lumbar_lordosis_angle","sacral_slope","pelvic_radius","degree_spondylolisthesis","class"]
data=pd.read_csv("../Datas/VertebralColumn/column_2C.dat",delimiter="\s+",names=names)
```

```python
data
```

      pelvic_incidence
      pelvic_tilt
      lumbar_lordosis_angle
      sacral_slope
      pelvic_radius
      degree_spondylolisthesis
      class

      0
      63.03
      22.55
      39.61
      40.48
      98.67
      -0.25
      AB

      1
      39.06
      10.06
      25.02
      29.00
      114.41
      4.56
      AB

      2
      68.83
      22.22
      50.09
      46.61
      105.99
      -3.53
      AB

      3
      69.30
      24.65
      44.31
      44.64
      101.87
      11.21
      AB

      4
      49.71
      9.65
      28.32
      40.06
      108.17
      7.92
      AB

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      305
      47.90
      13.62
      36.00
      34.29
      117.45
      -4.25
      NO

      306
      53.94
      20.72
      29.22
      33.22
      114.37
      -0.42
      NO

      307
      61.45
      22.69
      46.17
      38.75
      125.67
      -2.71
      NO

      308
      45.25
      8.69
      41.58
      36.56
      118.55
      0.21
      NO

      309
      33.84
      5.07
      36.64
      28.77
      123.95
      -0.20
      NO

310 rows × 7 columns

Podemos hacer primero una visión general de nuestros datos para entender cual es el compotamiento de las variables que tenemos en el estudio. Esto podemos hacerlo muy fácilmente con Python usando una librería de pandas y usando algunos de los algoritmos y funciones predefinidas.

```python
print("El número de variables es N_var =",len(data.columns))
print("El número de casos o instancias (o individuos estudiados) es N =",len(data))
```

    El número de variables es N_var = 7
    El número de casos o instancias (o individuos estudiados) es N = 310

Algunos estdísticos comunes y usuales. Por ejemplo podemos calcular la media, mediana y moda, así como también las desviaciones estándar y varianzas para cada una de las 7 variables.

- Media (Promedio):
\begin{equation}
\bar x=\frac{1}{N}\sum_{i=0}^{N-1}x_i
\end{equation}
- Mediana (Cuartil 50): Con datos ordenados en orden ascendente:
\begin{equation}
median=\left\{\begin{matrix}
x\left[\frac{N}{2}\right] &  \mbox{si $N$ es par}\\
\frac{1}{2}\left(x\left[\frac{N-1}{2}\right]+x\left[\frac{N+1}{2}\right]\right) &  \mbox{si $N$ es impar}\\
\end{matrix}\right.
\end{equation}
- Moda (dato con mayor frecuencia)
- varianza:
\begin{equation}
s^2(x)=\mbox{Var}(x)=\frac{1}{N}\sum_{i=0}^{N-1}(x_i-\bar x)^2
\end{equation}
- Desviación estándar:
\begin{equation}
\sigma_x=s(x)=\sqrt{\mbox{Var}(x)}=\sqrt{\frac{1}{N}\sum_{i=0}^{N-1}(x_i-\bar x)^2}
\end{equation}

```python
def average(var):
    return np.sum(var)/len(var)
def median(var):
    var_ord=np.sort(var)
    N=len(var_ord)
    if N%2==0:
        median=var_ord[N//2]
    if N%2!=0:
        median=(var_ord[(N-1)//2]+var_ord[(N+1)//2])/2
    return median
def mode(var):
    dat,frec=np.unique(var,return_counts=True)
    if len(dat[frec==frec.max()])>1:
        print("Existen varias modas (la variable es multimodal)")
    if len(dat[frec==frec.max()])==1:
        print("Existe una moda (la variable es unimodal)")
    return dat[frec==frec.max()]
```

```python
def variance(var):
    return np.sum((var-average(var))**2)/len(var)
def std(var):
    return np.sqrt(variance(var))
```

```python
for i in data.columns[:-1]:
    print("Para la variable "+str(i),"el promedio (la media) es <{}> =".format(str(i)),average(data[i]))
```

    Para la variable pelvic_incidence el promedio (la media) es <pelvic_incidence> = 60.49648387096773
    Para la variable pelvic_tilt el promedio (la media) es <pelvic_tilt> = 17.542903225806448
    Para la variable lumbar_lordosis_angle el promedio (la media) es <lumbar_lordosis_angle> = 51.93070967741936
    Para la variable sacral_slope el promedio (la media) es <sacral_slope> = 42.953870967741935
    Para la variable pelvic_radius el promedio (la media) es <pelvic_radius> = 117.92054838709676
    Para la variable degree_spondylolisthesis el promedio (la media) es <degree_spondylolisthesis> = 26.296741935483873

```python
for i in data.columns[:-1]:
    print("Para la variable "+str(i),"la mediana es med({}) =".format(str(i)),median(data[i]))
```

    Para la variable pelvic_incidence la mediana es med(pelvic_incidence) = 58.78
    Para la variable pelvic_tilt la mediana es med(pelvic_tilt) = 16.42
    Para la variable lumbar_lordosis_angle la mediana es med(lumbar_lordosis_angle) = 49.78
    Para la variable sacral_slope la mediana es med(sacral_slope) = 42.44
    Para la variable pelvic_radius la mediana es med(pelvic_radius) = 118.34
    Para la variable degree_spondylolisthesis la mediana es med(degree_spondylolisthesis) = 12.07

```python
for i in data.columns[:-1]:
    print("Para la variable "+str(i),"la moda es moda({}) =".format(str(i)),mode(data[i]))
```

    Existen varias modas (la variable es multimodal)
    Para la variable pelvic_incidence la moda es moda(pelvic_incidence) = [42.52 49.71 50.91 53.94 54.92 63.03 65.01 65.76 74.72]
    Existen varias modas (la variable es multimodal)
    Para la variable pelvic_tilt la moda es moda(pelvic_tilt) = [ 5.27  8.4  10.06 10.22 10.76 13.11 13.28 13.92 14.38 15.4  16.42 16.74
     17.44 19.44 21.12 23.08 26.33 33.28 37.52]
    Existen varias modas (la variable es multimodal)
    Para la variable lumbar_lordosis_angle la moda es moda(lumbar_lordosis_angle) = [35. 42. 47. 52. 58.]
    Existe una moda (la variable es unimodal)
    Para la variable sacral_slope la moda es moda(sacral_slope) = [56.31]
    Existen varias modas (la variable es multimodal)
    Para la variable pelvic_radius la moda es moda(pelvic_radius) = [110.71 116.56 116.59 116.8  117.98 119.32 129.39]
    Existen varias modas (la variable es multimodal)
    Para la variable degree_spondylolisthesis la moda es moda(degree_spondylolisthesis) = [-4.08 -2.01  1.01  3.09  4.96]

```python
for i in data.columns[:-1]:
    print("Para la variable "+str(i),"la varianza es Var({}) =".format(str(i)),variance(data[i]))
```

    Para la variable pelvic_incidence la varianza es Var(pelvic_incidence) = 296.12512021748176
    Para la variable pelvic_tilt la varianza es Var(pelvic_tilt) = 99.83976124869929
    Para la variable lumbar_lordosis_angle la varianza es Var(lumbar_lordosis_angle) = 343.13176723829343
    Para la variable sacral_slope la varianza es Var(sacral_slope) = 179.5889740478668
    Para la variable pelvic_radius la varianza es Var(pelvic_radius) = 176.7871277637877
    Para la variable degree_spondylolisthesis la varianza es Var(degree_spondylolisthesis) = 1406.119148417274

```python
for i in data.columns[:-1]:
    print("Para la variable "+str(i),"la desviación estándar es STD({}) =".format(str(i)),std(data[i]))
```

    Para la variable pelvic_incidence la desviación estándar es STD(pelvic_incidence) = 17.208286382364797
    Para la variable pelvic_tilt la desviación estándar es STD(pelvic_tilt) = 9.991984850303732
    Para la variable lumbar_lordosis_angle la desviación estándar es STD(lumbar_lordosis_angle) = 18.523816216921755
    Para la variable sacral_slope la desviación estándar es STD(sacral_slope) = 13.401081077579779
    Para la variable pelvic_radius la desviación estándar es STD(pelvic_radius) = 13.296132060256761
    Para la variable degree_spondylolisthesis la desviación estándar es STD(degree_spondylolisthesis) = 37.49825527164263

También podemos usar una función de Pandas que reune algunos estadísticos básicos:

```python
vertebral_stats=data.describe().T
```

```python
vertebral_stats
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      pelvic_incidence
      310.0
      60.496484
      17.236109
      26.15
      46.4325
      58.690
      72.8800
      129.83

      pelvic_tilt
      310.0
      17.542903
      10.008140
      -6.55
      10.6675
      16.360
      22.1200
      49.43

      lumbar_lordosis_angle
      310.0
      51.930710
      18.553766
      14.00
      37.0000
      49.565
      63.0000
      125.74

      sacral_slope
      310.0
      42.953871
      13.422748
      13.37
      33.3475
      42.405
      52.6925
      121.43

      pelvic_radius
      310.0
      117.920548
      13.317629
      70.08
      110.7100
      118.265
      125.4675
      163.07

      degree_spondylolisthesis
      310.0
      26.296742
      37.558883
      -11.06
      1.6000
      11.765
      41.2850
      418.54

también podemos usar las funciones que están predefinidas en Python para estas tareas:

```python
import scipy.stats as st
# La mediana puede ser también calculada, así como el coeficiente de asimetría y curtosis respectivamente
print(data.columns)
print("media =",np.mean(data.loc[:, data.columns != 'class'],axis=0))
print("median =",np.median(data.loc[:, data.columns != 'class'],axis=0))
print("varianza =",np.var(data.loc[:, data.columns != 'class'],axis=0))
print("desviación estándar =",np.std(data.loc[:, data.columns != 'class'],axis=0))
print("skewness =",st.skew(data.loc[:, data.columns != 'class'],axis=0))
print("kurtosis =",st.kurtosis(data.loc[:, data.columns != 'class'],axis=0))
```

    Index(['pelvic_incidence', 'pelvic_tilt', 'lumbar_lordosis_angle',
           'sacral_slope', 'pelvic_radius', 'degree_spondylolisthesis', 'class'],
          dtype='object')
    media = pelvic_incidence             60.496484
    pelvic_tilt                  17.542903
    lumbar_lordosis_angle        51.930710
    sacral_slope                 42.953871
    pelvic_radius               117.920548
    degree_spondylolisthesis     26.296742
    dtype: float64
    median = [ 58.69   16.36   49.565  42.405 118.265  11.765]
    varianza = pelvic_incidence             296.125120
    pelvic_tilt                   99.839761
    lumbar_lordosis_angle        343.131767
    sacral_slope                 179.588974
    pelvic_radius                176.787128
    degree_spondylolisthesis    1406.119148
    dtype: float64
    desviación estándar = pelvic_incidence            17.208286
    pelvic_tilt                  9.991985
    lumbar_lordosis_angle       18.523816
    sacral_slope                13.401081
    pelvic_radius               13.296132
    degree_spondylolisthesis    37.498255
    dtype: float64
    skewness = [ 0.51786409  0.67329913  0.5964695   0.78883655 -0.17607403  4.29696644]
    kurtosis = [ 0.2006561   0.64597047  0.13977901  2.94050906  0.90035703 37.4374569 ]

```python
vertebral_stats["median"]=np.median(data.loc[:, data.columns != 'class'],axis=0)
vertebral_stats["skewness"]=st.skew(data.loc[:, data.columns != 'class'],axis=0)
vertebral_stats["kurtosis"]=st.kurtosis(data.loc[:, data.columns != 'class'],axis=0)
```

```python
vertebral_stats
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max
      median
      skewness
      kurtosis

      pelvic_incidence
      310.0
      60.496484
      17.236109
      26.15
      46.4325
      58.690
      72.8800
      129.83
      58.690
      0.517864
      0.200656

      pelvic_tilt
      310.0
      17.542903
      10.008140
      -6.55
      10.6675
      16.360
      22.1200
      49.43
      16.360
      0.673299
      0.645970

      lumbar_lordosis_angle
      310.0
      51.930710
      18.553766
      14.00
      37.0000
      49.565
      63.0000
      125.74
      49.565
      0.596469
      0.139779

      sacral_slope
      310.0
      42.953871
      13.422748
      13.37
      33.3475
      42.405
      52.6925
      121.43
      42.405
      0.788837
      2.940509

      pelvic_radius
      310.0
      117.920548
      13.317629
      70.08
      110.7100
      118.265
      125.4675
      163.07
      118.265
      -0.176074
      0.900357

      degree_spondylolisthesis
      310.0
      26.296742
      37.558883
      -11.06
      1.6000
      11.765
      41.2850
      418.54
      11.765
      4.296966
      37.437457

```python

```

```python

```

Ahora entendamos el problema en el que nos embarcamos. Resulta que los datos pueden contener outliers, o en otras palabras, hay datos que parecen estar fuera del rango de valores del conjunto total. Veamos esto con otro conjunto de datos: https://archive.ics.uci.edu/ml/datasets/ionosphere

```python
#names=["pelvic_incidence","pelvic_tilt","lumbar_lordosis_angle","sacral_slope","pelvic_radius","degree_spondylolisthesis","class"]
data=pd.read_csv("../Datas/Ionosphere/ionosphere.data",delimiter=",",names=["Atr"+str(i) for i in range(35)])
```

```python
data
```

      Atr0
      Atr1
      Atr2
      Atr3
      Atr4
      Atr5
      Atr6
      Atr7
      Atr8
      Atr9
      ...
      Atr25
      Atr26
      Atr27
      Atr28
      Atr29
      Atr30
      Atr31
      Atr32
      Atr33
      Atr34

      0
      1
      0
      0.99539
      -0.05889
      0.85243
      0.02306
      0.83398
      -0.37708
      1.00000
      0.03760
      ...
      -0.51171
      0.41078
      -0.46168
      0.21266
      -0.34090
      0.42267
      -0.54487
      0.18641
      -0.45300
      g

      1
      1
      0
      1.00000
      -0.18829
      0.93035
      -0.36156
      -0.10868
      -0.93597
      1.00000
      -0.04549
      ...
      -0.26569
      -0.20468
      -0.18401
      -0.19040
      -0.11593
      -0.16626
      -0.06288
      -0.13738
      -0.02447
      b

      2
      1
      0
      1.00000
      -0.03365
      1.00000
      0.00485
      1.00000
      -0.12062
      0.88965
      0.01198
      ...
      -0.40220
      0.58984
      -0.22145
      0.43100
      -0.17365
      0.60436
      -0.24180
      0.56045
      -0.38238
      g

      3
      1
      0
      1.00000
      -0.45161
      1.00000
      1.00000
      0.71216
      -1.00000
      0.00000
      0.00000
      ...
      0.90695
      0.51613
      1.00000
      1.00000
      -0.20099
      0.25682
      1.00000
      -0.32382
      1.00000
      b

      4
      1
      0
      1.00000
      -0.02401
      0.94140
      0.06531
      0.92106
      -0.23255
      0.77152
      -0.16399
      ...
      -0.65158
      0.13290
      -0.53206
      0.02431
      -0.62197
      -0.05707
      -0.59573
      -0.04608
      -0.65697
      g

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      346
      1
      0
      0.83508
      0.08298
      0.73739
      -0.14706
      0.84349
      -0.05567
      0.90441
      -0.04622
      ...
      -0.04202
      0.83479
      0.00123
      1.00000
      0.12815
      0.86660
      -0.10714
      0.90546
      -0.04307
      g

      347
      1
      0
      0.95113
      0.00419
      0.95183
      -0.02723
      0.93438
      -0.01920
      0.94590
      0.01606
      ...
      0.01361
      0.93522
      0.04925
      0.93159
      0.08168
      0.94066
      -0.00035
      0.91483
      0.04712
      g

      348
      1
      0
      0.94701
      -0.00034
      0.93207
      -0.03227
      0.95177
      -0.03431
      0.95584
      0.02446
      ...
      0.03193
      0.92489
      0.02542
      0.92120
      0.02242
      0.92459
      0.00442
      0.92697
      -0.00577
      g

      349
      1
      0
      0.90608
      -0.01657
      0.98122
      -0.01989
      0.95691
      -0.03646
      0.85746
      0.00110
      ...
      -0.02099
      0.89147
      -0.07760
      0.82983
      -0.17238
      0.96022
      -0.03757
      0.87403
      -0.16243
      g

      350
      1
      0
      0.84710
      0.13533
      0.73638
      -0.06151
      0.87873
      0.08260
      0.88928
      -0.09139
      ...
      -0.15114
      0.81147
      -0.04822
      0.78207
      -0.00703
      0.75747
      -0.06678
      0.85764
      -0.06151
      g

351 rows × 35 columns

```python
len(data.columns)
```

    35

```python
plt.figure(figsize=(30,6))
data.iloc[:,2:].boxplot(grid=True,fontsize=15)
plt.legend(fontsize=16)
#plt.xticks(range(1,7),np.array(["pelvic \n incidence","pelvic \n tilt","lumbar \n lordosis \n angle","sacral \n slope","pelvic \n radius","degree \n spondylolisthesis"]),fontsize=16)
plt.yticks(fontsize=16)
plt.savefig("../../figures/ionosphere_boxplot1.png",bbox_inches ="tight")
plt.show()
```

```python
ionosphere_stats=data.describe().T
```

```python
ionosphere_stats["LII"]=ionosphere_stats["25%"]-((ionosphere_stats["75%"]-ionosphere_stats["25%"])*1.5)
ionosphere_stats["LIS"]=ionosphere_stats["75%"]+((ionosphere_stats["75%"]-ionosphere_stats["25%"])*1.5)
ionosphere_stats["LEI"]=ionosphere_stats["25%"]-((ionosphere_stats["75%"]-ionosphere_stats["25%"])*3.0)
ionosphere_stats["LES"]=ionosphere_stats["75%"]+((ionosphere_stats["75%"]-ionosphere_stats["25%"])*3.0)
```

```python
ionosphere_stats
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max
      LII
      LIS
      LEI
      LES

      Atr0
      351.0
      0.891738
      0.311155
      0.0
      1.000000
      1.00000
      1.000000
      1.0
      1.000000
      1.000000
      1.000000
      1.000000

      Atr1
      351.0
      0.000000
      0.000000
      0.0
      0.000000
      0.00000
      0.000000
      0.0
      0.000000
      0.000000
      0.000000
      0.000000

      Atr2
      351.0
      0.641342
      0.497708
      -1.0
      0.472135
      0.87111
      1.000000
      1.0
      -0.319663
      1.791797
      -1.111460
      2.583595

      Atr3
      351.0
      0.044372
      0.441435
      -1.0
      -0.064735
      0.01631
      0.194185
      1.0
      -0.453115
      0.582565
      -0.841495
      0.970945

      Atr4
      351.0
      0.601068
      0.519862
      -1.0
      0.412660
      0.80920
      1.000000
      1.0
      -0.468350
      1.881010
      -1.349360
      2.762020

      Atr5
      351.0
      0.115889
      0.460810
      -1.0
      -0.024795
      0.02280
      0.334655
      1.0
      -0.563970
      0.873830
      -1.103145
      1.413005

      Atr6
      351.0
      0.550095
      0.492654
      -1.0
      0.211310
      0.72873
      0.969240
      1.0
      -0.925585
      2.106135
      -2.062480
      3.243030

      Atr7
      351.0
      0.119360
      0.520750
      -1.0
      -0.054840
      0.01471
      0.445675
      1.0
      -0.805613
      1.196448
      -1.556385
      1.947220

      Atr8
      351.0
      0.511848
      0.507066
      -1.0
      0.087110
      0.68421
      0.953240
      1.0
      -1.212085
      2.252435
      -2.511280
      3.551630

      Atr9
      351.0
      0.181345
      0.483851
      -1.0
      -0.048075
      0.01829
      0.534195
      1.0
      -0.921480
      1.407600
      -1.794885
      2.281005

      Atr10
      351.0
      0.476183
      0.563496
      -1.0
      0.021120
      0.66798
      0.957895
      1.0
      -1.384043
      2.363058
      -2.789205
      3.768220

      Atr11
      351.0
      0.155040
      0.494817
      -1.0
      -0.065265
      0.02825
      0.482375
      1.0
      -0.886725
      1.303835
      -1.708185
      2.125295

      Atr12
      351.0
      0.400801
      0.622186
      -1.0
      0.000000
      0.64407
      0.955505
      1.0
      -1.433258
      2.388763
      -2.866515
      3.822020

      Atr13
      351.0
      0.093414
      0.494873
      -1.0
      -0.073725
      0.03027
      0.374860
      1.0
      -0.746602
      1.047737
      -1.419480
      1.720615

      Atr14
      351.0
      0.344159
      0.652828
      -1.0
      0.000000
      0.60194
      0.919330
      1.0
      -1.378995
      2.298325
      -2.757990
      3.677320

      Atr15
      351.0
      0.071132
      0.458371
      -1.0
      -0.081705
      0.00000
      0.308975
      1.0
      -0.667725
      0.894995
      -1.253745
      1.481015

      Atr16
      351.0
      0.381949
      0.618020
      -1.0
      0.000000
      0.59091
      0.935705
      1.0
      -1.403558
      2.339263
      -2.807115
      3.742820

      Atr17
      351.0
      -0.003617
      0.496762
      -1.0
      -0.225690
      0.00000
      0.195285
      1.0
      -0.857152
      0.826747
      -1.488615
      1.458210

      Atr18
      351.0
      0.359390
      0.626267
      -1.0
      0.000000
      0.57619
      0.899265
      1.0
      -1.348898
      2.248163
      -2.697795
      3.597060

      Atr19
      351.0
      -0.024025
      0.519076
      -1.0
      -0.234670
      0.00000
      0.134370
      1.0
      -0.788230
      0.687930
      -1.341790
      1.241490

      Atr20
      351.0
      0.336695
      0.609828
      -1.0
      0.000000
      0.49909
      0.894865
      1.0
      -1.342297
      2.237163
      -2.684595
      3.579460

      Atr21
      351.0
      0.008296
      0.518166
      -1.0
      -0.243870
      0.00000
      0.188760
      1.0
      -0.892815
      0.837705
      -1.541760
      1.486650

      Atr22
      351.0
      0.362475
      0.603767
      -1.0
      0.000000
      0.53176
      0.911235
      1.0
      -1.366853
      2.278087
      -2.733705
      3.644940

      Atr23
      351.0
      -0.057406
      0.527456
      -1.0
      -0.366885
      0.00000
      0.164630
      1.0
      -1.164157
      0.961902
      -1.961430
      1.759175

      Atr24
      351.0
      0.396135
      0.578451
      -1.0
      0.000000
      0.55389
      0.905240
      1.0
      -1.357860
      2.263100
      -2.715720
      3.620960

      Atr25
      351.0
      -0.071187
      0.508495
      -1.0
      -0.332390
      -0.01505
      0.156765
      1.0
      -1.066123
      0.890497
      -1.799855
      1.624230

      Atr26
      351.0
      0.541641
      0.516205
      -1.0
      0.286435
      0.70824
      0.999945
      1.0
      -0.783830
      2.070210
      -1.854095
      3.140475

      Atr27
      351.0
      -0.069538
      0.550025
      -1.0
      -0.443165
      -0.01769
      0.153535
      1.0
      -1.338215
      1.048585
      -2.233265
      1.943635

      Atr28
      351.0
      0.378445
      0.575886
      -1.0
      0.000000
      0.49664
      0.883465
      1.0
      -1.325197
      2.208662
      -2.650395
      3.533860

      Atr29
      351.0
      -0.027907
      0.507974
      -1.0
      -0.236885
      0.00000
      0.154075
      1.0
      -0.823325
      0.740515
      -1.409765
      1.326955

      Atr30
      351.0
      0.352514
      0.571483
      -1.0
      0.000000
      0.44277
      0.857620
      1.0
      -1.286430
      2.144050
      -2.572860
      3.430480

      Atr31
      351.0
      -0.003794
      0.513574
      -1.0
      -0.242595
      0.00000
      0.200120
      1.0
      -0.906668
      0.864193
      -1.570740
      1.528265

      Atr32
      351.0
      0.349364
      0.522663
      -1.0
      0.000000
      0.40956
      0.813765
      1.0
      -1.220648
      2.034413
      -2.441295
      3.255060

      Atr33
      351.0
      0.014480
      0.468337
      -1.0
      -0.165350
      0.00000
      0.171660
      1.0
      -0.670865
      0.677175
      -1.176380
      1.182690

```python
ionosphere_stats2=pd.DataFrame()
ionosphere_stats2.index=ionosphere_stats.index
atipicos,extremos=[],[]
for i in ionosphere_stats2.index:
    LII=ionosphere_stats[ionosphere_stats.index==i]["LII"][0]
    LIS=ionosphere_stats[ionosphere_stats.index==i]["LIS"][0]
    LEI=ionosphere_stats[ionosphere_stats.index==i]["LEI"][0]
    LES=ionosphere_stats[ionosphere_stats.index==i]["LES"][0]
    atipicos.append(len(data[((data[i]<=LII)&(data[i]>=LIS))|((data[i]>=LIS)&(data[i]<=LES))]))
    extremos.append(len(data[((data[i]<=LEI)|(data[i]>=LES))]))
ionosphere_stats2["Valores atipicos"]=atipicos
ionosphere_stats2["Valores extremos"]=extremos
ionosphere_stats2["Valores atipicos (%)"]=np.round(ionosphere_stats2["Valores atipicos"]/len(data)*100,2)
ionosphere_stats2["Valores extremos (%)"]=np.round(ionosphere_stats2["Valores extremos"]/len(data)*100,2)
ionosphere_stats2.loc['Total'] = ionosphere_stats2.sum(axis=0)
```

```python
ionosphere_stats2[24:]
```

      Valores atipicos
      Valores extremos
      Valores atipicos (%)
      Valores extremos (%)

      Atr24
      0.0
      0.0
      0.00
      0.0

      Atr25
      20.0
      0.0
      5.70
      0.0

      Atr26
      0.0
      0.0
      0.00
      0.0

      Atr27
      0.0
      0.0
      0.00
      0.0

      Atr28
      0.0
      0.0
      0.00
      0.0

      Atr29
      31.0
      0.0
      8.83
      0.0

      Atr30
      0.0
      0.0
      0.00
      0.0

      Atr31
      33.0
      0.0
      9.40
      0.0

      Atr32
      0.0
      0.0
      0.00
      0.0

      Atr33
      40.0
      0.0
      11.40
      0.0

      Total
      971.0
      755.0
      276.64
      215.1

Tomemos por ejemplo los datos que tienen valores atípicos en mayor cantidad, por ejemplo, los datos con  `Atr33`, y veamos como cambia la distribución cual eliminamos los valores atípicos.

```python
def filter_outlier(col):
    LII=ionosphere_stats[ionosphere_stats.index==col]["LII"][0]
    LIS=ionosphere_stats[ionosphere_stats.index==col]["LIS"][0]
    LEI=ionosphere_stats[ionosphere_stats.index==col]["LEI"][0]
    LES=ionosphere_stats[ionosphere_stats.index==col]["LES"][0]
    return data[~(((data[col]<=LII)&(data[col]>=LIS))|((data[col]>=LIS)&(data[col]<=LES)))][col]
```

```python
plt.figure(figsize=(16,6))
plt.subplot(121)
plt.hist(filter_outlier("Atr33"),label="Atr33 (Outliers removed) ({} rows)".format(len(filter_outlier("Atr33"))),alpha=0.4,color='b')
plt.hist(data["Atr33"],label="Atr33 (All data) ({} rows)".format(len(data)),alpha=0.4,color='r')
plt.vlines(np.mean(filter_outlier("Atr33")),0,120,label="Outliers removed mean's (<Atr33>={})".format(np.round(np.mean(filter_outlier("Atr33")),4)),alpha=0.99,color='b',linestyle="--",linewidth=4)
plt.vlines(np.mean(data["Atr33"]),0,120,label="All data mean's (<Atr33>={})".format(np.round(np.mean(data["Atr33"]),4)),alpha=0.99,color='r',linestyle="--",linewidth=4)
plt.legend(title="Absolute distribution",fontsize=11)
plt.ylabel("Counts",fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.subplot(122)
plt.hist(filter_outlier("Atr33"),label="Atr33 (Outliers removed) ({} rows)".format(len(filter_outlier("Atr33"))),alpha=0.4,cumulative=True,color='b')
plt.hist(data["Atr33"],label="Atr33 (All data) ({} rows)".format(len(data)),alpha=0.4,cumulative=True,color='r')
plt.vlines(np.median(filter_outlier("Atr33")),0,350,label="Outliers removed median's (<Atr33>={})".format(np.round(np.mean(filter_outlier("Atr33")),4)),alpha=0.99,color='b',linestyle="--",linewidth=6)
plt.vlines(np.median(data["Atr33"]),0,350,label="All data median's (<Atr33>={})".format(np.round(np.mean(data["Atr33"]),4)),alpha=0.99,color='r',linestyle="--",linewidth=3)

plt.legend(title="Cumulative distribution",fontsize=11)
plt.ylabel("Counts",fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.savefig("../../figures/ionosphere_hist2.png",bbox_inches ="tight")
plt.show()
```

Entonces como podemos ver cláramente la distribución de los datos cambian cuando eliminamos de nuestros datos los valores atípicos y por tanto también cambian las medidas de tendencia central, en particular la diferencia porcentual para el cálculo de la media en los datos originales y los datos cuando han sido removidos los valores atípicos es de:

```python
np.round(abs(np.mean(filter_outlier("Atr33"))-np.mean(data["Atr33"]))/np.mean(data["Atr33"])*100,2)
```

    794.06

```python
np.mean(filter_outlier("Atr33"))
```

    -0.10050096463022508

```python
np.mean(data["Atr33"])
```

    0.014480113960113956

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# Group Metrics

The `fairlearn` package contains algorithms which enable machine learning models to minimise disparity between groups. The `metrics` portion of the package provides the means required to verify that the mitigation algorithms are succeeding.

```python
import numpy as np
import pandas as pd
import sklearn.metrics as skm
```

## Ungrouped Metrics

At their simplest, metrics take a set of 'true' values $Y_{true}$ (from the input data) and predicted values $Y_{pred}$ (by applying the model to the input data), and use these to compute a measure. For example, the _recall_ or _true positive rate_ is given by
\begin{equation}
P( Y_{pred}=1 | Y_{true}=1 )
\end{equation}
That is, a measure of whether the model finds all the positive cases in the input data. The `scikit-learn` package implements this in [sklearn.metrics.recall_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html).

Suppose we have the following data:

```python
Y_true = [0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1]
Y_pred = [0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1]
```

we can see that the prediction is 1 in five of the ten cases where the true value is 1, so we expect the recall to be 0.0.5:

```python
skm.recall_score(Y_true, Y_pred)
```

## Metrics with Grouping

When considering fairness, each row of input data will have an associated group label $g \in G$, and we will want to know how the metric behaves for each $g$. To help with this, `fairlearn` provides wrappers, which take an existing (ungrouped) metric function, and apply it to each group within a set of data.

Suppose in addition to the $Y_{true}$ and $Y_{pred}$ above, we had the following set of labels:

```python
group_membership_data = ['d', 'a', 'c', 'b', 'b', 'c', 'c', 'c', 'b', 'd', 'c', 'a', 'b', 'd', 'c', 'c']

df = pd.DataFrame({ 'Y_true': Y_true, 'Y_pred': Y_pred, 'group_membership_data': group_membership_data})
df
```

We can see that for the groups 'a' and 'd' the recall is 0 (none of the true positives were identified), for 'b' the recall is 0.5 and for 'c' the recall is 0.75.

The `fairlearn.metrics.metric_by_group` routine can calculate all of these for us. This takes as arguments an _ungrouped_ metric (such as `sklearn.metrics.recall_score`), the arrays $Y_{true}$ and $Y_{pred}$ and an array of group labels (and optionally, if the ungrouped metric supports it, an array of sample weights), and produces `GroupMetricResult` object:

```python
import fairlearn.metrics as flm

group_metrics = flm.metric_by_group(skm.recall_score, Y_true, Y_pred, group_membership_data, sample_weight=None)

print("Overall recall = ", group_metrics.overall)
print("recall by groups = ", group_metrics.by_group)
```

Note that the overall recall is the same as that calculated above in the Ungrouped Metric section, while the `by_group` dictionary matches the values we calculated by inspection from the table above.

In addition to these basic scores, `metric_by_group` also records the maximum and minimum values of the metric, the groups for which these occurred, and also the difference and ratio between the maximum and minimum:

```python
print("min recall over groups = ", group_metrics.minimum)
print("occurred for groups: ", group_metrics.argmin_set)
print()
print("max recall over groups = ", group_metrics.maximum)
print("occurred for groups: ", group_metrics.argmax_set)
print()
print("range in recall = ", group_metrics.range)
print("ratio in recall = ", group_metrics.range_ratio)
```

## Supported Ungrouped Metrics

To be used by `metric_by_group` the supplied Python function must take arguments of `y_true` and `y_pred`:
```python
my_metric_func(y_true, y_pred)
```
An additional argument of `sample_weight` is also supported:
```python
my_metric_with_weight(y_true, y_pred, sample_weight)
```
The `sample_weight` argument is always invoked by name, and _only_ if the user supplies a `sample_weight` argument.

## Convenience Wrapper

Rather than require a call to `metric_by_group` each time, we also provide a function which turns an ungrouped metric into a grouped one. This is called `make_group_metric`:

```python
group_recall_score = flm.make_group_metric(skm.recall_score)

results = group_recall_score(Y_true, Y_pred, group_membership_data)

print("Overall recall = ", results.overall)
print("recall by groups = ", results.by_group)
```

```python

```


## Visualizing Convolutional Neural Networks and Neural Style Transfer

July 2019

**Author:** Matthew Stewart

```python
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import time
import numpy as np

from keras import backend as K
from keras.applications import vgg16, vgg19
from keras.preprocessing.image import load_img

from scipy.misc import imsave
from scipy.optimize import fmin_l_bfgs_b

# preprocessing
from utils import preprocess_image, deprocess_image

%matplotlib inline
```

### Part 1: Content loss

We can generate an image that combines the content and style of a pair with a loss function that incorporates this information. This is achieved with two terms, one that mimics the specific ativations of a certain layer for the content image, and a second term that mimics the style. The variable to optimize in the loss function will be a generated image that aims to minimize the proposed cost. Note that to optimize this function, we will perform gradient descent __on the pixel values__, rather than on the neural network weights.

We will load a trained neural network called VGG-16 proposed in [1](https://arxiv.org/pdf/1409.1556.pdf), who secured the first and second place in the localisation and classification tracks of ImageNet Challenge in 2014, respectively. This network has been trained to discriminate over 1000 classes over more than a million images. We will use the activation values obtained for an image of interest to represent the content and styles. In order to do so, we will feed-forward the image of interest and observe it's activation values at the indicated layer.

The content loss function measures how much the feature map of the generated image differs from the feature map of the source image. We will only consider a single layer to represent the contents of an image. The authors of this technique indicated they obtained better results when doing so. We denote the feature maps for layer $l$ with $a^{[l]} \in \mathbb{R}^{n_H^{[l]} \times n_W^{[l]} \times n_C^{[l]}}$. Parameter $n_C^{[l]}$ is the number of filters/channels in layer $l$, $n_H^{[l]}$ and $n_W^{[l]}$ are the height and width.

The content loss is then given by:
\begin{equation}
    J^{[l]}_C = \big\Vert a^{[l](G)} - a^{[l](C)} \big\Vert^2_{\mathcal{F}},
\end{equation}
where $a^{[l](G)}$ refers to the layer's activation values of the generated image, and $a^{[l](C)}$ to those of the content image.

  Part 1: Content loss

Implement funtion `feature_reconstruction_loss` that computes the loss of two feature inputs. You will need to use [keras backend functions](https://keras.io/backend/#backend-functions) to complete the exercise.

```python
def feature_reconstruction_loss(base, output):
    """
    Compute the content loss for style transfer.

    Inputs:
    - output: features of the generated image, Tensor with shape [height, width, channels]
    - base: features of the content image, Tensor with shape [height, width, channels]

    Returns:
    - scalar content loss
    """
    # YOUR CODE GOES HERE
    return K.sum(K.square(output - base))
```

Test your implementation:

```python
np.random.seed(1)
base = np.random.randn(10,10,3)
output = np.random.randn(10,10,3)
a = K.constant(base)
b = K.constant(output)
test = feature_reconstruction_loss(a, b)
print('Result:          ', K.eval(test))
print('Expected result: ', 605.62195)
```

### Part 2: Style loss

The style measures the similarity among filters in a set of layers. In order to compute that similarity, we will compute the Gram matrix of the activation values for the style layers, i.e., $a^{[l]}$ for some set $\mathcal{L}$. The Gram matrix is related to the empirical covariance matrix, and therefore, reflects the statistics of the activation values.

Given a feature map $a^{[l]}$ of shape $(n_H^{[l]}, n_W^{[l]}, n_C^{[l]})$, the Gram matrix has shape $(n_C^{[l]}, n_C^{[l]})$ and its elements are given by:
\begin{equation*}
    G^{[l]}_{k k'} = \sum_{i=1}^{n_H^{[l]}} \sum_{j=1}^{n_W^{[l]}} a^{[l]}_{ijk} a^{[l]}_{ijk'}.
\end{equation*}
The output is a 2-D matrix which approximately measures the cross-correlation among different filters for a given layer. This in essence constitutes the style of a layer.

  Part 2: Computing the Gram matrix

We implement a function that computes the Gram matrix of a given keras tensor. This can be accomplished efficiently if $x$ is reshaped as a tensor of shape ($n_C^{[l]} \times n_H^{[l]} n_W^{[l]}$) and then you compute the outer product of this matrix with itself. We need to use [keras backend functions](https://keras.io/backend/#backend-functions) for this.

```python
def gram_matrix(x):
    """
    Computes the outer-product of the input tensor x.

    Input:
    - x: input tensor of shape (H, W, C)

    Returns:
    - tensor of shape (C, C) corresponding to the Gram matrix of
    the input image.
    """
    # YOUR CODE GOES HERE
    features = K.batch_flatten(K.permute_dimensions(x, (2, 0, 1)))
    return K.dot(features, K.transpose(features))
```

Test your implementation:

```python
np.random.seed(1)
x_np = np.random.randn(10,10,3)
x = K.constant(x_np)
test = gram_matrix(x)
print('Result:\n', K.eval(test))
print('Expected:\n', np.array([[99.75723, -9.96186, -1.4740534], [-9.96186, 86.854324, -4.141108 ], [-1.4740534, -4.141108, 82.30106  ]]))
```

### Part 3: Style loss: layer's loss

Now we can tackle the style loss. For a given layer $l$, the style loss is defined as follows:
\begin{equation*}
    J^{[l]}_S = \frac{1}{4 (n^{[l]}_W n^{[l]}_H)^2} \Big\Vert G^{[l](S)} - G^{[l](G)}\Big\Vert^2_{\mathcal{F}}.
\end{equation*}

In practice we compute the style loss at a set of layers $\mathcal{L}$ rather than just a single layer $l$; then the total style loss is the sum of style losses at each layer:

$$J_S = \sum_{l \in \mathcal{L}} \lambda_l J^{[l]}_S$$
where $\lambda_l$ corresponds to a weighting parameter.

  Part 3: Computing the layer's loss

Implement `style_reconstruction_loss` that computes the loss for a given layer $l$. We again need to use [keras backend functions](https://keras.io/backend/#backend-functions) for this.

```python
def style_reconstruction_loss(base, output):
    """
    Computes the style reconstruction loss. It encourages the output img
    to have same stylistic features as style image.

    Inputs:
    - base: features at given layer of the style image.
    - output: features of the same length as base of the generated image.

    Returns:
    - style_loss: scalar style loss
    """
    # YOUR CODE GOES HERE
    H, W = int(base.shape[0]), int(base.shape[1])
    gram_base = gram_matrix(base)
    gram_output = gram_matrix(output)
    factor = 1.0 / float((2*H*W)**2)
    out = factor * K.sum(K.square(gram_output - gram_base))
    return out
```

Test your implementation:

```python
np.random.seed(1)
x = np.random.randn(10,10,3)
y = np.random.randn(10,10,3)
a = K.constant(x)
b = K.constant(y)
test = style_reconstruction_loss(a, b)
print('Result:  ', K.eval(test))
print('Expected:', 0.09799164)
```

### Part 4: Total-variation regularization
We will also encourage smoothness in the image using a total-variation regularizer. This penalty term will reduce variation among the neighboring pixel values.

The following expression constitues the regularization penalty over all pairs that are next to each other horizontally or vertically. The expression is independent among different RGB channels.
\begin{equation*}
    J_{tv} = \sum_{c=1}^3\sum_{i=1}^{n^{[l]}_H-1} \sum_{j=1}^{n^{[l]}_W-1} \left( (x_{i,j+1, c} - x_{i,j,c})^2 + (x_{i+1, j,c} - x_{i,j,c})^2  \right)
\end{equation*}

  Part 4: Total-variation regularization

In the next cell, fill in the definition for the TV loss term.

__Remark:__ $x$ has dimension $(1, n_H^{[l]}, n_W^{[l]}, n_C^{[l]})$, which is different from the 3D-tensors we used before.

```python
def total_variation_loss(x):
    """
    Total variational loss. Encourages spatial smoothness
    in the output image.

    Inputs:
    - x: image with pixels, has shape 1 x H x W x C.

    Returns:
    - total variation loss, a scalar number.
    """
    # YOUR CODE GOES HERE
    a = K.square(x[:, :-1, :-1, :] - x[:, 1:, :-1, :])
    b = K.square(x[:, :-1, :-1, :] - x[:, :-1, 1:, :])
    return K.sum(a + b)
```

Test your implementation. If you do not get exact results but similar, you may still have a correct implementation. The goal is that you write a smoother for neighboring pixels.

```python
np.random.seed(1)
x_np = np.random.randn(1,10,10,3)
x = K.constant(x_np)
test = total_variation_loss(x)
print('Result:  ', K.eval(test))
print('Expected:', 937.0538)
```

### Part 5: Style transfer

We now put it all together and generate some images! The `style_transfer` function below combines all the losses you coded up above and optimizes for an image that minimizes the total loss. Read the code and comments to understand the procedure.

```python
def style_transfer(base_img_path, style_img_path, output_img_path, convnet='vgg16',
        content_weight=3e-2, style_weights=(20000, 500, 12, 1, 1), tv_weight=5e-2, content_layer='block4_conv2',
        style_layers=['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1', 'block5_conv1'], iterations=50):

    print('\nInitializing Neural Style model...')

    # Determine the image sizes. Fix the output size from the content image.
    print('\n\tResizing images...')
    width, height = load_img(base_img_path).size
    new_dims = (height, width)

    # Preprocess content and style images. Resizes the style image if needed.
    content_img = K.variable(preprocess_image(base_img_path, new_dims))
    style_img = K.variable(preprocess_image(style_img_path, new_dims))

    # Create an output placeholder with desired shape.
    # It will correspond to the generated image after minimizing the loss function.
    output_img = K.placeholder((1, height, width, 3))

    # Sanity check on dimensions
    print("\tSize of content image is: {}".format(K.int_shape(content_img)))
    print("\tSize of style image is: {}".format(K.int_shape(style_img)))
    print("\tSize of output image is: {}".format(K.int_shape(output_img)))

    # Combine the 3 images into a single Keras tensor, for ease of manipulation
    # The first dimension of a tensor identifies the example/input.
    input_img = K.concatenate([content_img, style_img, output_img], axis=0)

    # Initialize the vgg16 model
    print('\tLoading {} model'.format(convnet.upper()))

    if convnet == 'vgg16':
        model = vgg16.VGG16(input_tensor=input_img, weights='imagenet', include_top=False)
    else:
        model = vgg19.VGG19(input_tensor=input_img, weights='imagenet', include_top=False)

    print('\tComputing losses...')
    # Get the symbolic outputs of each "key" layer (they have unique names).
    # The dictionary outputs an evaluation when the model is fed an input.
    outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])

    # Extract features from the content layer
    content_features = outputs_dict[content_layer]

    # Extract the activations of the base image and the output image
    base_image_features = content_features[0, :, :, :]  # 0 corresponds to base
    combination_features = content_features[2, :, :, :] # 2 coresponds to output

    # Calculate the feature reconstruction loss
    content_loss = content_weight * feature_reconstruction_loss(base_image_features, combination_features)

    # For each style layer compute style loss
    # The total style loss is the weighted sum of those losses
    temp_style_loss = K.variable(0.0)       # we update this variable in the loop
    weight = 1.0 / float(len(style_layers))

    for i, layer in enumerate(style_layers):
        # extract features of given layer
        style_features = outputs_dict[layer]
        # from those features, extract style and output activations
        style_image_features = style_features[1, :, :, :]   # 1 corresponds to style image
        output_style_features = style_features[2, :, :, :]  # 2 coresponds to generated image
        temp_style_loss += style_weights[i] * weight * \
                    style_reconstruction_loss(style_image_features, output_style_features)
    style_loss = temp_style_loss

    # Compute total variational loss.
    tv_loss = tv_weight * total_variation_loss(output_img)

    # Composite loss
    total_loss = content_loss + style_loss + tv_loss

    # Compute gradients of output img with respect to total_loss
    print('\tComputing gradients...')
    grads = K.gradients(total_loss, output_img)

    outputs = [total_loss] + grads
    loss_and_grads = K.function([output_img], outputs)

    # Initialize the generated image from random noise
    x = np.random.uniform(0, 255, (1, height, width, 3)) - 128.

    # Loss function that takes a vectorized input image, for the solver
    def loss(x):
        x = x.reshape((1, height, width, 3))   # reshape
        return loss_and_grads([x])[0]

    # Gradient function that takes a vectorized input image, for the solver
    def grads(x):
        x = x.reshape((1, height, width, 3))   # reshape
        return loss_and_grads([x])[1].flatten().astype('float64')

    # Fit over the total iterations
    for i in range(iterations+1):
        print('\n\tIteration: {}'.format(i+1))

        toc = time.time()
        x, min_val, info = fmin_l_bfgs_b(loss, x.flatten(), fprime=grads, maxfun=20)

        # save current generated image
        if i%10 == 0:
            img = deprocess_image(x.copy(), height, width)
            fname = output_img_path + '_at_iteration_%d.png' % (i)
            imsave(fname, img)
            print('\t\tImage saved as', fname)

        tic = time.time()

        print('\t\tLoss: {:.2e}, Time: {} seconds'.format(float(min_val), float(tic-toc)))
```

  Part 5: Generate pictures!

Find style and content images under `images/inputs/`.

* The `base_img_path` is the filename of content image.
* The `style_img_path` is the filename of style image.
* The `output_img_path` is the filename of generated image.
* The `convnet` is for the neural network weights, VGG-16 or VGG-19.
* The `content_layer` specifies which layer to use for content loss.
* The `content_weight` weights the content loss in the overall composite loss function. Increasing the value of this parameter will make the final image look more realistic (closer to the original content).
* `style_layers` specifies a list of which layers to use for the style loss.
* `style_weights` specifies a list of weights to use for each layer in style_layers (each of which will contribute a term to the overall style loss). We generally use higher weights for the earlier style layers because they describe more local/smaller scale features, which are more important to texture than features over larger receptive fields. In general, increasing these weights will make the resulting image look less like the original content and more distorted towards the appearance of the style image.
* `tv_weight` specifies the weighting of total variation regularization in the overall loss function. Increasing this value makes the resulting image look smoother and less jagged, at the cost of lower fidelity to style and content.

**CAUTION:** The script saves an image every 10 iterations.

### Great wave of Kanagawa + Chicago

```python
params = {
'base_img_path' : 'images/inputs/chicago.jpg',
'style_img_path' : 'images/inputs/great_wave_of_kanagawa.jpg',
'output_img_path' : 'images/results/wave_chicago',
'convnet' : 'vgg16',
'content_weight' : 500,
'style_weights' : (10, 10, 50, 10, 10),
'tv_weight' : 200,
'content_layer' : 'block4_conv2',
'style_layers' : ['block1_conv1',
                  'block2_conv1',
                  'block3_conv1',
                  'block4_conv1',
                  'block5_conv1'],
'iterations' : 50
}

style_transfer(**params)
```

### Starry night + Tübingen

```python
params = {
'base_img_path' : 'images/inputs/tubingen.jpg',
'style_img_path' : 'images/inputs/starry_night.jpg',
'output_img_path' : 'images/results/starry_tubingen',
'convnet' : 'vgg16',
'content_weight' : 100,
'style_weights' : (1000, 100, 12, 1, 1),
'tv_weight' : 200,
'content_layer' : 'block4_conv2',
'style_layers' : ['block1_conv1',
                  'block2_conv1',
                  'block3_conv1',
                  'block4_conv1',
                  'block5_conv1'],
'iterations' : 50
}

style_transfer(**params)
```

### Acknowledgments

- The implementation uses code from Francois Chollet's neural style transfer.
- The implementation uses code from Kevin Zakka's neural style transfer, under MIT license.
- The hierarchy borrows from Giuseppe Bonaccorso's gist, under MIT license.