```python
%load_ext autoreload
%autoreload 2

import sys
import time
import numpy as np
import matplotlib.pyplot as plt
sys.path.insert(0, "../")

from PDE_search import build_custom_Theta, BruteForceL0, TotalFiniteDiff_t, TotalFiniteDiff, build_Theta,\
TrainSTRidge, print_pde, build_linear_system, STRidgeScan

from scipy.signal import savgol_filter
from utils import scalar_pde_solver, save_dict, load_dict, get_u_term_from_descr
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

# In this example we consider evolution of a single magnon initial state in a ferromagnetic XXZ spin chain.
The Hamiltonian of the model reads:
$$ H = \sum_i \left[ J(S^{x}_i S^{x}_{i+1} + S^y_{i} S^y_{i+1})+\Delta S^z_i S^z_{i+1} \right] + \sum_i B_i S^z_i$$
We prepare the initial state as a wave packet given by a superposition of the product state and a single spin flip: $$|\psi_0\rangle  = \sum_{n}  f(n) |+\rangle_{n} \prod_{j \neq n} |\downarrow\rangle_{j} = \frac {1}{\sqrt{\pi\sigma^2}} \sum_{n} e^{-(n - x_0)^2/\sigma^2} | + \rangle_n \prod_{j \neq n} |\downarrow\rangle_{j}.$$

Here $f(n)$ is the Gaussian envelope function.

We introduce the following complex-valued function $u(t, x)$. At the sites of the spin chain, $x_i \equiv i a$, where $a$ is the lattice spacing, we set  the value of the function to
\begin{equation}
u(t, x_i) = \langle S^+_i(t) \rangle = \frac{1}{2}[\langle \sigma^x_i(t)\rangle + i\langle \sigma^y_i(t)\rangle],
\end{equation}
where $S^+_i=S^x_i+iS^y_i = \frac{1}{2} (\sigma^x_i + i \sigma^y_i)$ is the spin raising operator, $O(t) = \exp{(iHt)}O\exp{(-iHt)}$ is the time-dependent operator in the Heisenberg picture, and $\langle O \rangle \equiv \langle\psi_0|O|\psi_0\rangle$ is the expectation value taken in the initial state.

One can show that the dynamics of the complex field $u(t,x)$ is described by the following self-contained PDE:
$$i \partial_t u = J\cos{(i\partial_x)}u-\Delta u - B(x) u.$$
Assuming that the we assume that the magnetic field and the observables of the spin system change slowly, we arrive to the hydrodynamic-type PDE
$$
i\partial_t u =\frac{J}{2} \partial^2_x u + (J-\Delta) u - B(x) u + \mathcal{O(\partial_x^4 u)}.
$$

## Now we will learn the evolution PDE directly from data

## First, we load the dataset

```python
dict_data = load_dict('./data/single_magnon.npy')

stop = 500

u = dict_data['u'][:stop]
x = dict_data['x']
t = dict_data['t'][:stop]
```

## Visualize data

```python
xx, tt = np.meshgrid(x, t)
fig, ax = plt.subplots(1, 2, figsize = (12, 5));
plt_re = ax[0].pcolormesh(xx, tt, u.real, cmap='bwr', rasterized=True)
ax[0].set_title(r"$ \Re(u) = \langle S^x(t,x) \rangle$" )
ax[0].set_xlabel(r"$x$")
ax[0].set_ylabel(r"$t$")
ax[0].set_xticks([0,50,100])
plt.colorbar(plt_re, ax=ax[0])

plt_im = ax[1].pcolormesh(xx, tt, u.imag, cmap='bwr', rasterized=True)
ax[1].set_title(r"$ \Im(u) = \langle S^y(t,x) \rangle$" )
ax[1].set_xlabel(r"$x$")
ax[1].set_ylabel(r"$t$")
ax[1].set_xticks([0,50,100])
plt.colorbar(plt_im, ax=ax[1])
fig.tight_layout(pad=2.0)
```

### Dataset dimensions

```python
dt = t[1]-t[0]
dx = x[1]-x[0]
m, n = u.shape

print('dt: ', dt, 'dx: ', dx)
print('Dataset dimensions: ', u.shape)
print('Xmax, Tmax: ', x[-1], t[-1])
```

    dt:  0.1001001001001001 dx:  1
    Dataset dimensions:  (500, 100)
    Xmax, Tmax:  99 49.94994994994995

# Recovering PDE: $u_t = F(\cdot)$

```python
#-----------------------------------------------------------------------------
# Define a list of candidate terms, u_t = F(...)
#-----------------------------------------------------------------------------
descr = [
           'u',
           'u_x',
           'u_x^2',
           'u*u_x',
           'u^2*u_x',
           'u^3*u_x',
           'u^4*u_x',
           'u^5*u_x',
           'u_xx',
           'u_xxx',
           'u_xxxx',
           'u*u_xx',
           'u^2*u_xx',
        ]
#-----------------------------------------------------------------------------
# Define boundary conditions, needed for evaluation of spatial derivatives
bc = "periodic"
#-----------------------------------------------------------------------------
# Compute temporal derivative u_t from the data
ut = TotalFiniteDiff_t(u, dt).reshape((n*m,1))
#-----------------------------------------------------------------------------
# Method to compute spatial gradients (options: FD, Fourier)
deriv = "Fourier"
# Here we will be using "Fourier" method in order to capture high order derivative terms
#-----------------------------------------------------------------------------
u_data = []
for term in descr:
    u_data.append(get_u_term_from_descr(term, u, x, bc, deriv=deriv).reshape((n*m,1)))

u_data = np.hstack(u_data)
Theta, descr = build_custom_Theta(u_data, descr)
```

```python
print(f'Considering {len(descr)} candidate terms {descr}')
```

    Considering 10 candidate terms ['u', 'u_x', 'u_x^2', 'u*u_x', 'u^2*u_x', 'u^3*u_x', 'u^4*u_x', 'u^5*u_x', 'u_xx', 'u_xxxx']

## Now we are ready to perform PDE-learning

```python
xi, _ = BruteForceL0(Theta, ut, descr, l0_penalty=1e-3, verbose=False)
```

    Best PDE found with BruteForce:
    u_t = (-0.000001 +0.499600i)u
        + (0.000017 +0.489594i)u_xx

    Time elapsed (s): 2.3435070514678955

The inferred PDE matches theoretically expected equation:
$$i\partial_t u = -\frac{1}{2} \partial^2_x u - \frac{1}{2}u + \mathcal{O(\partial_x^4 u)}$$
for $J = -1$, $\Delta=-0.5$.

Lets decrease the L0 penalty constant

```python
xi, _ = BruteForceL0(Theta, ut, descr, l0_penalty=1e-6, verbose=False)
```

    Best PDE found with BruteForce:
    u_t = (-0.000000 +0.499793i)u
        + (-0.000000 +0.499301i)u_xx
        + (-0.000001 +0.040329i)u_xxxx

    Time elapsed (s): 5.337058067321777

We discovered correction terms containing the fourth spatial derivative!
$$i\partial_t u = -\frac{1}{2} \partial^2_x u - \frac{1}{24} \partial_x^4 u - \frac{1}{2}u + \mathcal{O(\partial_x^6 u)}$$

## Now we are comparing solution of the inferred PDE with the data

```python
# Define the initial condition
u0 = u[0]
# parameter num_integrator_steps corresponds to the number of Euler integrator steps between t and t+dt
u_ev = scalar_pde_solver(descr, xi, u0, t, x,
                  bc="periodic", num_integrator_steps=100)
```

```python
fig, ax = plt.subplots(1, 2, figsize = (12, 5));
for it in np.arange(0, 400, 100):
    ax[0].plot(u[it, :].real, c='royalblue')
    ax[0].plot(u_ev[it, :].real, ls='--', c='salmon')
    ax[0].set_xlabel(r'$x$')
    ax[0].set_title(r'$\Re[u]=\langle S^x (t,x) \rangle$')

    ax[1].plot(u[it, :].imag, c='royalblue')
    ax[1].plot(u_ev[it, :].imag, ls='--', c='salmon')
    ax[1].set_xlabel(r'$x$')
    ax[1].set_title(r'$\Im[u]=\langle S^y (t,x) \rangle$')

fig.tight_layout(pad=2.0)
```

```python
plt.pcolormesh(xx, tt, abs(u-u_ev), cmap='Blues', rasterized=True)
plt.colorbar()
plt.xlabel(r'$x$')
plt.ylabel(r'$t$')
plt.title(r'Difference $|u-u_{pde}|$')
plt.xticks([0,50,100])
plt.tight_layout()
```

### Plot Pareto frontier of inferred PDEs: scanning across the values of $\lambda_0$ penalty parameter

```python
# Use heuristic STRidge algorithm for faster performance
results_pareto = STRidgeScan(Theta, ut, descr, lam_l2=0, l0_fin=1, num_points=10, verbose=False)
```

Below we are reconstructing the PDE for various values of $\lambda_0$:

```python
for i, data in enumerate(results_pareto['train_data']):
    print('L0 penalty', results_pareto['l0_arr'][i])
    print_pde(data['best_xi'], descr)
```

    L0 penalty 1e-08
    u_t = (0.000000 +0.499793i)u
        + (-0.000186 +0.000501i)u_x^2
        + (0.000000 +0.499302i)u_xx
        + (-0.000002 +0.040337i)u_xxxx

    L0 penalty 7.742636826811277e-08
    u_t = (-0.000000 +0.499793i)u
        + (0.000000 +0.499302i)u_xx
        + (-0.000000 +0.040339i)u_xxxx

    L0 penalty 5.994842503189409e-07
    u_t = (-0.000000 +0.499793i)u
        + (0.000000 +0.499302i)u_xx
        + (-0.000000 +0.040339i)u_xxxx

    L0 penalty 4.641588833612773e-06
    u_t = (-0.000000 +0.499793i)u
        + (0.000000 +0.499302i)u_xx
        + (-0.000000 +0.040339i)u_xxxx

    L0 penalty 3.5938136638046256e-05
    u_t = (-0.000000 +0.499793i)u
        + (0.000000 +0.499302i)u_xx
        + (-0.000000 +0.040339i)u_xxxx

    L0 penalty 0.0002782559402207126
    u_t = (-0.000000 +0.499793i)u
        + (0.000000 +0.499302i)u_xx
        + (-0.000000 +0.040339i)u_xxxx

    L0 penalty 0.002154434690031882
    u_t = (-0.000000 +0.499599i)u
        + (0.000000 +0.489621i)u_xx

    L0 penalty 0.016681005372000558
    u_t = (-0.000000 +0.499599i)u
        + (0.000000 +0.489621i)u_xx

    L0 penalty 0.12915496650148828
    u_t = (-0.000000 +0.480014i)u

    L0 penalty 1.0
    u_t = (-0.000000 +0.480014i)u

For very small penalty parameter $\lambda=10^{-8}$ the recovered PDE contains a spurious term ($u_x^2$), which is the overfitting regime. For intermediate values of the penlty parameter $\lambda_0\in [10^{-7}, 10^{-2}]$ we find the correct PDEs matching Tailor expansion of the exact equation $$i \partial_t u = J\cos{(i\partial_x)}u-\Delta u.$$
Meanwhile for large values of the penalty coefficient $\lambda_0=1$ we enter the underfit region, where the inferred PDE misses gradient terms.

```python
num_terms = [np.count_nonzero(data['best_xi']) for data in results_pareto['train_data']]
l0_arr = results_ridge['l0_arr']
plt.plot(l0_arr, num_terms, marker='s', c='darkred')
plt.xscale('log')
plt.xlabel(r'$\lambda_0$ penalty')
plt.ylabel('# of terms');
```


# Tree based methods

J. Renero

January 2019

Trees construct prediction models from data. They recursively partition the data space to fit a boolean prediction model within each partition. The result is represented graphically as a decision tree. Trees can be used for regression of classification.

When the number of features, $p$, is large interpretability is easier than in distance based methods or probabilistic classification, where boundaries are much more difficult to be explained in higer dimensions.

For better visualizations of the trees, explore [dtreeviz](https://github.com/parrt/dtreeviz)

Table of Contents

1  Tree based methods

1.1  Regression Example

1.1.1  Load the data and split the dataset

1.1.2  Fit a simple regression tree (no CV)

1.1.3  Display the tree

1.1.4  Variables Importance

1.1.5  Understanding the tree structure (partially from here)

1.1.6  Evaluating the tree

1.2  Classification Example

1.2.1  Display the tree

1.2.2  Variables importance

1.2.3  Evaluating the tree

1.2.4  Cross Validation and Grid search

1.2.5  Working with the probabilities of the estimations

1.3  Optimization techniques

1.3.1  Bagging

1.3.2  Random Forests

1.3.3  Boosting

1.4  Hyperparameter tuning

1.5  Adding interpretability

1.5.1  Interpretability, through simple surrogate models

2  Isolation Trees

2.1  Example with Mopsi location dataset

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import pydotplus
import graphviz
import warnings

from sklearn.model_selection import cross_val_score, train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz
from sklearn.externals.six import StringIO
from sklearn.tree import export_graphviz
from IPython.display import Image
from sklearn import tree
from sklearn.exceptions import DataConversionWarning
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, GradientBoostingClassifier
from sklearn.utils.validation import column_or_1d

from lime.lime_tabular import LimeTabularExplainer

from dataset import Dataset

warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=DataConversionWarning)

%matplotlib inline
```

## Regression Example

We will start by building a simple regression example with the [wholesale customer dataset in UCI](https://archive.ics.uci.edu/ml/datasets/wholesale+customers). It contains 440 samples of 8 variables, and we will try to build a model that will help us to predict the annual spending on fresh products (first variable) with a regression tree.

### Load the data and split the dataset

```python
data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00292/Wholesale%20customers%20data.csv'
wholesale = Dataset(data_url)
wholesale.set_target('Fresh')
X, Y = wholesale.split()
```

```python
wholesale.data.head(5)
```

      Channel
      Region
      Fresh
      Milk
      Grocery
      Frozen
      Detergents_Paper
      Delicassen

      0
      2
      3
      12669
      9656
      7561
      214
      2674
      1338

      1
      2
      3
      7057
      9810
      9568
      1762
      3293
      1776

      2
      2
      3
      6353
      8808
      7684
      2405
      3516
      7844

      3
      1
      3
      13265
      1196
      4221
      6404
      507
      1788

      4
      2
      3
      22615
      5410
      7198
      3915
      1777
      5185

### Fit a simple regression tree (no CV)

```python
regressor = DecisionTreeRegressor(random_state=0, max_depth=3)
regressor.fit(X.train, Y.train)
```

    DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,
               max_leaf_nodes=None, min_impurity_decrease=0.0,
               min_impurity_split=None, min_samples_leaf=1,
               min_samples_split=2, min_weight_fraction_leaf=0.0,
               presort=False, random_state=0, splitter='best')

### Display the tree

Things get a bit complicated to plot the tree (this time, R is better than Python...). I pack eveything needed in a function so I can reuse it later on.

```python
def display_tree(tree, features):
    dot_data = StringIO()
    export_graphviz(tree,
                    out_file=dot_data,
                    feature_names=features,
                    filled=True,
                    rounded=True,
                    node_ids=True,
                    special_characters=True)
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
    image = Image(graph.create_png())
    display(image)
```

```python
display_tree(regressor, wholesale.names('features'))
```

### Variables Importance

```python
def features_importance(tree, X):
    """
    Plot the variables importance from the fitted tree and the
    training dataset used.
    """
    importances = tree.feature_importances_
    indices = np.argsort(importances)
    # Plot the feature importances of the forest
    plt.figure(figsize=(10,4))
    plt.title("Feature importances")
    plt.barh(range(X.train.shape[1]), importances[indices],
           color="r", align="center")
    index_names = [list(X.train)[i] for i in indices]
    plt.yticks(range(X.train.shape[1]), index_names)
    plt.ylim([-1, X.train.shape[1]])
    plt.xlim([0.0, 1.0])
    plt.show()

features_importance(regressor, X)
```

### Understanding the tree structure (partially from [here](https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html#sphx-glr-auto-examples-tree-plot-unveil-tree-structure-py))

From an example in SciKit Learn pages. Do not try this at home!

```python
def explain_tree(regressor):
    n_nodes = regressor.tree_.node_count
    children_left = regressor.tree_.children_left
    children_right = regressor.tree_.children_right
    feature = regressor.tree_.feature
    threshold = regressor.tree_.threshold

    # The tree structure can be traversed to compute various properties such
    # as the depth of each node and whether or not it is a leaf.
    node_depth = np.zeros(shape=n_nodes, dtype=np.int64)
    is_leaves = np.zeros(shape=n_nodes, dtype=bool)
    stack = [(0, -1)]  # seed is the root node id and its parent depth
    while len(stack) > 0:
        node_id, parent_depth = stack.pop()
        node_depth[node_id] = parent_depth + 1

        # If we have a test node
        if (children_left[node_id] != children_right[node_id]):
            stack.append((children_left[node_id], parent_depth + 1))
            stack.append((children_right[node_id], parent_depth + 1))
        else:
            is_leaves[node_id] = True

    print("The binary tree structure has %s nodes and has "
          "the following tree structure:"
          % n_nodes)
    for i in range(n_nodes):
        if is_leaves[i]:
            print("%snode=%s (leaf node)" % (node_depth[i] * "  ", i))
        else:
            print("%snode=%s: go to node %s if X[:, %s] <= %s else to "
                  "node %s."
                  % (node_depth[i] * "  ",
                     i,
                     children_left[i],
                     feature[i],
                     threshold[i],
                     children_right[i],
                     ))

explain_tree(regressor)
```

    The binary tree structure has 15 nodes and has the following tree structure:
    node=0: go to node 1 if X[:, 4] <= 15943.0 else to node 8.
      node=1: go to node 2 if X[:, 4] <= 2905.5 else to node 5.
        node=2: go to node 3 if X[:, 5] <= 2803.5 else to node 4.
          node=3 (leaf node)
          node=4 (leaf node)
        node=5: go to node 6 if X[:, 2] <= 2547.0 else to node 7.
          node=6 (leaf node)
          node=7 (leaf node)
      node=8: go to node 9 if X[:, 4] <= 17386.5 else to node 12.
        node=9: go to node 10 if X[:, 4] <= 16641.5 else to node 11.
          node=10 (leaf node)
          node=11 (leaf node)
        node=12: go to node 13 if X[:, 5] <= 155.0 else to node 14.
          node=13 (leaf node)
          node=14 (leaf node)

### Evaluating the tree

What's the $MSE$ of this tree when making predictions?

```python
def evaluate_regressor(tree, X, Y):
    """
    Evaluates a tree with the data values passed, returning the R2 and MSE
    """
    r2 = tree.score(X, Y)
    e = tree.predict(X)
    mse = np.average(np.power((e - Y.values), 2))
    return r2, mse

r2, mse = evaluate_regressor(regressor, X.test, Y.test)
mse_values = cross_val_score(regressor, X.test, Y.test, cv=10,
                             scoring="neg_mean_squared_error")
print('Avg. MSE: {:.04f}'.format(np.mean(mse_values)))
```

    Avg. MSE: -0.1591

If we evaluate the regressor with the function just described above, for a range of trees where the maximum depth ranges from 3 to 20, what we can see is the result oscilates a little bit but they're not specially good. The performance of a regression tree in this problem is specially bad. **Trees are not very good for regression problems** with features in a continuous space.

```python
# Plot the $R^2$ values obtained when varying the depth of the tree
min_depth = 1
max_depth = 15
r2s_train = []
r2s_test = []
for i in range(min_depth, max_depth):
    tree = DecisionTreeRegressor(
        random_state=0, max_depth=i).fit(X.train, Y.train)
    r2s_train.append(np.mean(cross_val_score(
        tree, X.train, Y.train, scoring='neg_mean_squared_error', cv=10)))
    r2s_test.append(np.mean(cross_val_score(
        tree, X.test, Y.test, scoring='neg_mean_squared_error', cv=10)))
    del(tree)

r2s_train = np.abs(r2s_train)
r2s_test = np.abs(r2s_test)
plt.plot(r2s_train, 'o-', label='Training $MSE$')
plt.plot(r2s_test, 'o-', label='Test $MSE$')
plt.xticks(range(min_depth-1, max_depth-1),
           ['{}'.format(i) for i in range(min_depth, max_depth)])
plt.xlabel('Tree depth')
plt.ylabel('$MSE$')
plt.legend()
plt.show()
```

## Classification Example

Let's build, using the same dataset a classification case. To do so the goal of this new model will be predicting the channel used by the customer (1 or 2), so we have a **binomial** classification problem.

```python
wholesale = Dataset(data_url)
wholesale.set_target('Channel')

X, Y = wholesale.split()

classifier = DecisionTreeClassifier(random_state=0, max_depth=3)
classifier.fit(X.train, Y.train)
```

    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,
                max_features=None, max_leaf_nodes=None,
                min_impurity_decrease=0.0, min_impurity_split=None,
                min_samples_leaf=1, min_samples_split=2,
                min_weight_fraction_leaf=0.0, presort=False, random_state=0,
                splitter='best')

### Display the tree

```python
display_tree(classifier, wholesale.names('features'))
```

```python
explain_tree(classifier)
```

    The binary tree structure has 11 nodes and has the following tree structure:
    node=0: go to node 1 if X[:, 5] <= 1755.5 else to node 6.
      node=1: go to node 2 if X[:, 3] <= 24578.0 else to node 5.
        node=2: go to node 3 if X[:, 4] <= 5557.0 else to node 4.
          node=3 (leaf node)
          node=4 (leaf node)
        node=5 (leaf node)
      node=6: go to node 7 if X[:, 2] <= 2609.0 else to node 8.
        node=7 (leaf node)
        node=8: go to node 9 if X[:, 5] <= 3500.5 else to node 10.
          node=9 (leaf node)
          node=10 (leaf node)

### Variables importance

Feature importance is calculated as the decrease in **node impurity weighted by the probability of reaching that node**. The node probability can be calculated by the number of samples that reach the node, divided by the total number of samples. The higher the value the more important the feature.

Scikit learns computes the node importance using the following expression to compute the node importance:

\begin{equation}
    ni_j = w_j C_j - w_{left(j)} C_{left(j)} - w_{right(j)} C_{right(j)}
\end{equation}

where:

- $I_j$: the importance of node $j$
- $w_j$: weighted nr. of samples reaching node $j$
- $C_j$: Impurity of node $j$
- $left(j)$ / $right(j)$: child node from left/right split on node $j$

and then, **features importance** is computed as follows:

\begin{equation}
    fi_i = \frac{ \sum_{j:node\ j\ splits\ on\ feature\ i} I_j }{ \sum_{k\ \in \ all\ nodes} I_k }
\end{equation}

Feature importance of a feature is a sum of importances of the nodes where that feature is used to take a split decision.

```python
features_importance(classifier, X)
```

### Evaluating the tree

This case, the evaluation metric is the GINI index. Let's look how the different "impurity" indicators behave with regard to the probability distribution of classes.

```python
# Plot the Entropy, Gini Index and missclassification error curves
def gini(p):
    return (p)*(1 - (p)) + (1 - p)*(1 - (1 - p))

def cross_entropy(p):
    return - p*np.log2(p) - (1 - p)*np.log2((1 - p))

def error(p):
    return 1 - np.max([p, 1-p])

x = np.arange(0.0, 1.0, 0.01)
ent = [cross_entropy(p) if p != 0 else None for p in x]
sc_ent = [e*0.5 if e else None for e in ent]
err = [error(i) for i in x]

plt.figure(figsize=(9,5))
for i, lab, ls, c in zip([ent, sc_ent, gini(x), err],
                        ['Cross-Entropy','Cross-Entropy(scaled)',
                        'Gini Impurity', 'Missclass. err.'],
                        ['-','-','--','-.'],
                        ['black','lightgrey','red','green','cyan']):
    line = plt.plot(x, i, label=lab, linestyle=ls, color=c)
plt.legend(loc='best')
plt.xlabel('p(i) = 1')
plt.ylabel('Impurity Index')
plt.show()
```

```python
mse_values = cross_val_score(classifier, X.test, Y.test, cv=10, scoring="f1")
print('Missclass err. rate: {:.4f}'.format(np.mean(mse_values)))
```

    Missclass err. rate: 0.8818

We can see that if we vary the depth of the tree from 3 up to 19, the trend is that the error rate grows.... which sounds counterintuitive, as we're making our tree better and better. The explanation is very easy: we're overfitting. Trees are prone to increase variance, and thus produce a worse generalization.

```python
# Plot the $R^2$ values obtained when varying the depth of the tree
min_depth = 1
max_depth = 20
mers_train = []
mers_test = []
for i in range(min_depth, max_depth):
    tree = DecisionTreeClassifier(
        random_state=0, max_depth=i).fit(X.train, Y.train)
    mer = np.mean(cross_val_score(tree, X.train, Y.train, cv=10,
                                  scoring='neg_mean_squared_error'))
    mers_train.append(mer)
    mer = np.mean(cross_val_score(tree, X.test, Y.test, cv=10,
                                  scoring='neg_mean_squared_error'))
    mers_test.append(mer)
    del(tree)

mers_train = np.abs(mers_train)
mers_test = np.abs(mers_test)
plt.figure(figsize=(8, 5))
plt.plot(mers_train, 'o-', label='Training Error')
plt.plot(mers_test, 'o-', label='Test Error')
plt.xticks(range(min_depth-1, max_depth-1),
           ['{}'.format(i) for i in range(min_depth, max_depth)])
plt.xlabel('Tree depth')
plt.ylabel('Error rate')
plt.legend()
plt.show()
```

### Cross Validation and Grid search

Now, we will use cross validation to check what is the average score when using different splits. To do so, instead of fitting a single split, we will use the method `cross_val_score` with `cv` param set to 10 (you can try different ones). For each value of the depth we're trying, an average accuracy will be produced over different splits, using _k_fold_ scheme, with $k = 10$.

```python
# Plot the accuracy values obtained when varying the depth of the tree
min_depth = 1
max_depth = 20
mers_train = []
mers_test = []
for i in range(min_depth, max_depth):
    tree = DecisionTreeClassifier(
        random_state=0, max_depth=i).fit(X.train, Y.train)
    mer = cross_val_score(tree, X.train, Y.train, cv=10)
    mers_train.append(np.mean(mer))
    mer = cross_val_score(tree, X.test, Y.test, cv=10)
    mers_test.append(np.mean(mer))
    del(tree)
plt.figure(figsize=(8, 5))
plt.plot(mers_train, 'o-', label='Training Accuracy')
plt.plot(mers_test, 'o-', label='Test Accuracy')
plt.xticks(range(min_depth-1, max_depth-1),
           ['{}'.format(i) for i in range(min_depth, max_depth)])
plt.xlabel('Tree depth')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
```

What is the optimal depth for this tree given the results obtained with CV? Instead of looking at a plot, we can use the grid search over the parameters we want to try, and for the range of values we decide. The result is a much simpler approach to the problem where you don't have to write the loop, and a single call does all the job for you.

```python
parameters = {'max_depth':range(min_depth, max_depth)}
grid_tree = GridSearchCV(DecisionTreeClassifier(), parameters, cv=5)
grid_tree.fit(X.train, Y.train)
best_tree = grid_tree.best_estimator_
print('Optimal accuracy: {:.4f}, for max_depth={}'.format(
    grid_tree.best_score_, grid_tree.best_params_['max_depth']))
```

    Optimal accuracy: 0.9261, for max_depth=4

What is the score, for the TEST set, of that optimal tree? Let's use `cross_val_score` again to check it over the test set, using the best model from the previous step.

```python
test_scores = cross_val_score(best_tree, X.test, Y.test, cv=5)
print('Average test score for 4-deep tree: {:.2f}'.format(
    np.mean(test_scores)))
```

    Average test score for 4-deep tree: 0.84

### Working with the probabilities of the estimations

The predicted class probability is the fraction of samples of the same class in a leaf. We can apply __laplace smoothing__ or play with the __threshold__ to be applied to consider when to assign a leaf to each class to find better results than those obtained with the default 0.5 probability.

In the code below, we obetain the prediction probabilities for the 23rd sample in the test set.

```python
sample = X.test.iloc[23, :].values.reshape(1, -1)
leaf_id = classifier.apply(sample)

print('Leaf node ID: ', leaf_id)
print('Decision path:\n', classifier.decision_path(sample), sep='')
print('Probabilities:', classifier.predict_proba(sample))
print('Nr. of samples of leaf node:', classifier.tree_.n_node_samples[leaf_id])
print('Impurity (Gini) of leaf node:', classifier.tree_.impurity[leaf_id])
print('Samples from each class:', classifier.tree_.value[leaf_id])
```

    Leaf node ID:  [9]
    Decision path:
      (0, 0)	1
      (0, 6)	1
      (0, 8)	1
      (0, 9)	1
    Probabilities: [[0.4137931 0.5862069]]
    Number of samples of leaf node: [29]
    Impurity (Gini) of leaf node: [0.48513674]
    Samples from each class: [[[12. 17.]]]

## Optimization techniques

### Bagging

A Bagging classifier is an ensemble meta-estimator that fits base classifiers each on random subsets of the original dataset and then aggregate their individual predictions (either by voting or by averaging) to form a final prediction. Such a meta-estimator can typically be used as a way to reduce the variance of a black-box estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.

We're producing `cv` splits, and for each of them we're growing 100 trees, each with a bootstrapped sample of the training set, to produce a classification, based on voting (hard or soft).

```python
bagging = BaggingClassifier(DecisionTreeClassifier(max_depth=4),
                            n_estimators=100,
                            max_samples=0.5,
                            max_features=len(wholesale.names('features')),
                            oob_score=True,
                            random_state=0, n_jobs=-1)
bagging.fit(X.train, column_or_1d(Y.train))
bagging_scores = cross_val_score(bagging, X.test, column_or_1d(Y.test), cv=10)
avg_score = np.mean(bagging_scores)
print('Avg. Score: {:.4f}'.format(avg_score))
```

    Avg. Score: 0.9181

The result is much better than the one obtained by simply applying grid search over the max_depth parameters with cross-validation.

### Random Forests

Now, we're not only going to pick subsets of the training set to train a number of trees, but also to select subset of features. In the end, the classification will be performed by the avearege prediction made by the grown trees.

```python
rf = RandomForestClassifier(n_estimators=100,
                            max_features=3,
                            max_depth=4,
                            oob_score=True,
                            random_state=0, n_jobs=-1)
rf.fit(X.train, column_or_1d(Y.train))
rf_scores = cross_val_score(rf, X.test, column_or_1d(Y.test), cv=10)
avg_score = np.mean(rf_scores)
print('Avg. Score: {:.4f}'.format(avg_score))
```

    Avg. Score: 0.8944

The result is much better than the one obtained by simply applying grid search over the max_depth parameters with cross-validation, but slightly worse than in the bagging case.

### Boosting

Gradient Tree Boosting or Gradient Boosted Trees (GBT) is a generalization of boosting to arbitrary differentiable loss functions. GBRT is an accurate and effective off-the-shelf procedure that can be used for both regression and classification problems. Gradient Tree Boosting models are used in a variety of areas including Web search ranking and ecology.

```python
gb = GradientBoostingClassifier(n_estimators=100,
                                learning_rate=1.0,
                                max_features=3,
                                max_depth=4,
                                random_state=0)
gb.fit(X.train, column_or_1d(Y.train))
gb_scores = cross_val_score(gb, X.test, column_or_1d(Y.test), cv=10)
avg_score = np.mean(gb_scores)
print('Avg. Score: {:.4f}'.format(avg_score))
```

    Avg. Score: 0.8819

```python
gb
```

    GradientBoostingClassifier(criterion='friedman_mse', init=None,
                  learning_rate=1.0, loss='deviance', max_depth=4,
                  max_features=3, max_leaf_nodes=None,
                  min_impurity_decrease=0.0, min_impurity_split=None,
                  min_samples_leaf=1, min_samples_split=2,
                  min_weight_fraction_leaf=0.0, n_estimators=100,
                  n_iter_no_change=None, presort='auto', random_state=0,
                  subsample=1.0, tol=0.0001, validation_fraction=0.1,
                  verbose=0, warm_start=False)

## Hyperparameter tuning

One of the most important phases in trees is deciding on what is the best combination of parameters that control the optimal values for

- depth of the tree,
- min number of samples per leaf,
- min number of samples to perform a split,
- max number of features to consider when looking for the best split,
- ...

All these parameters control the growth of the tree, and given the lack of other pruning mechanisms, this is the best way of considering that your tree model gets "**pruned**" to avoid overfitting.

How to perform gridsearch or hyper parameter tuning over Gradient Boosting trees?

```python
parameters = {
    "loss":["deviance"],
    "learning_rate": [0.1, 0.5, 1.0],
    "min_samples_split": np.linspace(0.1, 0.5, 5),
    "min_samples_leaf": np.linspace(0.1, 0.5, 5),
    "max_depth":[2, 3, 4],
    "max_features":["log2","sqrt"],
    "criterion": ["friedman_mse",  "mae"],
    "subsample":[0.5, 0.8, 1.0],
    "n_estimators":[10]
    }

grid_gb = GridSearchCV(GradientBoostingClassifier(), parameters, cv=10, n_jobs=-1)

# Grid search takes 10 or more minutes, so I disable it.
grid_gb.fit(X.train, column_or_1d(Y.train))

score = grid_gb.score(X.train, column_or_1d(Y.train))
print('Score: {:.4f}'.format(score))
print('Best Params:\n{}'.format(grid_gb.best_params_))
```

    Score: 0.9318
    Best Params:
    {'criterion': 'friedman_mse', 'learning_rate': 1.0, 'loss': 'deviance', 'max_depth': 3, 'max_features': 'log2', 'min_samples_leaf': 0.2, 'min_samples_split': 0.1, 'n_estimators': 10, 'subsample': 0.8}

```python
hpgb_scores = cross_val_score(
    grid_gb.best_estimator_,
    X.test,
    column_or_1d(Y.test),
    cv=10)
avg_score = np.mean(hpgb_scores)
print('Avg. Score: {:.4f}'.format(avg_score))
```

    Avg. Score: 0.8375

Parameters found are

    'criterion': 'friedman_mse',
    'learning_rate': 0.5,
    'loss': 'deviance',
    'max_depth': 2,
    'max_features': 'log2',
    'min_samples_leaf': 0.2,
    'min_samples_split': 0.2,
    'n_estimators': 10,
    'subsample': 1.0

Surprisingly, for this particular problem and this dataset, we're not getting better results with GBoost, or we couldn't find the proper set of parameters.

```python
optimal_gb = GradientBoostingClassifier(n_estimators=10,
                                        min_samples_leaf=0.2,
                                        min_samples_split=0.2,
                                        subsample=1.0,
                                        learning_rate=0.5,
                                        max_features='log2',
                                        max_depth=2,
                                        random_state=0)
optimal_gb.fit(X.train, column_or_1d(Y.train))
gb_scores = cross_val_score(optimal_gb, X.test, column_or_1d(Y.test), cv=10)
avg_score = np.mean(gb_scores)
print('Avg. Score: {:.4f}'.format(avg_score))
```

    Avg. Score: 0.8722

## Adding interpretability

The basic idea behind LIME and the general concept of bringing 'interpretability' to complex models is the following: train a model with the input features of the complex model, and learn the response given by the complex model to later explain what's the effect of each individual feature on the response.

The problem is that this approach is too simplistic and does not necessarily produces the right explanation to the phenomena learned by the complex model. In fact, any potential interaction between variables is not considered. Therefore, these ideas must be used with care.

Let's explore how LIME (Local Interpretable Model-agnostic Explanations) brings interpretability to our previously trained gradient boosting tree with the optimal parameters.

First of all print out the values we want to predict (random pick from the training set):

```python
idx=333
print('Sample being traced\n-------------------------\n',
      wholesale.data.iloc[idx, :], sep='')
```

    Sample being traced
    -------------------------
    Region                  2
    Fresh                8565
    Milk                 4980
    Grocery             67298
    Frozen                131
    Detergents_Paper    38102
    Delicassen           1215
    Name: 333, dtype: int64

This is the way LIME adds interpretability, this time, to a single prediction.

```python
explainer = LimeTabularExplainer(
    wholesale.features.values,
    mode='classification',
    feature_names=wholesale.names('features'))

exp = explainer.explain_instance(wholesale.features.iloc[idx, :],
    optimal_gb.predict_proba)

%matplotlib inline
p = exp.as_pyplot_figure();
```

### Interpretability, through simple surrogate models

Let's build a dataset with the input features as they're, and the target variable being the output from the gradient boosting model trained before.

```python
input_cols = wholesale.features
predictions = optimal_gb.predict(wholesale.features)
output_col = pd.Series(data=predictions, name='Channel')
predicted_df = pd.concat([input_cols, output_col], axis=1)

ws = Dataset.from_dataframe(predicted_df)
ws.set_target('Channel')

print('Sample being traced\n-------------------------\n',
      ws.data.iloc[idx, :], sep='')
```

    Sample being traced
    -------------------------
    Region                  2
    Fresh                8565
    Milk                 4980
    Grocery             67298
    Frozen                131
    Detergents_Paper    38102
    Delicassen           1215
    Name: 333, dtype: int64

Build a simple decision tree with the same input features as the gradient boosting tree, but with its predictions as target variable. The tree is a single tree, but will help us to see if the decision process is similar to that followed by the GB tree.

```python
X, Y = ws.split()
surrogate = DecisionTreeClassifier(
    min_samples_leaf=10,
    min_samples_split=10,
    random_state=0,
    max_depth=3)
surrogate.fit(X.train, Y.train)

display_tree(surrogate, ws.names('features'))
```

Let's reproduce what is the decision path for the entry selected, to see what nodes are shaping the decision in the surrogate model:

```python
print('Index selected: {}'.format(idx))
path = surrogate.decision_path(X=ws.features.iloc[idx, :].values.reshape(1, -1))
print('Decision Path:\n', path, sep='')
prediction = surrogate.predict(
    ws.features.iloc[idx, :].values.reshape(1, -1))
print('Prediction by surrogate model: {}'.format(prediction[0]))
print('Actual response for idx({}): {}'.format(idx, wholesale.target[idx]))
```

    Index selected: 333
    Decision Path:
      (0, 0)	1
      (0, 4)	1
      (0, 6)	1
      (0, 8)	1
    Prediction by surrogate model: 2
    Actual response for idx(333): 2

When plotting the features importance coming from the surrogate models, we can check that the two most important features that shape the classification decision match those coming from the LIME package: Detergents_paper and Grocery.

However, this simpler model is not able to add any importance to the rest of the features, probably because of its simplicity and low depth. We can also build an alternate interpretation with surrogate models by fitting a logistic regression and checking the coefficients.

```python
features_importance(surrogate, X)
```

# Isolation Trees

Return the anomaly score of each sample using the IsolationForest algorithm

The IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.

Since recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.

This path length, averaged over a forest of such random trees, is a measure of normality and our decision function.

```python
print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import IsolationForest

rng = np.random.RandomState(42)

# Generate train data
X = 0.3 * rng.randn(100, 2)
X_train = np.r_[X + 2, X - 2]
# Generate some regular novel observations
X = 0.3 * rng.randn(20, 2)
X_test = np.r_[X + 2, X - 2]
# Generate some abnormal novel observations
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))

# fit the model
clf = IsolationForest(behaviour='new', max_samples=100,
                      random_state=rng, contamination='auto')
clf.fit(X_train)
y_pred_train = clf.predict(X_train)
y_pred_test = clf.predict(X_test)
y_pred_outliers = clf.predict(X_outliers)

# plot the line, the samples, and the nearest vectors to the plane
xx, yy = np.meshgrid(np.linspace(-5, 5, 50), np.linspace(-5, 5, 50))
Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.figure(figsize=(9,6))
plt.title("IsolationForest")
plt.contourf(xx, yy, Z, cmap=plt.cm.Blues_r)

b1 = plt.scatter(X_train[:, 0], X_train[:, 1], c='white',
                 s=20, edgecolor='k')
b2 = plt.scatter(X_test[:, 0], X_test[:, 1], c='green',
                 s=20, edgecolor='k')
c = plt.scatter(X_outliers[:, 0], X_outliers[:, 1], c='red',
                s=20, edgecolor='k')
plt.axis('tight')
plt.xlim((-5, 5))
plt.ylim((-5, 5))
plt.legend([b1, b2, c],
           ["training observations",
            "new regular observations", "new abnormal observations"],
           loc="upper left")
plt.show();
```

## Example with Mopsi location dataset

Dataset and info [here](https://cs.joensuu.fi/sipu/datasets/)

```python
fi = Dataset('/Users/renero/Desktop/MopsiLocations2012-Joensuu.txt',
             sep=' ', header=None, names=['X1', 'X2'])
fi.data.head(5)
```

      X1
      X2

      0
      62.59809
      29.74448

      1
      62.59809
      29.74448

      2
      62.59809
      29.74448

      3
      62.61943
      29.69410

      4
      62.60326
      29.74494

```python
plt.scatter(fi.data.X1, fi.data.X2)
```

```python
# fit the model
clf = IsolationForest(behaviour='new', max_samples=100,
                      random_state=rng, contamination='auto')
clf.fit(fi.data)
y_pred_train = clf.predict(fi.data)

# plot the line, the samples, and the nearest vectors to the plane
xx, yy = np.meshgrid(np.linspace(fi.data.iloc[:, 0].min(), fi.data.iloc[:, 0].max(), 50),
                     np.linspace(fi.data.iloc[:, 1].min(), fi.data.iloc[:, 1].max(), 50))
Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.figure(figsize=(10, 8))
plt.title("IsolationForest")
plt.contourf(xx, yy, Z, cmap=plt.cm.Blues_r)
b1 = plt.scatter(fi.data.iloc[:, 0], fi.data.iloc[:, 1], c='white',
                 s=20, edgecolor='k')
plt.axis('tight')
plt.legend([b1, b2],
           ["training observations"],
           loc="upper left")
plt.show()
```

```python

```


# Burgersova rovnice

Nechť $u \in \mathbb C^1( \mathbb R^2 \to \mathbb R)$ a $f(u) = \frac{1}{2} u^2$. Potom rovnici
\begin{equation}
  \frac{\partial u(x,t)}{\partial t} + \frac{\partial f(u(x,t))}{\partial x} = 0
\end{equation}
nazveme **nelineární Burgersovou rovnicí**.

Poznámka: pro $u\in \mathbb C^1$ lze rovnici upravit do tvaru
$$
    u_t + u u_x = 0.
$$

## Počáteční úloha pro rovnici lineární advekce

Nechť $u_0 \in \mathbb C^1(\mathbb R \to \mathbb R)$. **Počáteční úlohou** pro Burgersovu rovnici pak rozumíme nalézt funkci $u \in \mathbb C^1( \mathbb R\times [0,+\infty) \to \mathbb R)$ takovou, že $\forall x \in \mathbb R, \forall t>0$ je splněna rovnice
\begin{equation}
  \frac{\partial u(x,t)}{\partial t} + \frac{\partial f(u(x,t))}{\partial x} = 0
\end{equation}
a navíc je $u(x,0)=u_0(x)$.

### Analytické řešení počáteční úlohy
Analytické řešení je možné získat například pomocí metody charakteristik. Řešení získáme v implicitním tvaru jako

$$
    u(x,t) = u_0(x - u(x,t) t).
$$

### Vznik nespojitosti
Klasické řešení existuje jen pro $t < t_{crit} = -1 / \min u_0'$, viz metoda charakteristik.

### Slabé řešení
Nechť $\phi \in \mathbb C^1( \mathbb R\times [0,+\infty) \to \mathbb R)$ nenulová na kompaktní množině. Potom

$$
    \int_0^{+\infty}\int_{-\infty}^{+\infty}\left( u_t + f(u)_x\right) \phi(x,t) \,dx\,dt = 0.
$$

Integrál rozdělíme na součet dvou sčítanců a integrujeme zvlášť pomocí metody per-partes

$$
    \int_0^{+\infty}\int_{-\infty}^{+\infty} u_t \phi \,dx\,dt =
    \int_{-\infty}^{+\infty} \left( \left[u \phi\right]_0^{+\infty}-
    \int_0^{+\infty} u \phi_t \,dt
    \right) \,dx
$$

a

$$
    \int_0^{+\infty}\int_{-\infty}^{+\infty} f(u)_x \phi \,dx\,dt =
    \int_0^{+\infty} \left( \left[f(u) \phi\right]_{-\infty}^{+\infty}-
    \int_{-\infty}^{+\infty} f(u) \phi_x \,dx
    \right) \,dt.
$$

Protože $\phi=0$ pro $t\to+\infty$ a $|x|\to+\infty$ a $u(x,0)=u_0(x)$, je

\begin{equation}
    \int_0^{+\infty}\int_{-\infty}^{+\infty}\left( u \phi_t + f(u)\phi_x\right) \phi(x,t) \,dx\,dt =
    - \int_{-\infty}^{+\infty} u_0(x)\phi(x,0)\,dx.
\end{equation}

Funkci $u \in \mathbb L^\infty( \mathbb R\times [0,+\infty) \to \mathbb R)$ nazveme *slabým řešením* počáteční úlohy pro nelinearní (Burgersovu) rovnici, pokud $\forall \phi \in \mathbb C^1( \mathbb R\times [0,+\infty) \to \mathbb R)$ nenulové na kompaktní množině platí posledně uvedený vztah.

### Numerické řešení počáteční úlohy pro Burgersovu rovnici metodou konečných objemů

```julia
using PyPlot
```

```julia
Nx = 51;                   # Pocet bodu site na intervalu [0,1]

x = range(0, stop=1, length=Nx);       # Souradnice bodu site
xc = (x[1:end-1] + x[2:end]) / 2.0

function u0(x)
    """Definice pocatecni podminky"""
    if x<0.25 || x>0.75
        return 0
    else
        return sin( 2 * π * (2*x-1.0) )
    end
end

plot(x,[u0(xi) for xi in x]);
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u");
```

**Protiproudové (upwind) schéma**

```julia
dx = 1.0 / (Nx-1);      # Velikost prostoroveho
dt = 0.5 * dx / 1.0  # Velikost casoveho kroku

u    = [u0(xi) for xi in xc]
uNew = copy(u)

t = 0
Tend = 0.25;

pocet_iteraci = round(Int, Tend / dt) # Vypocet provedeme do casu priblizne Tend

function flux_upwind(ul, ur)
    if ul+ur>0
        return ul^2/2
    elseif ul+ur<0
        return ur^2/2
    else
        return 0
    end
end

for n = 1:pocet_iteraci
    global t += dt

    for i = 2:Nx-2
        uNew[i] = u[i] - dt / dx * ( flux_upwind(u[i],u[i+1]) - flux_upwind(u[i-1],u[i]) )
    end

    for i = 2:Nx-2
        u[i] = uNew[i]
    end

end

u_upwind = copy(u)

#plot(x, [u0(xi) for xi in x], label="u0")
plot(xc,u, "o", label="u[i]");
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u"); legend(loc="upper left");

```

**Laxovo-Friedrichsovo (Rusanovovo) schéma**

```julia
u    = [u0(xi) for xi in xc]
uNew = copy(u)

t = 0

function flux_lf(ul, ur)
    q = dx / dt
    return (ul^2/2 + ur^2/2)/2 - q/2*(ur - ul)
end

for n =1:pocet_iteraci
    global t += dt

    for i = 2:Nx-2
        uNew[i] = u[i] - dt / dx * ( flux_lf(u[i],u[i+1]) - flux_lf(u[i-1],u[i]) )
    end

    for i = 2:Nx-2
        u[i] = uNew[i]
    end

end

u_lf = copy(u)

#plot(x, [u0(xi) for xi in x], label="u0")
plot(xc,u_lf, "o", label="u[i]");
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u"); legend(loc="upper left");

```

**Laxovo-Wendroffovo schéma**

```julia
u    = [u0(xi) for xi in xc]
uNew = copy(u)

t = 0

function flux_lw(ul, ur)
    a = (ul + ur) / 2
    q = dt / dx * a^2
    return (ul^2/2 + ur^2/2)/2 - q/2*(ur - ul)
end

for n = 1:pocet_iteraci
    global t += dt

    for i = 2:Nx-2
        uNew[i] = u[i] - dt / dx * ( flux_lw(u[i],u[i+1]) - flux_lw(u[i-1],u[i]) )
    end

    for i = 2:Nx-2
        u[i] = uNew[i]
    end
end

u_lw = copy(u)

#plot(x, [u0(xi) for xi in x], label="u0")
plot(xc,u_lw, "o", label="u[i]");
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u"); legend(loc="upper left");

```

**Laxovo-Wendroffovo schéma s přídavnou vazkostí**

```julia
# Umela vazkost rizena konstantou eps

u    = [u0(xi) for xi in xc]
uNew = copy(u)

t = 0

eps = 0.05

for n = 1:pocet_iteraci
    global t += dt

    for i = 2:Nx-2
        uNew[i] = u[i] - dt / dx * ( flux_lw(u[i],u[i+1]) - flux_lw(u[i-1],u[i]) )
        uNew[i] = uNew[i] + eps*( (u[i-1]-u[i]) - (u[i]-u[i+1]) )
    end

    for i = 2:Nx-2
        u[i] = uNew[i]
    end
end

u_lwe = copy(u)

#plot(x, [u0(xi) for xi in x], label="u0")
plot(xc,u_lwe, "o", label="u[i]");
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u"); legend(loc="upper left");

```

```julia
# Umela vazkost rizena konstantou eps s detekci "hladkosti" reseni

u    = [u0(xi) for xi in xc]
uNew = copy(u)

t = 0

eps = 0.25

for n = 1:pocet_iteraci
    global t += dt

    for i = 2:Nx-2
        uNew[i] = u[i] - dt / dx * ( flux_lw(u[i],u[i+1]) - flux_lw(u[i-1],u[i]) )

        eps_l = eps * abs(u[i-1]-u[i])
        eps_r = eps * abs(u[i]-u[i+1])
        uNew[i] = uNew[i] + eps_l*(u[i-1]-u[i]) - eps_r*(u[i]-u[i+1])
    end

    for i = 2:Nx-2
        u[i] = uNew[i]
    end

end

u_lwe = copy(u)

#plot(x, [u0(xi) for xi in x], label="u0")
plot(xc,u_lwe, "o", label="u[i]");
ylim(-1.5,1.5); grid(true); xlabel("x"); ylabel("u"); legend(loc="upper left");

```

## Řešení metodou charakteristik

"Přesné" hodnoty lze získat řešením nelineární rovnice

$$
    0 = u - u_0(x - ut)
$$

To ovšem funguje jen v případě, že je $u$ hladké.

```julia
using NLsolve
```

    ┌ Info: Precompiling NLsolve [2774e3e8-f4cf-5e23-947b-6d7e65073b56]
    └ @ Base loading.jl:1186

```julia
function exact_solution(x,t)

    function f!(fu,u)
        fu[1] = u[1] - u0(x-u[1]*t)
    end

    if x>=0.5
        result = nlsolve(f!, [1.0])
    else
        result = nlsolve(f!, [-1.0])
    end
    return result.zero[1]
end

exact_solution(0.55,0.0)

```

    0.5877852522908464

```julia
uex = [exact_solution(xi,0.08) for xi in x];

plot(x,uex); grid(true)
```

Metodu charakteristik nelze (a nebo lze jen obtížně) použít pro nespojitá řešení

```julia
uex = [exact_solution(xi,0.1) for xi in x];

plot(x,uex); grid(true)
```

```julia

```

```julia

```


# Initial Conditions (Experiences, Observables, Lagged Choices)

While working with **respy**, you often want to simulate the effects of policies in counterfactual environments. At the start of the simulation, you need to sample the characteristics of individuals. You normally want to do at least one of the following points:

- Individuals should start with nonzero years of experience for some choice.
- The previous (lagged) choice in the first period can only be a subset of all choices in the model.
- An observed characteristic is not evenly distributed in the population.

Taken together these assumptions are called the initial conditions of a model. An initial condition is also called a *seed value* and determines the value of a variable in the first period of a dynamic system.

In the following, we describe how to set the initial condition for each of the the three points for a small Robinson Crusoe Economy. A more thorough presentation of a similar model can be found [here](../tutorials/robinson_crusoe.ipynb).

```python
%matplotlib inline

import io
import pandas as pd
import respy as rp
import matplotlib.pyplot as plt
import seaborn as sns
```

## Experiences

In a nutshell, Robinson can choose between fishing and staying in the hammock every period. He can accumulate experience in fishing which makes him more productive. To describe such a simple model, we write the following parameterization and simulate data with ten periods.

```python
params = """
category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.01
nonpec_hammock,constant,1
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,0
"""

options = {
    "n_periods": 10,
    "simulation_agents": 1_000,
    "covariates": {"constant": "1"}
}
```

```python
params = pd.read_csv(io.StringIO(params), index_col=["category", "name"])
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.01

      nonpec_hammock
      constant
      1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      0.00

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

```python
fig, axs = plt.subplots(1, 2, figsize=(12.8, 4.8))

(df.groupby("Period").Choice.value_counts(normalize=True).unstack()
 .plot.bar(ax=axs[0], stacked=True, rot=0, title="Choice Probabilities"))
(df.groupby("Period").Experience_Fishing.value_counts(normalize=True).unstack().plot
 .bar(ax=axs[1], stacked=True, rot=0, title="Share of Experience Level per Period", cmap="Reds"))

axs[0].legend(["Fishing", "Hammock"], loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)
axs[1].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=5)

plt.show()
plt.close()
```

The figure on the left-hand-side shows the choice probabilities for Robinson in each period. On the right-hand-side, one can see the average experience in fishing. By default Robinson starts with zero experience in fishing.

What if Robinson has an equal probability start with zero, one or two periods of experience in fishing? The first way to add more complex initial conditions to the model is via **probability mass functions**. This type of distributions is handy if the probabilities do not depend on any information. To feed the information to **respy** use the keyword ``initial_exp_fishing_*`` in the category-level of the index. Replace ``*`` with the experience level. In the name-level, use ``probability`` to signal that the float in ``value`` is a probability. The new parameter specification is below.

Note that one probability is set to 0.34 such that all probabilities sum to one. If that is not the case, respy will emit a warning and normalize probabilities.

```python
params = """
category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.01
nonpec_hammock,constant,1
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,0
initial_exp_fishing_0,probability,0.33
initial_exp_fishing_1,probability,0.33
initial_exp_fishing_2,probability,0.34
"""
```

```python
params = pd.read_csv(io.StringIO(params), index_col=["category", "name"])
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.01

      nonpec_hammock
      constant
      1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      0.00

      initial_exp_fishing_0
      probability
      0.33

      initial_exp_fishing_1
      probability
      0.33

      initial_exp_fishing_2
      probability
      0.34

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

    C:\Users\tobia\git\respy\respy\state_space.py:84: UserWarning: Some choices in the model are not admissible all the time. Thus, respy applies a penalty to the utility for these choices which is -400000 by default. For the full solution, the penalty only needs to be larger than all other value functions to be effective. Choose a milder penalty for the interpolation which does not dominate the linear interpolation model.
      "Some choices in the model are not admissible all the time. Thus, respy"

```python
fig, axs = plt.subplots(1, 2, figsize=(12.8, 4.8))

(df.groupby("Period").Choice.value_counts(normalize=True).unstack()
 .plot.bar(ax=axs[0], stacked=True, rot=0, title="Choice Probabilities"))
(df.groupby("Period").Experience_Fishing.value_counts(normalize=True).unstack().plot
 .bar(ax=axs[1], stacked=True, rot=0, title="Share of Experience Level per Period", cmap="Reds"))

axs[0].legend(["Fishing", "Hammock"], loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)
axs[1].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=6)

plt.show()
plt.close()
```

One can clearly see the proportions of experiences in the first period.

## Lagged Choices

In the next step, we want to enrich the model with information on the choice in the previous period. Let us assume that Robinson has troubles storing food on a tropical island so that he incurs a penalty if he enjoys his life in the hammock two times in a row. We add the parameter ``"not_fishing_last_period"`` to the nonpecuniary reward and add a corresponding covariate to the options.

```python
params = """
category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.01
nonpec_hammock,constant,1
nonpec_hammock,not_fishing_last_period,-0.5
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,0
initial_exp_fishing_0,probability,0.33
initial_exp_fishing_1,probability,0.33
initial_exp_fishing_2,probability,0.34
"""

options = {
    "n_periods": 10,
    "simulation_agents": 1_000,
    "covariates": {
        "constant": "1",
        "not_fishing_last_period": "lagged_choice_1 != 'fishing'"
    }
}
```

```python
params = pd.read_csv(io.StringIO(params), index_col=["category", "name"])
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.01

      nonpec_hammock
      constant
      1.00

      not_fishing_last_period
      -0.50

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      0.00

      initial_exp_fishing_0
      probability
      0.33

      initial_exp_fishing_1
      probability
      0.33

      initial_exp_fishing_2
      probability
      0.34

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

    C:\Users\tobia\git\respy\respy\pre_processing\model_processing.py:470: UserWarning: The distribution of initial lagged choices is insufficiently specified in the parameters. Covariates require 1 lagged choices and parameters define 0. Missing lags have equiprobable choices.
      category=UserWarning,
    C:\Users\tobia\git\respy\respy\pre_processing\model_processing.py:470: UserWarning: The distribution of initial lagged choices is insufficiently specified in the parameters. Covariates require 1 lagged choices and parameters define 0. Missing lags have equiprobable choices.
      category=UserWarning,

The warning is raised because we forgot to specify what individuals in period 0 have been doing in the previous period. As a default, **respy** assumes that all choices have the same probability for being the previous choice. Note that, it might lead to inconsistent states where an individual should have accumulated experience in the previous period, but still starts with zero experience.

Below we see the choice probabilities on the left-hand-side and the shares of lagged choices on the right-hand-side. Without further information, **respy** makes all previous choices in the first period equiprobable.

If we had set the covariate with the lagged choice but not added a parameter using the covariate, **respy** would have discarded the covariate and created a model without lagged choices.

```python
fig, axs = plt.subplots(1, 2, figsize=(12.8, 4.8))

(df.groupby("Period").Choice.value_counts(normalize=True).unstack()
 .plot.bar(ax=axs[0], stacked=True, rot=0, title="Choice Probabilities"))
(df.groupby("Period").Lagged_Choice_1.value_counts(normalize=True).unstack().plot
 .bar(ax=axs[1], stacked=True, rot=0, title="Share of Lagged Choice per Period"))

axs[0].legend(["Fishing", "Hammock"], loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)
axs[1].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)

plt.show()
plt.close()
```

What if we wanted a more complex distribution of lagged choices. Remember that Robinson starts with 0, 1 or 2 periods of experience in fishing. Thus, it is natural to assume that higher experience levels correspond to a higher probability for having fishing as a previous choice.

This kind of distribution is more complex because it involves covariates. A flexible solution is to use a **multinomial logit** or **softmax function** to map parameters with covariates to probabilities. This allows to specify very complex distributions, ones which could have been estimated outside the structural model. Here is a short explanation.

For each lag, we have a set of parameters, $\beta^f$ and $\beta^h$, and their corresponding covariates, $x^f$ and $x^h$. The probability $p^f$ for one individual having fishing as their first lagged choice is

$$
    p^f = \frac{e^{x^f \beta^f}}{e^{x^f \beta^f} + e^{x^h \beta^h}}
$$

The probability for hammock as the lagged choice is defined analogously.

In the parameters, you can use the ``lagged_choice_1_fishing`` keyword to define the parameters for fishing as the first lag. You can also define higher order lags using, for example, ``lagged_choice_2_*``.

Let us assume that with each level of initial experience the probability for choosing fishing the previous period rises from 0.5 over 0.75 to 1. For zero experience in fishing, the resulting probabilities should be one halve for fishing and one halve for hammock. First, we create a covariate which is ``True`` if an individual has zero experience in fishing. This covariate is ``{"zero_exp_fishing": "exp_fishing == 0"}``. Then, the parameters for this covariate just have to be equal and can take any value. That is because the softmax function is shift-invariant. The resulting lines in the csv are

    lagged_choice_1_fishing,zero_exp_fishing,1
    lagged_choice_1_hammock,zero_exp_fishing,1

For one period of experience, the weights are 0.75 for fishing and 0.25 for the hammock. First, define a covariate ``{"one_exp_fishing": "exp_fishing == 1"}``. To get the corresponding softmax coefficients for the probabilities, note that you can rewrite the softmax formula replacing $x_i$ with 1. Recognize that the sum in the denominator, $C$, applies to all probabilities and can be discarded. Thus, the coefficients are the logs of the probabilities.

$$\begin{align}
    p_i      &= \frac{e^{x_i \beta_i}}{\sum_j e^{x_j \beta_j}} \\
             &= \frac{e^{\beta_i}}{\sum_j e^{\beta_j}} \\
    log(p_i) &= \beta_i - \log(\sum_j e^{\beta_j}) \\
             &= \beta_i - C
\end{align}$$

The lines in the parameters for $log(0.75)$ and $log(0.25)$ are

    lagged_choice_1_fishing,one_exp_fishing,-0.2877
    lagged_choice_1_hammock,one_exp_fishing,-1.3863

For two experience in fishing, we have to make sure that fishing receives a probability of one which can be achieved by using a fairly large value such as 6. By default, all missing choices receive a parameter with value -1e300.

The final parameters and options are the following:

```python
params = """
category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.01
nonpec_hammock,constant,1
nonpec_hammock,not_fishing_last_period,-0.5
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,0
initial_exp_fishing_0,probability,0.33
initial_exp_fishing_1,probability,0.33
initial_exp_fishing_2,probability,0.34
lagged_choice_1_fishing,zero_exp_fishing,1
lagged_choice_1_hammock,zero_exp_fishing,1
lagged_choice_1_fishing,one_exp_fishing,-0.2877
lagged_choice_1_hammock,one_exp_fishing,-1.3863
lagged_choice_1_fishing,two_exp_fishing,6
"""

options = {
    "n_periods": 10,
    "simulation_agents": 1_000,
    "covariates": {
        "constant": "1",
        "not_fishing_last_period": "lagged_choice_1 != 'fishing'",
        "zero_exp_fishing": "exp_fishing == 0",
        "one_exp_fishing": "exp_fishing == 1",
        "two_exp_fishing": "exp_fishing == 2",
    }
}
```

```python
params = pd.read_csv(io.StringIO(params), index_col=["category", "name"])
params
```

      value

      category
      name

      delta
      delta
      0.9500

      wage_fishing
      exp_fishing
      0.0100

      nonpec_hammock
      constant
      1.0000

      not_fishing_last_period
      -0.5000

      shocks_sdcorr
      sd_fishing
      1.0000

      sd_hammock
      1.0000

      corr_hammock_fishing
      0.0000

      initial_exp_fishing_0
      probability
      0.3300

      initial_exp_fishing_1
      probability
      0.3300

      initial_exp_fishing_2
      probability
      0.3400

      lagged_choice_1_fishing
      zero_exp_fishing
      1.0000

      lagged_choice_1_hammock
      zero_exp_fishing
      1.0000

      lagged_choice_1_fishing
      one_exp_fishing
      -0.2877

      lagged_choice_1_hammock
      one_exp_fishing
      -1.3863

      lagged_choice_1_fishing
      two_exp_fishing
      6.0000

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

    C:\Users\tobia\git\respy\respy\state_space.py:84: UserWarning: Some choices in the model are not admissible all the time. Thus, respy applies a penalty to the utility for these choices which is -400000 by default. For the full solution, the penalty only needs to be larger than all other value functions to be effective. Choose a milder penalty for the interpolation which does not dominate the linear interpolation model.
      "Some choices in the model are not admissible all the time. Thus, respy"

The parameters produce previous choice probabilities on the left-hand-side and the correct conditional probabilities on the right-hand-side. The differences in the probabilities occur due to the low number of simulated individuals.

```python
fig, axs = plt.subplots(1, 2, figsize=(12.8, 4.8))

(df.groupby("Period").Lagged_Choice_1.value_counts(normalize=True).unstack().plot
 .bar(ax=axs[0], stacked=True, rot=0, title="Share of Lagged Choice per Period"))

sns.heatmap((df.query("Period == 0").groupby(["Experience_Fishing"]).Lagged_Choice_1
 .value_counts(normalize="rows").unstack().fillna(0)), ax=axs[1], cmap="RdBu_r", annot=True)

axs[0].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=6)

plt.show()
plt.close()
```

## Observables

Now, we have talked enough about experiences and lagged choices. The last section is about observables which are other characteristics such as whether the island has rich fishing grounds or not. We can also express the distribution of observables as probability mass functions or softmax functions.

Note that observables are sampled first before all other characteristics of individuals. After that, experiences and lagged choices in chronological order (highest lag first) follow. Every group itself is sorted alphabetically. Keep this in mind if you want to condition a distribution on other characteristics.

For observables, it is also possible to use labels instead of numbers to identify levels which might be more convenient and intuitive. The evenly distributed levels are ``"rich"`` and ``"poor"``. The keyword is ``observable_*_*`` and everything after the last ``_`` is considered the name of the level.

```python
params = """
category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.01
nonpec_fishing,rich_fishing_grounds,0.5
nonpec_hammock,constant,1
nonpec_hammock,not_fishing_last_period,-0.5
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,0
initial_exp_fishing_0,probability,0.33
initial_exp_fishing_1,probability,0.33
initial_exp_fishing_2,probability,0.34
lagged_choice_1_fishing,zero_exp_fishing,1
lagged_choice_1_hammock,zero_exp_fishing,1
lagged_choice_1_fishing,one_exp_fishing,-0.2877
lagged_choice_1_hammock,one_exp_fishing,-1.3863
lagged_choice_1_fishing,two_exp_fishing,6
observable_fishing_grounds_rich,probability,0.5
observable_fishing_grounds_poor,probability,0.5
"""

options = {
    "n_periods": 10,
    "simulation_agents": 10_000,
    "covariates": {
        "constant": "1",
        "not_fishing_last_period": "lagged_choice_1 != 'fishing'",
        "zero_exp_fishing": "exp_fishing == 0",
        "one_exp_fishing": "exp_fishing == 1",
        "two_exp_fishing": "exp_fishing == 2",
        "rich_fishing_grounds": "fishing_grounds == 'rich'",
    }
}
```

```python
params = pd.read_csv(io.StringIO(params), index_col=["category", "name"])
params
```

      value

      category
      name

      delta
      delta
      0.9500

      wage_fishing
      exp_fishing
      0.0100

      nonpec_fishing
      rich_fishing_grounds
      0.5000

      nonpec_hammock
      constant
      1.0000

      not_fishing_last_period
      -0.5000

      shocks_sdcorr
      sd_fishing
      1.0000

      sd_hammock
      1.0000

      corr_hammock_fishing
      0.0000

      initial_exp_fishing_0
      probability
      0.3300

      initial_exp_fishing_1
      probability
      0.3300

      initial_exp_fishing_2
      probability
      0.3400

      lagged_choice_1_fishing
      zero_exp_fishing
      1.0000

      lagged_choice_1_hammock
      zero_exp_fishing
      1.0000

      lagged_choice_1_fishing
      one_exp_fishing
      -0.2877

      lagged_choice_1_hammock
      one_exp_fishing
      -1.3863

      lagged_choice_1_fishing
      two_exp_fishing
      6.0000

      observable_fishing_grounds_rich
      probability
      0.5000

      observable_fishing_grounds_poor
      probability
      0.5000

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

```python
fig, axs = plt.subplots(1, 2, figsize=(12.8, 4.8))

(df.groupby("Fishing_Grounds").Choice.value_counts(normalize=True).unstack().plot
 .bar(ax=axs[0], stacked=True, rot=0, title="Choice Probabilities"))

(df.Fishing_Grounds.value_counts(normalize=True)
 .fillna(0).plot.bar(stacked=True, ax=axs[1], rot=0, title="Observable Distribution"))

axs[0].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)
axs[1].legend(loc="upper center", bbox_to_anchor=(0.5, -0.15), ncol=2)

plt.show()
plt.close()
```

You can see in the figure on the right-hand-side that poor and rich fishing grounds are evenly distributed. The figure on the left-hand-side shows that rich fishing grounds lead to higher engagement in fishing.

## Conclusion

We showed ...

- how to express different distributions, probability mass functions or softmax functions, in the parameters of a respy model.
- that you can use numbers and labels for discrete levels of observables.

Happy modeling!


```python
%matplotlib inline
import re as re
import pandas as pd
import numpy as np
import seaborn as sbn
from scipy.stats import norm
import matplotlib.pyplot as plt
import warnings as warnings
warnings.filterwarnings('ignore')

sbn.set_style('white')
sbn.set_context('talk')

np.random.seed(12345678)
```

> Note: these are my personal opinions and I'm sure that lots of people would disagree with my assessment of available software packages.

We have been working on getting under the hood of Metropolis-Hastings and Monte Carlo Markov Chain.  But there are some very accessible Metropolis Markov Chain software packages out there.  These are my personal views on what is available:

1. Stata (`bayesianmh`)- just available in Stata 14.  To be honest, I have never heard of anyone using Stata for Bayesian Sampling Methods although I haven't really looked.
2. Matlab.  There are numerous user and Matlab written routines.  For example, there is a version of `emcee` that is implemented there (more on this later in the course).  Matlab is for people who want to possibly tweak their own sampler code and who need the fastest possible computation.
2. SAS.  A decent set of packages, not used very much in the social sciences (maybe in the medical sciences??), but is well suited for big data.
3. R.  For a long time R was the de-facto standard and probably still is, with packages like R-stan which interfaces the `STAN` C Library.  Also `Jags`, `Rbugs` and `Open BUGS` have been around for a long time. I almost elected to teach this course in R.
4. Python. The libraries aren't as rich as R, but you might view Python as sitting between Matlab and R.  Maybe (in my view probably) also easier to code your own samplers than R as I really prefer python syntax to R.  Python is the used a lot in many fields including physics and is strong in the big-data arena, more so than any of the other packages mentioned above. The major players on python are:
    * PyMC and PyMC3 (in beta)
    * PyStan
    * EMCEE

Today, we are going to focus on PyMC3, which is a very easy to use package now that we have a solid understanding of how posteriors are constructed.

We will start with our very simple one parameter model and then move to slightly more complicated settings:

```python
sigma = 3 # Note this is the std of our data
data = norm(10,sigma).rvs(100)
mu_prior = 8
sigma_prior = 1.5  # Note this is our prior on the std of mu

plt.hist(data,bins=20)
plt.show()
```

# PyMC3

Let's use PyMC3 to model this:

```python
import pymc3 as pm3
```

This is the model statement describing priors and the likelihood.  Here, `mu` is defined as a stochastic variable (we want a chain of sampled values for this variable) and we provide a prior distribution and hyper-parameters for it.  The likelihood function is chosen to be Normal, with one parameter to be estimated (`mu`), and we use known $\sigma$ (denoted as `sigma`).  Our "dependent variable" is given by `observed=data`, where `data` is generated above and shown in the histogram.  In more formal terms, the code below sets up a `basic_model` having the following form:

\begin{align}
Prob(\mu|\sigma,\mathbf{data}) \propto& Prob(\mathbf{data}|\mu,\sigma) \times Prob(\mu | \mu^0_\mu, \sigma^0_\mu) \\
Prob(\mu|\sigma,\mathbf{data}) \propto& \phi \left( \frac{\mathbf{data} - \mu}{\sigma} \right ) \times \phi \left(\frac{\mu - \mu^0_\mu}{\sigma^0_\mu} \right ) \\
Prob(\mu|\sigma,\mathbf{data}) \propto& \phi \left( \frac{\mathbf{data} - \mu}{3} \right ) \times \phi \left(\frac{\mu - 8}{1.5} \right )
\end{align}
where $\phi(.)$ is the standard normal pdf.

```python
basic_model = pm3.Model()

with basic_model:

    # Priors for unknown model parameters
    mu = pm3.Normal('Mean of Data',mu_prior,sigma_prior)

    # Likelihood (sampling distribution) of observations
    data_in = pm3.Normal('Y_obs', mu=mu, sd=sigma, observed=data)
```

This defines how the chain will be constructed.  We will set startvals at MAP and use a Metropolis step method (which is Random Walk MH that we have studied).

```python
chain_length = 10000

with basic_model:
    # obtain starting values via MAP
    startvals = pm3.find_MAP(model=basic_model)
    print(startvals)
    # instantiate sampler
    step = pm3.Metropolis()

    # draw 5000 posterior samples
    trace = pm3.sample(chain_length, step=step, start=startvals)
```

    logp = -290.67, ||grad|| = 31.153: 100%|██████████| 4/4 [00:00<00:00, 1555.17it/s]

    {'Mean of Data': array(10.69594596)}

    Multiprocess sampling (4 chains in 4 jobs)
    Metropolis: [Mean of Data]
    Sampling 4 chains: 100%|██████████| 42000/42000 [00:04<00:00, 9224.18draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
pm3.traceplot(trace,figsize=(20,5));
```

```python
pm3.summary(trace)
```

      mean
      sd
      mc_error
      hpd_2.5
      hpd_97.5
      n_eff
      Rhat

      Mean of Data
      10.698323
      0.298304
      0.003388
      10.127138
      11.290343
      8491.809956
      1.00013

## Understanding the PyMC3 Results Object

All the results are contained in the `trace` variable.  This is a pymc3 results object. It contains some information that we might want to extract at times.  Varnames tells us all the variable names setup in our model.

```python
trace.varnames
```

    ['Mean of Data']

With the variable names, we can extract chain values for each variable:

```python
trace['Mean of Data']
```

    array([10.47837212, 10.47837212, 10.47837212, ..., 10.65640358,
           10.65640358, 10.47800609])

So for each variable, we have an array of values, which is our chain.

## Diagnostics

### Autocorrelation Plots

```python
pm3.plots.autocorrplot(trace,figsize=(17,5));
```

### Acceptance Rate

```python
accept = np.sum(trace['Mean of Data'][1:] != trace['Mean of Data'][:-1])
print("Acceptance Rate: ", accept/trace['Mean of Data'].shape[0])
```

    Acceptance Rate:  0.3393

### Geweke Score

```python
score=pm3.geweke(trace, first=0.1, last=0.5, intervals=20)
score
```

    {0: {'Mean of Data': array([[ 0.00000000e+00,  3.98089881e-02],
             [ 2.63000000e+02,  3.62035946e-02],
             [ 5.26000000e+02, -2.17552281e-02],
             [ 7.89000000e+02, -4.24670150e-02],
             [ 1.05200000e+03, -3.11835563e-02],
             [ 1.31500000e+03,  1.68953273e-02],
             [ 1.57800000e+03, -4.88126007e-02],
             [ 1.84100000e+03, -3.36627389e-02],
             [ 2.10400000e+03, -1.12498869e-01],
             [ 2.36700000e+03, -9.30590373e-02],
             [ 2.63000000e+03, -5.75132684e-02],
             [ 2.89300000e+03, -7.35140470e-02],
             [ 3.15600000e+03, -4.69262730e-02],
             [ 3.41900000e+03,  2.63093107e-02],
             [ 3.68200000e+03,  7.54162558e-02],
             [ 3.94500000e+03,  1.94482040e-02],
             [ 4.20800000e+03, -1.93971697e-02],
             [ 4.47100000e+03, -4.05472731e-02],
             [ 4.73400000e+03,  7.72650011e-02],
             [ 4.99700000e+03,  1.40478948e-01]])},
     1: {'Mean of Data': array([[ 0.00000000e+00,  5.70312640e-04],
             [ 2.63000000e+02,  1.85112715e-02],
             [ 5.26000000e+02, -2.74031618e-02],
             [ 7.89000000e+02,  4.82398836e-02],
             [ 1.05200000e+03,  9.18104690e-02],
             [ 1.31500000e+03,  8.91433530e-02],
             [ 1.57800000e+03,  1.02104854e-02],
             [ 1.84100000e+03, -4.78043809e-02],
             [ 2.10400000e+03, -6.67302844e-03],
             [ 2.36700000e+03, -7.82674713e-03],
             [ 2.63000000e+03,  4.37846176e-02],
             [ 2.89300000e+03,  9.97546903e-02],
             [ 3.15600000e+03,  1.31289828e-01],
             [ 3.41900000e+03,  3.92879936e-02],
             [ 3.68200000e+03,  2.38198177e-02],
             [ 3.94500000e+03, -3.02272942e-02],
             [ 4.20800000e+03, -2.16042161e-02],
             [ 4.47100000e+03, -4.52155240e-02],
             [ 4.73400000e+03, -3.60629631e-02],
             [ 4.99700000e+03,  6.10076770e-02]])},
     2: {'Mean of Data': array([[ 0.00000000e+00, -2.47037184e-02],
             [ 2.63000000e+02, -3.91283185e-03],
             [ 5.26000000e+02, -7.22673002e-03],
             [ 7.89000000e+02,  5.16491920e-02],
             [ 1.05200000e+03,  2.52421875e-02],
             [ 1.31500000e+03,  3.86004067e-02],
             [ 1.57800000e+03,  3.56050647e-02],
             [ 1.84100000e+03, -1.39679233e-02],
             [ 2.10400000e+03, -2.67245021e-02],
             [ 2.36700000e+03, -3.21385286e-02],
             [ 2.63000000e+03,  3.72225487e-02],
             [ 2.89300000e+03,  4.24695530e-02],
             [ 3.15600000e+03,  7.05407670e-02],
             [ 3.41900000e+03, -2.46672820e-02],
             [ 3.68200000e+03, -8.04268212e-02],
             [ 3.94500000e+03, -1.12151011e-01],
             [ 4.20800000e+03, -1.60494897e-03],
             [ 4.47100000e+03, -9.20122764e-03],
             [ 4.73400000e+03, -2.16318581e-02],
             [ 4.99700000e+03,  2.45134345e-02]])},
     3: {'Mean of Data': array([[ 0.00000000e+00, -4.38108672e-03],
             [ 2.63000000e+02, -2.38006177e-02],
             [ 5.26000000e+02,  1.01550281e-02],
             [ 7.89000000e+02, -2.16886777e-02],
             [ 1.05200000e+03, -1.93968113e-02],
             [ 1.31500000e+03, -9.44745864e-03],
             [ 1.57800000e+03, -6.53750008e-03],
             [ 1.84100000e+03, -5.39005173e-02],
             [ 2.10400000e+03, -7.98831893e-02],
             [ 2.36700000e+03, -7.28944353e-02],
             [ 2.63000000e+03, -4.80856383e-02],
             [ 2.89300000e+03,  2.45622360e-02],
             [ 3.15600000e+03, -8.76509902e-03],
             [ 3.41900000e+03, -6.44125023e-02],
             [ 3.68200000e+03, -1.37897022e-01],
             [ 3.94500000e+03, -1.22975064e-01],
             [ 4.20800000e+03, -1.46453088e-01],
             [ 4.47100000e+03, -1.42797365e-01],
             [ 4.73400000e+03, -5.50912522e-02],
             [ 4.99700000e+03, -7.57302253e-02]])}}

```python
score=pm3.geweke(trace, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1], marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
plt.title('Geweke Plot Comparing first 10% and Slices of the Last 50% of Chain\nDifference in Mean Z score')
plt.show()
```

# Gelmen-Rubin

For GR, we need multiple chains in order to test if each of the chains collapse to the limiting distribution which is our posterior.  PyMC3 makes it very easy to generate these.

```python
chain_length = 100000

with basic_model:
    # obtain starting values via MAP
    startvals = pm3.find_MAP(model=basic_model)
    print(startvals)
    # instantiate sampler
    step = pm3.Metropolis()

    # draw 5000 posterior samples
    trace = pm3.sample(chain_length, step=step, start=startvals, njobs=2)
```

    logp = -290.67, ||grad|| = 31.153: 100%|██████████| 4/4 [00:00<00:00, 2157.56it/s]

    {'Mean of Data': array(10.69594596)}

    Multiprocess sampling (2 chains in 2 jobs)
    Metropolis: [Mean of Data]
    Sampling 2 chains: 100%|██████████| 201000/201000 [00:24<00:00, 8200.02draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
pm3.traceplot(trace,figsize=(20,5));
```

```python
pm3.gelman_rubin(trace)
```

    {'Mean of Data': 0.9999951110055701}

Based on Gelman-Rubin and Geweke, we can be confident we have a chain that has converged to the limiting distribution (although perhaps for Gelman-Rubin, we'd want to try different starting values rather than MAP).

# In class excercise

Team up and modify the code below to estimate sigma as well as the mean:

```python
basic_model = pm3.Model()

with basic_model:

    # Priors for unknown model parameters
    mu = pm3.Normal('Mean of Data',mu_prior,sigma_prior)

    # Likelihood (sampling distribution) of observations
    data_in = pm3.Normal('Y_obs', mu=mu, sd=sigma, observed=data)
```

```python
chain_length = 10000

with basic_model:
    # obtain starting values via MAP
    startvals = pm3.find_MAP(model=basic_model)
    print(startvals)
    # instantiate sampler
    step = pm3.Metropolis()

    # draw 5000 posterior samples
    trace = pm3.sample(chain_length, step=step, start=startvals)
```

    logp = -290.67, ||grad|| = 31.153: 100%|██████████| 4/4 [00:00<00:00, 1577.25it/s]
    Multiprocess sampling (4 chains in 4 jobs)
    Metropolis: [Mean of Data]

    {'Mean of Data': array(10.69594596)}

    Sampling 4 chains: 100%|██████████| 42000/42000 [00:05<00:00, 7569.48draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
score=pm3.geweke(trace, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1], marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
plt.title('Geweke Plot Comparing first 10% and Slices of the Last 50% of Chain\nDifference in Mean Z score')
plt.show()
```


# 8: Likelihood inference

*(note: in order to run this notebook, first run 20 creating the data and 30, adding a BDT)*

In most analysis, fitting a model to data is a crucial, last step in the chain in order to extract information about the parameters of interest.

**Model**: First, this involves building a model that depends on various parameters - some that are of immediate interest to us (Parameter Of Interest, POI) and some that help to describe the observables that we expect in the data but do not have any relevant physical meaning such as detectoreffects (Nuisance Parameter).

**Loss**: With the model and the data, a likelihood (or sometimes also a $\chi^2$ loss) is built. This is a function of the parameters and reflects the _likelihood of finding the data given the parameters_. The most likely parameter combination is retrieved when maximising the likelihood; also called the Maximum Likelihood (ML) estimator. Except for trivial cases, this has to be done numerically.

While this procedure returns an estimate of the _best-fit_, it does not say anything about the _uncertainty_ of our measurement. Therefore, advanced statistical methods are needed that perform toy studies; these are fits to generated samples under different conditions such as fixing a parameter to a certain value.

## Scope of this tutorial

Here, we focus on the libraries that are available to perform likelihood fits. For unbinned fits in Python, the libraries that we will consider here are [zfit](https://github.com/zfit/zfit) (likelihood model fitting library) and [hepstats](https://github.com/scikit-hep/hepstats) (higher level statistical inference; can use `zfit` models), both which are relatively young and written in pure Python.

An alternative to be mentioned is [RooFit](http://roofit.sourceforge.net/) and [RooStats](https://twiki.cern.ch/twiki/bin/view/RooStats/WebHome), an older, well-proven, reliable C++ framework that has Python bindings and acts as a standard in HEP for this kind of fits.

In case your analysis involved pure binned and templated fits, [pyhf](https://github.com/scikit-hep/pyhf) provides a specialized library for this kind of fits.

## Getting started

In order to get started with `zfit`, it is recommendable to have a look at the [zfit-tutorials](https://zfit-tutorials.readthedocs.io/en/latest/index.html) or the [recorded](https://www.youtube.com/playlist?list=PLAuzjeTfC3tNeP6o2sbCr3Jsa6ATvEYR6) [tutorials](https://www.youtube.com/watch?v=YDW-XxrSbns), which also contains `hepstats` tutorials; more the latter can be found [here](https://github.com/scikit-hep/hepstats/tree/master/notebooks).

Before we continue, it is highly recommended to follow the [zfit introduction tutorial](https://zfit-tutorials.readthedocs.io/en/latest/tutorials/introduction/README.html)

```python
%store -r bkg_df
%store -r mc_df
%store -r data_df
```

```python
import hepstats
import matplotlib.pyplot as plt
import mplhep
import numpy as np
import zfit
```

```python
# apply cuts
query = 'BDT > 0.4'
data_df.query(query, inplace=True)
mc_df.query(query, inplace=True)

# reduce the datasize for this example to make the fit more interesting
fraction = 0.1  # how much to take of the original data
data_df = data_df.sample(frac=0.1)
```

**Exercise**: write a fit to the data using `zfit`

```python
obs = zfit.Space('Jpsi_M', limits=(2.8, 3.5))  # defining the observable
```

```python
# bkg = zfit.Data.from_pandas(bkg_df['Jpsi_M'], obs=obs)
# OR
# obs_bkg = zfit.Space('Jpsi_M', limits=(2.8, 3.0)) + zfit.Space('Jpsi_M', limits=(3.2, 3.5))
# bkg_two = zfit.Data.from_pandas(data_df['Jpsi_M'], obs=obs_bkg)
```

```python
mc = zfit.Data.from_pandas(mc_df['Jpsi_M'], obs=obs)
data = zfit.Data.from_pandas(data_df['Jpsi_M'], obs=obs)
```

## Difference of the two spaces

While the first space is defined over the whole space from 2.8 to 3.5, the second consists of two distinct regions. Therefore we can use the original space and zfit applies the cut, the same as we did before to the `bkg_df`.

The difference comes when using the normalization in the PDF: we can either normalize it over the whole range or only over part of it.

```python
lambd = zfit.Parameter('lambda', -0.1, -2, 2)
bkg_yield = zfit.Parameter('bkg_yield', 5000, 0, 200000, step_size=1)

mu = zfit.Parameter('mu', 3.1, 2.9, 3.3)
sigma = zfit.Parameter('sigma', 0.1, 0, 0.5)
sig_yield = zfit.Parameter('sig_yield', 200, 0, 10000, step_size=1)
```

```python
bkg_pdf = zfit.pdf.Exponential(lambd, obs=obs)
bkg_pdf.set_yield(bkg_yield)
```

```python
sig_pdf = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma)
sig_pdf.set_yield(sig_yield)
```

```python
model = zfit.pdf.SumPDF([bkg_pdf, sig_pdf])
```

## Plotting

Plots can simply be made with `matplotlib` and `mplhep`.

```python
def plot_fit(model, data, nbins=30, ax=None):
    # The function will be reused.
    if ax is None:
        ax = plt.gca()

    lower, upper = data.space.limit1d

    # Creates and histogram of the data and plots it with mplhep.
    counts, bin_edges = np.histogram(data.unstack_x(), bins=nbins)
    mplhep.histplot(counts, bins=bin_edges, histtype="errorbar", yerr=True,
                    label="Data", ax=ax, color="black")

    binwidth = np.diff(bin_edges)[0]
    x = np.linspace(lower, upper, num=1000)  # or tf.linspace

    # Line plots of the total pdf and the sub-pdfs.
    y = model.ext_pdf(x) * binwidth
    ax.plot(x, y, label="total", color="royalblue")
    for m, l, c in zip(model.get_models(), ["background", "signal"], ["forestgreen", "crimson"]):
        ym = m.ext_pdf(x) * binwidth
        ax.plot(x, ym, label=l, color=c)

    plt.xlabel('$J/\\psi$ mass [GeV]')
    ax.set_title(data.data_range.obs[0])
    ax.set_xlim(lower, upper)
    ax.legend(fontsize=15)

    return ax
```

```python
plot_fit(model, data)  # before the fit
```

## Loss

Since we have now the models and the datasets, we can e.g. pre-fit the signal PDF to the simulation.

```python
sig_nll = zfit.loss.UnbinnedNLL(sig_pdf, mc)
```

It warns us that we are using a non-extended loss. The extended loss also includes to fit the yield while the normal one does not. Since we want to fit the _shape_ here only, we use the non-extended one.

```python
minimizer = zfit.minimize.Minuit()
# minimizer = zfit.minimize.NLoptLBFGSV1()  # can be changed but maybe not as powerful as iminuit
# minimizer = zfit.minimize.ScipySLSQPV1()
```

```python
minimizer.minimize(sig_nll)
```

### Fixing parameters

Sometimes we want to fix parameters obtained from MC, such as tailes. Here we will fix the `sigma`, just for demonstration purpose.

```python
sigma.floating = False
```

```python
nll = zfit.loss.ExtendedUnbinnedNLL(model, data)
```

```python
result = minimizer.minimize(nll)
```

```python
result
```

```python
result.hesse()  # calculate hessian error
result.errors()  # profile, using minos like uncertainty
print(result)
```

```python
plot_fit(model, data)
```

**Exercise**: use `hepstats` to check the significance of our signal

```python
from hepstats.hypotests.calculators import AsymptoticCalculator
from hepstats.hypotests.parameters import POI

# the null hypothesis
sig_yield_poi = POI(sig_yield, 0)
calculator = AsymptoticCalculator(input=result, minimizer=minimizer)
```

There is another calculator in `hepstats` called `FrequentistCalculator` which constructs the test statistic distribution $f(q_{0} |H_{0})$ with pseudo-experiments (toys), but it takes more time.

The `Discovery` class is a high-level class that takes as input a calculator and a `POI` instance representing the null hypothesis, it basically asks the calculator to compute the p-value and also computes the signifance as

\begin{equation}
Z = \Phi^{-1}(1 - p_0).
\end{equation}

```python
from hepstats.hypotests import Discovery

discovery = Discovery(calculator=calculator, poinull=sig_yield_poi)
discovery.result()
```

**Exercise** play around! First things first: repeat the fit. The difference we will see is statistical fluctuation from the resampling of the data; we take only a fraction at random.

Change the fraction of data that we have, the BDT cut, the signal model...

Attention: it is easy to have a significance of `inf` here