# Huawei Research France

## Acrobot RAMP: system identification

### Forecast the state of the altered acrobot system from history and actions

_Balázs Kégl, Gabriel Hurtado, Jianfeng Zhang, Albert Thomas, Ludovic Dos Santos (Huawei Research, Noah's Ark Laboratory, France)_

What is the probability that the world ends if I press this button?

## Introduction

In recent year we have witnessed a phenomenal success of deep reinforcement learning algorithms beating top human players in [go](https://en.wikipedia.org/wiki/AlphaGo), [chess](https://en.wikipedia.org/wiki/AlphaZero) and even [poker](https://ai.facebook.com/blog/pluribus-first-ai-to-beat-pros-in-6-player-poker/). Why don't we see the widespread use of these algorithms in application domains half of the world is after, such as engineering system optimization or robotics? It turns out that one of the main conditions of these successes is the access to a simulator that can generate unlimited data to learn from.

_"The real world will not become faster in a few years, contrary to computers"_ ([Chatzilygeroudis et al. 2019](https://arxiv.org/pdf/1807.02303.pdf))

If we want to control engineering systems, we need to be able to learn to control them with limited access to data. In this regime, one of the promising directions is [model-based reinforcement learning](https://arxiv.org/abs/1907.02057). The main component of these algorithms is a learnable system model. Its objective is to learn to forecast system behavior, given past history and control actions.

Classical time series forecasting methods provide point predictions: "the minimum temperature tomorrow is 5 degrees Celsius". Without knowing the _certainty_ of the prediction, it is hard to train a controller, especially when extreme events need to be modeled ("OK, but what's the probability that the plants on my balcony will freeze?")

On the other hand, powerful and popular Bayesian forecasters do exist but they do not scale with either the length of the history or the system state dimensions.

**The goal of this challenge to develop scalable and powerful generative probabilistic time series forecasters.** We will provide you with training and testing traces from a physical system based on the [acrobot benchmark](https://gym.openai.com/envs/Acrobot-v1). We are challenging you to develop generic learning algorithms, so we made manual reverse engineering harder by changing the physics of the original system.

### Formal task

You will be given a training multivariate time series ${\cal T} = \big[(o_t, a_t)\big]_{t=1}^T$ of observables $o_t \in \mathbb{R}^4$ and actions $a_t \in \{0, 1, 2\}$. You will need to learn to predict the density function of the observables at next step $o_{t+1}$, conditioned on the history:

\begin{equation}
  p\big(o_{t+1} | (o_1, a_1), \ldots, (o_t, a_t)\big).
\end{equation}

You will be evaluated on a test sequence ${\cal T}^\prime = \big[(o_t^\prime, a_t^\prime)\big]_{t=1}^{T^\prime}$ using the likelihood ratio

\begin{equation}
LR({\cal T}^\prime; p) = \frac{\prod_{t=1}^{T^\prime-1} p\big(o^\prime_{t+1} | (o^\prime_1, a^\prime_1), \ldots, (o^\prime_t, a^\prime_t)\big)}{L_\text{b}({\cal T}^\prime)},
\end{equation}

where $L_\text{b}({\cal T}^\prime)$ is a baseline (multivariate spherical Gaussian) likelihood.

${\cal T}$ and ${\cal T}^\prime$ will always come from the same dynamical system. However, to make manual reverse engineering hard, the time series in this public starting kit and the times series on the data challenge server will come from different physics, so your model should be prepared to be transferred to a (slightly) different system.

We will keep the training set size the same, $T \simeq 5000$ in both the public and private training sets.

### Challenges
1. We do not know whether the system observables $o_t$ and action $a_t$ at time $t$ are sufficient to predict $o_{t+1}$. Therefore we will explicitly ask you to develop (or learn) a feature extractor $f_\text{FE}$ wich will convert the history up to time $t$ into a fixed length state vector $s_t$. Formally,
\begin{equation}
  s_t = f_\text{FE}\big((o_1, a_1), \ldots, (o_t, a_t)\big),
\end{equation}
and the predictor will then simplify to
\begin{equation}
  p\big(o_{t+1} | (o_1, a_1), \ldots, (o_t, a_t)\big) = p(o_{t+1} | s_t).
\end{equation}
2. The predictor $p(o_{t+1} | s_t)$ is not a classical regressor outputting a point estimate $\hat{o}_{t+1}$ but a $d$-dimensional density ($d=4$). We needed to find a numerical representation for such a function with two requirements: 1) computationally easy evaluation of the likelihood and 2) computationally easy simulation (sampling). The first trick is that we represent the multivariate density $p(o_{t+1} | s_t)$ by $d$ univariate densities using the chain rule
\begin{equation}
  p(o_{t+1} | s_t) = p_1(o^1_{t+1} | s_t) \prod_{j=2}^d p_j\big(o^j_{t+1} |o^1_{t+1}, \ldots, o^{j-1}_{t+1}, s_t\big),
\end{equation}
where $o_{t+1} = \big(o^1_{t+1},\ldots,o^d_{t+1}\big)$. This means that you will be asked to learn $d$ (generative) regressors $p_1, \ldots, p_d$.
3. The second trick is that we will represent each generative regressor $p_j$ by a mixture of $L_j$ ($L_j \le 100$) simple parametric components
\begin{equation}
  p_j\big(o^j_{t+1} |o^1_{t+1}, \ldots, o^{j-1}_{t+1}, s_t\big) = \sum_{\ell=1}^{L_j} w_\ell {\cal P}_\ell(o^j_{t+1}; \theta_\ell).
\end{equation}
The $j$th regressor $p_j$ will thus map its vector input $\big(o^1_{t+1}, \ldots, o^{j-1}_{t+1}, s_t\big)$ onto a sequence of weights, densities, and density parameters $\big[(w, {\cal P}, \theta)_\ell\big]_{\ell=1}^{L_j}$.

This may sound complicated, but in fact it is relatively easy to convert both random forests and neural nets into generative regressors of this kind, and we will provide starting kit examples for both of these popular function classes. Also note that the intricacies of training set generation for each regressor will be taken care of the workflow behind the scenes; all you need to do is to parameterize the regressors.

## Competition rules

* Submissions will be trained on a time series of roughly 5000 time steps and tested on a time series of roughly 20000 time steps.
* The competition will end on November 22, 2019 at 19h UTC (20h in Paris).
* All models will be trained on the same cloud server allowing 4 CPUs (with shared memory of 128GB RAM).
* Participants will be given a total of 20 machine hours. Submissions of a given participant will be ordered by submission timestamp. We will make an attempt to train all submissions, but starting from (and including) the first submission that makes the participant's total training time exceed 20 hours, all submissions will be disqualified from the competition (but can enter into the collaborative phase). Testing time will not count towards the limit. Training time will be displayed on the leaderboard for all submissions, rounded to second. If a submission raises an exception, its training time will not count towards the total.
* There is a timeout of 1 day between submissions.
* Submissions submitted after the end of the competition will not qualify for prizes.
* The public leaderboard will display validation scores running a cross-validation. The official scores will be calculated on the hidden test set and will be published after the closing of the competition. We will rank submissions according to their likelihood ratio score.
* The organizers will do their best so that the provided backend runs flawlessly. We will communicate with participants in case of concerns and will try to resolve all issues, but we reserve the right to make unilateral decisions in specific cases, not covered by this set of minimal rules.
* The organizers reserve the right to disqualify any participant found to violate the fair competitive spirit of the challenge. Possible reasons, without being exhaustive, are multiple accounts, attempts to access the test data, etc.
* The challenge is essentially an individual contest, so there is no way to form official teams. Participants can form teams outside the platform before submitting any model individually, and submit on a single team member's account. However, submitting on one's own and participating in such a team at the same time is against the "no multiple accounts" rule, so, if discovered, may lead to disqualification.
* Participants retain copyright on their submitted code and grant reuse under BSD 3-Clause License.

Participants accept these rules automatically when making a submission at the RAMP site.

### Installation instructions are found in the [README](https://github.com/ramp-kits/acrobot/blob/master/README.md)

```python
%matplotlib inline
import os
import glob
import numpy as np
from scipy import io
import pandas as pd
import rampwf as rw
import xarray as xr

import altair as alt
alt.renderers.enable('notebook');
```

You can load `problem.py` to have access the RAMP setup.

```python
problem = rw.utils.assert_read_problem()
```

## Exploratory data analysis

### Loading the data

Let's load the data. `X_ds` is `xr.Dataset` which is a versatile container. We will not use many features of it so most of the time it will be converted to `pandas.Dataframe`. `y_array` is a numpy array.

```python
X_ds, y_array = problem.get_train_data()
X_ds
```

    <xarray.Dataset>
    Dimensions:      (time: 4999)
    Coordinates:
      * time         (time) datetime64[ns] 2019-08-05 2019-08-06 ... 2036-07-24
    Data variables:
        thetaDot2    (time) float64 -0.025 -0.0666 -0.4516 ... -11.23 -8.425 -11.7
        theta2       (time) float64 0.05051 0.04131 -0.01101 ... -1.367 3.055 1.109
        thetaDot1    (time) float64 0.004686 -0.02917 0.09464 ... -2.737 0.6118
        theta1       (time) float64 0.03029 0.02784 0.03448 ... -0.448 -0.7157
        action       (time) float64 1.0 0.0 1.0 0.0 0.0 1.0 ... 2.0 2.0 2.0 2.0 2.0
        restart      (time) int64 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0 0
        y_thetaDot2  (time) float64 -0.0666 -0.4516 -0.4365 ... -8.425 -11.7 -10.98
        y_theta2     (time) float64 0.04131 -0.01101 -0.1006 ... 3.055 1.109 -1.311
        y_thetaDot1  (time) float64 -0.02917 0.09464 0.04833 ... 0.6118 0.3024
        y_theta1     (time) float64 0.02784 0.03448 0.04905 ... -0.7157 -0.4942
    Attributes:
        n_burn_in:  0

`X_ds` is indexed by time. The unit (day) is arbitrary, in reality acrobot is moving much faster. The first four variables are the system observables $o_t \in \mathbb{R}^4$. For more information, we refer to the [acrobot site](https://gym.openai.com/envs/Acrobot-v1). The semantics of the variables is the same as in the original system, but **we altered the physics that generated the time series to make reverse engineering harder**. Also note that the data in the starting kit and the data in the challenge server were also generated by different physics since we would like you to learn from data not to manually figure out the dynamics and hard code it in your solutions.

`actions` is a ternary action: `0.0` means kick to the left, `1.0` means no kick, `2.0` means kick to the right. Indeed, the change of `thetaDot2` (the angular speed of the second joint) is roughly correlated with the action.

```python
print((X_ds.thetaDot2[1:].values - X_ds.thetaDot2[:-1].values)[X_ds.action[:-1]==0.0].mean())
print((X_ds.thetaDot2[1:].values - X_ds.thetaDot2[:-1].values)[X_ds.action[:-1]==1.0].mean())
print((X_ds.thetaDot2[1:].values - X_ds.thetaDot2[:-1].values)[X_ds.action[:-1]==2.0].mean())
```

    -0.287894118268408
    0.0023133801874445645
    0.2884619102470254

`restart` is a binary helper column:  `0.0` means the time step is a continuation of the previous trace, `1.0` means that we start a new trace from a (roughly) vertically hanging starting state. In the public training data there are several restarts, meaning that we have several traces. The cross validation will be done by these traces, so in the public training we will have seven folds. In the private training data at the challenge server we may have a different number of restarts. The test data can also contain several traces, delineated in the same way.

```python
X_ds.restart.values.sum()
```

    6

The rest of the columns are identical to `y_array`, which is the prediction target. It corresponds to the system observables $o_t$ (first six columns), shifted backwards by one, meaning that the target is to predict the system observable $o_{t+1}$ at time $t+1$. The reason for `X_ds` (the input) containing the target will be clear later.

### Visualizing the data

We now plot the system variables against time. You can explore different segments of training (and test) sets by changing `start_time` and `end_time` (and loading the test set on the top of the notebook).

```python
figs = []
start_time = 1303
end_time = 1680
for observable_col in problem._target_column_observation_names:
    ground_truth = X_ds.to_dataframe().reset_index().reset_index()
    line_gt = alt.Chart(ground_truth[start_time:end_time]).mark_line(color='black').encode(
        x=alt.X('index:Q', title='time step'),
        y=alt.Y(observable_col + ':Q', scale=alt.Scale(zero=False)),
    )
    fig = line_gt
    figs.append(fig)
alt.vconcat(alt.hconcat(*figs[:2]), alt.hconcat(*figs[2:]))
```

## The pipeline: what you need to develop

For submitting at the challenge site, you will have to write two classes, saved in two different files:
* the class `FeatureExtractor` in `ts_feature_extractor.py`, which will be used to extract features for representing the system state at time $t$.
* the class `GenerativeRegressor` in `generative_regressor.py` to predict the distribution of the six system variables at time $t+1$ from the features returned by the feature extractor.

### Feature extractor

The feature extractor implements a `transform` member function. It is saved in the file [`submissions/starting_kit/ts_feature_extractor.py`](submissions/starting_kit/ts_feature_extractor.py). It receives an xarray Dataset `X_ds` defined at the beginning of the notebook. It should produce a pandas DataFrame containing the extracted features, which will then be used for regression.

Be careful not to use any information from the future (`X_ds[t + 1:]`) when constructing `X_df[t]`. We have implemented a check for that in the training workflow, but any attempt to purposefully work around those checks may lead to disqualification from the challenge.

The following simple feature extractor in the starting kit simply makes $s_t = (o_t, a_t)$.

```python
# %load submissions/starting_kit/ts_feature_extractor.py
class FeatureExtractor():
    def __init__(self, restart_name):
        """
        Parameters
        ----------
        restart_name : str
            The name of the 0/1 column indicating restarts in the time series.
        """
        self.restart_name = restart_name

    def transform(self, X_ds):
        """Transform time series into list of states.
        We simply use the observables at time t as the state.
        Be careful not to use any information from the future (X_ds[t + 1:])
        when constructing X_df[t].
        Parameters
        ----------
        X_ds : xarray.Dataset
            The raw time series.
        Return
        ------
        X_df : pandas Dataframe
            The list of states.
        """
        X_df = X_ds.to_dataframe()
        # Since we do not use the restart information in our regressor, we
        # remove it
        restart = X_df[self.restart_name].values
        X_df = X_df.drop(columns=self.restart_name)
        return X_df
```

### Generative regressor

The generative regressor follows a classical scikit-learn regressor template. It should be saved in the file [`submissions/starting_kit/generative_regressor.py`](submissions/starting_kit/generative_regressor.py).

The `__init__` function receives two parameters that you can use in `fit` and `predict`. `max_dists` is the maximum number of components in the mixture distribution, and `target_dim` is the index of the dimension being predicted ($j$ in the introduction, $j \in [0,5]$).

The `fit` function receives the input array `X_array` and the ground truth target numpy array `y_array`. The number of rows is the number of time steps. The columns of `X_array` are the features you extracted in the feature extractor followed by the `target_dim` (= $j-1$) system observables $\big(o^1_{t+1}, \ldots, o^{j-1}_{t+1}\big)$. The number of columns in `y_array` is one representing the target $o^j_{t+1}$.

The particular regressor learns a classical linear regressor and saves the standard deviation of the residuals.

The `predict` function receives the input test array `X_array`, the format is the same as in `fit`. It then constructs the one-dimensional density for its target $o^j_{t+1}$ ($j=$ `target_dim` $+ 1$). As in the feature extractor, be careful not to use any information from the future (`X_array[t + 1:]`) when constructing the output.

The particular regressor constructs a single Gaussian (so $w_1 = 1$), centered by the output of the linear regressor, with a fixed sigma (the standard deviation of the residuals in the training set). The type (code) of the Gaussian is 0.

We have other component types implemented in [rampwf.utils.distributions_dict](https://github.com/paris-saclay-cds/ramp-workflow/blob/generative_regression_clean/rampwf/utils/generative_regression.py): uniform (1) and beta (2). If you would like to use other component types, either let us know which one, or (better) implement it and make a pull request into the [generative_regression_clean](https://github.com/paris-saclay-cds/ramp-workflow/tree/generative_regression_clean) branch on ramp-workflow. After checks we will merge the PR, install it on the server and make an announcement in Slack to make it available to everybody. You can then submit your solution with the new kernel types.

```python
# %load submissions/starting_kit/generative_regressor.py
from sklearn.base import BaseEstimator
import numpy as np
from sklearn.linear_model import LinearRegression

class GenerativeRegressor(BaseEstimator):
    def __init__(self, max_dists, target_dim):
        """
        Parameters
        ----------
        max_dists : int
            The maximum number of distributions (kernels) in the mixture.
        target_dim : int
            The index of the target column to be predicted [0..d-1]
        """
        pass

    def fit(self, X_array, y_array):
        """Linear regression + residual sigma.

        Parameters
        ----------
        X_array : numpy array
            The input array. The features extracted by the feature extractor,
            plus `target_dim` system observables from time step t+1.
        y_array : numpy array
            The ground truth array (system observable j=`target_dim` + 1
            at time step t+1: o^j_{t+1}).
        """
        self.reg = LinearRegression()
        self.reg.fit(X_array, y_array)
        y_pred = self.reg.predict(X_array)
        y_pred = np.array([y_pred]).reshape(-1, 1)
        residuals = y_array - y_pred
        # Estimate a single sigma from residual variance
        self.sigma = np.sqrt(
            (1 / (X_array.shape[0] - 1)) * np.sum(residuals ** 2))

    def predict(self, X_array):
        """Construct a conditional mixture distribution.

        Be careful not to use any information from the future
        (X_array[t + 1:]) when constructing the output.

        Parameters
        ----------
        X_array : numpy array
            The input array. The features extracted by the feature extractor,
            plus `target_dim` system observables from time step t+1.

        Return
        ------
        weights : np.array of float
            discrete probabilities of each component of the mixture
        types : np.array of int
            integer codes referring to component types
            see rampwf.utils.distributions_dict
        params : np.array of float tuples
            parameters for each component in the mixture
        """
        types = np.array([[0], ] * len(X_array))

        # Normal
        y_pred = self.reg.predict(X_array)
        sigmas = np.array([self.sigma] * len(X_array))
        sigmas = sigmas[:, np.newaxis]
        params = np.concatenate((y_pred, sigmas), axis=1)
        weights = np.array([[1.0], ] * len(X_array))
        return weights, types, params
```

## Training and scoring

You can train and test the starting kit in a terminal or here in the notebook using the `ramp-test` script. By default it tests the starting kit, if you would like to test another submission, you can specify it with `--submission <name>` where `<name>` is what you named the directory in [`submissions/`](submissions).

We are using as many cross validation folds as episodes in the training file (number of restarts + 1). With $N$ episodes, we train on $N-1$ in each fold and test on the remaining one. The number of folds in the training set is 7. The training script prints the episode bounds and the fold indices, then trains, validates (on the validation fold), and tests (on the test set loaded by `problem.get_test_data{}`). Finally the mean and bagged scores are displayed.

On the server, the public leaderboard will display your bagged validation score during the competition. Your final ranking will be determined by the bagged test score. The data is  different in this public starting kit and on the server. We use different physics to make reverse engineering harder.

```python
!ramp-test
```

    Testing Acrobot system identification
    Reading train and test files from ./data ...
    Reading cv ...
    episode bounds: [0, 333, 1303, 1678, 2093, 2532, 3288, 4999]
    CV fold 0: train 333..4998,  valid 0..332,
    CV fold 1: train 0..332, 1303..4998,  valid 333..1302,
    CV fold 2: train 0..1302, 1678..4998,  valid 1303..1677,
    CV fold 3: train 0..1677, 2093..4998,  valid 1678..2092,
    CV fold 4: train 0..2092, 2532..4998,  valid 2093..2531,
    CV fold 5: train 0..2531, 3288..4998,  valid 2532..3287,
    CV fold 6: train 0..3287,  valid 3288..4998,
    Training submissions/starting_kit ...
    CV fold 0
    	score  likelihood_ratio      time
    	train              4.59  0.075871
    	valid              3.96  0.069595
    	test               3.86  0.070474
    CV fold 1
    	score  likelihood_ratio      time
    	train              4.16  0.030928
    	valid              3.70  0.064255
    	test               3.61  0.067980
    CV fold 2
    	score  likelihood_ratio      time
    	train              4.22  0.029149
    	valid              2.58  0.064103
    	test               3.42  0.066786
    CV fold 3
    	score  likelihood_ratio      time
    	train              4.17  0.029267
    	valid              3.03  0.062825
    	test               3.40  0.064849
    CV fold 4
    	score  likelihood_ratio      time
    	train              4.22  0.028860
    	valid              2.72  0.062818
    	test               3.43  0.066875
    CV fold 5
    	score  likelihood_ratio      time
    	train              4.10  0.047163
    	valid              3.29  0.064688
    	test               3.31  0.067226
    CV fold 6
    	score  likelihood_ratio      time
    	train              4.21  0.027983
    	valid              2.75  0.063567
    	test               3.77  0.061401
    ----------------------------
    Mean CV scores
    ----------------------------
    	score likelihood_ratio        time
    	train     4.24 ± 0.148  0.0 ± 0.02
    	valid     3.15 ± 0.488   0.1 ± 0.0
    	test      3.54 ± 0.193   0.1 ± 0.0
    ----------------------------
    Bagged scores
    ----------------------------
    	score  likelihood_ratio
    	valid              3.73
    	test               3.58

### What does `ramp test` do? Step-by-step.

First, let's load the fold indices of the first fold.

```python
get_cv = problem.get_cv
train_is, valid_is = list(get_cv(X_ds, y_array))[0]
```

    episode bounds: [0, 333, 1303, 1678, 2093, 2532, 3288, 4999]
    CV fold 0: train 333..4998,  valid 0..332,
    CV fold 1: train 0..332, 1303..4998,  valid 333..1302,
    CV fold 2: train 0..1302, 1678..4998,  valid 1303..1677,
    CV fold 3: train 0..1677, 2093..4998,  valid 1678..2092,
    CV fold 4: train 0..2092, 2532..4998,  valid 2093..2531,
    CV fold 5: train 0..2531, 3288..4998,  valid 2532..3287,
    CV fold 6: train 0..3287,  valid 3288..4998,

Now load the workflow and train the model.

```python
workflow = problem.workflow
fe, reg = workflow.train_submission(
    'submissions/starting_kit', X_ds, y_array, train_is)
```

Now test the trained model, first on the full training data then on the test data.

```python
y_preds = workflow.test_submission((fe, reg), X_ds)
X_test_ds, y_test_array = problem.get_test_data()
y_test_preds = workflow.test_submission((fe, reg), X_test_ds)
```

Finally, load the score type (metrics) and compute the training, validation, and test scores.

```python
score_type = problem.score_types[0]
train_score = score_type(y_array[train_is], y_preds[train_is])
valid_score = score_type(y_array[valid_is], y_preds[valid_is])
test_score = score_type(y_test_array, y_test_preds)
print(train_score, valid_score, test_score)
```

    4.588134679241945 3.963022781790907 3.8630718396285877

## Submitting to the hackathon server

Once you found a good feature extractor and classifier, you can submit them to the hackathon server. First, if it is your first time using RAMP, sign up, otherwise log in. The system sends you a mail where you can validate your signup, then the site admins will validate it. Once this is done, click on the "Huawei Hackathon" event under the "Acrobot system identification" problem, and sign up.

Once you are signed up, you can go to your sandbox (menu at the left) and copy-paste  [`ts_feature_extractor.py`](/submissions/starting_kit/ts_feature_extractor.py) and [`generative_regressor.py`](/submissions/starting_kit/generative_regressor.py) from `submissions/starting_kit`. Save it, rename it, then submit it. The submission is trained and tested on our backend in the same way as `ramp-test` does it locally. While your submission is waiting in the queue and being trained, you can find it in the "New submissions (pending training)" table in my submissions. Once it is trained, your submission shows up on the leaderboard.

If there is an error, it will show up in the "Failed submissions" table in my submissions. You can click on the error to see the trace.

After submission, do not forget to give credits to the previous submissions you reused or integrated into your submission. This will be especially interesting in the collaborative phase (after the competition is over) when you can look at and reuse other solutions.

The data set we use at the backend is different from what you find in the starting kit, so the score may be different.

# Other models in the starting kit

You can also keep several other submissions in your work directory [`submissions`](/submissions). [`generative_rf`](/submissions/generative_rf) uses a feature extractor that concatenates to $o_t$ the mean of the observables of the last time steps, handling restarts intelligently (not using the observables of the previous episode).

```python
# %load submissions/generative_rf/ts_feature_extractor.py
import pandas as pd
import numpy as np

class FeatureExtractor():
    def __init__(self, restart_name):
        """
        Parameters
        ----------
        restart_name : str
            The name of the 0/1 column indicating restarts in the time series.
        """
        self.restart_name = restart_name
        pass

    def transform(self, X_df_raw):
        """Transform time series into list of states.
        We use the observables at time t as the state, concatenated to the mean
        of the last ten time steps, handling restarts.

        Be careful not to use any information from the future (X_ds[t + 1:])
        when constructing X_df[t].
        Parameters
        ----------
        X_ds : xarray.Dataset
            The raw time series.
        Return
        ------
        X_df : pandas Dataframe

        """
        X_df = X_df_raw.to_dataframe()

        restart = X_df[self.restart_name].values

        # Since we do not use the restart information in our regressor, we have
        # to remove it
        X_df = X_df.drop(columns=self.restart_name)

        tail = 10
        result_array = []
        curr_tail = tail
        for i in range(len(X_df)):
            if restart[i] == 1:
                # If we encounter a restart, tail is set to 0
                curr_tail = 0
            elif curr_tail < tail:
                # And it goes back up to it's normal length if no other restarts
                curr_tail += 1

            X_temp = X_df.iloc[
                    [idx for idx in range(i - curr_tail, i + 1) if idx >= 0]
                                ].mean(axis=0)
            result_array.append(X_temp)
        result_array = np.vstack(result_array)
        additional_dim = pd.DataFrame(result_array)

        new_names = [item + "_engineered" for item in list(X_df_raw.keys())]
        additional_dim.rename(
            columns={i: item for i, item in enumerate(new_names)}, inplace=True)

        date = X_df.index.copy()
        X_df.reset_index(drop=True, inplace=True)
        X_array = pd.concat([X_df, additional_dim], axis=1)

        X_array.set_index(date, inplace=True)

        # We return a dataframe with additional features (with no clashing names)
        # based on previous values' mean, being mindful about restarts

        return X_array
```

The generative regressor learns 99 bagged trees (a "manual" random forest), and puts Gaussians centered in the tree predictions, sigmas estimated in different ways.

```python
# %load submissions/generative_rf/generative_regressor.py
from sklearn.base import BaseEstimator
from sklearn.ensemble import BaggingRegressor
from sklearn.tree import DecisionTreeRegressor
import numpy as np
from rampwf.hyperopt import Hyperparameter

# RAMP START HYPERPARAMETERS
#sigma_multiplier = Hyperparameter(
#    dtype='float', default=0.2, values=[
#            0.03, 0.05, 0.07, 0.1, 0.15, 0.2, 0.3])
#n_estimators = Hyperparameter(
#    dtype='int', default=200, values=[30, 50, 100, 200, 300])
#max_leaf_nodes = Hyperparameter(
#    dtype='int', default=5000, values=[1000, 2000, 5000, 10000])
# RAMP END HYPERPARAMETERS

method_proba = 'estimate_sigma'
EPSILON =1e-8

class GenerativeRegressor(BaseEstimator):
    def __init__(self, max_dists, model_index):
        self.max_dists = min(100, max_dists)
        self.clf = BaggingRegressor(
            base_estimator=DecisionTreeRegressor(
                max_leaf_nodes=5000),
            n_estimators=self.max_dists-1, # The last dist is uniform
            max_samples=0.2,
        )
        self.sigma = None
        self.a = None
        self.b = None

    def fit(self, X, y):
        self.clf.fit(X, y.ravel())
        yGuess = self.clf.predict(X)
        yGuess = np.array([yGuess]).reshape(-1, 1)
        error = y - yGuess
        self.sigma = np.sqrt((1 / X.shape[0]) * np.sum(error ** 2))
        self.a = np.min(y) - 10
        self.b = np.max(y) + 10

    def predict(self, X):

        # We give every distribution the same weight
        eps = 10 ** -10
        w = (1.0 - EPSILON)/(self.max_dists - 1)
        weights = np.stack([[w] * len(X)
                            for _ in range(self.max_dists)],
                           axis=1)
        weights[:, 0] = EPSILON
        # The first generative regressors are gaussian, the last is uniform
        types = np.zeros(self.max_dists)
        types[-1] = 1
        types = np.array([types] * len(X))

        # Gaussians
        mus = np.zeros((len(X), len(self.clf.estimators_),))
        for i, est in enumerate(self.clf.estimators_):
            mus[:, i] = est.predict(X)

        if method_proba == 'estimate_sigma':
            # Third method, uses the information from the trees to estimate
            # sigma and estimate gaussian noise around outpus
            sigma = mus.std(axis=1)
            sigma = np.clip(sigma, EPSILON, None, out=sigma)
            sigmas = np.stack([sigma for _ in range(len(self.clf.estimators_))],
                              axis=1)

        elif method_proba == 'standard':
            sigmas = np.stack([[self.sigma] * len(X)
                               for _ in range(len(self.clf.estimators_))],
                              axis=1)
        else:
            sigmas = np.nan

        #sigmas *= float(sigma_multiplier)
        sigmas /= float(np.sqrt(self.max_dists - 1))

        # We put each mu next to its sigma
        params_normal = np.empty((len(X), len(self.clf.estimators_)*2))
        params_normal[:, 0::2] = mus
        params_normal[:, 1::2] = sigmas

        # Uniform
        a_array = np.array([self.a] * len(X))
        b_array = np.array([self.b] * len(X))
        params_uniform = np.stack((a_array, b_array), axis=1)

        # We concatenate the params
        params = np.concatenate((params_normal, params_uniform), axis=1)
        return weights, types, params

```

You can also uncomment the hyperparameter section at the top and `sigmas *= float(sigma_multiplier)` further down, then try the experimental `ramp-hyperopt` command, see what happens.

```python
!ramp-hyperopt --submission generative_rf --n-iter 21
```

    episode bounds: [0, 333, 1303, 1678, 2093, 2532, 3288, 4999]
    CV fold 0: train 333..4998,  valid 0..332,
    CV fold 1: train 0..332, 1303..4998,  valid 333..1302,
    CV fold 2: train 0..1302, 1678..4998,  valid 1303..1677,
    CV fold 3: train 0..1677, 2093..4998,  valid 1678..2092,
    CV fold 4: train 0..2092, 2532..4998,  valid 2093..2531,
    CV fold 5: train 0..2531, 3288..4998,  valid 2532..3287,
    CV fold 6: train 0..3287,  valid 3288..4998,
                      n_folds  train_likelihood_ratio_m  ...   n_train_s   n_valid_s
    sigma_multiplier                                     ...
    1.3                     7                 20.069907  ...  497.730994  497.730994
    1.5                     7                 18.347998  ...  497.730994  497.730994
    1.7                     7                 16.972466  ...  497.730994  497.730994

    [3 rows x 13 columns]
    Best hyperparameters:  1.7

## More information

You can find more information in the [ramp-workflow documentation](https://paris-saclay-cds.github.io/ramp-workflow).

## Contact

Use Slack to contact us.

```python

```


# Problema de selección de portafolio con preferencias media-varianza

En la clase pasada hablamos acerca de:
- preferencias,
- funciones de utilidad,
- la actitud de los inversionistas de cara al riesgo,
- la aversión al riesgo, entre otros.

Todas ellas son piezas que necesitamos para responder la pregunta de ¿cómo un inversionista toma la decisión óptima de selección de portafolio?

En esta clase al fin estamos listos para ensamblar estos conceptos y escribir el problema de selección de portafolios.

En el camino aprenderemos acerca del concepto de **utilidad esperada**, que nos permite trabajar con incertidumbre en el modelado económico (una de las ideas más importantes en economía). Esta idea tiene más de 60 años, y básicamente dice que los individuos, cuando están de cara a incertidumbre, maximizan el valor esperado de su utilidad (solo cierto si somos homo economicus).

Además del concepto de utilidad esperada, aprenderemos acerca de **preferencias media-varianza**. Es decir, supondremos que los inversionistas toman decisiones basados en un tipo particular de preferencias.

Con lo anterior, estableceremos el problema de selección de portafolios.

**Objetivos:**

- ¿Qué es utilidad esperada?
- ¿Qué son las preferencias media-varianza?
- Funciones de utilidad media-varianza.
- Enunciado y solución del problema básico de selección de portafolio.

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Utilidad esperada
- Básicamente combina las probabilidades de los resultados con cómo los inversionistas se sienten con dichos resultados.
- En otras palabras, la utilidad esperada multiplica la probabilidad de suceso de un evento con la utilidad que genera dicho evento.

Recordemos que las *funciones de utilidad* permiten a los inversionistas expresar cómo se sienten con los resultados, especialmente en los malos ratos.

Entonces la *utilidad esperada* es una herramienta que nos permite cuantificar cómo nos sentimos en nuestros malos momentos económicos, capturando el riesgo con la probabilidad de ocurrencia de dichos malos momentos.

Dado este marco de trabajo, cualquier decisión se puede escribir como la maximización de la utilidad esperada:
\begin{align}
\max_{\theta} & \quad E[U(W)], \\
\end{align}
mediante la escogencia de cierta variable $\theta$ (gastos, planes de ahorro, compra de activos, planes de producción, etc.).

Para nuestros propósitos, la variable de decisión serán los pesos o ponderaciones del portafolio.

Adicionalmente, en el contexto de la decisión de distribución de la riqueza entre activos, el problema de maximización tendrá comúnmente las siguientes restricciones:
- universo de inversión,
- posición en los activos dados.

**Ejemplo.**

Supongamos que un inversionista debe determinar la composición óptima de su portafolio, que contiene activos y bonos. Supongamos que son los únicos instrumentos disponibles.

Sean:
- $w_s$: peso o ponderación de activos en el portafolio,
- $w_b$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

De manera que podemos escribir el problema de selección de portafolios como la maximización de la utilidad esperade de nuestra riqueza futura, la cual dependerá de nuestros rendimientos:

\begin{align}
\max_{w_s,w_b} &\quad E[U(W)]\\
\text{s. a.}   &\quad W=W_0(1+w_sr_s+w_br_b)\\
               &\quad w_s+w_b=1\\
               &\quad w_s,w_b\geq0
\end{align}

Preguntas:
- ¿Qué significan las restricciones?
- Ya que tenemos planteado este problema básico, ¿qué haría falta para empezar a resolverlo?
___

## 2. Preferencias media-varianza

### 2.1. Utilidad media-varianza

Entonces, ¿qué funciones de utilidad deberíamos de usar en este problema de selección de portafolios?

- La respuesta es: **preferencias media-varianza**.
- Éstas serán representadas en términos de funciones de utilidad como: **utilidad media-varianza**.

Usamos la *utilidad media-varianza* en el problema de selección de portafolios dado que ésta decribe el "trade-off" entre riesgo y rendimiento que enfrentan los inversionistas. La *utilidad media-varianza* está dada por la siguiente expresión:

$$U=E[r_p]-\frac{1}{2}\gamma\sigma_p^2,$$

donde
- $E[r_p]$ es el rendimiento esperado del portafolio,
- $\sigma_p^2$ es la varianza del portafolio, y
- $\gamma$ es el coeficiente de aversión al riesgo.

#### Intuición acerca de la función de utilidad media-varianza:
- Sólo se preocupa por medias :) y varianzas :(.
- Incrementa con: rendimiento esperado del portafolio.
- Decrece con: varianza del portafolio.
- Malos tiempos: rendimientos son bajos y las volatilidades son altas.
- Conecta bastante bien con la teoría moderna de portafolios, la cual caracteriza los rendimientos con medias y varianzas únicamente.
- Criticada por su limitación: supone que los inversionistas sólo se preocupan por medias y varianzas.

### 2.2. Curvas de indiferencia

*¿Recuerdan las curvas de nivel que se ven en cálculo de varias variables?*
- Bien, acá nos servirán para representar la utilidad media-varianza gráficamente.
- En el contexto de utilidad media-varianza, las curvas de nivel se llaman **curvas de indiferencia**.

Dados ciertos niveles de utilidad $U_1>U_2>U_3$, las curvas de indiferencia relativas a estos niveles de utilidad, son los lugares geométricos en el espacio de rendimiento esperado vs. volatilidad representados por las siguientes expresiones

$$U_1=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_1,$$

$$U_2=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_2,$$

$$U_3=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_3.$$

**Gráficamente**

```python
# Importar numpy y pyplot

```

```python
# Coeficiente de aversión al riesgo (entre 1 y 10 comúnmente)

# Niveles de utilidad

# Vector de volatilidades (sugerido 0%-60%)

# Curvas de indiferencia

```

```python
# Gráfica

```

Bueno, ¿y porqué se llaman curvas de indiferencia?, ¿qué representa una cuva de indiferencia?

- Porque sobre una misma curva el nivel de utilidad es el mismo (es indiferente).
- Son todas las combinaciones de riesgo y rendimiento que producen un mismo nivel de utilidad.

Volviendo al problema de selección de portafolios, queremos la utilidad más alta.
- ¿Cuál de las anteriores curvas de indiferencia corresponde a la utilidad más alta?
- Intuitivamente, ¿porqué?
- Curvas de indiferencia para niveles de utilidad más altos, estarán...

Notamos además que las anteriores curvas de indiferencia son *paralelas* una con otra. Claro, las dibujamos con el mismo coeficiente de aversión al riesgo.

$$\frac{d E[r_p]}{d\sigma_p}=\frac{d }{d\sigma_p}\left[\frac{1}{2}\gamma\sigma_p^2+U\right]=\gamma\sigma_p.$$

¿Cómo cambian estas curvas para coeficientes de aversión al riesgo más altos?

```python
# Coeficientes de aversión al riesgo (entre 1 y 10 comúnmente)

# Nivel de utilidad

# Vector de volatilidades (sugerido 0%-60%)

# Curvas de indiferencia

```

```python
# Gráfica

```

¿Cómo interpretamos las anteriores gráficas?, ¿qué pasa con las personas más aversas al riesgo?
- Se puede ver de dos maneras: para un mismo nivel de rendimiento esperado, una persona más aversa al riesgo soporta un nivel menor de riesgo; equivalentemente, para un mismo nivel de riesgo, una persona más aversa al riesgo requerirá un nivel de rendimiento esperado más alto.

Con todo lo anterior, el problema de selección de portafolios se puede plantear como *encontrar la curva de indeferencia más alta dado el conjunto de oportunidades de inversión y restricciones*.

## 3. Problema de selección de portafolios: una ilustración

Ahora ilustraremos el problema de selección de portafolios con algunos datos.
- Por ahora solo queremos ilustrar gráficamente cómo se resuelve este problema. Trabajar en la intuición.
- En las siguientes dos clases nos enfocaremos en cómo resolverlo analíticamente.

Acá tenemos el rendimiento medio anual y la volatilidad para dos instrumentos usando datos de EU: instrumentos de deuda (bonos) y acciones. Supondremos que el inversionista solo puede invertir en estas dos clases de instrumentos.

```python
# Importamos pandas
import pandas as pd
```

```python
# Datos
data = pd.DataFrame(index=['Stocks','Bonds', 'CorrSB'], columns=['Mean', 'Std'])
data['Mean'] = [0.119, 0.0591, 0.113]
data['Std'] = [0.1915, 0.0833, None]
data
```

Entonces, ¿cuál es la distribución de riqueza óptima?, o más bien, ¿cuál es la composición óptima del portafolio para un inversionista dado su nivel de aversión al riesgo?

**Primero.** Recordamos que, para dos activos, podemos trazar la frontera de mínima varianza tomando todas las posibles combinaciones de los dos activos.

De nuevo, sean:
- $w_s=w$: peso o ponderación de activos en el portafolio,
- $w_b=1-w$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

Entonces

$$E[r_p]=wE[r_{s}]+(1-w)E[r_b]$$

$$\sigma_p^2=w^2\sigma_{s}^2+(1-w)^2\sigma_b^2+2w(1-w)\rho_{s,b}\sigma_s\sigma_b$$

```python
# Vector de w variando entre 0 y 1 con n pasos

# Rendimientos esperados individuales

# Volatilidades individuales

# Correlacion

```

```python
# Crear un DataFrame cuyas columnas sean rendimiento
# y volatilidad del portafolio para cada una de las w
# generadas

```

```python
# Gráfica

```

**Segundo.** Graficamos en la misma ventana, curvas de indiferencia.

```python
# Niveles de utilidad

# Coeficiente de aversión al riesgo

# Curvas de indiferencia

```

```python
# Gráfica

```

**Tercero.** La elección óptima está dada por la curva de indiferencia para el nivel de utilidad más alto que es tangente a la frontera media-varianza.
- Claramente, esta selección depende del coeficiente de aversión al riesgo.

```python
# Gráfica con zoom

```

# Anuncios parroquiales

## 1. Quiz la siguiente clase.

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez.
</footer>


## Subsurface Data Analytics

### Kriging vs. Simulation for Spatial Predictions in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

Let's first cover spatial estimation and simulation.

#### Spatial Estimation

Consider the case of making an estimate at some unsampled location, $𝑧(\bf{u}_0)$, where $z$ is the property of interest (e.g. porosity etc.) and $𝐮_0$ is a location vector describing the unsampled location.

How would you do this given data, $𝑧(\bf{𝐮}_1)$, $𝑧(\bf{𝐮}_2)$, and $𝑧(\bf{𝐮}_3)$?

It would be natural to use a set of linear weights to formulate the estimator given the available data.

\begin{equation}
z^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} z(\bf{u}_{\alpha})
\end{equation}

We could add an unbiasedness constraint to impose the sum of the weights equal to one.  What we will do is assign the remainder of the weight (one minus the sum of weights) to the global average; therefore, if we have no informative data we will estimate with the global average of the property of interest.

\begin{equation}
z^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} z(\bf{u}_{\alpha}) + \left(1-\sum^{n}_{\alpha = 1} \lambda_{\alpha} \right) \overline{z}
\end{equation}

We will make a stationarity assumption, so let's assume that we are working with residuals, $y$.

\begin{equation}
y^{*}(\bf{u}) = z^{*}(\bf{u}) - \overline{z}(\bf{u})
\end{equation}

If we substitute this form into our estimator the estimator simplifies, since the mean of the residual is zero.

\begin{equation}
y^{*}(\bf{u}) = \sum^{n}_{\alpha = 1} \lambda_{\alpha} y(\bf{u}_{\alpha})
\end{equation}

while satisfying the unbaisedness constraint.

#### Kriging

Now the next question is what weights should we use?

We could use equal weighting, $\lambda = \frac{1}{n}$, and the estimator would be the average of the local data applied for the spatial estimate. This would not be very informative.

We could assign weights considering the spatial context of the data and the estimate:

* **spatial continuity** as quantified by the variogram (and covariance function)
* **redundancy** the degree of spatial continuity between all of the available data with themselves
* **closeness** the degree of spatial continuity between the avaiable data and the estimation location

The kriging approach accomplishes this, calculating the best linear unbiased weights for the local data to estimate at the unknown location.  The derivation of the kriging system and the resulting linear set of equations is available in the lecture notes.  Furthermore kriging provides a measure of the accuracy of the estimate!  This is the kriging estimation variance (sometimes just called the kriging variance).

\begin{equation}
\sigma^{2}_{E}(\bf{u}) = C(0) - \sum^{n}_{\alpha = 1} \lambda_{\alpha} C(\bf{u}_0 - \bf{u}_{\alpha})
\end{equation}

What is 'best' about this estimate? Kriging estimates are best in that they minimize the above estimation variance.

#### Properties of Kriging

Here are some important properties of kriging:

* **Exact interpolator** - kriging estimates with the data values at the data locations
* **Kriging variance** can be calculated before getting the sample information, as the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.
* **Spatial context** - kriging takes into account, furthermore to the statements on spatial continuity, closeness and redundancy we can state that kriging accounts for the configuration of the data and structural continuity of the variable being estimated.
* **Scale** - kriging may be generalized to account for the support volume of the data and estimate. We will cover this later.
* **Multivariate** - kriging may be generalized to account for multiple secondary data in the spatial estimate with the cokriging system. We will cover this later.
* **Smoothing effect** of kriging can be forecast. We will use this to build stochastic simulations later.

I have more on this topic at [Krigign YouTube Lecture](https://youtu.be/CVkmuwF8cJ8).

#### Sequential Gaussian Simulation

With sequential Gaussian simulation we build on kriging by:

* adding a random residual with the missing variance

* sequentially adding the simulated values as data to correct the covariance between the simulated values

The resulting model corrects the issues of kriging, as we now:

* reproduce the global feature PDF / CDF

* reproduce the global variogram

* while providing a model of uncertainty through multiple realizations

I have more on this topic at [Simulation YouTube Lecture](https://www.youtube.com/watch?v=3cLqK3lR56Y&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=45&t=813s).

This is a tutorial for / comparing spatial estimation and simulation we use **Simple Kriging** and **Sequential Gaussian Simulation in Python with the GeostatsPy package (Pyrcz et al., 2021), GSLIB's KB2D and SGSIM programs translated to Python from the original FORTRAN GSLIB: Geostatistical Library methods** (Deutsch and Journel, 1997).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_MV_biased.csv available at https://git.io/fhgu0.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the Required Libraries

The following code loads the required libraries.

```python
import os                                                 # to set current working directory
import numpy as np                                        # arrays and matrix math
import pandas as pd                                       # DataFrames
import matplotlib.pyplot as plt                           # plotting
cmap = plt.cm.inferno                                     # color map
import geostatspy.geostats as geostats
import geostatspy.GSLIB as GSLIB
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Declare Functions

Here's a convenience function for plotting variograms.

```python
def vargplot(feature,lags,gamma_maj,gamma_min,npps_maj,npps_min,vmodel,azi,atol,sill):     # plot the variogram
    index_maj,lags_maj,gmod_maj,cov_maj,ro_maj = geostats.vmodel(nlag=100,xlag=10,azm=azi,vario=vmodel);
    index_min,lags_min,gmod_min,cov_min,ro_min = geostats.vmodel(nlag=100,xlag=10,azm=azi+90.0,vario=vmodel);

    plt.scatter(lags,gamma_maj,color = 'black',s = npps_maj*0.03,label = 'Major Azimuth ' +str(azi), alpha = 0.8)
    plt.plot(lags_maj,gmod_maj,color = 'black')
    plt.scatter(lags,gamma_min,color = 'red',s = npps_min*0.03,label = 'Minor Azimuth ' +str(azi+90.0), alpha = 0.8)
    plt.plot(lags_min,gmod_min,color = 'red')
    plt.plot([0,2000],[sill,sill],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    if atol < 90.0:
        plt.title('Directional ' + feature + ' Variogram')
    else:
        plt.title('Omni Directional NSCORE ' + feature + ' Variogram')
    plt.xlim([0,1000]); #plt.ylim([0,1.8])
    plt.legend(loc="lower right")
    plt.grid(True)
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see above) GSLIB executables in this directory or a location identified in the environmental variable *Path*.

```python
#os.chdir("c:/PGE383")                                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object. We will also extra a limited sample so that the spatial samples are not too dense.  This way we can observe more of the heterogeneity from the simulation with the spatial continuity model, rather than mostly data driven heterogeneity.

```python
#df = pd.read_csv("sample_data_MV_biased.csv")             # read a .csv file in as a DataFrame
#df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data_MV_biased.csv") # from Dr. Pyrcz's GitHub repo
df = pd.read_csv(r"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/spatial_nonlinear_MV_facies_v1.csv")
df = df.rename(columns = {'Por':'Porosity'})            # rename feature(s)
df = df.loc[:,['X','Y','Porosity']]; #df['Porosity'] = df['Porosity']*100.0
df.describe()                                             # summary statistics
#df = df.sample(50)                                        # extract 50 samples
#df = df.reset_index()                                     # reset the record index
#df.head()
```

      X
      Y
      Porosity

      count
      457.000000
      457.000000
      457.000000

      mean
      544.215378
      524.292267
      13.017708

      std
      295.136181
      282.502247
      4.695742

      min
      1.219681
      10.006391
      2.666539

      25%
      301.455681
      296.648407
      9.453725

      50%
      580.575433
      535.737818
      13.250704

      75%
      795.864122
      776.153691
      16.090334

      max
      1006.952489
      1000.088794
      25.653330

#### Set Limits for Plotting

This is applied for data and model visualization.

```python
pormin = 0.0; pormax = 22.0; pormean = 12.7; porvar = 22.0
xmin = 0.0; xmax = 1000.0
ymin = 0.0; ymax = 1000.0
tmin = -9999.9; tmax = 9999.9
nx = 100; xmn = 5.0; xsiz = 10.0
ny = 100; ymn = 5.0; ysiz = 10.0
```

#### Data Analytics and Visualization

Let's take a look at the available data:

* location map
* histogram
* variogram

```python
plt.subplot(131)
GSLIB.locmap_st(df,'X','Y','Porosity',0,1000,0,1000,0,25,'Porosity Location Map','X (m)','Y (m)','Porosity',cmap=cmap)

plt.subplot(132)
plt.hist(df['Porosity'].values,bins=np.linspace(pormin,pormax,50),color='red',alpha=0.2,edgecolor='black')
plt.xlabel('Porosity (%)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')

df['Npor'], tvPor, tnsPor = geostats.nscore(df,'Porosity')
lags, gamma_maj, npps_maj = geostats.gamv(df,"X","Y",'Porosity',tmin,tmax,xlag=20,xltol=20,nlag=100,azm=0.0,atol=22.5,bandwh=9999.9,isill=0);
lags, gamma_min, npps_min = geostats.gamv(df,"X","Y",'Porosity',tmin,tmax,xlag=20,xltol=20,nlag=100,azm=90.0,atol=22.5,bandwh=9999.9,isill=0);

nug = 0.0; nst = 2                                             # 2 nest structure variogram model parameters
it1 = 2; cc1 = 19.0; azi1 = 0; hmaj1 = 150; hmin1 = 150
it2 = 2; cc2 = 3.0; azi2 = 0; hmaj2 = 1000; hmin2 = 150

vmodel = GSLIB.make_variogram(nug,nst,it1,cc1,azi1,hmaj1,hmin1,it2,cc2,azi2,hmaj2,hmin2); # make model object
vmodel_sim = GSLIB.make_variogram(nug,nst,it1,cc1/(cc1+cc2),azi1,hmaj1,hmin1,it2,cc2/(cc1+cc2),azi2,hmaj2,hmin2); # make model object

plt.subplot(133)
vargplot('Porosity',lags,gamma_maj,gamma_min,npps_maj,npps_min,vmodel,azi=0.0,atol=22.5,sill=porvar)     # plot the variogram

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.1, wspace=0.2, hspace=0.2)
plt.show()
```

#### Simple Kriging

Let's make a simple kriging, estimation map and calculate the same statistics again to check the reproduction.

```python
por_kmap, por_vmap = geostats.kb2d(df,'X','Y','Porosity',tmin,tmax,nx,xmn,xsiz,ny,ymn,ysiz,nxdis=1,nydis=1,
         ndmin=0,ndmax=10,radius=500,ktype=0,skmean=pormean,vario=vmodel)

plt.subplot(131)                                          # plot the results
GSLIB.locpix_st(por_kmap,xmin,xmax,ymin,ymax,xsiz,pormin,pormax,df,'X','Y','Porosity','Simple Kriging Porosity','X(m)','Y(m)','Porosity (%)',cmap)

plt.subplot(132)
plt.hist(df['Porosity'].values,density=True,bins=np.linspace(pormin,pormax,50),color='red',alpha=0.2,edgecolor='black',label='Data')
plt.hist(por_kmap.flatten(),density=True,bins=np.linspace(pormin,pormax,50),color='blue',alpha=0.2,edgecolor='black',label='Kriging')
plt.xlabel('Porosity (%)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram'); plt.legend(loc='upper right')

lags, sk_gamma_maj, npps_maj = geostats.gam(por_kmap,tmin,tmax,xsiz,ysiz,ixd=1,iyd=-1,nlag=100,isill=0.0);
lags, sk_gamma_min, npps_min = geostats.gam(por_kmap,tmin,tmax,xsiz,ysiz,ixd=1,iyd=1,nlag=100,isill=0.0);

plt.subplot(133)
vargplot('Porosity',lags,sk_gamma_maj,sk_gamma_min,npps_maj,npps_min,vmodel,azi=0.0,atol=22.5,sill=porvar)     # plot the variogram

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.1, wspace=0.2, hspace=0.2)
plt.show()
```

#### Sequential Gaussian Simulation

Let's jump right to building a variety of models with simulation and visualizing the results.  We will start with a test, comparision of simulation with simple and ordinary kriging.

```python
# Simple Kriging vs. Ordinary Kriging

por_sim = geostats.sgsim(df,'X','Y','Porosity',wcol=-1,scol=-1,tmin=tmin,tmax=tmax,itrans=1,ismooth=0,dftrans=0,tcol=0,
            twtcol=0,zmin=pormin,zmax=pormax,ltail=1,ltpar=0.0,utail=1,utpar=0.3,nsim=1,
            nx=nx,xmn=xmn,xsiz=xsiz,ny=ny,ymn=ymn,ysiz=ysiz,seed=73073,
            ndmin=0,ndmax=20,nodmax=20,mults=0,nmult=2,noct=-1,radius=500,radius1=500,sang1=0,
            mxctx=41,mxcty=41,ktype=0,colocorr=0.0,sec_map=0,vario=vmodel_sim)

plt.subplot(131)                                          # plot the results
GSLIB.locpix_st(por_sim,xmin,xmax,ymin,ymax,xsiz,pormin,pormax,df,'X','Y','Porosity','Sequential Gaussian Simulation Porosity','X(m)','Y(m)','Porosity (%)',cmap)

plt.subplot(132)
plt.hist(df['Porosity'].values,density=True,bins=np.linspace(pormin,pormax,50),color='red',alpha=0.2,edgecolor='black',label='Data')
plt.hist(por_sim.flatten(),density=True,bins=np.linspace(pormin,pormax,50),color='yellow',alpha=0.2,edgecolor='black',label='Simulation')
plt.xlabel('Porosity (%)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram'); plt.legend(loc='upper right')

lags, sim_gamma_maj, npps_maj = geostats.gam(por_sim,tmin,tmax,xsiz,ysiz,ixd=1,iyd=-1,nlag=100,isill=0.0);
lags, sim_gamma_min, npps_min = geostats.gam(por_sim,tmin,tmax,xsiz,ysiz,ixd=1,iyd=1,nlag=100,isill=0.0);

plt.subplot(133)
vargplot('Porosity',lags,sim_gamma_maj,sim_gamma_min,npps_maj,npps_min,vmodel,azi=0.0,atol=22.5,sill=porvar)     # plot the variogram

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.1, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comments

This was a basic demonstration comparison of spatial estimation and simulation.

Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
%load_ext autoreload
%autoreload 2

%load_ext nb_black
```

    <IPython.core.display.Javascript object>

```python
#export
from exp.nb_06 import *
```

    <IPython.core.display.Javascript object>

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

```python
x_train, y_train, x_valid, y_valid = get_data()

x_train, x_valid = normalize_to(x_train, x_valid)
train_ds, valid_ds = Dataset(x_train, y_train), Dataset(x_valid, y_valid)

nh, bs = 50, 512
c = y_train.max().item() + 1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

    <IPython.core.display.Javascript object>

```python
mnist_view = view_tfm(1, 28, 28)
cbfs = [
    Recorder,
    partial(AvgStatsCallback, accuracy),
    partial(BatchTransformXCallback, mnist_view)
]
```

    <IPython.core.display.Javascript object>

```python
nfs = [8, 16, 32, 64, 64]
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [1.52496328125, tensor(0.4912)]
    valid: [0.29743173828125, tensor(0.9101)]
    train: [0.23856525390625, tensor(0.9270)]
    valid: [0.134212939453125, tensor(0.9579)]
    train: [0.13307724609375, tensor(0.9591)]
    valid: [0.11205748291015626, tensor(0.9638)]
    CPU times: user 32 s, sys: 2.77 s, total: 34.8 s
    Wall time: 17.7 s

    <IPython.core.display.Javascript object>

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch. In the case of computer vision, the average and std are by channel. Therefore, it is as if we concatenate all images with their heights and width by channel. Therefore, the output would be channels x (batches * height * width).

`self.register_buffer`:

`self.register_buffer('vars',  torch.ones(1,nf,1,1))` is almost the same as `self.vars = torch.ones(1,nf,1,1))` but has the advantage that if we move everything to GPU, they will be moved to GPU too; otherwise, the won't and we will get an error when we try to do some calculations because you can't do calculations on stuff that are on CPU and GPU. It also allows us to save them when we save the model to disk because we need them at inference time.

```python
class BatchNorm(nn.Module):
    __constants__ = ['eps', 'mom']

    def __init__(self, nf, mom=.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        # This means the beta in exponential weighted average is
        # actually 1 - mom. Therefore, beta is 0.9 here.
        self.eps = eps
        self.mom = mom
        self.gamma = nn.Parameter(torch.ones(nf, 1, 1))
        self.beta = nn.Parameter(torch.zeros(nf, 1, 1))
        self.register_buffer('means', torch.ones(1, nf, 1, 1))
        self.register_buffer('vars', torch.ones(1, nf, 1, 1))

    def update_stats(self, x):
        # Average across batches, height and width --> average across channels.
        mean = x.mean((0, 2, 3), keepdim=True)
        var = x.var((0, 2, 3), keepdim=True)

        # means = (1 - mom) x means + mom x m; exponentially weighted average
        self.means.lerp_(mean, self.mom)
        self.vars.lerp_(var, self.mom)
        return mean, var

    def forward(self, x):
        if self.training:
            mean, var = self.update_stats(x)
        else:
            mean, var = self.means, self.vars
        x = (x - mean) / (var + self.eps).sqrt()
        return self.gamma * x + self.beta
```

    <IPython.core.display.Javascript object>

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [
        nn.Conv2d(ni, nf, ks, padding=ks // 2, stride=stride, bias=not bn),
        GeneralRelu(**kwargs)
    ]
    if bn:
        layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None:
            m.bias.data.zero_()
    for l in m.children():
        init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs,
                  data,
                  lr,
                  layer,
                  cbs=None,
                  opt_func=None,
                  uniform=False,
                  **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

    <IPython.core.display.Javascript object>

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
learn, run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
learn.model
```

    Sequential(
      (0): Sequential(
        (0): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
        (1): GeneralRelu()
        (2): BatchNorm()
      )
      (1): Sequential(
        (0): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): BatchNorm()
      )
      (2): Sequential(
        (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): BatchNorm()
      )
      (3): Sequential(
        (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): BatchNorm()
      )
      (4): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): BatchNorm()
      )
      (5): AdaptiveAvgPool2d(output_size=1)
      (6): Lambda()
      (7): Linear(in_features=64, out_features=10, bias=True)
    )

    <IPython.core.display.Javascript object>

```python
with Hooks(learn.model, compute_stats, True) as hooks:
    run.fit(1, learn)
    fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(10, 4))
    for h in hooks[:-1]:
        ms, ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6))

    fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(10, 4))
    for h in hooks[:-1]:
        ms, ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```python
learn, run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [0.212331796875, tensor(0.9339)]
    valid: [0.08413507690429688, tensor(0.9751)]
    train: [0.0620726171875, tensor(0.9807)]
    valid: [0.06035379638671875, tensor(0.9833)]
    train: [0.04231357421875, tensor(0.9870)]
    valid: [0.06713604736328126, tensor(0.9784)]
    CPU times: user 57.5 s, sys: 4.73 s, total: 1min 2s
    Wall time: 31.3 s

    <IPython.core.display.Javascript object>

### Builtin batchnorm

```python
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [
        nn.Conv2d(ni, nf, ks, padding=ks // 2, stride=stride, bias=not bn),
        GeneralRelu(**kwargs)
    ]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [0.213002734375, tensor(0.9338)]
    valid: [0.09222581787109375, tensor(0.9714)]
    train: [0.060083720703125, tensor(0.9812)]
    valid: [0.06774615478515625, tensor(0.9796)]
    train: [0.04006894775390625, tensor(0.9873)]
    valid: [0.06115048828125, tensor(0.9799)]
    CPU times: user 47 s, sys: 2.93 s, total: 49.9 s
    Wall time: 25.1 s

    <IPython.core.display.Javascript object>

### With scheduler

Now let's add the usual warm-up/annealing.

```python
sched = combine_scheds([0.3, 0.7], [lin_sched(0.6, 2.), lin_sched(2., 0.1)])
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs,
                           data,
                           0.9,
                           conv_layer,
                           cbs=cbfs + [partial(ParamScheduler, 'lr', sched)])
```

    <IPython.core.display.Javascript object>

```python
run.fit(3, learn)
```

    /Users/imad/Documents/courses/fastai-courses/dl2/notebooks/exp/nb_05.py:81: UserWarning: This overload of nonzero is deprecated:
    	nonzero()
    Consider using one of the following signatures instead:
    	nonzero(*, bool as_tuple) (Triggered internally at  /Users/distiller/project/conda/conda-bld/pytorch_1595629430416/work/torch/csrc/utils/python_arg_parser.cpp:766.)
      idx = (pos >= pcts).nonzero().max()

    train: [0.26774140625, tensor(0.9175)]
    valid: [0.10132388916015625, tensor(0.9699)]
    train: [0.069213349609375, tensor(0.9783)]
    valid: [0.056937982177734374, tensor(0.9831)]
    train: [0.03427287109375, tensor(0.9895)]
    valid: [0.049336358642578126, tensor(0.9853)]

    <IPython.core.display.Javascript object>

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

The main challenges to **BatchNorm** are:
- How do we handle very small batches because we would have infinite variance or unstable training.
- How do we handle RNNs.

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

```python
class LayerNorm(nn.Module):
    __constants__ = ['eps']

    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.gamma = nn.Parameter(tensor(1.))
        self.beta = nn.Parameter(tensor(0.))

    def forward(self, x):
        mean = x.mean((1, 2, 3), keepdim=True)
        var = x.var((1, 2, 3), keepdim=True)
        x = (x - mean) / ((var + self.eps).sqrt())
        return x * self.gamma + self.beta
```

    <IPython.core.display.Javascript object>

```python
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [
        nn.Conv2d(ni, nf, ks, padding=ks // 2, stride=stride, bias=True),
        GeneralRelu(**kwargs)
    ]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [0.4233333984375, tensor(0.8652)]
    valid: [0.1118630859375, tensor(0.9647)]
    train: [0.085477177734375, tensor(0.9743)]
    valid: [0.08900013427734375, tensor(0.9751)]
    train: [0.0546854052734375, tensor(0.9825)]
    valid: [0.07122665405273437, tensor(0.9788)]
    CPU times: user 54.1 s, sys: 4.75 s, total: 58.8 s
    Wall time: 29.6 s

    <IPython.core.display.Javascript object>

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

It would be hard to distinguish between them because it forces the means and variances to be the same after normalizing. It throws away the difference in activations and if at inference time there is a difference in distribution that we care about, it would throw that away as well.

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

```python
class InstanceNorm(nn.Module):
    __constants__ = ['eps']

    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.gamma = nn.Parameter(torch.ones(nf, 1, 1))
        self.beta = nn.Parameter(torch.zeros(nf, 1, 1))

    def forward(self, x):
        mean = x.mean((2, 3), keepdim=True)
        var = x.var((2, 3), keepdim=True)
        res = (x - mean) / ((var + self.eps).sqrt())
        return res * self.gamma + self.beta
```

    <IPython.core.display.Javascript object>

```python
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [
        nn.Conv2d(ni, nf, ks, padding=ks // 2, stride=stride, bias=True),
        GeneralRelu(**kwargs)
    ]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986)]
    valid: [nan, tensor(0.0991)]
    train: [nan, tensor(0.0986)]
    valid: [nan, tensor(0.0991)]
    train: [nan, tensor(0.0986)]
    valid: [nan, tensor(0.0991)]
    CPU times: user 1min 8s, sys: 4.98 s, total: 1min 13s
    Wall time: 36.9 s

    <IPython.core.display.Javascript object>

*Question*: why can't this classify anything?

Because it throws the difference in the means and variances of each image and each channel. It is useful for style transfer.

### Group norm

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

    <IPython.core.display.Javascript object>

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [
        nn.Conv2d(ni, nf, ks, padding=ks // 2, stride=stride, bias=not bn),
        GeneralRelu(**kwargs)
    ]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(1, learn)
```

    train: [2.35215484375, tensor(0.1710)]
    valid: [1548761117.4912, tensor(0.1431)]
    CPU times: user 1min 51s, sys: 361 ms, total: 1min 51s
    Wall time: 1min 51s

    <IPython.core.display.Javascript object>

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

```python
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom, self.eps = mom, eps
        self.mults = nn.Parameter(torch.ones(nf, 1, 1))
        self.adds = nn.Parameter(torch.zeros(nf, 1, 1))
        self.register_buffer('sums', torch.zeros(1, nf, 1, 1))
        self.register_buffer('sqrs', torch.zeros(1, nf, 1, 1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('factor', tensor(0.))
        self.register_buffer('offset', tensor(0.))

    def update_stats(self, x):
        bs, nc, *_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0, 2, 3)
        s = x.sum(dims, keepdim=True)
        ss = (x * x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel() / nc)
        mom1 = 1 - (1 - self.mom) / math.sqrt(bs-1)
        self.sums.lerp_(s, mom1)
        self.sqrs.lerp_(ss, mom1)
        self.count.lerp_(c, mom1)
        self.batch += bs
        means = self.sums / self.count
        var = (self.sqrs / self.count).sub_(means * means)

        if bool(self.batch < 20):
            var.clamp_min_(0.01)

        self.factor = self.mults / (var + self.eps).sqrt()
        self.offset = self.adds - means * self.factor

    def forward(self, x):
        if self.training:
            self.update_stats(x)
        return x * self.factor + self.offset
```

    <IPython.core.display.Javascript object>

```python
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

    <IPython.core.display.Javascript object>

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
learn.model
```

    Sequential(
      (0): Sequential(
        (0): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (1): Sequential(
        (0): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (2): Sequential(
        (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (3): Sequential(
        (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (4): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (5): AdaptiveAvgPool2d(output_size=1)
      (6): Lambda()
      (7): Linear(in_features=64, out_features=10, bias=True)
    )

    <IPython.core.display.Javascript object>

```python
%time run.fit(1, learn)
```

    train: [0.4170171875, tensor(0.8936)]
    valid: [0.3150417236328125, tensor(0.9235)]
    CPU times: user 1min 51s, sys: 418 ms, total: 1min 52s
    Wall time: 1min 52s

    <IPython.core.display.Javascript object>

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 512), c)
```

    <IPython.core.display.Javascript object>

```python
learn, run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [0.22383076171875, tensor(0.9292)]
    valid: [0.09052369384765625, tensor(0.9712)]
    train: [0.0656801171875, tensor(0.9794)]
    valid: [0.06957890014648438, tensor(0.9794)]
    train: [0.044756376953125, tensor(0.9858)]
    valid: [0.07446788940429687, tensor(0.9781)]
    CPU times: user 47.1 s, sys: 2.92 s, total: 50.1 s
    Wall time: 25.2 s

    <IPython.core.display.Javascript object>

```python
# learn,run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
#                           +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

    <IPython.core.display.Javascript object>

```python
%time run.fit(3, learn)
```

    train: [0.032991181640625, tensor(0.9894)]
    valid: [0.0602272705078125, tensor(0.9817)]
    train: [0.02374192626953125, tensor(0.9923)]
    valid: [0.06647666015625, tensor(0.9819)]
    train: [0.0173309326171875, tensor(0.9948)]
    valid: [0.08294573974609375, tensor(0.9761)]
    CPU times: user 46.8 s, sys: 2.88 s, total: 49.7 s
    Wall time: 25 s

    <IPython.core.display.Javascript object>

## Export

```python
nb_auto_export()
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>