# Machine learning ("decoding") analyses
This week's tutorial is about how to implement "decoding" analyses in Python!

The term "decoding" is often used to denote analyses that aim to predict a single experimental feature (which can be either within-subject or between-subject) based on patterns of neuroimaging data. Some more advanced techniques make it possible to predict more than one experimental feature at once (with [Bayesian "reconstruction" techniques](https://www.sciencedirect.com/science/article/pii/S0896627309006850) and ["inverted encoding models"](https://www.sciencedirect.com/science/article/pii/S0896627315006352)), but these are beyond the scope of this course. Here, we'll focus on machine learning/statistical models and techniques that allow you to predict a single experimental feature.

We'll make heavy use of the awesome [scikit-learn](http://scikit-learn.org/stable/) library — the go-to library for machine learning in Python.

**What you'll learn**: at the end of this tutorial, you will be able to:

* use and implement feature-selection/extraction methods;
* fit machine learning models and predict (new) samples;
* implement cross-validation routines;
* statistically evaluate model performance estimates;

**Estimated time needed to complete**: 8-12 hours

**Credits**: if you use scikit-learn in your research, please cite the corresponding [paper](http://www.jmlr.org/papers/v12/pedregosa11a)

## Data representation
Decoding analyses always need two sets of data: the brain patterns, which we'll refer to as $\mathbf{R}$ (for **R**esponse), and a single experimental feature that we want to predict, which we'll refer to as $\mathbf{S}$ (which traditionally refers to **S**timulus). Note that $\mathbf{s}$ could be a within-subject experimental factor (such as stimulus or response-related factors) or a between-subject variable (such as age or depressed vs. healthy control). Moreover, the to-be-predicted variable\* can be either continuous (e.g., reaction time) or categorical (e.g., object category), which are associated with different types of models, regression and classification models respectively (more about this later). Note that "direction" of analysis is the exact opposite of what is done in enconding analyses. In encoding analyses, we try to predict the brain data (dependent variable) using experimental features (independent variables), while in decoding analyses we try to predict an experimental feature (dependent variable) using a a set of brain patterns (independent variables)! But essentially, encoding and decoding models are mathematically the same (they just use different inputs).

In the first part of this lab, we'll work with simulated data. For now, we'll assume that our data is from a simple face perception experiment in which participants viewed images with either male (condition: "M") or female faces (condition: "F") across four different fMRI runs. So, our experimental feature of interest is a categorical variable with two levels ("M" and "F"), making this a classification analysis (which is more common than regression in the context of cognitive neuroscience). Each run, participants saw fourty images (twenty for each condition) presented in a random order.

We'll simulate the patterns ($\mathbf{R}$) and experimental feature ($\mathbf{S}$, "M" vs. "F") below. For now, we'll generate random data (with some autocorrelation), for reasons that will become clear later. We'll assume that we are restricting our analysis to a single region-of-interest containing a 1000 voxels.

---
\* Often, the to-be-predicted variable is called the "target" or "dependent variable". Here, we'll use the term "target".

```python
import numpy as np
from scipy.linalg import toeplitz
from niedu.utils.nipa import generate_labels

N_per_run = 40
M = 4  # nr of runs
K = 1000  # nr of voxels

# Generate random data drawn for a multivariate normal
# distribution with AR1 noise (with phi = 0.85) to
# simulate autocorrelated noise in the estimated patterns,
# which is plausible for designs with relatively short ISIs
mu = np.zeros(N_per_run)
V = 0.85 ** toeplitz(np.arange(N_per_run))

# R_runs is a list of M arrays of shape N_per_run x K
R_runs = [np.random.multivariate_normal(mu, V, size=K).T for _ in range(M)]

# S_runs is a list of M arrays of shape N_per_run
# The custom generate_label function creates slightly correlated
# labels
S_runs = [generate_labels(['M', 'F'], N_per_run / 2, [0.7, 0.3]) for _ in range(M)]

print("Example of patterns for run 1:\n", R_runs[0])
print("\nExample of target for run 1:\n", S_runs[0])
```

Alright, technically, we have everything we need for a decoding analysis. However, machine learning (ML) and statistical models often require all data to be represented numerically, so we need to convert our target (containing the values "M" and "F") into a numeric format.

    ToDo (optional; 0 points): If you want to practice your Python skills, try converting the string labels of run 1 (the <tt>S_run1</tt> variable below) to numeric labels. Specifically, use the integer 0 for trials of condition "F" and the integer 1 for trials of condition "M". Store the result in a new variable called <tt>S_run1_num</tt>; make sure it's a numpy array. (To convert a list to a numpy array, you can do: <tt>np.array(your_list)</tt> ).

```python
S_run1 = S_runs[0]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_lab2num
test_lab2num(S_run1, S_run1_num)
```

While converting labels to numeric values can be done quite easily using standard Python, it gives us with a nice excuse to introduce some scikit-learn functionality. Specifically, the `LabelEncoder` class, which allows you to *encode* your target *labels* into a numeric format. Let's start with importing it (from the `preprocessing` module):

```python
from sklearn.preprocessing import LabelEncoder
```

The way `LabelEncoder` is used is similar to some of the Nilearn and Nistats functionality you've seen. In short, you first need to initialize the object (with, optionally, some parameters), after which you can give it data to `fit` and `transform`. This pattern is something we'll encounter a lot in this lab when working with scikit-learn.

Let's initialize a `LabelEncoder` object below:

```python
# LabelEncoder objects are not initialized with any parameters
lab_enc = LabelEncoder()
```

Now, let's "fit" it on the labels of our first run:

```python
lab_enc.fit(S_runs[0])
```

Notice that the `fit` function doesn't *return* anything useful (well, technically, it returns it*self*). Instead, when calling `fit`, it stores some parameters in the object itself as *attributes* which are used when calling the transform. For the `LabelEncoder` specifically, it stores the unique conditions (often called *classes* in machine learning) in an attribute called `classes_`:

```python
print(lab_enc.classes_)
```

In general, most "things" that are inferred or computed in the `fit` method of scikit-learn objects (and are needed later when calling `transform`) are stored in attributes with a trailing underscore (like `classes_`). We know, this all sounds incredibly trivial, but explaning these things in detail will give you a better understanding of how scikit-learn works (which is going to help a lot when dealing with more complicated functionality).

Finally, after fitting the `LabelEncoder` object, we can call the `transform` method to actually transform the labels:

```python
S_run1_num = lab_enc.transform(S_runs[0])
print(S_run1_num)
```

Note that, unlike the `fit` method, the `transform` method actually returns something, i.e., the transformed labels. Also note that the numeric labels are assigned alphabetically (i.e., "F" gets assigned 0, "M" gets assigned 1).

Like you've seen in the Nilearn notebook, we can often call `fit` and `transform` at once using the `fit_transform` method:

```python
S_run1_num = lab_enc.fit_transform(S_runs[0])
print(S_run1_num)
```

Also, after the `LabelEncoder` has been fit, it can be reused on other data, i.e., you can call the `transform` method on new arrays. This is how many classes in scikit-learn are actually used (i.e., fit on a particular subset of data and then apply to another subset), as it allows for efficient *cross-validation* &mdash a topic that will discuss in detail later.

    ToDo (1 point): We just converted the labels from the first run only. For this ToDo, convert the labels from all runs (i.e., the <tt>S_runs</tt> variable) using the <tt>LabelEncoder</tt>. Store the results in a new variable named <tt>S_runs_num</tt>, which should be a list (of length four, i.e., four runs) of arrays with ones and zeros (instead of "M" and "F").

```python
''' Implement your ToDo here . '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_lab2num_all_runs
test_lab2num_all_runs(S_runs, lab_enc, S_runs_num)
```

Alright, now we have everything we need to start building our decoding pipeline!

## Standardization
In addition to the "preprocessing" steps for pattern analyses discussed in last week's lab, when doing decoding, you additionally need to standardize your brain features (i.e., the columns in your pattern matrix) on which you fit your model. With "standardization", we mean making sure each feature ($\mathbf{R}_{j}$ for column $j$) in your pattern matrix has 0 zero mean and unit (1) standard deviation, which can be achieved as follows for each feature $j$:

\begin{align}
\mathbf{R}_{j, norm} = \frac{\mathbf{R}_{j} - \bar{\mathbf{R}}_{j}}{\hat{\sigma}(\mathbf{R}_{j})}
\end{align}

where $\bar{\mathbf{R}_{j}}$ represents the *mean* of $\mathbf{R}_{j}$ and $\hat{\sigma}(\mathbf{R}_{j})$ represents the standard deviation of $\mathbf{R}_{j}$. In other words, for each value in $\mathbf{R}$, you subtract the mean from the column it belongs to and subsequently you divide the result by the standard deviation of the column it belongs to. This process is also known as *z-scoring*.

This standardization process is done for each brain feature (column) separately. Standardization is important for most ML/statistical models because it makes sure that each brain feature has the same *scale*, which often helps in efficiently estimating model parameters.

Importantly, when you have patterns from multiple runs (as is often the case in fMRI decoding), these patterns should also be independently standardized, even if you want to pool these patterns later on (see [Lee & Kable, 2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0207083)). This is because some runs may yield patterns with a relatively higher mean or standard deviation across samples (for example, because participants start moving more towards the end of the experiment, leading to more noisy pattern estimates).

    ToDo (1 point): In this ToDo, you have to standardize the data from run 1 (the <tt>R_run1</tt> variable below) using Numpy (afterwards, we'll show you how to do this using scikit-learn). Store the results (which should have columns with mean zero and unit standard deviation) in a new variable called <tt>R_run1_norm</tt>. Importantly, this can be done efficiently (without a for-loop) using broadcasting!

```python
''' Implement your ToDo here. '''
R_run1 = R_runs[0]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the ToDo above. '''
# ToThink: do you understand how we're testing your answer here?
np.testing.assert_array_almost_equal(R_run1_norm.mean(axis=0), np.zeros(R_run1.shape[1]))
np.testing.assert_array_almost_equal(R_run1_norm.std(axis=0), np.ones(R_run1.shape[1]))
print("Well done!")
```

We can do this similarly using the `StandardScaler` class from scikit-learn, which has the same `fit`/`transform` interface:

```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
R_run1_norm = scaler.fit_transform(R_run1)
```

In the fitting process, the `StandardScaler` computed the feature-wise mean and standard deviation, which it stores in the `mean_` and `_scale` attributes:

```python
print(scaler.mean_.shape)
print(scaler.scale_.shape)
```

For the initial run-wise standardization, we usually don't want to cross-validate our standardization process (i.e., to for example fit the `StandardScaler` on run 1 and subsequently transform the other runs) for reasons discussed before. As such, we need to call fit and transform on each run separately:

```python
# Below, we use a "list comprehension" to loop across our runs
# to standardize each run separately!
R_runs_norm = [scaler.fit_transform(this_R) for this_R in R_runs]
```

We're almost ready to start fitting models. Before we do so, we are going to concatenate our data ($\mathbf{R}$ and $S$) across runs because we want to give our model as much data as possible!

```python
# Load S_runs_num if you didn't manage to do the last ToDo
S_runs_num = np.load('S_runs_num.npy')

R_all = np.vstack(R_runs_norm)  # stack vertically
S_all = np.concatenate(S_runs_num)

print("Shape R_all:", R_all.shape)
print("Shape S_all:", S_all.shape)
```

## Fitting models
When fitting decoding models, we assume that we can approximate our target variable ($\mathbf{S}$) as a function of the input data ($\mathbf{R}$):

\begin{align}
\hat{\mathbf{S}} \approx f(\mathbf{R})
\end{align}

Usually, the models used in pattern analyses assume linear functions (especially with relatively little data), i.e., functions that approximate the target using a linear combination of input variables ($\mathbf{R}_{j}$) weighted by parameters ($\beta$). Note that the univariate GLM often used in encoding models is such a linear model. The process of model fitting is estimating parameters that minize the discrepancy (error) between the predicted values ($\hat{\mathbf{S}}$) and the actual values ($\mathbf{S}$) of the target variable, both for regression ($\mathbf{S}$ is continuous) and classification ($\mathbf{S}$ is categorical) models. Different models differ in how they exactly estimate their parameters, but the general process is the same (minimizing error between predictions and target). In this course, we won't go much into the differences across models (partly because in practice, we found that performance doesn't differ that much between models).

Alright, let's get to it. Below, we import the [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) class, a particular linear *classification* model (unlike the name suggests).

```python
from sklearn.linear_model import LogisticRegression
```

We are many "options" (often called "hyperparameters") we can set upon initialization of a `LogisticRegression` object, but for now, we will only set the "solver" (for no other reason that this will get rid of a warning during the fitting process):

```python
# clf = CLassiFier
clf = LogisticRegression(solver='lbfgs')
```

Now, the fitting process using this model (or actually, *any* model in scikit-learn) is as simple as, guess what, calling the `fit` method! Unlike the `LabelEncoder` and `StandardScaler` that we discussed before, models in scikit-learn (including `LogisticRegression`) require two arguments when calling their `fit` method: `X` and `y`, which represent the input data (in our case: $\mathbf{R}$) and the target variable (in our case: $\mathbf{S}$):

```python
# The text in the output cell is there because the
# fit model returns "itself" (you can just ignore this)
clf.fit(R_all, S_all)
```

After fitting, the model stores the estimated parameters ($\beta$) as an araay in the `coef_` ("coefficients", another term for parameters) attribute:

```python
print("Shape of coef_:", clf.coef_.shape)
```

As you can see, the model estimated one parameter for each brain feature (column in $\mathbf{R}$). Now, unlike the previously discussed `LabelEncoder` and `StandardScaler`, scikit-learn models do not have a `transform` method; instead, they have a `predict` method, which takes a single input (a 2D array with observations) and generates discrete\* predictions for this input. For now, we'll call `predict` on the same data we've fit the model on:

---
\* Some models, including the `LogisticRegression` model, have an additional method called `predict_proba` which returns probabilistic instead of discrete predictions.

```python
preds = clf.predict(R_all)
print("Predictions for samples in R_all:\n", preds)
```

## Model evaluation
Alright, we have some preditions for our data! But how do we evaluate these predictions? This actually depends on whether you have a classification model (with categorical predictions) or a regression model (with continuous predictions). Because classification analyses are most popular in cognitive neuroscience (and our example data is categorical) and you are already familiar with some regression metrics such as $R^2$ (discussed in the previous course), we'll focus on model evaluation metrics for classification here.

### Metrics for discrete predictions
There are many different metrics to evaluate the predictions of classification models. The most well known (but not necessarily always appropriate) metric for *discrete* predictions is *accuracy*, which is defined as follows:

\begin{align}
\mathrm{accuracy} = \frac{\mathrm{number\ of\ correct\ predictions}}{\mathrm{total\ number\ of\ predictions}}
\end{align}

For accuracy, the best possible score is 1 (predict all samples correctly) and "chance level" performance (i.e., the expected score when randomly guessing) is, in general, $\frac{1}{\mathrm{Number\ of\ classes}}$, so for our example with two classes ("M" and "F"), it is 0.5.

    ToDo (1 point): Using the predictions (the <tt>preds</tt> variable) and the true labels (the <tt>S_all</tt> variable), compute the associated accuracy and store this in a variable <tt>acc</tt>. Note: there is no need for a for-loop! You can use the fact that boolean values, <tt>True</tt> and <tt>False</tt>, evaluate to 1 and 0, respectively, in Python...

```python
''' Implement your ToDo here. '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_acc
test_acc(preds, S_all, acc)
```

### Metrics for probabilistic predictions
Some classifiers allow for *probabilistic* (instead of discrete) predictions. For those classifiers, an additional method called `predict_proba` exists, which outputs a probability for each class. So, in a two-class classification setting, the `predict_proba` method will not give a single discrete prediction (i.e., either `0` or `1`) but a probability distribution over classes (e.g., 0.92 for class `0` and 0.08 for class `1`).

The `LogisticRegression` model from scikit-learn actually allows for probabilistic prediction. In general, if we'll give it all $N$ trials for a target variable with $M$ classes, it will output a $N \times M$ array with probabilities:

```python
probas = clf.predict_proba(R_all)
# let's print the first five trials
print(np.round(probas[:5, :], 3))
```

One often-used performance metric for probabilistic predictions is the "Area Under the ROC curve" (often abbreviated as *AUROC* or just *AUC*). Fortunately, the scikit-learn library contains implementations of many performance metrics, including AUROC (called `roc_auc_score` in scikit-learn), which can be imported from the `metrics` module:

```python
from sklearn.metrics import roc_auc_score
```

AUROC is an excellent metric to use for probabilistic predictions, but there's one caveat: when evaluating probabilistic predictions (formatted as a $N \times M$ matrxi), it needs the *true* target values (dependent variable) in a *one-hot-encoded* format. One-hot-encoding (OHE) is a technique that transforms a $N \times 1$ vector with $M$ classes into a $N \times M$ binary matrix:

\begin{align}
S = \begin{bmatrix}
1 \\
2 \\
1 \\
3 \\
2 \\
4
\end{bmatrix}
\underset{\Longrightarrow}{\mathrm{OHE}} \begin{bmatrix}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1
\end{bmatrix}
\end{align}

You might know this technique under the name "dummy (en)coding".

    ToDo (optional): Complete the <tt>one_hot_encode</tt> function below that takes in a 1D vector representing a target variable with any number of classes and observations and outputs a one-hot-encoded version of that target variable.

```python
''' Implement the optional ToDo here. '''

def one_hot_encode(y):
    ''' One-hot-encodes a 1D target vector.

    Parameters
    ----------
    y : numpy array
        1D target vector with N observations and M classes

    Returns
    -------
    An NxM numpy array
    '''
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
# Test 1
y = np.array([0, 1])
out = one_hot_encode(y)
ans = np.eye(2)
np.testing.assert_array_equal(ans, out)

# Test 2
y = np.array([1, 2, 3, 2, 2, 1])
out = one_hot_encode(y)
ans = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 0], [0, 1, 0], [1, 0, 0]])
np.testing.assert_array_equal(ans, out)

# Test 3
y = np.array([3, 2, 1])
out = one_hot_encode(y)
ans = np.rot90(np.eye(3))
np.testing.assert_array_equal(ans, out)

print("Well done!")
```

While the above ToDo was a nice way to practice your Python skills, we nonethless recommend using the `OneHotEncoder` class from scikit-learn to one-hot-encode your target vector. It uses the `fit`-`transform` syntax you are familiar with by now. Importantly, as the `OneHotEncoder` is, in practice, often used to one-hot-encoded predictors (independent variables), it expects a 2D array (not a 1D vector). So, when one-hot-encoding a 1D target variable, you can add a singleton axis (with `np.newaxis`) to make it work:

```python
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(sparse=False)  # we don't want a "sparse" output
S_all_ohe = ohe.fit_transform(S_all[:, np.newaxis])
print(S_all_ohe.shape)
```

Finally, we can use the `roc_auc_score` function to compute our model performance. Like any metric implementation in scikit-learn, it is called as follows:

```Python
score = metric(true_labels, predicted_labels)
```

where `true_labels` and `predicted_labels` are either 1D vectors of length $N$ (in case of discrete predictions) or 2D $N \times M$ arrays (in case of probabilistic predictions). Note that by default these metrics output a single score (often the average of the class-specific scores); some (but not all) metrics (including the `roc_auc_score`) allow the function to return a class-specific score by setting the optional argument `average` to `None`:

```python
# Omit average=None to get a single (average) score
scores = roc_auc_score(S_all_ohe, probas, average=None)
print(scores)
```

    ToDo (optional, quite difficult!): Another class of classification metrics are "pseudo $R^2$" scores. These metrics are similar to $R^2$ in regression models in the sense that they are bounded between 0 (chance performance) and 1 (perfect performance). They often assume probabilistic predictions.

One of these pseudo $R^2$ metrics is "Tjur's pseudo $R^2$" which is defined as the difference between the average (across observations) probability of a particular class and the average probability of the other class(es) for a given label. So, suppose we're dealing with a two-class classification problem (with class 0 and class 1), and the average probability for class 1 of trials belonging to class 1 is 0.9, while the average probability for class 1 of trials belonging to class 0 is 0.3, then the Tjur's pseudo $R^2$ score for class 1 is $0.9-0.3=0.6$. Formally, the Tjur's pseudo $R^2$ score for class $m$ is defined as:

\begin{align}
R^2_{m} = \frac{1}{N}\sum_{i=1}^{N}p(\hat{s}^{m}_{i}) - \frac{1}{N}\sum_{i=1}^{N}p(\hat{s}^{\neg m}_{i})
\end{align}

for any set of trials belonging to class 1 ($p^{m}_{i}$) and complementary set of trials *not* beloning to class 1 ($p^{\neg m}_{i}$).

Complete the function <tt>tjur_r2</tt> below that takes two arguments — <tt>target</tt> (a 1D array with target labels) and <tt>probas</tt> (a 2D array with probabilities) — and should output an array of length $M$ with Tjur's pseudo $R^2$ scores for the $M$ classes in the target array.

```python
''' Implement your ToDo here. '''

def tjur_r2(target, probas):
    ''' Computes Tjur's R2 score for all classes in `target`.

    Parameters
    ----------
    target : numpy array
        A 1D array with numerical targets
    probas : numpy array
        A 2D array with target probabilities

    Outputs
    -------
    A numpy array of length M with R2 scores for all M classes
    '''

    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
''' Tests the ToDo above. '''
y = np.array([0, 0, 1, 1])
probas = np.array([[0.9, 0.1], [0.95, 0.05], [0.3, 0.7], [0.4, 0.6]])
ans = tjur_r2(y, probas)
np.testing.assert_array_almost_equal(ans, np.array([0.575, 0.575]))

y = np.array([1, 1, 3, 3, 2, 2])
probas = np.array([[0.8, 0.1, 0.1], [0.7, 0.2, 0.1], [0.4, 0.0, 0.6], [0.3, 0.1, 0.6], [0.1, 0.5, 0.4], [0.2, 0.6, 0.2]])
ans = tjur_r2(y, probas)
np.testing.assert_array_almost_equal(ans, np.array([0.5, 0.45, 0.4]))
print("Well done!")
```

    Tip: This article by Dinga and colleagues (2019) argues against "accuracy" as a performance metric for classification models and reviews several (often probabilistic) alternatives; a highly recommended read!

## Cross-validation
If you did the previous ToDos correctly, you should have found that the accuracy was 1 (the maximum possible score)! Amazing! But wait, how is this possible? We generated *random* data, right?

So, what is the issue here? Well, we fitted the model on the same data that we want to generate predictions for! While this is common practice in many statistical models in psychology and neuroscience (including standard univariate "activation-based" fMRI models), this is not advisable for decoding models. The reason for this is that our decoding models often have many more predictors (i.e., brain features) than observations (i.e., trials). The consequence is that the model has a hard time figuring out what is "signal" and what is "noise", which will often cause your model to capitalize on spurious correlations between your data ($\mathbf{R}$) and the target ($\mathbf{S}$). The result is that your model will be *overfitted* and your model performance estimate will be overly optimistic estimate of generalization performance.

One possible solution, common in decoding analyses, is to *cross-validate* your analysis. Cross-validation is, in it's broadest definition, the process of estimating analysis parameters on a different subset of your data than the data you want to generate predictions for. With "analysis parameters", we do not only mean the parameters of your statistical model ($\hat{\beta}$), but this may also involve parameters estimated during preprocessing and feature transformations (which we'll talk about later). Importantly, cross-validation (if done properly) allows you to derive an unbiased estimate of generalization performance, i.e., how well your analysis would generalize to a new dataset.

Usually, the subset of data you use to fit your analysis parameters on is called the *train* set and the subset you evaluate your model predictions on is often called the *test* set. Assuming that each observation (i.e., row in $\mathbf{R}$) is independent from all other observations, any spurious correlation that is capitalized upon in the train set will not generalize to the test set!

There are different cross-validation schemes (i.e., how you partition your data in a train and set set). For the example in the next cell, we'll use a simple *hold-out* scheme, in which we'll reserve 50% of the data for the test set (note that this could have been a different percentage). (We'll discuss more intricate cross-validation schemes such as K-fold in the next section).

```python
R_train = R_all[0::2, :]  # all even samples
S_train = S_all[0::2]

R_test = R_all[1::2, :]  # all odd samples
S_test = S_all[1::2]
```

After splitting the data into a train and test set, we have introduced a "problem" however: the features within the train and test set may not have 0 mean and unit (1) variance anymore! Given that the features were properly standardized across *all* samples in our simulated fMRI dataset, this is unlikely to be a problem for our classifier. It is still good practice to make sure your *train set* is properly standardized. So, before fitting our classifier, let's standardize the train set:

```python
R_train_norm = scaler.fit_transform(R_train)
```

Now, we can fit our model on the standardized train set ...

```python
clf.fit(R_train_norm, S_train)
```

Before predicting the test set, however, we need to decide whether we want to independently standardize the test set or whether to *cross-validate* our previously estimated standardized parameters (the feature-wise mean and standard deviation). Although opinions differ on this topic (see e.g. [this excellent article](https://jamanetwork.com/journals/jamapsychiatry/article-abstract/2756204)), if we want a truly unbiased estimate of generalization, we should also cross-validate our standardization procedure in addition to cross-validation of our model. So, to cross-validate our standardization procedure, we do the following for each feature $j$ of our test set:

\begin{align}
\mathbf{R}_{j, norm}^{\mathrm{test}} = \frac{\mathbf{R}_{j}^{\mathrm{test}} - \bar{\mathbf{R}}_{j}^{\mathrm{train}}}{\hat{\sigma}(\mathbf{R}_{j}^{\mathrm{train}})}
\end{align}

```python
# Note that we're *not* calling fit on the test set, i.e.,
# we're cross-validating our standardization procedure!
R_test_norm = scaler.transform(R_test)
```

So, now we can cross-validate our model and derive predictions for our test set:

```python
preds = clf.predict(R_test_norm)
print("Predictions for our test set samples:\n", preds)
```

These predictions (`preds`) are made independently from the fitting process. Now, let's evaluate the model performance on these predictions, this time we're going to be lazy and use the `accuracy_score` from the `metrics` module in scikit-learn:

```python
from sklearn.metrics import accuracy_score
acc_cv = accuracy_score(S_test, preds)
print("Cross-validated accuracy:", acc_cv)
```

As you can see, accuracy is not perfect (1.0) anymore, but it is still much higher than you'd expect on random data (for which chance-level performance would be 0.5).

In hold-out cross-validation (which we demonstrated previously), you use your train set *only* for fitting and your test set *only* to evaluate model predictions. In other words, you only fit and predict once, but on different subsets of your data. If you have a big dataset (i.e., many samples), your test set can be relatively large, and thus cross-validated accuracy on the test set will probably be a good estimate of how well our model will generalize to future/other data. However, if you have a relatively small dataset, you will probably have a relatively small test-set. If you then estimate cross-validated accuracy on this test-set, the chance of just getting a relatively good (or bad) score by coincidence is quite high (see e.g. [Varoquaux et al., 2017](https://www.sciencedirect.com/science/article/pii/S105381191630595X#s0055))! In other words, the estimate of cross-validation accuracy is not really robust. Fortunately, there are ways to increase robustness of cross-validation accuracy estimates; one of them is by using K-fold cross-validation instead of hold-out cross-validation, in which you divide your dataset into $K$ folds, which you iteratively use as train and test set.

ToThink (1 point)

Can you think of a (practical) reason to prefer hold-out cross-validation over K-fold cross-validation?

YOUR ANSWER HERE

As fMRI data-sets often contain few samples (trials/subjects), K-fold cross-validation is often used. Instead of writing our own K-fold cross-validation scheme, we'll use some of scikit-learn's functionality. Specifically, we are going to use the [StratifiedKFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) class from scikit-learn's `model_selection` module. Click the highlighted link above and read through the manual to see how it works.

Importantly, if you're dealing with a classification analysis, always use **Stratified**KFold (instead of the regular [KFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html)), because this version makes sure that each fold contains the same proportion of the different classes (here: 0 and 1).

Anyway, enough talking. Let's initialize a `StratifiedKFold` object with, let's say, 5 folds:

```python
from sklearn.model_selection import StratifiedKFold

# They call folds 'splits' in scikit-learn
skf = StratifiedKFold(n_splits=5)
```

Alright, we have a `StratifiedKFold` object now, but not yet any indices for our folds (i.e. indices to split our $\mathbf{R}$ and $S$ into different subsets). To do that, we need to call the `split` method, which takes two inputs: the data ($\mathbf{R}$) and the target ($S$):

```python
folds = skf.split(R_all, S_all)
```

Now, we created the variable `folds` which is, technically, a [generator](https://wiki.python.org/moin/Generators) object, but just think of it as a type of list (with indices) which is specialized for looping over it. Each entry in `folds` is a tuple with two elements: an array with train indices and an array with test indices. Let's demonstrate that\*:

-------------
\* Note that you can only run the cell below once. After running it, the `folds` generator object is "exhausted", and you'll need to call `skf.split(R_all, S_all)` again in the above cell.

```python
for i, fold in enumerate(folds):

    print("Processing fold %i" % (i + 1))
    # Here, we unpack fold (a tuple) to get the train and test indices
    train_idx, test_idx = fold

    print("Train indices:", train_idx)
    print("Test indices:", test_idx, end='\n\n')
```

In a proper analysis, you would fit a model on the train set, predict the labels for the test set, and compute the cross-validated accuracy for all $K$ folds separately. Note that, in the case of K-fold cross-validation, you technically estimating $K$ different models (i.e., the estimated model parameters are likely slightly different across folds). For most decoding purposes, this is not necessarily a problem, as we're often not interested in the model *parameters*, but the (cross-validated) model *performance*. As such, people usually compute the fold-wise model performance and subsequently average these values to get an average model score — which is exactly what you're going to do in the next ToDo.

    ToDo (2 points): In the code cell below, initialize a <tt>StratifiedKFold</tt> object with 4 folds and write a for-loop that iterates across the 4 folds. Use the following additional parameters when initializing the <tt>StratifiedKFold</tt> object: <tt>random_state=42</tt> (this is to be able to test your implementation) and <tt>shuffle=True</tt> (which will draw random folds). Store this object in a variable named <tt>skf_4f</tt>.

In every iteration, divide the data into a train and test set, apply (cross-validated) standardization, fit the  model (you can reuse the <tt>LogisticRegression</tt> object from before) on the train set, predict the test set, and compute the accuracy. Store the accuracy for each iteration. After the loop, you should have 4 cross-validated accuracy scores. Average these and store the result (a single number) in a variable named <tt>acc_cv_average</tt>.

```python
''' Implement the ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_skf_loop_with_seed
test_skf_loop_with_seed(R_all, S_all, scaler, clf, skf_4f, acc_cv_average)
```

    Tip: you might wonder how many folds you should choose. It is tempting to choose as many folds as possible, i.e. to make your train set as big as possible, such that your model has as much data as possible to train on. The flip side, however, is that your test set becomes smaller with larger train sets, which tends to lead to highly variable cross-validated model performance estimates (e.g., 0.9 in one fold, but 0.35 in the other fold). It has been recommended to use a test set size of about 10%-20% of the size of your entire dataset (i.e., using 5-10 folds; see this article).

### Sidenote: scikit-learn `Pipelines`
As you might have noticed in the previous ToDo, it takes quite some lines of code to fully cross-validate your standardization step and model: fit your scaler on the train set, transform the train set, transform the test set, fit your model on the train set, and finally predict your test set. This cross-validation routine will only become more complicated and cumbersome when you add extra preprocessing or transformation procedures to it (as we'll do in the next section). As such, let us introduce one of the most amazing features of scikit-learn: `Pipelines`.

    ToDo (0 points): Read through the documentation of the Pipeline class.

Scikit-learn `Pipeline`s allow you to "bundle together" a sequence of analysis steps (which may include preprocessing and feature transformation operations) that usually ends in a model (e.g., a `LogisticRegression` object). Then, you can fit all steps on a particular subset of data by calling the `Pipeline`'s `fit` method and subsequently call the `predict` method on another subset of data, which will *automatically cross-validate every step in your analysis pipeline*. Instead of initializing `Pipeline` objects directly, we'll use the convenience function `make_pipeline`:

```python
from sklearn.pipeline import make_pipeline
```

Now, the `make_pipeline` function accepts an arbitrary number of arguments which should all be either preprocessing or feature transformation objects (i.e., so-called [transformator](https://scikit-learn.org/stable/data_transforms.html) objects) or model objects (i.e., so-called `estimator` objects), such as a `LogisticRegression` object. Note that you can only have a single model object in your pipeline, which should be the *last* step in your pipeline.

Let's create a very simple pipeline that involves standardization and a logistic regression model (like you implemented in the previous ToDo):

```python
# We re-initialize these objects for clarity
scaler = StandardScaler()
clf = LogisticRegression(solver='lbfgs')

# The make_pipeline function returns a Pipeline object
pipe = make_pipeline(scaler, clf)
print(pipe)
```

Now, let's use the data from the simple hold-out split from before to demonstrate the fiting and cross-validation of our complete pipeline. Just like a normal model, we can call the `fit` and `predict` methods to do so:

```python
R_train = R_all[0::2, :]
R_test = R_all[1::2, :]
S_train = S_all[0::2]
S_test = S_all[1::2]

# First, let's fit *all* the steps
pipe.fit(R_train, S_train)

# And now cross-validate *all* the steps
preds = pipe.predict(R_test)
```

Awesome, right? Using pipelines saves you many lines of code and allows you to easily cross-validate entire pipelines. You'll practice with pipelines in the upcoming ToDo.

Now, back to cross-validation routines. One notable variant of K-fold cross-validation is *repeated* stratified K-fold cross-validation, in which the cross-validation loop is repeated several times with different (random) folds. This way, the cross-validated model performance estimates usually become more stable (i.e., less variance). (Of course, scikit-learn contains a [RepeatedStratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html#sklearn.model_selection.RepeatedStratifiedKFold) class.)

Another notable cross-validation scheme, especially for fMRI-based decoding analyses, is group-based cross-validation, in which folds are created based on a particular grouping variable. In fMRI-based decoding analyses, this type of cross-validation is often applied to cross-validate models across runs. Specifically, the leave-one-run-out technique is often used, in which a model is fit on all trials except the trials from a single run (the train set) and is cross-validated to the trials of the left-out run (the test set).

This functionality is implemented in the `LeaveOneGroupOut` class in scikit-learn:

```python
from sklearn.model_selection import LeaveOneGroupOut
logo = LeaveOneGroupOut()
```

This cross-validation object is very similar to the other objects (e.g., `StratifiedKfold`) you have seen, except that when calling the `split` method, you need to provide an additional parameter `groups`, which should be an array/list with integers denoting the different groups:

```python
folds = logo.split(data, target, groups)
```

For our dataset, we can create a groups-variable based on the different runs as follows:

```python
groups =  np.concatenate([[i] * N_per_run for i in range(M)])
print(groups)
```

    ToDo (3 points): In this ToDo, you'll practice with both the Pipeline class as well as leave-on-group-out cross-validation. For this ToDo, you're going to create a new pipeline with a variant of the <tt>StandardScaler</tt> class — the <tt>RobustScaler</tt> class — and another model — the <tt>RidgeClassifier</tt> class (also from the <tt>linear_model</tt> module).

1. Create a new <tt>Pipeline</tt> object with the aforementioned <tt>RobustScaler</tt> and <tt>RidgeClassifier</tt> objects (which you have to import yourself) using the <tt>make_pipeline</tt> function;

2. Use the previously defined <tt>logo</tt> object to create a loop across folds, in which you should cross-validate your entire pipeline and compute each fold's cross-validated accuracy.

3. After the loop, average the four accuracy values and store this in a variable named <tt>acc_cv_logo</tt>.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_logo_loop
test_logo_loop(R_all, S_all, logo, groups, acc_cv_logo)
```

    ToThink (1 point): The average performance for the leave-one-run-out cross-validation analysis is, unlike the previous stratified K-fold results (which should be considerably above chance), near chance-level, as you would expect for random noise data. Explain why run-wise cross-validation solves the bias that we observed when ignoring the trial structure across runs. Hint: look at the way we simulated data (which reflects how real data may look like).

YOUR ANSWER HERE

## Class imbalance and model performance revisited
Before moving on to more exciting aspects of decoding pipelines, let's discuss class imbalance. Class imbalance, the situation in which not all classes within your target variable have the same number of samples, can have a big impact on your classification model (note: this is not relevant for regression models, as they don't have *classes*!). This is not uncommon in decoding models, especially when you don't have control over the feature of interest, such as response-based features (e.g., whether the participant pressed left or right) or between-subjects variables (e.g., when dealing with clinical populations, which cannot always be perfectly balanced due to practical reasons).

To illustrate this, let's look at what happens with random data and a imbalanced target variable. We'll simulate some random data (like our previously used data, this contains no "signal" at all: you'd expect 50% accuracy) a couple of times and we'll calculate the cross-validated accuracy.

```python
# Simulate random (there is no effect!) data
N = 100  # samples
K = 3  # brain features
ratio01 = 0.8  # ratio class 0 / class 1
clf = LogisticRegression(solver='lbfgs')

iters = 10
for i in range(iters):
    R_random = np.random.normal(0, 1, size=(N, K))

    # Simulate random target with prespecified imbalance ('ratio01')
    S0 = np.zeros(int(np.round(N * ratio01)))  # 80% is class 0
    S1 = np.ones(int(np.round(N * (1 - ratio01)))) # 20% is class 1
    S_random = np.concatenate((S0, S1))

    # Now, let's split it in a train- test-set
    R_train = R_random[::2, :]
    R_test = R_random[1::2, :]
    S_train = S_random[::2]
    S_test = S_random[1::2]

    clf.fit(R_train, S_train)
    S_pred = clf.predict(R_test)
    this_acc = accuracy_score(S_test, S_pred)
    print("Accuracy: %.2f" % this_acc)
```

When running the cell above, you should find that the models is able to consistently yield strong above-chance performance, which shouldn't happen because the data that we simulated is just random noise!

    ToDo (0 points): Try setting the <tt>ratio01</tt> variable to other numbers (in between 0.1 - 1), for example 0.9, 0.2, or 0.5, and see what happens with the cross-validated accuracy score!

ToThink (1 point): Suppose that you have a dataset with a binary target variable ($S = \{0, 1\}$) a train set in which 80% of the samples are of class 0 and a test set in which 80% of the samples are of class 1. Assuming similar parameters ($N$, $K$, type of classifier, etc.) as used in the previous simulation, what do you think will be the (cross-validated) accuracy on the test set? Why?

YOUR ANSWER HERE

As you can see, the model "learns" to just predict the majority class (the class with the most samples)! In a way, class imbalance also provides the classifier with a source of "information" that it can use to derive accurate (but theoretically meaningless) predictions.

Therefore, imbalanced datasets therefore need a different evaluation metric than accuracy. Fortunately, scikit-learn has many more performance metrics you can use, including metrics that "correct" for the (potential) bias due to class imbalance (including [f1-score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score), [ROC-AUC-score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score), [balanced-accuracy score](http://scikit-learn.org/dev/modules/generated/sklearn.metrics.balanced_accuracy_score.html), and [Cohen's Kappa](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html#sklearn.metrics.cohen_kappa_score)).

Let's check out what happens with our performance estimate if we use a different ('class-imbalance-aware') metric, the "ROC-AUC-score" we discussed previously, which should take care of the bias induced by imbalance.

```python
from sklearn.metrics import roc_auc_score

for i in range(iters):
    R_random = np.random.normal(0, 1, size=(N, K))

    # Simulate random target with prespecified imbalance ('ratio01')
    S0 = np.zeros(int(np.round(N * ratio01)))  # 80% is class 0
    S1 = np.ones(int(np.round(N * (1 - ratio01)))) # 20% is class 1
    S_random = np.concatenate((S0, S1))

    # Now, let's split it in a train- test-set
    R_train = R_random[::2, :]
    R_test = R_random[1::2, :]
    S_train = S_random[::2]
    S_test = S_random[1::2]

    clf.fit(R_train, S_train)
    S_pred = clf.predict(R_test)
    this_acc = roc_auc_score(S_test, S_pred)
    print("Accuracy: %.2f" % this_acc)
```

Much better! In addition to `roc_auc_score`, there are other metrics (such as `balanced_accuracy` and `f1_score`) that deal appropriately with class imbalance.

However, using class-imbalance-aware metrics only makes sure that the model performance estimate is unbiased (i.e., is not affected by class imbalance), but it doesn't prevent the model from actually "learning" the useless "information" related to class imbalance (instead of learning the true/useful signal in the data)! In our previous examples which used completely random (null) data, this is not a problem, but it *is* a problem when the data actually contains some effect. One way to counter this is by using the `class_weight` parameter that is available in mode scikit-learn models. Setting the parameter to "balanced" will weigh samples from the minority class more strongly than samples from the majority class, forcing the model to learn information that is independent from class frequency. Below, we initialize a logistic regression model with this setting enabled and show that it effectively reduces the influence of class imbalance:

```python
clf = LogisticRegression(solver='lbfgs', class_weight='balanced')
for i in range(iters):
    R_random = np.random.normal(0, 1, size=(N, K))

    # Simulate random target with prespecified imbalance ('ratio01')
    S0 = np.zeros(int(np.round(N * ratio01)))  # 80% is class 0
    S1 = np.ones(int(np.round(N * (1 - ratio01)))) # 20% is class 1
    S_random = np.concatenate((S0, S1))

    # Now, let's split it in a train- test-set
    R_train = R_random[::2, :]
    R_test = R_random[1::2, :]
    S_train = S_random[::2]
    S_test = S_random[1::2]

    clf.fit(R_train, S_train)
    S_pred = clf.predict(R_test)
    this_acc = accuracy_score(S_test, S_pred)
    print("Accuracy: %.2f" % this_acc)
```

```python
# Running this will remove all numpy arrays up to this point
# from memory
%reset -f array
```

## Feature selection/extraction
Now that we've dicussed cross-validation, which allows you to report an unbiased estimate of generalization performance. However, especially when your data contains many brain features (e.g., voxels), your model might still perform poorly simply because it has a hard time distinguishing signal from noise. One way to "help" our model a little is to apply feature selection and/or extraction techniques, which are meant to reduce the number of features to a smaller subset which, hopefully, contain more signal (and less noise).

Feature reduction can be achieved in two principled ways:

* feature extraction: transform your features into a set of lower-dimensional components;
* feature selection: select a subset of features

Examples of feature extraction are PCA (i.e. transform voxels to orthogonal components) and averaging features within brain regions from an atlas.

Examples of feature selection are ROI-analysis (i.e. restricting your patterns to a specific ROI in the brain) and "univariate feature selection" (UFS). This latter method is an often-used data-driven method to select features based upon their univariate difference, which is basically like using a traditional whole-brain mass-univariate analysis to select potentially useful features!

ToThink (1 point): Suppose a researcher wants to decode gratings with two different orientations from V1. To delineate V1, the subject underwent a retinotopy session in a different fMRI run. The data from this retinotopy session was subsequently used to extract ("mask") V1 by excluding non-significant voxels; the significant voxels were in turn used to base the orientation decoding analysis on. Is masking V1 using the retinotopy data a form of feature selection or feature extraction? Why?

YOUR ANSWER HERE

Fortunately, scikit-learn has a bunch of feature selection/extraction objects for us to use. These objects ("transformers" in scikit-learn lingo) work similarly to models ("estimators"): they also have a `fit(R, S)` method, in which for example the univariate differences (in UFS) or PCA-components (in PCA-driven feature extraction) are computed. Then, instead of having a `predict(R)` method, transformers have a `transform(R)` method.

Before going on, let's actually load in some *real* data. We'll use the data from a single "face" run from subject 03. We already estimated the single-trial patterns on Fmriprep-preprocessed data for you using Nilearn (you can check out the code [here](https://github.com/lukassnoek/NI-edu-data/blob/master/code/nipa/estimate_patterns.py)). Let's download these patterns:

```python
import os
data_dir = os.path.join(os.path.expanduser('~'), 'NI-edu-data')
print("Downloading patterns from sub-03, ses-1, run 1 (+- ... MB) ...")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003477 {data_dir} --exclude "*" --include "derivatives/pattern_estimation/sub-03/ses-1/*task-face*run-1*"
print("\nDone!")
```

We just downloaded two nifti files: one ending in `_beta.nii.gz` and one ending in `_varbeta.nii.gz`, which represent the parameter estimates ($\hat{\beta}$) of the single-trial model and the _variance_ of the parameter estimates ($\mathrm{var}[\hat{\beta}]$) respectively:

```python
# Get directory with data
pattern_dir = os.path.join(data_dir, 'derivatives', 'pattern_estimation', 'sub-03', 'ses-1', 'patterns')

# Check what's in the nibetaseries output directory
print('\n'.join(sorted(os.listdir(pattern_dir))))
```

These two files are both 4D nifti files, in which the fourth dimension does not represent time (as you may be used to), but trials! Below, we load them in and print its shape:

```python
from nilearn import image
betas_path = os.path.join(pattern_dir, 'sub-03_ses-1_task-face_run-1_space-MNI152NLin2009cAsym_desc-trial_beta.nii.gz')
R_4D = image.load_img(betas_path)

print(f"Shape of betas: {R_4D.shape}")
```

Note that the data is, technically, not yet in the right format: it is formatted as an $X \times Y \times Z \times N$ array ($N$ = number of trials), while we should format our data as a $N \times K$ array (where $K$ is the product of the $X$, $Y$, and $Z$ dimensions). We could reshape our data ourselves (e.g., by casting the data to a numpy array and using the `reshape` method) but instead, we'll postpone this until the next section.

Also, lastly, we need to define a target variable. For the rest of the notebook, we'll try to decode "smiling" faces from "neutral" faces (the variable "expression" in the event files). Note that the event file is included in the `pattern_estimation` directory we just downloaded.

```python
import pandas as pd

# Load events-file corresponding to run 1
events_file = os.path.join(data_dir, 'sub-03', 'ses-1', 'func', 'sub-03_ses-1_task-face_run-1_events.tsv')
events = pd.read_csv(events_file, sep='\t')

# Filter out 'response' and 'rating' events
events = events.loc[events['trial_type'].str.contains('STIM'), :]

# Extract the "expression" column (containing either "smiling" or "neutral")
S_expr = events['expression'].to_numpy()

# Encode the string labels into integers
S_expr = lab_enc.fit_transform(S_expr)
S_expr  # smiling = 1, neutral = 0
```

    ToThink (1 point): The patterns we just loaded in were already normalized to a common (MNI) template. This is convenient when we want to apply masks derived from atlases (which are often defined in MNI space). But can you also think of an argument for analyzing the data in subject-native space (i.e., data that has not been normalized to a common template)?

YOUR ANSWER HERE

### ROI selection
An often used method to reduce the number of brain features is to restrict analyses to a particular region-of-interest (ROI), which can either be defined anatomically (e.g., the amygdala) or functionally (e.g., the voxels that activate significantly more to faces than to houses in a separate localizer run). For this section, we'll focus on selecting a subset of voxels based on an anatomical ROI. What ROI we can use depends on the *space* of the data, i.e., whether the data has been normalized to a common template (usually MNI152) or not. Previously, we loaded in data that has been normalized to "MNI152NLin2009cAsym" space (the specific MNI flavor used by Fmriprep). As such, we can use ROIs from atlases that are defined in MNI space, such as the Harvard-Oxford atlas, which we can load using Nilearn:

```python
from nilearn.datasets import fetch_atlas_harvard_oxford

# ho_atlas is a dictionary with the keys "labels" and "maps"
# We use the subcortical "maxprob" atlas, thresholded at 25 probability
ho_atlas = fetch_atlas_harvard_oxford('sub-maxprob-thr25-2mm')
```

Now, as you (hopefully) remember from the Nilearn notebook, we can check out which ROIs this atlas contains by looking at the 'labels' key from the atlas object:

```python
ho_atlas['labels']
```

To demonstrate ROI selection, let's focus on the right amygdala. To get the index of the right amygdala, we can do the following:

```python
r_amygd_idx = ho_atlas['labels'].index('Right Amygdala')
print("Index of right amygdala:", r_amygd_idx)
```

Let's check the shape of the map stored in the "maps" key-value pair in the `ho_map` variable (a `Nifti1Image`):

```python
import nibabel as nib
ho_map = ho_atlas['maps']
print("Shape of map:", ho_map.shape)
```

But there seems to be a problem! The map has more voxels than our data ($73 \times 86 \times 66$):

```python
print("Shape of our data:", R_4D.shape)
```

This is because the Harvard-Oxford atlas is defined in 2 millimeter space, while Fmriprep outputs preprocessed data in the original resolution of the functional data (here: 2.7 mm$^3$). To fix this, we can downsample the map to the resolution of our data using `resample_to_img` from Nilearn:

```python
from nilearn import image
ho_map_resamp = image.resample_to_img(ho_map, R_4D, interpolation='nearest')
print("Shape of resampled map:", ho_map_resamp.shape)
```

Now, we can create an ROI by looking at which voxels in the Harvard-Oxford contain the label corresponding to our right amygdala label (i.e., 20):

```python
# Compare map with right amygdala label, creating a boolean array (True, False)
r_amygd_roi_bool = ho_map_resamp.get_fdata() == r_amygd_idx
```

In Nilearn, however, (ROI) masks should be Nifti objects with only two values: ones for voxels included in the mask and zeros for excluded voxels. Our `r_amygd_roi` now has boolean values, `True` and `False`. Let's convert this boolean array to integers (0, 1) and then make it a `Nifti1Image` object:

```python
# The method .astype allows you to cast data into a different data type (here: int)
# We'll reuse the affine from the `ho_map_resamp` object
r_amygd_roi = nib.Nifti1Image(r_amygd_roi_bool.astype(int), affine=ho_map_resamp.affine)
print("Shape of amygdala ROI:", r_amygd_roi.shape)
```

And let's plot it using Nilearn to check if everything worked as expected:

```python
from nilearn import plotting
plotting.plot_roi(r_amygd_roi);
```

Looks good! Now, *finally*, we can apply the ROI to our data in the same way we'd apply a mask: using the `apply_mask` function from Nilearn:

```python
from nilearn import masking
R_ramyg = masking.apply_mask(R_4D, r_amygd_roi)
print("Shape of indexed data:", R_ramyg.shape)
```

Alright, that took some code, but we finally achieved what we wanted: a subset of voxels for our ROI (128 in total) for all our trials (40), neatly packaged in an $N \times K$ array!

    ToDo (1 point): Repeat the ROI indexing process for our data (<tt>R_4D</tt>), but this time using an ROI of the left hippocampus. Store the results (a 2D array) in a variable named <tt>R_lhippocampus</tt>.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
assert(R_lhippocampus.shape == (40, 258))
print("Well done!")
```

### PCA
There are many methods for feature extraction, like "downsampling" to ROI-averages (i.e. averaging voxel patterns in brain regions) and dimensionality-reduction techniques like PCA. Scikit-learn provides some of these techniques as "transformer" objects, which again have a `fit()` and `transform` method. We'll showcase this on the amygdala-restricted patterns we defined earlier.

```python
from sklearn.decomposition import PCA
pca = PCA(n_components=10)

pca.fit(R_ramyg)
R_ramyg_pca10 = pca.transform(R_ramyg)
print("Shape after PCA:", R_ramyg_pca10.shape)
```

ToDo (2 points): In the previous cell, we fitted the PCA to the entire dataset (all samples in <tt>R_ramyg</tt>), but again, ideally we'd add this a a step to our analysis pipeline and also cross-validate this procedure! Write a completely ) cross-validated decoding pipeline with 5 stratified folds that includes standardization (using <tt>RobustScaler</tt>), PCA dimenensionality reduction (using 10 components), and a logistic regression model which aims to predict expression (smiling vs. neutral). Keep track of the foldwise cross-validated accuracy on the test set and store the average accuracy in a variable named <tt>acc_av_todo</tt>.

```python
''' Implement the ToDo here.'''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_pca_pipe
test_pca_pipe(R_ramyg, S_expr, acc_av_todo)
```

### Region averaging
So far, we have restricted our analyses mostly to voxel patterns within a single ROI. For some research questions, however, you might want to include whole-brain patterns (for example, if you believe your experimental feature of interest is represented in whole-brain networks). This means, however, that you might be dealing with a lot of voxels (i.e., columns in your pattern matrix)! Again, you could reduce the number of features using dimensionality reduction techniques such as PCA. An alternative is to not use the activity of *voxels* as features, but the region-average activity! These regions can for example be derived from an existing anatomical atlas or parcellation or even from your own data (by fitting a clustering algorithm on the underlying timeseries).

You can do both (using existing atlases/parcellations or creating your own parcellation) using Nilearn. Here, we'll  demonstrate how to "downsample" your whole-brain voxel patterns to region-average patterns using the existing [Shaefer et al. (2018)](https://academic.oup.com/cercor/article/28/9/3095/3978804) parcellation, which we can load in using Nilearn.

```python
from nilearn.datasets import fetch_atlas_schaefer_2018
# It can return different number of ROIs; we'll set it to 100
schaefer_parc = fetch_atlas_schaefer_2018(n_rois=100, resolution_mm=2, verbose=False)
```

Note that the Schaefer parcellation is defined MNI space with a 2 millimeter resolution. Our data has the original BOLD resolution ($2.7 \times 2.7 \times 2.97$), so in order to use it on our data, we need to resample it:

```python
# Set interpolation to nearest, because we're dealing with discrete integers
schaefer_rois = image.resample_to_img(schaefer_parc['maps'], R_4D, interpolation='nearest')

# Let's also plot it:
plotting.plot_roi(schaefer_rois);
```

Note that, unlike the probabilistic ROIs that we used, the Schaefer parcellation is discrete: every voxel belongs to only a single region. As such, the 'maps' map in the `schaef_parc` object is a single volume with integers:

```python
roi_ints = schaefer_rois.get_fdata()

# Let's check out the unique values in the volume
np.unique(roi_ints)
```

Importantly, here, voxels with the value 0 represent background voxels (not belonging to any region). Now, using the `NiftiLabelsMasker` class from Nilearn, we can easly compute the ROI-average activity for each of our samples across the 100 ROIs in the Schaefer parcellation.

    ToDo (1 point): Read through the documentation of the <tt>NiftiLabelsMasker</tt> class. Then, initialize a <tt>NiftiLabelsMasker</tt> object with <tt>labels_img=schaefer_rois</tt> and a "mean" <tt>strategy</tt>. Then, using this object, transform our whole-brain 4D patterns (the variable <tt>R_4D</tt>) into a 2D array (shape $40 \times 100$) and store it in a variable named <tt>R_schaefer</tt>.

```python
''' Implement your ToDo here. '''
from nilearn.input_data import NiftiLabelsMasker

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_niftilabelsmasker
test_niftilabelsmasker(schaefer_rois, R_4D, R_schaefer)
```

### Searchlight analysis
One last technique that we want to discuss before moving on to statistical testing of decoding results is *searchlight analysis* (although it's not *really* a feature selection/extraction technique). [Searchlight analysis](https://www.pnas.org/content/103/10/3863.short) is a decoding-based analysis that is kind of a hybrid between univariate and multivariate analysis: it analyzes local voxel *patterns* but does that for every location in the brain. It does so by defining a spherical "searchlight" of voxels around a particular center voxel (with a particular radius). Within this searchlight, a (cross-validated) decoding analysis is performed and the result, usually a foldwise average model performance score (e.g., accuracy), is assigned to the center voxel. Then, the searchlight moved to the next voxel and a new decoding analysis performed; this process is repeated until each voxel has been the center voxel (and has been assigned a model performance score). As such, at the end of the searchlight analysis, you have an entire volume of (average) model performances, not just a single one!

Nilearn contains an efficient implementation of a searchlight analysis:

```python
from nilearn.decoding import SearchLight
```

    ToDo (0 points): Read through the documentation of the Searchlight estimator.

Just like other Nilearn (and scikit-learn) classes, we first need to initialize a `Searchlight` object. Upon initialization, the `Searchlight` class needs a couple of parameters:
- `mask_img`: a binary mask that indicates which voxels contain actual brain;
- `process_mask_img`: a binary mask that indicates which voxels should be analyzed;
- `radius`: radius (in mm) of the searchlight sphere;
- `estimator`: a scikit-learn compatible estimator (which can be a classifier/regression model or a `Pipeline`!)
- `n_jobs`: how many CPUs to use
- `scoring`: a string indicating which scoring method to (e.g., 'accuracy', 'roc_auc', etc.)
- `cv`: either an integer (referring to the number of desired folds) or a cross-validation object (e.g., a `StratifiedKFold` object);
- `verbose`: whether it should print out stuff while performing the fit (`True` or `False`)

Now, let's go through these arguments one by one. First, to determine a brain mask, let's just simply include all voxels that contain at least *some* signal across trials (i.e., where the sum across trials is not 0):

```python
brain_mask = image.math_img('(img.sum(axis=3) != 0).astype(int)', img=R_4D)
plotting.plot_roi(brain_mask);
```

    ToThink (1 point): Why don't the inferior temporal lobes and the orbitofrontal cortex include any signal? Write your explanation in the text box below. Hint: think about the preprocessing tutorials from the previous course!

YOUR ANSWER HERE

For the `process_mask_img`, you could of course select the same mask (`brain_mask`), but for this tutorial, that would take too long. Instead, we'll only analyze a single axial slice (the 22nd slice):

```python
# Create empty volume
sl_mask = np.zeros(brain_mask.shape)

# Set the 22nd slice to 1
sl_mask[:, :, 21] = 1

# Create a Nifti1Image object from it using the "image.new_img_like" function
# from Nilearn
sl_mask = image.new_img_like(brain_mask, sl_mask)

# Of course, we only want to analyze the voxels from that slice that
# are ALSO in the brain mask, so let's intersect them:
sl_mask = masking.intersect_masks((brain_mask, sl_mask), threshold=1)

# Let's plot it:
plotting.plot_roi(sl_mask);
```

Lastly, we need to create an "estimator" and a cross-validation object. Let's go for a simple pipeline with scaling and a `LogisticRegression` classifier and a `StratifiedKFold` CV scheme:

```python
sl_cv = StratifiedKFold(n_splits=5, shuffle=True)
sl_estimator = make_pipeline(StandardScaler(), LogisticRegression(solver='lbfgs'))
```

Finally, we can initialize the `Searchlight` object:

```python
sl = SearchLight(
    mask_img=brain_mask,
    process_mask_img=sl_mask,
    radius=5,  # we'll use a 5 mm radius
    estimator=sl_estimator,
    n_jobs=1,  # use only 1 core (for your own analyses, you might want to increase this!)
    scoring='roc_auc',  # use AUROC as model performance metric
    cv=sl_cv,
    verbose=True  # print a progressbar while fitting
);
```

To actually perform the searchlight analysis, we call the `fit` method with our data (a 4D nifti file, with the samples, $N$, in the fourth dimension) and target (the variable it needs to predict). The cell below might take a minute or two to run!

```python
sl.fit(R_4D, S_expr);
```

After fitting, the `Searchlight` object's `scores_` attribute contains the voxelwise decoding scores (here: average AUROC scores across the five folds) as a 3D numpy array (with the same shape as our brain mask):

```python
print("Shape scores_:", sl.scores_.shape)
```

Let's convert it to a `Nifti1Image` object so that we can plot it using Nilearn. We'll set the (somewhat arbitrarily chosen) threshold to 0.65:

```python
score_img = image.new_img_like(brain_mask, sl.scores_)
plotting.plot_stat_map(score_img, threshold=0.65);
```

If you want to know more about searchlight analyses, we recommend [this article](https://www.sciencedirect.com/science/article/pii/S1053811913002917?casa_token=R268PUFghLUAAAAA:-BQTf_rGHdG-0HrGdfb23nwzUzyt65oO9cZhwNGfH7DSOAx7ZQGqRwqfhTrzKO7BZWHI-lWEW5I). That said, let's continue with the last part of this tutorial: permutation testing!

## Significance tests and permutation testing
Suppose you ran your decode pipeline on twenty subjects and accordingly collected a set of twenty model performance scores (e.g., accuracy for a two-class target). Inevitably, some scores will be higher (e.g., 0.9) than others (e.g., 0.7), and some may even be at or below chance level (e.g., 0.45). In the same vein, suppose you ran your decode pipeline across the gray-matter density patterns of a 100 subjects (of which 50 are diagnosed with major depression and 50 are healthy controls) in a 10-fold cross-validation scheme, yielding tne (foldwise) model performance scores. Again, the magnitude of these scores may differ. How do we determine if these scores are, in fact, significantly higher than we'd expect by chance?

Significance tests for within-subject decoding analyses can be done in two ways. The "easy" way is to simply do a one-sample $t$-test against chance-level (e.g., 0.5 for a two-class target) on the subject-wise model performance scores. For example, suppose that our analyses have produced the following model performance scores (e.g., accuracy, averaged across folds):

```python
# acc_scores are the average (across folds) accuracy score
acc_scores = np.array([0.5, 0.65, 0.55, 0.7, 0.85, 0.65, 0.9, 0.45, 0.55, 0.50])
```

As it is reasonable to assume that subjects are independent, we can do a one-sample $t$-test against a $H_{0}$ of 0.5:

```python
from scipy.stats import ttest_1samp
tvalue, pvalue = ttest_1samp(acc_scores, popmean=0.5)
print("T-value: %.3f, p-value: %.3f" % (tvalue, pvalue))
```

If we'd use a significance level ($\alpha$) of 0.05, this would mean that our result is statistically significant — yay! This simple approach is, however, technically not a completely valid (random effects) analysis as shown by [Allefeld and colleagues (2016)](https://www.sciencedirect.com/science/article/pii/S1053811916303470?casa_token=XqSuI7NmrQMAAAAA:AKGyrgT-Z12RwzyRSzE7L5cTsezjtSu3VRSKddahRSNDsz4XltgsMp7kjnxlz5avIqsltySRQbw), because a one-sample t-test assumes a symmetric distribution centered at chance-level, but below chance-level model performance *in the population* is impossible. This is like testing commute time of people against a null-hypothesis of 0, which doesn't make sense, because you cannot have a negative commute time! Allefeld and colleagues propose a solution, which they call "prevalence inference", which involves intricate permutation testing procedures and is a bit too complicated for this course. If you ever need to evaluate decoding performance at the group-level, though, we highly recommend trying to implement this method.

    Tip: If you want to know more about prevalence inference and want to see some example code, you can check out this notebook.

That said, let's look at how we'd statistically test the results of *between-subject* analyses. We cannot use a simple one-sample $t$-test for between-subject results, because data points in between-subject analyses — usually the model performance scores associated with different folds in K-fold CV — are *not independent*, which is an important assumption of all parametric statistics (such as $t$-tests). Dependence between data points is created because the same samples may be included in different folds. As such, we need a different, non-parameteric approach to evaluate statistical significance: permutation testing.

In the previous sections of this tutorial, we worked with within-subject data from a single subject, but for this section, we'll need some between-subject data. For this, we'll use voxel-based morphometry (VBM) data from 47 subjects. We'll use these voxelwise gray matter volume patterns in the bilateral caudate to predict subject gender (male vs. female). Although theoretically meaningless, this allows us to demonstrate permutation testing.

First, let's load in and prepare the data:

```python
with open('vbm_analysis_data.npz', 'rb') as f_in:
    data = np.load(f_in)
    R, S = data['R'], data['S']

print("R shape:", R.shape)
print("S shape:", S.shape)
```

As you can see, we have data from 47 subjects with 901 voxels from the bilateral caudate.

### Getting your *observed* performance score
Before you start with anything related to permutation testing, you first need to get your performance score that you would like to get a $p$-values for. In between-subject decoding analyses, this is usually the average accuracy (or whatever other metric) across folds. So, when you would implement a 3-fold cross-validation scheme, your observed performance score would be the average of the accuracy-estimates across those 3 folds. You know what? Let's implement exactly that:

```python
pipe = make_pipeline(StandardScaler(), LogisticRegression(solver='lbfgs', class_weight='balanced'))

np.random.seed(42)  # for reproducibility

# We'll pick 3 folds
skf = StratifiedKFold(n_splits=3)

performance = []
for train_idx, test_idx in skf.split(R, S):
    R_train, R_test = R[train_idx, :], R[test_idx, :]
    S_train, S_test = S[train_idx], S[test_idx]

    pipe.fit(R_train, S_train)
    preds = pipe.predict(R_test)
    performance.append(accuracy_score(S_test, preds))

observed_acc = np.mean(performance)
print("Mean accuracy across folds = %.3f" % observed_acc)
```

Alright, not bad! 63.6% correct is higher than chance, but is it also *significantly* higher than chance? For that, we need to "simulate" a null-distribution through permutation!

### Getting your *permuted* performance distribution
The goal of permutation tests are to create a null-distribution of your observed measure, in this case: (average) classification accuracy. The null-distribution refers to the distribution of classification accuracies that would arise if you would repeat an experiment with noisy data in which there is no effect (i.e. the null-hypothesis is true).

Thus, we need to somehow repeat the above "experiment" (i.e. the analysis which yielded the observed performance of 51.1%) yet while the null-hypothesis is true (i.e. there is no effect, only noise). Well, as the term "permutation" suggest, we can simply randomly shuffly the train-labels (`S_train`) to generate random labels. Now, if we would fit the classifier on {`R_train`, and `S_train`}, then practically it would fit on noise. In fact, by random shuffling of the train-labels, we simulated the scenario in which (on average) the null-hypothesis would be true.

So, what we need to do in order to create a null-distribution of classification-accuracies is to run our original analysis (in the code cell above) many times (i.e. "repeat the experiment"), yet with random labeling of our train-samples. We expect that the mean of these null-accuracies would center around .5 (assumed chance level), but simply due to random noise, our simulated null-distribution will contain values (sometimes substantially) above and below chance level.

Anyway, look at the code-cell below. It contains exactly the same analysis as in the above code-block, yet now it is looped 100 times, and just before fitting (within every fold separately) we shuffle the train-labels to generate a random mapping between `R_train` and `S_train`. After every "experiment" (or simply "permutation") we average the permuted accuracy scores across folds and store them (in the variable `permuted_accuracies`). Now, let's run the code cell below (may take minute or two):

```python
from tqdm.notebook import tqdm

n_permutations = 100
permuted_accuracies = np.zeros(n_permutations)

# the tqdm thingie will print a nice progressbar below
for i in tqdm(range(n_permutations)):

    folds = skf.split(R, S)
    fold_accuracies = np.zeros(skf.get_n_splits())

    for ii, (train_idx, test_idx) in enumerate(folds):

        X_train, X_test = R[train_idx], R[test_idx]
        y_train, y_test = S[train_idx], S[test_idx]

        # Here we shuffle the S-train labels!!!
        np.random.shuffle(S_train)

        pipe.fit(R_train, S_train)
        preds = pipe.predict(R_test)
        fold_accuracies[ii] = accuracy_score(S_test, preds)

    permuted_accuracies[i] = np.mean(fold_accuracies)
```

Okay, we now got our simulated null-distribution values stored in the variable `permuted_accuracies`. Let's look at how the distribution actually looks like (and how the observed accuracy relates to the permuted scores):

```python
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 5))
plt.title('Permuted null-distribution')
plt.hist(permuted_accuracies)
plt.xlabel('Average accuracy across folds')
plt.ylabel('Frequency')
plt.axvline(observed_acc, c='r', ls='--')
plt.legend(['Observed score'], frameon=False)
plt.show()
```

Now, the 'formula' for the $p$-value in a permutation test is as follows: given $P$ permutations, it's the following (in pseudo-notation):

\begin{align}
p = \frac{\sum_{i=1}^{P}{(\mathrm{permuted}_{i} \geq \mathrm{observed})} + 1}{P+1}
\end{align}

where $(\mathrm{permuted}_{i} \geq \mathrm{observed})$ evaluates to 1 if true, otherwise 0. Put differently: the $p$-value, here, expresses the proportion of permutation-values that is equal to or higher than the observed value. Graphically, we can visualize the $p$-value as the area of the (simulated!) null-distribution right of the dotted red line in the figure above.

ToDo (1 point):
Calculate the $p$-value corresponding to the observed accuracy given our permuted null-distribution. Store it in a variable named <tt>pval_permutation</tt>.

```python
# calculate p-value

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo.'''
from niedu.tests.nipa.week_2 import test_perm_pval
test_perm_pval(permuted_accuracies, observed_acc, pval_permutation)
```

    ToDo (2 points): For this final ToDo, we are going to throw you in at the deep end. Instead of a classification analysis, you are going to implement a regression analysis, in which you'll try to predict 'average attractiveness' (i.e., the attractiveness ratings for our stimuli given by an external set of subjects), which we defined for you already (the variable <tt>S</tt>, below). For the patterns, you can reuse the <tt>R_4D</tt> variable. However, you need to index this with a "Frontal Orbital Cortex" ROI from the Harvard-Oxford atlas (also defined below: <tt>ho_atlas</tt>). Note, you have to define the ROI yourself (including resampling to match the resolution of the patterns).

Then, construct a pipeline with a <tt>StandardScaler</tt> and a <tt>Ridge</tt> regression model (from the <tt>linear_model</tt> module; do not use any specific arguments during initialization). Using a 4-fold cross-validation routine (using the <tt>KFold</tt> class from the <tt>model_selection</tt> module), compute the cross-validated mean squared error (either manually or using the <tt>mean_squared_error</tt> function from the <tt>metrics</tt> module) each iteration. Finally, average the four mean squared error values and store this in a variable named <tt>av_mse</tt>.

A couple of pointers:
- You need to import all (new) classes and functions yourself;
- Realize that scikit-learn uses a consistent interface for its classes, whether they are regression-related or classification-related (e.g., a regression model has largely the same methods as classification models);

Good luck!

```python
''' Implement your ToDo here. '''
S = events['average_attractiveness'].values
ho_atlas = fetch_atlas_harvard_oxford('cort-maxprob-thr50-2mm')

np.random.seed(42)

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_2 import test_regression_todo
test_regression_todo(R_4D, events, av_mse)
```

This notebook discussed the basics of decoding analyses. There are many more topics that we didn't discuss (such as regularization, hyperparameter optimalization, prevalence inference), but hopefully you can get started with implementing your own decoding analyses after having completed this notebook!


# ForestDML, ForestDRLearner, OrthoForest and CausalForest: Basic Example

We depict the performance of our `ForestDML`, `ForestDRLearner`, `OrthoForest` and `CausalForest` estimators on the same data generating process as the one used in the tutorial page of the grf package (see https://github.com/grf-labs/grf#usage-examples). This is mostly for qualitative comparison and verification purposes among our implementation of variants of Causal Forests and the implementation in the grf R package.

```python
# Helper imports
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline
```

# DGP

We use the following data generating process (DGP) from [here](https://github.com/grf-labs/grf#usage-examples):

\begin{align}
X \sim& \text{Normal}(0,\, I_{p})\\
T =& \text{Binomial}(1, .4 + .2 \cdot 1\{X[0] > 0\})\\
Y =& (X[0] \cdot 1\{X[0] > 0\}) \cdot T + X[1] + X[2] \cdot 1\{X[2] < 0\} + \epsilon, &\; \epsilon \sim \text{Normal}(0, 1)\\
\end{align}

We use $p=10$ and draw $n=2000$ samples from this DGP.

```python
import numpy as np
import scipy.special
np.random.seed(123)
n = 2000
p = 10
X = np.random.normal(size=(n, p))
true_propensity = lambda x: .4 + .2 * (x[:, 0] > 0)
true_effect = lambda x: (x[:, 0] * (x[:, 0] > 0))
true_conf = lambda x: x[:, 1] + np.clip(x[:, 2], - np.inf, 0)
T = np.random.binomial(1, true_propensity(X))
Y =  true_effect(X) * T + true_conf(X) + np.random.normal(size=(n,))
```

### Cross-Validated Forest Nuisance Models

We use forest based estimators (Gradient Boosted Forests or Random Forests) as nuisance models. For the meta-learner versions of our forest based estimators, we also use a generic forest estimator even as a final model. The hyperparameters of the forest models (e.g. number of estimators, max depth, min leaf size) is chosen via cross validation. We also choose among Gradient or Random Forests via cross validation

```python
from econml.sklearn_extensions.model_selection import GridSearchCVList
from sklearn.linear_model import Lasso, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.base import clone
from econml.sklearn_extensions.linear_model import WeightedLasso

def first_stage_reg():
    return GridSearchCVList([Lasso(),
                             RandomForestRegressor(n_estimators=100, random_state=123),
                             GradientBoostingRegressor(random_state=123)],
                             param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
                                               {'max_depth': [3, None],
                                               'min_samples_leaf': [10, 50]},
                                              {'n_estimators': [50, 100],
                                               'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

def first_stage_clf():
    return GridSearchCVList([LogisticRegression(),
                             RandomForestClassifier(n_estimators=100, random_state=123),
                             GradientBoostingClassifier(random_state=123)],
                             param_grid_list=[{'C': [0.01, .1, 1, 10, 100]},
                                              {'max_depth': [3, 5],
                                               'min_samples_leaf': [10, 50]},
                                              {'n_estimators': [50, 100],
                                               'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

def final_stage():
    return GridSearchCVList([WeightedLasso(),
                             RandomForestRegressor(n_estimators=100, random_state=123)],
                             param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
                                              {'max_depth': [3, 5],
                                               'min_samples_leaf': [10, 50]}],
                             cv=5,
                             scoring='neg_mean_squared_error')
```

```python
model_y = clone(first_stage_reg().fit(X, Y).best_estimator_)
model_y
```

    GradientBoostingRegressor(min_samples_leaf=30, n_estimators=50,
                              random_state=123)

```python
model_t = clone(first_stage_clf().fit(X, T).best_estimator_)
model_t
```

    RandomForestClassifier(max_depth=3, min_samples_leaf=10, random_state=123)

# DML Estimators

```python
from econml.dml import CausalForestDML
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
from sklearn.dummy import DummyRegressor, DummyClassifier

est = CausalForestDML(model_y=model_y,
                      model_t=model_t,
                      discrete_treatment=True,
                      n_crossfit_splits=3,
                      n_estimators=4000,
                      min_samples_leaf=10,
                      verbose=0,
                      min_weight_fraction_leaf=.005)
est.fit(Y, T, X=X)
```

    <econml.dml.causal_forest.CausalForestDML at 0x249557e7208>

```python
from econml.dml import NonParamDML
est2 = NonParamDML(model_y=model_y,
                   model_t=model_t,
                   n_splits=3,
                   discrete_treatment=True,
                   model_final=final_stage())
est2.fit(Y, T, X=X)
```

    <econml.dml.dml.NonParamDML at 0x249686f4da0>

```python
X_test = np.zeros((100, p))
X_test[:, 0] = np.linspace(-2, 2, 100)
```

```python
pred = est.effect(X_test)
lb, ub = est.effect_interval(X_test, alpha=0.01)
```

```python
pred2 = est2.effect(X_test)
```

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred2, label='nonparamdml')
plt.plot(X_test[:, 0], pred, label='forestdml (causal forest)')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='honestrf_ci')
plt.legend()
plt.show()
```

### First Stage Learned Models

```python
# Model T
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdl in est.models_t:
    plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdl in est2.models_t:
    plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()
plt.show()
```

```python
# Model Y
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdl in est.models_y:
    plt.plot(X_test[:, 0], mdl.predict(X_test))
plt.plot(X_test[:, 0], true_effect(X_test) * true_propensity(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdl in est2.models_y:
    plt.plot(X_test[:, 0], mdl.predict(X_test))
plt.plot(X_test[:, 0], true_effect(X_test) * true_propensity(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()
plt.show()
```

### Interpretability of CATE Model of NonParamDML with SHAP

```python
import shap
import string

feature_names = list(string.ascii_lowercase)[:X.shape[1]]
# explain the model's predictions using SHAP values
shap_values = est.shap_values(X[:100], feature_names=feature_names)
```

     96%|=================== | 96/100 [00:24<00:01]

```python
# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)
shap.force_plot(shap_values["Y0"]["T0"][0], matplotlib=True)
```

```python
shap.summary_plot(shap_values["Y0"]["T0"])
```

# DRLearner

```python
model_regression = clone(first_stage_reg().fit(np.hstack([T.reshape(-1, 1), X]), Y).best_estimator_)
model_regression
```

    GradientBoostingRegressor(min_samples_leaf=30, n_estimators=50,
                              random_state=123)

```python
from econml.drlearner import ForestDRLearner
from sklearn.dummy import DummyRegressor, DummyClassifier

est = ForestDRLearner(model_regression=model_y,
                      model_propensity=model_t,
                      n_crossfit_splits=3,
                      n_estimators=4000,
                      min_samples_leaf=10,
                      verbose=0,
                      min_weight_fraction_leaf=.005)
est.fit(Y, T, X=X)
```

    <econml.drlearner.ForestDRLearner at 0x2496cd1b860>

```python
from econml.drlearner import DRLearner
est2 = DRLearner(model_regression=model_y,
                 model_propensity=model_t,
                 model_final=final_stage(),
                 n_splits=3)
est2.fit(Y, T.reshape((-1, 1)), X=X)
```

    <econml.drlearner.DRLearner at 0x2496cd9c2e8>

```python
X_test = np.zeros((100, p))
X_test[:, 0] = np.linspace(-2, 2, 100)
```

```python
pred = est.effect(X_test)
lb, ub = est.effect_interval(X_test, alpha=0.01)
```

```python
pred2 = est2.effect(X_test)
```

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred2, label='nonparamdml')
plt.plot(X_test[:, 0], pred, label='forestdml (causal forest)')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='honestrf_ci')
plt.legend()
plt.show()
```

### First stage nuisance models

```python
# Model T
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdl in est.models_propensity:
    plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdl in est2.models_propensity:
    plt.plot(X_test[:, 0], mdl.predict_proba(X_test)[:, 1])
plt.plot(X_test[:, 0], true_propensity(X_test), '--', label='truth')
plt.legend()
plt.show()
```

```python
# Model Y
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.title('honestrf')
for mdl in est.models_regression:
    plt.plot(X_test[:, 0], mdl.predict(np.hstack([X_test, np.ones((X_test.shape[0], 1))])))
plt.plot(X_test[:, 0], true_effect(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()

plt.subplot(1, 2, 2)
plt.title('rf')
for mdl in est2.models_regression:
    plt.plot(X_test[:, 0], mdl.predict(np.hstack([X_test, np.ones((X_test.shape[0], 1))])))
plt.plot(X_test[:, 0], true_effect(X_test) + true_conf(X_test), '--', label='truth')
plt.legend()
plt.show()
```

### Interpretability of CATE Model of DRLearner with SHAP

```python
# explain the model's predictions using SHAP values
shap_values = est.shap_values(X[:100], feature_names=feature_names)
```

     98%|===================| 98/100 [00:25<00:00]

```python
# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)
shap.force_plot(shap_values["Y0"]["T0"][0], matplotlib=True)
```

```python
shap.summary_plot(shap_values["Y0"]["T0"])
```

# OrthoForest

```python
from econml.ortho_forest import DROrthoForest
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV
from econml.sklearn_extensions.linear_model import WeightedLassoCV

est3 = DROrthoForest(model_Y=Lasso(alpha=0.01),
                     propensity_model=LogisticRegression(C=1),
                     model_Y_final=WeightedLassoCV(cv=3),
                     propensity_model_final=LogisticRegressionCV(cv=3),
                     n_trees=1000, min_leaf_size=10)
est3.fit(Y, T, X=X)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   15.8s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   17.9s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   21.8s
    [Parallel(n_jobs=-1)]: Done 496 tasks      | elapsed:   27.0s
    [Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:   33.5s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:   38.6s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.3s
    [Parallel(n_jobs=-1)]: Done 208 tasks      | elapsed:    4.6s
    [Parallel(n_jobs=-1)]: Done 528 tasks      | elapsed:   12.0s
    [Parallel(n_jobs=-1)]: Done 976 tasks      | elapsed:   24.5s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:   25.0s finished

    <econml.ortho_forest.DROrthoForest at 0x24970f68780>

```python
pred3 = est3.effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   19.7s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:   25.0s finished

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.plot(X_test[:, 0], true_effect(X_test), '--')
plt.plot(X_test[:, 0], pred, label='forestdr')
plt.plot(X_test[:, 0], pred2, label='nonparamdr')
plt.plot(X_test[:, 0], pred3, label='discreteorf')
plt.fill_between(X_test[:, 0], lb, ub, alpha=.4, label='forest_dr_ci')
plt.legend()
plt.show()
```

```python

```