Starting point: https://cloud.google.com/blog/products/gcp/intro-to-text-classification-with-keras-automatically-tagging-stack-overflow-posts

```python
import pandas as pd
import numpy as np
import seaborn as sns
from sklearn import metrics
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import SGDClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
```

# Data, data, data

We have 5000 StackOverflow posts that have only one tag. We want to use the post to predict what the tag is.

```python
raw_data = pd.read_csv('../data/stackOverflow.csv')
raw_data.head(10)
```

      post
      tag

      0
      getting absolute path of directory  how do i g...
      python

      1
      css  page header/colours show up in safari but...
      css

      2
      position absolute is not within the relative p...
      css

      3
      extracting individual items resulting from a s...
      python

      4
      resizable backgrounds  ok i need to make a who...
      css

      5
      why aren t these two divs displayed how i expe...
      html

      6
      strange height size top menu  in my top menu f...
      css

      7
      what is the space between these elements   wha...
      html

      8
      javascript missing semicolon   so i am using j...
      javascript

      9
      how can we link a html button   i would like t...
      html

```python
raw_data.tag.unique()
```

    array(['python', 'css', 'html', 'javascript', 'jquery'], dtype=object)

```python
raw_data.post[0]
```

    'getting absolute path of directory  how do i get the absolute path of a directory (it has to be any directory  not the current one!) in python     i tried with os.path.listdir() but it yields only the relative one. thanks!'

The tag words probably often occur in the posts, so let's strip them out to make things a bit more challenging.

```python
data = raw_data.copy()
tags = data.tag.unique()

def tag_stripper(s):
    return ' '.join(filter((lambda w: w not in tags), s.split()))

data['post'] = data.post.apply(tag_stripper)
```

```python
data.post[0]
```

    'getting absolute path of directory how do i get the absolute path of a directory (it has to be any directory not the current one!) in i tried with os.path.listdir() but it yields only the relative one. thanks!'

### How long are the posts?

```python
post_lengths = data.post.apply(lambda s: len(s.split()))
sns.distplot(post_lengths, axlabel="number of words")
```

```python
avg_length = np.mean(post_lengths)
median_length = np.median(post_lengths)

print(f"The average length of the posts in {avg_length} words, with half the posts having less than {median_length} words.")
post_lengths.describe()
```

    The average length of the posts in 138.532 words, with half the posts having less than 105.0 words.

    count    5000.00000
    mean      138.53200
    std       139.37451
    min         8.00000
    25%        68.00000
    50%       105.00000
    75%       162.00000
    max      3019.00000
    Name: post, dtype: float64

### Split into training data and testing data

It is **crucial** that we test our classifier using data that was **not** used to build. We split the data into two chunks, the data used to train (ie. build) the classifier, and the data we will later use to test it.

```python
train_data = data[:4000]
test_data = data[4000:]
print(f"Training on {len(train_data)} rows")
print(f"Testing on {len(test_data)} rows")
```

    Training on 4000 rows
    Testing on 1000 rows

# Build a classifier!

### Convert posts to word counts

We will use a **bag of words** approach. Here we ignore what the posts mean, and we even ignore the order of the words in the post! All that matters is the number of times each word appears.

We convert the words into numbers, and the posts into vectors of word counts.

```python
vectorizer = CountVectorizer()
word_counts = vectorizer.fit_transform(train_data.post)
```

```python
print(f"Number of unique words in the training data: {len(vectorizer.get_feature_names())}")
print(f"first 10: {', '.join(vectorizer.get_feature_names()[:10])}")
print(f"middle 10: {', '.join(vectorizer.get_feature_names()[13000:13010])}")
```

    Number of unique words in the training data: 26878
    first 10: 00, 000, 0000, 00000, 000000, 0000000, 000000000, 00001, 0000121593475342, 000016
    middle 10: ideal, ideally, idealy, ideas, ident, identical, identically, identifications, identified, identifier

Lets look at the first post.

```python
print(f" tag: {train_data.tag.tolist()[0]}")
print(f"post: {train_data.post.tolist()[0]}")
```

     tag: python
    post: getting absolute path of directory how do i get the absolute path of a directory (it has to be any directory not the current one!) in i tried with os.path.listdir() but it yields only the relative one. thanks!

```python
word_counts[0]
```

    <1x26878 sparse matrix of type '<class 'numpy.int64'>'
    	with 28 stored elements in Compressed Sparse Row format>

The first post has 28 different words. We encode it as a sparse vector rather than one that's almost all zeros.

```python
print(word_counts[0])
```

      (0, 23915)	1
      (0, 20271)	1
      (0, 17659)	1
      (0, 26612)	1
      (0, 5930)	1
      (0, 15095)	1
      (0, 17903)	1
      (0, 6946)	2
      (0, 26169)	1
      (0, 24558)	1
      (0, 13266)	1
      (0, 17627)	2
      (0, 7983)	1
      (0, 17186)	1
      (0, 4309)	1
      (0, 5159)	1
      (0, 24237)	1
      (0, 12420)	1
      (0, 13925)	2
      (0, 23923)	3
      (0, 11721)	1
      (0, 9133)	1
      (0, 12803)	1
      (0, 8914)	3
      (0, 17515)	2
      (0, 18367)	3
      (0, 3495)	2
      (0, 11901)	1

```python
vectorizer.get_feature_names()[18367]
```

    'path'

### Naively classifying

Our classifier will have two steps. First we convert the data into word counts as above. Then we use these word counts in a Naive Bayes classifier.

This is pretty much the simplest classifier we can use. It has few parameters but makes lots of assumptions about the data that are totally wrong.

For example, it assumes that the words appear independently, ie one word appearing in a post doesn't change the likelihood of one of the other words appearing.

Since our posts are sentences, this is really wrong.

```python
classifier = Pipeline([
    ('vect', CountVectorizer()),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
```

    Pipeline(memory=None,
         steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',
            dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
            lowercase=True, max_df=1.0, max_features=None, min_df=1,
            ngram_range=(1, 1), preprocessor=None, stop_words=None,
            strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
            tokenizer=None, vocabulary=None)), ('clf', MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))])

The proportion of predictions that you get right is called the **accuracy score**.

```python
predictions = classifier.predict(test_data.post)
accuracy_4000 = metrics.accuracy_score(test_data.tag, predictions)
print(f"Accuracy score: {accuracy_4000 * 100:.0f}%")
print("Not terrible!")
```

    Accuracy score: 73%
    Not terrible!

A **confusion matrix** shows us how many things got classified as other things.

```python
tags = test_data.tag.unique()
row_index = pd.MultiIndex.from_tuples([('actual', x) for x in tags])
col_index = pd.MultiIndex.from_tuples([('predicted', x) for x in tags])
confusion_matrix = metrics.confusion_matrix(test_data.tag, predictions, labels=tags)

df = pd.DataFrame(confusion_matrix, index=row_index, columns=col_index)
df['Count'] = df[df.columns].sum(axis=1)
s = df.sum()
s.name = ('actual', "Count")
df = df.append(s)

confusion_matrix_4000 = df
confusion_matrix_4000
```

      predicted
      Count

      python
      jquery
      javascript
      css
      html

      actual
      python
      184
      0
      4
      2
      0
      190

      jquery
      5
      133
      37
      4
      16
      195

      javascript
      23
      28
      138
      6
      13
      208

      css
      3
      8
      6
      162
      23
      202

      html
      19
      24
      15
      35
      112
      205

      Count
      234
      193
      200
      209
      164
      1000

Three other important measures of how good a classifier is are **precision**, **recall** and **f1-score**. Values of one are good, zero are bad.

\begin{align}
\text{precision of tag} & = \frac {\text {number with that tag guessed correctly}}
                                        {\text{total number of times we guessed that tag}} \\
                             & = \text {how much of our guesses of that tag were correct} \\
                             \\
\text{recall of tag} &= \frac {\text {number with that tag guessed correctly}}
                                        {\text{number with that tag}} \\
                          &= \text {how much of that tag we guessed correctly} \\
                          \\
\text{f1-score} &= \text{combination of precision and recall} \\
\end{align}

```python
print(metrics.classification_report(y_true=test_data.tag, y_pred=predictions))
```

                  precision    recall  f1-score   support

             css       0.78      0.80      0.79       202
            html       0.68      0.55      0.61       205
      javascript       0.69      0.66      0.68       208
          jquery       0.69      0.68      0.69       195
          python       0.79      0.97      0.87       190

       micro avg       0.73      0.73      0.73      1000
       macro avg       0.72      0.73      0.73      1000
    weighted avg       0.72      0.73      0.72      1000

### What if we don't have much data?

```python
train_data = data[:100]
print(f"Training on {len(train_data)} rows")
print(f"Testing on {len(test_data)} rows")
```

    Training on 100 rows
    Testing on 1000 rows

```python
classifier = Pipeline([
    ('vect', CountVectorizer()),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
print(f"Accuracy score for 4000 training posts: {accuracy_4000 * 100:.0f}%")
predictions = classifier.predict(test_data.post)
accuracy_100 = metrics.accuracy_score(test_data.tag, predictions)
print(f"Accuracy score for 100 training posts: {accuracy_100 * 100:.0f}%")
```

    Accuracy score for 4000 training posts: 73%
    Accuracy score for 100 training posts: 56%

```python
print("If we guessed randomly we'd have a 1 / 5 = 20% chance of being right.")
print(f"With very little data we boosted that to {accuracy_100 * 100:.0f}%.")
```

    If we guessed randomly we'd have a 1 / 5 = 20% chance of being right.
    With very little data we boosted that to 56%.

# Tweak things

```python
train_data = data[:4000]
test_data = data[4000:]
```

```python
print(f"Original accuracy: {accuracy_4000 * 100:.0f}%")
```

    Original accuracy: 73%

### Original Naive Bayes

### Normalize word counts

If a document is really long it will in general have higher word counts, so it might seem (to the classifier) to be more likely to have a given tag (Think of just repeating the text in a post).

To counter this we usually normalize the counts by the length of the document in some fashion.

Also, if a word occurs in most of the documents, it probably doesn't have much information value. So we want to reduce its score.

A common way to do both of these is to calculate the Term Frequency Inverse Document Frequency (TFIDF).

```python
classifier = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"Accuracy with TDIDF: {accuracy * 100:.0f}%")
```

    Accuracy with TDIDF: 74%

### Stop words

**Stop words** are words that have no information value, such as `the`, `a`, or `or`. Removing them from our data might improve things.

```python
classifier = Pipeline([
    ('vect', CountVectorizer(stop_words='english')),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"Accuracy with stop words removed: {accuracy * 100:.0f}%")
```

    Accuracy with stop words removed: 73%

### n-grams

We used word counts. What if instead we counted pairs of words? Or triples?

```python
classifier = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1,2))),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"Accuracy with 1,2 -grams: {accuracy * 100:.0f}%")
```

    Accuracy with 1,2 -grams: 76%

```python
classifier = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1,3))),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"Accuracy with 1,2,3 -grams: {accuracy * 100:.0f}%")
```

    Accuracy with 1,2,3 -grams: 76%

### All of the above?

```python
classifier = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1,2), stop_words='english')),
    ('tfidf', TfidfTransformer()),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"Accuracy with stopwords, TFIDF, and 1,2-grams: {accuracy * 100:.0f}%")
```

    Accuracy with stopwords, TFIDF, and 1,2-grams: 74%

So best option here is to use 1,2-grams.

### Try on small data set

```python
train_data = data[:100]
test_data = data[4000:]
```

```python
classifier = Pipeline([
    ('vect', CountVectorizer()),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"1-gram accuracy: {accuracy * 100:.0f}%")
```

    1-gram accuracy: 56%

```python
classifier = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1,2))),
    ('clf', MultinomialNB()),
])

classifier.fit(train_data.post, train_data.tag)
predictions = classifier.predict(test_data.post)
accuracy = metrics.accuracy_score(test_data.tag, predictions)

print(f"1,2-gram accuracy: {accuracy * 100:.0f}%")
```

    1,2-gram accuracy: 53%

## Try other classifiers

```python
train_data = data[:4000]
test_data = data[4000:]
```

```python
classifiers = [
    MultinomialNB(),
    SGDClassifier(alpha=0.1, random_state=42, max_iter=20, tol=0.3),
    RandomForestClassifier(n_estimators=100),
]
```

```python
result_cols = ["Classifier", "Accuracy"]
result_frame = pd.DataFrame(columns=result_cols)

for clf in classifiers:
    name = clf.__class__.__name__
    text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1,2))),
                         ('clf', clf),])
    text_clf.fit(train_data.post, train_data.tag)

    predicted = text_clf.predict(test_data.post)
    acc = metrics.accuracy_score(test_data.tag, predicted)
    print (f'{name} accuracy = {acc*100:.0f}%')
    acc_field = pd.DataFrame([[name, acc*100]], columns=result_cols)
    result_frame = result_frame.append(acc_field)

result_frame
```

    MultinomialNB accuracy = 76%
    SGDClassifier accuracy = 80%
    RandomForestClassifier accuracy = 75%

      Classifier
      Accuracy

      0
      MultinomialNB
      76.1

      0
      SGDClassifier
      80.4

      0
      RandomForestClassifier
      75.2

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

notebooks/tutorial 1  - StackOverflow.ipynb


# 00_setup_colab.ipynb

```python
%matplotlib inline
```

# Setup Google Colab

In this script, we setup a Google Colab environment. This script will only work
when run from `Google Colab <https://colab.research.google.com/>`_). You can
skip it if you run the tutorials on your machine.

```python

```

## Change runtime to use a GPU

This tutorial is much faster when a GPU is available to run the computations.
In Google Colab you can request access to a GPU by changing the runtime type.
To do so, click the following menu options in Google Colab:

(Menu) "Runtime" -> "Change runtime type" -> "Hardware accelerator" -> "GPU".

## Download the data and install all required dependencies

Uncomment and run the following cell to download the tutorial data and
install the required dependencies.

```python
# ![ -f "vim-5-for-ccn.tar.gz" ] || gdown --id 1b0I0Ytj06m6GCmfxfNrZuyF97fDo3NZb
# ![ -d "vim-5" ] || tar xzf vim-5-for-ccn.tar.gz
# ![ -d "pycortex" ] || git clone --quiet https://github.com/gallantlab/pycortex
# !apt-get install -qq inkscape > /dev/null
# !pip install -q voxelwise_tutorials
# ![ -f "ngrok-stable-linux-amd64.zip" ] || wget -q https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
# ![ -f "ngrok" ] || unzip ngrok-stable-linux-amd64.zip
```

For the record, here is what each command does:

- Download the dataset archive
- Extract the dataset archive
- Clone Pycortex to fix some filestore issues with Colab
- Install Inkscape, to use more features from Pycortex
- Install the tutorial helper package, and all the required dependencies
- Download ngrok to create a tunnel for pycortex 3D brain viewer
- Extract the ngrok archive

Now run the following cell to set up the environment variables for the
tutorials and pycortex.

```python
import os
os.environ['VOXELWISE_TUTORIALS_DATA'] = "/content"

import cortex
cortex.db.__init__("/content/pycortex/filestore/")  # change filestore inplace
cortex.options.config['webgl']['colormaps'] = "/content/pycortex/filestore/colormaps"

import sklearn
sklearn.set_config(assume_finite=True)
```

Your Google Colab environment is now set up for the voxelwise tutorials.

```python
%reset -f
```

# 01_plot_explainable_variance.ipynb

```python
%matplotlib inline
```

# Compute the explainable variance

Before fitting any voxelwise model to fMRI responses, it is good practice to
quantify the amount of signal in the test set that can be predicted by an
encoding model. This quantity is called the *explainable variance*.

The measured signal can be decomposed into a sum of two components: the
stimulus-dependent signal and noise. If we present the same stimulus multiple
times and we record brain activity for each repetition, the stimulus-dependent
signal will be the same across repetitions while the noise will vary across
repetitions. In voxelwise modeling, the features used to model brain activity
are the same for each repetition of the stimulus. Thus, encoding models will
predict only the repeatable stimulus-dependent signal.

The stimulus-dependent signal can be estimated by taking the mean of
brain responses over repeats of the same stimulus or experiment. The variance
of the estimated stimulus-dependent signal, which we call the explainable
variance, is proportional to the maximum prediction accuracy that can be
obtained by a voxelwise encoding model in the test set.

Mathematically, let $y_i, i = 1 \dots N$ be the measured signal in
a voxel for each of the $N$ repetitions of the same stimulus and
$\bar{y} = \frac{1}{N}\sum_{i=1}^Ny_i$ the average brain response
across repetitions. For each repeat, we define the residual timeseries
between brain response and average brain response as $r_i = y_i - \bar{y}$.
The explainable variance (EV) is estimated as

\begin{align}\text{EV} = \frac{1}{N}\sum_{i=1}^N\text{Var}(y_i) - \frac{N}{N-1}\sum_{i=1}^N\text{Var}(r_i)\end{align}

In the literature, the explainable
variance is also known as the *signal power*. For more information, see these
references [1]_ [2]_ [3]_.

```python

```

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Compute the explainable variance

```python
import numpy as np
from voxelwise_tutorials.io import load_hdf5_array
```

First, we load the fMRI responses on the test set, which contains brain
responses to ten (10) repeats of the same stimulus.

```python
file_name = os.path.join(directory, 'responses', f'{subject}_responses.hdf')
Y_test = load_hdf5_array(file_name, key="Y_test")
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

Then, we compute the explainable variance for each voxel.

```python
from voxelwise_tutorials.utils import explainable_variance
ev = explainable_variance(Y_test)
print("(n_voxels,) =", ev.shape)
```

To better understand the concept of explainable variance, we can plot the
measured signal in a voxel with high explainable variance...

```python
import matplotlib.pyplot as plt

voxel_1 = np.argmax(ev)
time = np.arange(Y_test.shape[1]) * 2  # one time point every 2 seconds
plt.figure(figsize=(10, 3))
plt.plot(time, Y_test[:, :, voxel_1].T, color='C0', alpha=0.5)
plt.plot(time, Y_test[:, :, voxel_1].mean(0), color='C1', label='average')
plt.xlabel("Time (sec)")
plt.title("Voxel with large explainable variance (%.2f)" % ev[voxel_1])
plt.yticks([])
plt.legend()
plt.tight_layout()
plt.show()
```

... and in a voxel with low explainable variance.

```python
voxel_2 = np.argmin(ev)
plt.figure(figsize=(10, 3))
plt.plot(time, Y_test[:, :, voxel_2].T, color='C0', alpha=0.5)
plt.plot(time, Y_test[:, :, voxel_2].mean(0), color='C1', label='average')
plt.xlabel("Time (sec)")
plt.title("Voxel with low explainable variance (%.2f)" % ev[voxel_2])
plt.yticks([])
plt.legend()
plt.tight_layout()
plt.show()
```

We can also plot the distribution of explainable variance over voxels.

```python
plt.hist(ev, bins=np.linspace(0, 1, 100), log=True, histtype='step')
plt.xlabel("Explainable variance")
plt.ylabel("Number of voxels")
plt.title('Histogram of explainable variance')
plt.grid('on')
plt.show()
```

We see that many voxels have low explainable variance. This is
expected, since many voxels are not driven by a visual stimulus, and their
response changes over repeats of the same stimulus.
We also see that some voxels have high explainable variance (around 0.7). The
responses in these voxels are highly consistent across repetitions of the
same stimulus. Thus, they are good targets for encoding models.

## Map to subject flatmap

To better understand the distribution of explainable variance, we map the
values to the subject brain. This can be done with `pycortex
<https://gallantlab.github.io/pycortex/>`_, which can create interactive 3D
viewers to be displayed in any modern browser. ``pycortex`` can also display
flattened maps of the cortical surface to visualize the entire cortical
surface at once.

Here, we do not share the anatomical information of the subjects for privacy
concerns. Instead, we provide two mappers:

- to map the voxels to a (subject-specific) flatmap
- to map the voxels to the Freesurfer average cortical surface ("fsaverage")

The first mapper is 2D matrix of shape (n_pixels, n_voxels) that maps each
voxel to a set of pixel in a flatmap. The matrix is efficiently stored in a
``scipy`` sparse CSR matrix. The function ``plot_flatmap_from_mapper``
provides an example of how to use the mapper and visualize the flatmap.

```python
from voxelwise_tutorials.viz import plot_flatmap_from_mapper

mapper_file = os.path.join(directory, 'mappers', f'{subject}_mappers.hdf')
plot_flatmap_from_mapper(ev, mapper_file, vmin=0, vmax=0.7)
plt.show()
```

This figure is a flattened map of the cortical surface. A number of regions of
interest (ROIs) have been labeled to ease interpretation. If you have
never seen such a flatmap, we recommend taking a look at a `pycortex brain
viewer <https://www.gallantlab.org/brainviewer/Deniz2019>`_, which displays
the brain in 3D. In this viewer, press "I" to inflate the brain, "F" to
flatten the surface, and "R" to reset the view (or use the ``surface/unfold``
cursor on the right menu). Press "H" for a list of all keyboard shortcuts.
This viewer should help you understand the correspondance between the flatten
and the folded cortical surface of the brain.

On this flatmap, we can see that the explainable variance is mainly located
in the visual cortex, in early visual regions like V1, V2, V3, or in
higher-level regions like EBA, FFA or IPS. This is expected since this
dataset contains responses to a visual stimulus.

## Map to "fsaverage"

The second mapper we provide maps the voxel data to a Freesurfer
average surface ("fsaverage"), that can be used in ``pycortex``.

First, let's download the "fsaverage" surface.

```python
import cortex

surface = "fsaverage"

if not hasattr(cortex.db, surface):
    cortex.utils.download_subject(subject_id=surface,
                                  pycortex_store=cortex.db.filestore)
    cortex.db._subjects = None  # force filestore reload
    assert hasattr(cortex.db, surface)
```

Then, we load the "fsaverage" mapper. The mapper is a matrix of shape
(n_vertices, n_voxels), which maps each voxel to some vertices in the
fsaverage surface. It is stored as a sparse CSR matrix. The mapper is applied
with a dot product ``@`` (equivalent to ``np.dot``).

```python
from voxelwise_tutorials.io import load_hdf5_sparse_array
voxel_to_fsaverage = load_hdf5_sparse_array(mapper_file,
                                            key='voxel_to_fsaverage')
ev_projected = voxel_to_fsaverage @ ev
print("(n_vertices,) =", ev_projected.shape)
```

We can then create a ``Vertex`` object in ``pycortex``, containing the
projected data. This object can be used either in a ``pycortex`` interactive
3D viewer, or in a ``matplotlib`` figure showing only the flatmap.

```python
vertex = cortex.Vertex(ev_projected, surface, vmin=0, vmax=0.7, cmap='viridis')
```

To start an interactive 3D viewer in the browser, we can use the ``webshow``
function in pycortex. If you are running the notebook on Colab, you first
need to tunnel the pycortex application out of Colab. To do so, use the
following cell to start a tunnel with ``ngrok`` and to get an address where
the pycortex viewer will be made accessible.

```python
try:
    import google.colab  # noqa
    in_colab = True
except ImportError:
    in_colab = False
print(in_colab)

if in_colab:
    from IPython import get_ipython
    get_ipython().system_raw('./ngrok http 8050 &')
    plt.pause(1)

    command = """
        curl -s http://localhost:4040/api/tunnels | python3 -c \
        "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
        """
    result = get_ipython().getoutput(command, split=True)
    print("Use the following address to connect to the brain viewer:\n"
          f"{result}\n"
          "and not the one proposed by pycortex ('Open viewer: ...')\n")
```

Now you can start an interactive 3D viewer by changing ``run_webshow`` to
``True`` and running the following cell.  If you are using Colab, remember to
use the address returned by ngrok in the cell above rather than the address
returned by this cell.

```python
run_webshow = False
if run_webshow:
    cortex.webshow(vertex, open_browser=False, port=8050)
```

Alternatively, to plot a flatmap in a ``matplotlib`` figure, use the
`quickshow` function.

(This function requires Inkscape to be installed. The rest of the tutorial
does not use this function, so feel free to ignore.)

```python
from cortex.testing_utils import has_installed

fig = cortex.quickshow(vertex, colorbar_location='right',
                       with_rois=has_installed("inkscape"))
plt.show()
```

## References

.. [1] Sahani, M., & Linden, J. F. (2003). How linear are auditory cortical
   responses?. Advances in neural information processing systems, 125-132.

.. [2] Hsu, A., Borst, A., & Theunissen, F. E. (2004). Quantifying
   variability in neural responses and its application for the validation of
   model predictions. Network: Computation in Neural Systems, 15(2), 91-109.

.. [3] Schoppe, O., Harper, N. S., Willmore, B. D., King, A. J., & Schnupp,
       J. W. (2016). Measuring the performance of neural models. Frontiers in
       computational neuroscience, 10, 10.

```python
%reset -f
```

# 02_plot_wordnet_model.ipynb

```python
%matplotlib inline
```

# Fit a ridge model with wordnet features

In this example, we model the fMRI responses with semantic "wordnet" features,
manually annotated on each frame of the movie stimulus. The model is a
regularized linear regression model, known as ridge regression. Since this
model is used to predict brain activity from the stimulus, it is called a
(voxelwise) encoding model.

This example reproduces part of the analysis described in Huth et al (2012)
[1]_. See this publication for more details about the experiment, the wordnet
features, along with more results and more discussions.

*Wordnet features:* The features used in this example are semantic labels
manually annotated on each frame of the movie stimulus. The semantic labels
include nouns (such as "woman", "car", or "building") and verbs (such as
"talking", "touching", or "walking"), for a total of 1705 distinct category
labels. To interpret our model, labels can be organized in a graph of semantic
relashionship based on the `Wordnet <https://wordnet.princeton.edu/>`_ dataset.

*Summary:* We first concatenate the features with multiple temporal delays to
account for the slow hemodynamic response. We then use linear regression to fit
a predictive model of brain activity. The linear regression is regularized to
improve robustness to correlated features and to improve generalization
performance. The optimal regularization hyperparameter is selected over a
grid-search with cross-validation. Finally, the model generalization
performance is evaluated on a held-out test set, comparing the model
predictions to the corresponding ground-truth fMRI responses.

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Load the data

We first load the fMRI responses. These responses have been preprocessed as
decribed in [1]_. The data is separated into a training set ``Y_train`` and a
testing set ``Y_test``. The training set is used for fitting models, and
selecting the best models and hyperparameters. The test set is later used
to estimate the generalization performance of the selected model. The
test set contains multiple repetitions of the same experiment to estimate
an upper bound of the model prediction accuracy (cf. previous example).

```python
import numpy as np
from voxelwise_tutorials.io import load_hdf5_array

file_name = os.path.join(directory, "responses", f"{subject}_responses.hdf")
Y_train = load_hdf5_array(file_name, key="Y_train")
Y_test = load_hdf5_array(file_name, key="Y_test")

print("(n_samples_train, n_voxels) =", Y_train.shape)
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

If we repeat an experiment multiple times, part of the fMRI responses might
change. However the modeling features do not change over the repeats, so the
voxelwise encoding model will predict the same signal for each repeat. To
have an upper bound of the model prediction accuracy, we keep only the
repeatable part of the signal by averaging the test repeats.

```python
Y_test = Y_test.mean(0)

print("(n_samples_test, n_voxels) =", Y_test.shape)
```

We fill potential NaN (not-a-number) values with zeros.

```python
Y_train = np.nan_to_num(Y_train)
Y_test = np.nan_to_num(Y_test)
```

Then, we load the semantic "wordnet" features, extracted from the stimulus at
each time point. The features corresponding to the training set are noted
``X_train``, and the features corresponding to the test set are noted
``X_test``.

```python
feature_space = "wordnet"

file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
X_train = load_hdf5_array(file_name, key="X_train")
X_test = load_hdf5_array(file_name, key="X_test")

print("(n_samples_train, n_features) =", X_train.shape)
print("(n_samples_test, n_features) =", X_test.shape)
```

## Define the cross-validation scheme

To select the best hyperparameter through cross-validation, we must define a
cross-validation splitting scheme. Since fMRI time-series are autocorrelated
in time, we should preserve as much as possible the temporal correlation.
In other words, since consecutive time samples are correlated, we should not
put one time sample in the training set and the immediately following time
sample in the validation set. Thus, we define here a leave-one-run-out
cross-validation split that keeps each recording run intact.

```python
from sklearn.model_selection import check_cv
from voxelwise_tutorials.utils import generate_leave_one_run_out

# indice of first sample of each run
run_onsets = load_hdf5_array(file_name, key="run_onsets")
print(run_onsets)
```

We define a cross-validation splitter, compatible with ``scikit-learn`` API.

```python
n_samples_train = X_train.shape[0]
cv = generate_leave_one_run_out(n_samples_train, run_onsets)
cv = check_cv(cv)  # copy the cross-validation splitter into a reusable list
```

## Define the model

Now, let's define the model pipeline.

We first center the features, since we will not use an intercept. The mean
value in fMRI recording is non-informative, so each run is detrended and
demeaned independently, and we do not need to predict an intercept value in
the linear model.

However, we prefer to avoid normalizing by the standard deviation of each
feature. If the features are extracted in a consistent way from the stimulus,
their relative scale is meaningful. Normalizing them independently from each
other would remove this information. Moreover, the wordnet features are
one-hot-encoded, which means that each feature is either present (1) or not
present (0) in each sample. Normalizing one-hot-encoded features is not
recommended, since it would scale disproportionately the infrequent features.

```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler(with_mean=True, with_std=False)
```

Then we concatenate the features with multiple delays to account for the
hemodynamic response. Due to neurovascular coupling, the recorded BOLD signal
is delayed in time with respect to the stimulus onset. With different delayed
versions of the features, the linear regression model will weigh each delayed
feature with a different weight to maximize the predictions. With a sample
every 2 seconds, we typically use 4 delays [1, 2, 3, 4] to cover the
hemodynamic response peak. In the next example, we further describe this
hemodynamic response estimation.

```python
from voxelwise_tutorials.delayer import Delayer
delayer = Delayer(delays=[1, 2, 3, 4])
```

Finally, we use a ridge regression model. Ridge regression is a linear
regression with L2 regularization. The L2 regularization improves robustness
to correlated features and improves generalization performance. However, the
L2 regularization is controlled by a hyperparameter ``alpha`` that needs to
be tuned for each dataset. This regularization hyperparameter is usually
selected over a grid search with cross-validation, selecting the
hyperparameter that maximizes the predictive performances on the validation
set. More details about cross-validation can be found in the `scikit-learn
documentation
<https://scikit-learn.org/stable/modules/cross_validation.html>`_.

For computational reasons, when the number of features is larger than the
number of samples, it is more efficient to solve ridge regression using the
(equivalent) dual formulation [2]_. This dual formulation is equivalent to
kernel ridge regression with a linear kernel. Here, we have 3600 training
samples, and 1705 * 4 = 6820 features (we multiply by 4 since we use 4 time
delays), therefore it is more efficient to use kernel ridge regression.

With one target, we could directly use the pipeline in ``scikit-learn``'s
``GridSearchCV``, to select the optimal regularization hyperparameter
(``alpha``) over cross-validation. However, ``GridSearchCV`` can only
optimize a single score across all voxels (targets). Thus, in the
multiple-target case, ``GridSearchCV`` can only optimize (for example) the
mean score over targets. Here, we want to find a different optimal
hyperparameter per target/voxel, so we use the package `himalaya
<https://github.com/gallantlab/himalaya>`_ which implements a
``scikit-learn`` compatible estimator ``KernelRidgeCV``, with hyperparameter
selection independently on each target.

```python
from himalaya.kernel_ridge import KernelRidgeCV
```

``himalaya`` implements different computational backends,
including two backends that use GPU for faster computations. The two
available GPU backends are "torch_cuda" and "cupy". (Each backend is only
available if you installed the corresponding package with CUDA enabled. Check
the ``pytorch``/``cupy`` documentation for install instructions.)

Here we use the "torch_cuda" backend, but if the import fails we continue
with the default "numpy" backend. The "numpy" backend is expected to be
slower since it only uses the CPU.

```python
from himalaya.backend import set_backend
backend = set_backend("torch_cuda", on_error="warn")
print(backend)
```

To speed up model fitting on GPU, we use single precision float numbers.
(This step probably does not change significantly the performances on non-GPU
backends.)

```python
X_train = X_train.astype("float32")
X_test = X_test.astype("float32")
```

Since the scale of the regularization hyperparameter ``alpha`` is unknown, we
use a large logarithmic range, and we will check after the fit that best
hyperparameters are not all on one range edge.

```python
alphas = np.logspace(1, 20, 20)
```

We also indicate some batch sizes to limit the GPU memory.

```python
kernel_ridge_cv = KernelRidgeCV(
    alphas=alphas, cv=cv,
    solver_params=dict(n_targets_batch=500, n_alphas_batch=5,
                       n_targets_batch_refit=100))
```

Finally, we use a ``scikit-learn`` ``Pipeline`` to link the different steps
together. A ``Pipeline`` can be used as a regular estimator, calling
``pipeline.fit``, ``pipeline.predict``, etc. Using a ``Pipeline`` can be
useful to clarify the different steps, avoid cross-validation mistakes, or
automatically cache intermediate results. See the ``scikit-learn``
`documentation <https://scikit-learn.org/stable/modules/compose.html>`_ for
more information.

```python
from sklearn.pipeline import make_pipeline
pipeline = make_pipeline(
    scaler,
    delayer,
    kernel_ridge_cv,
)
```

We can display the ``scikit-learn`` pipeline with an HTML diagram.

```python
from sklearn import set_config
set_config(display='diagram')  # requires scikit-learn 0.23
pipeline
```

## Fit the model

We fit on the training set..

```python
_ = pipeline.fit(X_train, Y_train)
```

..and score on the test set. Here the scores are the $R^2$ scores, with
values in $]-\infty, 1]$. A value of $1$ means the predictions
are perfect.

Note that since ``himalaya`` is implementing multiple-targets
models, the ``score`` method differs from ``scikit-learn`` API and returns
one score per target/voxel.

```python
scores = pipeline.score(X_test, Y_test)
print("(n_voxels,) =", scores.shape)
```

If we fit the model on GPU, scores are returned on GPU using an array object
specfic to the backend we used (such as a ``torch.Tensor``). Thus, we need to
move them into ``numpy`` arrays on CPU, to be able to use them for example in
a ``matplotlib`` figure.

```python
scores = backend.to_numpy(scores)
```

## Plot the model prediction accuracy

To visualize the model prediction accuracy, we can plot it for each voxel on
a flattened surface of the brain. To do so, we use a mapper that is specific
to the each subject's brain. (Check previous example to see how to use the
mapper to Freesurfer average surface.)

```python
import matplotlib.pyplot as plt
from voxelwise_tutorials.viz import plot_flatmap_from_mapper

mapper_file = os.path.join(directory, "mappers", f"{subject}_mappers.hdf")
ax = plot_flatmap_from_mapper(scores, mapper_file, vmin=0, vmax=0.4)
plt.show()
```

We can see that the "wordnet" features successfully predict part of the
measured brain activity, with $R^2$ scores as high as 0.4. Note that
these scores are generalization scores, since they are computed on a test set
that was not used during model fitting. Since we fitted a model independently
in each voxel, we can inspect the generalization performances at the best
available spatial resolution: individual voxels.

The best-predicted voxels are located in visual semantic areas like EBA, or
FFA. This is expected since the wordnet features encode semantic information
about the visual stimulus. For more discussions about these results, we refer
the reader to the original publication [1]_.

## Plot the selected hyperparameters

Since the scale of alphas is unknown, we plot the optimal alphas selected by
the solver over cross-validation. This plot is helpful to refine the alpha
grid if the range is too small or too large.

Note that some voxels might be at the maximum regularization value in the
grid search. These are voxels where the model has no predictive power, thus
the optimal regularization parameter is large to lead to a prediction equal
to zero. We do not need to extend the alpha range for these voxels.

```python
from himalaya.viz import plot_alphas_diagnostic
best_alphas = backend.to_numpy(pipeline[-1].best_alphas_)
plot_alphas_diagnostic(best_alphas=best_alphas, alphas=alphas)
plt.show()
```

## Visualize the regression coefficients

Here, we go back to the main model on all voxels. Since our model is linear,
we can use the (primal) regression coefficients to interpret the model. The
basic intuition is that the model will use larger coefficients on features
that have more predictive power.

Since we know the meaning of each feature, we can interpret the large
regression coefficients. In the case of wordnet features, we can even build a
graph that represents the features that are linked by a semantic
relationship.

We first get the (primal) ridge regression coefficients from the fitted
model.

```python
primal_coef = pipeline[-1].get_primal_coef()
primal_coef = backend.to_numpy(primal_coef)
print("(n_delays * n_features, n_voxels) =", primal_coef.shape)
```

Here, we are only interested in the voxels with good generalization
performances. We select an arbitrary threshold of 0.05 (R^2 score).

```python
primal_coef_selection = primal_coef[:, scores > 0.05]
```

Then, we aggregate the coefficients across the different delays.

```python
# split the ridge coefficients per delays
delayer = pipeline.named_steps['delayer']
primal_coef_per_delay = delayer.reshape_by_delays(primal_coef_selection,
                                                  axis=0)
print("(n_delays, n_features, n_voxels) =", primal_coef_per_delay.shape)

# average over delays
average_coef = np.mean(primal_coef_per_delay, axis=0)
print("(n_features, n_voxels) =", average_coef.shape)
```

Even after averaging over delays, the coefficient matrix is still too large
to interpret it. Therefore, we use principal component analysis (PCA) to
reduce the dimensionality of the matrix.

```python
from sklearn.decomposition import PCA

pca = PCA(n_components=4)
pca.fit(average_coef.T)
components = pca.components_
print("(n_components, n_features) =", components.shape)
```

We can check the ratio of explained variance by each principal component.
We see that the first four components already explain a large part of the
coefficients variance.

```python
print("PCA explained variance =", pca.explained_variance_ratio_)
```

Similarly to [1]_, we correct the coefficients of features linked by a
semantic relationship. When building the wordnet features, if a frame was
labeled with `wolf`, the authors automatically added the semantically linked
categories `canine`, `carnivore`, `placental mammal`, `mamma`, `vertebrate`,
`chordate`, `organism`, and `whole`. The authors thus argue that the same
correction needs to be done on the coefficients.

```python
from voxelwise_tutorials.wordnet import load_wordnet
from voxelwise_tutorials.wordnet import correct_coefficients
_, wordnet_categories = load_wordnet(directory=directory)
components = correct_coefficients(components.T, wordnet_categories).T
components -= components.mean(axis=1)[:, None]
components /= components.std(axis=1)[:, None]
```

Finally, we plot the first principal component on the wordnet graph. In such
graph, edges indicate "is a" relationships (e.g. an `athlete` "is a"
`person`). Each marker represents a single noun (circle) or verb (square).
The area of each marker indicates the principal component magnitude, and the
color indicates the sign (red is positive, blue is negative).

```python
from voxelwise_tutorials.wordnet import plot_wordnet_graph
from voxelwise_tutorials.wordnet import apply_cmap

first_component = components[0]
node_sizes = np.abs(first_component)
node_colors = apply_cmap(first_component, vmin=-2, vmax=2, cmap='coolwarm',
                         n_colors=2)

plot_wordnet_graph(node_colors=node_colors, node_sizes=node_sizes)
plt.show()
```

According to the authors of [1]_, "this principal component distinguishes
between categories with high stimulus energy (e.g. moving objects like
`person` and `vehicle`) and those with low stimulus energy (e.g. stationary
objects like `sky` and `city`)".

In this example, because we use only a single subject and we perform a
different voxel selection, our result is slightly different than in [1]_. We
also use a different regularization parameter in each voxel, while in [1]_
all voxels had the same regularization parameter. We do not aim at
reproducing exactly the results in [1]_, but we rather describe the general
approach.

To project the principal component on the cortical surface, we first need to
use the fitted PCA to transform the primal weights of all voxels.

```python
# split the ridge coefficients per delays
primal_coef_per_delay = delayer.reshape_by_delays(primal_coef, axis=0)
print("(n_delays, n_features, n_voxels) =", primal_coef_per_delay.shape)
del primal_coef

# average over delays
average_coef = np.mean(primal_coef_per_delay, axis=0)
print("(n_features, n_voxels) =", average_coef.shape)
del primal_coef_per_delay

# transform with the fitted PCA
average_coef_transformed = pca.transform(average_coef.T).T
print("(n_components, n_voxels) =", average_coef_transformed.shape)
del average_coef

# We make sure vmin = -vmax, so that the colormap is centered on 0.
vmax = np.percentile(np.abs(average_coef_transformed), 99.9)

# plot the primal weights projected on the first principal component.
ax = plot_flatmap_from_mapper(average_coef_transformed[0], mapper_file,
                              vmin=-vmax, vmax=vmax, cmap='coolwarm')
plt.show()
```

This flatmap shows in which brain regions the model has the largest
projection on the first component. Again, this result is different from the
one in [1]_, and should only be considered as reproducing the general
approach.

Following [1]_, we also plot the next three principal components on the
wordnet graph, mapping the three vectors to RGB colors.

```python
from voxelwise_tutorials.wordnet import scale_to_rgb_cube

next_three_components = components[1:4].T
node_sizes = np.linalg.norm(next_three_components, axis=1)
node_colors = scale_to_rgb_cube(next_three_components)
print("(n_nodes, n_channels) =", node_colors.shape)

plot_wordnet_graph(node_colors=node_colors, node_sizes=node_sizes)
plt.show()
```

According to the authors of [1]_, "this graph shows that categories thought
to be semantically related (e.g. athletes and walking) are represented
similarly in the brain".

Finally, we project these principal components on the cortical surface.

```python
from voxelwise_tutorials.viz import plot_3d_flatmap_from_mapper

voxel_colors = scale_to_rgb_cube(average_coef_transformed[1:4].T, clip=3).T
print("(n_channels, n_voxels) =", voxel_colors.shape)

ax = plot_3d_flatmap_from_mapper(voxel_colors[0], voxel_colors[1],
                                 voxel_colors[2], mapper_file=mapper_file,
                                 vmin=0, vmax=1, vmin2=0, vmax2=1, vmin3=0,
                                 vmax3=1)
plt.show()
```

Again, our results are different from the ones in [1]_, for the same reasons
mentioned earlier.

## References

.. [1] Huth, A. G., Nishimoto, S., Vu, A. T., & Gallant, J. L. (2012).
   A continuous semantic space describes the representation of thousands of
   object and action categories across the human brain. Neuron, 76(6),
   1210-1224.

.. [2] Saunders, C., Gammerman, A., & Vovk, V. (1998).
   Ridge regression learning algorithm in dual variables.

```python
%reset -f
```

# 03_plot_hemodynamic_response.ipynb

```python
%matplotlib inline
```

# Visualize the hemodynamic response

In this example, we describe how the hemodynamic response function was
estimated in the previous model. We fit the same ridge model as in the previous
example, and further describe the need to delay the features in time to account
for the delayed BOLD response.

Because of the temporal dynamics of neurovascular coupling, the recorded BOLD
signal is delayed in time with respect to the stimulus. To account for this
lag, we fit encoding models on delayed features. In this way, the linear
regression model weighs each delayed feature separately and recovers the shape
of the hemodynamic response function in each voxel separately. In turn, this
method (also known as a Finite Impulse Response model, or FIR) maximizes the
model prediction accuracy. With a repetition time of 2 seconds, we typically
use 4 delays [1, 2, 3, 4] to cover the peak of the the hemodynamic response
function. However, the optimal number of delays can vary depending on the
experiment and the brain area of interest, so you should experiment with
different delays.

In this example, we show that a model without delays performs far worse than a
model with delays. We also show how to visualize the estimated hemodynamic
response function (HRF) from a model with delays.

```python

```

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Load the data

We first load the fMRI responses.

```python
import numpy as np
from voxelwise_tutorials.io import load_hdf5_array

file_name = os.path.join(directory, "responses", f"{subject}_responses.hdf")
Y_train = load_hdf5_array(file_name, key="Y_train")
Y_test = load_hdf5_array(file_name, key="Y_test")

print("(n_samples_train, n_voxels) =", Y_train.shape)
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

We average the test repeats, to remove the non-repeatable part of fMRI
responses.

```python
Y_test = Y_test.mean(0)

print("(n_samples_test, n_voxels) =", Y_test.shape)
```

We fill potential NaN (not-a-number) values with zeros.

```python
Y_train = np.nan_to_num(Y_train)
Y_test = np.nan_to_num(Y_test)
```

Then, we load the semantic "wordnet" features.

```python
feature_space = "wordnet"

file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
X_train = load_hdf5_array(file_name, key="X_train")
X_test = load_hdf5_array(file_name, key="X_test")

print("(n_samples_train, n_features) =", X_train.shape)
print("(n_samples_test, n_features) =", X_test.shape)
```

## Define the cross-validation scheme

We define the same leave-one-run-out cross-validation split as in the
previous example.

```python
from sklearn.model_selection import check_cv
from voxelwise_tutorials.utils import generate_leave_one_run_out

# indice of first sample of each run
run_onsets = load_hdf5_array(file_name, key="run_onsets")
print(run_onsets)
```

We define a cross-validation splitter, compatible with ``scikit-learn`` API.

```python
n_samples_train = X_train.shape[0]
cv = generate_leave_one_run_out(n_samples_train, run_onsets)
cv = check_cv(cv)  # copy the cross-validation splitter into a reusable list
```

## Define the model

We define the same model as in the previous example. See the previous
example for more details about the model definition.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from voxelwise_tutorials.delayer import Delayer
from himalaya.kernel_ridge import KernelRidgeCV
from himalaya.backend import set_backend
backend = set_backend("torch_cuda", on_error="warn")

X_train = X_train.astype("float32")
X_test = X_test.astype("float32")

alphas = np.logspace(1, 20, 20)

pipeline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=500, n_alphas_batch=5,
                           n_targets_batch_refit=100)),
)
```

```python
from sklearn import set_config
set_config(display='diagram')  # requires scikit-learn 0.23
pipeline
```

## Fit the model

We fit on the train set, and score on the test set.

```python
pipeline.fit(X_train, Y_train)

scores = pipeline.score(X_test, Y_test)
scores = backend.to_numpy(scores)
print("(n_voxels,) =", scores.shape)
```

## Intermission: understanding delays

To have an intuitive understanding of what we accomplish by delaying the
features before model fitting, we will simulate one voxel and a single
feature. We will then create a ``Delayer`` object (which was used in the
previous pipeline) and visualize its effect on our single feature. Let's
start by simulating the data.

```python
# number of total trs
n_trs = 50
# repetition time for the simulated data
TR = 2.0
rng = np.random.RandomState(42)
y = rng.randn(n_trs)
x = np.zeros(n_trs)
# add some arbitrary value to our feature
x[15:20] = .5
x += rng.randn(n_trs) * 0.1  # add some noise

# create a delayer object and delay the features
delayer = Delayer(delays=[0, 1, 2, 3, 4])
x_delayed = delayer.fit_transform(x[:, None])
```

In the next cell we are plotting six lines. The subplot at the top shows the
simulated BOLD response, while the other subplots show the simulated feature
at different delays. The effect of the delayer is clear: it creates multiple
copies of the original feature shifted forward in time by how many samples we
requested (in this case, from 0 to 4 samples, which correspond to 0, 2, 4, 6,
and 8 s in time with a 2 s TR).

When these delayed features are used to fit a voxelwise encoding model, the
brain response $y$ at time $t$ is simultaneously modeled by the
feature $x$ at times $t-0, t-2, t-4, t-6, t-8$. In the remaining
of this example we will see that this method improves model prediction accuracy
and it allows to account for the underlying shape of the hemodynamic response
function.

```python
import matplotlib.pyplot as plt
fig, axs = plt.subplots(6, 1, figsize=(8, 6.5), constrained_layout=True,
        sharex=True)
times = np.arange(n_trs)*TR

axs[0].plot(times, y, color="r")
axs[0].set_title("BOLD response")
for i, (ax, xx) in enumerate(zip(axs.flat[1:], x_delayed.T)):
  ax.plot(times, xx, color='k')
  ax.set_title("$x(t - {0:.0f})$ (feature delayed by {1} sample{2})".format(
      i*TR, i, "" if i == 1 else "s"))
for ax in axs.flat:
  ax.axvline(40, color='gray')
  ax.set_yticks([])
_ = axs[-1].set_xlabel("Time [s]")
plt.show()
```

## Compare with a model without delays

We define here another model without feature delays (i.e. no ``Delayer``).
Because the BOLD signal is inherently slow due to the dynamics of
neuro-vascular coupling, this model is unlikely to perform well.

```python
pipeline_no_delay = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=500, n_alphas_batch=5,
                           n_targets_batch_refit=100)),
)
pipeline_no_delay
```

We fit and score the model as the previous one.

```python
pipeline_no_delay.fit(X_train, Y_train)
scores_no_delay = pipeline_no_delay.score(X_test, Y_test)
scores_no_delay = backend.to_numpy(scores_no_delay)
print("(n_voxels,) =", scores_no_delay.shape)
```

Then, we plot the comparison of model prediction accuracies with a 2D
histogram. All ~70k voxels are represented in this histogram, where the
diagonal corresponds to identical prediction accuracy for both models. A
distibution deviating from the diagonal means that one model has better
prediction accuracy than the other.

```python
from voxelwise_tutorials.viz import plot_hist2d

ax = plot_hist2d(scores_no_delay, scores)
ax.set(
    title='Generalization R2 scores',
    xlabel='model without delays',
    ylabel='model with delays',
)
plt.show()
```

We see that the model with delays performs much better than the model without
delays. This can be seen in voxels with scores above 0. The distribution
of scores below zero is not very informative, since it corresponds to voxels
with poor predictive performance anyway, and it only shows which model is
overfitting the most.

## Visualize the HRF

We just saw that delays are necessary to model BOLD responses. Here we show
how the fitted ridge regression weights follow the hemodynamic response
function (HRF).

Fitting a kernel ridge regression results in a set of coefficients called the
"dual" coefficients $w$. These coefficients differ from the "primal"
coefficients $\beta$ obtained with a ridge regression, but the primal
coefficients can be computed from the dual coefficients using the training
features $X$:

\begin{align}\beta = X^\top w\end{align}

To better visualize the HRF, we will refit a model with more delays, but only
on a selection of voxels to speed up the computations.

```python
# pick the 10 best voxels
voxel_selection = np.argsort(scores)[-10:]

# define a pipeline with more delays
pipeline_more_delays = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[0, 1, 2, 3, 4, 5, 6]),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=500, n_alphas_batch=5,
                           n_targets_batch_refit=100)),
)

pipeline_more_delays.fit(X_train, Y_train[:, voxel_selection])

# get the (primal) ridge regression coefficients
primal_coef = pipeline_more_delays[-1].get_primal_coef()
primal_coef = backend.to_numpy(primal_coef)

# split the ridge coefficients per delays
delayer = pipeline_more_delays.named_steps['delayer']
primal_coef_per_delay = delayer.reshape_by_delays(primal_coef, axis=0)
print("(n_delays, n_features, n_voxels) =", primal_coef_per_delay.shape)

# select the feature with the largest coefficients for each voxel
feature_selection = np.argmax(np.sum(np.abs(primal_coef_per_delay), axis=0),
                              axis=0)
primal_coef_selection = primal_coef_per_delay[:, feature_selection,
                                              np.arange(len(voxel_selection))]

plt.plot(delayer.delays, primal_coef_selection)
plt.xlabel('Delays')
plt.xticks(delayer.delays)
plt.ylabel('Ridge coefficients')
plt.title(f'Largest feature for the {len(voxel_selection)} best voxels')
plt.axhline(0, color='k', linewidth=0.5)
plt.show()
```

We see that the hemodynamic response function (HRF) is captured in the model
weights. Note that in this dataset, the brain responses are recorded every
two seconds.

```python
%reset -f
```

# 04_plot_motion_energy_model.ipynb

```python
%matplotlib inline
```

# Fit a ridge model with motion energy features

In this example, we model the fMRI responses with motion-energy features
extracted from the movie stimulus. The model is a regularized linear regression
model.

This tutorial reproduces part of the analysis described in Nishimoto et al
(2011) [1]_. See this publication for more details about the experiment, the
motion-energy features, along with more results and more discussions.

*Motion-energy features:* Motion-energy features result from filtering a video
stimulus with spatio-temporal Gabor filters. A pyramid of filters is used to
compute the motion-energy features at multiple spatial and temporal scales.
Motion-energy features were introduced in [1]_.

*Summary:* As in the previous example, we first concatenate the features with
multiple delays, to account for the slow hemodynamic response. A linear
regression model then weights each delayed feature with a different weight, to
build a predictive model of BOLD activity. Again, the linear regression is
regularized to improve robustness to correlated features and to improve
generalization. The optimal regularization hyperparameter is selected
independently on each voxel over a grid-search with cross-validation. Finally,
the model generalization performance is evaluated on a held-out test set,
comparing the model predictions with the ground-truth fMRI responses.

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Load the data

We first load the fMRI responses.

```python
import numpy as np

from voxelwise_tutorials.io import load_hdf5_array

file_name = os.path.join(directory, "responses", f"{subject}_responses.hdf")
Y_train = load_hdf5_array(file_name, key="Y_train")
Y_test = load_hdf5_array(file_name, key="Y_test")

print("(n_samples_train, n_voxels) =", Y_train.shape)
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

We average the test repeats, to remove the non-repeatable part of fMRI
responses.

```python
Y_test = Y_test.mean(0)

print("(n_samples_test, n_voxels) =", Y_test.shape)
```

We fill potential NaN (not-a-number) values with zeros.

```python
Y_train = np.nan_to_num(Y_train)
Y_test = np.nan_to_num(Y_test)
```

Then we load the precomputed "motion-energy" features.

```python
feature_space = "motion_energy"
file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
X_train = load_hdf5_array(file_name, key="X_train")
X_test = load_hdf5_array(file_name, key="X_test")

print("(n_samples_train, n_features) =", X_train.shape)
print("(n_samples_test, n_features) =", X_test.shape)
```

## Define the cross-validation scheme

We define the same leave-one-run-out cross-validation split as in the
previous example.

```python
from sklearn.model_selection import check_cv
from voxelwise_tutorials.utils import generate_leave_one_run_out

# indice of first sample of each run
run_onsets = load_hdf5_array(file_name, key="run_onsets")
print(run_onsets)
```

We define a cross-validation splitter, compatible with ``scikit-learn`` API.

```python
n_samples_train = X_train.shape[0]
cv = generate_leave_one_run_out(n_samples_train, run_onsets)
cv = check_cv(cv)  # copy the cross-validation splitter into a reusable list
```

## Define the model

We define the same model as in the previous example. See the previous
example for more details about the model definition.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from voxelwise_tutorials.delayer import Delayer
from himalaya.kernel_ridge import KernelRidgeCV
from himalaya.backend import set_backend
backend = set_backend("torch_cuda", on_error="warn")

X_train = X_train.astype("float32")
X_test = X_test.astype("float32")

alphas = np.logspace(1, 20, 20)

pipeline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=500, n_alphas_batch=5,
                           n_targets_batch_refit=100)),
)
```

```python
from sklearn import set_config
set_config(display='diagram')  # requires scikit-learn 0.23
pipeline
```

## Fit the model

We fit on the train set, and score on the test set.

```python
pipeline.fit(X_train, Y_train)

scores_motion_energy = pipeline.score(X_test, Y_test)
scores_motion_energy = backend.to_numpy(scores_motion_energy)

print("(n_voxels,) =", scores_motion_energy.shape)
```

## Plot the model performances
The performances are computed using the $R^2$ scores.

```python
import matplotlib.pyplot as plt
from voxelwise_tutorials.viz import plot_flatmap_from_mapper

mapper_file = os.path.join(directory, "mappers", f"{subject}_mappers.hdf")
ax = plot_flatmap_from_mapper(scores_motion_energy, mapper_file, vmin=0,
                              vmax=0.5)
plt.show()
```

The motion-energy features lead to large generalization scores in the
early visual cortex (V1, V2, V3, ...). For more discussions about these
results, we refer the reader to the original publication [1]_.

## Compare with the wordnet model

Interestingly, the motion-energy model performs well in different brain
regions than the semantic "wordnet" model fitted in the previous example. To
compare the two models, we first need to fit again the wordnet model.

```python
feature_space = "wordnet"
file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
X_train = load_hdf5_array(file_name, key="X_train")
X_test = load_hdf5_array(file_name, key="X_test")

X_train = X_train.astype("float32")
X_test = X_test.astype("float32")
```

We can create an unfitted copy of the pipeline with the ``clone`` function,
or simply call fit again if we do not need to reuse the previous model.

```python
if False:
    from sklearn.base import clone
    pipeline_wordnet = clone(pipeline)
    pipeline_wordnet
```

```python
pipeline.fit(X_train, Y_train)
scores_wordnet = pipeline.score(X_test, Y_test)
scores_wordnet = backend.to_numpy(scores_wordnet)

ax = plot_flatmap_from_mapper(scores_wordnet, mapper_file, vmin=0,
                              vmax=0.5)
plt.show()
```

We can also plot the comparison of model prediction accuracies with a 2D
histogram. All ~70k voxels are represented in this histogram, where the
diagonal corresponds to identical prediction accuracy for both models. A
distibution deviating from the diagonal means that one model has better
predictive performance than the other.

```python
from voxelwise_tutorials.viz import plot_hist2d

ax = plot_hist2d(scores_wordnet, scores_motion_energy)
ax.set(title='Generalization R2 scores', xlabel='semantic wordnet model',
       ylabel='motion energy model')
plt.show()
```

Interestingly, the well predicted voxels are different in the two models.
To further describe these differences, we can plot both performances on the
same flatmap, using a 2D colormap.

```python
from voxelwise_tutorials.viz import plot_2d_flatmap_from_mapper

mapper_file = os.path.join(directory, "mappers", f"{subject}_mappers.hdf")
ax = plot_2d_flatmap_from_mapper(scores_wordnet, scores_motion_energy,
                                 mapper_file, vmin=0, vmax=0.25, vmin2=0,
                                 vmax2=0.5, label_1="wordnet",
                                 label_2="motion energy")
plt.show()
```

The blue regions are well predicted by the motion-energy features, the orange
regions are well predicted by the wordnet features, and the white regions are
well predicted by both feature spaces.

A large part of the visual semantic areas are not only well predicted by the
wordnet features, but also by the motion-energy features, as indicated by the
white color. Since these two features spaces encode quite different
information, two interpretations are possible. In the first interpretation,
the two feature spaces encode complementary information, and could be used
jointly to further increase the generalization performance. In the second
interpretation, both feature spaces encode the same information, because of
spurious stimulus correlations. For example, imagine that the visual stimulus
contained faces that appeared consistetly in the same portion of the visual
field. In this case, position in the visual field would be perfectly
correlated with the "face" semantic category. Thus, motion-energy features
could predict responses in face-responsive areas without encoding any
semantic information.

To better disentangle the two feature spaces, we developed a joint model
called `banded ridge regression` [2]_, which fits multiple feature spaces
simultaneously with optimal regularization for each feature space. This model
is described in the next example.

## References

.. [1] Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu,
    B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain
    activity evoked by natural movies. Current Biology, 21(19), 1641-1646.

.. [2] Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019).
    Voxelwise encoding models with non-spherical multivariate normal priors.
    Neuroimage, 197, 482-492.

```python
%reset -f
```

# 05_plot_banded_ridge_model.ipynb

```python
%matplotlib inline
```

# Fit a banded ridge model with both wordnet and motion energy features

In this example, we model the fMRI responses with a `banded ridge regression`,
with two different feature spaces: motion energy and wordnet categories.

*Banded ridge regression:* Since the relative scaling of both feature spaces is
unknown, we use two regularization hyperparameters (one per feature space) in a
model called banded ridge regression [1]_. Just like with ridge regression, we
optimize the hyperparameters over cross-validation. An efficient implementation
of this model is available in the `himalaya
<https://github.com/gallantlab/himalaya>`_ package.

*Running time:* This example is more computationally intensive than the
previous examples. With a GPU backend, model fitting takes around 6 minutes.
With a CPU backend, it can last 10 times more.

```python

```

## Path of the data directory

```python
import os
from voxelwise_tutorials.io import get_data_home
directory = os.path.join(get_data_home(), "vim-5")
print(directory)
```

```python
# modify to use another subject
subject = "S01"
```

## Load the data

As in the previous examples, we first load the fMRI responses, which are our
regression targets.

```python
import numpy as np

from voxelwise_tutorials.io import load_hdf5_array

file_name = os.path.join(directory, "responses", f"{subject}_responses.hdf")
Y_train = load_hdf5_array(file_name, key="Y_train")
Y_test = load_hdf5_array(file_name, key="Y_test")

print("(n_samples_train, n_voxels) =", Y_train.shape)
print("(n_repeats, n_samples_test, n_voxels) =", Y_test.shape)
```

We also compute the explainable variance, to exclude voxels with low
explainable variance from the fit, and speed up the model fitting.

```python
from voxelwise_tutorials.utils import explainable_variance
ev = explainable_variance(Y_test)
print("(n_voxels,) =", ev.shape)

mask = ev > 0.1
print("(n_voxels_mask,) =", ev[mask].shape)
```

We average the test repeats, to remove the non-repeatable part of fMRI
responses.

```python
Y_test = Y_test.mean(0)

print("(n_samples_test, n_voxels) =", Y_test.shape)
```

We fill potential NaN (not-a-number) values with zeros.

```python
Y_train = np.nan_to_num(Y_train)
Y_test = np.nan_to_num(Y_test)
```

And we make sure the targets are centered.

```python
Y_train -= Y_train.mean(0)
Y_test -= Y_test.mean(0)
```

Then we load both feature spaces, that are going to be used for the
linear regression model.

```python
feature_names = ["wordnet", "motion_energy"]

Xs_train = []
Xs_test = []
n_features_list = []
for feature_space in feature_names:
    file_name = os.path.join(directory, "features", f"{feature_space}.hdf")
    Xi_train = load_hdf5_array(file_name, key="X_train")
    Xi_test = load_hdf5_array(file_name, key="X_test")

    Xs_train.append(Xi_train.astype(dtype="float32"))
    Xs_test.append(Xi_test.astype(dtype="float32"))
    n_features_list.append(Xi_train.shape[1])

# concatenate the feature spaces
X_train = np.concatenate(Xs_train, 1)
X_test = np.concatenate(Xs_test, 1)

print("(n_samples_train, n_features_total) =", X_train.shape)
print("(n_samples_test, n_features_total) =", X_test.shape)
print("[n_features_wordnet, n_features_motion_energy] =", n_features_list)
```

## Define the cross-validation scheme

We define again a leave-one-run-out cross-validation split scheme.

```python
from sklearn.model_selection import check_cv
from voxelwise_tutorials.utils import generate_leave_one_run_out

# indice of first sample of each run
run_onsets = load_hdf5_array(file_name, key="run_onsets")
print(run_onsets)
```

We define a cross-validation splitter, compatible with ``scikit-learn`` API.

```python
n_samples_train = X_train.shape[0]
cv = generate_leave_one_run_out(n_samples_train, run_onsets)
cv = check_cv(cv)  # copy the cross-validation splitter into a reusable list
```

## Define the model

The model pipeline contains similar steps than the pipeline from previous
examples. We remove the mean of each feature with a ``StandardScaler``,
and add delays with a ``Delayer``.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from voxelwise_tutorials.delayer import Delayer
from himalaya.backend import set_backend
backend = set_backend("torch_cuda", on_error="warn")
```

To fit the banded ridge model, we use ``himalaya``'s
``MultipleKernelRidgeCV`` model, with a separate linear kernel per feature
space. Similarly to ``KernelRidgeCV``, the model optimizes the
hyperparameters over cross-validation. However, while ``KernelRidgeCV`` has
to optimize only one hyperparameter (``alpha``), ``MultipleKernelRidgeCV``
has to optimize ``m`` hyperparameters, where ``m`` is the number of feature
spaces (here ``m = 2``). To do so, the model implements two different
solvers, one using hyperparameter random search, and one using hyperparameter
gradient descent. For large number of targets, we recommend using the
random-search solver.

The class takes a number of common parameters during initialization, such as
``kernels``, or ``solver``. Since the solver parameters vary depending on the
solver used, they are passed as a ``solver_params`` dictionary.

```python
from himalaya.kernel_ridge import MultipleKernelRidgeCV

# Here we will use the "random_search" solver.
solver = "random_search"

# We can check its specific parameters in the function docstring:
solver_function = MultipleKernelRidgeCV.ALL_SOLVERS[solver]
print("Docstring of the function %s:" % solver_function.__name__)
print(solver_function.__doc__)
```

The hyperparameter random-search solver separates the hyperparameters into a
shared regularization ``alpha`` and a vector of positive kernel weights which
sum to one. This separation of hyperparameters allows to explore efficiently
a large grid of values for ``alpha`` for each sampled kernel weights vector.

We use *20* random-search iterations to have a reasonably fast example. To
have better results, especially for larger number of feature spaces, one
might need more iterations. (Note that there is currently no stopping
criterion in the random-search method.)

```python
n_iter = 20

alphas = np.logspace(1, 20, 20)
```

Batch parameters, used to reduce the necessary GPU memory. A larger value
will be a bit faster, but the solver might crash if it is out of memory.
Optimal values depend on the size of your dataset.

```python
n_targets_batch = 200
n_alphas_batch = 5
n_targets_batch_refit = 200
```

We put all these parameters in a dictionary ``solver_params``, and define
the main estimator ``MultipleKernelRidgeCV``.

```python
solver_params = dict(n_iter=n_iter, alphas=alphas,
                     n_targets_batch=n_targets_batch,
                     n_alphas_batch=n_alphas_batch,
                     n_targets_batch_refit=n_targets_batch_refit)

mkr_model = MultipleKernelRidgeCV(kernels="precomputed", solver=solver,
                                  solver_params=solver_params, cv=cv)
```

We need a bit more work than in previous examples before defining the full
pipeline, since the banded ridge model requires `multiple` precomputed
kernels, one for each feature space. To compute them, we use the
``ColumnKernelizer``, which can create multiple kernels from different
column of your features array. ``ColumnKernelizer`` works similarly to
``scikit-learn``'s ``ColumnTransformer``, but instead of returning a
concatenation of transformed features, it returns a stack of kernels,
as required in ``MultipleKernelRidgeCV(kernels="precomputed")``.

First, we create a different ``Kernelizer`` for each feature space.
Here we use a linear kernel for all feature spaces, but ``ColumnKernelizer``
accepts any ``Kernelizer``, or ``scikit-learn`` ``Pipeline`` ending with a
``Kernelizer``.

```python
from himalaya.kernel_ridge import Kernelizer
from sklearn import set_config
set_config(display='diagram')  # requires scikit-learn 0.23

preprocess_pipeline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    Kernelizer(kernel="linear"),
)
preprocess_pipeline
```

The column kernelizer applies a different pipeline on each selection of
features, here defined with ``slices``.

```python
from himalaya.kernel_ridge import ColumnKernelizer

# Find the start and end of each feature space in the concatenated ``X_train``.
start_and_end = np.concatenate([[0], np.cumsum(n_features_list)])
slices = [
    slice(start, end)
    for start, end in zip(start_and_end[:-1], start_and_end[1:])
]
slices
```

```python
kernelizers_tuples = [(name, preprocess_pipeline, slice_)
                      for name, slice_ in zip(feature_names, slices)]
column_kernelizer = ColumnKernelizer(kernelizers_tuples)
column_kernelizer

# (Note that ``ColumnKernelizer`` has a parameter ``n_jobs`` to parallelize
# each ``Kernelizer``, yet such parallelism does not work with GPU arrays.)
```

Then we can define the model pipeline.

```python
pipeline = make_pipeline(
    column_kernelizer,
    mkr_model,
)
pipeline
```

## Fit the model

We fit on the train set, and score on the test set.

To speed up the fit and to limit the memory peak in Colab, we only fit on
voxels with explainable variance above 0.1.

With a GPU backend, the fitting of this model takes around 6 minutes. With a
CPU backend, it can last 10 times more.

```python
pipeline.fit(X_train, Y_train[:, mask])

scores_mask = pipeline.score(X_test, Y_test[:, mask])
scores_mask = backend.to_numpy(scores_mask)
print("(n_voxels_mask,) =", scores_mask.shape)

# Then we extend the scores to all voxels, giving a score of zero to unfitted
# voxels.
n_voxels = Y_train.shape[1]
scores = np.zeros(n_voxels)
scores[mask] = scores_mask
print("(n_voxels,) =", scores.shape)
```

## Compare with a ridge model

We can compare with a baseline model, which does not use one hyperparameter
per feature space, but instead shares the same hyperparameter for all feature
spaces.

```python
from himalaya.kernel_ridge import KernelRidgeCV

pipeline_baseline = make_pipeline(
    StandardScaler(with_mean=True, with_std=False),
    Delayer(delays=[1, 2, 3, 4]),
    KernelRidgeCV(
        alphas=alphas, cv=cv,
        solver_params=dict(n_targets_batch=n_targets_batch,
                           n_alphas_batch=n_alphas_batch,
                           n_targets_batch_refit=n_targets_batch_refit)),
)
pipeline_baseline
```

```python
pipeline_baseline.fit(X_train, Y_train[:, mask])
scores_baseline_mask = pipeline_baseline.score(X_test, Y_test[:, mask])
scores_baseline_mask = backend.to_numpy(scores_baseline_mask)

# extend to unfitted voxels
n_voxels = Y_train.shape[1]
scores_baseline = np.zeros(n_voxels)
scores_baseline[mask] = scores_baseline_mask
```

Here we plot the comparison of model prediction accuracies with a 2D
histogram. All 70k voxels are represented in this histogram, where the
diagonal corresponds to identical model prediction accuracy for both models.
A distibution deviating from the diagonal means that one model has better
predictive performance than the other.

```python
import matplotlib.pyplot as plt
from voxelwise_tutorials.viz import plot_hist2d

ax = plot_hist2d(scores_baseline, scores)
ax.set(title='Generalization R2 scores', xlabel='KernelRidgeCV',
       ylabel='MultipleKernelRidgeCV')
plt.show()
```

We see that the banded ridge model (``MultipleKernelRidgeCV``) outperforms
the ridge model (``KernelRidegeCV``). Indeed, banded ridge regression is able
to find the optimal scalings of each feature space, independently on each
voxel. Banded ridge regression is thus able to perform a soft selection
between the available feature spaces, based on the cross-validation
performances.

## Plot the banded ridge split

On top of better prediction accuracy, banded ridge regression also gives a
way to disentangle the contribution of the two feature spaces. To do so, we
take the kernel weights and the ridge (dual) weights corresponding to each
feature space, and use them to compute the prediction from each feature space
separately.

\begin{align}\hat{y} = \sum_i^m \hat{y}_i = \sum_i^m \gamma_i K_i \hat{w}\end{align}

Then, we use these split predictions to compute split $\tilde{R}^2_i$
scores. These scores are corrected so that their sum is equal to the
$R^2$ score of the full prediction $\hat{y}$.

```python
from himalaya.scoring import r2_score_split

Y_test_pred_split = pipeline.predict(X_test, split=True)
split_scores_mask = r2_score_split(Y_test[:, mask], Y_test_pred_split)

print("(n_kernels, n_samples_test, n_voxels_mask) =", Y_test_pred_split.shape)
print("(n_kernels, n_voxels_mask) =", split_scores_mask.shape)

# extend to unfitted voxels
n_kernels = split_scores_mask.shape[0]
n_voxels = Y_train.shape[1]
split_scores = np.zeros((n_kernels, n_voxels))
split_scores[:, mask] = backend.to_numpy(split_scores_mask)
print("(n_kernels, n_voxels) =", split_scores.shape)
```

We can then plot the split scores on a flatmap with a 2D colormap.

```python
from voxelwise_tutorials.viz import plot_2d_flatmap_from_mapper

mapper_file = os.path.join(directory, "mappers", f"{subject}_mappers.hdf")
ax = plot_2d_flatmap_from_mapper(split_scores[0], split_scores[1],
                                 mapper_file, vmin=0, vmax=0.25, vmin2=0,
                                 vmax2=0.5, label_1=feature_names[0],
                                 label_2=feature_names[1])
plt.show()
```

The blue regions are better predicted by the motion-energy features, the
orange regions are better predicted by the wordnet features, and the white
regions are well predicted by both feature spaces.

Compared to the last figure of the previous example, we see that most white
regions have been replaced by either blue or orange regions. The banded ridge
regression disentangled the two feature spaces in voxels where both feature
spaces had good prediction accuracy (see previous example). For example,
motion-energy features predict brain activity in early visual cortex, while
wordnet features predict in semantic visual areas. For more discussions about
these results, we refer the reader to the original publication [1]_.

## References

.. [1] Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019).
    Voxelwise encoding models with non-spherical multivariate normal priors.
    Neuroimage, 197, 482-492.