# Temporal preprocessing
Preprocessing of (f)MRI data is quite a complex topic, involving techniques from signal processing (filtering), linear algebra/statistics (prewhitening, autocorrelation correction), and numerical optimization (image registration). Additionally, there is ongoing discussion about which preprocessing steps are necessary/sufficient and what preprocessing parameters are optimal. As always, this depends on your specific research question, the type and quality of your data, and the intended analysis.

In this week, we'll discuss a couple (not all!) of preprocessing steps that are common in univariate fMRI analyses across two notebooks, one discussing temporal processing (the current notebook) and one discussing spatial preprocessing (the next notebook, `spatial_preprocessing.ipynb`).

**What you'll learn**: after this lab, you'll ...

- be able to explain the influence of preprocessing on the measured effects using the t-value formula
- understand (the advantage of) temporal filtering from both the time-domain and frequency-domain
- understand the necessity of prewhitening given the assumptions of the GLM
- understand the advantage of spatial filtering (smoothing)
- understand how to handle outliers
- know how to implement the concepts above in Python

**Estimated time needed to complete**: 8-12 hours

## Introduction
As we said before, preprocessing is a topic that almost warrants its own course. Nonetheless, we'll try to show you (and let you practice with) some of the most common and important preprocessing operations. Additionally, we'll introduce the concept of the fast fourier transform, which allows us to analyze our signal in the frequency domain, which helps to understand several preprocessing steps, such as temporal filtering.

### The *t*-value formula — yet again
The previous two weeks, you have learned that, essentially, we want to find large effects (calculated as *t*-values) of our contrasts by optimizing various parts of the *t*-value formula. Conceptually, the *t*-value formula for a particular contrast ($\mathbf{c}$) can be written as:

\begin{align}
t\mathrm{-value} = \frac{\mathrm{effect}}{\mathrm{uncertainty}} = \frac{\mathrm{effect}}{\sqrt{\mathrm{noise} \cdot \mathrm{design\ variance}}} = \frac{\mathbf{c}\hat{\beta}}{\sqrt{\hat{\sigma}^{2}\mathbf{c}(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{c}^{T}}}
\end{align}

Last week you've learned that by ensuring low design variance (through *high* predictor variance and *low* predictor correlations) leads to larger normalized effects (higher *t*-values). This week, we will discuss the other term of the denominator of this formula: the noise (also called the residual variance or unexplained variance), which is defined in the *t*-value formula as follows:

\begin{align}
\mathrm{noise} = \frac{\mathrm{SSE}}{\mathrm{DF}} = \frac{\sum_{i=1}^{N}(y_{i} - \hat{y}_{i})^{2}}{N - P}
\end{align}

Through preprocessing, we aim to reduce the difference between our prediction ($\hat{\mathbf{y}}$) and our true signal ($\mathbf{y}$), thus reducing the noise-term of the formula and thereby optimizing our ("normalized") effects.

### On temporal vs. spatial preprocessing
Roughly speaking, you can divide fMRI preprocessing into two types of operations: temporal and spatial. Temporal preprocessing involves operations that filter or otherwise affect the properties of your data across the time dimension (i.e., the time series). This is, of course, only applicable to *functional* MRI data (not structural or diffusion MRI). Examples of temporal operations are high-pass filtering and slice-time correction. Spatial preprocessing involves operations that filter or otherwise affect the spatial properties of your data (such as spatial orienatation, resolution, and shape). Examples of these spatial operations are spatial filtering ("smoothing"), distortion correction, motion correction (realignment), and spatial normalization/resampling.

In this lab, we'll discuss these operations per theme: we'll start with temporal operations and then move to spatial operations. Note that this *is not* the order in which data is usually preprocessed (see page 35 of the "Handbook of functional MRI Data Anslysis" textbook).

Let's start with temporal preprocessing.

```python
# First, some imports!
import numpy as np
from numpy.linalg import inv
from nilearn.glm.first_level.hemodynamic_models import glover_hrf
from scipy.interpolate import interp1d

import matplotlib.pyplot as plt
%matplotlib inline
```

## Temporal preprocessing
In this section, we will discuss how temporal preprocessing may greatly reduce the error term of our statistics. We'll also delve into a variant of the GLM that deals with autocorrelation in our signals appropriately ("generalized least squares", or GLS).

### Slice-time correction
Slice-time correction is a temporal resampling technique that corrects for the fact that, in (most) BOLD-MRI scan sequences, volumes are acquired slice by slice. This means that each slices is acquired at a slightly different time. For example, suppose that we are acquiring an fMRI run with a TR of 2 seconds and our volumes consist of 40 slices (acquired axially, inferior → superior). In this case, the acquisition of each slice takes $\frac{TR}{N_{slice}} = \frac{2}{40} = 0.05$ seconds (let's call this `dt`). This means that, for the very first volume, the "onset" of the first slice is at 0 seconds (and lasts until 0.05 seconds), while the onset of the last slice is at 1.95 (and lasts until 2.00 seconds). In code, we can calculate these slice onsets **within a volume** using the `np.linspace(start, stop, n_steps)` function:

```python
TR = 2
n_slices = 40
slice_onsets_within_volume = np.linspace(0, TR, n_slices, endpoint=False)
# Note, this would have worked as well: np.arange(0, TR, TR / 40)

print("Length of slice onsets: %i" % slice_onsets_within_volume.size, end='\n\n')
print("Onsets of slices for first volume: %s" % (slice_onsets_within_volume,), end='\n\n')
print("Acquisition of first slice (of first volume) started at %.2f sec." % slice_onsets_within_volume[0])
print("Acquisition of last slice (of first volume) started at %.2f sec." % slice_onsets_within_volume[-1])
```

Now, let's assume we do a very simple experiment, lasting 32 seconds (i.e., 16 volumes with a TR of 2), in which we show a single stimulus at $t=0$. Furthermore, let's assume that the *entire brain* responds to this stimulus with an idealized (noiseless response). In other words, the response of each voxel will look exactly like the HRF:

```python
oversampling = 100
exp_length = 32
n_vols = 32 // TR

ideal_response = glover_hrf(tr=TR, time_length=exp_length, oversampling=TR * oversampling)
ideal_response /= ideal_response.max()
t = np.linspace(0, exp_length, ideal_response.size, endpoint=False)

plt.figure(figsize=(15, 4))
plt.plot(t, ideal_response)
plt.grid()
plt.title("Idealized response of all voxels", fontsize=25)
plt.xlabel("Time (seconds)", fontsize=20)
plt.ylabel("Activation (A.U.)", fontsize=20)
plt.show()
```

We must, however, take into accout that, even when our data would represent an idealized response, it would be sampled at different timepoints due to a difference in onsets for different slices. The first slice would for example be sampled at $[0, 2, 4, 6, ..., 30]$ seconds, while, for example, the middle slice would be sampled at $[1, 3, 5, ... 31]$ seconds. We can calculate the onset per slice **across volumes** using `np.linspace` again:

```python
# Note: this is the onset of slice number "slice_num" **across volumes**
onsets_slice_across_volumes = np.linspace(dt * slice_num, length_exp + dt * slice_num, n_vols, endpoint=False)
```

where `dt` is the time it takes per slice (i.e., $dt = \frac{TR}{N_{slice}}$). So, to get the slice times for, e.g. slice number 23, for the experiment we have so far (i.e., with a TR of 2, 40 slices, and 32 second duration), we would compute the onsets as follows:

```python
dt = TR / n_slices
slice_num = 23

# note the slice_num - 1, as Python uses 0-based indexing
start = dt * (slice_num - 1)
end = exp_length + dt * (slice_num - 1)
onsets_slice23_across_volumes = np.linspace(start, end, n_vols, endpoint=False)
print(onsets_slice23_across_volumes)
```

So, even with a noiseless response, the data will look different per slice. We'll plot this below for the first slice and the middle slice (on top of the idealized response):

```python
onsets_s0_across_volumes = np.linspace(0, exp_length, n_vols, endpoint=False)
onsets_s20_across_volumes = np.linspace(dt * 19, exp_length + dt * 19, n_vols, endpoint=False)

print("Onset acquisition of slice 0 (first 10 volumes): %s" % (onsets_s0_across_volumes[:10],))
print("Onset acquisition of slice 20 (first 10 volumes): %s" % (onsets_s20_across_volumes[:10],))

# "Fit" the interpolation
resampler = interp1d(t, ideal_response)

plt.figure(figsize=(15, 4))
plt.plot(t, ideal_response)
plt.grid()
plt.title("Resampled slicewise responses", fontsize=25)
plt.xlabel("Time (seconds)", fontsize=20)
plt.ylabel("Activation (A.U.)", fontsize=20)

for onsets in [onsets_s0_across_volumes, onsets_s20_across_volumes]:
    # Do not use this code as an example to solve the next ToDo!
    # Because it does it the other way around.
    slice_sig = resampler(onsets)  # resample to slice times! kind of like inverse slice-time correction
    plt.plot(onsets, slice_sig, marker='*', ls='None', ms=10)

plt.legend(['Idealized response', 'Actual response slice 0', 'Actual response slice 20'],
           fontsize=15)
plt.show()
```

Now, if we would ignore the fact that these signals were acquired at different times (by assuming they were all acquired at $t=0, t=2, t=4$ etc.), our model will be suboptimal for all slices (except for slice 0). We'll show this below by plotting the resampled data from  slice 0, 20, and 39 on top of the idealized response (which represents our predictor):

```python
plt.figure(figsize=(15, 4))

slice_data = np.zeros((slice_sig.size, 3))
slice_onsets_across_volumes = np.zeros((slice_sig.size, 3))
for i, slc in enumerate([0, 19, 39]):
    t_slice = np.linspace(dt * slc, exp_length + dt * slc, n_vols, endpoint=False)
    slice_resp = resampler(t_slice)
    slice_data[:, i] = slice_resp  # save for later
    slice_onsets_across_volumes[:, i] = t_slice
    plt.subplot(1, 3, i+1)
    plt.plot(t, ideal_response)
    plt.plot(onsets_s0_across_volumes, slice_resp)
    plt.title("Slice %i" % (slc + 1), fontsize=25)
    plt.grid()

    if i == 0:
        plt.ylabel("Activation (A.U.)", fontsize=20)

    if i == 1:
        plt.xlabel("Time (seconds)", fontsize=20)
        plt.legend(['Model', 'Actual response'], fontsize=12)

plt.tight_layout()
plt.show()
```

As you can see, "later" slices are quite misaligned relative to the idealized response/model (which assume that each slice is acquired at the start of the volume). We can fix this by *resampling the data from different slices to the onset times of a reference slice*. Basically, this amounts to saying: "what would the signal from slice x look like if it was acquired at the onsets of a reference slice". This process is called *slice-time correction*.

Remember the temporal resampling of our "high precision" predictors that we did in week 2? We're going to do something similar to our data, here. Basically, we are going to resample our slice-wise timesieres corresponding to a particular set of onsets (let's call these the `original_onsets`) to the set of onsets of a particular reference slice (let's call these the `desired_onsets`). Using the same `interp1d` function from week 2, we can do the following:

```python
resampler = interp1d(original_onsets, original_signal)
std_signal = resampler(desired_onsets)
```

where `original_onsets` is a numpy array with onsets of each slice **across volumes** and the `original_signal` refers to the actual signal of that voxel (here: slice). They should be the same size. The `desired_onsets` is another numpy array with, *in the case of slice-time correction*, the same size as `original_onsets`.

Below, we'll pick the first slice as the reference slice.

```python
plt.figure(figsize=(15, 8))

plt.subplot(2, 1, 1)
for i in range(slice_data.shape[1]):
    plt.plot(onsets_s0_across_volumes, slice_data[:, i])
    plt.grid()
    plt.ylabel("Activation (A.U.)", fontsize=20)
    plt.title("Original slicewise responses", fontsize=25)

# Define reference slice as the first one (with onsets [0, 2, 4, 6, etc])
desired_onsets = onsets_s0_across_volumes

plt.subplot(2, 1, 2)
for i in range(slice_data.shape[1]):

    # Resample slice-specific onsets to desired onsets
    resampler = interp1d(slice_onsets_across_volumes[:, i], slice_data[:, i], fill_value='extrapolate')
    stc_slice = resampler(desired_onsets)

    # Plot relative to the reference onsets, i.e., onsets_s0 (first slice)
    plt.plot(onsets_s0_across_volumes, stc_slice)
    plt.grid()
    plt.title("Slice-time corrected data", fontsize=25)
    plt.xlabel("Time (seconds)", fontsize=20)
    plt.ylabel("Activation (A.U.)", fontsize=20)

plt.tight_layout()
plt.show()
```

As you can see, the resampled time series resemble each other much more after slice-time correction! It's showing some "edge artifacts" which are caused by extrapolation of the time series. In actual neuroimaging software packages, higher-order resampling (such as "cubic" or "spline" resampling) are used to somewhat mitigate this. Moreover, instead of the first slice, usually the middle slice is chosen as the reference slice, which reduces the amount of resampling (and extrapolation) that has to be done, further mitigating artifacts.

ToDo (2 points):

*Note*: this is a difficult ToDo!

Below, we load in new data (<tt>slice_data</tt>) from a very short experiment (60 seconds, TR = 3). The data are from 1 voxel from every of the 32 slices. Implement slice-time correction by resampling each slice to the onsets of the middle slice (slice no. 16). Use "cubic" interpolation (by setting <tt>kind="cubic"</tt> when initializing your resampler). Please store the results in the pre-allocated array <tt>stc_data</tt>.

Tip 1: define the TR, number of volumes, length of experiment, and <tt>dt</tt>.

Tip 2: before looping over the different slices, define the onsets of the reference slice (remember: Python is 0-indexed).

Tip 3: you have to initialize your resampler again for every iteration of your loop!

Tip 4: when initializing your resampler, set <tt>fill_value='extrapolate'</tt>.
Tip 5: plot your data before and after slice-time correction to see whether it worked properly (it should show you more-or-less aligned timeseries after STC, which may still show some differences in amplitude). Not part of the ToDo, but a good way to check your implementation.

Note: this is a ToDo which requires a little more code than usual!

```python
slice_data = np.load('stc_data.npy')
print("Slice data is of shape (time x slices): %s" % (slice_data.shape,))

stc_data = np.zeros(slice_data.shape)

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
from niedu.tests.nii.week_4 import test_stc_todo
test_stc_todo(slice_data, stc_data)
```

### A short primer on the frequency domain and the Fourier transform
Now, before we'll delve into important temporal preprocessing operations, let's discuss how we can represent time series in the frequency domain using the Fourier transform.

Thus far, we've always looked at our fMRI-signal as activity that varies across **time**. In other words, we're always looking at the signal in the *time domain*. However, there is also a way to look at a signal in the *frequency domain* (also called 'spectral domain') through transforming the signal using the *Fourier transform*.

Basically, the fourier transform calculates to which degree sine waves of different frequencies are present in your signal. If a sine wave of a certain frequency (let's say 2 hertz) is (relatively) strongly present in your signal, it will have a (relatively) high *power* in the frequency domain. Thus, looking at the frequency domain of a signal can tell you something about the frequencies of the (different) sources underlying your signal.

This may sound quite abstract, so let's look at some examples.

```python
# start with importing the python packages we'll need
from niedu.utils.nii import create_sine_wave
from numpy.linalg import inv
```

Sine waves are oscillating signals that have (for our purposes) two important characteristics: their *frequency* and their *amplitude*. Frequency reflects how fast a signal is oscillating (how many cycles it completes in a given time period) and the amplitude is the (absolute) height of the peaks and troughs of the signal. To illustrate this, we generate a couple of sine-waves (with a sampling rate of 500 Hz, i.e., 500 samples per second) with different amplitudes and frequencies, which we plot below:

```python
max_time = 5
sampling_rate = 500
timepoints = np.arange(0, max_time, 1.0 / sampling_rate)

amplitudes = np.arange(1, 4)
frequencies = np.arange(1, 4)
sines = []

fig, axes = plt.subplots(3, 3, sharex=True, sharey=True, figsize=(13, 8))
for i, amp in enumerate(amplitudes):

    for ii, freq in enumerate(frequencies):
        this_ax = axes[i, ii]

        if ii == 0:
            this_ax.set_ylabel('Activity (A.U.)', fontsize=14)

        if i == 2:
            this_ax.set_xlabel('Time (seconds)', fontsize=14)

        sine = create_sine_wave(timepoints, frequency=freq, amplitude=amp)
        sines.append((sine, amp, freq))
        this_ax.plot(timepoints, sine)
        this_ax.set_xlim(0, 5)
        this_ax.set_title('Amp = %i, freq = %i' % (amp, freq), fontsize=18)
        this_ax.set_ylim(-3.5, 3.5)
        this_ax.grid()

fig.tight_layout()
```

As you can see, the signals vary in their amplitude (from 1 to 3) and their frequency (from 1 - 3). Make sure you understand these characteristics! Now, we are going to use the fast fourier transform to plot the same signals in the *frequency domain*. We're not going to use a function to compute the FFT-transformation, but we're going to use a function that computes the "power spectrum density" directly (which makes life a little bit easier): the `periodogram` function from `scipy.signal`:

```python
from scipy.signal import periodogram
```

Now, the `periodogram` function takes two arguments, the signal and the sampling frequency (the sampling rate in Hz with which you recorded the signal), and returns both the reconstructed frequencies and their associated power values. An example:

```python
freqs, power = periodogram(some_signal, 1000)  # sampling_rate = 1000 Hz
```

We'll use the `periodogram` function to plot the 9 sine-waves (from the previous plot) again, but this time in the frequency domain:

```python
fig, axes = plt.subplots(3, 3, sharex=True, sharey=True, figsize=(13, 8))

for i, ax in enumerate(axes.flatten()):
    sine, amp, freq = sines[i]
    title = 'Sine with amp = %i and freq = %i' % (amp, freq)
    freq, power = periodogram(sine, sampling_rate)
    ax.plot(freq, power)
    ax.set_xlim(0, 4)
    ax.set_xticks(np.arange(5))
    ax.set_ylim(0, 25)

    if i > 5:
        ax.set_xlabel('Frequency (Hz)', fontsize=15)

    if i % 3 == 0:
        ax.set_ylabel('Power', fontsize=15)

    ax.set_title(title, fontsize=15)
    ax.grid()

fig.tight_layout()
plt.show()
```

As you can see, the frequency domain correctly 'identifies' the amplitudes and frequencies from the signals. But the real 'power' from fourier transforms is that they can reconstruct a signal in *multiple underlying oscillatory sources*. Let's see how that works. We're going to load in a time-series recorded for 5 seconds of which we don't know the underlying oscillatory sources. First, we'll plot the signal in the time-domain:

```python
mystery_signal = np.load('mystery_signal.npy')
plt.figure(figsize=(15, 5))
plt.plot(np.arange(0, 5, 0.001), mystery_signal)
plt.title('Time domain', fontsize=25)
plt.xlim(0, 5)
plt.xlabel('Time (sec.)', fontsize=20)
plt.ylabel('Activity (A.U.)', fontsize=20)
plt.grid()
plt.show()
```

ToDo (2 points): It's hard to see which frequencies (and corresponding amplitudes) are present in this 'mystery signal'. Get the frequencies and power of the signal using the <tt>periodogram</tt> function (you have to deduce the sampling rate of the signal yourself! It is *not* the variable <tt>sampling_rate</tt> from before). Set the x-limit of the x-axis to (0, 8) (<tt>plt.xlim(0, 8)</tt>). Also, give the plot appropriate labels for the axes.

```python
# Implement your ToDo here

# YOUR CODE HERE
raise NotImplementedError()
```

Now you know that you can use visualization of the signal in the frequency domain to help you understand from which underlying frequencies your signal is built up. Unfortunately, real fMRI data is not so 'clean' as the simulated sine waves we have used here, but the frequency representation of the fMRI signal can still tell us a lot about the nature and contributions of different (noise- and signal-related) sources!

### Frequency characteristics of fMRI data
Now, we will load a (much noisier) voxel signal and the corresponding design-matrix (which has just one predictor apart from the intercept). The signal was measured with a TR of 2 seconds and contains 300 volumes (timepoints), so the duration was 600 seconds. The predictor reflects an experiment in which we showed 15 stimuli in intervals of 40 seconds (i.e., one stimulus every 40 seconds).

We'll plot both the signal ($y$) and the design-matrix ($X$; without intercept):

```python
from niedu.utils.nii import simulate_signal

onsets = np.arange(0, 600, 40)
sig, X = simulate_signal(
    onsets,
    ['stim'] * onsets.size,
    duration=600,
    TR=2,
    icept=0,
    params_canon=[10],
    std_noise=5,
    rnd_seed=29,
    phi=0.95,
    plot=False
)
X = X[:, :-1]  # trim off the temporal derivative

print("Shape of X: %s" % (X.shape,))
print("Shape of y (sig): %s" % (sig.shape,))

plt.figure(figsize=(15, 8))
plt.subplot(2, 1, 1)
plt.plot(sig)
plt.xlim(0, sig.size)
plt.title('Signal in time domain', fontsize=25)
plt.ylabel('Activity (a.u.)', fontsize=20)
plt.grid()

plt.subplot(2, 1, 2)
plt.plot(np.arange(sig.size), X[:, 1], c='tab:orange')
plt.title('Predictor in time domain', fontsize=25)
plt.xlabel('Time (TR)', fontsize=15)
plt.xlim(0, sig.size)
plt.ylabel('Activity (a.u.)', fontsize=20)
plt.ylim(-0.5, 1.5)
plt.tight_layout()
plt.grid()
plt.show()
```

    ToDo (2 points): Run linear regression using the variable <tt>X</tt> (which already contains an intercept) to explain the variable <tt>sig</tt>. Calculate the model's MSE, and store this in a variable named <tt>mse_no_filtering</tt>. Then, in the next code-cell, plot the signal and the predicted signal ($\hat{y}$) in a single figure. Give the axes sensible labels and add a legend.

**Tip** (feel free to ignore): This tutorial, you'll be asked to compute t-values, R-squared, and MSE of several models quite some times. To make your life easier, you could (but certainly don't have to!) write a function that runs, for example, linear regression and returns the R-squared, given a design (X) and signal (y). For example, this function could look like:

```python
def compute_mse(X, y):
    # you implement the code here (run lstsq, calculate yhat, etc.)
    # ...
    # and finally, after you've computed the model's MSE, return it
    return r_squared
```

If you're ambitious, you can even write a single function that calculates t-values, MSE, and R-squared. This could look something like this:

```python
def compute_all_statistics(X, y, cvec):
    # Implement everything you want to know and return it
    # ...
    return t_value, MSE, r_squared # and whatever else you've computed!
```

Doing this will save you a lot of time and may prevent you from making unneccesary mistakes (like overwriting variables, typos, etc.). Lazy programmers are the best programmers!

(Note: writing these functions is *optional*!)

```python
# Implement the linear regression part of the ToDo here:

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above linear regression ToDo. '''
from niedu.tests.nii.week_4 import test_mse_no_filtering
test_mse_no_filtering(X, sig, mse_no_filtering)
```

```python
# Implement your y/yhat plot here (this is manually graded)

# YOUR CODE HERE
raise NotImplementedError()
```

ToThink (1 point): In your plot above, you should see that the fit of your model is "off" due to some low frequency drift. Name two potential causes of drift.

YOUR ANSWER HERE

### Two approaches to temporal preprocessing

Basically, there are *two* ways to preprocess your data:
1. Manipulating the signal ($\mathbf{y}$) **directly** *before* fitting your GLM-model;
2. Including "noise predictors" in your design ($\mathbf{X}$) when fitting your model;

Often, preprocessing steps can be done both by method 1 (manipulating the signal directly) and by method 2 (including noise predictors). For example, one of the videos showed that you could apply a high-pass filter by applying a "gaussian weighted running line smoother" (the method FSL employs) *directly* on the signal (method 1) **or** you could add "low-frequency (drift) predictors" to the design matrix (method 2; in the video they used a 'discrete cosine basis set'; the SPM method). In practice, both methods often yield very similar resuls. The most important thing to understand is that both methods are trying to accomplish the same goal: reduce the noise term of the model.

First, we will discuss how temporal and spatial filtering can *directly* filter the signal (method 1) to reduce error. Later in the tutorial, we will discuss including adding outlier-predictors and motion-predictors to the design to reduce noise (method 2).

### High-pass filtering of fMRI data (option 1)
From the previous ToDo, you probably noticed that the fit of the predictor to the model was not very good. The cause for this is the slow 'drift' — a low-frequency signal — that prevents the model from a good fit. Using a high-pass filter — meaning that you *remove* the low-frequency signals and thus *pass only the high frequencies* — can, for this reason, improve the model fit. But before we go on with actually high-pass filtering the signal, let's take a look at the frequency domain representation of our voxel signal:

```python
plt.figure(figsize=(17, 5))
TR = 2
sampling_frequency = 1 / TR  # our sampling rate is 0.5, because our TR is 2 sec!
freq, power = periodogram(sig, fs=0.5)
plt.plot(freq, power)
plt.xlim(0, freq.max())
plt.xlabel('Frequency (Hz)', fontsize=15)
plt.ylabel('Power (dB)', fontsize=15)
plt.axvline(x=0.01,color='r',ls='dashed', lw=2)

plt.show()
```

In the frequency-domain plot above, you can clearly see a low-frequency drift component at frequencies approximately below 0.01 Hz (i.e., left of the dashed red line).

ToThink (1 point): Apart from the low frequency drift component around 0.01 Hz, there is also a component visible at 0.025 Hz (and its harmonics at 0.05, 0.075, 0.1, etc.). What does this component represent? Please explain (concisely).

YOUR ANSWER HERE

Now, let's get rid of that pesky low frequency drift that messes up our model! There is no guideline on how to choose the cutoff of your high-pass filter, but most recommend to use a cutoff of around 100 seconds (i.e., of 0.01 Hz). This means that any oscillation slower than 100 seconds (one cycle in 100 seconds) is removed from your signal.

Anyway, as you've seen in the videos, there are many different ways to high-pass your signal (e.g., frequency-based filtering methods vs. time-based filtering methods). Here, we demonstrate a time-based 'gaussian running line smoother', which is used in FSL. As you've seen in the videos, this high-pass filter is estimated by convolving a gaussian "kernel" with the signal (taking the element-wise product and summing the values) across time, which is schematically visualized in the image below:

One implementation of this filter is included in the scipy "ndimage" subpackage. Let's import it\*:

---
\***Note**: if you're going to restart your kernel during the lab for whatever reason, make sure to re-import this package to avoid `NameErrors` (i.e., the error that you get when you call a function that isn't imported).

```python
from scipy.ndimage import gaussian_filter
```

The `gaussian_filter` function takes two mandatory input: some kind of (n-dimensional) signal and a cutoff, "sigma", that refers to the width of the gaussian filter in standard deviations. "What? We decided to define our cutoff in seconds (or, equivalently, Hz), right?", you might think. For some reason neuroimaging packages seem to define cutoff for their temporal filters in **seconds** while more 'low-level' filter implementations (such as in scipy) define cutoffs (of gaussian filters) in **the width of the gaussial filter**, i.e., **sigma**. Fortunately, there is an easy way to convert a cutoff in seconds to a cutoff in sigma, given a particular TR (in seconds):

\begin{align}
\sigma = \frac{\mathrm{cutoff}_{sec}}{\sqrt{8\ln{2}} \cdot \mathrm{TR}_{sec}}
\end{align}

where $\ln{2}$ refers to the natural logarithm (i.e., log with base $e$).

ToDo (1 point): Suppose I acquire some fMRI data (200 volumes) with a sampling frequency of 0.25 Hz and I would like to apply a high-pass filter of 80 seconds. What sigma should I choose? Calculate sigma and store it in a variable named <tt>sigma_todo</tt>. You can use the function <tt>np.log(some_number)</tt> to evaluate the natural logarithm of a number.

```python
# Implement your ToDo here

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo.'''
from niedu.tests.nii.week_4 import test_sec2sigma
test_sec2sigma(sec=80, ans=sigma_todo)
```

Importantly, the gaussian filter does not return the filtered signal itself, but the estimated low-frequency component of the data. As such, to filter the signal, we have to subtract this low-frequency component from the original signal to get the filtered signal!

Below, we estimate the low-frequency component using the high-pass filter first and plot it together with the original signal, which shows that it accurately captures the low-frequency drift (upper plot). Then, we subtract the low-frequency component from the original signal to create the filtered signal, and plot it together with the original signal to highlight the effect of filtering (lower plot):

```python
filt = gaussian_filter(sig, 8.5)

plt.figure(figsize=(17, 10))

plt.subplot(2, 1, 1)
plt.plot(sig, lw=2)
plt.plot(filt, lw=4)
plt.xlim(0, sig.size)
plt.legend(['Original signal', 'Low-freq component'], fontsize=20)
plt.title("Estimated low-frequency component using HP-filter", fontsize=25)
plt.ylabel("Activation (A.U.)", fontsize=20)
plt.grid()

# IMPORTANT: subtract filter from signal
filt_sig = sig - filt

plt.subplot(2, 1, 2)
plt.plot(sig, lw=2)
plt.plot(filt_sig, lw=3, c='tab:green')
plt.xlim(0, sig.size)
plt.legend(['Original signal', 'Filtered signal'], fontsize=20)
plt.title("Effect of high-pass filtering", fontsize=25)
plt.xlabel("Time (TR)", fontsize=20)
plt.ylabel("Activation (A.U.)", fontsize=20)
plt.grid()

plt.tight_layout()
plt.show()
```

The signal looks much better, i.e., it doesn't display much drift anymore. But let's check this by plotting the original and filtered signal in the frequency domain:

```python
plt.figure(figsize=(17, 5))

freq, power = periodogram(sig, fs=0.5)
plt.plot(freq, power, lw=2)

freq, power = periodogram(filt_sig, fs=0.5)
plt.plot(freq, power, lw=2)

plt.xlim(0, freq.max())
plt.ylabel('Power (dB)', fontsize=15)
plt.xlabel('Frequency (Hz)', fontsize=15)
plt.title("The effect of high-pass filtering in the frequency domain", fontsize=20)
plt.legend(["Original signal", "Filtered signal"], fontsize=15)
plt.grid()

plt.show()
```

Sweet! It seems that the high-pass filtering worked as expected! But does it really improve our model fit?

ToDo (1 point): We've claimed several times that high-pass filtering improves model fit, but is that really the case in our case? To find out, fit the same design (variable <tt>X</tt>) on the filtered signal (variable <tt>filt_sig</tt>) using linear regression. Calculate MSE and store it in the variable <tt>mse_with_filter</tt>.

```python
# Implement linear regression of X on filt_sig here!

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_mse_with_filter
test_mse_with_filter(X, filt_sig, mse_with_filter)
```

ToDo (2 points)

So far, we've filtered only a single (simulated) voxel timeseries. Normally, you want to temporally filter *all* your voxels in your 4D fMRI data, of course. Below, we load in such a 4D fMRI file (<tt>data_4d</tt>), which has $50$ timepoints and ($80 \cdot 80 \cdot 44 = $) $281600$ voxels.

For this assignment, you need to apply the high-pass filter (i.e., the <tt>gaussian_filter</tt> function; use <tt>sigma=25</tt>) on each and every voxel separately, which means that you need to loop through all voxels (which amounts to three nested for-loops across all three spatial dimensions). Below, we've already loaded in the data. Now it's up to you write the loops (across spatial dimensions) to filter the signal in the inner-most loop and store it in the pre-allocated <tt>data_4d_filt</tt> variable (the loop may, if implemented correctly, take about 20 seconds!).

```python
import os.path as op
import nibabel as nib

f = op.join(op.dirname(op.abspath('')), 'week_1', 'func.nii.gz')
print("Loading data from %s ..." % f)
data_4d = nib.load(f).get_fdata()

print("Shape of the original 4D fMRI scan: %s" % (data_4d.shape,))

# Here, we pre-allocate a matrix of the same shape as data_4d, in which
# you need to store the filtered timeseries
data_4d_filt = np.zeros(data_4d.shape)

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
# Test some random indices
np.testing.assert_almost_equal(data_4d_filt[20, 20, 20, 20], -1366.2907714, decimal=3)
np.testing.assert_almost_equal(data_4d_filt[30, 30, 30, 30], -381.6953125, decimal=3)
np.testing.assert_almost_equal(data_4d_filt[40, 40, 40, 40], -269.359375, decimal=3)
```

    Note: when temporally filtering your fMRI data ($\mathbf{y}$), it is important to apply the same filter to your design matrix ($\mathbf{X}$)! This makes sure that your design matrix does not contain any "information" that is removed from the fMRI time series by the filter, anyway. Note that most neuroimaging software packages (including FSL) do this automatically.

Technically, by filtering each column (predictor) in your design matrix as well, you're orthogonalizing your design matrix with respect to your temporal high-pass filter. If you want to know more about this, check out this excellent paper.

### High-pass filtering of fMRI data (option 2)
As we've seen, high-pass filtering using a "running line smoother" is an operation that is applied to the signal directly (before model fitting) to reduce the noise term. Another way to reduce the noise term is to include *noise regressors* (also called 'nuisance variables/regressors') in the design matrix. As such, we can subdivide our design matrix into "predictors of interest" (which are included to model the task/stimuli) and "noise predictors" (which aim to model the thus-far unmodelled variance). These "noise predictors" are also sometimes called "nuisance" predictors/regressors/covariates. We can now slightly reformulate our linear regression equation by dividing our design into two components, $\mathbf{X}_{\mathrm{interest}}$ and $\mathbf{X}_{\mathrm{noise}}$:

\begin{align}
y = \mathbf{X}_{\mathrm{interest}}\beta_{\mathrm{interest}} + \mathbf{X}_{\mathrm{noise}}\beta_{\mathrm{noise}} + \epsilon
\end{align}

Importantly, the difference between $\mathbf{X}_{\mathrm{noise}}$ and $\epsilon$ is that the $\mathbf{X}_{\mathrm{noise}}$ term refers to noise-related activity that you *are able to model* while the $\epsilon$ term refers to the noise that you *can't model* (this is often called the "irreducible noise/error" term).

We can use this technique, which we'll call "nuisance regression" (which we'll discuss in more detail later), as an alternative to directly high-pass filtering the signal ($\mathbf{y}$). One example of this (with respect to high-pass filtering) is including a series of cosines with varying frequencies in your design, which have the same effect as a high-pass filter. This type of filter is called a "discrete cosine (basis) set". Basically, for any given high-pass cutoff (in hertz), the "discrete cosine transform" (DTC) will yield a set of cosine regressors that is sufficient to filter out any frequency slower than your cutoff.

Fortunately, the `nilearn` package contains a function to calculate discrete cosine sets:

```python
from nilearn.glm.first_level.design_matrix import _cosine_drift as discrete_cosine_transform
```

This function takes two arguments: `high_pass` (in hertz) and `frame_times` (an array with volume onsets). The signal from the previous examples (`sig`) was from an experiment lasting 600 seconds and with a TR of 2, so the volume onsets can be defined as follows:

```python
frame_times = np.linspace(0, 600, 300, endpoint=False)
```

    ToThink (0 points): Here, we assumed our volume onset at the start of the TR. But technically, the exact definition/computation of our "frame times" depends on a preprocessing step that we discussed previously. Which one?

Now, let's compute a discrete cosine set for a highpass cutoff of 100 seconds (i.e., 0.01 hertz):

```python
dc_set = discrete_cosine_transform(high_pass=0.01, frame_times=frame_times)
dc_set = dc_set[:, :-1]  # remove the (extra) intercept
print(sig.shape)
print(dc_set.shape)
```

As you can see, the function returns a numpy array with the same number of timepoints as our signal (300) and 12 predictors (we removed the last one, because that's an intercept). Note that it's not super important to know how, mathematically, a discrete cosine set is created; it's more important to understand the idea of adding these low-frequency cosine predictors to your design matrix in order to account for ("explain") the low-frequency parts of your data.

Let's plot the discrete cosine set that we created (note: we only plot the first 6 for clarity):

```python
plt.figure(figsize=(15, 5))
plt.plot(dc_set[:, :6], lw=3)
plt.xlim(0, sig.size)
plt.grid()
plt.title("Discrete cosine set for a high-pass filter of 0.01 Hz", fontsize=25)
plt.xlabel("Time (TRs)", fontsize=20)
plt.ylabel("Activation (A.U)", fontsize=20)
plt.show()
```

    ToDo (2  points): Add the discrete cosine set as predictors to the design (<tt>X</tt>) and store it in a new variable named <tt>X_dct</tt> (do *not* overwrite the <tt>X</tt> variable). Make sure the first two columns of <tt>X_dct</tt> are the original predictors from <tt>X</tt> followed by the DCT set. Then, run linear regression. Save the parameters ("betas") in a variable named <tt>betas_dct</tt>. Plot the predicted signal ($\hat{\mathbf{y}}$) and the signal (<tt>sig</tt>) in the same plot. Name the axis labels appropriately.

```python
""" Implement your ToDo here. """
# YOUR CODE HERE
raise NotImplementedError()
```

```python
from niedu.tests.nii.week_4 import test_dct_betas
test_dct_betas(X, dc_set, sig, betas_dct)
```

As you've seen in the previous ToDos, it doesn't really matter which strategy you choose, filtering the signal directly or adding nuisance regressors to the design: both (usually) work equally well.

## Autocorrelation and prewhitening
As you (should) have seen in the previous ToDos, the model fit increases tremendously after high-pass filtering! This surely is the most important reason why you should apply a high-pass filter. But there is another important reason: high-pass filters reduce the signal's autocorrelation!

"Sure, but why should we care about autocorrelation?", you might think? Well this has to with the estimation of the standard error of our model, i.e., $\hat{\sigma}^{2}\mathbf{c}(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{c}^{T}$. As you've seen in the videos, the Gauss-Markov theorem states that in order for OLS to yield valid estimates (including estimates of the parameters' standard errors) *the errors (residuals) have a mean of 0, have 0 covariance (i.e., are uncorrelated), and have equal variance*.

Let's go through these three assumptions step by step. We'll use the previously filtered signal for this.

### Assumption of zero-mean of the residuals
First, let's check whether the mean of the residuals is zero:

```python
b = inv(X.T @ X) @ X.T @ filt_sig
y_hat = X @ b
resids = filt_sig - y_hat
mean_resids = resids.mean()
print("Mean of residuals: %3.f" % mean_resids)
```

ToThink (1 point): What component of the design-matrix ($\mathbf{X}$) ensures that the mean of the residuals is zero? Explain (concisely) why.

YOUR ANSWER HERE

### Equal variance of the residuals
Alright, sweet — the first assumption seems valid for our data. Now, the next two assumptions — about equal variance of the residuals and no covariance between residuals — are trickier to understand and deal with. In the book (and videos), these assumptions are summarized in a single mathemtical statement: the covariance-matrix of the residuals not differ substantially from the identity-matrix ($\mathbf{I}$) scaled by the noise-term ($\hat{\sigma}^{2}$). Or, put in a formula:

\begin{align}
\mathrm{cov}[\epsilon] = \hat{\sigma}^{2}\mathbf{I}
\end{align}

This sounds difficult, so let's break it down. First off all, the covariance matrix of the residuals is always a symmetric matrix of shape $N \times N$, in which the *diagonal represents the variances* and the *off-diagonal represents the covariances*. For example, at index $[i, i]$, the value represents the variance of the residual at timepoint $i$. At index $[i, j]$, the value represents the covariance between the residuals at timepoints $i$ and $j$.

In OLS, we assume that the covariance matrix of the residuals ($\mathrm{cov}[\epsilon]$) matches the
identity-matrix ($\mathbf{I}$) times the noise-term ($\hat{\sigma}^{2}$). The identity-matrix is simply a matrix with all zeros except for the diagonal, which contains ones. For example, the identity-matrix for a residual-array of length $8$ looks like:

```python
identity_mat = np.eye(8)  # makes an "eye"dentity matrix
print(identity_mat)
```

We can also represent this visually:

```python
plt.figure(figsize=(8, 8))
plt.imshow(identity_mat, cmap='gray', aspect='auto')
plt.xlabel("Time", fontsize=15)
plt.ylabel("Time", fontsize=15)
plt.title("Assumed covariance matrix of residuals", fontsize=20)
```

Now, suppose we calculated that the noise-term of a model explaining this hypothetical signal of length $8$ equals 2.58 ($\hat{\sigma}^{2} = 2.58$). Then, OLS *assumes* that the noise stems from a covariance matrix estimated from the residuals:

```python
noise_term = 2.58
assumed_cov_resid = noise_term * identity_mat
print(assumed_cov_resid)
```

In other words, this assumption about the covariance matrix of the residuals states that the *variance across residuals (the diagonal of the matrix) should be equal* and the *covariance between residuals (the off-diagonal values of the matrix) should be 0* (in the population).

Now, we won't explicitly estimate the covariance matrix of the residuals (which is usually estimated using techniques that fall beyond the scope of this course); however, we *do* want you to understand *conceptually* how fMRI data might invalidate the assumptions about the covariance matrix of the residuals and how fMRI analyses deal with this (i.e., using prewhitening, which is explained later).

So, let's check *visually* whether the assumption of equal variance of our residuals roughly holds for our (simulated) fMRI data. Now, when we consider this assumption in the context of our fMRI data, the assumption of "equal variance of the residuals" (also called homoskedasticity) means that we assume that the "error" in the model is equally big across our timeseries data. In other words, the mis-modelling (error) should be constant over time.

Let's check this for our data:

```python
plt.figure(figsize=(15, 5))
plt.plot(resids, marker='.')
plt.xlim(0, resids.size)
plt.xlabel("Time (TR)", fontsize=15)
plt.ylabel("Activation (A.U.)", fontsize=15)
plt.title("Residuals", fontsize=20)
plt.axhline(0, ls='--', c='black')
plt.grid()
plt.show()
```

Looks quite alright! Sure, there is some variation here and there, but given that our estimates (including the residuals and their variance!) are imperfect, this suffices.

Just to give you some intuition about serious issues with homoskedasticity, check out the (hypothetical) timeseries residuals below:

```python
mfactor = np.linspace(0, 2, sig.size)
example_resids = resids * mfactor

plt.figure(figsize=(15, 5))
plt.xlim(0, sig.size)
plt.xlabel("Time (TR)", fontsize=15)
plt.ylabel("Activation (A.U.)", fontsize=15)
plt.title("An example of residuals with (problematic) unequal variance", fontsize=20)
plt.axhline(0, ls='--', c='black')
plt.plot(example_resids, marker='.')
plt.grid()
plt.show()
```

ToThink (1 point): What could cause unequal variance in the residuals of an fMRI signal, *given that autocorrelation (i.e. low-frequency components) is filtered out appropriately*? In other words, can you think of something that might cause larger (or smaller) errors across the duration of an fMRI run?

YOUR ANSWER HERE

### Zero covariance between residuals
The last assumption of zero covariance between residuals (corresponding to the assumption of all zeros on the off-diagonal elements of the covariance-matrix of the residuals) basically refers to the assumption that *there is no autocorrelation (correlation in time) in the residuals*. In other words, knowing the residual at timepoint $i$ does not tell you anything about the residual at timepoint $i+\tau$, where $\tau$ reflects a particular "lag" and can be any positive number (up to $N$). For example, the "lag 1" autocorrelation ($\tau = 1$) is the correlation between the data at timepoints $i$ and $i+1$. In OLS, we assume that there is no autocorrelation across all possible lags.

Take for example the residuals of our unfiltered signal from before, which looked like:

```python
b = inv(X.T @ X) @ X.T @ sig
resids_new = sig - X @ b

plt.figure(figsize=(15, 8))
plt.subplot(2, 1, 1)
plt.plot(resids_new, marker='.')
plt.axhline(0, ls='--', c='black')
plt.xlim(0, 200)
plt.xlabel("Time (TR)", fontsize=15)
plt.title('Residuals (containing unmodelled drift!)', fontsize=20)
plt.ylabel('Activity (a.u.)', fontsize=15)
plt.grid()
plt.show()
```

In the above plot, there is clear and strong autocorrelation in the residuals. For example, the residuals are overall getting larger across time ("drift") and it contains other low-frequency (oscillatory) patterns. As such, we *do know something about the residual at timepoint $i+1$ (and other lags) given the residual at timepoint $i$, namely that it is likely that the residual at timpoint $i+1$ is likely __lower__ than the residual at timepoint $i$*! Therefore, drift is a perfect example of something that (if not modelled) causes autocorrelation in the residuals (i.e. covariance between residuals)! In other words, autocorrelation (e.g. caused by drift) will cause the values of the covariance matrix of the residuals at the indices $[i, i+1]$ to be non-zero, violating the third assumption of Gauss-Markov's theorem!

ToDo (1  point)

We stated that autocorrelation captures the information that you have of the residual at timepoint $i+\tau$ given that you know the residual at timepoint $i$. Practically, you can compute the autocorrelation (or actually, autocovariance) for a particular lag $\tau$ by computing the covariance of the residuals with the lag-$\tau$ shifted version of itself. In general, the autocovariance for the residuals $\epsilon$ with lag $\tau$ is calculated as:

\begin{align}
\mathrm{cov}[\epsilon_{i}, \epsilon_{i+\tau}] = \frac{1}{N-\tau-1}\sum_{i=1}^{N-\tau}(\epsilon_{i}\cdot\epsilon_{i+\tau})
\end{align}

Jeanette Mumford explains how to do this quite clearly in her [video on prewhitening](https://www.youtube.com/watch?v=4VSzZKO0k_w) (around minute 10). For this ToDo, calculate the covariance (with $\tau = 1$) between the residuals (i.e., using the variable <tt>resids_new</tt>) and store this in a variable named <tt>lag1_cov</tt>.

```python
# Implement your ToDo here

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_lag1_cov
test_lag1_cov(resids_new, lag1_cov)
```

### Accounting for unequal variance and autocorrelation: prewhitening
So, in summary, if the covariance matrix of your residuals appear to significantly deviate from the identity-matrix scaled by the noise-term ($\mathrm{cov}[\epsilon] = \hat{\sigma}^{2}\mathbf{I}$) — either due to unequal variance or non-zero covariance — your estimate of the variance term of your effects ($\mathrm{var}[c\hat{\beta}]$) will be incorrect.

Unfortunately, even after high-pass filtering (which corrects for *most* but not *all* autocovariance), the covariance matrix of the residuals of fMRI timeseries usually do no conform to the Markov-Gauss assumptions of equal variance and zero covariance. Fortunately, some methods have been developed by statisticians that transform the data such that the OLS assumptions hold again. One such technique is called *prewhitening*.

Prewhitening uses an estimate of the error covariance matrix — usually denoted by $\mathbf{V}$ — to account for possible unequal variance and/or autocovariance of the residuals. The matrix $\mathbf{V}$ is an $N \times N$ matrix ($N$ referring to the number of timepoints of your signal), and  may be estimated using different techniques, with names such as "ARMA", "AR(1)", and "REML". In this course, we won't discuss these techniques and instead assume that your software of choice (FSL, AFNI, SPM) has computed an accurate estimate of $\mathbf{V}$ for you already. But if you're up for a (programming) challenge, you can do the *optional* ToDo below.

    ToDo (optional, ungraded)

The "AR(1)" method is a relatively "easy" way to estimate $\mathbf{V}$. It computes only a single parameter, the lag-1 correlation (not covariance!). Then, it assumes that the correlation decreases exponentially as a function of lag:

\begin{align}
\mathrm{autocor}_{\tau} = \phi^{\tau + 1}
\end{align}

where $\phi$ is the estimated lag-1 correlation. For example, for $\phi = 0.9$, $\mathbf{V}$ would look like:

\begin{align}
\mathbf{V} = \begin{bmatrix}
                 1.0 & 0.9 & 0.9^{2} & 0.9^{3} & \dots & 0.9^{N-1} \\
                 0.9 & 1.0 & 0.9 & 0.9^{2} & \dots & 0.9^{N-2} \\
                 0.9^{2} & 0.9 & 1.0 & 0.9 & \dots & 0.9^{N-3} \\
                 0.9^{3} & 0.9^{2} & 0.9 & 1.0 & \dots & 0.9^{N-4} \\
                 \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
                 0.9^{N-1} & 0.9^{N-2} & 0.9^{N-3} & 0.9^{N-4} & \dots & 1.0
             \end{bmatrix}
\end{align}

Compute the lag-1 correlation below and create the corresponding AR(1) matrix of the residuals (<tt>resids_new</tt>) and store this in a variable named <tt>V_ar1</tt> (do not use the <tt>toeplitz</tt> function for this).

```python
''' Implement the (optional) ToDo here.'''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the optional ToDo above. '''
from scipy.linalg import toeplitz
phi = (t0 - t0.mean()) @ (t1 - t1.mean()) / np.sqrt(np.sum((t0 - t0.mean()) ** 2) * np.sum((t1 - t1.mean()) ** 2))
V_ans = phi ** toeplitz(np.arange(resids_new.size))
np.testing.assert_array_almost_equal(V_ans, V_ar1)
print("Well done!")
```

Suppose you have a signal of 20 timepoints (an irrealistically low number, but just ignore that for now) and that you already estimated the covariance matrix of the residuals of this signal. Now, suppose you take a look at it and you notice that it looks faaaaar from the identity-matrix ($\mathbf{I}$) that we need for OLS.

For example, you might see this:

```python
N = 20
phi = 0.7
V = phi ** toeplitz(np.arange(N))
# This will increase variance over time
V[np.diag_indices_from(V)] += np.linspace(0, 1, V.shape[0])

fig, axes = plt.subplots(ncols=2, sharex=True, sharey=True, figsize=(15, 8))
axes[0].imshow(V, vmax=2, cmap='gray', aspect='auto')
axes[0].set_title("V (actual covariance matrix)", fontsize=25)
axes[0].set_xlabel('Time (volumes)', fontsize=20)
axes[0].set_ylabel('Time (volumes)', fontsize=20)

axes[1].imshow(np.eye(N), vmax=2, cmap='gray', aspect='auto')
axes[1].set_title("Identity-matrix (assumed matrix)", fontsize=25)
#axes[1].colorbar()
axes[1].set_xlabel('Time (volumes)', fontsize=20)

fig.tight_layout()
```

    ToThink (0 points): In the above cell, the <tt>phi</tt> variable controls the amount of autcorrelation (technically, it is the $\phi$ parameter of an AR(1) autocorrelation model). Try changing the value of this variable. Do you understand the way the plotted $V$ matrix is changing as a function of $\phi$?

Well, shit. We have both unequal variance (different values on the diagonal) *and* non-zero covariance (some non-zero values on the off-diagonal). So, what to do now? Well, we can use the technique of prewhitening to make sure our observed covariance matrix ($\mathbf{V}$) will be "converted" to the identity matrix! Basically, this amounts to plugging in some extra terms to formula for ordinary least squares. As you might have seen in the book/videos, the *original* OLS solution (i.e., how OLS finds the beta-parameters is as follows):

\begin{align}
\hat{\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}y
\end{align}

Now, given that we've estimated our covariance matrix of the residuals, $\mathbf{V}$, we can rewrite the OLS solution such that it prewhitens the data (and thus the covariance matrix of the residuals will approximate $\hat{\sigma}^{2}\mathbf{I}$) as follows:

\begin{align}
\hat{\beta} = (\mathbf{X}^{T}\mathbf{V}^{-1}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{V}^{-1}y
\end{align}

Then, accordingly, the standard-error of any contrast of the estimated beta-parameters becomes:

\begin{align}
SE_{\mathbf{c}\hat{\beta}} = \sqrt{\hat{\sigma}^{2} \cdot \mathbf{c}(\mathbf{X}^{T}\mathbf{V}^{-1}\mathbf{X})^{-1}\mathbf{c}^{T}}
\end{align}

This "modification" of OLS is also called "generalized least squares" (GLS) and is central to most univariate fMRI analyses! You *don't* have to understand how this works mathematically; again, you should only understand *why* prewhitening makes sure that our data behaves according to the assumptions of the Gauss-Markov theorem.

(Fortunately for us, there is usually an option to 'turn on' prewhitening in existing software packages, so we don't have to do it ourselves. But it is important to actually turn it on whenever you want to meaningfully and in an unbiased way interpret your statistics in fMRI analyses!)

    ToDo (1 point): Given the target signal (<tt>some_sig</tt>), design-matrix (<tt>some_X</tt>), and the (hypothetical) covariance-matrix of the residuals from before (the variable <tt>V</tt>), calculate the beta-parameters using the prewhitened version of OLS (i.e., 'generalized least squares'; the formula above). Also, calculate the $t$-value of the contrast <tt>[0, 1]</tt> given the appropriate (GLS) computation of the standard-error. Store your results in the variable <tt>betas_gls</tt> and <tt>tval_gls</tt>, respectively.

```python
# Implement your ToDo here!
some_sig = sig[:20]  # y
some_X = X[:20, :]   # X
c_vec = np.array([0, 1])  # the contrast you should use

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the optional ToDo above '''
from niedu.tests.nii.week_4 import test_gls_todo
test_gls_todo(some_sig, some_X, V, c_vec, betas_gls, tval_gls)
```

    ToDo (optional! 0 points)

When it comes to estimating parameters from data with unequal (co)variance, OLS actually still gives you unbiased parameters: on average, they will be correct. However, OLS is not the estimator with least variance anymore, meaning that it is less "precise" (it is not the "Best Linear Unbiased Estimator" anymore; still unbiased, but not the "best"). In fact, with unequal (co)variance, GLS is the best unbiased linear estimator. A good way to build intuition about this is to iteratively generate data with known parameters (the "true betas", $\beta$, "sigma squared", $\sigma^{2}$, and $V$) and to estimate the parameters back from the generated data. Then, you can plot the histograms of the parameters and you'll see that, on average, the estimated parameters are the same as the true parameters.

Below, we set up such a "simulation" loop for you. We define the true parameters (<tt>true_betas</tt>, <tt>siqsq</tt>, <tt>V</tt>). Now, if you're up to the challenge, complete the loop by:
1. Generating some random design matrix (e.g., using <tt>np.random.normal</tt> of size (N, 1));
2. Stack an intercept
3. Generate correlated noise using the <tt>np.random.multivariate_normal</tt> function (with <tt>cov=V</tt>);
4. Generate the data using the formula $X\beta + \mathrm{noise}$;
5. Estimate the OLS parameters, and store in <tt>betas_ols</tt>;
6. Estimate the GLS parameters, and store in <tt>betas_gls</tt>;
7. In the next cell, plot both parameters (<tt>betas_ols[:, 1]</tt> and <tt>betas_gls[:, 1]</tt>) as histograms

Do the histograms look like you expected? Also, try changing the <tt>phi</tt> parameter, which controls the amount of autocorrelation in the data (it's the AR1 parameter which is used to create <tt>V</tt>). What happens to the difference between OLS and GLS?

```python
true_betas = np.array([0, 1])
iters = 100

N = 50
phi = 0.8  # AR1 parameter
sigsq = 2
V = sigsq * phi ** toeplitz(np.arange(N))

betas_ols = np.zeros((iters, 2))
betas_gls = np.zeros((iters, 2))

for i in range(iters):
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# YOUR CODE HERE
raise NotImplementedError()
```

## (More on) nuisance regression
Let's go back to the technique of nuisance regression. We have seen before that this technique can be used to model low-frequency components in our data (effectively functioning as a high-pass filter), but it can, in general, be used to model *any* thus-far unmodelled variance in the signal that would otherwise end up in the noise term. For example, people use this technique to model variance due to physiological processes (such cardiac and respiratory related signals; see e.g. [Glover et al., 2000](https://www.ncbi.nlm.nih.gov/pubmed/10893535)), motion-related variance (which we'll discuss later), and high-intensity "spikes". To get a better feel for nuisance regression and its consequences, let's look at this process of removing high-intensity spikes (which is sometimes called "despiking").

### Using nuisance regression for despiking
This technique of adding noise-predictors to the design matrix is sometimes used to model 'gradient artifacts', which are also called 'spikes' (which you've heard about in one of the videos for this week). This technique is also sometimes called "despiking". These spikes reflect sudden large intensity increases in the signal across the entire brain that likely reflect scanner instabilities. One way to deal with these artifacts is to "censor" bad timepoints (containing the spike) in your signal using a noise predictor.

But what defines a 'spike'/bad timepoint? One way is to compute the normalized "root mean square successive differences" (RMSSD), normalizing this, and imposing some threshold above which a timepoint is marked as a spike (technically, it's a little bit more complex, but we'll ignore that for now).

We'll delve into the details of this computation later. For now, let's take a look at some example data that we're going to use for this section:

```python
with np.load('spike_data.npz') as spike_data:
    all_sig = spike_data['all_sig']
    pred = spike_data['pred']

print("Shape of all_sig: %s" % (all_sig.shape,))
print("Shape of pred: %s" % (pred.shape,))
```

The example data `all_sig` is a (simulated) 4D fMRI scan with $10 \times 10 \times 10$ voxels and 500 timepoints (assuming a TR of 2, this amounts to a duration of 1000 seconds). The predictor reflects a design in which the participants was shown a stimulus every 100 seconds (50 TRs). Let's plot the predictor:

```python
plt.figure(figsize=(15, 5))
plt.plot(pred)
plt.grid()
plt.xlabel('Time (TRs)', fontsize=20)
plt.ylabel('Activation (A.U.)', fontsize=20)
plt.show()
```

Alright, now let's take a look at how you would calculate the "root mean square successive differences" (RMSSD). This quantity reflects the difference between every timepoint $t$ of your signal and the signal at timepoints $t-1$, which is then squared, averaged (across all voxels, $1 \dots K$), after which the square root is taken:

\begin{align}
\mathrm{RMSSD}_{t} = \sqrt{\frac{1}{K}\sum_{K}(s_{t, k} - s_{t-1, k})^2}
\end{align}

First, let's focus on the "successive differences". Suppose we have only one "signal" of length 5:

```python
ex_sig = np.array([1, 3, -2, 0, 5])
```

The "successive differences" are the difference between 3 and 1, -2 and 3, 0 and -2, and 5 and 0. Note that the successive difference for the first timepoint ($t=0$) is not defined! In code, we can compute this with:

```python
succ_diff = ex_sig[1:] - ex_sig[:-1]
print(succ_diff)
```

After computing the successive differences, we need to do another thing. As we noted earlier, RMSSD is not defined for $t=0$ (because there is no $t-1$ for the first timepoint). We can insert a duplicate of the first value here for convenience.

```python
succ_diff = np.insert(succ_diff, 0, succ_diff[0])
```

ToDo (optional; 0 point): Compute the RMSSD of the data. Note that our data (<tt>all_sig</tt>) has 4 dimensions, with the fourth dimension representing time. Store the result in a variable named <tt>all_sig_rmssd</tt>, which should have the same shape as the original signal. Also, try to do this without a for-loop by making using of vectorization.

```python
# Compute the sucessive differences here
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo (optional). '''
from niedu.tests.nii.week_4 import test_rmssd
test_rmssd(all_sig, all_sig_rmssd)
```

ToDo (optional; 0 points)

Now, in order to identify spikes, we need to identify timepoints that have a RMSSD-values that differ more than (let's say) 7 standard deviations from the mean RMSSD value. As such, you need to 'z-score' the RMSSD values: subtract the mean value from each individual value and divide each of the resulting 'demeaned' values by the standard deviation ($\mathrm{std}$) of the values. In other words, the z-transform of any signal $s$ with mean $\bar{s}$ is defined as:

\begin{align}
z(s) = \frac{(s - \bar{s})}{\mathrm{std}(s)}
\end{align}

Implement this z-score transform for the variable <tt>all_sig_rmssd</tt> and store it in the variable <tt>z_rmssd</tt> (1 point). Then, plot the z-scored RMSSD signal (with appropriate axis labels; 1 point).

```python
# Let's load the correct answer from the previous optional ToDo
# in case you didn't do it
all_sig_rmssd = test_rmssd(all_sig, None, check=False)

# Implement the z-scoring below
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests part 1 of ToDo. '''
from niedu.tests.nii.week_4 import test_zscore_rmssd
test_zscore_rmssd(all_sig_rmssd, z_rmssd)
```

```python
# Now, plot the zscored rmssd signal (z_rmssd)!
# YOUR CODE HERE
raise NotImplementedError()
```

Now, we can set a threshold above which we define timepoints as "spikes". Let's say we do this for $z > 7$.

```python
# Load the correct answer from the previous (optional) ToDo
from niedu.tests.nii.week_4 import test_zscore_rmssd
z_rmssd = test_zscore_rmssd(all_sig_rmssd, None, check=False)

identified_spikes = z_rmssd > 7  # creates array with True/False
n_spike = identified_spikes.sum()
print("There are %i spikes in the data!" % n_spike)
```

Now, to remove this influence, we can simply add a nuisance predictor for each spike, in which the predictor contains zeros at timepoints without the spike and 1 at the timepoint with a spike.

```python
spike_pred = np.zeros((pred.size, n_spike))
t_spikes = np.where(identified_spikes)[0]
for i, t in enumerate(t_spikes):
    print("Creating spike predictor for t = %i" % t)
    spike_pred[t, i] = 1

plt.figure(figsize=(15, 5))
plt.plot(spike_pred)
plt.xlabel("Time (TRs)", fontsize=20)
plt.ylabel("Activation (A.U.)", fontsize=20)
plt.grid()
plt.xlim(0, pred.size)
plt.show()
```

    ToThink (1 point):

Why do you think we do not convolve the spike regressors with an HRF (or basis set)? Write your answer in the text-cell below.

YOUR ANSWER HERE

    ToDo (2 points): Calculate the t-value of the stimulus-predictor-against-baseline contrast in a model with both the stimulus predictor (<tt>pred</tt>) and the spike predictors (<tt>spike_pred</tt>). Also, stack an intercept. Store the t-value in the variable <tt>tval_spike_model</tt>. Use <tt>spike_sig</tt> (defined below) as your target, i.e., $y$.

```python
# Calculate the t-value of the model with only an intercept + stim predictor
spike_sig = all_sig[5, 5, 5, :]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_spike_model
test_spike_model(pred, spike_pred, spike_sig, tval_spike_model)
```

    ToThink (1 point): An eager researcher might think that adding more and more (nuisance) predictors will always improve the amount of variance explained and thus will improve his/her chances of finding significant effects (i.e., $t$-values). Argue why this is not the case.

YOUR ANSWER HERE

    Tip!
    Before handing in your notebooks, we recommend restarting your kernel (Kernel → Restart & Clear Ouput) and running all your cells again (manually, or by Cell → Run all). By running all your cells one by one (from "top" to "bottom" of the notebook), you may spot potential errors that are caused by accidentally overwriting your variables or running your cells out of order (e.g., defining the variable 'x' in cell 28 which you then use in cell 15).


# Description of the problem and solution

The task1 was to predict a person's age from the brain image data: a standard regression problem. The original dataset included 832 features as well as a lot of NaN values and a few outliers. A good preprocessing stage was necessary in order to have a well defined dataset that could be used in our regression model. First step was the imputation of the dataset. Filling each NaN value with the median of each feature column. The use of the median instead of other value (e.g. mean) is justified since a lot of outliers are included in the dataset. (e.g. 1 2 _ 5 20 median: 3 mean: 7). Next step was the feature extraction. By using the "autofeat" library (paper: https://arxiv.org/pdf/1901.07329.pdf), we extracted the 21 most important features. The way the algorithm works is going through a loop of correlation of features with target, select promising features, train Lasso regression model with promising features, filter the good features keeping the ones with non-zero regression weights. We updated the datasets by keeping only the 21 most important features. Finally, we used these updated datasets for the training of our final regression model. A lot of outlier detection techniques were used but we decided to keep the outliers and use a tree-based method for our final model. Tree-methods have been proved to be robust to outliers and we avoid risking excluded important features / points from the dataset. The "ExtraTreesRegressor" model from the "sklearn" package was used and fine tuned based on the R2 score performance in our validation set. The final model had a score >0.6 in the validation sets using cross-validation and in the submission leaderboard of ETH scored 0.6812 while the hard baseline was set to 0.65 by the Advanced Machine Learning Task1 team.

# Include all the necessary packages

```python
!pip install autofeat

from sklearn.metrics import r2_score

try:
  %tensorflow_version 2.x
except Exception:
  pass
import tensorflow as tf

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from autofeat import FeatureSelector

from sklearn.model_selection import train_test_split
from sklearn.ensemble import ExtraTreesRegressor
```

    Requirement already satisfied: autofeat in /usr/local/lib/python3.6/dist-packages (0.2.5)
    Requirement already satisfied: pint in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.9)
    Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.25.2)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.21.3)
    Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from autofeat) (1.1.1)
    Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.14.0)
    Requirement already satisfied: numpy in /tensorflow-2.0.0/python3.6 (from autofeat) (1.17.3)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.16.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.0->autofeat) (2018.9)
    Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.0->autofeat) (2.6.1)
    Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->autofeat) (1.3.1)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy->autofeat) (1.1.0)
    Requirement already satisfied: six>=1.5 in /tensorflow-2.0.0/python3.6 (from python-dateutil>=2.6.1->pandas>=0.24.0->autofeat) (1.12.0)

# Load the data from the CSV files

```python
column_names_x = ['id']
for i in range(832):
  column_names_x.append('x'+str(i))

raw_dataset_x = pd.read_csv('/content/X_train.csv', names=column_names_x,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_x = raw_dataset_x.copy()
dataset_x.tail()
```

      id
      x0
      x1
      x2
      x3
      x4
      x5
      x6
      x7
      x8
      x9
      x10
      x11
      x12
      x13
      x14
      x15
      x16
      x17
      x18
      x19
      x20
      x21
      x22
      x23
      x24
      x25
      x26
      x27
      x28
      x29
      x30
      x31
      x32
      x33
      x34
      x35
      x36
      x37
      x38
      ...
      x792
      x793
      x794
      x795
      x796
      x797
      x798
      x799
      x800
      x801
      x802
      x803
      x804
      x805
      x806
      x807
      x808
      x809
      x810
      x811
      x812
      x813
      x814
      x815
      x816
      x817
      x818
      x819
      x820
      x821
      x822
      x823
      x824
      x825
      x826
      x827
      x828
      x829
      x830
      x831

      1207
      1207.0
      NaN
      5395.719279
      95668.548818
      1125.414599
      10341.757613
      10.547165
      108249.187400
      1.073291e+06
      108672.758838
      2.365013
      1.042126e+06
      21031.538201
      1059.704248
      103860.409465
      107.805931
      100.350289
      14040.029112
      NaN
      90414.095308
      10.108470
      3.215609
      102144.219482
      10.087478
      3825.046407
      89.538601
      4550.239929
      3542.470382
      206892.186644
      1.011027e+06
      11432.036864
      10.957698
      976.093967
      84.689946
      2500.095879
      1068.544855
      8.868114
      3.695648
      10381.626637
      8.641068e+05
      ...
      6.934015
      2.05604
      9.362337
      908.498493
      94.234452
      10.319140
      4155.345879
      8.498432e+16
      10977.013897
      93.766125
      8.120353
      10595.018829
      9.594486e+05
      10.485803
      10.601033
      101328.492269
      2.417968
      48.359296
      9486.144826
      2.185578
      966.094287
      9.039254e+05
      10.510759
      1049.753163
      10856.049561
      9109.098919
      10320.748617
      10892.743222
      106528.008864
      294.801642
      10.519831
      10.234396
      987.456317
      978.661701
      109520.061346
      102914.439553
      8144.701025
      10.442410
      102380.867791
      2.236101

      1208
      1208.0
      93669.580198
      3564.454295
      96937.341346
      1040.828378
      8415.112792
      10.721800
      100628.318687
      1.027671e+06
      102269.472299
      2.398768
      1.015631e+06
      10073.888339
      1005.671797
      106924.048275
      98.512805
      104.366843
      11719.094168
      115055.546469
      95750.178334
      NaN
      3.446025
      100455.697718
      10.303138
      3502.012112
      84.167435
      3462.164372
      3411.771480
      206892.234016
      1.082826e+06
      9957.026155
      11.155088
      1127.806594
      107.745872
      2298.057201
      1031.637383
      8.778034
      3.477553
      10627.490011
      1.007256e+06
      ...
      10.732723
      2.14034
      10.930583
      956.306694
      117.419062
      10.361670
      3436.194754
      6.249342e+16
      9866.685225
      104.420975
      9.171168
      10762.025617
      9.205254e+05
      10.627956
      10.101622
      105133.566646
      2.740250
      55.449159
      10959.655054
      2.160671
      1111.015635
      1.118527e+06
      10.342601
      1030.540259
      10097.006670
      9039.074820
      10895.768148
      10315.088493
      106898.299599
      195.245438
      10.618901
      10.456550
      1112.699713
      NaN
      NaN
      106685.647900
      7428.338174
      10.405107
      107051.806312
      2.297040

      1209
      1209.0
      94119.048262
      NaN
      88398.586879
      1298.024079
      8905.109893
      10.294486
      103347.543915
      1.087571e+06
      106082.959731
      2.345450
      1.074165e+06
      49595.639929
      1057.051949
      103147.267707
      NaN
      104.099172
      11275.080883
      115055.523086
      105201.547958
      10.358605
      2.872759
      107834.745605
      10.070086
      2708.039945
      90.315251
      4502.083820
      2267.035496
      206892.199058
      1.016735e+06
      8545.036877
      NaN
      818.975938
      95.255808
      2614.046035
      1002.028549
      11.262573
      3.016052
      10912.480045
      1.016656e+06
      ...
      10.447547
      NaN
      10.502198
      967.481047
      88.216016
      10.535296
      3548.968573
      3.358649e+16
      8140.800029
      98.172030
      11.651412
      8779.078077
      7.920442e+05
      10.525088
      10.468168
      104071.834129
      2.615953
      130.309663
      10513.170240
      2.228708
      728.052692
      1.147362e+06
      9.968195
      1077.035231
      7580.079750
      8509.033486
      10228.351982
      10327.801413
      100144.263391
      310.183226
      10.887052
      10.836007
      1124.755976
      988.193910
      102076.305244
      101964.413583
      6470.242459
      9.786964
      NaN
      2.222794

      1210
      1210.0
      NaN
      3356.949591
      97081.843123
      1002.778571
      10864.284014
      10.797471
      102894.681201
      NaN
      100102.393811
      2.436208
      1.072574e+06
      37587.588730
      1047.896435
      105241.465707
      93.329187
      102.255382
      11668.076593
      115055.525882
      85619.926316
      10.732048
      2.422995
      104044.942484
      NaN
      2945.014802
      112.182044
      3202.074919
      4883.134639
      206892.208501
      1.095387e+06
      7036.096752
      8.290676
      878.350888
      93.612258
      2560.094596
      1011.331590
      10.580875
      2.822595
      NaN
      1.085728e+06
      ...
      10.115872
      2.15810
      9.868292
      1004.134878
      110.999990
      10.713020
      3394.243287
      1.335970e+17
      8371.821670
      89.703650
      9.220535
      8998.055593
      1.113826e+06
      10.302250
      10.883418
      101283.417629
      2.186525
      366.750195
      11376.166095
      2.186734
      1120.066965
      1.013800e+06
      11.734373
      1046.939030
      9132.087999
      5896.007658
      10706.113864
      10369.803817
      102404.821085
      NaN
      10.661966
      10.569144
      1010.143125
      1064.316139
      101477.589227
      104517.364548
      4922.920835
      8.566389
      103968.235402
      2.071553

      1211
      1211.0
      101744.439395
      3416.474452
      111000.425491
      893.080460
      9250.598106
      10.935695
      102906.878188
      NaN
      107003.164135
      2.547171
      1.087477e+06
      32084.633800
      1075.700568
      104998.478525
      118.678047
      106.993698
      13226.038641
      115055.543174
      97282.156471
      10.208538
      3.710694
      107439.936587
      10.753281
      4719.085520
      96.505582
      3419.271979
      3796.393911
      206892.165049
      1.006348e+06
      11518.011808
      10.803776
      1103.388499
      106.356520
      2583.097413
      1081.964569
      11.453524
      3.203942
      10257.152912
      9.292065e+05
      ...
      8.157006
      2.16880
      8.844200
      1005.897320
      96.253172
      NaN
      3289.749206
      9.364744e+16
      9170.136304
      97.067706
      NaN
      NaN
      1.148141e+06
      10.989940
      10.204516
      100896.544241
      2.720932
      49.567830
      8013.156899
      2.550404
      916.073375
      1.080536e+06
      9.101128
      1013.211838
      12517.066013
      10990.064488
      10349.678752
      10096.876996
      NaN
      322.445715
      10.599132
      10.350638
      962.755003
      1093.734877
      105353.550546
      106061.798352
      9302.374002
      10.949418
      109317.619776
      2.496408

5 rows × 833 columns

```python
column_names_y = ['id','y']
raw_dataset_y = pd.read_csv('/content/y_train.csv', names=column_names_y,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_y = raw_dataset_y.copy()
dataset_y.tail()
```

      id
      y

      1207
      1207.0
      66.0

      1208
      1208.0
      73.0

      1209
      1209.0
      74.0

      1210
      1210.0
      78.0

      1211
      1211.0
      64.0

# Print the missing values

```python
missing_values = dataset_x.isna().sum()
print (missing_values)
```

    id        0
    x0       81
    x1      103
    x2       92
    x3       91
           ...
    x827     83
    x828     78
    x829     98
    x830     84
    x831     92
    Length: 833, dtype: int64

# Split the data into training and test data

```python
# Split using sklearn.model_selection
x_train, x_test, y_train, y_test = train_test_split(dataset_x, dataset_y, test_size=0.2, random_state = 100)
```

```python
train_stats = x_train.describe()
train_stats.pop("id")
train_stats = train_stats.transpose()
train_stats
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      x0
      909.0
      99849.359545
      9534.020258
      65533.368423
      93818.485147
      100183.062423
      105994.290528
      130226.576502

      x1
      885.0
      3698.730375
      943.683864
      180.312021
      3076.550570
      3651.110055
      4303.892503
      7265.213902

      x2
      891.0
      99975.389109
      9540.065988
      68544.573581
      93937.346571
      99386.035114
      106102.200889
      132221.045067

      x3
      898.0
      999.944996
      100.903669
      694.745271
      935.303439
      999.571797
      1068.606823
      1434.200505

      x4
      890.0
      10001.743350
      1001.473353
      6681.561828
      9339.312428
      10021.924636
      10646.003276
      13560.223285

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      x827
      906.0
      104990.967566
      2761.209013
      100015.768596
      102783.100024
      104986.305216
      107352.370223
      109999.847537

      x828
      907.0
      6827.704539
      1387.835714
      1696.036569
      6002.316474
      6835.947954
      7652.607118
      11276.075121

      x829
      884.0
      10.021817
      0.982265
      6.899008
      9.378562
      9.977236
      10.676450
      13.188278

      x830
      902.0
      104960.353706
      2845.423469
      100003.049706
      102653.373914
      104838.184005
      107428.898901
      109993.046071

      x831
      893.0
      2.269127
      0.169559
      1.589261
      2.173057
      2.291077
      2.374205
      2.846222

832 rows × 8 columns

# Fill the NaN in the training data set with the median values of each column

```python
x_train = x_train.fillna(x_train.median())
x_test = x_test.fillna(x_test.median())
missing_values = x_train.isna().sum()
print (missing_values)
```

    id      0
    x0      0
    x1      0
    x2      0
    x3      0
           ..
    x827    0
    x828    0
    x829    0
    x830    0
    x831    0
    Length: 833, dtype: int64

# Remove the unnecessary "id" label

```python
y_train.pop("id")
y_test.pop("id")
```

    1143    1143.0
    941      941.0
    365      365.0
    467      467.0
    615      615.0
             ...
    156      156.0
    689      689.0
    28        28.0
    69        69.0
    1198    1198.0
    Name: id, Length: 243, dtype: float64

# Feature Extraction using autofeat

```python
fsel = FeatureSelector(featsel_runs=4,
        max_it=150,
        w_thr=1e-6,
        keep=None,
        n_jobs=1,
        verbose=1)

new_X = fsel.fit_transform(pd.DataFrame(x_train, columns=column_names_x), y_train)
print(new_X.columns)

df_train = pd.DataFrame(x_train, columns=column_names_x)
df_test = pd.DataFrame(x_test, columns=column_names_x)
```

    /usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

    [featsel] Scaling data...done.
    [featsel] 220/833 features after univariate filtering
    [featsel] Feature selection run 1/4
    [featsel] Feature selection run 2/4
    [featsel] Feature selection run 3/4
    [featsel] Feature selection run 4/4
    [featsel] 28 features after 4 feature selection runs
    [featsel] 28 features after correlation filtering
    [featsel] 21 features after noise filtering
    [featsel] 21 final features selected (including 0 original keep features).
    Index(['x400', 'x635', 'x757', 'x516', 'x809', 'x214', 'x556', 'x617', 'x93',
           'x346', 'x596', 'x255', 'x309', 'x252', 'x292', 'x738', 'x537', 'x593',
           'x474', 'x614', 'x502'],
          dtype='object')

# Keep only the necessary features

```python
dataset_selected_x = x_train.copy()
x_test_selected= x_test.copy()

#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,
       #556, 617, 93, 346, 596, 309, 252, 292, 474,
       #593, 614]

accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,
       346, 596, 255, 309, 252, 292, 738, 537, 593,
       474, 614, 502]

for j in range(832):
        if (j in accepted):
          print (j)
        else:
          del dataset_selected_x['x'+str(j)]
          del x_test_selected['x'+str(j)]

train_stats = dataset_selected_x.describe()
train_stats.pop("id")
train_stats = train_stats.transpose()
train_stats
```

    93
    214
    252
    255
    292
    309
    346
    400
    474
    502
    516
    537
    556
    593
    596
    614
    617
    635
    738
    757
    809

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      x93
      969.0
      3.519955e+03
      6.123839e+02
      1.789851e+03
      3.122307e+03
      3.492040e+03
      3.882096e+03
      5.902057e+03

      x214
      969.0
      1.686814e+03
      4.246669e+02
      3.857756e+02
      1.438003e+03
      1.657997e+03
      1.929010e+03
      3.719025e+03

      x252
      969.0
      5.153136e+03
      6.443209e+03
      -1.066389e+04
      2.036529e+03
      3.051817e+03
      5.396992e+03
      5.218476e+04

      x255
      969.0
      1.063172e+04
      1.847941e+03
      3.258064e+03
      9.580092e+03
      1.053304e+04
      1.170004e+04
      1.742768e+04

      x292
      969.0
      1.289525e+05
      1.561997e+04
      7.115248e+04
      1.201669e+05
      1.282218e+05
      1.377145e+05
      1.982293e+05

      x309
      969.0
      1.360934e+04
      2.104294e+03
      1.787475e+03
      1.241801e+04
      1.355888e+04
      1.480791e+04
      2.142239e+04

      x346
      969.0
      7.264234e+03
      1.238998e+03
      2.195927e+03
      6.577637e+03
      7.355575e+03
      8.085946e+03
      1.121581e+04

      x400
      969.0
      2.419177e+00
      1.566607e-01
      1.441218e+00
      2.325897e+00
      2.416256e+00
      2.507563e+00
      3.029658e+00

      x474
      969.0
      2.138704e+05
      3.363320e+04
      6.580233e+04
      1.951336e+05
      2.113702e+05
      2.311579e+05
      4.824998e+05

      x502
      969.0
      7.341196e+13
      5.051213e+13
      -8.384285e+13
      4.129257e+13
      6.245895e+13
      9.198935e+13
      3.816907e+14

      x516
      969.0
      2.633410e+00
      2.851324e-01
      1.586758e+00
      2.461447e+00
      2.632617e+00
      2.810227e+00
      3.709460e+00

      x537
      969.0
      2.131250e+05
      3.401347e+04
      6.499225e+04
      1.936266e+05
      2.105931e+05
      2.305157e+05
      4.810740e+05

      x556
      969.0
      6.058210e+03
      7.982839e+02
      3.040824e+03
      5.588225e+03
      5.998493e+03
      6.472923e+03
      9.103862e+03

      x593
      969.0
      6.306444e+04
      3.517018e+04
      2.381309e+04
      4.825502e+04
      5.195309e+04
      5.625009e+04
      2.232860e+05

      x596
      969.0
      1.994849e+04
      2.631193e+03
      9.693873e+03
      1.845468e+04
      1.980071e+04
      2.136755e+04
      3.037075e+04

      x614
      969.0
      1.219718e+06
      1.863793e+05
      4.090431e+05
      1.112420e+06
      1.220616e+06
      1.319038e+06
      1.976749e+06

      x617
      969.0
      1.528155e+03
      6.897489e+02
      -6.478282e+02
      1.039647e+03
      1.425202e+03
      1.899843e+03
      5.028253e+03

      x635
      969.0
      2.552238e+00
      2.211141e-01
      1.536237e+00
      2.441985e+00
      2.571545e+00
      2.685138e+00
      3.319348e+00

      x738
      969.0
      4.070505e+05
      5.419321e+04
      1.500802e+05
      3.777463e+05
      4.051612e+05
      4.389639e+05
      6.508670e+05

      x757
      969.0
      2.556026e+00
      2.138400e-01
      1.513769e+00
      2.448038e+00
      2.577740e+00
      2.684761e+00
      3.292584e+00

      x809
      969.0
      8.301919e+01
      1.026332e+02
      -1.034091e+02
      3.267047e+01
      5.403747e+01
      9.901100e+01
      1.245880e+03

# Regression model

```python
rfr = ExtraTreesRegressor(n_jobs=1, max_depth=None, n_estimators=180, random_state=0, min_samples_split=3, max_features=None)

rfr.fit(dataset_selected_x, np.ravel(y_train))

y_pred = rfr.predict(x_test_selected)

score = r2_score(y_test, y_pred)
print(score)
```

    0.6037280104368132

# Export the file with our predictions for submission

```python
#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,
       #556, 617, 93, 346, 596, 309, 252, 292, 474,
       #593, 614]

accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,
       346, 596, 255, 309, 252, 292, 738, 537, 593,
       474, 614, 502]

column_names_x = ['id']
for i in range(832):
  column_names_x.append('x'+str(i))

raw_dataset_x_test = pd.read_csv('/content/X_test.csv', names=column_names_x,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_x_test = raw_dataset_x_test.copy()
dataset_x_test.tail()

dataset_x_test = dataset_x_test.fillna(dataset_x_test.median())

dataset_selected_x_test = dataset_x_test.copy()

for j in range(832):
        if (j in accepted):
          print (j)
        else:
          del dataset_selected_x_test['x'+str(j)]

predictions = rfr.predict(dataset_selected_x_test)

index = 0.0
with open('predictions.txt', 'w') as f:
    f.write("%s\n" % "id,y")
    for predict in predictions:
        writing_str = str(index)+','+str(predict.item(0))
        f.write("%s\n" % writing_str)
        index = index + 1
```

    93
    214
    252
    255
    292
    309
    346
    400
    474
    502
    516
    537
    556
    593
    596
    614
    617
    635
    738
    757
    809


```python
import numpy as np
import qutip as qt
import matplotlib.pyplot as plt
%matplotlib inline
```

# Dissipative Jaynes-Cummings Model

<br>
$$
\begin{equation}
H = \hbar\omega_{c}a^{\dagger}a + \frac{1}{2}\hbar\omega_{a}\sigma_{-}^{\dagger}\sigma_{-}+\hbar g(a^{\dagger}\sigma_{-}+a\sigma_{-}^{\dagger})
\end{equation}
$$

Build up the Hamiltonian same as before:

```python
N = 15                 # number of cavity fock states
wc = 1.0  * 2 * np.pi  # cavity frequency
wa = 1.0  * 2 * np.pi  # atom frequency
g  = 0.05 * 2 * np.pi  # coupling strength

# operators
a  = qt.tensor(qt.destroy(N), qt.qeye(2)) # qeye is short for identity
sm = qt.tensor(qt.qeye(N), qt.sigmam())

# Hamiltonian
H = wc * a.dag() * a + wa * sm.dag() * sm + g * (a.dag() * sm + a * sm.dag())
```

Now define the collapse operators:

```python
kappa = 0.005       # cavity dissipation rate
gamma = 0.05        # atom dissipation rate

c_ops = [np.sqrt(kappa)*a, np.sqrt(gamma) * sm]
```

Repeat initial Jaynes-Cummings simulation with dissipation.

```python
# intial state
psi0 = qt.tensor(qt.basis(N,0), qt.basis(2,0))    # start with an excited atom, cavity ground

# Times over which to evaluate system
tlist = np.linspace(0,25,101)

# Make list of expectation value operators
e_ops = [a.dag() * a, sm.dag() * sm]

#Perform evolution
output = qt.mesolve(H, psi0, tlist, c_ops, e_ops)
```

```python
plt.figure(figsize=(8,6))
plt.plot(tlist,output.expect[0], lw=3)
plt.plot(tlist,output.expect[1], lw=3)
plt.legend(['$\langle a^{\dagger}a\\rangle$',
            '$\langle \sigma_{-}^{\dagger}\sigma_{-}\\rangle$'],
           loc=1,fontsize=16)
plt.xlabel('Time', fontsize=16)
plt.ylabel('Occupation Number', fontsize=16);
```

## State of the cavity

Output states from master equation solver, and look at Wigner function of cavity.

```python
output = qt.mesolve(H, psi0, tlist, c_ops, [])
```

```python
t_idx = np.where([tlist == t for t in [0.0, 5, 15, 25]])[1]
cavity_rho_list = [output.states[kk].ptrace(0) for kk in t_idx]
```

```python
# loop over the list of density matrices

xvec = np.linspace(-3,3,200)

fig, axes = plt.subplots(1,len(cavity_rho_list), sharex=True,
                         figsize=(4*len(cavity_rho_list),3))

for idx, rho in enumerate(cavity_rho_list):
    # calculate its wigner function
    W = qt.wigner(rho, xvec, xvec)
    # plot its wigner function
    cf = axes[idx].contourf(xvec, xvec, W, 100, vmin=-0.4, vmax=0.4, cmap='RdBu')
    axes[idx].set_title(r"$t = %.1f$" % tlist[t_idx][idx], fontsize=16)
cbar = fig.colorbar(cf, ax=axes.ravel().tolist())
```

## What does super operator look like

```python
plt.figure(figsize=(7,7))
plt.spy(qt.liouvillian(H,c_ops).data,ms=2)
```

It is easy to see that the Liouvillian super operator is non-Hermitian.  This is to be expected as our evolution traces over the environment which, as we have seen, leads to mixed states independent of the purity of the initial state; something that cannot happen in unitary evolution:

$$
\mathrm{Tr}\left[\rho(t)^{2}\right] = \mathrm{Tr}\left[U(t)\rho(0)U^{\dagger}(t)U(t)\rho(0)U^{\dagger}(t)\right] = \mathrm{Tr}\left[\rho(0)^{2}\right]
$$

```python

```


```python
import numpy as np
import pydae.build as db
import sympy as sym
from pydae.models import grid2dae_dq
```

```python

```

```python
U_b = 20e3
V_b = U_b/np.sqrt(3)

R_ph_km   = 0.501
X_ph_km   = 0.716
B_ph_muS_km = 47.493

Length_12_13_km = 4.89
Length_13_14_km = 2.99

R_12_13 = Length_12_13_km*R_ph_km
X_12_13 = Length_12_13_km*X_ph_km
B_12_13 = Length_12_13_km*B_ph_muS_km*1e-6

R_13_14 = Length_13_14_km*R_ph_km
X_13_14 = Length_13_14_km*X_ph_km
B_13_14 = Length_13_14_km*B_ph_muS_km*1e-6

data = {
"system":{"f_hz":50,"model_type":"ode"},
"buses": [{"bus":"12"},{"bus":"13"},{"bus":"14"}],
"lines":     [
              # {"bus_j":"0", "bus_k":"12", "R":0.016, "X":1.92, "B":B_12_13/10},
                {"bus_j":"12", "bus_k":"13", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":4.89},
                {"bus_j":"13", "bus_k":"14", "R_km":0.501, "X_km":0.716, "B_muS_km":47.493, "km":2.99},
             ],
"loads" : [
        #{"bus": "12", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200},
        #{"bus": "13", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200},
        #{"bus": "14", "kVA":0e3, "pf": 0.95, "T_i":0.01,"I_max":1200}
        ],
"grid_formers" : [
                  #{"bus": "12","V_phph":20.0e3, "deg":0.0},
                 ]}
```

```python
15300
```

    15300

```python

```

```python
grid_dae  = grid2dae_dq(data,park_type = 'fisix')

'''
grid_formers = data["grid_formers"]
for grid_former in grid_formers:
    bus = grid_former['bus']
    idx_D = grid_dae['x_list'].index(f'v_{bus}_D')
    grid_dae['f'].pop(idx_D)
    grid_dae['x'].pop(idx_D)
    grid_dae['x_list'].pop(idx_D)
    grid_dae['u'].update({f'v_{bus}_D':grid_former["V_phph"]*np.sqrt(2/3)*np.sin(np.deg2rad(grid_former["deg"]))})
    idx_Q = grid_dae['x_list'].index(f'v_{bus}_Q')
    grid_dae['f'].pop(idx_Q)
    grid_dae['x'].pop(idx_Q)
    grid_dae['x_list'].pop(idx_Q)
    grid_dae['u'].update({f'v_{bus}_Q':grid_former["V_phph"]*np.sqrt(2/3)*np.cos(np.deg2rad(grid_former["deg"]))})
'''
f_list = grid_dae['f']
x_list = grid_dae['x']

g_list = grid_dae['g']
y_list = grid_dae['y']
params_dict = grid_dae['params']
u_run_dict = grid_dae['u']
u_run_dict.update({'ina':0})
a = sym.Symbol('a')
ina = sym.Symbol('ina')
y_list = ['a']
g_list = [1-a]
sys = {'name':'grid_3bus',
       'params_dict':params_dict,
       'f_list':f_list,
       'g_list':g_list,
       'x_list':x_list,
       'y_ini_list':y_list,
       'y_run_list':y_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_run_dict,
       'h_dict':{'i_R06_d':y_list[0]}
      }

sys = db.system(sys)
db.sys2num(sys)
```

```python
grid_dae
```

    {'f': [(-L_1213*i_l_1213_Q*omega - R_1213*i_l_1213_D + v_12_D - v_13_D)/L_1213,
      (L_1213*i_l_1213_D*omega - R_1213*i_l_1213_Q + v_12_Q - v_13_Q)/L_1213,
      (-L_1314*i_l_1314_Q*omega - R_1314*i_l_1314_D + v_13_D - v_14_D)/L_1314,
      (L_1314*i_l_1314_D*omega - R_1314*i_l_1314_Q + v_13_Q - v_14_Q)/L_1314,
      2*(-C_1213*omega*v_12_Q/2 + i_12_D - i_l_1213_D)/C_1213,
      2*(C_1213*omega*v_12_D/2 + i_12_Q - i_l_1213_Q)/C_1213,
      (i_13_D + i_l_1213_D - i_l_1314_D + omega*v_13_Q*(-C_1213/2 - C_1314/2))/(C_1213/2 + C_1314/2),
      (i_13_Q + i_l_1213_Q - i_l_1314_Q - omega*v_13_D*(-C_1213/2 - C_1314/2))/(C_1213/2 + C_1314/2),
      2*(-C_1314*omega*v_14_Q/2 + i_14_D + i_l_1314_D)/C_1314,
      2*(C_1314*omega*v_14_D/2 + i_14_Q + i_l_1314_Q)/C_1314],
     'g': [],
     'x': [i_l_1213_D,
      i_l_1213_Q,
      i_l_1314_D,
      i_l_1314_Q,
      v_12_D,
      v_12_Q,
      v_13_D,
      v_13_Q,
      v_14_D,
      v_14_Q],
     'y': [],
     'x_list': ['i_l_1213_D',
      'i_l_1213_Q',
      'i_l_1314_D',
      'i_l_1314_Q',
      'v_12_D',
      'v_12_Q',
      'v_13_D',
      'v_13_Q',
      'v_14_D',
      'v_14_Q'],
     'u': {'ina': 0},
     'params': {'R_1213': 2.44989,
      'L_1213': 0.011144793059021352,
      'C_1213': 7.392453306593591e-07,
      'R_1314': 1.4979900000000002,
      'L_1314': 0.006814505367377064,
      'C_1314': 4.520129935933505e-07,
      'i_12_D': 0.0,
      'i_12_Q': 0.0,
      'i_13_D': 0.0,
      'i_13_Q': 0.0,
      'i_14_D': 0.0,
      'i_14_Q': 0.0,
      'omega': 314.1592653589793},
     'v_list': [v_12_D, v_12_Q, v_13_D, v_13_Q, v_14_D, v_14_Q]}

```python
u_run_dict
```

    {'ina': 0}

```python

```

```python

```