# Working with MRI data in Python
In this tutorial we will discuss how to interact with *Nifti* files — the file format used most in the MRI community — using the Python package *Nibabel*. We assume that you have experience with basic Python syntax and Numpy and Matplotlib functionality. If not, please consult [this page](https://lukas-snoek.com/NI-edu/other/python_recap.html) for an overview of resources.

**What you'll learn**: after this week's lab ...

* you know how to load, manipulate, and analyze Nifti-files

**Estimated time to complete**: 1-2 hours

## What is "nifti"?
Usually, every scanner manufacturer (Philips, Siemens, GE, etc.) have their own proprietary data format. For example, here at the University of Amsterdam, we have a Philips scanner and we often export our data as PAR/REC files. However, to streamline the development and use of neuroimaging software packages, the Neuroimaging InFormatics Technology Initiative came up with a new, standardized format, *nifti*, that most neuroimaging packages should support. As such, usually the first step in any (f)MRI preprocessing pipeline is to convert the scanner-specific files (e.g., PAR/RECs) to nifti. This is beyond the scope of this course, but if at one point you need to do this yourself, we highly recommed the [dcm2niix](https://github.com/rordenlab/dcm2niix) package!

Throughout these tutorials you'll work with these nifti-files (also called nifti images), which you can recognize by their extension of *.nii* or its compressed version *.nii.gz*. This file-format is supported by most neuroimaging software packages (such as FSL, which you'll use later in this course!).

However, we'd like to inspect and analyze nifti images in Python as well! *Nibabel* is an awesome Python package that allows us to read and load nifti images, and convert them to numpy arrays in a straightforward manner.

```python
# Let's load some other packages we need
import os
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import nibabel as nib # common way of importing nibabel
```

We'll load in an example anatomical MRI scan (`anat.nii.gz`) from the current directory using nibabel below:

```python
mri_file = 'anat.nii.gz'
img = nib.load(mri_file)
```

Note the type of `img`: the `Nifti1Image` class. This is a custom class just like a Numpy array, with its own attributes and methods!

```python
print(type(img))
```

For example, one attribute of `Nifti1Image` objects is the `shape` (which is similar to the numpy array attribute with the same name):

```python
print(img.shape)
```

Here, the `shape` attribute is telling us that this is a 3D (anatomical) scan and has 240 voxels in the first dimension, 240 voxels in the second dimension, and 220 voxels in the third dimension.

## The three parts of nifti images
Nifti images can be roughly divided into three "parts":
1. The header with metadata;
2. The image data;
3. The affine matrix

All three parts are of course represented in nibabel's `Nifti1Image` class. Let's go through them one by one.

### The header
The header of nifti files contain metadata about the scan, such as the units of measurement, the voxel size, etc. In `Nifti1Images`, the header is an attribute:

```python
# here, we're storing the header attribute in a new variable, hdr, for easy of use
hdr = img.header
```

Perhaps confusingly, the header is a custom object (a `Nifti1Header` object) as well, with its own methods and attributes. For example, it has a method called `get_zooms()`, which returns the voxel size (and optionally the sampling rate, if it's a fMRI file):

```python
hdr.get_zooms()  # it's a 1x1x1 mm MRI file!
```

Another useful method is the `get_xyzt_units` which returns the units of the measurements (here: voxel size in millimeter and time in seconds):

```python
hdr.get_xyzt_units()
```

Let's also load in a functional MRI file (`func.nii.gz`) to see the difference with an anatomical MRI file:

```python
fmri_file = 'func.nii.gz'
f_img = nib.load(fmri_file)
print(f_img.shape)
print(f_img.header.get_zooms())
print(f_img.header.get_xyzt_units())
```

In case of fMRI files, the fourth dimension (almost) always represents the "time" dimension. So you can assume that a nifti image of an fMRI file has 4 dimensions, with the first three being the spatial dimensions (similar to the anatomical MRI file: $X \times Y \times Z$) and the last (fourth) being the time dimension ($T$).

So for the above file, you can assume that it has 50 timepoints and has a sampling rate of 0.7 seconds (i.e., a new volume was scanned every 0.7 seconds).

Moreover, you can infer that this file contains data from $80 \times 80 \times 44$ voxels with dimensions $2.7 \times 2.7 \times 2.97$ (in millimeters). To be honest, in practice, you won't deal a lot with the header (as you are generally aware of the dimensions/units of your data), so let's look at the *actual* data!

### The data
When we loaded in the data and created a `Nifti1Image` object, we actually didn't load in the *actual* data (i.e., voxel intensities) in memory yet! This is postponed because it's quite a memory-intensive operation: remember, loading in an fMRI scan with dimensions $80 \times 80 \times 44 \times 50$ is effectively loading in over 14 million numbers (which is, in this case, more than 50MB in size)! By postponing loading the data, you can, for example, peek at the image dimensions without loading all the data in memory.

Anyway, to actually load in the data, you can call the `get_fdata()` method, which will return a numpy array with the same dimensions as the image data. We'll take a look at the anatomical MRI data (`anat.nii.gz`):

```python
img_data = img.get_fdata()
print(type(img_data))  # it's a numpy array!
print(img_data.shape)
```

Okay, so `img_data` is a 3D numpy array with ... what exactly? Let's check it out:

```python
print(img_data)
```

It's just a bunch of numbers! This is important to remember: **like any image (like your standard png, jpeg, or gif), MRI scans are also just (3D or 4D) arrays of numbers**! The higher the number, the more signal the scanner recorded at that voxel. (It's actually a little more complex than this, but that's beyond the scope of this course!)

Often, the absolute value of the signal intensities is not necessarily most or only important thing. The *relative differences* between different voxels in space or within voxels across time is also important. For example, for anatomical scans, apart from a high overall signal intensity, you often want a good *contrast* between white and gray matter (i.e., different signal intensities between voxels in these two tissue types). For functional MRI, apart from a clear tissue contrast and a high overall signal intensity, you often also want little (non-experimentally related) differences in signal intensity within a voxel across time.

Anyway, when we printed this 3D array (of $240 \times 240 \times 220$), the notebook chose not to display all 12,672,000 numbers but instead truncated it. The reason it only printed zeros is because the first and last couple of numbers in each dimension likely doesn't contain any (signal related to) brain, just empty space!

So, let's index a small $3 \times 3 \times 3$ patch of voxels in the middle of the brain to check what values these voxels contain:

```python
mid_vox = img_data[118:121, 118:121, 108:111]
print(mid_vox)
```

That's better! The exact values are not necessarily directly interpretable, i.e., you cannot say that the value 61978.46 is good (or bad), because the exact *scale* of the signal intensities (whether it goes from 0-100 or from 0-100,000) depends on the specific scanner hardware and specific nifti conversion software.

Like any image, (f)MRI scans can be visualized by plotting the numbers and assigning colors to the numbers. Often we visualize brain images in brain and white, with higher signal intensities being brighter and lower signal intensities being darker. Importantly, remember that our data here cannot directly be plotted as a (2D) image, because our data (an anatomical scan) is 3D! However, we *can* just plot a single *slice* of the 3D volume, for example, the middle slice of our first voxel axis:

```python
mid_slice_x = img_data[119, :, :]
print(mid_slice_x.shape)
```

We can use matplotlib to plot this slice as an image using the `imshow` function you're seen before:

```python
# Note that the transpose the slice (using the .T attribute).
# This is because imshow plots the first dimension on the y-axis and the
# second on the x-axis, but we'd like to plot the first on the x-axis and the
# second on the y-axis. Also, the origin to "lower", as the data was saved in
# "cartesian" coordinates.
plt.imshow(mid_slice_x.T, cmap='gray', origin='lower')
plt.xlabel('First axis')
plt.ylabel('Second axis')
plt.colorbar(label='Signal intensity')
plt.show()
```

Alright, time to practice.

ToDo (1 point): Remember numpy indexing? For this ToDo, extract the second entry from the last dimension of <tt>img_data</tt> and store it in a new variable called <tt>second_slice</tt>. Tip: this variable should have a shape of $240 \times 240$!

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo '''
from niedu.tests.nii.week_1 import test_extract_second_slice
test_extract_second_slice(img_data, second_slice)
```

So far, we have only looked at *structural* (3D) MRI data. This course is mainly about analyzing *functional* data, so let's look at the data from an fMRI scan. We'll use the `Nifti1Image` object `func_img` from before. We'll load the memory with the `get_fdata` method. Note that this takes a little longer than loading in the structural data, because instead of ($240 \times 240 \times 220 = $) 12,672,000 datapoints, we're loading in ($80 \times 80 \times 44 \times 347 =$) 97,715,200 datapoints!

```python
f_img_data = f_img.get_fdata()
print(f_img_data.shape)
```

Now, you can think about this 4D array as a series of 3D arrays (volumes), in which the fourth dimension represents time (as shown in the image below).

*Source: [nilearn website](https://nilearn.github.io/_images/niimgs.jpg)*

These separate 3D arrays (from the first three dimensions) in fMRI files are often called "volumes" or "dynamics". So, for example, the first volume refers to the first 3D array that was recorded.

Just like the anatomical MRI data, the values in the fMRI data just represent numbers:

```python
# printing a small 3x3x3 volume of voxels from the first timepoint
print(f_img_data[38:41, 38:41, 20:23, 0])
```

Again, we can visualize these numbers as images, at least for the first three dimensions, as these represent the spatial dimensions (the fourth dimension represents time). For example, we can visualize a single slice (e.g., $x = 39$) of the first volume ($t = 0$) as follows:

```python
mid_slice_x_fmri = f_img_data[39, :, :, 0]  # x = 39, t = 0
print("Shape of slice: %s" % (mid_slice_x_fmri.shape,))

plt.imshow(mid_slice_x_fmri.T, cmap='gray', origin='lower')
plt.xlabel('First axis')
plt.ylabel('Second axis')
plt.colorbar(label='Signal intensity')
plt.show()
```

However, we can also look at fMRI data from a different perspective, that is, from the time dimension! For example, we could extract a single voxel's *time series* (i.e., how the signal intensity varies across time) and plot the signal intensity values of that voxel across time. First, let's extract the time series of one particular voxel (e.g., the middle one across all spatial dimensions):

```python
mid_vox_ts = f_img_data[39, 39, 21, :]  # note the ":", saying: give me ALL the timepoints
print("Voxel timeseries shape: %s" % (mid_vox_ts.shape,))
```

What we effectively did in the previous cell is to extract the signal intensity at the *same* spatial coordinates ($x=39, y=39, z=21$) across all 347 timepoints. We show this visually in the next cell, where the show the coordinates as a red box across 20 timepoints (not all 347, because that would clutter the notebook too much):

```python
from matplotlib import patches

fig, axes = plt.subplots(ncols=5, nrows=4, figsize=(20, 10))  # 20 timepoints
# Loop over the first 20 volumes/timepoints
for t, ax in enumerate(axes.flatten()):
    ax.imshow(f_img_data[39, :, :, t].T, cmap='gray', origin='lower')  # index with t!
    rect = patches.Rectangle((38, 20), 2, 2, linewidth=2, edgecolor='r', facecolor='none')
    ax.add_patch(rect)
    ax.axis('off')
    ax.set_title('t = %i' % t, fontsize=20)
fig.tight_layout()
```

Note that it is *really* hard to spot whether the image of the slice is actually different between different time points! To the naked eye, it just seems like the same image. This is because activity fluctations (over time) in fMRI are **very** small - most of the time just 1-3% compared to baseline (average) activity. That's why it's hard to spot activity differences by looking at the pictures alone (without any scaling).

It's actually easier to see these time-by-time fluctuations in signal intensity by plotting the time series directly:

```python
plt.figure(figsize=(18, 3))
plt.plot(mid_vox_ts, 'o-', ms=4)
plt.xlim(-1, mid_vox_ts.size)
plt.ylim(38000, 55000)
plt.ylabel('Signal intensity', fontsize=15)
plt.xlabel('Time (nr. of volumes)', fontsize=15)
plt.show()
```

In the plot above, the blue dots are the signal intensity values (y-axis), plotted across time (x-axis). The blue lines interconnecting the blue dots are just interpolated values in between the different datapoints (the dots) and serve mostly for visualization purposes (it just looks prettier).

Time for a ToDo!

    ToDo (1 point): From the the variable <tt>f_img_data</tt>, extract the odd timepoints (i.e., 1, 3, 5, etc.) of this 4D nummpy and store it in a new variable called <tt>f_img_data_odd</tt>. This should be of shape $80 \times 80 \times 44 \times 25$.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_1 import test_extract_voxel_timeseries
if 'f_img_data_odd' not in dir():
    raise ValueError("Could not find any variable named 'f_img_odd'; did you spell it correctly?")

test_extract_voxel_timeseries(f_img_data, f_img_data_odd)
```

### The affine
Each nifti file contains, in addition to the (meta)data, als an *affine* matrix, which relates the position of the image coordinates to real word coordinates. This may sound vague (it definitely did to us at first!), so let's sketch a scenario in which the image's affine becomes important.

Suppose you were given a nifti file by a colleague (we'll pretend that the `img` variable represents this mysterious nifti file). You have no idea whether it's even a brain scan, so you decide to plot three slices - one from each of the three axes. You pick slice 70 (index 69, because Python is zero-based; first axis), 100 (second axis), and again 100 (third axis).

```python
fig, ax = plt.subplots(ncols=3, figsize=(15, 5))

ax[0].imshow(img_data[69, :, :].T, origin='lower', cmap='gray')
ax[0].set_xlabel('Second dim voxel coords.', fontsize=12)
ax[0].set_ylabel('Third dim voxel coords', fontsize=12)
ax[0].set_title('First dimension, slice nr. 70', fontsize=15)

ax[1].imshow(img_data[:, 99, :].T, origin='lower', cmap='gray')
ax[1].set_xlabel('First dim voxel coords.', fontsize=12)
ax[1].set_ylabel('Third dim voxel coords', fontsize=12)
ax[1].set_title('Second dimension, slice nr. 100', fontsize=15)

ax[2].imshow(img_data[:, :, 99].T, origin='lower', cmap='gray')
ax[2].set_xlabel('First dim voxel coords.', fontsize=12)
ax[2].set_ylabel('Second dim voxel coords', fontsize=12)
ax[2].set_title('Third dimension, slice nr. 100', fontsize=15)

fig.tight_layout()
```

Alright, so, you figured out it's definitely a (structural) MRI scan.You can clearly see that the first voxel axis represents the sagittal dimension (left ←→ right), the second voxel axis represents the coronal dimension (anterior ←→ posterior), and the third voxel axis represents the axial dimension (inferior ←→ superior).

So far, so good!

Now, if we were to ask you: "In the first plot, are we looking at the left or right side of the brain?" Due to the left-right symmetry of the brain, this is actually impossible to tell! As such, we need a way to tell what is left and right (and anterior/posterior and inferior/superior) in our voxel space - this is what the affine is for! It will tell us, basically, whether "left" in our "voxel space" is also "left" in the "real world".

Here, the real world space refers to the position of the voxels (in millimeters) *relative to the isocenter of the scanner*. In this world space, the coordinate $(0, 0, 0)$ refers to the isocenter (see image below). Furthermore, nibabel `Nifti1Images` assume that the first axis goes from left to right, the second axis goes posterior to anterior, and the third axis from inferior to superior. A short-hand for this convention is *RAS+*, which basically says that coordinates to the Right, Anterior, and Superior of the isocenter should be positive (+). It follows that coordinates to the left, posterior, and inferior of the isocenter should be negative (-).

(Source: Graham Wideman, [link](http://www.grahamwideman.com/gw/brain/fs/coords/fscoords.htm))

Alright, but how does the affine help us recover the real word coordinates of our voxels? It turns out you only need a simple matrix operation: for a set of voxel coordinates $(i, j, k)$, appended with a single 1, and an $4 \times 4$ affine matrix $A$, you can get the real word coordinates (in RAS+) by the dot product (matrix multiplication) of $A$ and $(i, j, k, 1)$:

\begin{align}
\begin{bmatrix} x & y & z & 1 \end{bmatrix} = A\begin{bmatrix}
           i \\
           j \\
           k \\
           1
         \end{bmatrix}
\end{align}

So, our image's affine should be a $4 \times 4$ matrix. Let's see whether that's the case. The affine of a `Nifti1Image` is an attribute named, well, `affine`, and is a 2D numpy array:

```python
np.set_printoptions(suppress=True, precision=3)  # suppresses scientific notation
A = img.affine
print(A)
```

Now, suppose that you want to know whether the sagittal slice from the leftmost plot in the previous figure is on the left side or the right side. So we want to know the real world $x$ coordinate for our voxel coordinate $i=69$. Let's pick the middle voxel index for our other two dimensions ($j=119, k=109$) (we're plotting this 3D coordinate below in red).

```python
import matplotlib.patches as patches
fig, ax = plt.subplots(ncols=3, figsize=(15, 5))

ax[0].imshow(img_data[69, :, :].T, origin='lower', cmap='gray')
ax[0].set_xlabel('Second dim voxel coords.', fontsize=12)
ax[0].set_ylabel('Third dim voxel coords', fontsize=12)
ax[0].set_title('First dimension, slice nr. 70', fontsize=15)
rect = patches.Rectangle((119, 109), 3, 3, linewidth=2, edgecolor='r', facecolor='none')
ax[0].add_patch(rect)

ax[1].imshow(img_data[:, 99, :].T, origin='lower', cmap='gray')
ax[1].set_xlabel('First dim voxel coords.', fontsize=12)
ax[1].set_ylabel('Third dim voxel coords', fontsize=12)
ax[1].set_title('Second dimension, slice nr. 100', fontsize=15)
rect = patches.Rectangle((69, 109), 3, 3, linewidth=2, edgecolor='r', facecolor='none')
ax[1].add_patch(rect)

ax[2].imshow(img_data[:, :, 99].T, origin='lower', cmap='gray')
ax[2].set_xlabel('First dim voxel coords.', fontsize=12)
ax[2].set_ylabel('Second dim voxel coords', fontsize=12)
ax[2].set_title('Third dimension, slice nr. 100', fontsize=15)
rect = patches.Rectangle((69, 119), 3, 3, linewidth=2, edgecolor='r', facecolor='none')
ax[2].add_patch(rect)

fig.tight_layout()
```

To get the real world coordinate, we need to evaluate the following:
\begin{align}
x, y, z, 1 = A\begin{bmatrix}
           69 \\
           119 \\
           109 \\
           1
         \end{bmatrix}
\end{align}

Let's do that below:

```python
xyz1 = A @ np.array([69, 119, 109, 1])
print(xyz1)
```

Here we see that the real world $x$ coordinate is 48.472, which means that this coordinate is 48.472 millimeters to the *right* of the isocenter (remember *RAS+*!).

So, why is this important, you might ask? Well, actually, for your statistical analysis, it doesn't! The general linear model (GLM), which we'll use as our main statistical workhorse in this course, doesn't care whether a voxel is on the left or right side of the brain. But when it comes to visualization and reporting, it does! Suppose that you find unilateral amygdala activity for a particular experimental condition - you want to know whether this is on the left or right side, both for your figures and for your paper that you might write about it!

Moreover, the affine will become important when we'll talk about realignment of different fMRI volumes (in motion correction) and registration of the fMRI volumes to the structural MRI volume or standard space (which might be in different orientations!), which is the topic of week 4.

ToDo (1 point):

Suppose that you want to know whether the how much the middle of the image (i.e., the center in voxel coordinates) deviates from the isocenter of the scanner. In other words, you want to check whether the center of the image was in the isocenter of the scanner. You should do this for the <tt>img_data</tt> array!

Store these world coordinates in a new array named <tt>xyz_middle</tt>, which should be of size 3 (representing the three coordinates, so make sure to remove the extra <tt>1</tt>). Remember, indexing is Python is zero-based (so the first index is 0).

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo '''
from niedu.tests.nii.week_1 import test_affine_middle_coords
if len(xyz_middle) != 3:
    raise ValueError("Did you remove the last 1?")

test_affine_middle_coords(A, xyz_middle)
```

ToDo (3 points):

Alright, almost done! Let's practice some of the skills that you learned in this tutorial! For this assignment, you goal is to plot an histogram of the average activity value across time for all non-zero voxels in the brain from the <tt>f_img_data</tt> array. So, what you have to do is the following:

1. Average the <tt>func_img_data</tt> variable across time (resulting array should by of shape (80, 80, 44)
2. Remove all voxels which are zero (use boolean indexing!)
3. Make a histogram of all the resulting non-zero voxels (make sure the histogram has 50 bins)

Do this in the code-cell below. Also, give the plot sensible labels for the axes. This is a manually graded assignment, so there is no test-cell/feedback.

The resulting histogram should show a "bimodal distribution" of average activity values — roughly with a peak around $x = 0$ and a peak round $x = 42,000$.

```python
# Implement to ToDo here

# YOUR CODE HERE
raise NotImplementedError()
```

Alright, that's it! You finished the notebook of this week!

    Tip: In this notebook, we used several basic Python packages for data processing, analysis, manipulation and plotting (such as numpy, scipy, and matplotlib). We'll make heavy use of this "stack" for the rest of the course. Many procedures we'll discuss in the rest of the course (such as fMRI preprocessing and analysis) can also, and more efficiently, be implemented using Nilearn, a Python package for fast and easy statistical learning on neuroimaging data. This package and its functionality is discussed in detail in week 7.


```python
# The paper examines return-illiquidity relationship over time.
# Over time, ex ante stock excess return is increasing in the expected illiquidity of the stock market
# Illiquidity ratio: ILLIQ defined as the daily ratio of absolute stock return to dollar volume, average over xx days
# The ratio gives the daily price impact of the order flow
# consider verify the findings using spreads and PIN
```

\begin{align}
ILLILQ_{iy} = \frac{1}{D_{iy}}*\sum \limits_{t=1}^{D_{iy}} \frac{\mid{R_{iyd}}\mid}{VOLD_{ivyd}}
\end{align}

where $D_{iy}$ is the number of trading days for stock $i$ in year $y$.

```python
import pandas as pd
import numpy as np
import wrds
from appelpy.linear_model import OLS

```

    /usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```python
!pip install appelpy
```

    Collecting appelpy
      Downloading https://files.pythonhosted.org/packages/0a/58/ee586ae485138fadddc798aca51e30375538c080dbf814bbfb227c419ffd/appelpy-0.4.2-py3-none-any.whl
    Requirement already satisfied: seaborn>=0.9 in /usr/local/lib/python3.7/dist-packages (from appelpy) (0.11.1)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from appelpy) (2.11.3)
    Requirement already satisfied: statsmodels>=0.9 in /usr/local/lib/python3.7/dist-packages (from appelpy) (0.10.2)
    Requirement already satisfied: pandas>=0.24 in /usr/local/lib/python3.7/dist-packages (from appelpy) (1.1.5)
    Requirement already satisfied: patsy in /usr/local/lib/python3.7/dist-packages (from appelpy) (0.5.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from appelpy) (1.4.1)
    Requirement already satisfied: matplotlib>=3 in /usr/local/lib/python3.7/dist-packages (from appelpy) (3.2.2)
    Requirement already satisfied: numpy>=1.16 in /usr/local/lib/python3.7/dist-packages (from appelpy) (1.19.5)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->appelpy) (2.0.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->appelpy) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->appelpy) (2.8.1)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from patsy->appelpy) (1.15.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3->appelpy) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3->appelpy) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3->appelpy) (0.10.0)
    Installing collected packages: appelpy
    Successfully installed appelpy-0.4.2

```python
# sample period: 1963-1997
# sample: NYSE-traded stocks (i.e., EXCHCD==1 or 31)

#connect to wrds
conn=wrds.Connection()

```

    Enter your WRDS username [root]:zz2585
    Enter your password:··········
    WRDS recommends setting up a .pgpass file.
    You can find more info here:
    https://www.postgresql.org/docs/9.5/static/libpq-pgpass.html.
    Loading library list...
    Done

```python
# get crsp monthly data
crsp_m = conn.raw_sql("""
                      select a.permno, a.permco, a.date, b.shrcd, b.exchcd,
                      a.ret, a.retx, a.shrout, a.prc, a.vol
                      from crsp.msf as a
                      left join crsp.msenames as b
                      on a.permno=b.permno
                      and b.namedt<=a.date
                      and a.date<=b.nameendt
                      where a.date between '01/01/1963' and '12/31/1997'
                      and b.exchcd =1 or b.exchcd =31
                      """, date_cols=['date'])

# get delisting return
dlret = conn.raw_sql("""
                     select permno, dlret, dlstdt, dlstcd
                     from crsp.msedelist
                     """, date_cols=['dlstdt'])

```

```python
crsp.describe()
```

      permno
      permco
      shrcd
      exchcd
      ret
      retx
      shrout
      prc
      vol
      dlret
      dlstcd
      retadj
      ILLIQ
      yr
      key

      count
      660990.000000
      660990.000000
      660990.000000
      660990.000000
      660990.000000
      658877.000000
      6.609900e+05
      660990.000000
      6.609900e+05
      319.000000
      325.000000
      660990.000000
      6.609900e+05
      660990.000000
      660990.000000

      mean
      41194.329541
      20313.578299
      12.720362
      1.001589
      0.012528
      0.009616
      3.208344e+04
      29.933656
      1.582012e+04
      0.015079
      265.818462
      0.012518
      9.004769e-06
      1981.944123
      43176.273664

      std
      21308.362148
      6630.289444
      8.627656
      0.218296
      0.110178
      0.110489
      7.991515e+04
      281.348613
      4.555754e+04
      0.162957
      99.842906
      0.110184
      4.505744e-04
      10.242245
      21313.968353

      min
      10006.000000
      4.000000
      10.000000
      1.000000
      -0.937500
      -0.937500
      0.000000e+00
      0.015625
      1.000000e+00
      -1.000000
      200.000000
      -1.000000
      0.000000e+00
      1926.000000
      11969.000000

      25%
      22787.000000
      20549.000000
      10.000000
      1.000000
      -0.042893
      -0.046414
      4.702000e+03
      13.125000
      9.310000e+02
      -0.000262
      231.000000
      -0.042869
      1.160560e-07
      1973.000000
      24763.000000

      50%
      39925.000000
      21430.000000
      11.000000
      1.000000
      0.006329
      0.000000
      1.122400e+04
      22.250000
      3.121000e+03
      0.008065
      231.000000
      0.006329
      6.647486e-07
      1983.000000
      41905.000000

      75%
      57665.000000
      23126.000000
      11.000000
      1.000000
      0.060606
      0.057935
      2.914000e+04
      34.625000
      1.161100e+04
      0.028826
      241.000000
      0.060606
      3.161549e-06
      1991.000000
      59658.750000

      max
      93201.000000
      56285.000000
      71.000000
      31.000000
      11.000000
      11.000000
      3.272726e+06
      47200.000000
      2.641605e+06
      1.250000
      588.000000
      11.000000
      2.666667e-01
      1999.000000
      95191.000000

```python
# change variable format to int
crsp_m[['permco','permno','shrcd','exchcd']]=crsp_m[['permco','permno','shrcd','exchcd']].astype(int)

# Line up date to be end of month
crsp_m['jdate']=crsp_m['date']+pd.offsets.MonthEnd(0)

dlret['dlstcd']=dlret['dlstcd'].astype(int)
dlret['permno']=dlret['permno'].astype(int)
dlret['dlstdt']=pd.to_datetime(dlret['dlstdt'])
dlret['jdate']=dlret['dlstdt']+pd.offsets.MonthEnd(0)

crsp = pd.merge(crsp_m, dlret, how='left',on=['permno','jdate'])

crsp['ret']=crsp['ret'].fillna(0)

```

```python
# adjust for delisting returns following Shumway(1997)
crsp['retadj'] = np.where(crsp.dlstcd.isna(), crsp.ret, np.nan)
crsp['retadj'] = np.where((crsp.dlstcd.notna())&(crsp.dlret.notna()), crsp.dlret, crsp['retadj'])
crsp['retadj'] = np.where((crsp.dlstcd==500) | (crsp.dlstcd==520)| (crsp.dlstcd==580)| (crsp.dlstcd==584)|((crsp.dlstcd >= 551) & (crsp.dlstcd <= 574)), -0.3, crsp['retadj'])
crsp['retadj'] = np.where(crsp['retadj'].isna(), -1, crsp['retadj'])

```

```python
#to be deleted
# retadj factors in the delisting returns
crsp['retadj']=(1+crsp['ret'])*(1+crsp['dlret'])-1

# calculate market equity
crsp['me']=crsp['prc'].abs()*crsp['shrout']
crsp=crsp.drop(['dlret','dlstdt','prc','shrout'], axis=1)
crsp=crsp.sort_values(by=['jdate','permco','me'])
```

```python
# filter data based on several criteria

# drop ADRs
crsp = crsp[~((crsp.shrcd==30) | (crsp.shrcd==31))]

# drop observations where vol is null
crsp = crsp[(crsp['vol'] > 0)& (crsp['prc'] > 0)]

# calculate Amihud ILLIQ measure
crsp['ILLIQ'] = abs(crsp['ret']) / (crsp['prc'] * crsp['vol'])

# (i) The stock has return and volume data for more than 200 days during year y. Also, the stock must be listed at the end of year y.
# zip permno and delist year in both dataset to remove delist firms
dlret['key'] = dlret['permno'] + dlret['dlstdt'].dt.year
crsp['key'] = crsp['permno'] + crsp['date'].dt.year
crsp_1 = crsp[~crsp.key.isin(dlret.key)].reset_index().drop(['jdate','dlret','dlstdt','dlstcd','index', 'key'],axis=1)

# count CRSP daily observations
crsp_d = conn.raw_sql(f"""
                        select permno, date, prc, abs(prc*shrout) as mcap, ret from crsp.dsf
                        where date between '01/01/1963' and '12/31/1997'
                        """, date_cols = ['date'])

crsp_1['yr'] = crsp_1['date'].dt.year
obs = crsp_1.groupby(['yr', 'permno']).date.count().reset_index()

# (ii) The stock price is greater than $5 at the end of year y.
# (iii) The stock has data on market capitalization at the end of year y 􏰣 1 in the CRSP database.
# (iv) Stock-year ILLIQ is winsorized at the 1% level

```

```python
df_trading_days_200 = df_counts[df_counts['counts'] > 200][['year', 'PERMNO']].reset_index(drop=True)
df_div_amihud = df_notnull.groupby(['year', 'PERMNO']).sum().reset_index()[['year', 'PERMNO', 'amihud_d', 'DIVAMT']]
df_div_amihud['counts'] = df_notnull.groupby(['year', 'PERMNO']).size().reset_index(name='counts')['counts']
df_div_amihud['sdret'] = df_notnull.groupby(['year', 'PERMNO']).std().reset_index()['RET'] * 100
df_step1 = df_trading_days_200.merge(df_div_amihud, 'left', on=['year', 'PERMNO'])
```


# Inference example for trained 2D U-Net model on BraTS.
Takes a trained model and performs inference on a few validation examples.

```python
import sys
import platform
import os

print("Python version: {}".format(sys.version))
print("{}".format(platform.platform()))
```

    Python version: 3.7.10 (default, Feb 26 2021, 18:47:35)
    [GCC 7.3.0]
    Linux-5.11.0-051100rc5-generic-x86_64-with-debian-buster-sid

```python
def test_intel_tensorflow():
    """
    Check if Intel version of TensorFlow is installed
    """
    import tensorflow as tf

    print("We are using Tensorflow version {}".format(tf.__version__))

    major_version = int(tf.__version__.split(".")[0])
    if major_version >= 2:
       from tensorflow.python import _pywrap_util_port
       print("Intel-optimizations (DNNL) enabled:", _pywrap_util_port.IsMklEnabled())
    else:
       print("Intel-optimizations (DNNL) enabled:", tf.pywrap_tensorflow.IsMklEnabled())

test_intel_tensorflow()
```

    We are using Tensorflow version 2.4.1
    Intel-optimizations (DNNL) enabled: True

```python
saved_model_dir = "./output/2d_unet_decathlon"
```

```python
# Create output directory for images
png_directory = "inference_examples"
if not os.path.exists(png_directory):
    os.makedirs(png_directory)

model_filename = os.path.join(saved_model_dir)
```

#### Define the DICE coefficient and loss function

The Sørensen–Dice coefficient is a statistic used for comparing the similarity of two samples. Given two sets, X and Y, it is defined as

\begin{equation}
dice = \frac{2|X\cap Y|}{|X|+|Y|}
\end{equation}

```python
import numpy as np

def calc_dice(target, prediction, smooth=0.0001):
    """
    Sorensen Dice coefficient
    """
    prediction = np.round(prediction)

    numerator = 2.0 * np.sum(target * prediction) + smooth
    denominator = np.sum(target) + np.sum(prediction) + smooth
    coef = numerator / denominator

    return coef

def calc_soft_dice(target, prediction, smooth=0.0001):
    """
    Sorensen (Soft) Dice coefficient - Don't round predictions
    """
    numerator = 2.0 * np.sum(target * prediction) + smooth
    denominator = np.sum(target) + np.sum(prediction) + smooth
    coef = numerator / denominator

    return coef
```

## Inference Time!

Inferencing in this example can be done in 3 simple steps:
1. Load the data
1. Load the Keras model
1. Perform a `model.predict` on an input image (or set of images)

#### Step 1 : Load data

```python
data_path = "/data/medical_decathlon/Task01_BrainTumour/"

crop_dim=128  # Original resolution (240)
batch_size = 128
seed=816
train_test_split=0.85
```

```python
from dataloader import DatasetGenerator, get_decathlon_filelist

trainFiles, validateFiles, testFiles = get_decathlon_filelist(data_path=data_path, seed=seed, split=train_test_split)

ds_test = DatasetGenerator(testFiles,
                           batch_size=batch_size,
                           crop_dim=[crop_dim, crop_dim],
                           augment=False,
                           seed=seed)

```

    ******************************
    ==============================
    Dataset name:         BRATS
    Dataset description:  Gliomas segmentation tumour and oedema in on brain images
    Tensor image size:    4D
    Dataset release:      2.0 04/05/2018
    Dataset reference:    https://www.med.upenn.edu/sbia/brats2017.html
    Dataset license:      CC-BY-SA 4.0
    ==============================
    ******************************
    Number of training files   = 411
    Number of validation files = 36
    Number of testing files    = 37

#### Step 2 : Load the model

```python
from model import unet

from tensorflow import keras as K
model = K.models.load_model(saved_model_dir, compile=False, custom_objects=unet().custom_objects)
```

    Data format = channels_last

#### Step 3: Perform prediction on some images.
The prediction results will be saved in the output directory for images, which is defined by the `png_directory` variable.

```python
import matplotlib.pyplot as plt
%matplotlib inline

import time

def plot_results(ds):

    plt.figure(figsize=(10,10))

    img, msk = next(ds.ds)

    idx = np.argmax(np.sum(np.sum(msk[:,:,:,0], axis=1), axis=1)) # find the slice with the largest tumor

    plt.subplot(1, 3, 1)
    plt.imshow(img[idx, :, :, 0], cmap="bone", origin="lower")
    plt.title("MRI {}".format(idx), fontsize=20)

    plt.subplot(1, 3, 2)
    plt.imshow(msk[idx, :, :], cmap="bone", origin="lower")
    plt.title("Ground truth", fontsize=20)

    plt.subplot(1, 3, 3)

    print("Index {}: ".format(idx), end="")

    # Predict using the TensorFlow model
    start_time = time.time()
    prediction = model.predict(img[[idx]])
    print("Elapsed time = {:.4f} msecs, ".format(1000.0*(time.time()-start_time)), end="")

    plt.imshow(prediction[0,:,:,0], cmap="bone", origin="lower")
    dice_coef = calc_dice(msk[idx], prediction)
    print("Dice coefficient = {:.4f}, ".format(dice_coef), end="")
    plt.title("Prediction\nDice = {:.4f}".format(dice_coef), fontsize=20)

```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

```python
plot_results(ds_test)
```

# Can we perform inference even faster? Hmm..

Let's find out. Move on the the next tutorial section.

*Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. SPDX-License-Identifier: EPL-2.0*

*Copyright (c) 2019-2020 Intel Corporation*

```python

```


```python
import numpy as np
import pandas as pd
import scipy, scipy.signal

from datetime import date
import time

import random
from random import seed
from random import random

import os, os.path
import shutil

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report

import matplotlib
import matplotlib.pyplot as plt
from pylab import imshow

import h5py
import sys
sys.path.append('/Users/hn/Documents/00_GitHub/Ag/NASA/Python_codes/')
import NASA_core as nc
# import NASA_plot_core.py as rcp
```

```python
from tslearn.metrics import dtw as dtw_metric

# https://dtaidistance.readthedocs.io/en/latest/usage/dtw.html
from dtaidistance import dtw
from dtaidistance import dtw_visualisation as dtwvis
```

```python

```

    0

# Read Fields Metadata

```python
meta_dir = "/Users/hn/Documents/01_research_data/NASA/parameters/"
meta = pd.read_csv(meta_dir+"evaluation_set.csv")
meta_moreThan10Acr=meta[meta.Acres>=10]
print (meta.shape)
print (meta_moreThan10Acr.shape)
meta.head(2)
```

    (6340, 8)
    (3613, 8)

      ID
      CropTyp
      Irrigtn
      DataSrc
      Acres
      ExctAcr
      LstSrvD
      county

      0
      100010_WSDA_SF_2017
      alfalfa hay
      center pivot
      wsda
      34
      34.310305
      2017/09/12
      Grant

      1
      100204_WSDA_SF_2017
      alfalfa hay
      center pivot
      wsda
      62
      61.826535
      2017/08/09
      Grant

```python

```

# Read Training Set Labels

```python
training_set_dir = "/Users/hn/Documents/01_research_data/NASA/ML_data/"
ground_truth_labels = pd.read_csv(training_set_dir+"train_labels.csv")
print ("Unique Votes: ", ground_truth_labels.Vote.unique())
print (len(ground_truth_labels.ID.unique()))
ground_truth_labels.head(2)
```

    Unique Votes:  [2 1]
    1849

      ID
      Vote

      0
      99837_WSDA_SF_2017
      2

      1
      114615_WSDA_SF_2017
      1

# Read the Data

```python
VI_idx = "EVI"
data_dir = "/Users/hn/Documents/01_research_data/NASA/VI_TS/04_regularized_TS/"
```

```python
file_names = ["regular_Walla2015_EVI_JFD.csv", "regular_AdamBenton2016_EVI_JFD.csv",
              "regular_Grant2017_EVI_JFD.csv", "regular_FranklinYakima2018_EVI_JFD.csv"]

data=pd.DataFrame()

for file in file_names:
    curr_file=pd.read_csv(data_dir + file)
    curr_file['human_system_start_time'] = pd.to_datetime(curr_file['human_system_start_time'])

    # These data are for 3 years. The middle one is the correct one
    all_years = sorted(curr_file.human_system_start_time.dt.year.unique())
    if len(all_years)==3 or len(all_years)==2:
        proper_year = all_years[1]
    elif len(all_years)==1:
        proper_year = all_years[0]

    curr_file = curr_file[curr_file.human_system_start_time.dt.year==proper_year]
    data=pd.concat([data, curr_file])

data.reset_index(drop=True, inplace=True)
data.head(2)
```

      ID
      human_system_start_time
      EVI

      0
      135073_WSDA_SF_2015
      2015-01-10
      0.049624

      1
      135073_WSDA_SF_2015
      2015-01-20
      0.026503

```python
ground_truth = data[data.ID.isin(list(ground_truth_labels.ID.unique()))].copy()
```

# Toss small fields

```python
ground_truth = ground_truth[ground_truth.ID.isin((list(meta_moreThan10Acr.ID)))].copy()
ground_truth_labels = ground_truth_labels[ground_truth_labels.ID.isin((list(meta_moreThan10Acr.ID)))].copy()

ground_truth.reset_index(drop=True, inplace=True)
ground_truth_labels.reset_index(drop=True, inplace=True)
```

# Sort the order of time-series and experts' labels identically

```python
ground_truth.sort_values(by=["ID", 'human_system_start_time'], inplace=True)
ground_truth_labels.sort_values(by=["ID"], inplace=True)

ground_truth.reset_index(drop=True, inplace=True)
ground_truth_labels.reset_index(drop=True, inplace=True)

assert (len(ground_truth.ID.unique()) == len(ground_truth_labels.ID.unique()))

print (list(ground_truth.ID)[0])
print (list(ground_truth_labels.ID)[0])
print ("____________________________________")
print (list(ground_truth.ID)[-1])
print (list(ground_truth_labels.ID)[-1])
print ("____________________________________")
print (list(ground_truth.ID.unique())==list(ground_truth_labels.ID.unique()))
```

    100048_WSDA_SF_2017
    100048_WSDA_SF_2017
    ____________________________________
    99909_WSDA_SF_2017
    99909_WSDA_SF_2017
    ____________________________________
    True

```python
mins = ground_truth.groupby("ID")[VI_idx].min()
maxs = ground_truth.groupby("ID")[VI_idx].max()
_ranges = maxs-mins
_ranges = pd.DataFrame(_ranges)
_ranges.reset_index(inplace=True)

mins = pd.DataFrame(mins)
mins.reset_index(inplace=True)

_ranges.rename(columns = {'EVI':'EVI_range'}, inplace = True)
mins.rename(columns = {'EVI':'EVI_min'}, inplace = True)

_ranges.head(2)
```

      ID
      EVI_range

      0
      100048_WSDA_SF_2017
      0.602470

      1
      100081_WSDA_SF_2017
      0.721754

```python
ground_truth = pd.merge(ground_truth, _ranges, on=['ID'], how='left')
ground_truth = pd.merge(ground_truth, mins, on=['ID'], how='left')
ground_truth["EVI_ratio"] = (ground_truth["EVI"]-ground_truth["EVI_min"])/ground_truth["EVI_range"]
ground_truth.head(2)
```

      ID
      human_system_start_time
      EVI
      EVI_range
      EVI_min
      EVI_ratio

      0
      100048_WSDA_SF_2017
      2017-01-06
      0.0
      0.60247
      0.0
      0.0

      1
      100048_WSDA_SF_2017
      2017-01-16
      0.0
      0.60247
      0.0
      0.0

```python
ID_1=ground_truth.ID.unique()[0]
ID_2=ground_truth.ID.unique()[12]

# _minimum = ground_truth[ground_truth.ID==ID_1].EVI.values.min()
# _range = ground_truth[ground_truth.ID==ID_1].EVI.values.max()-_minimum
# y_1 = (ground_truth[ground_truth.ID==ID_1].EVI.values-_minimum)/_range
# _minimum = ground_truth[ground_truth.ID==ID_2].EVI.values.min()
# _range = ground_truth[ground_truth.ID==ID_2].EVI.values.max()-_minimum
# y_2 = (ground_truth[ground_truth.ID==ID_2].EVI.values-_minimum)/_range

dtw_score = dtw_metric(ground_truth[ground_truth.ID==ID_1].EVI.values,
                ground_truth[ground_truth.ID==ID_2].EVI.values)

dtw_score_ratios = dtw_metric(ground_truth[ground_truth.ID==ID_1].EVI_ratio.values,
                       ground_truth[ground_truth.ID==ID_2].EVI_ratio.values)

# print ("dtw score is {:.2f}.".format(dtw_score))
plt.figure(figsize=(20,4))
plt.subplot(1, 1, 1)

# plot EVIs
plt.plot(range(len(ground_truth[ground_truth.ID==ID_1].human_system_start_time.values)),
         ground_truth[ground_truth.ID==ID_1].EVI.values,
        c='k', linewidth=5);

plt.plot(range(len(ground_truth[ground_truth.ID==ID_2].human_system_start_time.values)),
         ground_truth[ground_truth.ID==ID_2].EVI.values,
        c='red', linewidth=5);

# plot ratios
plt.plot(range(len(ground_truth[ground_truth.ID==ID_1].human_system_start_time.values)),
         ground_truth[ground_truth.ID==ID_1].EVI_ratio.values,
         ".-.", c='k', linewidth=2, label="ratios");

plt.plot(range(len(ground_truth[ground_truth.ID==ID_2].human_system_start_time.values)),
         ground_truth[ground_truth.ID==ID_2].EVI_ratio.values,
         ".-.", c='red', linewidth=2, label="ratios");

title = "dtw score for EVI is [{:.2f}] and for ratios is [{:.2f}].".format(dtw_score, dtw_score_ratios)
plt.ylim([-0.2, 1.05]);
plt.title(title , fontsize = 20);
```

```python
s1=ground_truth[ground_truth.ID==ID_1].EVI_ratio.values
s2=ground_truth[ground_truth.ID==ID_2].EVI_ratio.values
path = dtw.warping_path(s1, s2)
dtwvis.plot_warping(s1, s2, path)
distance = dtw.distance(s1, s2)
```

```python
# d, paths = dtw.warping_paths(s1, s2, window=10, use_pruning=True);
# best_path = dtw.best_path(paths);
# dtwvis.plot_warpingpaths(s1, s2, paths, best_path);
```

# Widen Ground Truth Table

```python
EVI_colnames = [VI_idx + "_" + str(ii) for ii in range(1, 37) ]
columnNames = ["ID"] + EVI_colnames
ground_truth_wide = pd.DataFrame(columns=columnNames,
                             index=range(len(ground_truth.ID.unique())))
ground_truth_wide["ID"] = ground_truth.ID.unique()

for an_ID in ground_truth.ID.unique():
    curr_df = ground_truth[ground_truth.ID==an_ID]

    ground_truth_wide_indx = ground_truth_wide[ground_truth_wide.ID==an_ID].index
    ground_truth_wide.loc[ground_truth_wide_indx, "EVI_1":"EVI_36"] = curr_df.EVI.values[:36]
```

```python
ground_truth_wide.head(2)
```

      ID
      EVI_1
      EVI_2
      EVI_3
      EVI_4
      EVI_5
      EVI_6
      EVI_7
      EVI_8
      EVI_9
      ...
      EVI_27
      EVI_28
      EVI_29
      EVI_30
      EVI_31
      EVI_32
      EVI_33
      EVI_34
      EVI_35
      EVI_36

      0
      100048_WSDA_SF_2017
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.072227
      0.088491
      ...
      0.111507
      0.119534
      0.13248
      0.061173
      0.055843
      0.050513
      0.045183
      0.039854
      0.045165
      0.050476

      1
      100081_WSDA_SF_2017
      0.014832
      0.167038
      0.319244
      0.47145
      0.378153
      0.284855
      0.198468
      0.290683
      0.398445
      ...
      0.096717
      0.117915
      0.159058
      0.121182
      0.060591
      0.0
      0.050004
      0.100009
      0.102269
      0.104529

2 rows × 37 columns

# Split Train and Test Set

#### Make sure rows of ```ground_truth_allBands``` and ```ground_truth_labels``` are in the same order

```python
ground_truth_labels.head(2)
```

      ID
      Vote

      0
      100048_WSDA_SF_2017
      1

      1
      100081_WSDA_SF_2017
      1

```python
ground_truth.head(2)
```

      ID
      human_system_start_time
      EVI
      EVI_range
      EVI_min
      EVI_ratio

      0
      100048_WSDA_SF_2017
      2017-01-06
      0.0
      0.60247
      0.0
      0.0

      1
      100048_WSDA_SF_2017
      2017-01-16
      0.0
      0.60247
      0.0
      0.0

```python
ground_truth_labels = ground_truth_labels.set_index('ID')
ground_truth_labels = ground_truth_labels.reindex(index=ground_truth_wide['ID'])
ground_truth_labels = ground_truth_labels.reset_index()
```

```python
ground_truth_labels.head(2)
```

      ID
      Vote

      0
      100048_WSDA_SF_2017
      1

      1
      100081_WSDA_SF_2017
      1

```python
x_train_df, x_test_df, y_train_df, y_test_df = train_test_split(ground_truth_wide,
                                                                ground_truth_labels,
                                                                test_size=0.2,
                                                                random_state=0,
                                                                shuffle=True,
                                                                stratify=ground_truth_labels.Vote.values)
```

```python
x_train_df.shape
```

    (1086, 37)

# Start SVM

# Definitions

  - **Precision** Of all instances we predict $\hat y = 1$, what fraction is actually 1.
     \begin{equation}\label{eq:precision}
        \text{Precision} = \frac{TP}{TP + FP}
     \end{equation}

  - **Recall** Of all instances that are actually $y = 1$, what fraction we predict 1.
     \begin{equation}\label{eq:recall}
         \text{Recall} = \text{TPR} = \frac{TP}{TP + FN}
     \end{equation}

  - **Specifity** Fraction of all negative instances that are incorrectly predicted positive.
     \begin{equation}\label{eq:specifity}
        \text{Specifity} = \text{FPR} = \frac{FP}{TN + FP}\\
     \end{equation}

  - **F-Score** Adjust $\beta$ for trade off between  precision and recall. For precision oriented task $\beta = 0.5$.
     \begin{equation}\label{eq:Fscore}
        F_\beta = \frac{(1+\beta^2) TP}{ (1+\beta^2) TP + \beta^2 FN + FP}
     \end{equation}

```python
from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC
```

```python
def DTW_prune(ts1, ts2):
    d,_ = dtw.warping_paths(ts1, ts2, window=10, use_pruning=True);
    return d
```

```python
parameters = {'C':[5, 10, 13, 14, 15, 16, 17, 20, 40, 80],
              'kernel':['linear', 'poly', 'rbf', 'sigmoid'] # 'linear', 'poly', 'rbf', 'sigmoid', 'precomputed'
              } # ,
SVM_classifier_balanced_00 = GridSearchCV(SVC(random_state=0, class_weight='balanced'),
                                          parameters, cv=5, verbose=1)

SVM_classifier_balanced_00.fit(x_train_df.iloc[:, 1:], y_train_df.Vote.values)

print (SVM_classifier_balanced_00.best_params_)
print (SVM_classifier_balanced_00.best_score_)
```

    Fitting 5 folds for each of 40 candidates, totalling 200 fits
    {'C': 5, 'kernel': 'rbf'}
    0.9696317591848814

```python

```

```python
parameters = {'C':[5, 10, 13, 14, 15, 16, 17, 20, 40, 80],
              'kernel':['linear', 'poly', 'rbf', 'sigmoid'] # 'linear', 'poly', 'rbf', 'sigmoid', 'precomputed'
              } # ,
SVM_classifier_NoneWeight_00 = GridSearchCV(SVC(random_state=0),
                                 parameters, cv=5, verbose=1)

SVM_classifier_NoneWeight_00.fit(x_train_df.iloc[:, 1:], y_train_df.Vote.values)

print (SVM_classifier_NoneWeight_00.best_params_)
print (SVM_classifier_NoneWeight_00.best_score_)
```

    Fitting 5 folds for each of 40 candidates, totalling 200 fits
    {'C': 10, 'kernel': 'rbf'}
    0.9723967361434067

```python
SVM_classifier_NoneWeight_00_predictions = SVM_classifier_NoneWeight_00.predict(x_test_df.iloc[:, 1:])
SVM_classifier_balanced_00_predictions = SVM_classifier_balanced_00.predict(x_test_df.iloc[:, 1:])
```

```python
balanced_y_test_df=y_test_df.copy()
None_y_test_df=y_test_df.copy()
balanced_y_test_df["prediction"] = list(SVM_classifier_balanced_00_predictions)
balanced_y_test_df.head(2)
```

      ID
      Vote
      prediction

      1321
      98828_WSDA_SF_2017
      1
      1

      299
      105536_WSDA_SF_2018
      1
      1

```python
None_y_test_df=y_test_df.copy()
None_y_test_df=y_test_df.copy()
None_y_test_df["prediction"] = list(SVM_classifier_NoneWeight_00_predictions)
None_y_test_df.head(2)
```

      ID
      Vote
      prediction

      1321
      98828_WSDA_SF_2017
      1
      1

      299
      105536_WSDA_SF_2018
      1
      1

```python
# Balanced Confusion Table

true_single_predicted_single=0
true_single_predicted_double=0

true_double_predicted_single=0
true_double_predicted_double=0

for index in balanced_y_test_df.index:
    curr_vote=list(balanced_y_test_df[balanced_y_test_df.index==index].Vote)[0]
    curr_predict=list(balanced_y_test_df[balanced_y_test_df.index==index].prediction)[0]
    if curr_vote==curr_predict:
        if curr_vote==1:
            true_single_predicted_single+=1
        else:
            true_double_predicted_double+=1
    else:
        if curr_vote==1:
            true_single_predicted_double+=1
        else:
            true_double_predicted_single+=1

balanced_confus_tbl_test = pd.DataFrame(columns=['None', 'Predict_Single', 'Predict_Double'],
                               index=range(2))
balanced_confus_tbl_test.loc[0, 'None'] = 'Actual_Single'
balanced_confus_tbl_test.loc[1, 'None'] = 'Actual_Double'
balanced_confus_tbl_test['Predict_Single']=0
balanced_confus_tbl_test['Predict_Double']=0

balanced_confus_tbl_test.loc[0, "Predict_Single"]=true_single_predicted_single
balanced_confus_tbl_test.loc[0, "Predict_Double"]=true_single_predicted_double
balanced_confus_tbl_test.loc[1, "Predict_Single"]=true_double_predicted_single
balanced_confus_tbl_test.loc[1, "Predict_Double"]=true_double_predicted_double
balanced_confus_tbl_test
```

      None
      Predict_Single
      Predict_Double

      0
      Actual_Single
      218
      4

      1
      Actual_Double
      5
      45

```python
# None Weight Confusion Table

true_single_predicted_single=0
true_single_predicted_double=0

true_double_predicted_single=0
true_double_predicted_double=0

for index in None_y_test_df.index:
    curr_vote=list(None_y_test_df[None_y_test_df.index==index].Vote)[0]
    curr_predict=list(None_y_test_df[None_y_test_df.index==index].prediction)[0]
    if curr_vote==curr_predict:
        if curr_vote==1:
            true_single_predicted_single+=1
        else:
            true_double_predicted_double+=1
    else:
        if curr_vote==1:
            true_single_predicted_double+=1
        else:
            true_double_predicted_single+=1

None_confus_tbl_test = pd.DataFrame(columns=['None', 'Predict_Single', 'Predict_Double'],
                               index=range(2))
None_confus_tbl_test.loc[0, 'None'] = 'Actual_Single'
None_confus_tbl_test.loc[1, 'None'] = 'Actual_Double'
None_confus_tbl_test['Predict_Single']=0
None_confus_tbl_test['Predict_Double']=0

None_confus_tbl_test.loc[0, "Predict_Single"]=true_single_predicted_single
None_confus_tbl_test.loc[0, "Predict_Double"]=true_single_predicted_double
None_confus_tbl_test.loc[1, "Predict_Single"]=true_double_predicted_single
None_confus_tbl_test.loc[1, "Predict_Double"]=true_double_predicted_double
None_confus_tbl_test
```

      None
      Predict_Single
      Predict_Double

      0
      Actual_Single
      218
      4

      1
      Actual_Double
      6
      44

```python
print(classification_report(y_test_df.Vote, SVM_classifier_NoneWeight_00_predictions))
```

                  precision    recall  f1-score   support

               1       0.97      0.98      0.98       222
               2       0.92      0.88      0.90        50

        accuracy                           0.96       272
       macro avg       0.94      0.93      0.94       272
    weighted avg       0.96      0.96      0.96       272

```python
print(classification_report(y_test_df.Vote, SVM_classifier_balanced_00_predictions))
```

                  precision    recall  f1-score   support

               1       0.98      0.98      0.98       222
               2       0.92      0.90      0.91        50

        accuracy                           0.97       272
       macro avg       0.95      0.94      0.94       272
    weighted avg       0.97      0.97      0.97       272

```python

```

```python
import pickle
model_dir = "/Users/hn/Documents/01_research_data/NASA/ML_Models/"
filename = model_dir + 'SVM_classifier_balanced_00.sav'
pickle.dump(SVM_classifier_balanced_00, open(filename, 'wb'))
```

```python
filename = model_dir + 'SVM_classifier_NoneWeight_00.sav'
pickle.dump(SVM_classifier_NoneWeight_00, open(filename, 'wb'))
```

```python

```