#Week 6: Dictionaries && Modules

Hello and welcome to the 6th part of the coding series. At the end of this lesson you should be able to:

*   Learn about Dictionaries.
*   Learn about Modules
*   Code your own Dictionaries.
*   Make use of existing Modules.
*   Practice using Dictionaries and Modules with previous lessons.
*   Read more technical terms.

Today you will be introduced to Dictionaries, and Modules. Adding these to our skill repertoire gets us closer to the end of the course. Be warned that as we learn more skills, the code might get slighlty longer than usual.
Remember if you have any problems or questions do not hesitate to ask the instructor ([Enrique](https://www.instagram.com/01000101.sh/?hl=en)) or to ask the hacker community on the SHPE Discord.

# Dictionaries:

A dictionary is a collection of unordered, changeable and indexed variables. You will know it is a dictionary for the use of curly braces {}. It also has the unique ability to contain **keys**, and **values**. This is extremely helpful while coding since it lets you be faster when looking for values in different problems. Lets take a look at the anatomy of a dictionary:

```
name_of_dictionary = {
    "key1" : value1,
    "key2" : value2,
    "key3" : value3
}
```

```
# Lets go ahead and start with something very familiar to you
# Contacts!
contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000
}

# lets display the contacts dictionary
print(contacts)
```

But wait, now you ran into your aunt Mary. She got a new phone, how do we add it?

```
contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000
}

# lets add a new key AND value.
contacts["Mary"] = 9561111111

# lets display the contacts dictionary
print(contacts)
```

But now you are no longer friends with Enrique, sadly, you need to delete his number. How do we do it?

```
contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000,
    "Mary"    : 9561111111
}

# Delete Enrique's key and value
del contacts["Enrique"]

print(contacts)
```

Lets make use of some of the skills learned in previous lessons to get a better grasp of Dictionaries:

```
# Loop thru the names (keys) of the dictionary using a FOR loop!

contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000,
    "Mary"    : 9561111111
}

# create loop
for x in contacts:
  print(x)
```

We can also display only the phone#'s (Values):

```
contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000,
    "Mary"    : 9561111111
}

# create loop to display phone #
for x in contacts.values():
  print(x)
```

But that's boring, we need to know BOTH the key and the values at times. How do we do it?

```
contacts = {
    "Antonio" : 9560000101,
    "Enrique" : 9560101010,
    "Carlos"  : 9561111000,
    "Mary"    : 9561111111
}

# create loop to display the Keys and the Values inside dictionary
for x,y in contacts.items():
  print(x,y)
```

Python comes preloaded with some very useful functions for Dictionaries, here are some of them:

     Function | Description
____________________
* **clear()** - Removes all the elements from the dictionary.
* **copy()** - Returns a copy of the dictionary.
* **fromkeys()**	- Returns a dictionary with the specified keys and values.
* **get()**	- Returns the value of the specified key.
* **items()**	- Returns a list containing the a tuple for each key value pair.
* **keys()**	- Returns a list containing the dictionary's keys.
* **pop()**	- Removes the element with the specified key.
* **popitem()**	- Removes the last inserted key-value pair.
* **setdefault()**	- Returns the value of the specified key. If the key does not exist: insert the key, with the specified value.
* **update()**	- Updates the dictionary with the specified key-value pairs.
* **values()**	- Returns a list of all the values in the dictionary.

Lets go thru some of the most useful examples, **Dictionaries** inside **Dictionaries**!

```
# Create 3 contact notebooks, inside a main contact dictionary

myContacts = {
    "work_contacts" : {
        "name"  : "Sheryl",
        "Phone" : 9561110000
    },

    "school_contacts" : {
        "name"  : "Jaime",
        "phone" : 9560001101
    },
    "friends_contacts" : {
        "name" : "Jerry",
        "phone" : 9561101010
    }
}

# Now display all your contacts!
print(myContacts)
```

You don't always have to do this much code to create a dictionary. Remember, reusability is part of the game. In the next exercise we make use of a 'constructor' to make a dictionary. Lets check it out:

```
# Reserved for in class example:

```

# Modules:

Hurray! This is it, the last of the skills that will serve you well beyond you could ever imagine. Modules are libraries or programs made by other people, and you can add them to your program for better functionality, as well as easy to access functions. Some of the most commonly used would be Tensorflow, Numpy, Pandas, Keras, and OS.

Anatomy of a module:

```
import name_of_module as short_name

  regular code
```

Some of these modules will required for you to download them beforehand on your computer. But once that is done, its just the same as we've been doing so far. Ready to start? lets use our first module for something fun.

We will be using the **calendar** module to display a given month given a year. If you find this interesting you can learn more about the module [here](https://docs.python.org/3/library/calendar.html).

```
# Display calendar of given month of the year

# Import the calendar module
import calendar

# Ask the user for the Month and Year!
y = int(input("Enter year: "))
m = int(input("Enter month: "))

# Display the calendar
print(calendar.month(y, m))
```

    Enter year: 2019
    Enter month: 9
       September 2019
    Mo Tu We Th Fr Sa Su
                       1
     2  3  4  5  6  7  8
     9 10 11 12 13 14 15
    16 17 18 19 20 21 22
    23 24 25 26 27 28 29
    30

Life is very random, and to mimic life situations we need to make our programs react randomly as well! In the following program we make use of a library called **random** to create a random number given a range. If you are interested in this library, you can findout more about it [here](https://docs.python.org/3/library/random.html).

```
# Import the random module
import random

# remove the comments ask the user for the numbers!
#low =  int(input("Enter the start of the range:"))
#high = int(input("Enter the end of the range: "))

# Display random number (notice this number changes everytime you run the program)
print(random.randint(0,1000))
```

What if i want to display some cool math functions? Well for that you can use **sympy**! this module can help you create and customize your formulas to display while programming. To learn more about sympy you can click [here](https://docs.sympy.org/latest/tutorial/index.html#tutorial).

```
# Import sympy module
import sympy
from sympy import *

# Declare the variables that we will use
x,y,z = symbols('x,y,z')

# Ignore this, this is just some google colab compatibility stuff.
# Only wizards read this
def custom_latex_printer(exp,**options):
    from google.colab.output._publish import javascript
    url = "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=default"
    javascript(url=url)
    return sympy.printing.latex(exp,**options)

# This is the line used to display the content
init_printing(use_latex="mathjax",latex_printer=custom_latex_printer)

# Function to be displayed
Integral(sqrt(1/x)*8, x)
```

$$\int 8 \sqrt{\frac{1}{x}}\, dx$$

Getting it so far? Awesome! Now that you see the power of python on its own, and with the extension of modules you can realize that your posibilities with it are endless! We are currently using a cloud environment to execute all of our code, and sadly not all modules work here. You can try them on your computer (just take your time installing the necessary files). However we will be focusing out attention into Data Science.

# Why Data Science?

Data Science is a rapidly growing field, and the demand for [jobs](https://www.hiringlab.org/2019/01/17/data-scientist-job-outlook/) is high. At the end of this course you will be presented with an oportunity to take an online certification from [EDX](https://www.edx.org/course/python-basics-for-data-science-2).

Next lesson we will be taking a look at examples of python used for Data Science, and many more applications (might require actual computer installation).

# See you next time!


# 3. Data Pre-processing

Data pre-processing techniques generally refer to the addition, deletion, or transformation of training set data. Different models have different sensitivities to the type of predictors in the model; *how* the predictors enter the model is also important.

The need for data pre-processing is determined by the type of model being used. Some procedures, such as tree-based models, are notably insensitive to the characteristics of the predictor data. Others, like linear regression, are not. In this chapter, a wide array of possible methodologies are discussed.

How the predictors are encoded, called *feature engineering*, can have a significant impact on model performance. Often the most effective encoding of the data is informed by the modeler's understanding of the problem and thus is not derived from any mathematical techniques.

## 3.1 Case Study: Cell Segmentation in High-Content Screening

This dataset is from Hill et al. (2007) that consists of 2019 cells. Of these cells, 1300 were judged to be poorly segmented (PS) and 719 were well segmented (WS); 1009 cells were reserved for the training set.

```python
import numpy as np
import pandas as pd

cell_segmentation = pd.read_csv("../datasets/segmentationOriginal/segmentationOriginal.csv")
```

```python
cell_segmentation.shape
```

    (2019, 120)

A first look at the dataset.

```python
cell_segmentation.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      0
      1
      207827637
      Test
      PS
      143.247705
      1
      185
      0
      15.711864
      3.954802
      ...
      12.474676
      7.609035
      2.714100
      0
      2
      2
      10.642974
      2
      42
      14

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      4
      5
      207932455
      Test
      PS
      2.887837
      2
      285
      0
      24.275735
      111.415441
      ...
      15.407972
      20.504288
      45.450457
      0
      0
      0
      13.198561
      0
      283
      159

5 rows × 120 columns

This chapter will use the training set samples to demonstrate data pre-processing techniques.

```python
cell_segmentation.groupby('Case').count()
```

      Unnamed: 0
      Cell
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      Case

      Test
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      ...
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010

      Train
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      ...
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009

2 rows × 119 columns

```python
# separate training and test data
cell_train = cell_segmentation.ix[cell_segmentation['Case'] == 'Train']
cell_test = cell_segmentation.ix[cell_segmentation['Case'] == 'Test']

cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      12
      207932484
      Train
      WS
      109.416426
      0
      256
      0
      18.828571
      125.938776
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      15
      207932459
      Train
      PS
      104.278654
      0
      258
      0
      17.570850
      124.368421
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 120 columns

## 3.2 Data Transformation for Individual Predictors

Transformations of predictor variables may be needed for several reasons. Some modeling techniques may have strict requirements, such as the predictors having a commom scale. In other cases, creating a good model may be difficult due to specific characteristics of the data (e.g., outliers).

### Centering and Scaling

To center a predictor variable, the average predictor value is substracted from all the values. As a result of centering, the predictor has a zero mean. Similarly, to scale the data, each value of the predictor variable is divided by its standard deviation. Scaling the data coerce the values to have a common standard deviation of one. These manipulations are generally used to improve the numerical stability of some calculations, such as PLS. The only real downside to these transformation is a loss of interpretability of the individual values.

### Transformations to Resolve Skewness

An un-skewed distribution is one that is roughly symmetric. A rule of thumb to consider is that skewed data whose ratio of the highest value to the lowest value is greater than 20 have significant skewness. The sample skewness statistic is defined $$\text{skewness} = {\sum (x_i - \bar{x})^3 \over (n - 1) v^{3/2}},$$ where $$v = {\sum (x_i - \bar{x})^2 \over (n - 1)}.$$ Note that the skewness for a normal distribution is zero.

The cell segmentation data contain a predictor that measures the standard deviation of the intensity of the pixels in the actin filaments.

```python
%matplotlib inline
import matplotlib.pyplot as plt

# Some nice default configuration for plots
plt.rcParams['figure.figsize'] = 10, 7.5
plt.rcParams['axes.grid'] = True
plt.gray()
```

```python
fig, (ax1, ax2, ax3) = plt.subplots(1, 3)

ax1.hist(cell_train['VarIntenCh3'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(np.log(cell_train['VarIntenCh3'].values), bins=20)
ax2.set_xlabel('Log Units')

ax3.hist(np.sqrt(cell_train['VarIntenCh3'].values), bins=20)
ax3.set_xlabel('Square Root Units')
```

The histogram shows a strong right skewness. The log transformation seems to work well for this dataset. The ratio of the smallest to largest value and the sample skewness statistic all agree with the histogram under natural units.

```python
from scipy.stats import skew

r = np.max(cell_train['VarIntenCh3'].values)/np.min(cell_train['VarIntenCh3'].values)
skewness = skew(cell_train['VarIntenCh3'].values)

print('Ratio of the smallest to largest value is {0} \nSample skewness statistic is {1}'.format(r, skewness))
```

    Ratio of the smallest to largest value is 870.8872472030321
    Sample skewness statistic is 2.395184181238347

Alternatively, statistical models can be used to empirically identify an appropriate transformation. One of the most famous transformations is the Box-Cox family, i.e.
\begin{equation}
x^* = \begin{cases} {x^{\lambda}-1 \over \lambda} & \text{if} \ \lambda \neq 0 \\ log(x) & \text{if} \ \lambda = 0 \end{cases}
\end{equation}
This family covers the log ($\lambda = 0$), square ($\lambda = 2$), square root ($\lambda = 0.5$), inverse ($\lambda = -1$), and others in-between. Using the training data, $\lambda$ can be estimated using maximum likelihood estimation (MLE). This procedure would be applied independently to each predictor data that contain values **greater than 0**.

The boxcox() in *scipy.stats* finds the estimated lambda and performs the transformation at the same time.

```python
from scipy.stats import boxcox

print('Estimated lambda is {0}'.format(boxcox(cell_train['VarIntenCh3'].values)[1]))
```

    Estimated lambda is 0.121531931959674

Take another predictor for example.

```python
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.hist(cell_train['PerimCh1'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(boxcox(cell_train['PerimCh1'].values)[0], bins=20)
ax2.set_xlabel('Transformed Data (lambda = {:1.4f})'.format(boxcox(cell_train['PerimCh1'].values)[1]))
```

## 3.3 Data Transformations for Multiple Predictors

These transformations act on groups of predictors, typically the entire set under consideration. Of primary importance are methods to resolve outliers and reduce the dimension of the data.

### Transformations to Resolve Outliers

We generally define outliers as samples that are exceptionally far from the mainstream of the data. Even with a thorough understanding of the data, outliers can be hard to define. However, we can often identify an unusual value by looking at a figure. When one or more samples are suspected to be outliers, the first step is to make sure that the values are scientifically valid and that no data recording errors have occured. Great care should be taken not to hastily remove or change values, especially if the sample size is small. With small sample sizes, apparent outliers might be a result of a skewed distribution where there are not yet enough data to see the skewness. Also, the outlying data may be an indication of a special part of the population under study that is just starting to be sampled. Depending on how the data were collected, a "cluster" of valid points that reside outside the mainstream of the data might belong to a different population than the other samples, e.g. *extrapolation* and *applicability domain*.

There are several predictive models that are resistant to outliers, e.g.
- Tree based classification models: create splits of the training set.
- Support Vector Machines (SVM) for classification: disregard a portion if the training set that may be far away from the decision boundary.

If a model is considered to be sensitive to outliers, one data transformation that can minimize the problem is the *spatial sign*. Mathematically, each sample is divided by its squared norm: $$x_{ij}^* = {x_{ij} \over \sqrt{\sum_{j=1}^p x_{ij}^2}}.$$ Since the denominator is intended to measure the squared distance to the center of the predictor's distribution, it is **important** to center and scale the predictor data prior to using this transformation. Note that, unlike centering and scaling, this manipulation of the predictors transform them as a group. Removing predictor variables after applying the spatial sign transformation may be problematic.

```python
# toy example
beta0 = -2.3 # intercept
beta1 = 0.8 # slope
n = 1000
x1_true = np.random.normal(4, 2, n)
x2_true = np.zeros(n)

# generate a random sample
for i in range(n):
    x2_true[i] = beta0 + beta1*x1_true[i] + np.random.normal(size = 1)

# generate outliers
x1_outliers = np.random.uniform(-4, -3, 8)
x2_outliers = np.zeros(8)
for i in range(8):
    x2_outliers[i] = x1_outliers[i] + np.random.normal(size = 1)

plt.scatter(x1_true, x2_true)
plt.plot(x1_outliers, x2_outliers, 'ro', markersize=8)
```

```python
from sklearn.preprocessing import scale
x1 = scale(np.concatenate([x1_true, x1_outliers]))
x2 = scale(np.concatenate([x2_true, x2_outliers]))
x = np.array(zip(x1, x2))

```

```python

# spatial sign
dist = x[:, 0]**2 + x[:, 1]**2
x1 = x[:, 0]/np.sqrt(dist)
x2 = x[:, 1]/np.sqrt(dist)

plt.scatter(x1[:-8], x2[:-8])
plt.plot(x1[-7:], x2[-7:], 'ro', markersize=8)
```

The *spatial sign* transformation brings the outliers towards the majority of the data.

### Data Reduction and Feature Extraction

These methods reduce the data by generating a smaller set of predictors that seek to capture a majority of the information in the original variables. For most data reduction techniques, the new predictors are functions of the original predictors; therefore, all the original predictors are still needed to create the surrogate variables. This class of methods is often called *signal extraction* or *feature extraction* techniques.

Principal component analysis (PCA) seeks to find linear combinations of the predictors, known as principal components (PCs), which capture the most possible variance. The first PC is defined as the linear combination of the predictors that captures the most variability of all possible linear combinations. Then, subsequent PCs are derived such that these linear combinations capture the most remaining variability while also being uncorrelated with all previous PCs. Mathematically,
$$\text{PC}_j = (a_{j1} \times \text{Predictor 1}) + \cdots + (a_{jP} \times \text{Predictor P}).$$
P is the number of predictors. The coefficients $a_{j1}, \cdots, a_{jP}$ are called component weights and help us understand which predictors are most important to each PC.

Let us look at an example from the previous dataset.

```python
cell_train_subset = cell_train[['Class', 'FiberWidthCh1', 'EntropyIntenCh1']]
```

```python
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = (cell_train_subset['Class'] == c[k])
    if k == 0:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Original Data')
plt.xlabel('Channel 1 Fiber Width')
plt.ylabel('Entropy intensity of Channel 1')
plt.legend(loc='upper right')
plt.show()
```

Calculate PCs

```python
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])
print('variance explained by PCs {0}'.format(pca.explained_variance_ratio_))
```

    variance explained by PCs [ 0.96900659  0.03099341]

The first PC summarizes 97% of the original variability, while the second summarizes 3%. Hence, it is reasonable to use only the first PC for modeling since it accounts for the majority of the information in the data.

```python
cell_train_subset_pca = pca.transform(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])

colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = np.where(cell_train_subset['Class'] == c[k])[0]
    if k == 0:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Transformed')
plt.xlabel('Principal Component #1')
plt.ylabel('Principal Component #2')
plt.legend(loc='upper right')
plt.show()
```

The primary advantage of PCA is that it creates components that are uncorrelated. PCA preprocessing creates new predictors with desirable characteristics for models that prefer predictors to be uncorrelated.

While PCA delivers new predictors with desirable characteristics, it must be used with understanding and care. PCA seeks predictor-set variation without regard to any further understanding of the predictors (i.e. measurement scales or distributions) or to knowledge of the modeling objectives (i.e. response variable). Hence, without proper guidance, PCA can generate components that summarize characteristics of the data that are irrelevant to the underlying structure of the data and also to the ultimate modeling objectives.

PCA was applied to the entire set of segmentation data predictors.

```python
cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      12
      207932484
      Train
      WS
      109.416426
      0
      256
      0
      18.828571
      125.938776
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      15
      207932459
      Train
      PS
      104.278654
      0
      258
      0
      17.570850
      124.368421
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 120 columns

```python
cell_train_feature = cell_train.iloc[:, 4:]
cell_train_feature.head(5)
```

      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      AvgIntenCh4
      AvgIntenStatusCh1
      AvgIntenStatusCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      133.752037
      0
      819
      1
      31.923274
      205.878517
      69.916880
      164.153453
      0
      0
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      106.646387
      0
      431
      0
      28.038835
      115.315534
      63.941748
      106.696602
      0
      0
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      69.150325
      0
      298
      0
      19.456140
      101.294737
      28.217544
      31.028070
      0
      0
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      109.416426
      0
      256
      0
      18.828571
      125.938776
      13.600000
      46.800000
      0
      0
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      104.278654
      0
      258
      0
      17.570850
      124.368421
      22.461538
      71.206478
      0
      0
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 116 columns

Because PCA seeks linear combinations of predictors that maximize variability, it will naturally first be drawn to summarizing predictors that have more variation. If the original predictors are on measurement scales that differ in orders of magnitude or have skewed distributions, PCA will be focusing its efforts on identifying the data structure based on measurement scales and distributional difference rather than based on the important relationships within the data for the current problem. Hence, it is best to first transform skewed predictors and then center and scale the predictors prior to performing PCA.

```python
# Box-Cox transformation on positive predictors
# separate positive and non-positive predictors
pos_indx = np.where(cell_train_feature.apply(lambda x: np.all(x > 0)))[0]
cell_train_feature_pos = cell_train_feature.iloc[:, pos_indx]
print("# of positive features is {0}".format(pos_indx.shape[0]))
cell_train_feature_nonpos = cell_train_feature.drop(cell_train_feature.columns[pos_indx], axis=1, inplace=False)
print("# of npn-positive features is {0}".format(cell_train_feature.shape[1] - pos_indx.shape[0]))

cell_train_feature_pos_tr = cell_train_feature_pos.apply(lambda x: boxcox(x)[0])

cell_train_feature_tr = np.c_[cell_train_feature_pos_tr, cell_train_feature_nonpos]
print("The shape before/after transformation is {0} and {1}".format(cell_train_feature.shape, cell_train_feature_tr.shape))
```

    # of positive features is 47
    # of npn-positive features is 69
    The shape before/after transformation is (1009, 116) and (1009, 116)

```python
# scale and center predictors
from sklearn.preprocessing import scale

cell_train_feature_tr = scale(cell_train_feature_tr, with_mean=True, with_std=True)
```

The second caveat of PCA is that it does not consider the modeling obejective or response variable when summarizing variability -- it is an *unsupervised technique*. If the predictive relationship between the predictors and response is not connected to the predictors' variability, then the derived PCs will not provide a suitable relationship with the response. In this case, a *supervised technique*, like PLS will derive components while simultaneously considering the corresponding response.

To decide how many components to retain after PCA, a heuristic approach is to create a scree plot, which contains the ordered component number (x-axis) and the amount of summarized variability (y-axis). Generally, the component number prior to the tapering off of variation is the maximal component that is retained. In an automated model building process, the optimal number of components can be determined by cross-validation.

```python
# conduct PCA to transformed predictors
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_feature_tr)

# generate scree plot
plt.plot(pca.explained_variance_ratio_)
plt.xlabel('Percent of Total Variance')
plt.ylabel('Component')
```

```python
print("The first four components account for {0} of the total variance".format(pca.explained_variance_ratio_[:4]))
print("All together they account for {0} of the total variance".format(np.sum(pca.explained_variance_ratio_[:4])))
```

    The first four components account for [ 0.14015262  0.12569722  0.09394676  0.06393814] of the total variance
    All together they account for 0.4237347251400651 of the total variance

Visually examining the principal components is a critical step for assessing data quality and gaining intuition for the problem. To do this, the first few PCs can be plotted against each other and the plot symbols can be colored by the relevant characteristics, such as the class labels. If PCA has captured a sufficient amount of the information in the data, this type of plot can demonstrate clusters of samples or outliers that may prompt a closer examination of the individual data points. Note that the scale of the components tend to become smaller as they account for less and less variation in the data. If axes are displayed on separate scales, there is the potential to over-interpret any patterns that might be seen for components that account for small amounts of variation.

```python
# look at the first 3 PCs
pca = PCA(n_components=3)
cell_train_feature_pca = pca.fit_transform(cell_train_feature_tr)
```

```python
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']

fig, axarr = plt.subplots(3, 3, sharex=True, sharey=True)

# PC1 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        line1= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        line2= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC2 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC1 vs PC2
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

axarr[2,0].text(0.5, -1.0, 'PC1', ha='center', va='center', fontsize=24)
axarr[1,1].text(0.5, -1.0, 'PC2', ha='center', va='center', fontsize=24)
axarr[0,2].text(0.5, -1.0, 'PC3', ha='center', va='center', fontsize=24)
fig.legend([line1, line2], ('PS', 'WS'), loc='upper center', ncol=2, frameon=False)
fig.subplots_adjust(hspace=0.12, wspace=0.1)
fig.text(0.5, 0.06, 'Scatter Plot Matrix', ha='center', va='center', fontsize=18)
```

Since the percentages of variation explained are not large for the first three components, it is important not to over-interpret the resulting image. From this plot, there appears to be some separation between the classes when plotting the first and second components. However, the distribution of the well-segmented cells is roughly contained within the distribution of the poorly identified cells. One conclusion is that the cell types are not easily separated.

Another exploratory use of PCA is characterizing which predictors are associated with each component. Recall that each component is a linear combination of the predictors and the coefficient for each predictor is called the loading. Loadings close to zero indicate that the predictor variable did not contribute much to that component.

```python
# loadings
pca.components_.shape
```

    (3, 116)

## 3.4 Dealing with Missing Values

In many cases, some predictors have no values for a given sample. It is important to understand *why* the values are missing. First and foremost, it is important to know if the pattern of missing data is related to the outcome. This is called *informative missingness* since the missing data pattern is instructional on its own. Informative missingness can induce significant bias in the model.

Missing data should not be confused with *censored* data where the exact value is missing but something is known about its value. When building traditional statistical models focused on interpretation or inference, the censoring is usually taken in to account in a formal manner by making assumptions about the censoring mechanism. For predictive models, it is more common to treat these data as simple missing data or use the censored value as the observed value.

Missing values are more often related to predictive variables than the sample. Because of this, amount of missing data may be concentrated in a subset of predictors rather than occuring randomly across all the predictors. In some cases, the percentage of missing data is substantial enough to remove this predictor from subsequent modeling activities.

There are cases where the missing values might be concentrated in specific samples. For large datasets, removal of samples based on missing values is not a problem, assuming that the missingness is not informative. In smaller datasets, there is a steep price in removing samples; some of alternative approaches described below may be more appropriate.

If we do not remove the missing data, there are two general approaches. First, a few predictive models, especially tree-based techniques, can specifically account for missing data. Alternatively, missing data can be imputed. In this case, we can use information in the training set predictors to, in essence, estimate the values of other predictors.

Imputation is just another layer of modeling where we try to estimate values of the predictor variables based on other predictor variables. The most relevant scheme for accomplishing this is to use the training set to built an imputation model for each predictor in the daa set. Prior to model training or the prediction of new samples, missing values are filled in using imputation. Note that this extra layer of models adds uncertainty. If we are using resampling to select tuning parameter values or to estimate performance, the imputation should be incorporated within the resampling. This will increase the computational time for building models, but it will also provide honest estimates of model performance.

If the number of predictors affected by missing values is small, an exploratory analysis of the relationships between the preditors is a good idea. For example, visulization or methods like PCA can be used to determine if there are strong relationships between the predictors. If a variable with missing values is highly correlated with another predictor that has few missing values, a focused model can often be effective for imputation.

One popular technique for imputation is a $K$-nearest neighbor model. A new sample is imputed by finding the samples in the training set "closest" to it and averages these nearby points to fill in the value. One advantage of this approach is that the imputed data are confined to be within the range of the training set values. One disadvantage is that the entire training set is required every time a missing value needs to be imputed. Also, the number of neighbors is a tuning parameter, as is the method for determining "closeness" of two points. However, Troyanskaya et al. (2001) found the nearest neighbor approach to be fairly robust to the tuning parameters, as well as the amount of missing data.

```python
# randomly sample 50 test set
import random
cell_test_subset = cell_test.iloc[np.sort(random.sample(range(cell_test.shape[0]), 50))]

# separate features
cell_test_subset_f = cell_test_subset.iloc[:, 4:].drop('VarIntenCh3', 1)
cell_test_subset_v = cell_test_subset.iloc[:, 4:]['VarIntenCh3']
cell_train_f = cell_train_feature.drop('VarIntenCh3', 1)
cell_train_v = cell_train_feature['VarIntenCh3']
```

```python
# scale and center before imputation
from sklearn.preprocessing import StandardScaler

# standardize based on training set
sc_f = StandardScaler()
cell_train_f_sc = sc_f.fit_transform(cell_train_f)
cell_test_subset_f_sc = sc_f.transform(cell_test_subset_f)

sc_v = StandardScaler()
cell_train_v_sc = sc_v.fit_transform(cell_train_v.reshape(-1, 1))
cell_test_subset_v_sc = sc_v.transform(cell_test_subset_v.reshape(-1, 1))
```

    /Users/aremirata/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:10: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
      # Remove the CWD from sys.path while we load stuff.
    /Users/aremirata/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:11: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
      # This is added back by InteractiveShellApp.init_path()

```python
# use 5-nearest neighbor
from sklearn.neighbors import NearestNeighbors

nbrs = NearestNeighbors(n_neighbors = 5)
nbrs.fit(cell_train_f_sc) # based on training set
distance, indices = nbrs.kneighbors(cell_test_subset_f_sc) # neighbors for test set

# imputation
cell_test_subset_v_pred_knn = np.empty(50)
for idx, i in enumerate(indices):
    cell_test_subset_v_pred_knn[idx] = np.mean(cell_train_v_sc[i[1:]])
```

Find the predictor with highest correlation.

```python
from scipy.stats.stats import pearsonr

print("corr('VarIntenCh3', 'DiffIntenDensityCh3') is {0}".format(pearsonr(cell_train_v, cell_train_f['DiffIntenDensityCh3'])[0]))
```

    corr('VarIntenCh3', 'DiffIntenDensityCh3') is 0.8948715047853233

```python
# use linear model
from sklearn.linear_model import LinearRegression

lm = LinearRegression()
lm.fit(cell_train_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis],
       cell_train_v_sc) # find the predictor with highest correlation
cell_test_subset_v_pred_lm = \
lm.predict(cell_test_subset_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis])
```

Correlation between the real and imputed values

```python
print("kNN: {0}".format(pearsonr(cell_test_subset_v_sc.ravel(), cell_test_subset_v_pred_knn)[0]))
print("Linear Model: {0}".format(pearsonr(cell_test_subset_v_sc, cell_test_subset_v_pred_lm)[0][0]))
```

    kNN: 0.9041754173339616
    Linear Model: 0.8714262815406963

Note that the better performance of linear model is because of the high correlation (0.895) between these two predictors. kNN is generally more robust since it takes all predictors into consideration.

```python
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_knn)
ax1.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax1.plot(ax1.get_xlim(), ax1.get_ylim(), ls="--", c=".3")
ax1.set_title('5NN')

ax2.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_lm)
ax2.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax2.plot(ax2.get_xlim(), ax2.get_ylim(), ls="--", c=".3")
ax2.set_title('Linear Model')

fig.text(0.5, 0.04, 'Original Value (centered and scaled)', ha='center', va='center')
fig.text(0.06, 0.5, 'Imputed', ha='center', va='center', rotation='vertical')
```

## 3.5 Removing Predictors

There are potential advantages to removing predictors prior to modeling. First, fewer predictors means decreased computational time and complexity. Second, if two predictors are highly correlated, this implies that they are measuring the same underlying information. Removing one should not compromise the performance of the model and might lead to a more parsimonious and interpretable model. Third, some models can be crippled by predictors with degenerate distributions, e.g. near-zero variance predictors. In these cases, there can be a significant improvement in model performance and/or stability without the problematic variables.

A rule of thumb for detecting near-zero variance predictors:
- The fraction of unique values over the sample size is low (say 10%)
- The ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20)

If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

### Between-Predictor Correlations

*Collinearity* is the technical term for the situation where a pair of predictor variables have a substantial correlation with each other. It is also possible to have relationships between multiple predictors at once (called *multicollinearity*).

A direct visualization of the correlation matrix from the training set.

```python
# calculate the correlation matrix
corr_dataframe = cell_train_feature.corr()

# compute hierarchical cluster on both rows and columns for correlation matrix and plot heatmap
def corr_heatmap(corr_dataframe):
    import scipy.cluster.hierarchy as sch

    corr_matrix = np.array(corr_dataframe)
    col_names = corr_dataframe.columns

    Y = sch.linkage(corr_matrix, 'single', 'correlation')
    Z = sch.dendrogram(Y, color_threshold=0, no_plot=True)['leaves']
    corr_matrix = corr_matrix[Z, :]
    corr_matrix = corr_matrix[:, Z]
    col_names = col_names[Z]
    im = plt.imshow(corr_matrix, interpolation='nearest', aspect='auto', cmap='bwr')
    plt.colorbar()
    plt.xticks(range(corr_matrix.shape[0]), col_names, rotation='vertical', fontsize=4)
    plt.yticks(range(corr_matrix.shape[0]), col_names[::-1], fontsize=4)

# plot
corr_heatmap(corr_dataframe)
```

Note that the predictor variables have been grouped using a clustering technique so that collinear groups of predictors are adjacent to one another.

When the data set consists of too many predictors to examine visually, techniques such as PCA can be used to characterize the magnitude of the problem. For example, if the first principal component accounts for a large percentage of the variance, this implies that there is at least one group of predictors that represent the same information. The PCA loadings can be used to understand which predictors are associated with each component to tease out this relationship.

In general, there are good reasons to avoid data with highly correlated predictors. First, redundant predictors frequently add more complexity to the model than information they provide to the model. In situations where obtaining the predictor data is costly, fewer variables is obviously better. Using highly correlated predictors in techniques like linear regression can result in highly unstable models, numerical values, and degraded predictive performances.

Classical regression analysis has several tools to diagnose multicollinearity for linear regression. A statistic called the variance inflation factor (VIF) can be used to identify predictors that are impacted. A common rule of thumb is that if VIF > 5, then multicollinearity is high. Note that this method is developed for linear models, it requires more samples than predictor variables and it does not determine which should be removed to resolve the problem

A more heuristic approach is to remove the minimum number of predictors to ensure that all pairwise correlation are below a certain threshold. The algorithm is as follows:
- Calculate the correlation matrix of the predictors.
- Determine the two predictors associated with the largest absolute pairwise correlation (A and B).
- Determine the average absolute correlation between A and the other variables. Do the same for predictor B.
- If A has a larger average correlation, remove it; otherwise, remove predictor B.
- Repeat Steps 2-4 until no absolute correlations are above the threshold.

Suppose we wanted to use a model that is particularly sensitive to between predictor correlations, we might apply a threshold of 0.75.

As previously mentioned, feature extraction methods (e.g., principal components) are another technique for mitigating the effect of strong correlations between predictors. However, these techniques make the connection between the predictors and the outcome more complex. Additionally, since signal extraction methods are usually unsupervised, there is no guarantee that the resulting surrogate preditors have any relationship with the outcome.

## 3.6 Adding Predictors

When a predictor is categorical, it is common to decompose the predictor into a set of more specific variables.

Look at the following example for the credit scoring data.

```python
credit_data = pd.read_csv("../datasets/GermanCredit/GermanCredit.csv")
credit_data.head(5)
```

      Unnamed: 0
      Duration
      Amount
      InstallmentRatePercentage
      ResidenceDuration
      Age
      NumberExistingCredits
      NumberPeopleMaintenance
      Telephone
      ForeignWorker
      ...
      OtherInstallmentPlans.Bank
      OtherInstallmentPlans.Stores
      OtherInstallmentPlans.None
      Housing.Rent
      Housing.Own
      Housing.ForFree
      Job.UnemployedUnskilled
      Job.UnskilledResident
      Job.SkilledEmployee
      Job.Management.SelfEmp.HighlyQualified

      0
      1
      6
      1169
      4
      4
      67
      2
      1
      0.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0

      1
      2
      48
      5951
      2
      2
      22
      1
      1
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0

      2
      3
      12
      2096
      2
      3
      49
      1
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0

      3
      4
      42
      7882
      2
      4
      45
      1
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

      4
      5
      24
      4870
      3
      4
      53
      2
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

5 rows × 63 columns

```python
credit_data.shape
```

    (1000, 63)

The predictor based on how much money was in the applicant's saving account is categorical coded into dummy variables.

```python
credit_data_saving = credit_data[['SavingsAccountBonds.lt.100', 'SavingsAccountBonds.100.to.500',
                                  'SavingsAccountBonds.500.to.1000', 'SavingsAccountBonds.gt.1000',
                                  'SavingsAccountBonds.Unknown']]
credit_data_saving.head(10)
```

      SavingsAccountBonds.lt.100
      SavingsAccountBonds.100.to.500
      SavingsAccountBonds.500.to.1000
      SavingsAccountBonds.gt.1000
      SavingsAccountBonds.Unknown

      0
      0.0
      0.0
      0.0
      0.0
      1.0

      1
      1.0
      0.0
      0.0
      0.0
      0.0

      2
      1.0
      0.0
      0.0
      0.0
      0.0

      3
      1.0
      0.0
      0.0
      0.0
      0.0

      4
      1.0
      0.0
      0.0
      0.0
      0.0

      5
      0.0
      0.0
      0.0
      0.0
      1.0

      6
      0.0
      0.0
      1.0
      0.0
      0.0

      7
      1.0
      0.0
      0.0
      0.0
      0.0

      8
      0.0
      0.0
      0.0
      1.0
      0.0

      9
      1.0
      0.0
      0.0
      0.0
      0.0

```python
credit_data_saving.apply(np.sum)
```

    SavingsAccountBonds.lt.100         603.0
    SavingsAccountBonds.100.to.500     103.0
    SavingsAccountBonds.500.to.1000     63.0
    SavingsAccountBonds.gt.1000         48.0
    SavingsAccountBonds.Unknown        183.0
    dtype: float64

| Value    |  n   | <100 | 100-500 | 500-1000 | >1000 | Unknown |
|:---------|:----:|:----:|:-------:|:--------:|:-----:|:-------:|
| < 100    | 603  |   1  |   0     |    0     |   0   |    0    |
| 100-500  | 100  |   0  |   1     |    0     |   0   |    0    |
| 500-1000 |  63  |   0  |   0     |    1     |   0   |    0    |
| >1000    |  48  |   0  |   0     |    0     |   1   |    0    |
| Unknown  | 183  |   0  |   0     |    0     |   0   |    1    |

Usually, each category gets its own dummy variable that is a zero/one indicator for that group. Only four dummy variables are needed here, the fifth can be inferred. However, the decision to include all of the dummy variables can depend on the choice of the model. Models that include an intercept term, such as simple linear model, would have numerical issues if each dummy variable was included in the model. The reason is that, for each sample, these variables all add up to one and this would provide the same information as the intercept. If the model is insensitive to this type of issue, using the complete set of dummy variables would help improve interpretation of the model.

Many of the advanced models automatically generate highly complex, nonlinear relationships between the predictors and the outcome. More simplistic models do not unless the user manually specifices which predictors should be nonlinear and in what way. Another technique to augment the prediction data for classification model is through the "*class centroids*", which are the centers of the predictor data for each class. For each predictor, the distance to each class centroid can be calculated and these distances can be added to the model.

## 3.7 Binning Predictors (to avoid)

There are many issues with the manual binning of continuous data. First, there can be a significant loss of performance in the model. Second, there is a loss of precision in the predictions when the predictors are categorized. Unfortunately, the predictive models that are most powerful are usually the least interpretable. The bottom line is that the perceived improvement in interpretability gained by manual categorization is usually offset by a significant loss in performance.

Note that the argument here is related to the *manual* categorization of predictors prior to model building. There are several models, such as classification/regression trees and multivariate adaptive regression splines, that estimate cut points in the process of model building. The difference between these methodologies and manual binning is that the models ues all the predictors to derive bins based on a single objective (such as maximizing accuracy). They evaluate many variable simultaneously and are usually based on statistically sound methodologies.


# Lecture 10: Audio Equalizers
Audio Processing, MED4, Aalborg University, 2020

By
- Jesper Kjær Nielsen (jkn@create.aau.dk), Audio Analysis Lab, Aalborg University, and
- Cumhur Erkut (cer@create.aau.dk), Multisensory Experience Lab, Aalborg University

Last edited: 2020-03-29

Table of Contents

1  Introduction

1.1  Why do we need audio equalizers?

1.2  A typical parametric equalizer

1.3  Example of a block diagram of a parametric equalizer

1.4  Summary

1.5  Active 5 minutes break

2  Notch and peak filters

2.1  Notch (bandstop) filter

2.2  Peak (bandpass) filter

2.3  Summary

3  Parametric equalizer filter

3.1  Summary

3.2  Active 5 minutes break

4  Shelving filters

4.1  Low frequency shelving filter

4.2  High frequency shelving filter

4.3  Multi-band parametric equalizer

4.4  Summary

## Introduction
In the next 20 minutes, you will learn

- what an audio equalizer is

- what it is used for

- what a parametric equalizer is

### Why do we need audio equalizers?
Many phenomena change the sound before it reaches our ears:
- amplifier
- loudspeaker
- room acoustics
- our hearing
- etc.

The main objective of audio equalizers is to **invert (some of) these changes**!

#### Example: hard surfaces boost the bass of a loudspeaker
We need an equalizer to compensate for the boost in bass.

#### Example: loudspeaker tuning
We need to tune the filters in the loudspeaker so that the listener hears what is intended.

#### Example: equalizing compensation for hearing loss

### A typical parametric equalizer
A common audio equalizer is a **parametric equalizer** which functions by
- dividing the frequency range into a number of bands
- apply filters in each band which can amplify/attenuate the frequency content in this band
---
Note that so-called graphic equalizers is an alternative to the parametric equalizer, but we will not cover that here.

### Example of a block diagram of a parametric equalizer

The filter in each band of the parametric equalizer is called a **parametric equalizer filter**. It can be either a
- **first and last band**: lowpass or highpass filter with adjustable cut-off frequency and gain (a **shelving filter**)
- **bands in the middle**: combination of a bandpass (peak) or a bandstop (notch) filter with adjustable center frequency and bandwidth

Note that
- both of these filters can be implemented using **a second order feedback (IIR) filter**
- we will go more into depth with these two kinds of parametric equalizer filters later

### Summary
1. Audio equalizers are useful in many applications such as room compensation and the tuning of hearing aids
2. A parametric equaliser is used to amplify/attenuate the frequency content within different bands

### Active 5 minutes break
As we see later, the basic ingredient is a **parametric equalizer filter** resulting in the **difference equation**
$$
    y_n = b_0x_n + b_1 x_{n-1} + b_2 x_{n-2} + a_1y_{n-1} + a_2 y_{n-2}
$$

1. Do you know this filter? Is this a feedforward or a feedback filter?
2. What are the feedforward and feedback filter coefficients?
3. Sketch the difference equations using the delay, summation, and multiplication blocks.

## Notch and peak filters
In the next 20 minutes, you will learn
- how a parametric equalizer filter can be designed using a notch (bandstop) and peak (bandpass) filter
- how the notch filter is designed and controlled
- how the peak filter is designed and controlled

The **transfer function** of the parametric equalizer filter $H_\text{eq}(z)$ is
$$
    H_\text{eq}(z) = G_0H_\text{notch}(z) + G H_\text{peak}(z)\ .
$$

### Notch (bandstop) filter
We will use the following notation:
- $\omega_1$ and $\omega_2$: lower and upper cutoff frequencies in radians/sample
- $\omega_0=\sqrt{\omega_1\omega_2}$: center frequency in radians/sample
- $\Delta\omega=\omega_2-\omega_1$: bandwidth in radians/sample
- $G_\text{B}$: gain at the cutoff frequencies.

It can be shown that the transfer function of a notch filter is given by
$$
    H_\text{notch}(z) = b\frac{1-2\cos(\omega_0)z^{-1}+z^{-2}}{1-2b\cos(\omega_0)z^{-1}+(2b-1)z^{-2}}
$$
where
- $\omega_0$ is the center frequency in radians/sample
- $b=(1+\beta)^{-1}$ where
$$
    \beta = \frac{\sqrt{1-G_\text{B}^2}}{G_\text{B}}\tan(\Delta\omega/2)
$$
- $G_\text{B}$ is the gain at the cutoff frequencies $\omega_1$ and $\omega_2$
- $\Delta\omega$ is the bandwidth (i.e., $\omega_2-\omega_1$) in radians/sample.

The notch filter results in the difference equation
$$
    y_n = bx_n -2b\cos(\omega_0)x_{n-1}+bx_{n-2} + 2b\cos(\omega_0)y_{n-1} - (2b-1) y_{n-2}
$$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

def computeNotchFilterParameters(digCenterFreq, digBandwidth, cutoffGain):
    beta = (np.sqrt(1-cutoffGain**2)/cutoffGain)*np.tan(digBandwidth/2)
    b0 = 1/(1+beta)
    b1 = -2*b0*np.cos(digCenterFreq)
    b2 = b0
    a1 = -b1
    a2 = -(2*b0-1)
    feedforwardParams = np.array([b0, b1, b2])
    feedbackParams = np.array([a1, a2])
    return feedforwardParams, feedbackParams
```

```python
samplingFreq = 44100 # Hz
centerFreq = 1000 # Hz
bandwidth = 250 # Hz
cutoffGain = np.sqrt(0.5)
nDtft = 2048
feedforwardParams, feedbackParams = computeNotchFilterParameters(centerFreq*2*np.pi/samplingFreq, \
    bandwidth*2*np.pi/samplingFreq, cutoffGain)
digFreqVector, freqResp = sig.freqz(feedforwardParams, np.r_[1,-feedbackParams],nDtft)
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,1)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{notch}(f)|$');
```

### Peak (bandpass) filter
We will use the following notation:
- $\omega_1$ and $\omega_2$: lower and upper cutoff frequencies in radians/sample
- $\omega_0=\sqrt{\omega_1\omega_2}$: center frequency in radians/sample
- $\Delta\omega=\omega_2-\omega_1$: bandwidth in radians/sample
- $G_\text{B}$: gain at the cutoff frequencies.

It can be shown that the transfer function of a notch filter is given by
$$
    H_\text{peak}(z) = (1-b)\frac{1-z^{-2}}{1-2b\cos(\omega_0)z^{-1}+(2b-1)z^{-2}}
$$
where
- $\omega_0$ is the center frequency in radians/sample
- $b=(1+\beta)^{-1}$ where
$$
    \beta = \frac{G_\text{B}}{\sqrt{1-G_\text{B}^2}}\tan(\Delta\omega/2)
$$
- $G_\text{B}$ is the gain at the cutoff frequencies $\omega_1$ and $\omega_2$
- $\Delta\omega$ is the bandwidth (i.e., $\omega_2-\omega_1$) in radians/sample.

The peak filter results in the difference equation
$$
    y_n = (1-b)x_n - (1-b)x_{n-2} + 2b\cos(\omega_0)y_{n-1} - (2b-1) y_{n-2}
$$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

def computePeakFilterParameters(digCenterFreq, digBandwidth, cutoffGain):
    beta = (cutoffGain/np.sqrt(1-cutoffGain**2))*np.tan(digBandwidth/2)
    b = 1/(1+beta)
    b0 = 1-b
    b1 = 0
    b2 = -b0
    a1 = 2*b*np.cos(digCenterFreq)
    a2 = -(2*b-1)
    feedforwardParams = np.array([b0, b1, b2])
    feedbackParams = np.array([a1, a2])
    return feedforwardParams, feedbackParams
```

```python
samplingFreq = 44100 # Hz
centerFreq = 1000 # Hz
bandwidth = 2500 # Hz
cutoffGain = np.sqrt(0.5)
nDtft = 2048
feedforwardParams, feedbackParams = computePeakFilterParameters(centerFreq*2*np.pi/samplingFreq, \
    bandwidth*2*np.pi/samplingFreq, cutoffGain)
digFreqVector, freqResp = sig.freqz(feedforwardParams, np.r_[1,-feedbackParams],nDtft)
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,1)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{peak}(f)|$');
```

### Summary
1. A notch (bandstop) filter can be used to remove frequencies

2. A peak (bandpass) filter can be used to remove frequencies

## Parametric equalizer filter
In the next 20 minutes, you will learn
- how we can combine a notch and a peak filter into a parametric equalizer filter

We can write the transfer function of the peak and notch filters as
\begin{align}
    H_\text{notch}(z) &= \frac{A_\text{notch}(z)}{B_\text{notch}(z)}\\
    H_\text{peak}(z) &= \frac{A_\text{peak}(z)}{B_\text{peak}(z)}
\end{align}
where
\begin{align}
    A_\text{notch}(z) &= b_\text{notch}(1-2\cos(\omega_0)z^{-1}+z^{-2})\\
    B_\text{notch}(z) &= 1-2b_\text{notch}\cos(\omega_0)z^{-1}+(2b_\text{notch}-1)z^{-2}\\
    A_\text{peak}(z) &= (1-b_\text{peak})(1-z^{-2})\\
    B_\text{peak}(z) &= 1-2b_\text{peak}\cos(\omega_0)z^{-1}+(2b_\text{peak}-1)z^{-2}\ .
\end{align}

Recall that the **transfer function** of the parametric equalizer filter $H_\text{eq}(z)$ is
$$
    H_\text{eq}(z) = G_0H_\text{notch}(z) + G H_\text{peak}(z)\ .
$$

Note that
- $G_0$ (which is sometimes called the level) is often set to 1
- $G>G_0$ results in a boost of some frequencies
- $G<G_0$ results in a cut of some frequencies

We have
\begin{align}
    H_\text{eq}(z) &= G_0H_\text{notch}(z) + G H_\text{peak}(z)\\
    &= \frac{G_0A_\text{notch}(z)B_\text{peak}(z)+GA_\text{peak}(z)B_\text{notch}(z)}{B_\text{notch}(z)B_\text{peak}(z)}
\end{align}
with three interesting special cases:
1. $G_0=G=1$: the signal passes unaltered, i.e., $H_\text{eq}(z)=1$
2. $G_0=1$ and $G=0$: the parametric equalizer filter is a notch filter, i.e., $H_\text{eq}(z)=H_\text{notch}(z)$
3. $G_0=0$ and $G=1$: the parametric equalizer filter is a peak filter, i.e., $H_\text{eq}(z)=H_\text{peak}(z)$

After a lot of math, it can be shown that
$$
    H_\text{eq}(z) = \frac{b_0 + b_1z^{-1}+b_2z^{-2}}{1 - a_1z^{-1}-a_2z^{-2}}
$$
where we have defined
\begin{alignat}{2}
    b_0 &= \frac{G_0+G\alpha}{1+\alpha}\ , &\qquad b_1 &= \frac{-2G_0\cos(\omega_0)}{1+\alpha}\\
    b_2 &= \frac{G_0-G\alpha}{1+\alpha}\ , &\qquad a_1 &= \frac{2\cos(\omega_0)}{1+\alpha}\\
    a_2 &= -\frac{1-\alpha}{1+\alpha}\ , &\qquad \alpha &= \sqrt{\frac{G_\text{B}^2-G_0^2}{G^2-G_\text{B}^2}}\tan(\Delta\omega/2)\ .
\end{alignat}

#### Example: design of parametric equalizer filter
Assume that the user can control the three parameters
1. center frequency $\omega_0$
2. bandwidth $\Delta\omega$
3. boost/cut $G$

When design the parametric equalizer filter as
1. calculate the cutoff gain as either
$$
    G_\text{B}^2 = G_0G \quad\text{or}\quad G_\text{B} = G_0^2/2+G^2/2
$$
with (typically) $G_0=1$.
2. Compute $\alpha$ and the filter coefficients $b_0$, $b_1$, $b_2$, $a_1$, and $a_2$ (see above)

The parametric equalizer filter results in the difference equation
$$
    y_n = b_0x_n + b_1 x_{n-1} + b_2 x_{n-2} + a_1y_{n-1} + a_2 y_{n-2}
$$

```python
def paramEqFilterCoefficients(digCenterFreq, digBandwidth, gain, level=1):
    if gain == level:
        feedforwardParams = np.array([1, 0, 0])
        feedbackParams = np.array([0, 0])
    else:
        cutoffGain = np.sqrt((gain**2+level**2)/2) # could also be the geometric mean instead
        alpha = np.sqrt((cutoffGain**2-level**2)/(gain**2-cutoffGain**2))*np.tan(digBandwidth/2)
        b0 = (level+gain*alpha)/(1+alpha)
        b1 = -2*level*np.cos(digCenterFreq)/(1+alpha)
        b2 = (level-gain*alpha)/(1+alpha)
        a1 = 2*np.cos(digCenterFreq)/(1+alpha)
        a2 = -(1-alpha)/(1+alpha)
        feedforwardParams = np.array([b0, b1, b2])
        feedbackParams = np.array([a1, a2])
    return feedforwardParams, feedbackParams
```

```python
samplingFreq = 44100 # Hz
centerFreq = 1000 # Hz
bandwidth = 2500 # Hz
gain = 1
nDtft = 2048
feedforwardParams, feedbackParams = paramEqFilterCoefficients(centerFreq*2*np.pi/samplingFreq, \
    bandwidth*2*np.pi/samplingFreq, gain)
digFreqVector, freqResp = sig.freqz(feedforwardParams, np.r_[1,-feedbackParams],nDtft)
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,2)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{notch}(f)|$');
```

### Summary
1. A parametric equalizer filter is a way of boosting or cutting some frequencies using a combination of peak and notch filters.
2. Typically, the user can control
 - $\omega_0$: the center frequency,
 - $\Delta\omega$: the bandwidth, and
 - $G$: the amount of boost/cut

### Active 5 minutes break
1. Together with your neighbour, explain as much as you can about the equalizer on the picture (i.e., number of bands, user parameters, etc.)

## Shelving filters
In the next 20 minutes, you will learn
- what a shelving filter is and why we need them
- that a shelving filter is a special case of the parametric equalizer filter

Recall that a **parametric equalizer** functions by
- dividing the frequency range into a number of bands
- apply filters in each band which can amplify/attenuate the frequency content in this band

What about the first (low frequencies) and last (high frequencies) band?

For the low and high frequencies, traditional low- and highpass filters are used instead of peak and notch filters. When used as shown below, the filter is called a **shelving filter**!

The shelving filter exists in two forms:
1. Low frequency shelving filter
2. High frequency shelving filter

### Low frequency shelving filter
The low frequency shelving filter is simply the **parametric equalizer filter** with $\omega_0 = 0$ which can be written as
\begin{align}
    H_\text{low}(z) &= \frac{(b_0-b_2z^{-1})(1-z^{-1})}{(1+a_2z^{-1})(1-z^{-1})}\\
    &= \frac{b_0-b_2z^{-1}}{1+a_2z^{-1}}
\end{align}
since $\cos(\omega_0)=1$ for $\omega_0=0$ where (as before)
\begin{alignat}{2}
    b_0 &= \frac{G_0+G\alpha}{1+\alpha}\ , &\qquad b_2 &= \frac{G_0-G\alpha}{1+\alpha}\\
    a_2 &= -\frac{1-\alpha}{1+\alpha}\ , &\qquad \alpha &= \sqrt{\frac{G_\text{B}^2-G_0^2}{G^2-G_\text{B}^2}}\tan(\Delta\omega/2)\ .
\end{alignat}

In the context of the **low frequency shelving filter**, the meaning of $\Delta\omega$ and $G_\text{B}$ are
- $\Delta\omega$: the cutoff frequency which is sometimes denoted as $\omega_\text{c}$
- $G_\text{B}$: the gain at the cutoff frequency which is sometimes denoted as $G_\text{C}$

```python
samplingFreq = 44100 # Hz
centerFreq = 1 # Hz - for low shelving filter
cutoffFreq = 250 # Hz
gain = 1
nDtft = 2048
feedforwardParams, feedbackParams = paramEqFilterCoefficients(centerFreq*2*np.pi/samplingFreq, \
    cutoffFreq*2*np.pi/samplingFreq, gain)
digFreqVector, freqResp = sig.freqz(feedforwardParams, np.r_[1,-feedbackParams],nDtft)
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,2)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{low}(f)|$');
```

### High frequency shelving filter
The high frequency shelving filter is simply the **parametric equalizer filter** with $\omega_0 = \pi$ which can be written as
\begin{align}
    H_\text{high}(z) &= \frac{(b_0+b_2z^{-1})(1-z^{-1})}{(1-a_2z^{-1})(1-z^{-1})}\\
    &= \frac{b_0+b_2z^{-1}}{1-a_2z^{-1}}
\end{align}
since $\cos(\omega_0)=-1$ for $\omega_0=\pi$ where (as before)
\begin{alignat}{2}
    b_0 &= \frac{G_0+G\alpha}{1+\alpha}\ , &\qquad b_2 &= \frac{G_0-G\alpha}{1+\alpha}\\
    a_2 &= -\frac{1-\alpha}{1+\alpha}\ , &\qquad \alpha &= \sqrt{\frac{G_\text{B}^2-G_0^2}{G^2-G_\text{B}^2}}\tan(\Delta\omega/2)\ .
\end{alignat}

In the context of the **high frequency shelving filter**, the meaning of $\Delta\omega$ and $G_\text{B}$ are
- $\Delta\omega$: the Nyquist frequency minus the cutoff frequency, i.e., $\Delta\omega=\pi-\omega_\text{c}$
- $G_\text{B}$: the gain at the cutoff frequency which is sometimes denoted as $G_\text{C}$

```python
samplingFreq = 44100 # Hz
centerFreq = samplingFreq/2 # Hz - for high shelving filter
cutoffFreq = 20000 # Hz
gain = 1.5
nDtft = 2048
feedforwardParams, feedbackParams = paramEqFilterCoefficients(centerFreq*2*np.pi/samplingFreq, \
    np.pi-cutoffFreq*2*np.pi/samplingFreq, gain)
digFreqVector, freqResp = sig.freqz(feedforwardParams, np.r_[1,-feedbackParams],nDtft)
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,2)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{high}(f)|$');
```

### Multi-band parametric equalizer
Building a multi-band parametric equalizer is simply a question of
- designing a number of parametric equalizer filters (possibly as low and high shelving filters)
- connect all the parametric equalizer filters in series

```python
def multibandParametricEq(digCenterFreqs, digBandwidths, gains, nDtft=0):
    nBands = np.size(digCenterFreqs)
    feedforwardParams = np.zeros((3,nBands))
    feedbackParams = np.zeros((2,nBands))
    if nDtft > 0:
        freqResp = np.ones(nDtft)
    for ii in np.arange(nBands):
        feedforwardParams[:,ii], feedbackParams[:,ii] = \
            paramEqFilterCoefficients(digCenterFreqs[ii], digBandwidths[ii], gains[ii])
        if nDtft > 0:
            digFreqVector, iifreqResp = \
                sig.freqz(feedforwardParams[:,ii], np.r_[1,-feedbackParams[:,ii]],nDtft)
            freqResp = freqResp*iifreqResp
    if nDtft > 0:
        return feedforwardParams, feedbackParams, digFreqVector, freqResp
    else:
        return feedforwardParams, feedbackParams
```

```python
samplingFreq = 44100 # Hz
centerFreqs = np.array([1, 3000, 10000, samplingFreq/2]) # Hz
bandwidths = np.array([1000, 1000, 300, 15000]) # Hz
gains = np.array([1.2, 0.2, 2, 0.8])
nDtft = 2048
feedforwardParams, feedbackParams, digFreqVector, freqResp = \
    multibandParametricEq(centerFreqs*2*np.pi/samplingFreq, bandwidths*2*np.pi/samplingFreq, gains, nDtft)
```

```python
freqVector = digFreqVector*samplingFreq/(2*np.pi)
plt.figure(figsize=(14,6))
plt.plot(freqVector, np.abs(freqResp))
plt.xlim((0,freqVector[-1])), plt.ylim((0,2)), plt.xlabel('$f$ [Hz]'), plt.ylabel('$|H_{high}(f)|$');
```

### Summary
1. Shelving filters are low and high pass filters which can either amplify or attenuate low and high frequencies.
2. Shelving filters are used only for the first and last band of an equalizer.
3. A shelving filter is a special case of the parametric equalizer filter.