# The Rainbow Method Part I. Encoding Categorical Variables.
> 'When nature gives you a rainbow, take it.'

- author: Anna Arakelyan, Dmytro Karabash
- categories: [python, data science, classification, encoding]
- image: ../images/jd-rincs-9GkYmKYVoGY-unsplash.jpg
- permalink: /rainbow/
- hide: true

## Introduction

"I have 2000 features and I need to get to 50 features and keep the model as good or better" was how this all started. This is familiar to anyone in institutional data science where you need to carefully vet every model and look at each feature to make sure it is fine from the standpoint of regulation. What is valuable about this story is that it is based on a real production model that was developed by Anna Arakelyan at MassMutual and the Rainbow method suggested by Dmytro Karabash. We will get detailed results at the end but here's a preview for anyone who wants to take a peek:

Figure 1

```python
#hide_input

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pd.set_option('display.max_columns', None)

import warnings
warnings.filterwarnings("ignore")
# Add 'rainbow_research.csv' and 'rainbow_research_feature_selection.csv' in the _notebooks folder
# Don't push data to repo - they must be ignored in .gitignore
results = pd.read_csv('rainbow_research.csv').rename(columns={'feature_selection_parameter':'Num_features',"transform_parameter": "method"})
feature_selection = pd.read_csv('rainbow_research_feature_selection.csv').rename(columns={'feature_selection_parameter':'Num_features',"transform_parameter": "method"})

results['method']=results['method'].replace({'one_hot': "One-hot", 'rainbow': "Rainbow"})

metrics=['accuracy', 'kappa', 'macro avg_f1-score', 'weighted avg_f1-score', 'macro avg_roc_auc', 'weighted avg_roc_auc']
tmp = feature_selection.groupby(['Num_features', 'method'])[metrics].mean().round(4).reset_index()

tmp['Num_features']=tmp['Num_features'].str.replace('top_',"").astype(int)
table1 = tmp.set_index(['Num_features', 'method']).sort_index()

table1=table1.rename(columns={'macro avg_f1-score': 'macro-f1',
                       'weighted avg_f1-score':'weighted-f1',
                       'macro avg_roc_auc':'macro-auc',
                       'weighted avg_roc_auc' : 'weighted-auc'
                      })
# table1[['macro-f1','accuracy','kappa','macro-auc']].round(2)
table11 = table1[['macro-f1','accuracy','kappa','macro-auc']].round(4).reset_index()

def autolabel(rects, i, j, decimals=3):
    """
    Attach a text label above each bar displaying its height
    """
    for rect in rects:
        height = rect.get_height()
        axs[i, j].text(rect.get_x() + rect.get_width()/2., 1.015*height,
                '%.{}f'.format(decimals) % height,
                ha='center', va='bottom', color = 'dimgrey')

def make_subplot(table11, metric, x, i, j, colors, ylimits=[]):
    y1 = table11[table11['method']=='one_hot'][metric]
    y2 = table11[table11['method']=='rainbow'][metric]
    rects1 = axs[i, j].bar(x-0.2, y1, width, label = 'One-hot', color = colors[0])
    rects2 = axs[i, j].bar(x+0.2, y2, width, label = 'Rainbow', color = colors[1])
    axs[i, j].set_ylim(ylimits)
    axs[i, j].set_xticks(x)
    axs[i, j].set_xticklabels(['10 features', '50 features', '100 features'], rotation=0)
    axs[i, j].set_title(metric)
    autolabel(rects1, i, j)
    autolabel(rects2, i, j)

x = np.arange(3)
fig, axs = plt.subplots(2, 2, figsize=(9,5))
width = 0.35
colors = ['lightsalmon', 'turquoise']
make_subplot(table11, 'macro-f1', x, 0, 0, colors, [0.2,0.45])
make_subplot(table11, 'kappa', x, 0, 1, colors, [0, 0.30])
make_subplot(table11, 'macro-auc', x, 1, 0, colors, [0.55, 0.80])
make_subplot(table11, 'accuracy', x, 1, 1, colors, [0.3, 0.50])
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.05, 2.4), loc='upper left', prop={'size': 12})
plt.show()
```

The rainbow method outperforms the One-hot easily. The rainbow method even shows better results for 50 features than One-hot does for 100. Also note that when dropping from 50 to 10 features, the reduction in macro-f1 if you use One-hot is 6 times that of the rainbow method (3 times for kappa and accuracy, 2 times for macro-auc).

>Both models keep improving as the number of features goes up. Though the improvement is substantially faster and larger for the rainbow method.

**In this series of articles, we will introduce the Rainbow encoding method and will discuss its advantages over the conventional One-hot.**

Part I below explains how the method works, shows that it preserves full data signal, and provides examples of its use for various kinds of categorical variables. Part II explores its empirical application to a real business data science problem - Mindset Segmentation model at MassMutual using the Acxiom database. Part III provides the mathematical foundation for the method and shows that it reduces model complexity and helps avoid overfitting.

## Background

Data encoding is a crucial part of any data science pipeline. No matter which machine learning algorithm is used - there is no avoiding of data cleaning and encoding. Especially with dirty and complex real data sets, ensuring the most appropriate and efficient way of feature encoding and engineering is a nontrivial problem. While encoding quantitative and binary columns is usually a straightforward task, the encoding of categorical variables deserves a deeper look.

One of the most popular encoding methods for categorical variables has been the One-hot procedure. It generates an indicator variable for every category and, thus, creates a set of K new features, where K is the number of categories.
One significant implication of this method is the dimensionality increase. Depending on the magnitude of K, it can have various undesirable technical consequences: substantial raise of computational complexity, loss of degrees of freedom (N-K, where N is the number of samples), multicollinearity, and, ultimately, an overfitted model. Non-technical consequences include unnecessary complications of the model that should be communicated to final users and that contradict the law of parsimony aka Occam's razor.

## Method

We call it rainbow from a very simple analogy: if you have a categorical column with value names "Red", "Orange", "Yellow", "Green", "Blue", "Indigo" and "Violet", in other words - colors of the rainbow, instead of One-hot encoding, you can simply create one feature with encoding:
***
0 = "Red"

1 = "Orange"

2 = "Yellow"

3 = "Green"

4 = "Blue"

5 = "Indigo"

6 = "Violet"

This nicely replaces seven One-hot features with one rainbow.

Most of the nominal variables, at the first glance, seem like they cannot be converted to a quantitative scale. This is where we suggest **finding a rainbow**.

The seemingly unordered categories acquire a perfect order if we find a scale, where each category fits into its unique place. With the color - the natural scale is Hue. But it is not the only one. We can think of, say, brightness, saturation, color temperature, etc. We invite you to experiment with a few different rainbows that might capture different nuances of the categorical quality.

You can actually make and use two or more rainbows out of one categorical variable depending on the number of categories K and the context.

> We don't recommend using more than $\log_2 {K}$ rainbows because we do not want to surpass the number of encodings in a binary One-hot.

Why is rainbow better than any other choice? It is very simple and intuitively sensible. In many cases it is not even that important which rainbow you choose (and by that we mean the color order), it would still be better than using One-hot. The more natural orders are just likely to be slightly better than others and easier to interpret.

> Hence our motto: "When nature gives you a rainbow, take it..."

**Important note**: this method is highly efficient in conjunction with the models that rely on variables as ranks rather than exact values. For example, decision trees, random forest, gradient boosting - these algorithms will output the same result if, say, the variable *Number of Children* is coded as
***
0 = "0 Children"

1 = "1 Child"

2 = "2 Children"

3 = "3 Children"

4 = "4 or more Children"
***
or as
***
1 = "0 Children"

2 = "1 Child"

3 = "2 Children"

4 = "3 Children"

5 = "4 or more Children"
***

or even as
***
-100 = "0 Children"

-85 = "1 Child"

0 = "2 Children"

10 = "3 Children"

44 = "4 or more Children"
***

as long as the correct order of the categories is preserved. The values themselves don't carry the quantitative function in these algorithms. It is the rank of the variable that matters, and a tree-based algorithm will use its
magic to make the most appropriate splits to introduce new tree nodes.

The application of our method for other algorithms such as Linear Regression, Logistic Regression is out of the scope of this article. We expect that such a way of feature engineering would be still beneficial, but that is a subject of a different investigation.

Note that by using a rainbow rather than One-hot we are not losing any of the data signal. Let's explore that question below.

## Rainbow keeps the full signal

Let's zoom in for a minute on a classical rainbow example with just 4 values:

***
*Color*

0 = "Red"

1 = "Yellow"

2 = "Green"

3 = "Blue"
***

In the case of One-hot, we would create 4 features:

*Color_Red* = 1 if *Color* = 0 and 0 otherwise,

*Color_Yellow* = 1 if *Color* = 1 and 0 otherwise,

*Color_Green* = 1 if *Color* = 2 and 0 otherwise,

*Color_Blue* = 1 if *Color* = 3 and 0 otherwise.

In the case of Rainbow, we would just use the *Color* itself.

Let's compare the possible models made using these two methods: 4 features vs 1 feature. Let's for simplicity build a single decision tree. Let's look at a few scenarios of the data generating process.

### Scenario 1
Assume all the categories are wildly different, and each one introduces a substantial gain to the model. That means each One-hot feature is indeed critical - the model should separate between all 4 groups created by One-hot.

In that case, an algorithm such as *xgboost* will just make the splits between all the values which is perfectly equivalent to One-hot. So the same exact result is achieved with just one feature instead of four.

Figure 2

One can clearly see that this example is easily generalized to One-hot with **any** number of categories. Also, note that for such data generating process the order of the categories in Rainbow does not even matter as splits will be made between **all** categories. In practice, (K-1) splits will be sufficient for both methods to separate between K categories.

Figure 3

The main takeaway is that not a bit of data signal is lost if one switches from One-hot to Rainbow. In addition to this, depending on the number of categories, a substantial dimensionality reduction happens which is beneficial for multiple reasons.

### Scenario 2

If the chosen rainbow (Figure 3) is not the best because, for example, the data generating process separates between the group of {Red, Green} and {Yellow, Blue}, then once again - the algorithm will make all the necessary splits.

Figure 4

Here we show that depending on the order of binary features picked by One-hot there can be different trees built. In the case of Rainbow, though - once again we demonstrate that no information is lost and a (K-1) splits tree is built even in this least favorable scenario.

### Scenario 3

Finally, if the data generating process is actually in agreement with the Rainbow order, then the Rainbow method would be superior to One-hot. Not only will it not lose any data signal, but it will also substantially reduce complexity, decrease dimensionality, and help avoid overfitting.

Suppose, the true model pattern only separates between {Red, Yellow} and {Green, Blue}. In that case, Rainbow has a clear advantage as it exploits these groupings while One-hot does not.

Figure 5

In this introductory article, we briefly show that by substituting One-hot with Rainbow, we save dimensionality without any informational loss. In Part III though we dive deeper into the mathematical foundation of the Rainbow method and prove that the Rainbow method **outperforms** One-hot in the worst, average, and best scenarios.

In short, if the rainbow order is even *somewhat* meaningful, the algorithm would capture the similarity in the adjacent values by making a smart decision about the number and location of splits. Thus, we will receive a simpler and better model.

## Examples of Categorical Variables

The application of our method depends highly on the level of measurement of the treated variables. While **quantitative** variables have a *ratio* scale, i.e. they have a meaningful 0, ordered values, and equal distances between values; **categorical** variables usually have either *interval*, or *ordinal*, or *nominal* scales.
Let us illustrate our method for each of these types of categorical variables.

**Interval** variables have ordered values, equal distances between values, but the values themselves are not necessarily meaningful, for example, 0 does not indicate an absence of some quality. The common examples of interval variables are Likert scales:

***
*How likely is the person to buy a smartphone mobile phone?*

1 = "Very Unlikely"

2 = "Somewhat Unlikely"

3 = "Neither Likely Nor Unlikely"

4 = "Somewhat Likely"

5 = "Very Likely"
***

Straightforwardly, if we simply use the raw values 1 through 5, that will be the best and most natural rainbow. The algorithm such as *xgboost* will make the appropriate splits instead of being forced to use the splits predetermined by One-hot, which in many cases is simply an overfit.

**Ordinal** variables have ordered meaningless values, and the distances between values are also not equal or not even explainable. An example:

***
*What is the highest level of Education completed by the person?*

1 = "No High School"

2 = "High School"

3 = "Associate Degree"

4 = "Bachelor's Degree"

5 = "Master's Degree"

6 = "Doctoral Degree"

***

Similar to interval variables, raw ordinal numeric codes can be used in the model without introducing any issues, as long as the order is correct. In some cases, a variable is intrinsically ordinal, but the given numeric codes don't follow the correct order - in these situations, we could simply reorder categories and then use this updated variable as a quantitative feature.

The use of a natural rainbow for interval and ordinal variables is likely not the cause for concern because it is clearly a perfect alternative to One-hot for tree-based algorithms. The more complicated and non-obvious question is how to treat nominal variables.

**Nominal** variables have no obvious order between categories and are almost always encoded with the One-hot method. This is where **finding a rainbow** becomes valuable. It will turn something, that seems completely nonquantitative, into a quantitative scale. In  our classical example, a nominal variable *Color*

***
A = "Red"

B = "Blue"

C = "Green"

D = "Yellow"

***

can be replaced by the newly engineered rainbow feature
***

1 = "Blue"

2 = "Green"

3 = "Yellow"

4 = "Red"

***

Lets us show a few other examples of creating rainbows.

***
*Vehicle Type*

C = "Compact Car"

F = "Full-size Car"

L = "Luxury Car"

M = "Mid-Size Car"

P = "Pickup Truck"

S = "Sports Car"

U = "SUV"

V = "Van"

***

For this variable, we can think of dozens of characteristics to make a rainbow - vehicle size, capacity, price category, average speed, fuel economy, costs of ownership, motor features, etc. Which one (or more than one) to pick? The choice depends on the context of the model and what are you trying to predict. The preference should be given to the ones that provide more explanatory power and/or make sense from the interpretation standpoint.

Consider another variable:
***

*Marital Status*

A = "Married"

B = "Single"

C = "Inferred Married"

D = "Inferred Single"

***

If we think about Single and Married as of the two ends of the spectrum, then Inferred Single could be between the two ends, closer to Single, while Inferred Married would be between the two ends, closer to Married. That would make sense because Inferred holds a certain degree of uncertainty. Thus, the following order would be reasonable:
***
1 = "Single"

2 = "Inferred Single"

3 = "Inferred Married"

4 = "Married"
***

In case there are any missing values here, the Unknown category fits exactly in the middle between Single and Married as there is no reason to prefer one end over the other. So the modified scale could look like this:
***
1 = "Single"

2 = "Inferred Single"

3 = "Unknown"

3 = "Inferred Married"

4 = "Married"
***

Another example:
***
*Occupation*

1 = "Professional/Technical"

2 = "Administration/Managerial"

3 = "Sales/Service"

4 = "Clerical/White Collar"

5 = "Craftsman/Blue Collar"

6 = "Student"

7 = "Homemaker"

8 = "Retired"

9 = "Farmer"

A = "Military"

B = "Religious"

C = "Self Employed"

D = "Other"

***

Finding a rainbow here might be harder, but here are a few ways to do it: we could order occupations by average annual salary, by their prevalence in the geographic area of interest, or by information from some other dataset. That might involve calling a Census API or some other data source, and might be complicated by the fact that these values are not static, but these are still viable solutions.

## What if there is no natural rainbow?

In some situations though, we cannot find a logical order for the rainbow because the variable itself is not interpretable. An example could be a black box column made by a third party:
***
*Financial Cluster of the Household*

1 = "Market Watchers"

2 = "Conservative Wealth"

3 = "Specific Savers"

4 = "Tried and True"

5 = "Trendy Inclinations"

6 = "Current Consumers"

7 = "Rural Trust"

8 = "City Spotlight"

9 = "Career Conscious"

10 = "Digital Financiers"

11 = "Financial Futures"

12 = "Stable Influentials"

13 = "Conservatively Rural"
***

In this case, we might not have a clear idea of how to order categories due to a lack of knowledge of what each category entails. What to do with such variables?

We recommend creating an artificial rainbow by looking at how each category is related to the target variable.

The simplest solution is to place categories in the order of correlation with the target variable.
So the category with the highest value of correlation with the dependent variable would acquire numeric code 1, and the category with the lowest correlation would acquire numeric code 13. So our rainbow in this case would mean the relationship between the cluster and the target variable. This method would work for both - classification and regression models - as it can be applied to a discrete and a continuous target variable.

Alternatively, you can construct your rainbows by merely utilizing certain statistical qualities of the categorical variable and the target variable.

For instance, in the case of a binary target variable, we could look at the proportion of ones given each of the categories. Suppose, among Market Watchers percent of positive targets is 0.67, while for Conservative Wealth it is 0.45. In that case, Market Watchers will be ordered higher than Conservative Wealth (or lower, if the target percent scale is ascending). In other words, this rainbow would reflect the prevalence of positive targets inside each category.

In the case of a multiclass classification, we could create rainbows for each class to represent a relationship between categories and each class separately.

In Part II we will demonstrate that the Rainbow method outperforms One-hot using an empirical example of a real data science project at MassMutual.

# The Rainbow Method Part II. Application to a Real Data Science Problem

We will illustrate the effectiveness of the rainbow encoding method using the real data science project developed in the Customer Journey domain of [Data Science at MassMutual](https://datascience.massmutual.com) - a life insurance company with over 170 years of history that takes pride in growing a large team of data scientists, engineers, and technologists to support most of the business decisions.

## Business Use Case

In a nutshell, the data task is a multiclass classification problem that aims to predict one of the five Mindset Segments for each of the prospective customers.

Figure 1

The segmentation framework represents five classes that reflect a person's age, financial stability, and attitude towards financial decisions. The predicted segments are then used by the MassMutual marketing team in different types of campaigns for targeting and customization. For example, Self-Assured customers would value more independence and autonomy in deciding on buying a life insurance policy whereas Day to Day customers would value having guidance and thorough explanations of different financial products by a dedicated advisor.

We have a small set of labeled individuals (17.5K persons). The labels are provided by the MassMutual vendor which ran Mindset surveys and designed segment assignment rules. For simplicity, let's assume we know nothing about these rules and take labels as given.

Then we attach the columns from our main prospect database to these 17.5K persons. The goal is to learn the best model using these target labels and the available features and then predict segments for all other (unlabelled) prospective customers.

The main database for this problem is provided by [Acxiom](https://www.acxiom.com/) and covers about 300 columns representing a rich set of demographic characteristics, the composition of the household, income and net worth, financial behavior, and digital savviness.

```python
#hide

size_of_data = int((results['n_train'] + results['n_test'])[0])
size_of_data
```

    17493

Using Acxiom data and the Mindset Segmentation problem, we will compare the conventional One-hot encoding with the Rainbow method. For this 5-class classification task, we will demonstrate a few standard metrics - Accuracy, Cohen's Kappa, Macro Avg F1 score, and Macro Avg AUC ROC.

Cohen's Kappa, F1, and AUC ROC are very helpful for an unbalanced multiclass classification problem. Accuracy is not the best metric for such task and is used merely for interpretation and comparison purposes.

## Categorical Variables

We took all the existing categorical variables in the Acxiom database - interval, ordinal, and nominal, and excluded quantitative and binary variables. The idea was to show the pure difference in the model performance between the two types of encoding for the same set of categorical factors.

We then applied a target stratified 4-fold Cross-Validation split. All the data processing from this point on is done inside the cross-validation loop, i.e. the creation of One-hot features and Rainbow features is learned from each fold train set and applied to each fold validation set.

The total set of 111 variables was transformed into **201** Rainbow features and, alternatively, into **2260** One-hot features (with very slight deviations in N in 4 different folds).

Table 1

| Type of variable | N raw | N Rainbow encoded | N One-hot encoded |
| :-:| :-: | :-: | :-:|
| Interval |64| 64 | 1670 |
| Ordinal | 14 | 14 | 178 |
| Nominal |33| 123 | 412 |
| Total | 111 | **201** | **2260** |

While interval and ordinal variables have a straightforward rainbow transformation - one can notice that 64 interval features turned into 64 rainbows and 14 ordinal features turned into 14 rainbows, the nominal variables transformation was a bit more involved. Out of 33 nominal variables, for 23 we found a natural rainbow, while for 10 remaining variables we applied correlation ordering and target percent ordering (see Part I). Since we deal with 5 classes, we made 10 new features for each of these variables.

For example, given the name of feature "Financial_Cluster" and 5 segment names, we made features

- *Financial_Cluster_Self-Assured_correlation_rank*
- *Financial_Cluster_Juggler_correlation_rank*
- *Financial_Cluster_Starter_correlation_rank*
- *Financial_Cluster_Day to Day_correlation_rank*
- *Financial_Cluster_Well-Establishes_correlation_rank*

 and

- *Financial_Cluster_Self-Assured_target_percent*
- *Financial_Cluster_Juggler_target_percent*
- *Financial_Cluster_Starter_target_percent*
- *Financial_Cluster_Day to Day_target_percent*
- *Financial_Cluster_Well-Establishes_target_percent*

In this way, 33 raw nominal variables turned into 123 Rainbows.

It is important to note that the search for natural or non-natural Rainbows is highly project and context-specific, and is more of an art than a craft. For instance, for a binary classification problem, there probably would have been only one or two Rainbows.

We ran all XGBoost models covering this hyperparameter space:
```
'objective': 'multi:softprob'
'eval_metric': 'mlogloss'
'num_class': 5
'subsample': 0.8
'max_depth': [2, 3, 5]
'eta': [0.1, 0.3, 0.5]
'n_estimators': [50, 100, 200]
```

All the results below in this article represent cross-validation average metrics.

## Aggregate Results

Let us start with overall averages for all runs. Clearly, the average metrics across all models are notably higher for Rainbow encoding.

Figure 2

```python
#hide_input
t=results.copy()
t=t.rename(columns={'macro avg_f1-score': 'macro-f1',
                       'macro avg_roc_auc':'macro-auc',
                      })
k = t[t['feature_group']=='all'].groupby(['method'])[['accuracy', 'kappa', 'macro-f1', 'macro-auc']].mean().round(4)

def make_subplot2(k, metric, x, i, j, colors, ylimits=[]):
    y = k[metric]
    rects = axs[i, j].bar(x, y, color = colors, width=1, align='center')
    axs[i, j].set_ylim(ylimits)
    axs[i, j].set_xticks(x)
    axs[i, j].set_xticklabels(['One-hot', 'Rainbow'])
    axs[i, j].set_title(metric)
    autolabel(rects, i, j, 4)

x = np.arange(2)
fig, axs = plt.subplots(2, 2)
colors = ['lightsalmon', 'turquoise']
make_subplot2(k, 'macro-f1', x, 0, 0, colors, [0.27,0.37])
make_subplot2(k, 'kappa', x, 0, 1, colors, [0.14, 0.23])
make_subplot2(k, 'macro-auc', x, 1, 0, colors, [0.65, 0.75])
make_subplot2(k, 'accuracy', x, 1, 1, colors, [0.38, 0.45])
fig.tight_layout()
plt.show()

```

## Hyperparameters

The following plots show metric dynamics depending on every hyperparameter keeping other hyperparameters constant. These plots also clearly demonstrate the superiority of the Rainbow method for every hyperparameter and every metric.

Figure 3

```python
#hide_input
def make_subplot3(t, i, j, feature_group, hyperparameter, hyperparameter_name, metric,
                  ylimit_bottom, ylimit_top, xticks_list):
    k1 = pd.DataFrame(t[t['feature_group']==feature_group][t['method']=='Rainbow'].groupby(
        hyperparameter)[metric].mean().round(3))
    k2 = pd.DataFrame(t[t['feature_group']==feature_group][t['method']=='One-hot'].groupby(
        hyperparameter)[metric].mean().round(3))
    ax = axs[i,j]
    k1[metric].plot(ax=ax, label = 'Rainbow', color=colors[1], linewidth=5)
    k2[metric].plot(ax=ax, label = 'One-hot', color = colors[0], linewidth=5)
    ax.set_ylim(ylimit_bottom, ylimit_top)
    ax.set_xticks(xticks_list)
    ax.grid()
    ax.set_xlabel(hyperparameter_name)
    ax.set_title(metric)

fig, axs = plt.subplots(2, 2, figsize = (7,5))
make_subplot3(t, 0, 0, 'all', 'n_trees', 'Number of Trees', 'macro-f1', 0.29, 0.37, [50, 100 ,200])
make_subplot3(t, 0, 1, 'all', 'n_trees', 'Number of Trees', 'kappa', 0.15, 0.23, [50, 100 ,200])
make_subplot3(t, 1, 0, 'all', 'n_trees', 'Number of Trees', 'macro-auc', 0.68, 0.72, [50, 100 ,200])
make_subplot3(t, 1, 1, 'all', 'n_trees', 'Number of Trees', 'accuracy', 0.39, 0.43, [50, 100 ,200])
fig.suptitle('Number of trees', fontsize=15)
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.1, 2.55), loc='upper left', prop={'size': 12})
plt.show()

fig, axs = plt.subplots(2, 2, figsize = (7,5))
make_subplot3(t, 0, 0, 'all', 'eta', 'Learning Rate', 'macro-f1', 0.29, 0.37, [0.1, 0.3 ,0.5])
make_subplot3(t, 0, 1, 'all', 'eta', 'Learning Rate', 'kappa', 0.15, 0.23, [0.1, 0.3 ,0.5])
make_subplot3(t, 1, 0, 'all', 'eta', 'Learning Rate', 'macro-auc', 0.68, 0.72, [0.1, 0.3 ,0.5])
make_subplot3(t, 1, 1, 'all', 'eta', 'Learning Rate', 'accuracy', 0.39, 0.43, [0.1, 0.3 ,0.5])
fig.suptitle('Learning Rate', fontsize=15)
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.1, 2.55), loc='upper left', prop={'size': 12})
plt.show()

fig, axs = plt.subplots(2, 2, figsize = (7,5))
make_subplot3(t, 0, 0, 'all',  'max_depth', 'Max Depth', 'macro-f1', 0.29, 0.37, [2, 3 ,5])
make_subplot3(t, 0, 1, 'all',  'max_depth', 'Max Depth', 'kappa', 0.15, 0.23, [2, 3 ,5])
make_subplot3(t, 1, 0, 'all',  'max_depth', 'Max Depth', 'macro-auc', 0.68, 0.72, [2, 3 ,5])
make_subplot3(t, 1, 1, 'all',  'max_depth', 'Max Depth', 'accuracy', 0.39, 0.43, [2, 3 ,5])
fig.suptitle('Max Depth', fontsize=15)
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.1, 2.55), loc='upper left', prop={'size': 12})
plt.show()

```

## Runtime

Next, let's compare the runtime for each method.
```
One-hot: 66.074 s
Rainbow:  7.444 s
 ```
The average time to run a single rainbow model is **9 times faster** than that of the single One-hot model! So, in addition to a substantial increase in model performance metrics, the Rainbow method can also save data scientists a huge amount of time.

```python
#hide
pd.DataFrame(t[t['feature_group']=='all'].groupby(['method'])['total_time'].mean().round(3))
```

      total_time

      method

      One-hot
      66.074

      Rainbow
      7.444

## Interval, Ordinal, and Nominal

Next, we ran the models that covered the bundles of interval, ordinal, and nominal features separately. Below are the results.

```python
#hide
from IPython.display import display, HTML

u = t[t['feature_group']!='all'].groupby(['feature_group','method'])[['macro-f1', 'kappa', 'macro-auc', 'accuracy']].mean().round(4)
u = u.reindex([('interval', 'One-hot'), ('interval', 'Rainbow'),  ('ordinal', 'One-hot'), ('ordinal', 'Rainbow'),
              ('nominal', 'One-hot'), ('nominal', 'Rainbow')])
u.index.set_names(['feature group', 'method'], inplace=True)

HTML('''
        <style>
            .df tbody tr:nth-child(even) { background-color: ''' + str(colors[1])+'''; }
        </style>
        ''' + u.to_html(classes="df"))
```

      macro-f1
      kappa
      macro-auc
      accuracy

      feature group
      method

      interval
      One-hot
      0.2965
      0.1563
      0.6849
      0.4003

      Rainbow
      0.3275
      0.1894
      0.7040
      0.4149

      ordinal
      One-hot
      0.2362
      0.1113
      0.6338
      0.3784

      Rainbow
      0.2500
      0.1247
      0.6385
      0.3827

      nominal
      One-hot
      0.2916
      0.1542
      0.6783
      0.3938

      Rainbow
      0.2940
      0.1544
      0.6729
      0.3915

Figure 5

```python
#hide_input
table2 = u.reset_index()
table12 = table2[['method','macro-f1','accuracy','kappa','macro-auc']].round(4)

def autolabel(rects, i, j, decimals=3):
    """
    Attach a text label above each bar displaying its height
    """
    for rect in rects:
        height = rect.get_height()
        axs[i, j].text(rect.get_x() + rect.get_width()/2., 1.015*height,
                '%.{}f'.format(decimals) % height,
                ha='center', va='bottom', color = 'dimgrey')

def make_subplot2(table12, metric, x, i, j, colors, ylimits=[]):
    y1 = table12[table12['method']=='One-hot'][metric].reset_index(drop=True)
    y2 = table12[table12['method']=='Rainbow'][metric].reset_index(drop=True)
    rects1 = axs[i, j].bar(x-0.2, y1, width, label = 'One-hot', color = colors[0])
    rects2 = axs[i, j].bar(x+0.2, y2, width, label = 'Rainbow', color = colors[1])
    axs[i, j].set_ylim(ylimits)
    axs[i, j].set_xticks(x)
    axs[i, j].set_xticklabels(['Interval', 'Ordinal', 'Nominal'], rotation=0)
    axs[i, j].set_title(metric)
    autolabel(rects1, i, j)
    autolabel(rects2, i, j)

x = np.arange(3)
fig, axs = plt.subplots(2, 2, figsize=(9,5))
width = 0.35
colors = ['lightsalmon', 'turquoise']
make_subplot2(table12, 'macro-f1', x, 0, 0, colors, [0.2,0.40])
make_subplot2(table12, 'kappa', x, 0, 1, colors, [0, 0.25])
make_subplot2(table12, 'macro-auc', x, 1, 0, colors, [0.55, 0.78])
make_subplot2(table12, 'accuracy', x, 1, 1, colors, [0.3, 0.47])
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.05, 2.4), loc='upper left', prop={'size': 12})
plt.show()
```

These results demonstrate again that Rainbow is preferred to One-hot. As expected, interval and ordinal features gain the most from Rainbow encoding, while nominal variables - less so.

## Feature Selection

Finally, to make a comparison fairer in terms of dimensionality, we picked top 10, top 50, and top 100 features from each feature set - Rainbow and One-hot. We used the feature importance attribute of the XGBoost model and aggregated feature importance scores for 4 cross-validation folds on the best hyperparameter set for each encoding type. Below are the results.

```python
#hide
# same as below, but just the table - chart is a nicer representation
w = table1[['macro-f1', 'kappa', 'macro-auc', 'accuracy']].reset_index().replace({'one_hot': 'One-hot', 'rainbow': 'Rainbow'})
w = w.set_index(['Num_features', 'method'])
w.index.set_names(['number of features', 'method'], inplace=True)
w

HTML('''
        <style>
            .df tbody tr:nth-child(even) { background-color: ''' + str(colors[1])+'''; }
        </style>
        ''' + w.to_html(classes="df"))
```

      macro-f1
      kappa
      macro-auc
      accuracy

      number of features
      method

      10
      One-hot
      0.2685
      0.1505
      0.6778
      0.3945

      Rainbow
      0.3273
      0.1898
      0.7010
      0.4122

      50
      One-hot
      0.3342
      0.1975
      0.7040
      0.4180

      Rainbow
      0.3428
      0.2070
      0.7092
      0.4240

      100
      One-hot
      0.3333
      0.1992
      0.7102
      0.4205

      Rainbow
      0.3477
      0.2105
      0.7130
      0.4245

Figure 6

```python
#hide_input
# this is a copy of cell 1 of the python code, the first chart shown in Part I

x = np.arange(3)
fig, axs = plt.subplots(2, 2, figsize=(9,5))
width = 0.35
colors = ['lightsalmon', 'turquoise']
make_subplot(table11, 'macro-f1', x, 0, 0, colors, [0.2,0.45])
make_subplot(table11, 'kappa', x, 0, 1, colors, [0, 0.30])
make_subplot(table11, 'macro-auc', x, 1, 0, colors, [0.55, 0.80])
make_subplot(table11, 'accuracy', x, 1, 1, colors, [0.3, 0.50])
fig.tight_layout()
plt.legend(bbox_to_anchor=(1.05, 2.4), loc='upper left', prop={'size': 12})
plt.show()
```

Every single metric is considerably higher for the Rainbow method, especially with the lower number of selected features.
As mentioned in Part I, rainbow encoding with 50 features is even better than One-hot with 100 features as shown by macro-f1, kappa, and accuracy. Also, note the drop from 50 to 10 features in One-hot versus rainbow - macro-f1 loses 0.065 points with One-hot compared to only 0.016 points with Rainbow.

## Conclusion

As we illustrated above, the Rainbow method is an elegant and simple way to encode categorical variables, that will significantly reduce data dimensionality without losing any part of valuable signal; that will likely cause substantial improvements in model performance metrics (or, at the very least, will not cause any reductions in metrics); and that will save a great amount of time for modelers.

In Part III we provide mathematical justification of the Rainbow method and demonstrate its superiority against One-hot.

# The Rainbow Method Part III. Mathematical Foundation.

In this article, we justify mathematically the advantages of the Rainbow method over One-hot. We start with a simple example, and then we proceed with a full generalization and a simulation. In the final part, we discuss various assumptions and limitations of the Rainbow method and provide actionable recommendations to get the best out of Rainbow. Please note that the discussion below applies to any tree-based algorithm that relies only on the order of the values. That includes any random forest and any gradient boosting algorithm such as *xgboost*.

## How exactly does rainbow help?

### Simple Example

Consider the following example:

- assume a binary classification problem, with output variable having values 0 and 1;
- assume we have only one categorical variable V and no other variables;
- assume V has exactly four categories with values v1 < v2 < v3 < v4;

    *In the case of the nominal variable, the values simply represent numeric codes assigned to each category.*
- assume each value of V has the same or close number of samples in the data;
- assume an ideal separation will give to {v1 & v2} samples f-score = 1 and to {v3 & v4} samples f-score = 2, where f-score represents some decision function used for separation (e.g. Gini impurity, information gain, etc.).

    *I.e. the data generating process would separate **between** the groups {v1 & v2} and {v3 & v4}, but would not distinguish values **within** each group*.

Let's consider two cases - Rainbow and One-hot. For Rainbow, we will use V directly to build a decision tree. For One-hot, we will need to create four binary variables - let's call them V1, V2, V3, V4, where
Vi = 1 if V=vi and 0 otherwise.

Let's for simplicity build a single decision tree. Assume a simple algorithm that picks each next feature randomly, and compares all values of that feature to make the best split. In the case of the rainbow, samples from two categories will be sent to the left branch and samples from two other categories will be sent to the right branch of the tree. In such a situation a natural split occurs between v2 and v3, and rainbow would need one split,

Figure 1

while One-hot would need two or three splits. Here are two examples:

Figure 2

 Figure 3

In other words, assuming a clear order between categories, the Rainbow method is strictly better at separating samples - it makes fewer splits. In this example, it finds the single perfect split between v2 and v3. The One-hot method goes through four scales instead of one and does not naturally incorporate a combination of information from a few features.

Figure 4

But what happens if the categories do not go in perfect order?

### Generalizing simple example

Let's look at the more general case, where the values of the categorical variable are in a different order. Below are some examples of 4 categories and their f-scores. The same f-score justifies combining values into a group. For instance, 1,1,2,2 represents our example above, where the first two categories get assigned f = 1, and the last two categories get assigned f = 2, and the split should happen in the middle.

The number of splits would depend on the order in which these categories appear.

For rainbow orders:

* 1,1,2,2 and 2,2,1,1 the number of splits would be 1;

Figure 5

* 1,2,2,1 and 2,1,1,2 the required number of splits would be 2;

Figure 6

* 1,2,1,2 and 2,1,2,1, the required number of splits would be 3.

Figure 7

For One-hot, the orders below would represent the order in which the One-hot features are selected and added to the tree:

* 1,1,2,2 and 2,2,1,1 the required number of splits would be 2;

As in our simple example above (Figure 2), the first split would be made on V1, the second on V2, and that would be sufficient to separate between the samples with f = 1 and f = 2.

* 1,2,2,1 and 2,1,1,2 and 1,2,1,2 and 2,1,2,1 the number of splits would be 3.

In these examples, as shown above (Figure 3), three splits would be needed given the order in which binary features enter the tree.

Note that the Rainbow and One-hot orders described here are **not the same orders**, so we cannot compare them directly. In the case of Rainbow, we look at the order at the top - in which categories appear on a single scale, while for One-hot we look at the order at the bottom - in which categories are picked by the model. Nevertheless, we see that on average Rainbow is better as it provides a lower number of splits - best case 1, average case 2, worst case 3; while One-hot gives best case 2, average case $\frac 83$, and worst case 3. In fact, this is true in general and we show it below.

## General Case: Random rainbow vs One-hot

Assume that a certain categorical variable $V$ has $K$ categories with values v1 < v2 < v3 ... < vk. Then:
- One-hot would create $K$ features $V1, V2, ..., VK$, where $Vi$ = 1 if $V$ = vi and 0 otherwise;
- Rainbow would create a single feature, where all the categories appear on one scale. We can keep the name of this variable $V$.

Assume $K$ categories optimally would need to be split into $g$ groups of size $G_1,G_2,...,G_g$. By size $G_i$ we mean a number of categories out of $K$ (not the number of samples).

For example, in the simple example above - the order is 1,1,2,2:

Figure 8

and $K=4$, $g=2$, $G_1=2$ consisted of {v1, v2}, and $G_2=2$ consisted of {v3, v4}.

Another example - if the order is 1,2,2,1:

Figure 9

Here $K=4$, $g=2$, $G_1=2$ consists of {v1, v4}, and $G_2=2$ consists of {v2, v3}.

Unlike the simple example of the perfect rainbow, in the general case, the $K$ categories would appear **in random order**, and our rainbow would be random. However, for the purposes of computation, we only care about the groups of the same f-values rather than about actual categories. In other words, if v1 and v2 samples produce the same f-score, it does not matter whether the order is (v1, v2) or (v2, v1).

For One-hot the order that would matter is the one in which One-hot splits are made. This order is also assumed to be random. This is the order that we would always refer to when talking about One-hot.

>In reality, the One-hot splits are not fully random and the order of them being picked by the model varies depending on the number of samples in each category, subsamples of columns chosen, and other hyperparameters. That can be both in favor and against the case for One-Hot. We will come back to this assumption later in the article (see Randomness Assumption), and for now, will proceed with assuming randomness for simplicity.

While the orders that matter for Rainbow and One-hot are not the same, it helps to couple them to make comparisons. Let's compute the required number of splits in best, average, and worst cases.

### Main Comparison Metric

As in the simple example above, in this general case of random rainbow vs One-hot, we will compare the number of splits made by the model to approximate the data generating process. Why is it our main metric? Less splits means:
- less time;
- less computational resources;
- simpler model;
- less overfit;
- more effective model.

### Best case

#### One-hot
In the best-case scenario for One-hot, the biggest group won't be split.

For example, in the case of 1,1,2,2,2,3,3 we would like the biggest group 2,2,2 to end up on the bottom of the tree, and thus, not introduce any extra splits.

Figure 10

Thus, in the best case, One-hot would require $K - G_{max}$ splits, where $G_{max} = max_i G_i$ is the maximum of $G_i$. This best-case $K - G_{max}$ is obtained when each group except the largest one gets one split per category and the remaining largest group has the same answer so doesn't require any additional splits.  To prove that one cannot do better, observe that if a few categories do not appear on any tree node, i.e. no splits are made, then they are "combined" - there is no way to distinguish the values in these two categories.

#### Rainbow

In the best case of a rainbow - the values are in perfect order, and the minimum splits are needed. Given $g$ groups, that would require $g-1$ splits. This happens when groups are together and it simply splits at $g-1$ points to separate $g$ groups.

For example, clearly, two splits are needed for the perfect rainbow with three groups 1,1,2,2,2,3,3.

One cannot do better since if there is no split between two groups there is no way to distinguish them.

#### Comparison

Clearly  $$K - G_{max}=(\sum_{i=1}^g G_i)-G_{max}=(\sum_{i!=max}G_i) \geq (\sum_{i!=max} 1) = g-1$$ and hence best rainbow is better than the best One-hot.

### Average Case

#### One-hot

From the logic presented in the best case, it follows that
> The number of splits that One-hot needs would be $N$ - the number of categories that are chosen before the remaining ones are in one group.

Figure 10 shows the best case - where all the remaining ones belong to the largest group. In the average case, the remaining group can be any group or any group's part. We cannot expect that a homogenous group won't be broken into pieces when building an average tree. What we care about is that the tree building should end when reaching some homogenous group.

In the best case $N = K - G_{max}$. In the average case $N = K - R$ where $R$ is the size of the remaining group.

For example, if the order in which One-hot categories are chosen for the tree is 1,1,2,2,3,3,2,2,2 then $R=3$ since the last group 2,2,2 is of size 3. The number of categories $K=9$, thus the number of splits would be $K-R=6$.

It is easiest to compute the expected value of $N$ by using less common formula $\mathbb E[X] = \sum_{j} \mathbb P[X \geq j]$ applied to $R$. We then utilize the formula of hypergeometric distribution to calculate each probability and achieve the result below.

\begin{align}
\mathbb E[N] &= \mathbb E[K-R] = K- \mathbb E[R] = K - \sum_{j=1}^{K} \mathbb P[R \geq j] =K- \sum_{j=1}^K \frac{\sum_i \frac{G_i!}{(G_i-j)!}} {\frac{K!}{(K-j)!}} = (K-1) - \sum_{j=2}^K \frac{\sum_i \frac{G_i!}{(G_i-j)!}} {\frac{K!}{(K-j)!}}\\
\end{align}

#### Rainbow

From the best case logic, it also follows that
> The number of splits that random Rainbow needs would always be $D$ - the number of consecutive pairs in rainbow order that are in distinct groups.

In the best case $D=g-1$.

For example, in the case of Rainbow order 1,1,2,2,3,3,2,2,2 the splits will happen in 3 places, and these are the only places where the group is changed and the values in consecutive pairs belong to distinct groups.

Let's calculate the expected value of $D$. We do this by calculating for $K-1$ consecutive pairs as potential places for splitting the probability that the f-score is different. It is easier to subtract from one the probabilities that f values in each consecutive pair are the same. We sum through all possible groups $i$ to compute the likelihood that two consecutive values belong to one group.

\begin{align}
\mathbb E[D] &= (K - 1) \cdot (1-\frac{ \sum_i G_i \cdot (G_i-1)} {K(K-1)}) \notag \\
             &= (K-1)-\frac{ \sum G_i \cdot (G_i-1)} {K} \notag
\end{align}

#### Comparison

An easy way to see that $\mathbb E[D]$ is larger than $\mathbb E[N]$ is the stochastic coupling since both $N$ and $D$ can be obtained as starting from $K-1$ and subtracting $1$ for each consecutive pair in the same group; except in case of $N$, all these pairs have to be in the string at the end.  Due to this restriction $N$ is always larger.

### Worst Case

#### One-hot

In the worst case for One-hot, the number of splits is $K-1$ when one goes through all but one category to get to one group remaining.

#### Rainbow

In this case, the number of needed splits is

$\min( (K - 1), 2 \cdot (K-G_{max}))$

as we can only get up to two splits per each category that is not in the largest group.

Indeed, if there are no consecutive pairs of values that belong to the same group then the number of splits would be $K-1$, for example here 3,2,1,3,1,2.

At the same time, if there is a prevalent group and some minority groups, for example, 1,1,1,2,1,1,3,1,1, the number of splits will equal to the number of minority groups - $K-G_{max}=2$ in this case (2 and 3) times 2 which is the maximum number of splits that each minority group introduces - one to the left and one to the right.

Thus the needed number of splits will be the minimum between the two cases above.

#### Comparison

Clearly, the number of splits for Rainbow is less than or equal to that of One-hot for the worst case as well.

## Simulation

We have shown above that *theoretically* in a general case Rainbow method outperforms One-hot in best, average, and worst-case scenarios.
Let's look at the *empirical* simulation: we applied the formulas for best, average, and worst-case number of splits for Rainbow and One-hot to compare the distribution of splits for some examples of the data generating processes.

```python
#hide

import pandas as pd
from collections import Counter
import numpy as np

s='12,112,1112,1122,11112,11122,111222,11123,11223,112233,11223344,111222333,1111122233,1111111122,1111111123'
#s+=','+('1'*10+'2'*10)+','+('1'*20+'2'*20)
cc=examples=s.split(',')
examples=[list(i) for i in examples]

# number of splits in each scenario
o_best = [len(e)-max(Counter(e).values()) for e in examples]
r_best = [len(Counter(e))-1 for e in examples]
o_worst = [len(e)-1 for e in examples]
r_worst = [min(len(e)-1,2*(len(e)-max(Counter(e).values())))
           for e in examples]

o_ave = []
r_ave = []
n_samples = 40000
# to have two precision digits
for e in examples:
    # o_sum and r_sum is the sum of number of splits over all samples
    # we want to get avg n of splits given various scenarios (avg over samples)
    o_sum = r_sum = 0
    for i in range(n_samples):
        p = np.random.permutation(e)
        #[::-1] omiting as same thing but to indicate we count from end for one-hot order
        o_res = r_res = len(e)-1
        for i in range(len(p)-1):
            if p[i]==p[i+1]:
                r_res -=1
        for i in range(len(p)-1):
            if p[i]==p[i+1]:
                o_res -=1
            else:
                break
        o_sum += o_res
        r_sum += r_res
    o_ave += [o_sum/n_samples]
    r_ave += [r_sum/n_samples]

ii = 'o_best,r_best,o_ave,r_ave,o_worst,r_worst'.replace('o_','One-hot ').replace('r_','Rainbow ')
ii = ii.replace('ave','Average').replace('best','Best').replace('worst','Worst')
ii = ii.split(',')
ii = [i+' Case' for i in ii]
rainbow_vs_onehot=pd.DataFrame([o_best,r_best,o_ave,r_ave,o_worst,r_worst],
                                   columns=cc,index=ii).round(2)
rainbow_vs_onehot=rainbow_vs_onehot.iloc[[4,5,2,3,0,1]]
rainbow_vs_onehot
```

      12
      112
      1112
      1122
      11112
      11122
      111222
      11123
      11223
      112233
      11223344
      111222333
      1111122233
      1111111122
      1111111123

      One-hot Worst Case
      1.0
      2.00
      3.00
      3.00
      4.00
      4.0
      5.00
      4.0
      4.0
      5.0
      7.00
      8.00
      9.00
      9.00
      9.00

      Rainbow Worst Case
      1.0
      2.00
      2.00
      3.00
      2.00
      4.0
      5.00
      4.0
      4.0
      5.0
      7.00
      8.00
      9.00
      4.00
      4.00

      One-hot Average Case
      1.0
      1.67
      2.24
      2.67
      2.79
      3.5
      4.50
      3.6
      3.8
      4.8
      6.85
      7.71
      8.57
      7.12
      7.14

      Rainbow Average Case
      1.0
      1.33
      1.49
      2.00
      1.60
      2.4
      3.01
      2.8
      3.2
      4.0
      6.00
      5.99
      6.20
      3.19
      3.40

      One-hot Best Case
      1.0
      1.00
      1.00
      2.00
      1.00
      2.0
      3.00
      2.0
      3.0
      4.0
      6.00
      6.00
      5.00
      2.00
      2.00

      Rainbow Best Case
      1.0
      1.00
      1.00
      1.00
      1.00
      1.0
      1.00
      2.0
      2.0
      2.0
      3.00
      2.00
      2.00
      1.00
      2.00

Figure 11

```python
#hide_input
rainbow_vs_onehot.T.plot(rot=45,figsize=(15,8))
plt.xticks(ticks = range(15),labels=cc, fontsize=14)
plt.xlabel('Data Generating Process', fontsize=14)
plt.yticks([1,2,3,4,5,6,7,8,9,10,11], fontsize=14)
plt.ylabel('Number of Splits', fontsize=14)
plt.grid()
plt.legend(loc=2, prop={'size': 14})
plt.show()
```

One can clearly see that the simulation confirms our theoretical finding:
- Best Rainbow always needs a less or equal number of splits than Best One-hot: the brown line is almost always below the purple line;
- Average Rainbow always outperforms Average One-hot: the red line is always strictly below the green line;
- Worst Rainbow needs the same or a lower number of splits than worst One-hot: the orange line is either below or coincides with the blue line.

## Conclusion

### Discussion of Assumptions

While the results above appear to be very optimistic, we should discuss the method limitations and assumptions we made.

> Single Tree Assumption: The use of a single decision tree vs forest.

That assumption does not really impact the outcomes, since a single tree is equivalent to a forest of trees with depth 1. In fact, if our method works on simpler models that only adds to its advantages.

> Randomness Assumption: One-hot picks columns randomly.

We assumed above that our machine learning algorithms pick features somewhat randomly, and then make a split by using full scales of values of those features. One might argue that in reality, before making any split, algorithms such as *xgboost* would look at many splits **across all features** and will pick the best.

We understand that this is a strong assumption and in reality, the One-hot splits are not fully random. However, they are also not fully deterministic. First, most of the algorithms don't run all possible comparisons within all features to make a choice about the node, rather some quick set of rough comparisons, which is done to avoid overfitting. Second, various hyperparameters have a substantial effect on which split will be chosen: subsample of columns, subsample of rows, maximum depth, etc. Third, if we are talking about bigger categorical variables with many categories - One-hot creates a lot of features, and randomness assumption would be not far from the truth. Finally, counts of samples for each category can be highly uneven, and that also gets us closer to randomness and is likely to yield worse results for One-hot than for Rainbow.

### How to find the best Rainbow?

As we have shown above the distribution of the number of splits is strictly better for Rainbow vs One-hot. However, one might argue that in reality we might be choosing between a *good* One-hot (assuming intelligent algorithms and relaxing randomness assumption) and a *random* rainbow.

In fact, the quality of a rainbow is an unknown. We can see that the Average Rainbow (red line) is between Best One-hot (purple line) and Average One-hot (green line). So it is important to try and find a good order rainbow, that approaches the data generating process.

The proposed solution here is very simple and clear - pick a few, choose the best. We invite you to try a few possible Rainbow encodings and compare them side by side to each other and One-hot. Your data and problem context will be the best tools here to choose the best encoding. We also recommend you to look at a few metrics, as well as model runtime. We expect that the Rainbow method would provide you with much better overall results, especially if you have many categorical variables with many values.

### Final Notes

This article just opens the conversation about the Rainbow method, and by no means exhausts the topic. In potential future investigations, we could explore some other aspects. To name a few: continuous target variables; comparison with other dimensionality reduction methods, such as PCA; missing values and how they fit the Rainbow framework; limitations and implications of combining seemingly unrelated features into a single Rainbow. We hope to open the gate for further questions and feedback on this method.

#### References

While the topic of One-hot and binary-One-hot encoding has been touched by many authors:

* https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159
* https://towardsdatascience.com/stop-One-hot-encoding-your-categorical-variables-bbb0fba89809

the approach presented here is rarely used in industry or academia, while has shown superior results and by definition creates fewer features.


# Cocco, Gomes, & Maenhout (2005)

 # "[Consumption and Portfolio Choice Over the Life Cycle](https://academic.oup.com/rfs/article-abstract/18/2/491/1599892)"

 - Notebook created by [Mateo Velásquez-Giraldo](https://github.com/Mv77) and [Matthew Zahn](https://sites.google.com/view/matthew-v-zahn/matthew-v-zahn).

This notebook uses the [Econ-ARK/HARK](https://github.com/econ-ark/hark) toolkit to describe the main results and reproduce the figures in the linked paper. The main HARK tool used here is the $\texttt{PortfolioConsumerType}$ class. For an introduction to this module, see the [HARK's Documentation](https://hark.readthedocs.io/en/latest/).

Our results using the HARK toolkit resemble those of the original paper, but we have not been able to replicate its results exactly for reasons that we have not been able to track down. We suspect there might be subtleties in the normalization or calibration that we have missed and that could be causing the discrepancies. We would be thankful if a reader (or the original authors) were able to discover the reason for the discrepancies, and could issue a pull request for a revision that reconciles them.

We thank Chris Carroll and Sylvain Catherine for comments and guidance.

```python
# This cell does some preliminary set up

# Packages
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# Import relevenat HARK tools
import HARK.ConsumptionSaving.ConsPortfolioModel as cpm

# This is a jupytext paired notebook that autogenerates BufferStockTheory.py
# which can be executed from a terminal command line via "ipython BufferStockTheory.py"
# But a terminal does not permit inline figures, so we need to test jupyter vs terminal
# Google "how can I check if code is executed in the ipython notebook"
from IPython import get_ipython # In case it was run from python instead of ipython

# If the ipython process contains 'terminal' assume not in a notebook
def in_ipynb():
    try:
        if 'terminal' in str(type(get_ipython())):
            return False
        else:
            return True
    except NameError:
        return False

# Determine whether to make the figures inline (for spyder or jupyter)
# vs whatever is the automatic setting that will apply if run from the terminal
if in_ipynb():
    # %matplotlib inline generates a syntax error when run from the shell
    # so do this instead
    get_ipython().run_line_magic('matplotlib', 'inline')
else:
    get_ipython().run_line_magic('matplotlib', 'auto')
```

### The base model

The authors' aim is to represent the life cycle of a consumer that is exposed to uninsurable labor income risk and how he chooses to allocate his savings between a risky and a safe asset, without the possibility to borrow or short-sell.

##### Dynamic problem

The problem of an agent $i$ of age $t$ in the base model is recursively represented as

\begin{split}
V_{i,t} =& \max_{0\leq C_{i,t} \leq X_{i,t}, \alpha_{i,t}\in[0,1]} U(C_{i,t}) + \delta p_t E_t\{ V_{i,t+1} (X_{i,t+1}) \}\\
&\text{s.t}\\
&X_{i,t+1} = Y_{i,t+1} + (X_{i,t} - C_{i,t})(\alpha_{i,t} R_{t+1} + (1-\alpha_{i,t})\bar{R}_f)
\end{split}

where $C_{i,t}$ is consumption, $\alpha_{i,t}$ is the share of savings allocated to the risky asset, $Y_{i,t}$ is labor income, and $X_{i,t}$ represents wealth. The utility function $U(\cdot)$ is assumed to be CRRA in the base model. Extensions beyond the baseline model include an additively separable bequest motive in the utility function. The discount factor is $\delta$ and $p_t$ is the probability of survival from $t$ to $t+1$. Death is certain at a maximum period $T$.

Note that the consumer cannot borrow or short-sell.

The control variables in the problem are $\{C_{it}, \alpha_{it}\}^T_{t=1}$ and the state variables are $\{t, X_{it}, v_{it} \}^T_{t=1}$. The agent solves for  policy rules as a function of the state variables—$C_{it}(X_{it}, v_{it})$ and $\alpha_{it}(X_{it}, v_{it})$.

#### Labor income

An important driver of the paper's results is the labor income process. It is specified as follows:

\begin{equation}
\log Y_{i,t} = f(t,Z_{i,t}) + v_{i,t} + \epsilon_{i,t}, \quad \text{for } t\leq K.
\end{equation}

where $K$ is the (exogenous) age of retirement, $Z_{i,t}$ are demographic characteristics, $\epsilon_{i,t}\sim \mathcal{N}(0,\sigma^2_\epsilon)$ is a transitory shock, and  $v_{i,t}$ is a permanent component following a random walk

\begin{equation}
v_{i,t} = v_{i,t-1} + u_{i,t} = v_{i,t-1} + \xi_t + \omega_{i,t}
\end{equation}

in which the innovation is decomposed into an aggregate ($\xi_t$) and an idiosyncratic component ($\omega_{i,t}$), both following mean-0 normal distributions.

Post-retirement income is a constant fraction $\lambda$ of permanent income in the last working year $K$.

A crucial aspect of the labor income process is that $f(\cdot,\cdot)$ is calibrated to match income profiles in the PSID, capturing the usual humped shape of income across lifetime.

#### Matching labor income in HARK

In HARK's consumption-saving models, the income process takes the form
\begin{equation}
    \ln Y_t = \ln P_t + \ln \theta_t
\end{equation}
where $P_t$ represents permanent income and $\ln \theta_t \sim N(0,\sigma_\theta)$ transitory shocks to income. Permanent income evolves according to
\begin{equation}
    \ln P_{t+1} = \ln \Gamma_{t+1} +  \ln \psi_{t+1} + \ln P_t
\end{equation}
where $\Gamma_{t+1}$ is a deterministic growth factor, and $\ln \psi_{t+1} \sim N(0,\sigma_\psi)$ a permanent income shock.

To represent the author's assumptions in HARK, we express both income processes as sums of deterministic components and i.i.d shocks
\begin{align}
\text{Cocco et. al} &\quad& \ln Y_{i,t} &=& f(t,Z_{i,t}) + v_{i,0} &+&\sum_{k=1}^t u_{i,k} &+&\qquad \varepsilon_{i,t} \\
\text{HARK}        &\quad& \ln Y_{i,t} &=& \ln P_{i,0} + \sum_{k=1}^t \ln \Gamma_k &+&\sum_{k=1}^t \ln \psi_{i,k} &+& \qquad \ln \theta_{i,t}.
\end{align}

These representations make evident the mapping that we use

|HARK | Cocco et. al |
| :---: | :-----------: |
| $\ln$ $P_{i,0}$ | $f(0,Z_{i,0})$ + $v_{i,0}$ |
| $\ln$ $\Gamma_{t+1}$| $f(t+1$, $Z_{i,t+1})$ - $f(t,Z_{i,t})$|
|$\ln$ $\psi_{i,k}$| $u_{i,k}$|
|$\ln$ $\theta_{i,t}$| $\varepsilon_{i,t}$|

and to achieve a retirement income that is equal to a fraction $\lambda$ of permanent income in the last working period $K$, we simply make $\Gamma_{K+1} = \lambda$ and $\Gamma_{t} = 1$ $\forall t>K+1$.

#### Assets and their returns

There are two assets available for consumers to allocate their savings.

- Bonds: paying a risk-free return $\bar{R}_f$.

- Stocks: paying a stochastic return $R_t = \bar{R}_f + \mu + \eta_t$, where the stochastic component $\eta_t \sim \mathcal{N}(0, \sigma^2_\eta)$ is allowed to be correlated with the aggregate labor income innovation $\xi_t$.

The borrowing and short-selling constraints ensure that agents cannot allocate negative dollars to either of these assets or borrow against future labor income or retirement wealth. Recall $\alpha_{i,t}$ is the proportion of the investor's savings that are invested in the risky asset. The model's constraints imply that $\alpha_{i,t}\in[0,1]$ and wealth is non-negative.

### Calibration—Summary

The paper defines and calibrates several parameters which can be broken down into the following categories:

__1. Preferences and life cycle__

 | Parameter | Description | Code | Value |
 |:---:| ---         | ---  | :---: |
 | $\delta$ | Time Preference Factor | $\texttt{DiscFac}$ | 0.96 |
 | $\gamma$ | Coefficient of Relative Risk Aversion| $\texttt{CRRA}$ | 10 |
 | $p_t$ | Survival Probability | $\texttt{LivPrb}$ | [0.6809,0.99845] |
 | $t_0$ | Starting Age | $\texttt{t_start}$ | 20 |
 | $t_r$ | Retirement Age | $\texttt{t_ret}$ | 65 |
 | $T$ | Maximum Age | $\texttt{t_end}$ | 100 |

__2. Income process and the finanical assets__

 | Parameter | Description | Code | Value|
 |:---:| ---         | ---  | :---: |
 | $f(t,Z_{i,t})$| Average income at each stage of life | $\texttt{det_income}$ | $ \exp($0.530339 + 0.16818 $t$+ (0.0323371/10) $t^2$ + (0.0019704/100) $t^3)$ |
 | $\lambda$ | Last Period Labor Income Share for Retirement | $\texttt{repl_fac}$ | 0.68212 |
 | $\log \Gamma$ | Permanent Income Growth Factor | $\texttt{PermGroFac}$ | $\{\log f_{t+1} - \log f_t\}^{t_r+1}_{t=20}$ |
 | $\mathsf{R}$ | Interest Factor | $\texttt{Rfree}$ | 1.02 |
 | $\mu$ | Average Stock Return | $\texttt{RiskyDstnFunc}$ \& $\texttt{RiskyDrawFunc}$ | 1.06 |
 | $\sigma_\eta$ | Std Dev of Stock Returns | $\texttt{RiskyDstnFunc}$ \& $\texttt{RiskyDrawFunc}$ | 0.157 |

__3. Shocks__

 | Parameter | Description | Code | Value |
 |:---:| ---         | ---  | :---: |
 | $\sigma_v$ | Std Dev of Log Permanent Shock| $\texttt{PermShkStd}$ | 0.102956 |
 | $\sigma_\epsilon$ | Std Dev of Log Transitory Shock| $\texttt{TranShkStd}$ | 0.27166 |

For reference, the authors' source Fortran code that includes these paramerization details is available on [Gomes' personal page](http://faculty.london.edu/fgomes/research.html). Code that solves the model is also available in [Julia](https://github.com/econ-ark/HARK/issues/114#issuecomment-371891418).

### Calibration in our implementation.

We replicate the previously defined parameter values and transition processes for state variables and shocks, and format them into the structures required by HARK.

This is done in the external file [Calibration/params.py](https://github.com/matthew-zahn/CGMPort/blob/master/CGMPort-Shared/Code/Python/Calibration/params.py), which we now import.

```python
# Calibrate the model in line with the information above
import sys,os
# The following code navigates to another directory where a python script with the parameters for the model is saved.
sys.path.append(os.path.realpath('Calibration/'))
# Loading the parameters from the Calibration/params.py script
from params import dict_portfolio, time_params, det_income, age_plot_params
```

All of the model's parameters are contained in the structure <tt>dict_portfolio<tt>, which can now be passed to HARK's <tt>PortfolioConsumerType<tt> to build and solve a representation of our problem.

```python
# Solve the model with the given parameters
agent = cpm.PortfolioConsumerType(**dict_portfolio)
agent.solve()
```

```python
len(dict_portfolio['PermGroFac'])
```

    80

### A note on normalization

The problem as specified above makes the value function homogeneous with respect to permanent labor income. This is convenient as it allows for a re-statement of the problem in variables that are normalized by permanent income or its random components, eliminating a state variable.

The authors report (page 497) taking the normalization $v_{i,t} = 1$. This amounts to defining normalized variables $\tilde{\cdot}$ as the original variable divided by $e^{v_{i,t}-1}$. For instance:

\begin{equation}
\tilde{Y_{i,t}} = \frac{Y_{i,t}}{\exp(v_{i,t}-1)} = \frac{\exp (f(t,Z_{i,t}) + v_{i,t} + \varepsilon_{i,t})}{\exp(v_{i,t}-1)} = \exp( f(t,Z_{i,t}) + 1 + \varepsilon_{i,t} )
\end{equation}

These normalized variables have the convenient interpretation of the state that things would be in if, it weren't for permanent shocks. The author's depictions of policy functions are presented in terms of these normalized variables.

On the other hand, HARK normalizes the problem by total permanent income $P_t = \exp (f(t,Z_{i,t}) + v_{i,t})$ and its solution objects are therefore in terms of normalized variables $\hat{\cdot}$, defined as
\begin{equation}
\hat{X_{i,t}} = \frac{X_{i,t}}{P_{i,t}} = \frac{X_{i,t}}{\exp (f(t,Z_{i,t}) + v_{i,t})}.
\end{equation}

Therefore, to present our results in a way consistent with that of the original authors, we would use the following relationship
\begin{equation}
\tilde{X_{i,t}} = \hat{X_{i,t}} \times \exp (f(t,Z_{i,t})+1)
\end{equation}

However, our results are much more consistent with those of the original authors when we take the normalization $v_{i,t} = 0$, which also make sense since it makes the random-walk multiplicative part of permanent income $\exp{v_{i,t}}=1$. We therefore assume this is a typo, take $v_{i,t} = 0$, document this issue in the \textbf{Puzzles} section below, and use the relationship
\begin{equation}
\tilde{X_{i,t}} = \hat{X_{i,t}} \times \exp (f(t,Z_{i,t})).
\end{equation}

The next cell defines our normalization factor

```python
# Define a normalization factor
norm_factor = det_income*np.exp(0)
```

### Key Results

#### The optimal risky asset share

The plot below shows the policy function for the risky portfolio share as a function of wealth at different ages.

The optimal risky share is decreasing in wealth. The authors argue this is due to the fact that, at low levels of wealth, relatively safe human wealth represents a higher fraction of the consumer's wealth, so he shifts his tradeable wealth towards riskier alternatives.

Analyzing the policy rule by age also shows that the risky share increases from young to middle age, and decreases from middle to old age. This is consistent with the previous interpretation: shares trace the humped shape of labor earnings.

These estimates are different from what is produced in the original paper, which are also reproduced below. Generally, the policy functions do not share the same curvature, which leads to greater reductions in the optimal portfolio share at lower levels of wealth.

```python
# Plot portfolio rule
eevalgrid = np.linspace(0,300,100)
plt.figure()
# In the last period of life you consume everything
# so portfolio choice is irrelevant

# Ages
ages = age_plot_params
age_born = time_params['Age_born']
for a in ages:
    plt.plot(eevalgrid,
             agent.solution[a-age_born].ShareFuncAdj(eevalgrid/norm_factor[a-age_born]),
             label = 'Age = %i' %(a))
plt.xlabel('Wealth')
plt.ylabel('Risky portfolio share')
plt.title('Risky Portfolio Policy Function')
plt.legend()
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

We present the author's originally reported policy functions for comparison

#### Consumption behavior

The plot below shows the policy function for consumption as a function of wealth at different ages.

At all age levels consumption increases with wealth. The consumption function also appears to shift upwards as life progresses.

Our consumption policy functions again do not match those of the original paper, which are also reproduced below. Consumption also appears to increase with age in our policy functions that does not come through in the results presented in the paper.

```python
# Plot consumption function
plt.figure()
ages = age_plot_params
for a in ages:
    plt.plot(eevalgrid,
             agent.solution[a-age_born].cFuncAdj(eevalgrid/norm_factor[a-age_born])*norm_factor[a-age_born],
             label = 'Age = %i' %(a))
plt.xlabel('Wealth')
plt.ylabel('Consumption')
plt.title('Consumption Policy Function')
plt.legend()
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

We again present the author's originally reported policy functions for comparison

### Simulations

Using the policy functions obtained from solving the model we present a series of simulations to highlight features of the model.

We first run a few simulations to verify the quality of our calibration.

The figures below show simulated levels of permanent income and risky portfolio shares for 5 agents over their life spans. We can see the model generates a heterogeneous permanent income distribution. Interestingly, all of these agents tend to follow the same general pattern for investing in the risky asset. Early in life, all of their portfolios are invested in the risky asset. This declines as the agent ages and converges to approximately 35% once they reach retirement.

```python
# Set up simulation parameters

# Number of agents and periods in the simulation.
agent.AgentCount = 5 # Number of instances of the class to be simulated.
# Since agents can die, they are replaced by a new agent whenever they do.

# Number of periods to be simulated
agent.T_sim = 80

# Set up the variables we want to keep track of.
agent.track_vars = ['aNrmNow','cNrmNow', 'pLvlNow', 't_age', 'ShareNow','mNrmNow']

# Run the simulations
agent.initializeSim()
agent.simulate()
agent.history

```

```python
# Set up simulation parameters

# Number of agents and periods in the simulation.
agent.AgentCount = 5 # Number of instances of the class to be simulated.
# Since agents can die, they are replaced by a new agent whenever they do.

# Number of periods to be simulated
agent.T_sim = 80

# Set up the variables we want to keep track of.
agent.track_vars = ['aNrmNow','cNrmNow', 'pLvlNow', 't_age', 'ShareNow','mNrmNow']

# Run the simulations
agent.initializeSim()
agent.simulate()

# Present diagnostic plots.
plt.figure()
plt.plot(agent.history['t_age']+time_params['Age_born'], agent.history['pLvlNow'],'.')
plt.xlabel('Age')
plt.ylabel('Permanent income')
plt.title('Simulated Income Paths')
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)

plt.figure()
plt.plot(agent.history['t_age']+time_params['Age_born'], agent.history['ShareNow'],'.')
plt.xlabel('Age')
plt.ylabel('Risky share')
plt.title('Simulated Risky Portfolio Shares')
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

#### The average life cycle patterns

We now increase the number of simulations to examine and compare the behavior of the mean values of variables of interest at different ages, conditional on survival. In each case we present the original plots from the paper for reference.

The plot below illustrates the average dynamics of permanent income, consumption, and market resources across all of the simulated agents. The plot follows the general pattern observed in the original paper. However, our results show that the agents are accumulating significantly more market resources.

```python
# Number of agents and periods in the simulation.
agent.AgentCount = 50 # Number of instances of the class to be simulated.
# Since agents can die, they are replaced by a new agent whenever they do.

# Number of periods to be simulated
agent.T_sim = 80*50

# Run the simulations
agent.initializeSim()
agent.simulate()

raw_data = {'Age': agent.history['t_age'].flatten()+time_params['Age_born'] - 1,
            'pIncome': agent.history['pLvlNow'].flatten(),
            'rShare': agent.history['ShareNow'].flatten(),
            'nrmM': agent.history['mNrmNow'].flatten(),
            'nrmC': agent.history['cNrmNow'].flatten()}

Data = pd.DataFrame(raw_data)
Data['Cons'] = Data.nrmC * Data.pIncome
Data['M'] = Data.nrmM * Data.pIncome

# Find the mean of each variable at every age
AgeMeans = Data.groupby(['Age']).mean().reset_index()
```

```python
plt.figure()
plt.plot(AgeMeans.Age, AgeMeans.pIncome,
         label = 'Income')
plt.plot(AgeMeans.Age, AgeMeans.M,
         label = 'Market resources')
plt.plot(AgeMeans.Age, AgeMeans.Cons,
         label = 'Consumption')
plt.legend()
plt.xlabel('Age')
plt.title('Variable Means Conditional on Survival')
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

```python
# Find age percentiles
AgePC5 = Data.groupby(['Age']).quantile(0.05).reset_index()
AgePC95 = Data.groupby(['Age']).quantile(0.95).reset_index()

# plot till death - 1
age_1 = time_params['Age_death'] - time_params['Age_born']

plt.figure()
plt.ylim([0, 1.1])
plt.plot(AgeMeans.Age[:age_1], AgeMeans.rShare[:age_1], label = 'Mean')
plt.plot(AgePC5.Age[:age_1], AgePC5.rShare[:age_1], '--r', label='Perc. 5')
plt.plot(AgePC95.Age[:age_1], AgePC95.rShare[:age_1], '--g', label = 'Perc. 95')
plt.legend()

plt.xlabel('Age')
plt.ylabel('Risky Share')
plt.title('Risky Portfolio Share Mean Conditional on Survival')
plt.grid()

if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

### Other results in the original paper

#### The welfare implications of different allocation rules

The authors next conduct a welfare analysis of different allocation rules, including popular heuristics. The rules are presented in the next figure.

The utility cost of each policy in terms of constant consumption streams with respect to the authors calculated optimal policy function is reported in the next table.

Interestingly, the "no-income" column corresponds to the usual portfolio choice result of the optimal share being the quotient of excess returns and risk times relative risk aversion, disregarding labor income. The experiment shows this allocation produces substantial welfare losses.

#### Heterogeneity and sensitivity analysis

The authors also considered a number of extensions to the baseline model. These are summarized below along with their main conclusions.

- Labor income risk: Income risk may vary across employment sectors relative to the baseline model. The authors examine extreme cases for industries that have a large standard deviation and temporary income shocks. While some differences appear across sectors, the results are generally in line with the baseline model.
- Disastrous labor income shocks: The authors find that even a small probability of zero labor income lowers the optimal portfolio allocation in stocks, while the qualitative features of the baseline model are preserved.
- Uncertain retirement income: The authors consider two types of uncertainty for retirement income; it is stochastic and correlated with current stock market performance and allowing for disastrous labor income draws before retirement. The first extension has results essentially the same as the baseline case. The second leads to more conservative portfolio allocations but is broadly consistent with the baseline model.
- Endogenous borrowing constraints: The authors add borrowing to their model by building on credit-market imperfections. They find that the average investor borrows about \$5,000 and are in debt for most of their working life. The agents eventually pay off this debt and save for retirement. Relative to the benchmark model, the investor has put less of their money in their portfolio and arrive at retirement with substantially less wealth. These results are particularly pronounced at the lower end of the income distribution relative to the higher end. Additional details are available in the text.
- Bequest motive: The authors introduce a bequest motive into the agent's utility function (i.e., $b>0$). Young investors are more impatient and tend to save less for bequests. As the agent ages, savings increases and is strongest once the agent retires. This leads to effects on the agent's portfolio allocation. Taking a step-back however, these effects are not very large unless $b$ is large.
- Educational attainment: The authors generally find that savings are consistent across education groups. They note that for a given age, the importance of future income is increasing with education level. This implies that riskless asset holdings are larger for these households.
- Risk aversion and intertemporal substitution: Lowering the level of risk aversion in the model leads to changes in the optimal portfolio allocation and wealth accumulation. Less risk-averse investors accumulate less precautionary savings and invest more in risky assets.

### Conclusion

This article provides a dynamic model with accurate lifetime income profiles in which labor income increases risky asset holdings, as it is seen as a closer substitute of risk-free assets. It finds an optimal risky asset share that decreases in wealth and with age, after middle age. The model is also used to show that ignoring labor income for portfolio allocation can generate substantial welfare losses.

### Puzzles and Questions

- Table 4 says stock returns are $0.06$. They might mean that the equity premium $\mu$ is $0.06$.
- The authors report taking the normalization $v_{i,t} = 1$. However the ranges of their results seem more consistent with $v_{i,t} = 0$ so that $\exp (v_{i,t}) = 1$, which also makes more sense for interpretation.

### Robustness Analyses

Given the differences between our results and the original paper, we did a number of checks to ensure our model was behaving consistently with well-established theoretical results. Specifically we checked:
- For an infinitely lived agent with log normal returns, that their optimal portfolio allocation converges to the Campbell-Viceira (2002) approximation to the optimal portfolio share in Merton-Samuelson (1969) model.
- For an infinitely lived agent with no labor income that can only invest in a single risky asset, that their marginal propensity to consumer converges to the theoretical MPC of Merton-Samuelson (1969).
- For an agent facing no labor income risk, that their consumption patterns precisely match the results from a perfect foresight solution.

In all three cases, we verified that our HARK model holds up to these results. More details and specific results are available upon request.

As the HARK toolkit continues to develop, there are additional sensitivities that we can perform to further check the credibility of our results. Specifically, once human wealth is available in the $\texttt{PortfolioConsumerType}$ class, we can perform the following additional checks, which were kindly suggested by Professor Sylvain Catherine:
- Shut down the income risk and remove retirement income. The solution to this new problem are provided by Merton 1971. Basically, you capitalize future earnings as an endowment of risk free asset. Then the equity share should be such that Equity/(Wealth+NPV of Human capital) is the same as the equity share in Merton 1969.
- Adding back the permanent income risk and check if the equity share is consistent with Viceira 2001. Viceira tells you something like this: $\pi = \frac{\mu - r}{\gamma \sigma^2_s} + \left(\frac{\mu - r}{\gamma \sigma^2_s} - \beta_{HC} \right) \frac{HC}{W}$, where $\beta_{HC} = \frac{\text{Cov}(r_{HC},r_s)}{\text{Var}(r_s)}$. In the CGM problem it is easy to compute $\beta_{HC}$ because earnings follow a simple random walk. HC is the NPV of human capital, which you can approximate very well by discounting expected earnings by $r+\beta_{HC}*(rm-r)$.

### Bibliographic entry of the original article

```python
### Bibtex entry
'''
@article{10.1093/rfs/hhi017,
    author = {Cocco, João F. and Gomes, Francisco J. and Maenhout, Pascal J.},
    title = "{Consumption and Portfolio Choice over the Life Cycle}",
    journal = {The Review of Financial Studies},
    volume = {18},
    number = {2},
    pages = {491-533},
    year = {2005},
    month = {02},
    abstract = "{This article solves a realistically calibrated life cycle model of consumption and portfolio choice with non-tradable labor income and borrowing constraints. Since labor income substitutes for riskless asset holdings, the optimal share invested in equities is roughly decreasing over life. We compute a measure of the importance of human capital for investment behavior. We find that ignoring labor income generates large utility costs, while the cost of ignoring only its risk is an order of magnitude smaller, except when we allow for a disastrous labor income shock. Moreover, we study the implications of introducing endogenous borrowing constraints in this incomplete-markets setting.}",
    issn = {0893-9454},
    doi = {10.1093/rfs/hhi017},
    url = {https://doi.org/10.1093/rfs/hhi017},
    eprint = {http://oup.prod.sis.lan/rfs/article-pdf/18/2/491/24421441/hhi017.pdf},
}
'''
```

    '\n@article{10.1093/rfs/hhi017,\n    author = {Cocco, João F. and Gomes, Francisco J. and Maenhout, Pascal J.},\n    title = "{Consumption and Portfolio Choice over the Life Cycle}",\n    journal = {The Review of Financial Studies},\n    volume = {18},\n    number = {2},\n    pages = {491-533},\n    year = {2005},\n    month = {02},\n    abstract = "{This article solves a realistically calibrated life cycle model of consumption and portfolio choice with non-tradable labor income and borrowing constraints. Since labor income substitutes for riskless asset holdings, the optimal share invested in equities is roughly decreasing over life. We compute a measure of the importance of human capital for investment behavior. We find that ignoring labor income generates large utility costs, while the cost of ignoring only its risk is an order of magnitude smaller, except when we allow for a disastrous labor income shock. Moreover, we study the implications of introducing endogenous borrowing constraints in this incomplete-markets setting.}",\n    issn = {0893-9454},\n    doi = {10.1093/rfs/hhi017},\n    url = {https://doi.org/10.1093/rfs/hhi017},\n    eprint = {http://oup.prod.sis.lan/rfs/article-pdf/18/2/491/24421441/hhi017.pdf},\n}\n'


```python
from sympy import *
init_printing(use_unicode=True)
from sympy.codegen.ast import Assignment
```

```python
C = Matrix( symarray('C', (2,2)) )
R = Matrix( symarray('R', (2,2)) )
n = Matrix( symarray('n', (2)) )
t = Matrix( symarray('t', (2)) )
```

```python
R[0,0]=t[0]
R[1,0]=t[1]
R[0,1]=n[0]
R[1,1]=n[1]
```

```python
C
```

```python
simplify(transpose(R)*C*R)
```

```python
TensRot = Matrix( symarray('r', (3,3)) )

TensRot[0,0]=t[0]*t[0]
TensRot[0,1]=t[1]*t[1]
TensRot[0,2]=t[0]*t[1] + t[1]*t[0]
TensRot[1,0]=n[0]*n[0]
TensRot[1,1]=n[1]*n[1]
TensRot[1,2]=n[0]*n[1] + n[1]*n[0]
TensRot[2,0]=t[0]*n[0]
TensRot[2,1]=t[1]*n[1]
TensRot[2,2]=t[0]*n[1] + t[1]*n[0]
TensRot
```

```python
InvTensRot = Matrix( symarray('r', (3,3)) )

InvTensRot[0,0]=t[0]*t[0]
InvTensRot[0,1]=n[0]*n[0]
InvTensRot[0,2]=t[0]*n[0] + t[0]*n[0]
InvTensRot[1,0]=t[1]*t[1]
InvTensRot[1,1]=n[1]*n[1]
InvTensRot[1,2]=t[1]*n[1] + t[1]*n[1]
InvTensRot[2,0]=t[0]*t[1]
InvTensRot[2,1]=n[0]*n[1]
InvTensRot[2,2]=t[0]*n[1] + t[1]*n[0]
InvTensRot
```

```python
simplify(R*C*transpose(R))
```

```python
print(latex(InvTensRot))
```

    \left[\begin{matrix}t_{0}^{2} & n_{0}^{2} & 2 n_{0} t_{0}\\t_{1}^{2} & n_{1}^{2} & 2 n_{1} t_{1}\\t_{0} t_{1} & n_{0} n_{1} & n_{0} t_{1} + n_{1} t_{0}\end{matrix}\right]

```python
print(latex(TensRot))
```

    \left[\begin{matrix}t_{0}^{2} & t_{1}^{2} & 2 t_{0} t_{1}\\n_{0}^{2} & n_{1}^{2} & 2 n_{0} n_{1}\\n_{0} t_{0} & n_{1} t_{1} & n_{0} t_{1} + n_{1} t_{0}\end{matrix}\right]

```python
theta = Symbol("theta")
n[0] = -sin(theta)
n[1] =  cos(theta)

t[0]= n[1]
t[1]= -n[0]
```

```python
InvTensRot[0,0]=t[0]*t[0]
InvTensRot[0,1]=n[0]*n[0]
InvTensRot[0,2]=t[0]*n[0] + t[0]*n[0]
InvTensRot[1,0]=t[1]*t[1]
InvTensRot[1,1]=n[1]*n[1]
InvTensRot[1,2]=t[1]*n[1] + t[1]*n[1]
InvTensRot[2,0]=t[0]*t[1]
InvTensRot[2,1]=n[0]*n[1]
InvTensRot[2,2]=t[0]*n[1] + t[1]*n[0]

TensRot[0,0]=t[0]*t[0]
TensRot[0,1]=t[1]*t[1]
TensRot[0,2]=t[0]*t[1] + t[1]*t[0]
TensRot[1,0]=n[0]*n[0]
TensRot[1,1]=n[1]*n[1]
TensRot[1,2]=n[0]*n[1] + n[1]*n[0]
TensRot[2,0]=t[0]*n[0]
TensRot[2,1]=t[1]*n[1]
TensRot[2,2]=t[0]*n[1] + t[1]*n[0]

```

```python
sigma = Matrix( symarray('\sigma', (3)) )
sigma[0]=1
sigma[1]=3
sigma[2]=2

(TensRot*sigma).subs(theta,20*pi/180).evalf()
```

```python
(Transpose(TensRot)*sigma).subs(theta,20*pi/180).evalf()
```

```python

(InvTensRot*sigma).subs(theta,-50*pi/180).evalf()
```

```python
(Transpose(TensRot)*TensRot).subs(theta,20*pi/180).evalf()
```

```python
(InvTensRot*TensRot).subs(theta,20*pi/180).evalf()
```

```python

```

```python
# EXTRA
```

```python
import numpy as np
import matplotlib.pyplot as plt

import math

def rotMatrix(angle):
    c = np.cos(np.radians(angle))
    s = np.sin(np.radians(angle))
    return np.array([[c, -s], [s, c]])

v = np.array([0,1])
v30 = rotMatrix(30).dot(v)

plt.arrow(0,0,v[0],v[1], head_width=0.1, head_length=0.1)
plt.arrow(0,0,v30[0],v30[1],head_width=0.1, head_length=0.1)

v = np.array([1,0])
v30 = rotMatrix(30).dot(v)

plt.arrow(0,0,v[0],v[1], head_width=0.1, head_length=0.1)
plt.arrow(0,0,v30[0],v30[1],head_width=0.1, head_length=0.1)

plt.axis([-2,2,-2,2])
plt.grid
plt.axis('square')
plt.show()
```

```python

```