# Model 1 ~ What sport are we most likely to succeed in?

In order to answer this question we will split this up in two smaller questions which we will be tackling individually:

1. **Which sport should we participate in the Olympics in?:**

2. **What probability do we have of winning a medal at a given sport?:**

First step however is to do some basic data pre-processing steps to adjust the data set to our benefit and obtain better final results (most of these data processing steps are done based on our own logic and considering that these steps would improve  our models).

We will be going step by step throughout this notebook, explaining at every step what we are doing and why.

We would also like to highlight that we have leveraged on a kaggle notebook which also looked into the first subquestion. We have used a lot of his/her ideas and code, but have extended the model based on two main things:

- Include additional feature (medals won per country per sport)
- Divide models based on sex

The kaggle notebook can be found under: https://www.kaggle.com/deanpower/what-sport-will-you-compete-in

Reviewing this notebook has allowed us to better understand how to do a "real" machine learning notebook and we consider we have been able to learn a lot from it despite having copied a lot of it's ideas.

##### Import libraries

```python
%load_ext autoreload
%autoreload 2
```

```python
import model1_funcs as M1
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import (
    classification_report,
    f1_score,
    make_scorer,
    precision_recall_fscore_support,
    precision_score,
    recall_score,
)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import LinearSVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
pd.options.mode.chained_assignment = None  # default='warn'
```

```python

```

##### Read data

```python
df = pd.read_csv('data/athlete_events.csv', index_col=0)
```

```python
df.head()
```

      Name
      Sex
      Age
      Height
      Weight
      Team
      NOC
      Games
      Year
      Season
      City
      Sport
      Event
      Medal

      ID

      1
      A Dijiang
      M
      24.0
      180.0
      80.0
      China
      CHN
      1992 Summer
      1992
      Summer
      Barcelona
      Basketball
      Basketball Men's Basketball
      NaN

      2
      A Lamusi
      M
      23.0
      170.0
      60.0
      China
      CHN
      2012 Summer
      2012
      Summer
      London
      Judo
      Judo Men's Extra-Lightweight
      NaN

      3
      Gunnar Nielsen Aaby
      M
      24.0
      NaN
      NaN
      Denmark
      DEN
      1920 Summer
      1920
      Summer
      Antwerpen
      Football
      Football Men's Football
      NaN

      4
      Edgar Lindenau Aabye
      M
      34.0
      NaN
      NaN
      Denmark/Sweden
      DEN
      1900 Summer
      1900
      Summer
      Paris
      Tug-Of-War
      Tug-Of-War Men's Tug-Of-War
      Gold

      5
      Christine Jacoba Aaftink
      F
      21.0
      185.0
      82.0
      Netherlands
      NED
      1988 Winter
      1988
      Winter
      Calgary
      Speed Skating
      Speed Skating Women's 500 metres
      NaN

```python

```

##### Basic explanatory analysis

```python
df.describe()
```

      Age
      Height
      Weight
      Year

      count
      261642.000000
      210945.000000
      208241.000000
      271116.000000

      mean
      25.556898
      175.338970
      70.702393
      1978.378480

      std
      6.393561
      10.518462
      14.348020
      29.877632

      min
      10.000000
      127.000000
      25.000000
      1896.000000

      25%
      21.000000
      168.000000
      60.000000
      1960.000000

      50%
      24.000000
      175.000000
      70.000000
      1988.000000

      75%
      28.000000
      183.000000
      79.000000
      2002.000000

      max
      97.000000
      226.000000
      214.000000
      2016.000000

```python
df.isna().sum()
```

    Name           0
    Sex            0
    Age         9474
    Height     60171
    Weight     62875
    Team           0
    NOC            0
    Games          0
    Year           0
    Season         0
    City           0
    Sport          0
    Event          0
    Medal     231333
    dtype: int64

```python
# Show missing evolution of Age, Height and Weight
M1.show_missing_evolution(df)
```

```python

```

##### Data Pre-processing

We will now apply all the data pre-processing steps in one single function. For transparency purposes we will however explain what this function does over the input dataframe:

- Remove observations before 1960 based on the previous plot.
- Split Athletics sport into sub-sports based on its events. This is due to the fact that athletics covers a very wide range of sport events which are probably not comparable.
- Replace missing values with the average feature values at Sport, Sex and Year level. This can give us an approximation of the height, age and year of each athlete that has missing data.
- We will only foocus on one olympic game season. For this analysis we have choosen the summer games since none of our team-mates is especially keen of winter sports.
- We will only focus our analysis on sports that have competed during the last Olympic games. This is motivated due to the fact that our test data will be the last Olympic games and we want both data set (train and test) to have the same sports.
- Construct one of the target variables (1 if an athlete has won a medal, 0 otherwise). This will be used to identify team sports as well.
- Convert team sports it one athlete. This is done to reduce the number of medals won by sport, and that it does not depend on the number of team members.
- Add column with the number of medals won per country at sport and sex level.

```python
df_clean = M1.data_preprocessing(df)
```

```python

```

## 1. What probability do we have of winning a medal at a given sport?

##### Generate Dictionary that maps country and sport to number of medals won

For later on, it will be very comfortable to have a dictionary where we can quickly access the number of medals won by a country, sport and sex.

```python
medals_country_dict = M1.generate_mapping_table_country_sport_sex(df_clean)
```

```python

```

##### Standardize features

We have standardized our data since the main features we will be using throughout our analyis have different units and magnituude

```python
df_standardized = M1.standardize_features(df_clean)
```

```python

```

##### Generate training and testing dataframe samples

We have decided to split the data as follows:

- **Training Sample:** All observations from 1960 until 2012
- **Testing Sample:** All observations from 2016

The motivation for this decision is that our model should be able to do predictions over the upcoming olympic games given a set of historical olympic game features. Any other random split would not have made too much sense from our point of view.

```python
df_train, df_test = M1.generate_sample_split(df_standardized)
```

```python

```

##### Generate a set of additional columns to increase the possible degrees of our logistic regression fit

In order to train the logistic regression model (to predict the probability of winning a medal or not) we will also investigate how well the model performs if we consider higher degree orders of our classification model

```python
temp_list = M1.extend_dataset_features([df_train, df_test])
df_train_extended = temp_list[0]
df_test_extended = temp_list[1]
```

```python

```

### Using all the different features (including the new ones we have just generated), which gives the best performances at sport and sex level?

1. Compute the optimized set of theta values for the given input training set
2. From all these potential theta values, which one gives the best results over the test data?

##### 1. Compute the optimized set of theta values for the given input training set

```python
feature_permutations = {'featureBag_1': ['Age', 'Height', 'Weight', 'Medals_per_country', 'Age^2', 'Height^2', 'Weight^2',
                                         'Medals_per_country^2', 'Age*Height', 'Age*Weight', 'Age*Medals_per_country',
                                         'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country', 'Age^3',
                                         'Height^3', 'Weight^3', 'Medals_per_country^3'],
                        'featureBag_2': ['Age', 'Height', 'Weight', 'Medals_per_country'],
                        'featureBag_3': ['Age', 'Height', 'Weight', 'Medals_per_country', 'Age^2', 'Height^2', 'Weight^2',
                                         'Medals_per_country^2', 'Age*Height', 'Age*Weight', 'Age*Medals_per_country',
                                         'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country'],
                        'featureBag_4': ['Age^2', 'Height^2', 'Weight^2', 'Medals_per_country^2'],
                        'featureBag_5': ['Height', 'Weight', 'Medals_per_country'],
                        'featureBag_6': ['Age', 'Height', 'Weight', 'Age^2', 'Height^2', 'Weight^2'],
                        'featureBag_7': ['Age*Height', 'Age*Weight', 'Age*Medals_per_country',
                                         'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country'],
                        'featureBag_8': ['Age^3', 'Height^3', 'Weight^3', 'Medals_per_country^3'],
                        'featureBag_9': ['Medals_per_country', 'Medals_per_country^2', 'Medals_per_country^3']}
```

```python
hyperparameter_thetas = M1.optimal_feature_combinations(df_train_extended, feature_permutations)
```

    Model 0
    Features inspected: ['Age', 'Height', 'Weight', 'Medals_per_country', 'Age^2', 'Height^2', 'Weight^2', 'Medals_per_country^2', 'Age*Height', 'Age*Weight', 'Age*Medals_per_country', 'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country', 'Age^3', 'Height^3', 'Weight^3', 'Medals_per_country^3']
    Sex: F
    Sex: M
    ====================
    Model 1
    Features inspected: ['Age', 'Height', 'Weight', 'Medals_per_country']
    Sex: F
    Sex: M
    ====================
    Model 2
    Features inspected: ['Age', 'Height', 'Weight', 'Medals_per_country', 'Age^2', 'Height^2', 'Weight^2', 'Medals_per_country^2', 'Age*Height', 'Age*Weight', 'Age*Medals_per_country', 'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country']
    Sex: F
    Sex: M
    ====================
    Model 3
    Features inspected: ['Age^2', 'Height^2', 'Weight^2', 'Medals_per_country^2']
    Sex: F
    Sex: M
    ====================
    Model 4
    Features inspected: ['Height', 'Weight', 'Medals_per_country']
    Sex: F
    Sex: M
    ====================
    Model 5
    Features inspected: ['Age', 'Height', 'Weight', 'Age^2', 'Height^2', 'Weight^2']
    Sex: F
    Sex: M
    ====================
    Model 6
    Features inspected: ['Age*Height', 'Age*Weight', 'Age*Medals_per_country', 'Height*Weight', 'Height*Medals_per_country', 'Weight*Medals_per_country']
    Sex: F
    Sex: M
    ====================
    Model 7
    Features inspected: ['Age^3', 'Height^3', 'Weight^3', 'Medals_per_country^3']
    Sex: F
    Sex: M
    ====================
    Model 8
    Features inspected: ['Medals_per_country', 'Medals_per_country^2', 'Medals_per_country^3']
    Sex: F
    Sex: M
    ====================

```python
hyperparameter_thetas['featureBag_2']['M']
```

      Intercept
      Age
      Height
      Weight
      Medals_per_country

      Archery
      -1.263956
      -0.338999
      -0.097871
      0.090982
      4.022280

      Athletics Endurance
      -2.873725
      -0.060190
      0.026015
      0.143287
      1.020414

      Athletics Sprints
      -3.316379
      -0.030952
      0.264165
      0.366035
      0.449914

      Athletics Throws
      -3.008692
      0.060549
      0.461448
      0.053348
      1.513966

      Badminton
      -1.069912
      -0.063780
      -0.031614
      -0.192227
      4.040050

      Basketball
      -3.326468
      -1.018428
      1.022995
      0.764390
      3.796012

      Beach Volleyball
      -3.095074
      0.215885
      0.252813
      0.540710
      0.606925

      Boxing
      -1.955068
      0.064236
      0.105702
      0.101900
      1.147090

      Canoeing
      -2.316584
      -0.003184
      -0.150159
      0.709292
      1.543597

      Cycling
      -3.037151
      -0.064725
      0.012476
      0.900109
      1.766603

      Diving
      -1.940929
      0.125516
      0.153418
      -0.345167
      2.660342

      Equestrianism
      -2.392121
      0.093966
      0.128429
      -0.305629
      2.866382

      Fencing
      -2.919638
      -0.021163
      0.294158
      0.102774
      1.467790

      Football
      -0.362940
      1.056066
      0.410131
      0.813241
      2.611478

      Gymnastics
      -3.707569
      -0.100438
      -0.306858
      0.121907
      1.244677

      Handball
      -2.400230
      0.674572
      0.601462
      0.581465
      1.492534

      Hockey
      -0.388756
      -0.008819
      0.492113
      0.340457
      3.016240

      Judo
      -1.949442
      -0.075557
      0.024617
      0.105297
      1.953922

      Modern Pentathlon
      -1.482327
      0.266981
      0.072233
      0.297456
      3.777742

      Rowing
      -2.157699
      0.453572
      0.036568
      0.429843
      2.081328

      Sailing
      -2.173301
      0.054126
      0.169097
      0.078541
      1.978570

      Shooting
      -2.545168
      -0.141328
      0.052823
      0.076070
      1.643497

      Swimming
      -3.715960
      -0.013741
      0.567182
      0.098443
      0.383196

      Table Tennis
      -1.434151
      -0.412283
      0.253649
      -0.010713
      4.474477

      Taekwondo
      -0.416472
      -0.272944
      0.862232
      -0.427708
      3.541313

      Tennis
      -2.098668
      -0.062102
      0.318348
      0.257920
      2.628818

      Trampolining
      -0.950576
      -0.443620
      0.352858
      0.048102
      0.907100

      Triathlon
      -2.472077
      -0.271690
      0.144110
      0.416322
      0.733031

      Volleyball
      -0.594638
      1.455339
      -0.415285
      0.704736
      1.901708

      Water Polo
      -1.657350
      -0.277180
      0.490428
      0.779911
      3.065504

      Weightlifting
      -2.287220
      -0.064086
      -0.322242
      0.273738
      2.136439

      Wrestling
      -2.230206
      0.029324
      0.025920
      0.043009
      0.854385

##### 2. From all these potential theta values, which one gives the best results over the test data?

##### We now have the theta values for each sport. Several observations can already be made from the previous:

- Ideal theta values will be estimated at sport level (for some sports the country might have more influence, in other sport we might prefer giving more weight to the variable height, etc.)
- In general, the variable that seems to have most weight on the prediction of winning a gold medal or not is Medals_per_country (this theta has the highest weight).
- The other features which should intuitevely also be relevant (age, height, weight) seem to be less relevant. This is also an expected (although undesired) result since already the best althelets arrive at the olympics, regardless of their physical features. Another relevant and probably more insightful variable would be the number of training hours, age at which they started practicing the sport, etc.

##### Apply model on test data and see how well is predicts medal winners (we will compute performance using f1-score, which considers both recall and precision)

```python
df_female_performance, df_male_performance = M1.model_performance(df_test_extended, hyperparameter_thetas,
                                                                  print_evolution=True)
df_performance = {'F': df_female_performance,
                  'M': df_male_performance}
```

    featureBag_1
    Sex:  M
    Sex:  F
    ===========================
    featureBag_2
    Sex:  M
    Sex:  F
    ===========================
    featureBag_3
    Sex:  M
    Sex:  F
    ===========================
    featureBag_4
    Sex:  M
    Sex:  F
    ===========================
    featureBag_5
    Sex:  M
    Sex:  F
    ===========================
    featureBag_6
    Sex:  M
    Sex:  F
    ===========================
    featureBag_7
    Sex:  M
    Sex:  F
    ===========================
    featureBag_8
    Sex:  M
    Sex:  F
    ===========================
    featureBag_9
    Sex:  M
    Sex:  F
    ===========================

```python
df_performance['M']
```

      Model features
      Archery
      Athletics Endurance
      Athletics Sprints
      Athletics Throws
      Badminton
      Basketball
      Beach Volleyball
      Boxing
      Canoeing
      ...
      Swimming
      Table Tennis
      Taekwondo
      Tennis
      Trampolining
      Triathlon
      Volleyball
      Water Polo
      Weightlifting
      Wrestling

      0
      featureBag_1
      0.444444
      0.066667
      0.055556
      0.000000
      0.2
      0.333333
      0.4
      0.095238
      0.108108
      ...
      0.072727
      0.666667
      0.105263
      0.000000
      0.000000
      0.333333
      0.666667
      0.333333
      0.148148
      0.039216

      1
      featureBag_2
      0.222222
      0.066667
      0.000000
      0.133333
      0.2
      0.666667
      0.8
      0.095238
      0.054054
      ...
      0.072727
      0.666667
      0.210526
      0.285714
      0.000000
      0.000000
      0.666667
      0.333333
      0.148148
      0.039216

      2
      featureBag_3
      0.444444
      0.000000
      0.111111
      0.000000
      0.2
      0.333333
      0.8
      0.047619
      0.054054
      ...
      0.072727
      0.666667
      0.105263
      0.000000
      0.333333
      0.666667
      0.666667
      0.333333
      0.148148
      0.000000

      3
      featureBag_4
      0.222222
      0.066667
      0.000000
      0.133333
      0.2
      0.333333
      0.8
      0.047619
      0.108108
      ...
      0.109091
      0.000000
      0.210526
      0.000000
      0.333333
      0.333333
      0.666667
      0.333333
      0.074074
      0.000000

      4
      featureBag_5
      0.222222
      0.066667
      0.000000
      0.133333
      0.2
      0.666667
      0.8
      0.095238
      0.054054
      ...
      0.109091
      0.666667
      0.210526
      0.285714
      0.000000
      0.000000
      1.000000
      0.333333
      0.148148
      0.039216

      5
      featureBag_6
      0.222222
      0.000000
      0.000000
      0.000000
      0.0
      0.333333
      0.8
      0.095238
      0.054054
      ...
      0.000000
      0.222222
      0.210526
      0.000000
      0.333333
      0.000000
      0.666667
      0.333333
      0.074074
      0.000000

      6
      featureBag_7
      0.000000
      0.066667
      0.000000
      0.133333
      0.0
      0.666667
      0.8
      0.095238
      0.054054
      ...
      0.036364
      0.000000
      0.210526
      0.285714
      0.333333
      0.000000
      0.666667
      0.333333
      0.148148
      0.000000

      7
      featureBag_8
      0.000000
      0.066667
      0.000000
      0.133333
      0.4
      0.333333
      0.8
      0.095238
      0.108108
      ...
      0.109091
      0.000000
      0.210526
      0.000000
      0.333333
      0.000000
      0.666667
      0.333333
      0.148148
      0.039216

      8
      featureBag_9
      0.444444
      0.000000
      0.055556
      0.000000
      0.2
      0.666667
      0.4
      0.095238
      0.108108
      ...
      0.109091
      0.666667
      0.315789
      0.000000
      0.666667
      0.666667
      1.000000
      0.333333
      0.074074
      0.000000

9 rows × 33 columns

```python

```

##### Evaluate Model over a set of input variables

```python
sex = 'M'
country = 'ESP'
sport = 'Basketball'
age = 20
height = 200
weight = 100

M1.compute_probability_medal(sex, country, sport, age, height, weight, medals_country_dict, df_performance,
                             hyperparameter_thetas, df_clean=df_clean)
```

    0.6429739404621343

```python

```

##### Inspect features against target variable

```python
M1.plot_sport_features('Swimming', df_test, x_feature='Age', y_feature='Height')
```

```python

```

## Which sport should we participate in the Olympics in?

We will now start looking into the models that can tell us what sport we are best suited for a given athlete. For this we have investigated the following models using a 5-fold cross validation approach, allowing us to decide which of the hyperparameters of each model would be best suited over the training sample (metric to choose which hyperparameter is best is the F1-score). From these best models we have then evaluated them over the test data sample (2016 Olympic games) to see which of the models would outperform the rest.

1. **Decision Tree Classifier:** Basic Classifier which sets up a set of decisions (controlled by the depth of the tree) which converts input features (branches) into categories or target values (leaves) through a set of boolean if statements. One of the advantages os such models is that it is easily reproducible and therefore can be validated easily.
2. **Random Forest Classifier:** The basic principal is very similar to the decision tree classifier. The main difference is that the random forest classifies based on some set of bootstraps over the initial data set. This allows our model to be more flexible when predicting unseen data since it trains over more data sets than the basic decision tree classifier.
3. **Logisitic Regression:** The main difference between our model and this logistic regression is that this model will be used to classify between multiple categories. In the previous model we were classifying a athelete between winning or not a medal. The method used in this case is the one-vs-rest approach, i.e. basically the logistic regression classifies a given catgegory into 1 and all the rest into 0 and generates a normal logisit regression over these binary categories.
4. **K-Nearest Neighbours:** One of the most popular classification models used throughout the data science community. The principal is very straight forward and easy: Given a set of clusters (choosen by the user if the "real" number of clusters is not known, or inferred by the data, in our case the number of groups should ideally be equal to the number of sport types), look for groups over which a specific distance measure is minimized.
5. **Linear SVC:** Algorithm to classify observations into categories according to some features by maximizing the margin between two different category examples.

These models will be evaluated similarly to the previous model, i.e. using the following metrics:

1. **Accuracy:** Percentage of atheletes correctly classified

\begin{align}
Accuracy = \frac{True Positives + True Negatives}{True Positives + True Negatives + False Positives + False Negatives}
\end{align}

2. **Weighted precision:** Percentage of athelets correctly classified out of all positive results

\begin{align}
Precision = \frac{True Positives}{True Positives + False Positives}
\end{align}

3. **Weighted recall:** Percentage of athlets correctly classified from all the athletes that have been assigned to a specific class

\begin{align}
Recall = \frac{True Positives}{True Positives + False Negatives}
\end{align}

4. **Weighted F1:** Similarly as in the previous model selection, this will be the utlimate metric over which we will decide which model is the best. This metric combines the recall and precision, and for this reason is sometimes considered to be the most accurate metric to use.

\begin{align}
F1 = 2 \cdot \frac{recall \cdot precision}{recall + precision}
\end{align}

```python

```

##### Convert sports into integer labels

```python
le = LabelEncoder()
df_standardized['Sport_label'] = le.fit_transform(df_standardized.Sport)
```

```python

```

##### Generate training and testing samples (we continue following the same approach as before):

As in the previous models, we will train our model over all the olympic game history and then apply the best hyper-parameters over the unseen 2016 Olympic Games

```python
df_train_Female_X, df_train_Male_X, df_train_Female_Y, df_train_Male_Y, \
df_test_Female_X, df_test_Male_X, df_test_Female_Y, df_test_Male_Y = M1.generate_sample_split_advanced(df_standardized)

data_sets = {'F': {'Train': [df_train_Female_X, df_train_Female_Y],
                   'Test':  [df_test_Female_X,  df_test_Female_Y]},
             'M': {'Train': [df_train_Male_X, df_train_Male_Y],
                   'Test':  [df_test_Male_X,  df_test_Male_Y]}}
```

```python
## Count sport labels
#train_counts_Female = df_train_Female_Y.value_counts()
#train_counts_Male = df_train_Male_Y.value_counts()
#
#test_counts_Female = df_test_Female_Y.value_counts()
#test_counts_Male = df_test_Male_Y.value_counts()
#
## Assign labels to actual sport names
#train_counts_Female.index = le.inverse_transform(train_counts_Female.index)
#train_counts_Male.index = le.inverse_transform(train_counts_Male.index)
#
#test_counts_Female.index = le.inverse_transform(test_counts_Female.index)
#test_counts_Male.index = le.inverse_transform(test_counts_Male.index)
```

```python
# Define scoring dict to make loop easier to read afterwards
scoring = {
    'accuracy': 'accuracy',
    'weighted_precision': make_scorer(precision_score, average='weighted'),
    'weighted_recall': make_scorer(recall_score, average='weighted'),
    'weighted_F1': make_scorer(f1_score, average='weighted')
}
```

```python
# Define model and hyperparameter list to make loop easier to read afterwards

classifiers = [("DT", DecisionTreeClassifier(),
                {"max_depth": [3, 5, 10, None]}),

               ("RF", RandomForestClassifier(),
                {"max_depth": [3, 5, 10, None]}),

               ("LOGREG", LogisticRegression(),
                {"C": np.logspace(-5, 5, 5 + 5 + 1, base=10)}),

               ("KNN", KNeighborsClassifier(),
                {"n_neighbors_male": {"n_neighbors": np.append(np.logspace(0, 3, 3 + 0 + 1, base=10).astype("int"),
                                          np.sqrt(len(df_train_Male_X)).astype("int"))},
                 "n_neighbors_female": {"n_neighbors": np.append(np.logspace(0, 3, 3 + 0 + 1, base=10).astype("int"),
                                          np.sqrt(len(df_train_Female_X)).astype("int"))}}),
               #("SVM", LinearSVC(),
               # {"C": np.logspace(-5, 5, 5 + 5 + 1, base=10)})
              ]

```

```python
models_Female, results_Female, models_Male, results_Male = M1.train_models(data_sets, classifiers, scoring)
```

    F
    ===================
    Model: DT
    Fitting 5 folds for each of 4 candidates, totalling 20 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    1.3s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    1.3s finished

    Model: RF
    Fitting 5 folds for each of 4 candidates, totalling 20 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:   41.5s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:   41.5s finished

    Model: LOGREG
    Fitting 5 folds for each of 11 candidates, totalling 55 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  24 tasks      | elapsed:  1.7min
    [Parallel(n_jobs=-1)]: Done  55 out of  55 | elapsed:  4.0min finished
    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\sklearn\linear_model\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    Model: KNN
    Fitting 5 folds for each of 5 candidates, totalling 25 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  25 out of  25 | elapsed:   15.5s finished

    M
    ===================
    Model: DT
    Fitting 5 folds for each of 4 candidates, totalling 20 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    3.1s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:    3.1s finished

    Model: RF
    Fitting 5 folds for each of 4 candidates, totalling 20 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:  1.8min finished
    [Parallel(n_jobs=-1)]: Done  20 out of  20 | elapsed:  1.8min remaining:    0.0s

    Model: LOGREG
    Fitting 5 folds for each of 11 candidates, totalling 55 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  24 tasks      | elapsed:  3.3min
    [Parallel(n_jobs=-1)]: Done  55 out of  55 | elapsed:  7.5min finished
    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\sklearn\linear_model\_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    Model: KNN
    Fitting 5 folds for each of 5 candidates, totalling 25 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  25 out of  25 | elapsed:   40.0s finished

```python
results_Male
```

      params
      accuracy
      weighted_precision
      weighted_recall
      weighted_F1
      Model
      dummy_accuracy
      dummy_weighted_precision
      dummy_weighted_recall
      dummy_weighted_F1

      0
      {'max_depth': 3}
      0.263260
      0.181940
      0.263260
      0.198626
      DT
      0.031035
      0.066609
      0.031035
      0.039431

      1
      {'max_depth': 5}
      0.292669
      0.213466
      0.292669
      0.220856
      DT
      0.031035
      0.066609
      0.031035
      0.039431

      2
      {'max_depth': 10}
      0.329772
      0.325275
      0.329772
      0.304810
      DT
      0.031035
      0.066609
      0.031035
      0.039431

      3
      {'max_depth': None}
      0.266584
      0.340210
      0.266584
      0.281883
      DT
      0.031035
      0.066609
      0.031035
      0.039431

      4
      {'max_depth': 3}
      0.290100
      0.214810
      0.290100
      0.198988
      RF
      0.031035
      0.066609
      0.031035
      0.039431

      5
      {'max_depth': 5}
      0.333677
      0.292048
      0.333677
      0.257399
      RF
      0.031035
      0.066609
      0.031035
      0.039431

      6
      {'max_depth': 10}
      0.383495
      0.405380
      0.383495
      0.343647
      RF
      0.031035
      0.066609
      0.031035
      0.039431

      7
      {'max_depth': None}
      0.346030
      0.424626
      0.346030
      0.353108
      RF
      0.031035
      0.066609
      0.031035
      0.039431

      8
      {'C': 1e-05}
      0.121759
      0.078399
      0.121759
      0.034903
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      9
      {'C': 0.0001}
      0.251662
      0.139222
      0.251662
      0.134024
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      10
      {'C': 0.001}
      0.274263
      0.148193
      0.274263
      0.170356
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      11
      {'C': 0.01}
      0.282726
      0.181215
      0.282726
      0.190921
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      12
      {'C': 0.1}
      0.280941
      0.160681
      0.280941
      0.187746
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      13
      {'C': 1.0}
      0.284076
      0.176380
      0.284076
      0.195517
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      14
      {'C': 10.0}
      0.283074
      0.166150
      0.283074
      0.190986
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      15
      {'C': 100.0}
      0.282015
      0.166119
      0.282015
      0.192997
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      16
      {'C': 1000.0}
      0.282029
      0.164965
      0.282029
      0.191891
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      17
      {'C': 10000.0}
      0.281710
      0.164849
      0.281710
      0.190716
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      18
      {'C': 100000.0}
      0.282058
      0.168326
      0.282058
      0.191111
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      19
      {'n_neighbors': 1}
      0.101292
      0.205611
      0.101292
      0.106831
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

      20
      {'n_neighbors': 10}
      0.104543
      0.177015
      0.104543
      0.101749
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

      21
      {'n_neighbors': 100}
      0.088591
      0.137339
      0.088591
      0.076270
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

      22
      {'n_neighbors': 1000}
      0.083147
      0.050824
      0.083147
      0.043962
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

      23
      {'n_neighbors': 262}
      0.076905
      0.112648
      0.076905
      0.059310
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

##### Evaluate best hyper-parameters

We now have run all the models and computed their metric values using a 5-fold cross validation approach. As we can observe the results are not ideal (most models are not able to classify corrcetly). However we are always able to improve the results of the dummy classification which randomly assigns sports to athlets without looking into aby of their features.

It is important to highlight that an accuracy  of ~30% is already pretty good since we are trying to classify athletes to some sport (we have more than 40 sports to classify between) solely based on a few features.

```python
best_models_Female =  results_Female.loc[results_Female.groupby('Model')['weighted_F1'].idxmax()]
best_models_Male =  results_Male.loc[results_Male.groupby('Model')['weighted_F1'].idxmax()]
```

```python
best_models_Female
```

      params
      accuracy
      weighted_precision
      weighted_recall
      weighted_F1
      Model
      dummy_accuracy
      dummy_weighted_precision
      dummy_weighted_recall
      dummy_weighted_F1

      2
      {'max_depth': 10}
      0.349474
      0.359594
      0.349474
      0.309587
      DT
      0.029419
      0.099966
      0.029419
      0.038228

      19
      {'n_neighbors': 1}
      0.130935
      0.230487
      0.130935
      0.132123
      KNN
      0.029419
      0.099966
      0.029419
      0.038228

      13
      {'C': 1.0}
      0.347646
      0.218627
      0.347646
      0.246400
      LOGREG
      0.029419
      0.099966
      0.029419
      0.038228

      7
      {'max_depth': None}
      0.362952
      0.426847
      0.362952
      0.355706
      RF
      0.029419
      0.099966
      0.029419
      0.038228

```python
best_models_Male
```

      params
      accuracy
      weighted_precision
      weighted_recall
      weighted_F1
      Model
      dummy_accuracy
      dummy_weighted_precision
      dummy_weighted_recall
      dummy_weighted_F1

      2
      {'max_depth': 10}
      0.329772
      0.325275
      0.329772
      0.304810
      DT
      0.031035
      0.066609
      0.031035
      0.039431

      19
      {'n_neighbors': 1}
      0.101292
      0.205611
      0.101292
      0.106831
      KNN
      0.031035
      0.066609
      0.031035
      0.039431

      13
      {'C': 1.0}
      0.284076
      0.176380
      0.284076
      0.195517
      LOGREG
      0.031035
      0.066609
      0.031035
      0.039431

      7
      {'max_depth': None}
      0.346030
      0.424626
      0.346030
      0.353108
      RF
      0.031035
      0.066609
      0.031035
      0.039431

```python

```

##### Let's now see how our champion models predict based on our test data at sport and sex level

```python
models = {'F': models_Female,
          'M': models_Male}
```

```python
evaluation_df = M1.evaluate_models(data_sets, models, le)
```

```python
evaluation_df
```

    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

      precision
      recall
      f1-score
      support
      Sex
      Model

      Archery
      0.000000
      0.000000
      0.000000
      76.0
      F
      DT

      Athletics Endurance
      0.438028
      0.677560
      0.532079
      459.0
      F
      DT

      Athletics Sprints
      0.205797
      0.554688
      0.300211
      512.0
      F
      DT

      Athletics Throws
      0.485000
      0.729323
      0.582583
      133.0
      F
      DT

      Badminton
      0.000000
      0.000000
      0.000000
      59.0
      F
      DT

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Volleyball
      0.105263
      0.166667
      0.129032
      12.0
      M
      KNN

      Water Polo
      0.100000
      0.083333
      0.090909
      12.0
      M
      KNN

      Weightlifting
      0.363128
      0.427632
      0.392749
      152.0
      M
      KNN

      Wrestling
      0.216312
      0.260684
      0.236434
      234.0
      M
      KNN

      overall
      0.255744
      0.240192
      0.243197
      NaN
      M
      KNN

288 rows × 6 columns

From the previous dataframe we can see that depending on the sport and sex, the evaulation metrics can behave better or worse. The champion model is evaluated over the overall performance of the model over all sport types

```python

```

##### Decide which is the champion model

```python
evaluation_df.loc['overall'][['precision', 'recall', 'f1-score', 'Sex', 'Model']]
```

    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)

      precision
      recall
      f1-score
      Sex
      Model

      overall
      0.274093
      0.341434
      0.283723
      F
      DT

      overall
      0.391214
      0.413366
      0.397887
      F
      RF

      overall
      0.144636
      0.236452
      0.148133
      F
      LOGREG

      overall
      0.259370
      0.242041
      0.247230
      F
      KNN

      overall
      0.298768
      0.340769
      0.303445
      M
      DT

      overall
      0.401694
      0.421731
      0.407771
      M
      RF

      overall
      0.146488
      0.270769
      0.184091
      M
      LOGREG

      overall
      0.255744
      0.240192
      0.243197
      M
      KNN

### From the previous  dataframe we can conclude that the best model to predict which sport an athlete participated in is the Random Forest, for both male and female

Apply such models over our FDS team members we would get the following results:

1. Apply random forest to obtain the ideal sport (without setting a fixed max_depth of the branches)
2. Applying a logisitic regression to compute the probability of winning a medal

```python
df_us = pd.DataFrame(index=['Tansel', 'Alessandro', 'Leonardo', 'Juan'],
                  columns=['Year', 'Age', 'Height', 'Weight', 'Country', 'Sex'])

df_us['Year'] = [2016, 2016, 2016, 2016]
df_us['Age'] = [23, 26, 24, 25]
df_us['Height'] = [172, 173, 165, 180]
df_us['Weight'] = [57, 69, 55, 80]
df_us['Country'] = ['TUR', 'ITA', 'ITA', 'ESP']
df_us['Sex'] = ['F', 'M', 'M', 'M']
```

```python
df = M1.predict_ideal_sport_and_prob(df_us, models, le, medals_country_dict, df_clean, df_performance,
                                     hyperparameter_thetas)
df
```

      Year
      Age
      Height
      Weight
      Country
      Sex
      Sport_predict
      P_Medal

      Tansel
      2016
      23
      172
      57
      TUR
      F
      Taekwondo
      0.321129

      Alessandro
      2016
      26
      173
      69
      ITA
      M
      Sailing
      0.111271

      Leonardo
      2016
      24
      165
      55
      ITA
      M
      Wrestling
      0.101105

      Juan
      2016
      25
      180
      80
      ESP
      M
      Athletics Sprints
      0.039841

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# The *grmpy*  package
This notebook demonstrates the current capabilities of the *grmpy* package. *grmpy* is an open source package for the programming language python that enables researchers to simulate datasets andestimate parameters using already existing data within the structure of the generalized Roy model. Currently the package serves as a teaching tool for a course on the econometrics of policy evaluation at the University of Bonn. The corresponding lecture materials can be found on [GitHub](https://github.com/HumanCapitalAnalysis/econometrics).  Morover it is thought of as a promotion for the conceptual framework as well as a showcase for basic software engineering practices.

For a more detailed overview on the economic background as well as the installation routine feel free to take a look on the [online documentation](https://grmpy.readthedocs.io/en/develop/).

The notebook itself is divided in three parts. Firstly we provide a basic outline on how to use the package and introduce the core features. Next we will show that the results obtained by the package's estimation process withstand a critical examination by comparing its performance in the presence of essential heterogeneity with several other estimation approaches like Ordinary Least Squares and Instrumental variables. We conclude by conducting a replication of results from

Carneiro, P., Heckman, J. J., & Vytlacil, E. J. (2011). [Estimating marginal returns to education.](https://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.101.6.2754)
*American Economic Review, 101*(6), 2754-81.

# The Framework

As mentioned before the package makes use of the normal-linear-in-parameters version generalized Roy model. In addition we assume that the unobservable terms $\{U_1, U_0, V\}$ are normally distributed according to the covariance matrix $\Sigma$. The following set of equations characterize the underlying model:

\begin{align*}
 &\textbf{Potential Outcomes} & & \textbf{Choice}  &\\
 & Y_1 = \beta_1 X + U_{1} &  &D_i = \mathbf{1}\{\Phi(\gamma Z) > u_D\} &\\
 & Y_0 = \beta_0 X + U_{0} &  & \text{with $u_D = \Phi(V)$}  &\\
&&&&\\
&\textbf{Distributional Characteristics}&&&\\
&\{U_{1}, U_{0}, V\} \sim \mathcal{N}\left(0, \Sigma\right)&&\Sigma =  \begin{bmatrix}
    \sigma_1^{2} & \sigma_{1,0} & \sigma_{1,V} \\
    \sigma_{1,0} & \sigma_0^{2} & \sigma_{0,V} \\
    \sigma_{1,V} & \sigma_{0,V} & \sigma_V^{2} \\
  \end{bmatrix}&\\
&&&&\\
& \textbf{Observed Outcome} &&&\\
& Y = D Y_1 + (1-D) Y_0 &&&
\end{align*}

# Preliminaries

The first step consists of some style and import improving settings.

```python
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:90% !important; }</style>"))
%load_ext autoreload
%autoreload 2
```

Before we can start with the application examples we have to import several libaries that we rely on during this presentation.

```python
import matplotlib.pyplot as plt
import statsmodels.api as sm
import pandas as pd
import numpy as np
import json

import grmpy
```

In addition we will import some custom functions for plotting several figures of interest as well as some data management purposes. The code is provided in the auxiliary.py file.

```python
from auxiliary import plot_joint_distribution_unobservables
from auxiliary import plot_joint_distribution_potential
from auxiliary import plot_marginal_effects
from auxiliary import process_data
from auxiliary import plot_est_mte
from auxiliary import monte_carlo
from auxiliary import effects
```

# Part I - Introduction

This part provides some basic information about the general application of the package's core methods.

## The initialization file

Currently the package has two core features. The first one is the simulation process. It enables users to simulate data sets along a pre-specified parameterization. The data generating process follows the structure of the previously introduced parametric Roy model framework. Therefore users specify their desired parametrization in an initialization file which allows altering any aspects of the model according to their preferences. The ability to estimate parameters on already existing data sets is the second core feature. As before the initialization file is the starting point for the estimation process.

The initialization file is structured as follows:

* **Simulation**: The simulation section contains information for the estimation process. For instance users are able to change the number of individuals for which the program runs a simulation through setting the agents option to a specific value.

* **Estimation**: Accordingly information regarding the estimation process are stored in the estimation section. Here, users can set the type of optimizer which should be used for running the process as well as the maximum number of iterations that the process is allowed to perform. In addition the flags *dependent* and *indicator* determine which variable of the input data frame is seen as the dependent variable and the treatment indicator variable, respectively.

* **Treated, Untreated, Choice**: These sections are essential for both methods. They contain the parameterization which is used for the simulation process. Additionally the second column indicates the variable labels well as for the simulated data set. This column is also essential for the estimation process because it will only include variables that are pre-specified in a specific section for the following optimization.

* **Dist**: The Dist section determines the distributional characteristics of the unobservable variables. Therefore the section reports the upper triangle of the covariance matrix of $(U_1, U_0,V)$.

* **SCIPY-BFGS/SCIPY-POWELL**: The SCIPY-related sections contain options for the relevant optimization algorithms.

```python
%%file files/tutorial.grmpy.yml
---
SIMULATION:
    agents: 10000
    seed: 2356
    source: data
ESTIMATION:
    file: data.grmpy.txt
    output_file: output/est.grmpy.info
    optimizer: SCIPY-BFGS
    start: auto
    maxiter: 6383
    agents: 165
    dependent: Y
    indicator: D
    comparison: 1
    print_output: 0
TREATED:
    params:
    - 1.0
    - 0.555
    order:
    - const
    - X2
UNTREATED:
    params:
    - 0.5
    - 0.25
    order:
    - const
    - X2
CHOICE:
    params:
    - 0.378
    - -0.39
    order:
    - const
    - X3
DIST:
    params:
    - 0.1
    - 0.0
    - 0.0
    - 0.1
    - 0.0
    - 1.0
VARTYPES:
    const: nonbinary
    X2: nonbinary
    X3: nonbinary
SCIPY-BFGS:
    gtol: 1.0e-05
    eps: 1.4901161193847655e-08
SCIPY-POWELL:
    xtol: 9.147777614048603e-05
    ftol: 9.749582129043358e-05
```

    Overwriting files/tutorial.grmpy.yml

## The Simulation Process

For simulating a dataset we only have to provide the package with the related inititalization file.

```python
data = grmpy.simulate('files/tutorial.grmpy.yml')
data.head(20)
```

      const
      X2
      X3
      U1
      U0
      V
      Y1
      Y0
      D
      Y

      0
      1.0
      -0.014710
      0.023439
      0.002676
      0.061452
      0.767310
      0.994512
      0.557775
      0.0
      0.557775

      1
      1.0
      1.941447
      -1.375891
      0.087087
      0.066451
      -0.567979
      2.164590
      1.051813
      1.0
      2.164590

      2
      1.0
      -0.220668
      1.555453
      0.078557
      0.008140
      -0.307828
      0.956087
      0.452973
      1.0
      0.956087

      3
      1.0
      0.531899
      0.192161
      -0.005574
      0.055052
      -1.129202
      1.289630
      0.688027
      1.0
      1.289630

      4
      1.0
      -0.295161
      1.663835
      -0.111534
      0.077755
      -0.024787
      0.724652
      0.503965
      0.0
      0.503965

      5
      1.0
      2.030613
      0.845409
      -0.096886
      -0.086180
      0.117710
      2.030104
      0.921473
      0.0
      0.921473

      6
      1.0
      -0.189934
      -0.830659
      0.019401
      -0.045266
      -0.037156
      0.913988
      0.407250
      1.0
      0.913988

      7
      1.0
      0.590819
      1.731056
      -0.087072
      0.032617
      0.719759
      1.240833
      0.680322
      0.0
      0.680322

      8
      1.0
      0.790466
      1.133287
      0.049938
      -0.200924
      0.749826
      1.488647
      0.496692
      0.0
      0.496692

      9
      1.0
      -1.378429
      0.020067
      0.151613
      0.015832
      -1.119519
      0.386585
      0.171225
      1.0
      0.386585

      10
      1.0
      -0.089817
      -0.279000
      0.062335
      -0.053459
      0.049270
      1.012487
      0.424087
      1.0
      1.012487

      11
      1.0
      0.212042
      2.217055
      0.009479
      -0.024719
      0.391382
      1.127162
      0.528291
      0.0
      0.528291

      12
      1.0
      0.140421
      -0.949804
      0.050072
      0.002839
      0.658555
      1.128005
      0.537945
      1.0
      1.128005

      13
      1.0
      -0.152488
      0.997728
      0.191663
      -0.072238
      -0.608002
      1.107032
      0.389639
      1.0
      1.107032

      14
      1.0
      -0.008147
      0.812770
      -0.060911
      0.025155
      0.732511
      0.934567
      0.523118
      0.0
      0.523118

      15
      1.0
      0.962076
      -1.210027
      0.260456
      0.008918
      -1.209136
      1.794408
      0.749437
      1.0
      1.794408

      16
      1.0
      -2.212943
      0.613103
      0.002101
      0.084170
      0.483983
      -0.226082
      0.030934
      0.0
      0.030934

      17
      1.0
      -0.243372
      -0.256826
      0.065807
      -0.007584
      0.906569
      0.930736
      0.431573
      0.0
      0.431573

      18
      1.0
      0.882084
      -0.619958
      0.059393
      -0.044658
      -1.719871
      1.548950
      0.675863
      1.0
      1.548950

      19
      1.0
      0.363549
      1.137413
      0.150281
      0.109781
      -0.421398
      1.352051
      0.700668
      1.0
      1.352051

The simulation process generates an info file that provides additional information about the distribution of outcomes and effects, respectively. Additionally it contains the criterion function value, information about the corresponding marginal treatment effects as well as the paramterization.

```python
!cat "data.grmpy.info"
```

     Number of Observations

                                Count

      All                       10000
      Treated                    6391
      Untreated                  3609

     Distribution of Outcomes

                                 Mean             Std-Dev.                  25%                  50%                  75%

      All                      0.8196               0.5328               0.4401               0.7467               1.1652
      Treated                  1.0018               0.5580               0.6305               1.0059               1.3721
      Untreated                0.4968               0.2689               0.3218               0.5001               0.6782

     Distribution of Effects

                                 Mean             Std-Dev.                  25%                  50%                  75%

      All                      0.4998               0.3340               0.2739               0.4974               0.7208
      Treated                  0.5007               0.3354               0.2731               0.5016               0.7233
      Untreated                0.4982               0.3315               0.2761               0.4905               0.7177

     Criterion Function

      Value           -0.270418786930

     Marginal Treatment Effect

        Quantile                Value

              1%               0.5006
              5%               0.5006
             10%               0.5006
             15%               0.5006
             20%               0.5006
             25%               0.5006
             30%               0.5006
             35%               0.5006
             40%               0.5006
             45%               0.5006
             50%               0.5006
             55%               0.5006
             60%               0.5006
             65%               0.5006
             70%               0.5006
             75%               0.5006
             80%               0.5006
             85%               0.5006
             90%               0.5006
             95%               0.5006
             99%               0.5006

     Parameterization

     Section       Identifier           Coef

      TREATED
                   const                1.0000
                   X2                   0.5550

      UNTREATED
                   const                0.5000
                   X2                   0.2500

      CHOICE
                   const                0.3780
                   X3                  -0.3900

      DIST
                   sigma1               0.1000
                   sigma10              0.0000
                   sigma1v              0.0000
                   sigma0               0.1000
                   sigma0v              0.0000
                   sigmaV               1.0000

The generated data set is specififed so that the treatment selection process is not affected by essential heterogeneity. Therefore the conventional effects are nearly identical.

```python
effects(data)
```

### Essential Heterogeneity
For providing an example on how essential heterogeneity biases the results obtained by naive comparison between treated and untreated individuals, we alter the initialization file. Specifically we will introduce correlation between the unobservable terms. This leads to the situation where individuals select into treatment based on unobservable gains. Therefore the treatment decision $D$ is no longer independent from the outcomes $Y_1$ and $Y_0$.

\begin{align}
Y_1,Y_0\;\; {\perp\!\!\!\!\!\!\diagup\!\!\!\!\!\!\!\perp} \;\; D
\end{align}

Unlike in the absence of essential heterogeneity individuals who select themselves into treatment differ from
individuals who do not with respect to their unobservable characteristics. From this follows that

\begin{align}
B^{ATE} \neq B^{TT} \neq B^{TUT}
\end{align}

```python
%%file files/tutorial_eh.grmpy.yml
---
SIMULATION:
    seed: 2356
    agents: 10000
    source: data_eh
ESTIMATION:
    file: data_eh.grmpy.txt
    start: auto
    agents: 165
    optimizer: SCIPY-BFGS
    maxiter: 6383
    dependent: Y
    indicator: D
    output_file: output/est.grmpy.info
    comparison: 1
    print_output: 0

TREATED:
    params:
    - 1.0
    - 0.555
    order:
    - const
    - X2
UNTREATED:
    params:
    - 0.5
    - 0.25
    order:
    - const
    - X2
CHOICE:
    params:
    - 0.378
    - -0.39
    order:
    - const
    - X3
DIST:
    params:
    - 0.1
    - 0.0
    - 0.0524
    - 0.1
    - -0.0216
    - 1.0
VARTYPES:
    const: nonbinary
    X2: nonbinary
    X3: nonbinary
SCIPY-BFGS:
    gtol: 1.0e-05
    eps: 1.4901161193847655e-08
SCIPY-POWELL:
    xtol: 9.147777614048603e-05
    ftol: 9.749582129043358e-05
```

    Overwriting files/tutorial_eh.grmpy.yml

```python
data_eh = grmpy.simulate('files/tutorial_eh.grmpy.yml')
data_eh.head(20)
```

      const
      X2
      X3
      U1
      U0
      V
      Y1
      Y0
      D
      Y

      0
      1.0
      -0.014710
      0.023439
      -0.015029
      0.072663
      -0.767427
      0.976807
      0.568986
      1.0
      0.976807

      1
      1.0
      1.941447
      -1.375891
      0.121409
      0.021829
      0.563892
      2.198912
      1.007191
      1.0
      2.198912

      2
      1.0
      -0.220668
      1.555453
      0.078986
      -0.023754
      0.304143
      0.956515
      0.421079
      0.0
      0.421079

      3
      1.0
      0.531899
      0.192161
      0.076321
      0.028085
      1.129450
      1.371525
      0.661060
      0.0
      0.661060

      4
      1.0
      -0.295161
      1.663835
      -0.053739
      0.106255
      0.030013
      0.782446
      0.532464
      0.0
      0.532464

      5
      1.0
      2.030613
      0.845409
      -0.112613
      -0.046794
      -0.113168
      2.014377
      0.960859
      1.0
      2.014377

      6
      1.0
      -0.189934
      -0.830659
      -0.000561
      -0.048730
      0.036247
      0.894026
      0.403786
      1.0
      0.894026

      7
      1.0
      0.590819
      1.731056
      -0.091649
      0.073059
      -0.715671
      1.236255
      0.720764
      1.0
      1.236255

      8
      1.0
      0.790466
      1.133287
      -0.078217
      -0.185083
      -0.752158
      1.360492
      0.512533
      1.0
      1.360492

      9
      1.0
      -1.378429
      0.020067
      0.180205
      -0.057158
      1.112402
      0.415177
      0.098234
      0.0
      0.098234

      10
      1.0
      -0.089817
      -0.279000
      0.024354
      -0.067862
      -0.052191
      0.974506
      0.409684
      1.0
      0.974506

      11
      1.0
      0.212042
      2.217055
      -0.022872
      -0.017309
      -0.391822
      1.094811
      0.535702
      0.0
      0.535702

      12
      1.0
      0.140421
      -0.949804
      0.004355
      0.001276
      -0.660895
      1.082289
      0.536381
      1.0
      1.082289

      13
      1.0
      -0.152488
      0.997728
      0.150062
      -0.139993
      0.599014
      1.065431
      0.321885
      0.0
      0.321885

      14
      1.0
      -0.008147
      0.812770
      -0.075304
      0.058249
      -0.729648
      0.920175
      0.556212
      1.0
      0.920175

      15
      1.0
      0.962076
      -1.210027
      0.264935
      -0.099564
      1.196918
      1.798888
      0.640955
      0.0
      0.640955

      16
      1.0
      -2.212943
      0.613103
      0.008139
      0.087685
      -0.484076
      -0.220044
      0.034450
      1.0
      -0.220044

      17
      1.0
      -0.243372
      -0.256826
      -0.000754
      -0.007892
      -0.909643
      0.864174
      0.431265
      1.0
      0.864174

      18
      1.0
      0.882084
      -0.619958
      0.118808
      -0.097278
      1.717069
      1.608364
      0.623243
      0.0
      0.623243

      19
      1.0
      0.363549
      1.137413
      0.178168
      0.045299
      0.414351
      1.379938
      0.636186
      0.0
      0.636186

```python
!cat "data_eh.grmpy.info"
```

     Number of Observations

                                Count

      All                       10000
      Treated                    6384
      Untreated                  3616

     Distribution of Outcomes

                                 Mean             Std-Dev.                  25%                  50%                  75%

      All                      0.7945               0.5279               0.4237               0.7200               1.1301
      Treated                  0.9721               0.5557               0.6012               0.9698               1.3457
      Untreated                0.4809               0.2674               0.3072               0.4807               0.6615

     Distribution of Effects

                                 Mean             Std-Dev.                  25%                  50%                  75%

      All                      0.4987               0.3328               0.2705               0.5020               0.7210
      Treated                  0.4607               0.3279               0.2391               0.4578               0.6831
      Untreated                0.5658               0.3308               0.3415               0.5696               0.7857

     Criterion Function

      Value           -0.322419326705

     Marginal Treatment Effect

        Quantile                Value

              1%               0.3285
              5%               0.3789
             10%               0.4058
             15%               0.4239
             20%               0.4383
             25%               0.4507
             30%               0.4618
             35%               0.4721
             40%               0.4819
             45%               0.4913
             50%               0.5006
             55%               0.5099
             60%               0.5194
             65%               0.5291
             70%               0.5394
             75%               0.5505
             80%               0.5629
             85%               0.5773
             90%               0.5955
             95%               0.6223
             99%               0.6728

     Parameterization

     Section       Identifier           Coef

      TREATED
                   const                1.0000
                   X2                   0.5550

      UNTREATED
                   const                0.5000
                   X2                   0.2500

      CHOICE
                   const                0.3780
                   X3                  -0.3900

      DIST
                   sigma1               0.1000
                   sigma10              0.0000
                   sigma1v              0.0524
                   sigma0               0.1000
                   sigma0v             -0.0216
                   sigmaV               1.0000

```python
effects(data_eh)
```

Next we illustrate how essential heterogeneity is refelected in the joint distribution of the error term. For that purpose we created two figures that show the relationship between $U_1$ and $V$ for both data sets which we simulated before.

```python
plot_joint_distribution_unobservables(data, data_eh)
```

Simulating data sets enables us to explore more facets of the impact of essential heterogeneity on the marginal effect of treatment. The marginal effect of treatment, henceforth denoted as $B^{MTE}$ is defined as the effect of treatment for individuals that are indifferent between taking the treatment or not along the distribution of the unobservable variable $V$. This means that instead of assigning explicitly one value as the effect of treatment the $B^{MTE}$ provides a continuum of effects along the distribution of the unobservable $V$. More formally:

\begin{align*}
B^{MTE} = E[Y_1-Y_0|X=x, U_D = u_D] \\
\end{align*}

In the abscence of essential heterogeneity $B^{MTE}$ provides a constant value over the whole distribution of $V$. This implies that the individual's decision for treatment participation is independent of their unobservable benefits so that each individual on the margin faces the same average benefit of treatment. This can be seen in the figure below. The flat line captures the $B^{MTE}$ for the simulated data without essential heterogeneity, wheras the increasing cure illustrates the marginal effect for the one that is affected by essential heterogeneity.

```python
plot_marginal_effects('data.grmpy.info', 'data_eh.grmpy.info')
```

Using simulated data allows us to tackle issues and explore additional objects of interest for which we are not able to obtain reliable information if we use empirical data sets. For instance we are able to construct the joint distribution of potential outcomes from the previously simulated data sets because our simulated data allows us to evade the evaluation problem. This means that we have information about the whole space of potential outcomes for each individual.

```python
plot_joint_distribution_potential(data_eh)
```

## The Estimation Process

grmpy enables users to estimate parameters on data sets. Executing an estimation is simple. Just setup your estimation specifications in the initialization file and provide the estimation process with the resulting initialization file.

```python
rslt = grmpy.fit('files/tutorial_eh.grmpy.yml')
```

The estimation process returns a detailed overview of the results via an output file

```python
!cat output/est.grmpy.info
```

      Optimization Information

                 Optimizer:                    SCIPY-BFGS
                 Start Values:                 auto
                 Success:                      1
                 Status:                       0
                 Message:                      Optimization terminated successfully.
                 Number of Evaluations:        276
                 Criterion:                    -0.3230
                 Observations:                 10000
                 Warning:                      The optimization algorithm has failed to provide the parametrization that leads to the minimal criterion function value.
                                               The estimation output is automatically adjusted and provides the parameterization with the smallest criterion function value
                                               that was reached during the optimization.

      Criterion Function

                                            Start               Finish

                              -0.3187406802544943  -0.3229750696988651

      Economic Parameters

                                        Start          Finish

     Section       Identifier           Coef           Coef           Std err        t              P>|t|          95% Conf. Int.

      TREATED
                   const                0.9708         0.9981         0.0031       320.4145         0.0000         0.9920    1.0042
                   X2                   0.5533         0.5533         0.0012       470.7484         0.0000         0.5510    0.5556

      UNTREATED
                   const                0.4805         0.4953         0.0064        77.2435         0.0000         0.4827    0.5078
                   X2                   0.2477         0.2477         0.0017       148.9133         0.0000         0.2445    0.2510

      CHOICE
                   const                0.3795         0.3796         0.0133        28.5424         0.0000         0.3535    0.4056
                   X3                  -0.4175        -0.4145         0.0141       -29.4912         0.0000        -0.4420   -0.3870

      DIST
                   sigma1               0.0936         0.0998         0.0016        63.0250         0.0000         0.0967    0.1029
                   rho1                 0.0000         0.5055         0.0457        11.0522         0.0000         0.4159    0.5952
                   sigma0               0.1002         0.1009         0.0013        75.1643         0.0000         0.0983    0.1036
                   rho0                 0.0000        -0.1528         0.0631        -2.4197         0.0156        -0.2765   -0.0290

In addition the process is capable of simulating a new data set according to the estimation results. This enables users to verify their obtained results easily.

```python
!cat comparison.grmpy.txt
```

     Number of Observations

        Sample                                Observed   Simulated (finish)    Simulated (start)

        All                                      10000                10000                10000
        Treated                                   6384                 6374                 6398
        Untreated                                 3616                 3626                 3602

     Distribution of Outcomes

       All

                                                  Mean             Std-Dev.                  25%                  50%                  75%

        Observed Sample                         0.7945               0.5279               0.4237               0.7200               1.1301
        Simulated Sample (finish)               0.7941               0.5261               0.4249               0.7196               1.1309
        Simulated Sample (start)                0.7944               0.5281               0.4204               0.7235               1.1358

       Treated

                                                  Mean             Std-Dev.                  25%                  50%                  75%

        Observed Sample                         0.9721               0.5557               0.6012               0.9698               1.3457
        Simulated Sample (finish)               0.9721               0.5538               0.5985               0.9700               1.3458
        Simulated Sample (start)                0.9726               0.5546               0.6072               0.9752               1.3414

       Untreated

                                                  Mean             Std-Dev.                  25%                  50%                  75%

        Observed Sample                         0.4809               0.2674               0.3072               0.4807               0.6615
        Simulated Sample (finish)               0.4812               0.2658               0.3086               0.4804               0.6620
        Simulated Sample (start)                0.4778               0.2673               0.3040               0.4816               0.6576

     MTE Information

        Quantile                Value

              1%               0.3503
              5%               0.3951
             10%               0.4191
             15%               0.4352
             20%               0.4480
             25%               0.4590
             30%               0.4689
             35%               0.4781
             40%               0.4868
             45%               0.4952
             50%               0.5035
             55%               0.5117
             60%               0.5201
             65%               0.5288
             70%               0.5380
             75%               0.5479
             80%               0.5589
             85%               0.5717
             90%               0.5878
             95%               0.6118
             99%               0.6566

## Software Engineering Practices

# Part II - Monte Carlo Simulation

For illustrating the advantages of grmpy's estimation process in the presence of essential heterogeneity we conduct a Monte Carlo exercise. As before the starting point of the exericise is an initialization file over which we iterate several times during the process. The distributional characteristics are such that the unobservable variables are distributed according to the following covariance matrix

\begin{align}
\Sigma =  \begin{bmatrix}
    0.01 & 0 & \frac{\rho_{1,V}}{0.1}  \\
    0 & 0.01 & 0 \\
    \frac{\rho_{1,V}}{0.1}  & 0 & 1 \\
  \end{bmatrix}
\end{align}

During each step of the iteration we increase the correlation between $U_1$ and $V$. We will start from a value of $\rho_1 =0.0$ and end at $\rho_1 = 0.99$. This increase is equivalent to the observation of incremental reverse selection behavior, because individuals with a low value of $V$ which are most likely to select into treatment have on average a lower value of $U_1$ than individuals that have larger values of $V$. in addition we estimate the average effect of treatment during each step. For this purpose we use the grmpy estimation process, an ordinary least squares regression and an instrumental variables approach as well as a naive comparison of outputs between treated and untreated individuals.

### The Initialization file

```python
%%file files/mc.grmpy.yml
---
SIMULATION:
    seed: 5133
    agents: 10000
    source: mc
ESTIMATION:
    file: mc.grmpy.txt
    start: auto
    agents: 165
    optimizer: SCIPY-BFGS
    maxiter: 6383
    dependent: wage
    indicator: state
    output_file: mc_rslt.grmpy.info
    comparison: 0
    print_output: 0

TREATED:
    params:
    - 0.99
    - 0.555
    - -0.555
    - 0.755
    - 0.155
    order:
    - const
    - X2
    - X3
    - X4
    - X5
UNTREATED:
    params:
    - 0.5
    - 0.255
    - -0.255
    - 0.1768
    - 0.0987

    order:
    - const
    - X2
    - X3
    - X4
    - X5
CHOICE:
    params:
    - 0.28
    - -0.39
    - 0.59
    - -0.89
    - -0.73

    order:
    - const
    - X6
    - X7
    - X8
    - X9

DIST:
    params:
    - 0.2
    - 0.0
    - 0.0
    - 0.2
    - 0.0
    - 1.0
VARTYPES:
    const: nonbinary
    X2: nonbinary
    X3: nonbinary
    X4: nonbinary
    X5: nonbinary
    X6: nonbinary
    X7: nonbinary
    X8: nonbinary
    X9: nonbinary
SCIPY-BFGS:
    gtol: 1.0e-05
    eps: 1.4901161193847655e-08
SCIPY-POWELL:
    xtol: 9.147777614048603e-05
    ftol: 9.749582129043358e-05
```

    Overwriting files/mc.grmpy.yml

```python
grmpy.simulate('files/mc.grmpy.yml')
monte_carlo('files/mc.grmpy.yml', 10)
```

As can be seen from the figure, the OLS estimator and the naive comparison of outcomes between the treated and untreated subpopulation underestimate the effect significantly. The stronger the correlation between the unobservable variables the more or less stronger bias. Moreover the IV estimates become upward biased as soon the impact of essential heterogeneity increases. Conversely to the other estimation approaches the grmpy estimate of the average effect is close to the true value even if the unobservables are almost perfectly correlated.

# Part III - Replication Carneiro & Heckman & Vytlacil 2011

Since the current version of grmpy is not capable of estimating non-parametric versions of the Roy models, our
replication of Carneiro et al. (2011) will focus on reproducing the results for the marginal treatment effect of the parametric selection model. Due to reasons of privacy regarding local variables, we are not able to merge the data provided by the authors so that they fully coincide with the original data set. Therefore our replication setup makes use of a mock data set. For this purpose we randomly merge the individual specific data with the local characteristics.

```python
basic = pd.read_stata('data/basicvariables.dta')
local = pd.read_stata('data/localvariables.dta')
df = pd.concat([basic, local], axis = 1)
process_data(df,'data/aer-replication-mock')
```

In the next step we have to create a inititalization file that fully coincides with the setup by Carneiro et. al. (2011). Therefore we use the information that the authors provide in their appendix to create the following init file:

```python
%%file files/replication.grmpy.yml
---
SIMULATION:
    seed: 5062
    agents: 991
    source: 8EF73AA0
ESTIMATION:
    file: data/aer-replication-mock.pkl
    start: auto
    agents: 1000
    optimizer: SCIPY-BFGS
    maxiter: 80000
    dependent: wage
    indicator: state
    output_file: replication.grmp.info
    comparison: 0
    print_output: 0

TREATED:
    params:
    - 1.0
    order:
    - const
    - exp
    - expsq
    - lwage5
    - lurate
    - cafqt
    - cafqtsq
    - mhgc
    - mhgcsq
    - numsibs
    - numsibssq
    - urban14
    - lavlocwage17
    - lavlocwage17sq
    - avurate
    - avuratesq
    - d57
    - d58
    - d59
    - d60
    - d61
    - d62
    - d63
UNTREATED:
    params:
    - 1.0
    order:
    - const
    - exp
    - expsq
    - lwage5
    - lurate
    - cafqt
    - cafqtsq
    - mhgc
    - mhgcsq
    - numsibs
    - numsibssq
    - urban14
    - lavlocwage17
    - lavlocwage17sq
    - avurate
    - avuratesq
    - d57
    - d58
    - d59
    - d60
    - d61
    - d62
    - d63
CHOICE:
    params:
    - 1.0
    order:
    - const
    - cafqt
    - cafqtsq
    - mhgc
    - mhgcsq
    - numsibs
    - numsibssq
    - urban14
    - lavlocwage17
    - lavlocwage17sq
    - avurate
    - avuratesq
    - d57
    - d58
    - d59
    - d60
    - d61
    - d62
    - d63
    - lwage5_17numsibs
    - lwage5_17mhgc
    - lwage5_17cafqt
    - lwage5_17
    - lurate_17
    - lurate_17numsibs
    - lurate_17mhgc
    - lurate_17cafqt
    - tuit4c
    - tuit4cnumsibs
    - tuit4cmhgc
    - tuit4ccafqt
    - pub4
    - pub4numsibs
    - pub4mhgc
    - pub4cafqt
DIST:
    params:
    - 0.1
    - 0.0
    - 0.0
    - 0.1
    - 0.0
    - 1.0
VARTYPES:
    const: nonbinary
SCIPY-BFGS:
    gtol: 1.0e-05
    eps: 1.4901161193847656e-08
SCIPY-POWELL:
    xtol: 0.0001
    ftol: 0.0001
```

    Overwriting files/replication.grmpy.yml

We then conduct an estimation based on the initialization file.

```python
rslt = grmpy.fit('files/replication.grmpy.yml')
```

Next we plot $B^{MTE}$ based on our estimation results. As shown in the figure below the results are really close to the original results. The deviation seems to be negligible because we use a mock dataset.

```python
mte = plot_est_mte(rslt, 'files/replication.grmpy.yml')
```

For a detailed overview on the theoretical economic background, more application examples as well as contact informations see the [online documentation](https://grmpy.readthedocs.io/en/latest/index.html). In addition, the most current code is available on [GitHub](https://github.com/OpenSourceEconomics/grmpy).