```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from sympy import *
import seaborn as sns
```

#### Get Data & Inspect Columns (Gapminder Dataset)

```python
life_expectancy= pd.read_excel(r'C:\Users\denni\Desktop\data_vault_2021\visual_data_analysis\gapminder_lifeexpectancy.xlsx', index_col=0)
population = pd.read_excel(r'C:\Users\denni\Desktop\data_vault_2021\visual_data_analysis\gapminder_population.xlsx', index_col=0)
fertility = pd.read_csv(r'C:\Users\denni\Desktop\data_vault_2021\visual_data_analysis\gapminder_total_fertility.csv', index_col=0)
```

```python
life_expectancy.index.rename('life_expectancy', inplace=True)
life_expectancy
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015
      2016

      life_expectancy

      Abkhazia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Afghanistan
      28.21
      28.20
      28.19
      28.18
      28.17
      28.16
      28.15
      28.14
      28.13
      28.12
      ...
      52.4
      52.8
      53.3
      53.6
      54.0
      54.4
      54.8
      54.9
      53.8
      52.72

      Akrotiri and Dhekelia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Albania
      35.40
      35.40
      35.40
      35.40
      35.40
      35.40
      35.40
      35.40
      35.40
      35.40
      ...
      76.6
      76.8
      77.0
      77.2
      77.4
      77.5
      77.7
      77.9
      78.0
      78.10

      Algeria
      28.82
      28.82
      28.82
      28.82
      28.82
      28.82
      28.82
      28.82
      28.82
      28.82
      ...
      75.3
      75.5
      75.7
      76.0
      76.1
      76.2
      76.3
      76.3
      76.4
      76.50

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Yugoslavia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Zambia
      32.60
      32.60
      32.60
      32.60
      32.60
      32.60
      32.60
      32.60
      32.60
      32.60
      ...
      49.0
      51.1
      52.3
      53.1
      53.7
      54.7
      55.6
      56.3
      56.7
      57.10

      Zimbabwe
      33.70
      33.70
      33.70
      33.70
      33.70
      33.70
      33.70
      33.70
      33.70
      33.70
      ...
      46.4
      47.3
      48.0
      49.1
      51.6
      54.2
      55.7
      57.0
      59.3
      61.69

      Åland
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      South Sudan
      26.67
      26.67
      26.67
      26.67
      26.67
      26.67
      26.67
      26.67
      26.67
      26.67
      ...
      55.5
      55.6
      55.8
      56.0
      55.9
      56.0
      56.0
      56.1
      56.1
      56.10

260 rows × 217 columns

```python
population.index.rename('total_population', inplace=True)
population
```

      1800
      1810
      1820
      1830
      1840
      1850
      1860
      1870
      1880
      1890
      ...
      2006
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015

      total_population

      Abkhazia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Afghanistan
      3280000.0
      3280000.0
      3323519.0
      3448982.0
      3625022.0
      3810047.0
      3973968.0
      4169690.0
      4419695.0
      4710171.0
      ...
      25183615.0
      25877544.0
      26528741.0
      27207291.0
      27962207.0
      28809167.0
      29726803.0
      30682500.0
      31627506.0
      32526562.0

      Akrotiri and Dhekelia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      15700.0
      15700.0
      15700.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Albania
      410445.0
      423591.0
      438671.0
      457234.0
      478227.0
      506889.0
      552800.0
      610036.0
      672544.0
      741688.0
      ...
      3050741.0
      3010849.0
      2968026.0
      2929886.0
      2901883.0
      2886010.0
      2880667.0
      2883281.0
      2889676.0
      2896679.0

      Algeria
      2503218.0
      2595056.0
      2713079.0
      2880355.0
      3082721.0
      3299305.0
      3536468.0
      3811028.0
      4143163.0
      4525691.0
      ...
      33749328.0
      34261971.0
      34811059.0
      35401790.0
      36036159.0
      36717132.0
      37439427.0
      38186135.0
      38934334.0
      39666519.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Northern Marianas
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      South Georgia and the South Sandwich Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      US Minor Outlying Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Virgin Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      West Bank
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

275 rows × 81 columns

```python
fertility.index.rename('total_fertility_rate', inplace=True)
fertility
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2006
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015

      total_fertility_rate

      Abkhazia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Afghanistan
      7.00
      7.00
      7.00
      7.00
      7.00
      7.00
      7.00
      7.00
      7.00
      7.00
      ...
      6.70
      6.46
      6.20
      5.93
      5.66
      5.40
      5.14
      4.90
      4.68
      4.47

      Akrotiri and Dhekelia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Albania
      4.60
      4.60
      4.60
      4.60
      4.60
      4.60
      4.60
      4.60
      4.60
      4.60
      ...
      1.85
      1.80
      1.76
      1.74
      1.74
      1.75
      1.76
      1.77
      1.78
      1.78

      Algeria
      6.99
      6.99
      6.99
      6.99
      6.99
      6.99
      6.99
      6.99
      6.99
      6.99
      ...
      2.58
      2.66
      2.73
      2.78
      2.82
      2.83
      2.82
      2.80
      2.76
      2.71

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Yugoslavia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Zambia
      6.71
      6.71
      6.71
      6.71
      6.71
      6.71
      6.71
      6.71
      6.71
      6.71
      ...
      5.93
      5.91
      5.88
      5.85
      5.81
      5.77
      5.73
      5.69
      5.64
      5.59

      Zimbabwe
      6.75
      6.75
      6.75
      6.75
      6.75
      6.75
      6.75
      6.75
      6.75
      6.75
      ...
      3.94
      3.90
      3.85
      3.79
      3.72
      3.64
      3.56
      3.49
      3.41
      3.35

      Åland
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Åland
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

260 rows × 216 columns

```python
print(life_expectancy.isnull().sum())
print(population.isnull().sum())
print(fertility.isnull().sum())
```

    1800    59
    1801    59
    1802    59
    1803    59
    1804    59
            ..
    2012    52
    2013    52
    2014    52
    2015    52
    2016    52
    Length: 217, dtype: int64
    1800    46
    1810    46
    1820    46
    1830    46
    1840    46
            ..
    2011    42
    2012    42
    2013    42
    2014    44
    2015    44
    Length: 81, dtype: int64
    1800    59
    1801    59
    1802    59
    1803    59
    1804    59
            ..
    2011    59
    2012    59
    2013    59
    2014    61
    2015    61
    Length: 216, dtype: int64

```python
life_without_na=life_expectancy.dropna()
population_without_na=population.dropna()
fertility_without_na=fertility.dropna()
```

```python
life_without_na.isna().sum()
```

    1800    0
    1801    0
    1802    0
    1803    0
    1804    0
           ..
    2012    0
    2013    0
    2014    0
    2015    0
    2016    0
    Length: 217, dtype: int64

```python
print(life_without_na.shape)
print(population_without_na.shape)
print(fertility_without_na.shape)
```

    (201, 217)
    (229, 81)
    (199, 216)

#### Inspect variables (descriptive statistics & exploratory plots)

```python
life_expectancy.describe()
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015
      2016

      count
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      ...
      208.000000
      208.000000
      208.000000
      208.000000
      208.000000
      208.000000
      208.000000
      208.000000
      208.00000
      208.000000

      mean
      31.486020
      31.448905
      31.463483
      31.377413
      31.446318
      31.562537
      31.615970
      31.573134
      31.376766
      31.310448
      ...
      70.139712
      70.447163
      70.767740
      70.969904
      71.324375
      71.663077
      71.916106
      72.088125
      72.32101
      72.556635

      std
      3.763585
      3.755739
      3.878204
      3.901966
      3.877156
      3.947957
      4.039261
      3.917339
      4.017228
      3.972970
      ...
      8.953255
      8.800452
      8.610341
      8.898859
      8.376540
      8.217466
      8.120852
      8.074990
      7.90202
      7.738535

      min
      23.390000
      23.390000
      23.390000
      19.600000
      23.390000
      23.390000
      23.390000
      23.390000
      12.480000
      13.430000
      ...
      43.300000
      44.500000
      45.500000
      32.200000
      46.700000
      46.100000
      45.600000
      45.400000
      47.10000
      48.860000

      25%
      29.000000
      28.950000
      28.900000
      28.900000
      28.950000
      29.000000
      29.000000
      29.000000
      28.950000
      28.820000
      ...
      64.825000
      64.875000
      65.225000
      65.475000
      65.600000
      66.075000
      66.475000
      66.775000
      67.05000
      67.175000

      50%
      31.800000
      31.700000
      31.600000
      31.500000
      31.600000
      31.700000
      31.800000
      31.800000
      31.600000
      31.500000
      ...
      72.750000
      73.000000
      73.350000
      73.700000
      73.750000
      74.050000
      74.150000
      74.300000
      74.40000
      74.500000

      75%
      33.900000
      33.900000
      33.900000
      33.800000
      33.870000
      33.900000
      34.000000
      34.000000
      33.870000
      33.800000
      ...
      76.925000
      77.150000
      77.425000
      77.650000
      77.825000
      78.125000
      78.300000
      78.400000
      78.50000
      78.650000

      max
      42.850000
      40.300000
      44.370000
      44.840000
      42.830000
      44.270000
      45.820000
      43.560000
      43.550000
      41.740000
      ...
      84.500000
      84.600000
      84.600000
      84.700000
      84.700000
      84.700000
      84.800000
      84.800000
      84.80000
      84.800000

8 rows × 217 columns

```python
population.describe()
```

      1800
      1810
      1820
      1830
      1840
      1850
      1860
      1870
      1880
      1890
      ...
      2006
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015

      count
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      2.290000e+02
      ...
      2.560000e+02
      2.560000e+02
      2.560000e+02
      2.380000e+02
      2.380000e+02
      2.330000e+02
      2.330000e+02
      2.330000e+02
      2.310000e+02
      2.310000e+02

      mean
      4.137832e+06
      4.330692e+06
      4.561516e+06
      4.822455e+06
      5.066513e+06
      5.250450e+06
      5.398263e+06
      5.609144e+06
      5.945787e+06
      6.376547e+06
      ...
      2.813600e+07
      2.846652e+07
      2.880011e+07
      2.876698e+07
      2.911667e+07
      3.010052e+07
      3.046134e+07
      3.082280e+07
      3.135238e+07
      3.171457e+07

      std
      2.422799e+07
      2.601555e+07
      2.789334e+07
      2.941603e+07
      3.018315e+07
      2.995424e+07
      2.898546e+07
      2.846866e+07
      2.904625e+07
      3.022539e+07
      ...
      1.149097e+08
      1.160131e+08
      1.171146e+08
      1.212191e+08
      1.223602e+08
      1.247404e+08
      1.258755e+08
      1.270003e+08
      1.286516e+08
      1.297556e+08

      min
      1.000000e+02
      1.000000e+02
      1.000000e+02
      1.000000e+02
      1.000000e+02
      1.090000e+02
      1.390000e+02
      1.910000e+02
      2.590000e+02
      3.520000e+02
      ...
      5.000000e+01
      5.000000e+01
      5.000000e+01
      5.000000e+01
      5.000000e+01
      7.990000e+02
      7.990000e+02
      7.990000e+02
      7.990000e+02
      8.000000e+02

      25%
      4.390800e+04
      4.390800e+04
      4.467300e+04
      4.689400e+04
      4.764300e+04
      5.340500e+04
      6.446900e+04
      6.542200e+04
      6.817000e+04
      6.922200e+04
      ...
      2.868232e+05
      2.928408e+05
      2.989525e+05
      3.174302e+05
      3.243505e+05
      3.994430e+05
      4.055120e+05
      4.114990e+05
      4.175585e+05
      4.209290e+05

      50%
      4.010000e+05
      4.133260e+05
      4.235270e+05
      4.535070e+05
      4.782270e+05
      5.443020e+05
      5.959280e+05
      6.599300e+05
      7.139570e+05
      7.432100e+05
      ...
      4.521912e+06
      4.561442e+06
      4.636136e+06
      4.523432e+06
      4.581304e+06
      4.953945e+06
      5.018367e+06
      5.240088e+06
      5.307188e+06
      5.373502e+06

      75%
      1.894389e+06
      1.904962e+06
      1.983383e+06
      2.101048e+06
      2.186704e+06
      2.236808e+06
      2.399360e+06
      2.596105e+06
      2.822978e+06
      3.088808e+06
      ...
      1.711520e+07
      1.764428e+07
      1.819443e+07
      1.676626e+07
      1.691918e+07
      2.011166e+07
      1.994495e+07
      1.932259e+07
      1.894310e+07
      1.900687e+07

      max
      3.216750e+08
      3.505430e+08
      3.800553e+08
      4.023735e+08
      4.112134e+08
      4.027113e+08
      3.800475e+08
      3.636612e+08
      3.655442e+08
      3.771353e+08
      ...
      1.312601e+09
      1.319625e+09
      1.326691e+09
      1.333807e+09
      1.340969e+09
      1.348174e+09
      1.355387e+09
      1.362514e+09
      1.369436e+09
      1.376049e+09

8 rows × 81 columns

```python
fertility.describe()
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2006
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015

      count
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      ...
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      201.000000
      199.000000
      199.000000

      mean
      6.084080
      6.081244
      6.084328
      6.083383
      6.083682
      6.083433
      6.079353
      6.078159
      6.070647
      6.059701
      ...
      2.999403
      2.971642
      2.943881
      2.915771
      2.884925
      2.853433
      2.823284
      2.794179
      2.775176
      2.744523

      std
      0.777465
      0.780857
      0.775431
      0.774806
      0.774999
      0.771939
      0.776931
      0.777962
      0.791615
      0.817953
      ...
      1.591173
      1.561309
      1.531775
      1.502406
      1.475264
      1.447739
      1.418326
      1.389848
      1.362167
      1.332827

      min
      4.040000
      4.040000
      3.910000
      4.050000
      3.940000
      4.060000
      4.070000
      4.050000
      4.000000
      3.210000
      ...
      0.870000
      0.910000
      0.940000
      0.970000
      0.900000
      1.030000
      1.060000
      1.070000
      1.110000
      1.130000

      25%
      5.620000
      5.620000
      5.620000
      5.620000
      5.640000
      5.620000
      5.620000
      5.620000
      5.620000
      5.620000
      ...
      1.820000
      1.830000
      1.830000
      1.840000
      1.840000
      1.820000
      1.800000
      1.800000
      1.795000
      1.785000

      50%
      6.150000
      6.150000
      6.150000
      6.150000
      6.150000
      6.150000
      6.150000
      6.150000
      6.130000
      6.130000
      ...
      2.450000
      2.440000
      2.430000
      2.410000
      2.380000
      2.350000
      2.330000
      2.290000
      2.260000
      2.230000

      75%
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      6.690000
      ...
      3.910000
      3.830000
      3.760000
      3.740000
      3.720000
      3.640000
      3.560000
      3.490000
      3.565000
      3.495000

      max
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      8.100000
      ...
      7.600000
      7.590000
      7.590000
      7.590000
      7.580000
      7.580000
      7.570000
      7.560000
      7.540000
      7.510000

8 rows × 216 columns

```python
life_exp_tr=life_without_na.transpose()
sns.heatmap(life_exp_tr)
life_exp_tr
```

```python
life_expectancy.loc[life_expectancy[1800]==life_expectancy[1800].min(),:]
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015
      2016

      life_expectancy

      Yemen
      23.39
      23.39
      23.39
      23.39
      23.39
      23.39
      23.39
      23.39
      23.39
      23.39
      ...
      65.2
      65.7
      66.2
      66.6
      66.6
      66.7
      67.1
      67.1
      66.0
      64.92

1 rows × 217 columns

```python
life_expectancy.loc[life_expectancy[1800]==life_expectancy[1800].max(),:]
```

      1800
      1801
      1802
      1803
      1804
      1805
      1806
      1807
      1808
      1809
      ...
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015
      2016

      life_expectancy

      Iceland
      42.85
      33.88
      27.62
      19.6
      24.76
      30.85
      45.82
      43.56
      43.55
      40.42
      ...
      82.1
      82.4
      82.5
      82.8
      82.9
      83.1
      83.2
      83.3
      83.3
      83.3

1 rows × 217 columns

```python
population
```

      1800
      1810
      1820
      1830
      1840
      1850
      1860
      1870
      1880
      1890
      ...
      2006
      2007
      2008
      2009
      2010
      2011
      2012
      2013
      2014
      2015

      total_population

      Abkhazia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Afghanistan
      3280000.0
      3280000.0
      3323519.0
      3448982.0
      3625022.0
      3810047.0
      3973968.0
      4169690.0
      4419695.0
      4710171.0
      ...
      25183615.0
      25877544.0
      26528741.0
      27207291.0
      27962207.0
      28809167.0
      29726803.0
      30682500.0
      31627506.0
      32526562.0

      Akrotiri and Dhekelia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      15700.0
      15700.0
      15700.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Albania
      410445.0
      423591.0
      438671.0
      457234.0
      478227.0
      506889.0
      552800.0
      610036.0
      672544.0
      741688.0
      ...
      3050741.0
      3010849.0
      2968026.0
      2929886.0
      2901883.0
      2886010.0
      2880667.0
      2883281.0
      2889676.0
      2896679.0

      Algeria
      2503218.0
      2595056.0
      2713079.0
      2880355.0
      3082721.0
      3299305.0
      3536468.0
      3811028.0
      4143163.0
      4525691.0
      ...
      33749328.0
      34261971.0
      34811059.0
      35401790.0
      36036159.0
      36717132.0
      37439427.0
      38186135.0
      38934334.0
      39666519.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Northern Marianas
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      South Georgia and the South Sandwich Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      US Minor Outlying Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      Virgin Islands
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      West Bank
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

275 rows × 81 columns

```python
print(population.loc[population[1800] > 100000000].count())
print(population.loc[population[2015] > 100000000].count())
```

    1800    2
    1810    2
    1820    2
    1830    2
    1840    2
           ..
    2011    2
    2012    2
    2013    2
    2014    2
    2015    2
    Length: 81, dtype: int64
    1800    12
    1810    12
    1820    12
    1830    12
    1840    12
            ..
    2011    12
    2012    12
    2013    12
    2014    12
    2015    12
    Length: 81, dtype: int64

#### Merge the different Datasources & Data wrangling

```python
fert = fertility
life= life_expectancy
pop= population
```

```python
fert.shape
```

    (260, 216)

```python
life.shape

```

    (260, 217)

```python
pop.shape
```

    (275, 81)

```python
fert.columns
```

    Index(['1800', '1801', '1802', '1803', '1804', '1805', '1806', '1807', '1808',
           '1809',
           ...
           '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014',
           '2015'],
          dtype='object', length=216)

```python
ncol=[int(x) for x in fert.columns]
fert.set_axis(axis=1, labels=ncol, inplace=True)
```

```python
life.columns
```

    Int64Index([1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809,
                ...
                2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016],
               dtype='int64', length=217)

```python
fert.columns
```

    Int64Index([1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809,
                ...
                2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015],
               dtype='int64', length=216)

```python
pop.columns
```

    Int64Index([1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900,
                1910, 1920, 1930, 1940, 1950, 1951, 1952, 1953, 1954, 1955, 1956,
                1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967,
                1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978,
                1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989,
                1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000,
                2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
                2012, 2013, 2014, 2015],
               dtype='int64')

```python
sfert=fert.stack()
slife=life.stack()
spop=pop.stack()
```

```python
d={'fertility': sfert, 'lifeexp': slife, 'population':spop}
df2=pd.DataFrame(data=d)
```

```python
df3=df2.stack()
```

```python
df4=df3.unstack((0,2))
```

```python
df4.head()
```

      Afghanistan
      Akrotiri and Dhekelia
      Albania
      Algeria
      ...
      Yemen
      Yugoslavia
      Zambia
      Zimbabwe
      Åland

      fertility
      lifeexp
      population
      population
      fertility
      lifeexp
      population
      fertility
      lifeexp
      population
      ...
      population
      population
      fertility
      lifeexp
      population
      fertility
      lifeexp
      population
      population
      lifeexp

      1800
      7.0
      28.21
      3280000.0
      NaN
      4.6
      35.4
      410445.0
      6.99
      28.82
      2503218.0
      ...
      2593000.0
      NaN
      6.71
      32.6
      747000.0
      6.75
      33.7
      1085814.0
      NaN
      NaN

      1801
      7.0
      28.20
      NaN
      NaN
      4.6
      35.4
      NaN
      6.99
      28.82
      NaN
      ...
      NaN
      NaN
      6.71
      32.6
      NaN
      6.75
      33.7
      NaN
      NaN
      NaN

      1802
      7.0
      28.19
      NaN
      NaN
      4.6
      35.4
      NaN
      6.99
      28.82
      NaN
      ...
      NaN
      NaN
      6.71
      32.6
      NaN
      6.75
      33.7
      NaN
      NaN
      NaN

      1803
      7.0
      28.18
      NaN
      NaN
      4.6
      35.4
      NaN
      6.99
      28.82
      NaN
      ...
      NaN
      NaN
      6.71
      32.6
      NaN
      6.75
      33.7
      NaN
      NaN
      NaN

      1804
      7.0
      28.17
      NaN
      NaN
      4.6
      35.4
      NaN
      6.99
      28.82
      NaN
      ...
      NaN
      NaN
      6.71
      32.6
      NaN
      6.75
      33.7
      NaN
      NaN
      NaN

5 rows × 667 columns

```python
df6=df3.unstack(1)
df6s=df6
df6=df6[1880]
df6=df6.unstack(1)
df6.plot.scatter('fertility', 'lifeexp', s=0.1)
# immer df6.head um die Daten zu sehen!
```

```python
df6
```

      fertility
      lifeexp
      population

      Afghanistan
      7.00
      27.39
      4419695.0

      Akrotiri and Dhekelia
      NaN
      NaN
      NaN

      Albania
      4.60
      35.40
      672544.0

      Algeria
      6.99
      28.82
      4143163.0

      American Samoa
      NaN
      NaN
      6582.0

      ...
      ...
      ...
      ...

      Yemen
      6.88
      23.39
      2960773.0

      Yugoslavia
      NaN
      NaN
      NaN

      Zambia
      6.71
      32.60
      744157.0

      Zimbabwe
      6.75
      33.70
      1661683.0

      Åland
      NaN
      NaN
      NaN

256 rows × 3 columns

```python
df6.sort_values(by=['population', 'lifeexp'], ascending=False)
```

      fertility
      lifeexp
      population

      China
      5.50
      32.00
      365544192.0

      India
      5.95
      25.44
      223020377.0

      Russia
      6.80
      30.20
      53996807.0

      United States
      4.75
      39.41
      51256498.0

      Germany
      5.06
      38.91
      43577358.0

      ...
      ...
      ...
      ...

      USSR
      NaN
      NaN
      NaN

      United Korea (former)
      NaN
      NaN
      NaN

      West Germany
      NaN
      NaN
      NaN

      Yugoslavia
      NaN
      NaN
      NaN

      Åland
      NaN
      NaN
      NaN

256 rows × 3 columns

```python
def country_rating(fertility,population):
    if (fertility>=3) and (population>=100000000):
        return 1
    else:
        return 0
```

```python
df6['dynamic'] = df6.apply(lambda x: country_rating(x['fertility'],x['population']),axis=1)
```

```python
df6.loc[df6.loc[:,"dynamic"]==1]
```

      fertility
      lifeexp
      population
      dynamic

      China
      5.50
      32.00
      365544192.0
      1

      India
      5.95
      25.44
      223020377.0
      1

#### Visualize findings (relevant facts & explanatory plots & key findings)

```python
sns.set_context("talk", font_scale=1.1)
plt.figure(figsize=(10,6))
sns.scatterplot(x="fertility",
                y="lifeexp",
                size="population",
                data=df6)
plt.xlabel("fertility")
plt.ylabel("life_expectancy")
plt.tight_layout()
#plt.savefig("gapminder1880.png",format='png',dpi=150)
```

```python
import matplotlib.pyplot as plt
life_expectancy[1800].hist(bins=15)
life_expectancy[2015].hist(bins=15, alpha=0.5)
#plt.savefig('histo.png')
```

```python
sns.distplot(life_expectancy[2015] ,  kde=True, color="red", bins=30)
sns.distplot(life_expectancy[1800] ,  kde=True, color="blue", bins=30)
```

```python
life_expectancy_stack=life_expectancy.stack()
life_expectancy_stack['Yemen'].hist(bins=15)
life_expectancy_stack['Iceland'].hist(bins=15)
#plt.savefig('histo1.png')
#Life_exp1990stack['Ethiopia','Andorra']
```

```python
life_expectancy[2015].hist(facecolor='green',alpha=0.75,histtype='bar',bins=5)
plt.axis([50.0, 85.0,0.0, 80.0])
plt.title('Life expectancy 2015')
plt.xlabel('Years')
plt.ylabel('countries')

```

```python
plt.plot(life_expectancy.loc['Yemen'])
plt.plot(life_expectancy.loc['Iceland'])
```

```python
# Reproduce a Plot
subset2 = life_expectancy[[1880, 1900, 2000, 2015]]
#stichprobe lander mit subset
subset2.plot(kind='hist')
```

```python
#Create a Boxplot
#pd.DataFrame.boxplot()
#Slicing!
#x und y label schreiben!
life_without_na.boxplot(column=[1880,2015],rot=90)
plt.figure(figsize=(100,16))
```

```python
df4[['Germany', 'France', 'Sweden']].plot()
```

```python
df5=df3.unstack(2)
df5.plot.scatter('fertility', 'lifeexp', s=0.1)
```

```python
plt.figure(figsize=(20,20))
cmap = plt.get_cmap('tab20', lut = len(df6)).colors
sns.lmplot(x="fertility", y="lifeexp",data=df6,height=10)
plt.legend(loc=0)
plt.axis((0,10,0,90))
```

#### Create an animated scatterplot

```python
df6
```

      fertility
      lifeexp
      population

      Afghanistan
      7.67
      26.85
      7752118.0

      Akrotiri and Dhekelia
      NaN
      NaN
      10661.0

      Albania
      5.80
      54.48
      1263171.0

      Algeria
      7.65
      42.77
      8872247.0

      American Samoa
      NaN
      NaN
      18937.0

      ...
      ...
      ...
      ...

      Yemen
      7.27
      23.51
      4402320.0

      Yugoslavia
      NaN
      NaN
      16285527.0

      Zambia
      6.71
      42.85
      2316950.0

      Zimbabwe
      6.75
      48.46
      2746854.0

      Åland
      NaN
      NaN
      21649.0

256 rows × 3 columns

```python
cmap = plt.get_cmap('tab20', lut=len(df6)). colors
df6.plot.scatter('fertility', 'lifeexp', s=0.1, c=cmap)
plt.legend(loc=0)
plt.axis((0,10,0,90))
#plt.savefig("lifeexp_1960.png")
```

```python
for i in range(1950,2016):
    df6=df6s[i]
    df6=df6.unstack(1)

    cmap = plt.get_cmap('tab20', lut=len(df6)). colors
    df6.plot.scatter('fertility', 'lifeexp', s=3, c=cmap)
    plt.legend(loc=0)
    plt.axis((0,10,0,90))
    filename=str("lifeexp_"+str(i)+ ".png")
    plt.savefig(filename)

    save as lifeexp_[i].png
```

```python
import imageio
images = []

for i in range(1950, 2016):
    filename = 'lifeexp_{}.png'.format(i)
    images.append(imageio.imread(filename))

imageio.mimsave('output.gif', images, fps=20)
```


# Deep Cox Mixtures with Heterogenous Effects (CMHE) Demo

Author: ***Mononito Goswami***  <mgoswami@cs.cmu.edu>

# Contents

### 1. [Introduction](#introduction)

### 2. [Synthetic Data](#syndata)
####                   1.1 [Generative Process for the Synthetic Dataset.](#gensyndata)
####                   1.2 [Loading and Visualizing the Dataset.](#vissyndata)
####                   1.2 [Split Dataset into Train and Test.](#splitdata)

### 3. [Counterfactual Phenotyping](#phenotyping)

####                   3.1 [Phenotyping with CMHE](#phenocmhe)

####                   3.1 [Comparison with Clustering](#clustering)

### 4. [Factual Regression](#regression)

####                 3.1 [Factual Regression with CMHE](#regcmhe)

####                   3.1 [Comparison with a Deep Cox Proportional Hazards Model](#deepcph)

## 1. Introduction

Figure A: Schematic Description of CMHE:The set of features (confounders) $\mathbf{x}$ are passed through an encoder to obtain deep non-linear representations. These representations then describe the latent phenogroups $\mathbf{P}(Z|X=\mathbf{x})$ and $\mathbf{P}(\mathbf{\phi}|X=\mathbf{x})$ that determine the base survival rate and the treatment effect respectively. Finally, the individual level hazard (survival) curve under an intervention $A=\mathbf{a}$ is described by marginalizing over $Z$ and $\mathbf{\phi}$ as $\mathbf{S}(t|X=x, A=a) = \mathbf{E}_{(Z,\mathbf{\phi)}\sim \mathbf{P}(\cdot|X)}\big[ \mathbf{S}(t|A=\mathbf{a}, X, Z, \mathbf{\phi})\big]$.

 Cox Mixture with Heterogenous Effects (CMHE) is a flexible approach to recover counterfactual phenotypes of individuals that demonstrate heterogneous effects to an intervention in terms of censored Time-to-Event outcomes. CMHE is not restricted by the strong Cox Proportional Hazards assumption or any parametric assumption on the time to event distributions. CMHE achieves this by describing each individual as belonging to two different latent groups,
$Z$ that mediate the base survival rate and $\phi$ the effect of the treatment. CMHE can also be employed to model individual level counterfactuals or for standard factual survival regression.

**Figure B (Right)**: CMHE in Plate Notation. $\mathbf{x}$ confounds treatment assignment $A$ and outcome $T$ (Model parameters and censoring distribution have been abstracted out).

*For full details on Cox Mixtures with Heterogenous Effects, please refer to our preprint*:

[Counterfactual Phenotyping with Censored Time-to-Events, arXiv preprint, C. Nagpal, M. Goswami, K. Dufendach, A. Dubrawski](https://arxiv.org/abs/2202.11089)

## 2. Synthetic Data Example

```python
import pandas as pd
import torch
from tqdm import tqdm
import sys
sys.path.append('../')

from auton_survival.datasets import load_dataset
from cmhe_demo_utils import *
```

### 2.1. Generative Process for the Synthetic Data

1. Features $x_1$, $x_2$ and the base survival phenotypes $Z$ are sampled from $\texttt{scikit-learn's make_blobs(...)}$ function which generates isotropic Gaussian blobs:
$$[x_1, x_2], Z \sim \texttt{sklearn.datasets.make_blobs(K = 3)}$$
2. Features $x_3$ and $x_4$ are sampled uniformly, whereas the underlying treatment effect phenotypes $\phi$ are defined according to an $L_1$-ball:
$$ [x_1, x_2] \sim \texttt{Uniform}(-2, 2) $$
$$ \phi \triangleq \mathbb{1}\{|x_3| + |x_3| > 2\} $$
3. We then sample treat assignments from a Bernoulli distribution:
$$ A \sim  \texttt{Bernoulli}(\frac{1}{2}) $$
4. Next, the time-to-event $T$ conditioned on the confounders $x$, latent $Z$ and latent effect group $\phi$ are generated from a Gompertz distribution:
$$ T^{*}| (Z=k, {\phi}=m, A={a}) \sim \nonumber \texttt{Gompertz}\big({\beta}_{k}^{\top}{x} +({-a}^m)\big) $$
5. Finally, the observed time $T$ is obtained after censoring some of the events and censoring time is chosen uniformly at random upto $T^*$:
$$\delta \sim \texttt{Bernoulli}(\frac{3}{4}), \quad C \sim \texttt{Uniform}(0, {T}^{*})$$
$$ T =  \begin{cases} T^*, & \text{if } \delta = 1 \\ C, & \text{if } \delta = 0 \end{cases} $$

```python
# Load the synthetic dataset
outcomes, features, interventions = load_dataset(dataset='SYNTHETIC')

# Let's take a look at take the dataset
features.head(5)
```

      X1
      X2
      X3
      X4
      X5
      X6
      X7
      X8

      0
      0.148745
      1.892484
      0.195254
      0.860757
      0.696523
      0.483697
      0.339551
      0.374794

      1
      1.139439
      -0.943330
      0.411054
      0.179533
      0.428686
      0.683057
      0.600948
      0.070483

      2
      -0.961237
      0.782706
      -0.305381
      0.583576
      0.157478
      0.070556
      0.034590
      0.776005

      3
      0.466508
      0.694348
      -0.249651
      1.567092
      0.850959
      0.416178
      0.968841
      0.863598

      4
      -0.249002
      -0.552091
      1.854651
      -0.466234
      0.860385
      0.367184
      0.954347
      0.748930

### 2.2. Visualizing the Synthetic Data

```python
plot_synthetic_data(outcomes, features, interventions)
```

### 2.3 Split Dataset into  Train and Test folds

```python
# Hyper-parameters
random_seed = 0
test_size = 0.25

# Split the synthetic data into training and testing data
import numpy as np

np.random.seed(random_seed)
n = features.shape[0]

test_idx = np.zeros(n).astype('bool')
test_idx[np.random.randint(n, size=int(n*test_size))] = True

features_tr = features.iloc[~test_idx]
outcomes_tr = outcomes.iloc[~test_idx]
interventions_tr = interventions[~test_idx]
print(f'Number of training data points: {len(features_tr)}')

features_te = features.iloc[test_idx]
outcomes_te = outcomes.iloc[test_idx]
interventions_te = interventions[test_idx]
print(f'Number of test data points: {len(features_te)}')

x_tr = features_tr.values.astype('float32')
t_tr = outcomes_tr['time'].values.astype('float32')
e_tr = outcomes_tr['event'].values.astype('float32')
a_tr = interventions_tr.values.astype('float32')

x_te = features_te.values.astype('float32')
t_te = outcomes_te['time'].values.astype('float32')
e_te = outcomes_te['event'].values.astype('float32')
a_te = interventions_te.values.astype('float32')

print('Training Data Statistics:')
print(f'Shape of covariates: {x_tr.shape} | times: {t_tr.shape} | events: {e_tr.shape} | interventions: {a_tr.shape}')
```

    Number of training data points: 3899
    Number of test data points: 1101
    Training Data Statistics:
    Shape of covariates: (3899, 8) | times: (3899,) | events: (3899,) | interventions: (3899,)

```python
def find_max_treatment_effect_phenotype(g, zeta_probs, factual_outcomes):
    """
    Find the group with the maximum treatement effect phenotype
    """
    mean_differential_survival = np.zeros(zeta_probs.shape[1]) # Area under treatment phenotype group
    outcomes_train, interventions_train = factual_outcomes

    # Assign each individual to their treatment phenotype group
    for gr in range(g): # For each treatment phenotype group
        # Probability of belonging the the g^th treatment phenotype
        zeta_probs_g = zeta_probs[:, gr]
        # Consider only those individuals who are in the top 75 percentiles in this phenotype
        z_mask = zeta_probs_g>np.quantile(zeta_probs_g, 0.75)

        mean_differential_survival[gr] = find_mean_differential_survival(
            outcomes_train.loc[z_mask], interventions_train.loc[z_mask])

    return np.nanargmax(mean_differential_survival)
```

## 3. Counterfactual Phenotyping

### 3.1 Counterfactual Phenotyping with CMHE

```python
# Hyper-parameters to train model
k = 1 # number of underlying base survival phenotypes
g = 2 # number of underlying treatment effect phenotypes.
layers = [50, 50] # number of neurons in each hidden layer.

random_seed = 3
iters = 100 # number of training epochs
learning_rate = 0.001
batch_size = 128
vsize = 0.15 # size of the validation split
patience = 3
optimizer = "Adam"
```

```python
from auton_survival.models.cmhe import DeepCoxMixturesHeterogenousEffects

torch.manual_seed(random_seed)
np.random.seed(random_seed)

# Instantiate the CMHE model
model = DeepCoxMixturesHeterogenousEffects(k=k, g=g, layers=layers)

model = model.fit(x_tr, t_tr, e_tr, a_tr, vsize=vsize, val_data=None, iters=iters,
                  learning_rate=learning_rate, batch_size=batch_size,
                  optimizer=optimizer, random_state=random_seed,
                  patience=patience)
```

     88%|████████████████████████████████████▉     | 88/100 [00:41<00:05,  2.12it/s]

```python
print(f'Treatment Effect for the {g} groups: {model.torch_model[0].omega.detach()}')

zeta_probs_train = model.predict_latent_phi(x_tr)
zeta_train =  np.argmax(zeta_probs_train, axis=1)
print(f'Distribution of individuals in each treatement phenotype in the training data: \
{np.unique(zeta_train, return_counts=True)[1]}')

max_treat_idx_CMHE = find_max_treatment_effect_phenotype(
    g=2, zeta_probs=zeta_probs_train, factual_outcomes=(outcomes_tr, interventions_tr))
print(f'\nGroup {max_treat_idx_CMHE} has the maximum restricted mean survival time on the training data!')
```

    Treatment Effect for the 2 groups: tensor([-0.5053,  0.3926])
    Distribution of individuals in each treatement phenotype in the training data: [2006 1893]

    Group 1 has the maximum restricted mean survival time on the training data!

    /Users/chiragn/anaconda3/lib/python3.8/site-packages/lifelines/fitters/__init__.py:200: ApproximationWarning: Approximating using linear interpolation`.

      warnings.warn("Approximating using linear interpolation`.\n", exceptions.ApproximationWarning)

### Evaluate CMHE on Test Data

```python
# Now for each individual in the test data, let's find the probability that
# they belong to the max treatment effect group

zeta_probs_test_CMHE = model.predict_latent_phi(x_te)
zeta_test =  np.argmax(zeta_probs_test_CMHE, axis=1)
print(f'Distribution of individuals in each treatement phenotype in the test data: \
{np.unique(zeta_test, return_counts=True)[1]}')

# Now let us evaluate our performance
plot_phenotypes_roc(outcomes_te, zeta_probs_test_CMHE[:, max_treat_idx_CMHE])
```

### 3.3 Comparison with the Clustering phenotyper

We compare the ability of CMHE against dimensionality reduction followed by clustering for counterfactual phenotyping. Specifically, we first perform dimensionality reduction of the input confounders, $\mathbf{x}$, followed by clustering. Due to a small number of confounders in the synthetic data, in the following experiment, we directly perform clustering using a Gaussian Mixture Model (GMM) with 2 components and diagonal covariance matrices.

```python
from phenotyping import ClusteringPhenotyper
from sklearn.metrics import auc

clustering_method = 'gmm'
dim_red_method = None # We would not perform dimensionality reduction for the synthetic dataset
n_components = None
n_clusters = 2 # Number of underlying treatment effect phenotypes

# Running the phenotyper
phenotyper = ClusteringPhenotyper(clustering_method=clustering_method,
                                  dim_red_method=dim_red_method,
                                  n_components=n_components,
                                  n_clusters=n_clusters)
```

```python
zeta_probs_train = phenotyper.fit_phenotype(features_tr.values)
zeta_train = np.argmax(zeta_probs_train, axis=1)
print(f'Distribution of individuals in each treatement phenotype in the training data: \
{np.unique(zeta_train, return_counts=True)[1]}')

max_treat_idx_CP = find_max_treatment_effect_phenotype(
    g=2, zeta_probs=zeta_probs_train, factual_outcomes=(outcomes_tr, interventions_tr))
print(f'\nGroup {max_treat_idx_CP} has the maximum restricted mean survival time on the training data!')
```

    No Dimensionaity reduction specified...
     Proceeding to learn clusters with the raw features...
    Fitting the following Clustering Model:
     GaussianMixture(covariance_type='diag', n_components=3)
    Distribution of individuals in each treatement phenotype in the training data: [1438 1312 1149]

    Group 2 has the maximum restricted mean survival time on the training data!

    /Users/chiragn/anaconda3/lib/python3.8/site-packages/lifelines/fitters/__init__.py:200: ApproximationWarning: Approximating using linear interpolation`.

      warnings.warn("Approximating using linear interpolation`.\n", exceptions.ApproximationWarning)

### Evaluate Clustering Phenotyper on Test Data

```python
# Now for each individual in the test data, let's find the probability that
# they belong to the max treatment effect group

# Use the phenotyper trained on training data to phenotype on testing data
zeta_probs_test_CP = phenotyper.phenotype(x_te)
zeta_test_CP =  np.argmax(zeta_probs_test_CP, axis=1)
print(f'Distribution of individuals in each treatement phenotype in the test data: \
{np.unique(zeta_test_CP, return_counts=True)[1]}')

# Now let us evaluate our performance
plot_phenotypes_roc(outcomes_te, zeta_probs_test_CP[:, max_treat_idx_CP])
```

## 4. CMHE for Factual Regression

For completeness, we further evaluate the performance of CMHE in estimating factual risk over multiple time horizons using the standard survival analysis metrics, including:

1. $\textbf{Brier Score} \ (\textrm{BS})$: Defined as the Mean Squared Error (MSE) around the probabilistic prediction at a certain time horizon.
\begin{align}
\text{BS}(t) = \mathop{\mathbf{E}}_{x\sim\mathcal{D}}\big[ ||\mathbf{1}\{ T > t \} - \widehat{\mathbf{P}}(T>t|X)\big)||_{_\textbf{2}}^\textbf{2}  \big]
\end{align}
2. $ \textbf{Time Dependent Concordance Index} \ (C^{\text{td}}$): A rank order statistic that computes model performance in ranking patients based on their estimated risk at a specfic time horizon.
\begin{align}
C^{td }(t) = \mathbf{P}\big( \hat{F}(t| \mathbf{x}_i) > \hat{F}(t| \mathbf{x}_j)  | \delta_i=1, T_i<T_j, T_i \leq t \big)
\end{align}

We compute the censoring adjusted estimates of the Time Dependent Concordance Index (Antolini et al., 2005; Gerds et al., 2013) and the Integrated Brier Score (i.e. Brier Score integrated over 1, 3 and 5 years. $\text{IBS} = \mathop{\sum}_t  \frac{t}{t_\text{max}}  \cdot \text{BS}(t)$) (Gerds and Schumacher, 2006; Graf et al., 1999) to assess both discriminative performance and model calibration at multiple time horizons.

*We find that CMHE had similar or better discriminative performance than a simple Cox Model with MLP hazard functions. CMHE was also better calibrated as evidenced by overall lower Integrated Brier Score, suggesting utility for factual risk estimation.*

```python
model.predict_survival(x_te, a_te, t=horizons)
```

    array([[0.73384696, 0.29896823, 0.08328623],
           [0.9574444 , 0.83792984, 0.6892881 ],
           [0.93582904, 0.7631118 , 0.56567705],
           ...,
           [0.97567105, 0.9040196 , 0.8069841 ],
           [0.8633335 , 0.5535729 , 0.29013065],
           [0.71951264, 0.28196204, 0.08862075]], dtype=float32)

### 4.1 Factual Regression Performance of  CMHE

```python
horizons = [1, 3, 5]

# Now let us predict survival using CMHE
predictions_test_CMHE = model.predict_survival(x_te, a_te, t=horizons)

CI1, CI3, CI5, IBS = factual_evaluate((x_tr, t_tr, e_tr, a_tr), (x_te, t_te, e_te, a_te),
                                      horizons, predictions_test_CMHE)
print(f'Concordance Index (1 Year): {np.around(CI1, 4)} (3 Year) {np.around(CI3, 4)}: (5 Year): {np.around(CI5, 4)}')
print(f'Integrated Brier Score: {np.around(IBS, 4)}')
```

    Concordance Index (1 Year): 0.6894 (3 Year) 0.6978: (5 Year): 0.6987
    Integrated Brier Score: 0.1524

```python
x_te
```

    array([[ 0.1487453 ,  1.8924836 ,  0.19525401, ...,  0.4836966 ,
             0.33955073,  0.37479353],
           [ 2.2802675 , -0.73033774, -1.7158557 , ...,  0.47515342,
             0.8169348 ,  0.59493804],
           [ 1.4902165 , -0.91186345,  0.7905248 , ...,  0.08489922,
             0.37587634,  0.62569857],
           ...,
           [ 2.0787652 , -2.0501418 , -0.36273366, ...,  0.63403505,
             0.97710544,  0.81890947],
           [-1.5852283 , -0.79666543,  0.9420089 , ...,  0.6343607 ,
             0.5449456 ,  0.01124444],
           [ 0.25414062,  1.5835027 , -0.41139466, ...,  0.369376  ,
             0.05497523,  0.5677395 ]], dtype=float32)

### 4.2 Comparison with Deep Cox-Proportional Hazards Model

```python
from auton_survival.estimators import SurvivalModel

# Now let us train a Deep Cox-proportional Hazard model with two linear layers and tanh activations
random_seed = 0
hyperparams = {'layers':[[50, 50]],
               'lr':[1e-3],
               'bs':[128],
               'activation':['tanh']}
```

```python
dcph_model = SurvivalModel(model='dcph', random_seed=0, hyperparams=hyperparams)

interventions_tr.name, interventions_te.name = 'treat', 'treat'
features_tr_dcph = pd.concat([features_tr, interventions_tr], axis=1)
features_te_dcph = pd.concat([features_te, interventions_te], axis=1)

# Train the DCPH model
dcph_model = dcph_model.fit(features_tr_dcph, outcomes_tr)
```

    0:	[0s / 0s],		train_loss: 3.4539,	val_loss: 3.4712
    1:	[0s / 0s],		train_loss: 3.3983,	val_loss: 3.4378
    2:	[0s / 0s],		train_loss: 3.3631,	val_loss: 3.4173
    3:	[0s / 0s],		train_loss: 3.3495,	val_loss: 3.4090
    4:	[0s / 0s],		train_loss: 3.3353,	val_loss: 3.3997
    5:	[0s / 0s],		train_loss: 3.3275,	val_loss: 3.3949
    6:	[0s / 0s],		train_loss: 3.3231,	val_loss: 3.3874
    7:	[0s / 0s],		train_loss: 3.3233,	val_loss: 3.3898
    8:	[0s / 0s],		train_loss: 3.3157,	val_loss: 3.3892
    9:	[0s / 0s],		train_loss: 3.3086,	val_loss: 3.3821
    10:	[0s / 0s],		train_loss: 3.2997,	val_loss: 3.3888
    11:	[0s / 0s],		train_loss: 3.2978,	val_loss: 3.3864
    12:	[0s / 0s],		train_loss: 3.2952,	val_loss: 3.3854
    13:	[0s / 0s],		train_loss: 3.3009,	val_loss: 3.3910
    14:	[0s / 0s],		train_loss: 3.2954,	val_loss: 3.3866
    15:	[0s / 0s],		train_loss: 3.2926,	val_loss: 3.3890
    16:	[0s / 0s],		train_loss: 3.2925,	val_loss: 3.3853
    17:	[0s / 0s],		train_loss: 3.2928,	val_loss: 3.3838
    18:	[0s / 0s],		train_loss: 3.2794,	val_loss: 3.3780
    19:	[0s / 0s],		train_loss: 3.2956,	val_loss: 3.3802
    20:	[0s / 0s],		train_loss: 3.2847,	val_loss: 3.3787
    21:	[0s / 0s],		train_loss: 3.2823,	val_loss: 3.3785
    22:	[0s / 0s],		train_loss: 3.2827,	val_loss: 3.3784
    23:	[0s / 0s],		train_loss: 3.2775,	val_loss: 3.3852
    24:	[0s / 0s],		train_loss: 3.2831,	val_loss: 3.3793
    25:	[0s / 0s],		train_loss: 3.2764,	val_loss: 3.3689
    26:	[0s / 0s],		train_loss: 3.2639,	val_loss: 3.3785
    27:	[0s / 0s],		train_loss: 3.2696,	val_loss: 3.3704
    28:	[0s / 0s],		train_loss: 3.2651,	val_loss: 3.3758
    29:	[0s / 0s],		train_loss: 3.2729,	val_loss: 3.3691
    30:	[0s / 0s],		train_loss: 3.2721,	val_loss: 3.3706
    31:	[0s / 1s],		train_loss: 3.2599,	val_loss: 3.3760
    32:	[0s / 1s],		train_loss: 3.2659,	val_loss: 3.3685
    33:	[0s / 1s],		train_loss: 3.2632,	val_loss: 3.3655
    34:	[0s / 1s],		train_loss: 3.2614,	val_loss: 3.3643
    35:	[0s / 1s],		train_loss: 3.2646,	val_loss: 3.3647
    36:	[0s / 1s],		train_loss: 3.2546,	val_loss: 3.3661
    37:	[0s / 1s],		train_loss: 3.2581,	val_loss: 3.3605
    38:	[0s / 1s],		train_loss: 3.2541,	val_loss: 3.3634
    39:	[0s / 1s],		train_loss: 3.2509,	val_loss: 3.3617
    40:	[0s / 1s],		train_loss: 3.2515,	val_loss: 3.3631
    41:	[0s / 1s],		train_loss: 3.2558,	val_loss: 3.3641
    42:	[0s / 1s],		train_loss: 3.2572,	val_loss: 3.3597
    43:	[0s / 1s],		train_loss: 3.2445,	val_loss: 3.3635
    44:	[0s / 1s],		train_loss: 3.2508,	val_loss: 3.3605
    45:	[0s / 1s],		train_loss: 3.2433,	val_loss: 3.3623
    46:	[0s / 1s],		train_loss: 3.2479,	val_loss: 3.3560
    47:	[0s / 1s],		train_loss: 3.2356,	val_loss: 3.3593
    48:	[0s / 1s],		train_loss: 3.2412,	val_loss: 3.3614
    49:	[0s / 1s],		train_loss: 3.2410,	val_loss: 3.3589

### Evaluate DCPH on Test Data

```python
# Find suvival scores in the test data
predictions_test_DCPH = dcph_model.predict_survival(features_te_dcph, horizons)

CI1, CI3, CI5, IBS = factual_evaluate((x_tr, t_tr, e_tr, a_tr), (x_te, t_te, e_te, a_te),
                                      horizons, predictions_test_DCPH)
print(f'Concordance Index (1 Year): {np.around(CI1, 4)} (3 Year) {np.around(CI3, 4)}: (5 Year): {np.around(CI5, 4)}')
print(f'Integrated Brier Score: {np.around(IBS, 4)}')
```

    Concordance Index (1 Year): 0.6894 (3 Year) 0.6925: (5 Year): 0.6942
    Integrated Brier Score: 0.1535

```python

```

```python

```

examples/Demo of CMHE on Synthetic Data.ipynb