#  ĐỒ ÁN 01: REGRESSION

__TÊN MÔN HỌC:__ NHẬP MÔN HỌC MÁY

__ĐỀ TÀI:__ CHI PHÍ SỬ DỤNG DỊCH VỤ Y TẾ

__GIẢNG VIÊN:__ NGUYỄN TIẾN HUY

__THỨ TỰ NHÓM:__ 07

__THÀNH VIÊN:__

- 18120184 Nguyễn Nguyên Khang
- 18120189 Trần Đăng Khoa
- 18120264 Nguyễn Duy Vũ
- 18120283 Nguyễn Chiêu Bản
- 18120286 Nguyễn Quốc Bảo

__PHÂN CÔNG:__

Công việc | Thực hiện | Mức độ hoàn thành
------------ | ------------- | ------------
Khám phá dữ liệu cơ bản | Vũ | 100%
Tiền xử lý dữ liệu | Vũ | 100%
Mô hình hóa dữ liệu | Bản, Bảo | 100%
Phân tích dữ liệu tìm Insight| Khang, Khoa | 100%

## Get Data

```python
!gdown --id 19ig1GFDCioN_O2LjhF_JSzhwl_erengT
!gdown --id 1xvHczHpMFgPIQhyMKWT__fHimq0yVVK9
```

    Downloading...
    From: https://drive.google.com/uc?id=19ig1GFDCioN_O2LjhF_JSzhwl_erengT
    To: /content/train.csv
    100% 41.5k/41.5k [00:00<00:00, 15.3MB/s]
    Downloading...
    From: https://drive.google.com/uc?id=1xvHczHpMFgPIQhyMKWT__fHimq0yVVK9
    To: /content/test.csv
    100% 13.9k/13.9k [00:00<00:00, 23.0MB/s]

```python
!pip install --upgrade scikit-learn
```

    Collecting scikit-learn
      Downloading https://files.pythonhosted.org/packages/a8/eb/a48f25c967526b66d5f1fa7a984594f0bf0a5afafa94a8c4dbc317744620/scikit_learn-0.24.2-cp37-cp37m-manylinux2010_x86_64.whl (22.3MB)
         |████████████████████████████████| 22.3MB 1.3MB/s
    Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.0.1)
    Collecting threadpoolctl>=2.0.0
      Downloading https://files.pythonhosted.org/packages/f7/12/ec3f2e203afa394a149911729357aa48affc59c20e2c1c8297a60f33f133/threadpoolctl-2.1.0-py3-none-any.whl
    Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.19.5)
    Requirement already satisfied, skipping upgrade: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.4.1)
    Installing collected packages: threadpoolctl, scikit-learn
      Found existing installation: scikit-learn 0.22.2.post1
        Uninstalling scikit-learn-0.22.2.post1:
          Successfully uninstalled scikit-learn-0.22.2.post1
    Successfully installed scikit-learn-0.24.2 threadpoolctl-2.1.0

## Import

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, StandardScaler, OrdinalEncoder
from sklearn.preprocessing import PowerTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.compose import ColumnTransformer, make_column_transformer
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn import metrics
from sklearn.compose import TransformedTargetRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.cross_decomposition import PLSRegression
from sklearn.svm import SVR
from sklearn import set_config
set_config(display='diagram')

```

## Data Exploration

```python
train_df = pd.read_csv('train.csv')
train_df.head(5)
```

      age
      sex
      bmi
      children
      smoker
      region
      charges

      0
      24
      male
      23.655
      0
      no
      northwest
      2352.96845

      1
      28
      female
      26.510
      2
      no
      southeast
      4340.44090

      2
      51
      male
      39.700
      1
      no
      southwest
      9391.34600

      3
      47
      male
      36.080
      1
      yes
      southeast
      42211.13820

      4
      46
      female
      28.900
      2
      no
      southwest
      8823.27900

### Kích thước của tập dữ liệu

```python
train_df.shape
```

    (1003, 7)

### Ý nghĩa của mỗi cột

```python
columns = train_df.columns
mean_column = ["Tuổi", "Giới tính", "Chỉ số khối cơ thể",
               "Số lượng trẻ con/người phụ thuộc",
               'Tình trạng hút thuốc',
               'Khu vực sinh sống',
               'Chi phí y tế cá nhân']
mean_df = pd.DataFrame(list(zip(columns, mean_column)), columns=['Column', 'Meaning of Column'])
mean_df
```

      Column
      Meaning of Column

      0
      age
      Tuổi

      1
      sex
      Giới tính

      2
      bmi
      Chỉ số khối cơ thể

      3
      children
      Số lượng trẻ con/người phụ thuộc

      4
      smoker
      Tình trạng hút thuốc

      5
      region
      Khu vực sinh sống

      6
      charges
      Chi phí y tế cá nhân

### Dữ liệu có bị lặp hay không ?

```python
train_df.duplicated().sum()
```

    0

### Dữ liệu có bị thiếu hay không ?

```python
train_df.isna().sum()
```

    age         0
    sex         0
    bmi         0
    children    0
    smoker      0
    region      0
    charges     0
    dtype: int64

### Các cột có kiểu dữ liệu gì ?

```python
train_df.dtypes
```

    age           int64
    sex          object
    bmi         float64
    children      int64
    smoker       object
    region       object
    charges     float64
    dtype: object

Các cột "sex" "smoker" "region" mang kiểu object -> Cần chuyển quả kiểu numeric để thuận lợi cho việc huấn luyện các mô hình

### Giá trị của các cột có kiểu dữ liệu object

**Sex**

```python
train_df['sex'].value_counts()
```

    male      506
    female    497
    Name: sex, dtype: int64

Có thể thấy giá trị male và female cách biệt nhau không quá lớn điều này sẽ không ảnh hưởng đến dữ liệu

**Smoker**

```python
train_df['smoker'].value_counts()
```

    no     797
    yes    206
    Name: smoker, dtype: int64

Lượng người hút thuốc và không hút thuốc chênh lệch nhau rất lớn

**Region**

```python
train_df['region'].value_counts()
```

    southeast    259
    northeast    251
    southwest    247
    northwest    246
    Name: region, dtype: int64

Không có giá trị bất thường ở đây

### Xem xét sự phân bổ của các cột dạng số

```python
def missing_ratio(df):
    return (df.isna().mean() * 100).round(1)
def lower_quartile(df):
    return df.quantile(0.25).round(1)
def median(df):
    return df.quantile(0.5).round(1)
def upper_quartile(df):
    return df.quantile(0.75).round(1)
col_numeric = train_df[['age', 'bmi', 'children', 'charges']]
col_numeric.agg([missing_ratio, 'min', lower_quartile, median, upper_quartile, 'max']).round(1)
```

      age
      bmi
      children
      charges

      missing_ratio
      0.0
      0.0
      0.0
      0.0

      min
      18.0
      16.0
      0.0
      1121.9

      lower_quartile
      27.0
      26.2
      0.0
      4780.8

      median
      39.0
      30.2
      1.0
      9447.4

      upper_quartile
      51.0
      34.4
      2.0
      16840.7

      max
      64.0
      53.1
      5.0
      62592.9

## Preprocessing

### Tách tập input output

```python
train_y_df = train_df['charges']
train_X_df = train_df.drop('charges', axis = 1)
train_X_df.shape, train_y_df.shape
```

    ((1003, 6), (1003,))

```python
X, y = train_df.iloc[:, :-1], train_df.iloc[:, -1]
```

### Xử lý các cột có kiểu dữ liệu Categorical

Ở bước khám phá dữ liệu ta có thể thấy cột "sex" "smoker" "region" mang kiểu object. Vì vậy ta sẽ tiến hành xử lý các cột này

```python
onehot_features = ['region']
onehot_transformer = Pipeline(
    [
        ('imputer_cat', SimpleImputer(strategy = 'constant')),
        ('onehot', OneHotEncoder(handle_unknown = 'ignore'))
    ]
)

imp_mode = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
```

```python
ordinal_features = ['sex', 'smoker']
ordinal_transformer = Pipeline(
    [
        ('imputer_cat', SimpleImputer(strategy = 'constant')),
        ('ordinal', OrdinalEncoder())
    ]
)
```

### Xử lý các cột có kiểu dữ liệu Numerical

```python
numeric_features = ['age', 'bmi', 'children']
numeric_transformer = Pipeline(
    [
        ('imputer_num', SimpleImputer(strategy = 'median')),
        ('scaler', StandardScaler())
    ]
)

numeric_transformer_insight = Pipeline(
    [
        ('imputer_num', SimpleImputer(strategy = 'median')),
    ]
)
```

### Preprocessor

```python
preprocessor = ColumnTransformer(
    [
        ('ordinals', ordinal_transformer, ordinal_features),
        ('onehot', onehot_transformer, onehot_features),
        ('numericals', numeric_transformer, numeric_features)
    ],
    remainder = 'drop'
)

preprocessor_insight = ColumnTransformer(
    [
        ('ordinals', ordinal_transformer, ordinal_features),
        ('numericals', numeric_transformer_insight, numeric_features),
        ('unorder_categorical', imp_mode, onehot_features)
    ],
    remainder = 'drop'
)

```

```python
preprocessed_train_X = preprocessor_insight.fit_transform(X)
insight_df = pd.DataFrame(preprocessed_train_X)
columns = ['sex', 'smoker', 'age', 'bmi', 'children', 'region']
insight_df.columns = columns

insight_df['sex'] = insight_df['sex'].astype(int)
insight_df['smoker'] = insight_df['smoker'].astype(int)
insight_df['age'] = insight_df['age'].astype(int)
insight_df['bmi'] = insight_df['bmi'].astype(float)
insight_df['children'] = insight_df['children'].astype(int)

insight_X_df = insight_df.copy()
insight_X_no_object_df = insight_df.drop('region', 1)
insight_df['charges'] = y
insight_df
# X.columns
```

      sex
      smoker
      age
      bmi
      children
      region
      charges

      0
      1
      0
      24
      23.655
      0
      northwest
      2352.96845

      1
      0
      0
      28
      26.510
      2
      southeast
      4340.44090

      2
      1
      0
      51
      39.700
      1
      southwest
      9391.34600

      3
      1
      1
      47
      36.080
      1
      southeast
      42211.13820

      4
      0
      0
      46
      28.900
      2
      southwest
      8823.27900

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      998
      0
      0
      18
      31.350
      4
      northeast
      4561.18850

      999
      0
      0
      39
      23.870
      5
      southeast
      8582.30230

      1000
      1
      0
      58
      25.175
      0
      northeast
      11931.12525

      1001
      0
      1
      37
      47.600
      2
      southwest
      46113.51100

      1002
      1
      0
      55
      29.900
      0
      southwest
      10214.63600

1003 rows × 7 columns

## **Phân tích dữ liệu**

### Vẽ biểu đồ một biến và nhận xét

```python
sns.boxplot(data=insight_df, y='charges')
```

Nhận xét: Biến charges có phân bố bị lệch trái, nhiều outlier

```python
sns.boxplot(data=insight_df, y='age')
```

Nhận xét: Biến age có phân bố chuẩn

```python
sns.boxplot(data=insight_df, y='bmi')
```

Nhận xét: Biến bmi có phân bố chuẩn, tồn tại outlier

```python
sns.distplot(insight_df['sex'], hist=True, kde=False,
             bins=20, color = 'blue',
             hist_kws={'edgecolor':'black'})
plt.xlabel('Sex')
plt.ylabel('Frequency')
```

Nhận xét: Tỉ lệ nam nữ bằng nhau

```python
sns.distplot(insight_df['smoker'], hist=True, kde=False,
             bins=20, color = 'blue',
             hist_kws={'edgecolor':'black'})
plt.xlabel('Smoker')
plt.ylabel('Frequency')
```

Nhận xét: Tỉ lệ người không hút thuốc gấp 4 lần người hút thuốc

```python
sns.distplot(insight_df['children'], hist=True, kde=False,
             bins=20, color = 'blue',
             hist_kws={'edgecolor':'black'})
plt.xlabel('Children')
plt.ylabel('Frequency')
```

Nhận xét: Tỉ lệ người có càng nhiều con giảm dần

### Vẽ biểu đồ các biến tương quan và nhận xét

Tính ma trận tương quan

```python
corr = insight_df.corr()
plt.figure(figsize=(7, 7))
sns.heatmap(corr,square=True, annot=True, cbar=False, fmt='.3f', cmap='Reds')
```

Có thể thấy những thuộc tính như age (yếu), bmi (yếu), smoker (mạnh) có tương quan với thuộc tính charges

```python
sns.boxplot(data=insight_df, x='smoker', y='charges')
```

Biểu đồ trên cho ta thấy người hút thuốc thì có chi phí y tế cao hơn, cụ thể :
- hơn 75% người hút thuốc trả chi phí cao hơn hầu hết tất cả người không hút thuốc
- chi phí thấp nhất của người hút thuốc chỉ nhỉnh hơn một chút so với chi phí của 75% người không hút thuốc.
- nếu chi phí dưới 10k, xác suất cao là người đó không hút thuốc
- nếu chi phí trên 20k, xác suất cao là người đó hút thuốc

```python
sns.scatterplot(data=insight_df, x='age', y='charges', hue='smoker')
```

Nhìn vào biểu đồ trên, ta thấy
- người càng cao tuổi thì số tiền chi cho y tế càng nhiều
- Nếu dưới 35 tuổi và không hút thuốc thì khả năng cao chi phí dưới 6k

```python
sns.scatterplot(data=insight_df, x='bmi', y='charges', hue='smoker')
```

- Người hút thuốc và có chỉ số BMI lớn hơn 30 thì chi phí tổi thiểu là khoảng 30k

### VIF

```python
from statsmodels.stats.outliers_influence import variance_inflation_factor

# VIF dataframe
vif_data = pd.DataFrame()
vif_data["feature"] = insight_X_no_object_df.columns

# calculating VIF for each feature
vif_data["VIF"] = [variance_inflation_factor(insight_X_no_object_df.values, i)
                          for i in range(len(insight_X_no_object_df.columns))]

print(vif_data)
```

        feature       VIF
    0       sex  1.966855
    1    smoker  1.254563
    2       age  7.658193
    3       bmi  8.638958
    4  children  1.816248

    /usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

1 = Không tương quan

Giữa 1 và 5 = Tương quan vừa

Lớn hơn 5 = Tương quan mạnh

(Tham khảo : [1])

Ta thấy các biến `sex`, `smoker`, `children` tương quan vừa với các biến còn lại.

`age` và `bmi` có sự tương quan mạnh với các biến còn lại

Nên thu thập thêm data để giảm sự phụ thuộc giữa các biến

### Insight: Sex có ảnh hưởng đến Smoker?

${H_0}$: sex và smoker độc lập nhau

${H_A}$: sex và smoker phụ thuộc nhau

Đặt:

${A =}$  sex, ${A_1 =}$ `male`, ${A_2}$ = `female`

${B =}$ smoker, ${B_1 =}$ `yes`, ${B_2 =}$ `no`

Ta có:

${H_0}$: ${P(A_i\cap B_j) = P(A_i)P(B_j)}$

${H_A}$: ${P(A_i\cap B_j) \neq P(A_i)P(B_j)}$

Phần dưới sẽ trình bày về mặt toán học lẫn sử dụng thư viện scipy.stats để tính toán

```python
from scipy.stats import chi2_contingency
from scipy.stats import chi2
```

```python
insight_df_1 = insight_df.copy()
insight_df_1['sex'] = insight_df_1['sex'].replace(1,'male')
insight_df_1['sex'] = insight_df_1['sex'].replace(0,'female')
insight_df_1['smoker'] = insight_df_1['smoker'].replace(1,'yes')
insight_df_1['smoker']= insight_df_1['smoker'].replace(0,'no')
```

```python
contigency= pd.crosstab(insight_df_1['sex'], insight_df_1['smoker'])
contigency
```

      smoker
      no
      yes

      sex

      female
      406
      91

      male
      391
      115

```python
contigency.columns = ['no', 'yes']
```

```python
contigency['Pr(Ai)'] = contigency['no'] + contigency['yes']
sum_table = sum(contigency['Pr(Ai)'])
contigency['Pr(Ai)'] = contigency['Pr(Ai)']/sum_table
contigency
```

      no
      yes
      Pr(Ai)

      sex

      female
      406
      91
      0.495513

      male
      391
      115
      0.504487

```python

Pr_No = sum(contigency['no'])/sum_table
Pr_Yes = sum(contigency['yes'])/sum_table

PrBj = [Pr_No,Pr_Yes]
print(f'Pr_No = {Pr_No}\nPr_Yes = {Pr_Yes}\n')
print(sum(PrBj))
```

    Pr_No = 0.7946161515453639
    Pr_Yes = 0.2053838484546361

    1.0

Ta đã tính được ${Pr(A_i)}$ như bảng trên và

${Pr(B_1)}$ = 0.2053838484546361

${Pr(B_2)}$ =  0.7946161515453639

---

Đến đây ta có thể tính:

Giá trị mong đợi ${E}$:

\begin{equation}
\text{Do kỳ vọng A và B độc lập:}\\
E_{ij} = Pr(A_i) \times Pr(B_j) \times N [2]\\
\text{hay}\\
E_{ij} = \frac{\text{(Tổng dòng} \times \text{Tổng cột)}}{\text{Tổng bảng}} [3]   \\
\text{với bảng là bảng contingency}
\end{equation}

Giá trị ${\chi^2}$:

\begin{equation}
\chi^2=\Sigma\frac{(O-E)^2}{E} [2][3]\\
\text{với O là giá trị thực sự và E là giá trị mong đợi}
\end{equation}

Giá trị dof: Degree of freedom

dof cho ${\chi^2}$ độc lập:

\begin{equation}
dof = v = rc - 1 - (r-1) - (c-1) = (r-1)(c-1) [2]\\ = 1
\end{equation}

Chọn mức ý nghĩa:

\begin{equation}
\alpha = 0.05
\end{equation}

Tra bảng Chi Squared với ${\alpha = 0.05, dof = 1}$ ta được critical value ${ = 3.841459}$

Chấp nhận ${H_0}$ nếu
\begin{equation}
\chi^2_v <= 3.841459
\end{equation}

Ta có thể sử dụng `chi2_contingency` của thư viện spicy để tính toán:

```python
# [3]Source:  https://towardsdatascience.com/gentle-introduction-to-chi-square-test-for-independence-7182a7414a95
chi, pval, dof, expected = chi2_contingency(contigency.drop(['Pr(Ai)'],1))
print('p-value là: ', pval)
significance = 0.05
p = 1 - significance
critical_value = chi2.ppf(p, dof)
print('chi = %.6f, critical value = %.6f\n' % (chi, critical_value))
if chi > critical_value:
    print("""Với mức ý nghĩa %.2f, ta bác bỏ Ho và chấp nhận HA.
Kết luận: sex và smoker không độc lập.""" % (significance))
else:
    print("""Với mức ý nghĩa %.2f, ta bác bỏ HA và chấp nhận H0.
Kết luận: sex và smoker độc lập.""" % (significance))
```

    p-value là:  0.09827321674727184
    chi = 2.733346, critical value = 3.841459

    Với mức ý nghĩa 0.05, ta bác bỏ HA và chấp nhận H0.
    Kết luận: sex và smoker độc lập.

```python
chi
```

    2.7333457250617705

Ta kiểm tra, không dùng thư viện:

```python
append_data = []
for sex in insight_df_1['sex'].unique():
  append_data.append(insight_df_1['smoker'][insight_df_1['sex'] == f'{sex}'].value_counts().rename_axis('smoker').reset_index(name='count'))
# items_df['Accidents'].value_counts().rename_axis('Accidents').reset_index(name='Frequency')
contigency_1 = pd.concat(append_data)
contigency_1.reset_index(inplace = True)
contigency_1

contigency_1['sex'] = np.nan
i = 0
pd.options.mode.chained_assignment = None  # default='warn'

for sex in insight_df_1['sex'].unique():
  contigency_1['sex'][i] = f'{sex}'
  contigency_1['sex'][i+1] = f'{sex}'
  i = i + 2

contigency_1 = contigency_1.reindex(columns = ['smoker', 'sex', 'count'])
contigency_1
```

      smoker
      sex
      count

      0
      no
      male
      391

      1
      yes
      male
      115

      2
      no
      female
      406

      3
      yes
      female
      91

```python
expectedValue = []

for Ai in contigency['Pr(Ai)'][::-1]: #female, male
  for Bj in PrBj: # no, yes
      expectedValue.append(Ai*Bj*sum_table)

chiij = []
oij = np.array(contigency_1['count'])

for ij in range(0, len(expectedValue)):
  chiij.append((oij[ij] - expectedValue[ij])**2/expectedValue[ij])

contigency_1['Expected value'] = expectedValue
contigency_1['(O_ij - E_ij)^2/E_ij'] = chiij

chi_square = sum(contigency_1['(O_ij - E_ij)^2/E_ij'])
print('chi_square = ', chi_square)
contigency_1.head()
```

    chi_square =  2.997908815661011

      smoker
      sex
      count
      Expected value
      (O_ij - E_ij)^2/E_ij

      0
      no
      male
      391
      402.075773
      0.305099

      1
      yes
      male
      115
      103.924227
      1.180406

      2
      no
      female
      406
      394.924227
      0.310623

      3
      yes
      female
      91
      102.075773
      1.201781

Ta thấy:

\begin{equation}
\chi^2_v = 2.997908815661011 < 3.841459
\end{equation}

Vậy bác bỏ ${H_A}$ với mức ý nghĩa 0.05, chấp nhận ${H_0}$

Kết luận: sex và smoker độc lập

### Insight: Trung bình của 'age', 'bmi', 'children' giữa người có hút thuốc và người không hút thuốc có bằng nhau

```python
numeric_features = ['age', 'bmi', 'children']

```

```python
from statsmodels.stats.weightstats import ztest

significance = 0.05

for i in range(len(numeric_features)):

    print(f'\nHo: trung bình `{numeric_features[i]}` của người có hút thuốc = trung bình `{numeric_features[i]}` của người không hút thuốc\n')
    print(f'\nHa: trung bình `{numeric_features[i]}` của người có hút thuốc khác trung bình `{numeric_features[i]}` của người không hút thuốc\n')
    stat, p = ztest(insight_df[numeric_features[i]][insight_df['smoker'] == 1],insight_df[numeric_features[i]][insight_df['smoker'] == 0])
    print('stat=%.3f, p=%.3f' % (stat, p))
    print('\nKẾT LUẬN: ')
    if p >= significance:
      print(f'Với mức ý nghĩa {significance}, ta chấp nhận Ho, bác bỏ Ha.\nTrung bình `{numeric_features[i]}` của người có hút thuốc = trung bình `{numeric_features[i]}` của người không hút thuốc\n---')
    else:
      print(f'Với mức ý nghĩa {significance}, ta chấp nhận Ha, bác bỏ Ho.\nTrung bình `{numeric_features[i]}` của người có hút thuốc khác trung bình `{numeric_features[i]}` của người không hút thuốc\n---')
```

    Ho: trung bình `age` của người có hút thuốc = trung bình `age` của người không hút thuốc

    Ha: trung bình `age` của người có hút thuốc khác trung bình `age` của người không hút thuốc

    stat=-1.200, p=0.230

    KẾT LUẬN:
    Với mức ý nghĩa 0.05, ta chấp nhận Ho, bác bỏ Ha.
    Trung bình `age` của người có hút thuốc = trung bình `age` của người không hút thuốc
    ---

    Ho: trung bình `bmi` của người có hút thuốc = trung bình `bmi` của người không hút thuốc

    Ha: trung bình `bmi` của người có hút thuốc khác trung bình `bmi` của người không hút thuốc

    stat=-0.047, p=0.962

    KẾT LUẬN:
    Với mức ý nghĩa 0.05, ta chấp nhận Ho, bác bỏ Ha.
    Trung bình `bmi` của người có hút thuốc = trung bình `bmi` của người không hút thuốc
    ---

    Ho: trung bình `children` của người có hút thuốc = trung bình `children` của người không hút thuốc

    Ha: trung bình `children` của người có hút thuốc khác trung bình `children` của người không hút thuốc

    stat=0.807, p=0.420

    KẾT LUẬN:
    Với mức ý nghĩa 0.05, ta chấp nhận Ho, bác bỏ Ha.
    Trung bình `children` của người có hút thuốc = trung bình `children` của người không hút thuốc
    ---

### Insight: Sự phụ thuộc của `charges` vào `sex`, `smoker`, `age`, `bmi`, `children`

```python
import statsmodels.api as sm
import statsmodels.formula.api as smf
```

```python
results = sm.OLS(y, sm.add_constant(insight_X_no_object_df)).fit()
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                charges   R-squared:                       0.744
    Model:                            OLS   Adj. R-squared:                  0.743
    Method:                 Least Squares   F-statistic:                     580.0
    Date:                Fri, 14 May 2021   Prob (F-statistic):          3.98e-292
    Time:                        15:44:27   Log-Likelihood:                -10164.
    No. Observations:                1003   AIC:                         2.034e+04
    Df Residuals:                     997   BIC:                         2.037e+04
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const       -1.23e+04   1121.474    -10.968      0.000   -1.45e+04   -1.01e+04
    sex           66.0697    386.570      0.171      0.864    -692.515     824.655
    smoker      2.363e+04    478.849     49.344      0.000    2.27e+04    2.46e+04
    age          260.0358     13.870     18.748      0.000     232.818     287.254
    bmi          327.5600     32.307     10.139      0.000     264.162     390.958
    children     434.8043    160.590      2.708      0.007     119.671     749.938
    ==============================================================================
    Omnibus:                      241.068   Durbin-Watson:                   2.068
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              592.918
    Skew:                           1.268   Prob(JB):                    1.78e-129
    Kurtosis:                       5.785   Cond. No.                         299.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Kết luận:

- Biến `sex` không có ý nghĩa (có thể loại bỏ)
- Biến `smoker` có ý nghĩa đối với mô hình về mặt thống kê (với mức ý nghĩa (***) hay p-value = 0.000)
- Biến `age` có ý nghĩa đối với mô hình về mặt thống kê (với mức ý nghĩa (***) hay p-value = 0.000)
- Biến `bmi` có ý nghĩa đối với mô hình về mặt thống kê (với mức ý nghĩa (***) hay p-value = 0.000)
- Biến `children` không có ý nghĩa (có thể loại bỏ)
- Mô hình có thể giải thích được 74.3% sự thay đổi của biến `charges`
- Mô hình tương đối tốt (p-value = 1.78e-129)

Ta huấn luyện lại mô hình dựa theo kết luận trên

```python
results = sm.OLS(y, sm.add_constant(insight_X_no_object_df.drop(['sex', 'children'],1))).fit()
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                charges   R-squared:                       0.742
    Model:                            OLS   Adj. R-squared:                  0.741
    Method:                 Least Squares   F-statistic:                     959.1
    Date:                Fri, 14 May 2021   Prob (F-statistic):          1.63e-293
    Time:                        15:44:27   Log-Likelihood:                -10168.
    No. Observations:                1003   AIC:                         2.034e+04
    Df Residuals:                     999   BIC:                         2.036e+04
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const      -1.185e+04   1097.537    -10.799      0.000    -1.4e+04   -9698.979
    smoker      2.367e+04    479.257     49.386      0.000    2.27e+04    2.46e+04
    age          262.1450     13.884     18.881      0.000     234.900     289.390
    bmi          326.7252     32.392     10.086      0.000     263.160     390.290
    ==============================================================================
    Omnibus:                      238.957   Durbin-Watson:                   2.060
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              580.364
    Skew:                           1.263   Prob(JB):                    9.45e-127
    Kurtosis:                       5.741   Cond. No.                         291.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
print('Parameters: ', results.params)
```

    Parameters:  const    -11852.720452
    smoker    23668.497446
    age         262.144961
    bmi         326.725200
    dtype: float64

Ta thấy:

- Cứ tăng 1 tuổi thì chi phí y tế cá nhân tăng 262.144961, tăng 1 chỉ số bmi thì tăng 326.725200 chi phí y tế cá nhân
- Riêng với smoker, người có hút thuốc thì có chi phí y tế cá nhân cao hơn người không hút thuốc đến 23668.497446

## TIỀN XỬ LÝ + MÔ HÌNH HÓA

reference: [machinelearningmastery](https://machinelearningmastery.com/how-to-transform-target-variables-for-regression-with-scikit-learn/)

### Sử dụng SVR với kernel non-linear: RBF (mặc định)

### Chọn tham số cho mô hình

#### Hàm svr_r2

Hàm này để tính độ chính xác trung bình khi sử dụng phương pháp đánh giá lỗi K-fold của 1 bộ tham số C, gamma

```python
def svr_r2(X, y, kernel, C, gamma, cv):
  svr = SVR(kernel=kernel, C=C, gamma=gamma)
  pipeline = Pipeline(steps=[('Column Transformer', preprocessor), ('model', svr),])
  model = TransformedTargetRegressor(regressor=pipeline, transformer=StandardScaler())
  scores = cross_val_score(model, X, y, scoring='r2', cv=cv, n_jobs=-1)
  return np.mean(scores)
```

#### Hàm optimal_svr

Để chọn ra chọn siêu tham số làm cho mô hình có độ chính xác cao nhất trên tập huấn luyện

```python
def optimal_svr(X, y, kernel, c_range, gamma_range, train_r2=[]):
  max_score = 0
  cv = KFold(n_splits=10, shuffle=True, random_state=1)
  for c in c_range:
    scores = []
    for g in gamma_range:
      score = svr_r2(X, y, kernel, c, g, cv)
      scores.append(score)
      if score > max_score:
        max_score = score
        res = (c, g)
    train_r2.append(scores)
  return res

```

```python
C_range = [0.05,0.1,1,10, 50]
gamma_range = [0.01,0.05,0.1,1,10]
kernel = 'rbf'
train_r2 = []
best_c, best_g = optimal_svr(X, y, kernel, C_range, gamma_range, train_r2)
```

#### Trực quan kết quả

```python
train_r2 = pd.DataFrame(train_r2, columns=gamma_range, index=C_range)
```

```python
plt.figure(figsize=(7, 7))
sns.heatmap(train_r2,square=True, annot=True, cbar=False, fmt='.3f', cmap='Reds')
plt.xlabel('Gamma'); plt.ylabel('C');plt.title('R-square of training set')
plt.show()
```

Vậy các tham số tìm được là:

```python
print('C= %s và gamma= %s' %(best_c, best_g))
```

    C= 10 và gamma= 0.05

### Dùng các siêu tham số tìm được ở trên để huấn luyện mô hình

```python
df_test = pd.read_csv('test.csv')
X_test, y_test = df_test.iloc[:, :-1], df_test.iloc[:, -1]
```

```python
svr = SVR(kernel= kernel, C=best_c, gamma=best_g)
pipeline = Pipeline(steps=[('Column Transformer', preprocessor), ('model', svr),])
model = TransformedTargetRegressor(regressor=pipeline, transformer=StandardScaler())
model.fit(X,y)
```

<input class="sk-toggleable__control sk-hidden--visually" id="abd9613c-d5b6-4430-b189-32a82b60c5c4" type="checkbox" >TransformedTargetRegressor

TransformedTargetRegressor(regressor=Pipeline(steps=[('Column Transformer',
                                                      ColumnTransformer(transformers=[('ordinals',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('ordinal',
                                                                                OrdinalEncoder())]),
                                                                                ['sex',
                                                                                'smoker']),
                                                                                ('onehot',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('onehot',
                                                                                OneHotEncoder(handle_unknown='ignore'))]),
                                                                                ['region']),
                                                                                ('numericals',
                                                                                Pipeline(steps=[('imputer_num',
                                                                                SimpleImputer(strategy='median')),
                                                                                ('scaler',
                                                                                StandardScaler())]),
                                                                                ['age',
                                                                                'bmi',
                                                                                'children'])])),
                                                     ('model',
                                                      SVR(C=10, gamma=0.05))]),
                           transformer=StandardScaler())

<input class="sk-toggleable__control sk-hidden--visually" id="4a84e96e-87ad-4405-9cf9-261d56ff8e56" type="checkbox" >Column Transformer: ColumnTransformer

ColumnTransformer(transformers=[('ordinals',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('ordinal',
                                                  OrdinalEncoder())]),
                                 ['sex', 'smoker']),
                                ('onehot',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('onehot',
                                                  OneHotEncoder(handle_unknown='ignore'))]),
                                 ['region']),
                                ('numericals',
                                 Pipeline(steps=[('imputer_num',
                                                  SimpleImputer(strategy='median')),
                                                 ('scaler', StandardScaler())]),
                                 ['age', 'bmi', 'children'])])

<input class="sk-toggleable__control sk-hidden--visually" id="4147a348-20fa-484a-8399-79e74fa48a2f" type="checkbox" >ordinals

['sex', 'smoker']

<input class="sk-toggleable__control sk-hidden--visually" id="a86918f4-fc86-473d-bed1-dc520a93d3ee" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="d1643781-d9ed-4504-ab77-f654e04d488f" type="checkbox" >OrdinalEncoder

OrdinalEncoder()

<input class="sk-toggleable__control sk-hidden--visually" id="700453d0-f42b-4e95-8b1d-680a893f7a3b" type="checkbox" >onehot

['region']

<input class="sk-toggleable__control sk-hidden--visually" id="81adac7e-996e-4c53-994b-b7adcd7e06dc" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="79922e74-d56c-4328-b788-3f92bafaaf33" type="checkbox" >OneHotEncoder

OneHotEncoder(handle_unknown='ignore')

<input class="sk-toggleable__control sk-hidden--visually" id="76e19f77-4e38-4fe9-873f-f57d6adc40ad" type="checkbox" >numericals

['age', 'bmi', 'children']

<input class="sk-toggleable__control sk-hidden--visually" id="4042eb4c-7ec9-42a2-828e-1bab367739a5" type="checkbox" >SimpleImputer

SimpleImputer(strategy='median')

<input class="sk-toggleable__control sk-hidden--visually" id="7b90f6d3-344e-4c47-a0cc-6c64a12adb0f" type="checkbox" >StandardScaler

StandardScaler()

<input class="sk-toggleable__control sk-hidden--visually" id="aafab106-4584-401e-b1ec-5b5d1df26331" type="checkbox" >SVR

SVR(C=10, gamma=0.05)

<input class="sk-toggleable__control sk-hidden--visually" id="97e93c96-4687-43a8-bc89-1877dd58d027" type="checkbox" >StandardScaler

StandardScaler()

```python
preds = model.predict(X_test)
```

```python
score = model.score(X_test,y_test)
score
```

    0.856584307140871

### Sử dụng SVR với kernel linear

#### Lựa chọn siêu tham số tốt nhất cho mô hình

```python
kernel = 'linear'
train_r2 = []
best_c, best_g = optimal_svr(X, y, kernel, C_range, gamma_range, train_r2)
```

#### Trực quan kết quả

```python
train_r2 = pd.DataFrame(train_r2, columns=gamma_range, index=C_range)
```

```python
plt.figure(figsize=(7, 7))
sns.heatmap(train_r2,square=True, annot=True, cbar=False, fmt='.3f', cmap='Reds')
plt.xlabel('Gamma'); plt.ylabel('C');plt.title('R-square of training set')
plt.show()
```

Vậy các siêu tham số tốt nhất tìm được là:

```python
print('C= %s và gamma= %s' %(best_c, best_g))
```

    C= 1 và gamma= 0.01

**Nhận xét:**
- Siêu tham số gamma chỉ có giá trị trong các mô hình phi tuyến (bao gồm: ‘rbf’, ‘poly’ và ‘sigmoid’)

#### Dùng các siêu tham số tìm được để huấn luyện mô hình

```python
svr = SVR(kernel= kernel, C=best_c, gamma=best_g)
pipeline = Pipeline(steps=[('Column Transformer', preprocessor), ('model', svr),])
model = TransformedTargetRegressor(regressor=pipeline, transformer=StandardScaler())
model.fit(X,y)
```

<input class="sk-toggleable__control sk-hidden--visually" id="4cf3fc3c-edab-472a-ae32-e46bb429e19b" type="checkbox" >TransformedTargetRegressor

TransformedTargetRegressor(regressor=Pipeline(steps=[('Column Transformer',
                                                      ColumnTransformer(transformers=[('ordinals',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('ordinal',
                                                                                OrdinalEncoder())]),
                                                                                ['sex',
                                                                                'smoker']),
                                                                                ('onehot',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('onehot',
                                                                                OneHotEncoder(handle_unknown='ignore'))]),
                                                                                ['region']),
                                                                                ('numericals',
                                                                                Pipeline(steps=[('imputer_num',
                                                                                SimpleImputer(strategy='median')),
                                                                                ('scaler',
                                                                                StandardScaler())]),
                                                                                ['age',
                                                                                'bmi',
                                                                                'children'])])),
                                                     ('model',
                                                      SVR(C=1, gamma=0.01,
                                                          kernel='linear'))]),
                           transformer=StandardScaler())

<input class="sk-toggleable__control sk-hidden--visually" id="2ff1bcf2-fca5-4445-9734-b97beb2f16a8" type="checkbox" >Column Transformer: ColumnTransformer

ColumnTransformer(transformers=[('ordinals',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('ordinal',
                                                  OrdinalEncoder())]),
                                 ['sex', 'smoker']),
                                ('onehot',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('onehot',
                                                  OneHotEncoder(handle_unknown='ignore'))]),
                                 ['region']),
                                ('numericals',
                                 Pipeline(steps=[('imputer_num',
                                                  SimpleImputer(strategy='median')),
                                                 ('scaler', StandardScaler())]),
                                 ['age', 'bmi', 'children'])])

<input class="sk-toggleable__control sk-hidden--visually" id="26ea4881-0402-44a6-958b-542c87288673" type="checkbox" >ordinals

['sex', 'smoker']

<input class="sk-toggleable__control sk-hidden--visually" id="f3b1af12-aca2-4e55-9fef-bb6a00e49cbc" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="59f13eb2-aa4c-41ce-bd5b-d8ee0d789d94" type="checkbox" >OrdinalEncoder

OrdinalEncoder()

<input class="sk-toggleable__control sk-hidden--visually" id="8f8b41f2-4a05-4ffd-a422-6feedab1cd2c" type="checkbox" >onehot

['region']

<input class="sk-toggleable__control sk-hidden--visually" id="f970f3e7-fe42-4dba-bf62-5f62f82a17b5" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="6688b0b3-a3a9-40ed-a839-063dd3e9deec" type="checkbox" >OneHotEncoder

OneHotEncoder(handle_unknown='ignore')

<input class="sk-toggleable__control sk-hidden--visually" id="3e4d4a6c-f77b-4a45-b857-7e45ceb6f1b9" type="checkbox" >numericals

['age', 'bmi', 'children']

<input class="sk-toggleable__control sk-hidden--visually" id="24a856bc-098c-4ab4-987f-cdb67fc2954e" type="checkbox" >SimpleImputer

SimpleImputer(strategy='median')

<input class="sk-toggleable__control sk-hidden--visually" id="773eeac8-19cc-4844-8108-cc925fa8f8fe" type="checkbox" >StandardScaler

StandardScaler()

<input class="sk-toggleable__control sk-hidden--visually" id="dd404939-da94-4377-9abc-e891fcf4507f" type="checkbox" >SVR

SVR(C=1, gamma=0.01, kernel='linear')

<input class="sk-toggleable__control sk-hidden--visually" id="4776dc4b-a851-4870-a918-f86cfd3d4046" type="checkbox" >StandardScaler

StandardScaler()

```python
preds = model.predict(X_test)
```

```python
score = model.score(X_test,y_test)
score
```

    0.7264543523698831

**Nhận xét:**
- Ta thấy vấn đề dự đoán chi phí y tế được giải quyết tốt hơn trên mô hình phi tuyến
- Thuật toán SVR ở kernel phi tuyến cho độ chính xác cao hơn rất nhiều so với kernel tuyến tính
- Độ chính xác của mô hình còn khá thấp, nên ta thử các các tối ưu khác

### Sử dụng mô hình SVR với kernel non-linear trên dữ liệu được xóa outliers

Từ biểu đồ hộp ở phần phân tích dữ liệu, ta thấy ở các bệnh nhân không hút thuốc thì có sự biến động khá lớn trong chi phí điều trị bệnh. Có thể những người này mắc các bệnh bẩm sinh hoặc không liên quan đến thuốc lá.

Giả sử các bệnh nhân không hút thuốc có chi phí điều trị bệnh cao bất thường là các outliers ta tiến hành xóa các bệnh nhân này ra khỏi tập huấn luyện

Theo biểu đồ hộp, thì khoảng lớn hơn 1.5 lần ipr là các outliers

```python
q25 = train_df[train_df.smoker == 'no']['charges'].agg(lower_quartile)
q75 = train_df[train_df.smoker == 'no']['charges'].agg(upper_quartile)
iqr = q75 - q25
cut_off = iqr * 1.5
lower, upper = q25 - cut_off, q75 + cut_off
```

#### Xóa các outliers

```python
removed_outliers = train_df[((train_df.smoker == 'no') & (train_df.charges > lower) & (train_df.charges < upper)) | (train_df.smoker == 'yes')]
```

Ta xem lại biểu đồ để xác nhận không còn những điểm outliers

```python
sns.boxplot(data= removed_outliers, x='smoker', y='charges');
```

Tách tập train đã bỏ các outlier

```python
removed_outliers_X, removed_outliers_y = removed_outliers.iloc[:, :-1], removed_outliers.iloc[:, -1]
```

#### Tìm các siêu tham số tốt nhất với dữ liệu hiện tại

Các bước thực hiện tương tự như trên

```python
kernel = 'rbf'
train_r2 = []
best_c, best_g = optimal_svr(removed_outliers_X, removed_outliers_y, kernel, C_range, gamma_range, train_r2)
```

```python
train_r2 = pd.DataFrame(train_r2, columns=gamma_range, index=C_range)
```

```python
plt.figure(figsize=(7, 7))
sns.heatmap(train_r2,square=True, annot=True, cbar=False, fmt='.3f', cmap='Reds')
plt.xlabel('Gamma'); plt.ylabel('C');plt.title('R-square of training set')
plt.show()
```

```python
print('C= %s và gamma= %s' %(best_c, best_g))
```

    C= 10 và gamma= 0.05

```python
svr = SVR(kernel= kernel, C=best_c, gamma=best_g)
pipeline = Pipeline(steps=[('Column Transformer', preprocessor), ('model', svr),])
model = TransformedTargetRegressor(regressor=pipeline, transformer=StandardScaler())
model.fit(removed_outliers_X,removed_outliers_y)
```

<input class="sk-toggleable__control sk-hidden--visually" id="6290af69-5422-4782-a591-26bf65f10614" type="checkbox" >TransformedTargetRegressor

TransformedTargetRegressor(regressor=Pipeline(steps=[('Column Transformer',
                                                      ColumnTransformer(transformers=[('ordinals',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('ordinal',
                                                                                OrdinalEncoder())]),
                                                                                ['sex',
                                                                                'smoker']),
                                                                                ('onehot',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('onehot',
                                                                                OneHotEncoder(handle_unknown='ignore'))]),
                                                                                ['region']),
                                                                                ('numericals',
                                                                                Pipeline(steps=[('imputer_num',
                                                                                SimpleImputer(strategy='median')),
                                                                                ('scaler',
                                                                                StandardScaler())]),
                                                                                ['age',
                                                                                'bmi',
                                                                                'children'])])),
                                                     ('model',
                                                      SVR(C=10, gamma=0.05))]),
                           transformer=StandardScaler())

<input class="sk-toggleable__control sk-hidden--visually" id="a876c177-07ef-458d-acb0-33c70e569e17" type="checkbox" >Column Transformer: ColumnTransformer

ColumnTransformer(transformers=[('ordinals',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('ordinal',
                                                  OrdinalEncoder())]),
                                 ['sex', 'smoker']),
                                ('onehot',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('onehot',
                                                  OneHotEncoder(handle_unknown='ignore'))]),
                                 ['region']),
                                ('numericals',
                                 Pipeline(steps=[('imputer_num',
                                                  SimpleImputer(strategy='median')),
                                                 ('scaler', StandardScaler())]),
                                 ['age', 'bmi', 'children'])])

<input class="sk-toggleable__control sk-hidden--visually" id="f3549ca7-0bc2-4bf3-b35a-d0d3ded25b0b" type="checkbox" >ordinals

['sex', 'smoker']

<input class="sk-toggleable__control sk-hidden--visually" id="8edf01a1-3a83-4abe-b41d-d6d92effd007" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="09ab7340-8f3d-468d-b095-cc6f2a48f030" type="checkbox" >OrdinalEncoder

OrdinalEncoder()

<input class="sk-toggleable__control sk-hidden--visually" id="4aeb26d4-c7ba-4e9e-8473-7c847a9e30fd" type="checkbox" >onehot

['region']

<input class="sk-toggleable__control sk-hidden--visually" id="3f3b218f-aec2-4e3a-81d5-27eafcaf9865" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="47b4e5af-6590-47d7-8b76-2e005d85eb24" type="checkbox" >OneHotEncoder

OneHotEncoder(handle_unknown='ignore')

<input class="sk-toggleable__control sk-hidden--visually" id="c35eb755-3e20-416d-b3c4-857a7455c5e7" type="checkbox" >numericals

['age', 'bmi', 'children']

<input class="sk-toggleable__control sk-hidden--visually" id="9e06c8c1-ff14-42a1-926f-ee3c0d9785f2" type="checkbox" >SimpleImputer

SimpleImputer(strategy='median')

<input class="sk-toggleable__control sk-hidden--visually" id="620fd9f8-e9d6-4c5f-b3ff-1eb5835c36ab" type="checkbox" >StandardScaler

StandardScaler()

<input class="sk-toggleable__control sk-hidden--visually" id="2a3dce83-9c1c-42ad-89a9-e7d2aff12d0f" type="checkbox" >SVR

SVR(C=10, gamma=0.05)

<input class="sk-toggleable__control sk-hidden--visually" id="32ce18a1-411d-417c-891d-fa4a4ab4ec41" type="checkbox" >StandardScaler

StandardScaler()

```python
preds = model.predict(X_test)
score = model.score(X_test,y_test)
score
```

    0.8532437921534919

**Nhận xét:**
- Khi xóa các dữ liệu bệnh nhân bất thường thì độ chính xác trên tập train tăng lên đến gần 90%
- Tuy nhiên khi dự đoán trên tập test thì độ chính xác của mô hình không tăng mà lại thấp đi một chút
- Có vẻ là lớp bệnh nhân này cũng có tỉ lệ cao trong thực tế nên nó không hẵn là các outlier mà có thể là các đối tượng khá đặc biệt mà cần quan sát thêm các đặc trưng khác để có thể dự đoán chính xác

### Dùng Simple Regression

```python
from sklearn.linear_model import LinearRegression
reg = LinearRegression()
pipeline = Pipeline(steps=[('Column Transformer', preprocessor), ('model', reg),])
model = TransformedTargetRegressor(regressor=pipeline, transformer=StandardScaler())
model.fit(X,y)
model.score(X, y)
```

    0.744221765190394

```python
model
```

<input class="sk-toggleable__control sk-hidden--visually" id="600b4cd3-361a-47d4-8ffe-bda7ba79d07d" type="checkbox" >TransformedTargetRegressor

TransformedTargetRegressor(regressor=Pipeline(steps=[('Column Transformer',
                                                      ColumnTransformer(transformers=[('ordinals',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('ordinal',
                                                                                OrdinalEncoder())]),
                                                                                ['sex',
                                                                                'smoker']),
                                                                                ('onehot',
                                                                                Pipeline(steps=[('imputer_cat',
                                                                                SimpleImputer(strategy='constant')),
                                                                                ('onehot',
                                                                                OneHotEncoder(handle_unknown='ignore'))]),
                                                                                ['region']),
                                                                                ('numericals',
                                                                                Pipeline(steps=[('imputer_num',
                                                                                SimpleImputer(strategy='median')),
                                                                                ('scaler',
                                                                                StandardScaler())]),
                                                                                ['age',
                                                                                'bmi',
                                                                                'children'])])),
                                                     ('model',
                                                      LinearRegression())]),
                           transformer=StandardScaler())

<input class="sk-toggleable__control sk-hidden--visually" id="41052e77-ca99-4ded-a82d-646b4a07cf3a" type="checkbox" >Column Transformer: ColumnTransformer

ColumnTransformer(transformers=[('ordinals',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('ordinal',
                                                  OrdinalEncoder())]),
                                 ['sex', 'smoker']),
                                ('onehot',
                                 Pipeline(steps=[('imputer_cat',
                                                  SimpleImputer(strategy='constant')),
                                                 ('onehot',
                                                  OneHotEncoder(handle_unknown='ignore'))]),
                                 ['region']),
                                ('numericals',
                                 Pipeline(steps=[('imputer_num',
                                                  SimpleImputer(strategy='median')),
                                                 ('scaler', StandardScaler())]),
                                 ['age', 'bmi', 'children'])])

<input class="sk-toggleable__control sk-hidden--visually" id="6fd56200-5830-4385-8b6d-dc1e943a8028" type="checkbox" >ordinals

['sex', 'smoker']

<input class="sk-toggleable__control sk-hidden--visually" id="8d624528-417e-430f-a198-10915eb2df86" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="3b6498f4-3116-4704-95cc-ee054dd79b80" type="checkbox" >OrdinalEncoder

OrdinalEncoder()

<input class="sk-toggleable__control sk-hidden--visually" id="f4db03fa-d164-4cba-b7b7-362d767273c7" type="checkbox" >onehot

['region']

<input class="sk-toggleable__control sk-hidden--visually" id="dbb0238a-7f3d-4d59-8677-aedeb91e5509" type="checkbox" >SimpleImputer

SimpleImputer(strategy='constant')

<input class="sk-toggleable__control sk-hidden--visually" id="b96a46d2-868b-4609-8e74-f4a53a75d252" type="checkbox" >OneHotEncoder

OneHotEncoder(handle_unknown='ignore')

<input class="sk-toggleable__control sk-hidden--visually" id="99f2f9c9-c5c4-4c29-8b57-a08d2d08d269" type="checkbox" >numericals

['age', 'bmi', 'children']

<input class="sk-toggleable__control sk-hidden--visually" id="98f316f4-1499-4b90-8fba-faea114b1063" type="checkbox" >SimpleImputer

SimpleImputer(strategy='median')

<input class="sk-toggleable__control sk-hidden--visually" id="c60ead9f-24ff-4618-8ef7-e4c0e87acd8b" type="checkbox" >StandardScaler

StandardScaler()

<input class="sk-toggleable__control sk-hidden--visually" id="b5ee5a73-0bc8-492b-9a24-29c16045e525" type="checkbox" >LinearRegression

LinearRegression()

<input class="sk-toggleable__control sk-hidden--visually" id="c530071c-08fb-4add-8832-352f3102ca78" type="checkbox" >StandardScaler

StandardScaler()

```python
preds = model.predict(X_test)
```

```python
score = model.score(X_test,y_test)
score
```

    0.7658201226718288

#### Giảm chiều và trực quan

```python
Xpre = preprocessor.fit_transform(X)
```

```python
pls = PLSRegression(n_components=1)
pls.fit(Xpre, y)
```

<input class="sk-toggleable__control sk-hidden--visually" id="66357638-51ff-41a9-b0cf-afd35a962a0d" type="checkbox" checked>PLSRegression

PLSRegression(n_components=1)

```python
X_test_pre = preprocessor.transform(X_test)
```

```python
plt.scatter(x=pls.transform(X_test_pre), y= y_test, alpha = 0.5,color= 'r', label= 'True')
plt.scatter(x=pls.transform(X_test_pre), y = pls.predict(X_test_pre), alpha= 0.5, label= 'Predictions')
plt.legend()
plt.show()
```

### Kết luận:

- Đối với dữ liệu này, trong các mô hình được thực hiện thì mô hình SVR với kernel là RBF cho độ chính xác cao nhất là khoảng 86% trên tập test và lên đến 90% trên tập train khi xóa đi các bệnh nhân không hút thuốc nhưng chi phí y tế cao
- Đối tượng bệnh nhân không hút thuốc nhưng lại mất nhiều chi phí cho dịch vụ y tế cần được quan sát thêm các đặc trưng để dự đoán được chính xác hơn

## Tham khảo

[1]. [Stephanie - Variance Inflation Factor - Statisticshowto.com](https://www.statisticshowto.com/variance-inflation-factor/)

[2]. https://www3.nd.edu/~rwilliam/stats1/x51.pdf

[3]. https://towardsdatascience.com/gentle-introduction-to-chi-square-test-for-independence-7182a7414a95

```python

```


# Customer Segmentation: Estimate Individualized Responses to Incentives

Nowadays, business decision makers rely on estimating the causal effect of interventions to answer what-if questions about shifts in strategy, such as promoting specific product with discount, adding new features to a website or increasing investment from a sales team. However, rather than learning whether to take action for a specific intervention for all users, people are increasingly interested in understanding the different responses from different users to the two alternatives. Identifying the characteristics of users having the strongest response for the intervention could help make rules to segment the future users into different groups. This can help optimize the policy to use the least resources and get the most profit.

In this case study, we will use a personalized pricing example to explain how the [EconML](https://aka.ms/econml) and [DoWhy](https://github.com/microsoft/dowhy) libraries could fit into this problem and provide robust and reliable causal solutions.

### Summary

1. [Background](#background)
2. [Data](#data)
3. [Create Causal Model and Identify Causal Effect with DoWhy](#identify)
4. [Get Causal Effects with EconML](#estimate)
5. [Test Estimate Robustness with DoWhy](#robustness)
    1. [Add Random Common Cause](#random-common-cause)
    2. [Add Unobserved Common Cause](#unobserved-common-cause)
    3. [Replace Treatment with a Random (Placebo) Variable](#placebo-variable)
    4. [Remove a Random Subset of the Data](#subset)
6. [Understand Treatment Effects with EconML](#interpret)
7. [Make Policy Decisions with EconML](#policy)
8. [Conclusions](#conclusion)

# Background

The global online media market is growing fast over the years. Media companies are always interested in attracting more users into the market and encouraging them to buy more songs or become members. In this example, we'll consider a scenario where one experiment a media company is running is to give small discount (10%, 20% or 0) to their current users based on their income level in order to boost the likelihood of their purchase. The goal is to understand the **heterogeneous price elasticity of demand** for people with different income level, learning which users would respond most strongly to a small discount. Furthermore, their end goal is to make sure that despite decreasing the price for some consumers, the demand is raised enough to boost the overall revenue.

The EconML and DoWhy libraries complement each other in implementing this solution. On one hand, the DoWhy library can help [build a causal model, indentify the causal effect](#identify) and [test causal assumptions](#robustness). On the other hand, EconML’s `DML` based estimators can be used to take the discount variation in existing data, along with a rich set of user features, to [estimate heterogeneous price sensitivities](#estimate) that vary with multiple customer features. Then, the `SingleTreeCateInterpreter` provides a [presentation-ready summary](#interpret) of the key features that explain the biggest differences in responsiveness to a discount, and the `SingleTreePolicyInterpreter` recommends a [policy](#policy) on who should receive a discount in order to increase revenue (not only demand), which could help the company to set an optimal price for those users in the future.

```python
# Some imports to get us started
# Utilities
import os
import urllib.request
import numpy as np
import pandas as pd
from networkx.drawing.nx_pydot import to_pydot
from IPython.display import Image, display

# Generic ML imports
from sklearn.preprocessing import PolynomialFeatures
from sklearn.ensemble import GradientBoostingRegressor

# DoWhy imports
import dowhy
from dowhy import CausalModel

# EconML imports
from econml.dml import LinearDML, ForestDML
from econml.cate_interpreter import SingleTreeCateInterpreter, SingleTreePolicyInterpreter

import matplotlib.pyplot as plt

%matplotlib inline
```

# Data

The dataset* has ~10,000 observations and includes 9 continuous and categorical variables that represent user's characteristics and online behaviour history such as age, log income, previous purchase, previous online time per week, etc.

We define the following variables:

Feature Name|Type|Details
:--- |:---|:---
**account_age** |W| user's account age
**age** |W|user's age
**avg_hours** |W| the average hours user was online per week in the past
**days_visited** |W| the average number of days user visited the website per week in the past
**friend_count** |W| number of friends user connected in the account
**has_membership** |W| whether the user had membership
**is_US** |W| whether the user accesses the website from the US
**songs_purchased** |W| the average songs user purchased per week in the past
**income** |X| user's income
**price** |T| the price user was exposed during the discount season (baseline price * small discount)
**demand** |Y| songs user purchased during the discount season

**To protect the privacy of the company, we use the simulated data as an example here. The data is synthetically generated and the feature distributions don't correspond to real distributions. However, the feature names have preserved their names and meaning.*

The treatment and outcome are generated using the following functions:
$$
T =
\begin{cases}
  1 & \text{with } p=0.2,  \\
  0.9 & \text{with }p=0.3, & \text{if income}<1 \\
  0.8 & \text{with }p=0.5, \\
  \\
    1 & \text{with }p=0.7, \\
  0.9 & \text{with }p=0.2, & \text{if income}\ge1 \\
  0.8 & \text{with }p=0.1, \\
\end{cases}
$$

\begin{align}
\gamma(X) & = -3 - 14 \cdot \{\text{income}<1\} \\
\beta(X,W) & = 20 + 0.5 \cdot \text{avg_hours} + 5 \cdot \{\text{days_visited}>4\} \\
Y &= \gamma(X) \cdot T + \beta(X,W)
\end{align}

```python
# Import the sample pricing data
file_url = "https://msalicedatapublic.blob.core.windows.net/datasets/Pricing/pricing_sample.csv"
train_data = pd.read_csv(file_url)
```

```python
# Data sample
train_data.head()
```

      account_age
      age
      avg_hours
      days_visited
      friends_count
      has_membership
      is_US
      songs_purchased
      income
      price
      demand

      0
      3
      53
      1.834234
      2
      8
      1
      1
      4.903237
      0.960863
      1.0
      3.917117

      1
      5
      54
      7.171411
      7
      9
      0
      1
      3.330161
      0.732487
      1.0
      11.585706

      2
      3
      33
      5.351920
      6
      9
      0
      1
      3.036203
      1.130937
      1.0
      24.675960

      3
      2
      34
      6.723551
      0
      8
      0
      1
      7.911926
      0.929197
      1.0
      6.361776

      4
      4
      30
      2.448247
      5
      8
      1
      0
      7.148967
      0.533527
      0.8
      12.624123

```python
# Define estimator inputs
train_data["log_demand"]=np.log(train_data["demand"])
train_data["log_price"]=np.log(train_data["price"])

X = train_data[["income"]]  # features
```

```python
# Get test data
X_test = np.linspace(0, 5, 100).reshape(-1, 1)
X_test_data = pd.DataFrame(X_test, columns=["income"])
```

# Create Causal Model and Identify Causal Effect with DoWhy

We define the causal assumptions with DoWhy. For example, we can include features we believe as confounders and features we think will influence the heterogeneity of the effect. With these assumptions defined, DoWhy can generate a causal graph for us, and use that graph to first identify the causal effect.

```python
common_causes=["account_age","age","avg_hours","days_visited","friends_count","has_membership","is_US","songs_purchased","income"]
```

```python
model = CausalModel(
    data=train_data,
    treatment='log_price',
    outcome='log_demand',
    common_causes=common_causes,
    effect_modifiers='income'
)
```

    WARNING:dowhy.causal_model:Causal Graph not provided. DoWhy will construct a graph based on data inputs.
    INFO:dowhy.causal_graph:If this is observed data (not from a randomized experiment), there might always be missing confounders. Adding a node named "Unobserved Confounders" to reflect this.
    INFO:dowhy.causal_model:Model to find the causal effect of treatment ['log_price'] on outcome ['log_demand']

```python
# Visualize causal graph
try:
    # Try pretty printing the graph. Requires pydot and pygraphviz
    display(
        Image(to_pydot(model._graph._graph).create_png())
    )
except:
    # Fall back on default graph view
    model.view_model()
```

```python
identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)
print(identified_estimand)
```

    INFO:dowhy.causal_identifier:Common causes of treatment and outcome:['income', 'songs_purchased', 'U', 'avg_hours', 'is_US', 'age', 'days_visited', 'friends_count', 'account_age', 'has_membership']
    WARNING:dowhy.causal_identifier:If this is observed data (not from a randomized experiment), there might always be missing confounders. Causal effect cannot be identified perfectly.
    INFO:dowhy.causal_identifier:Continuing by ignoring these unobserved confounders because proceed_when_unidentifiable flag is True.
    INFO:dowhy.causal_identifier:Instrumental variables for treatment and outcome:[]

    Estimand type: nonparametric-ate
    ### Estimand : 1
    Estimand name: backdoor
    Estimand expression:
         d
    ────────────(Expectation(log_demand|income,songs_purchased,avg_hours,is_US,age
    d[log_price]

    ,days_visited,friends_count,account_age,has_membership))

    Estimand assumption 1, Unconfoundedness: If U→{log_price} and U→log_demand then P(log_demand|log_price,income,songs_purchased,avg_hours,is_US,age,days_visited,friends_count,account_age,has_membership,U) = P(log_demand|log_price,income,songs_purchased,avg_hours,is_US,age,days_visited,friends_count,account_age,has_membership)
    ### Estimand : 2
    Estimand name: iv
    No such variable found!

# Get Causal Effects with EconML

Based on the identified causal effect above, we fit the model as follows using EconML:

\begin{align}
log(Y) & = \theta(X) \cdot log(T) + f(X,W) + \epsilon \\
log(T) & = g(X,W) + \eta
\end{align}

where $\epsilon, \eta$ are uncorrelated error terms.

The models we fit here aren't an exact match for the data generation function above, but if they are a good approximation, they will allow us to create a good discount policy.  Although the model is misspecified, we hope to see that our `DML` based estimators can still capture the right trend of $\theta(X)$ and that the recommended policy beats other baseline policies (such as always giving a discount) on revenue.  Because of the mismatch between the data generating process and the model we're fitting, there isn't a single true $\theta(X)$ (the true elasticity varies with not only X but also T and W), but given how we generate the data above, we can still calculate the range of true $\theta(X)$ to compare against.

```python
# Define underlying treatment effect function given DGP
def gamma_fn(X):
    return -3 - 14 * (X["income"] < 1)

def beta_fn(X):
    return 20 + 0.5 * (X["avg_hours"]) + 5 * (X["days_visited"] > 4)

def demand_fn(data, T):
    Y = gamma_fn(data) * T + beta_fn(data)
    return Y

def true_te(x, n, stats):
    if x < 1:
        subdata = train_data[train_data["income"] < 1].sample(n=n, replace=True)
    else:
        subdata = train_data[train_data["income"] >= 1].sample(n=n, replace=True)
    te_array = subdata["price"] * gamma_fn(subdata) / (subdata["demand"])
    if stats == "mean":
        return np.mean(te_array)
    elif stats == "median":
        return np.median(te_array)
    elif isinstance(stats, int):
        return np.percentile(te_array, stats)
```

```python
# Get the estimate and range of true treatment effect
truth_te_estimate = np.apply_along_axis(true_te, 1, X_test, 1000, "mean")  # estimate
truth_te_upper = np.apply_along_axis(true_te, 1, X_test, 1000, 95)  # upper level
truth_te_lower = np.apply_along_axis(true_te, 1, X_test, 1000, 5)  # lower level
```

    INFO:numexpr.utils:NumExpr defaulting to 8 threads.

## Parametric heterogeneity
First of all, we can try to learn a **linear projection of the treatment effect** assuming a polynomial form of $\theta(X)$. We use the `LinearDML` estimator. Since we don't have any priors on these models, we use a generic gradient boosting tree estimators to learn the expected price and demand from the data.

```python
lineardml_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.econml.dml.LinearDML",
    control_value=0,
    treatment_value=1,
    target_units=X_test_data,
    method_params={
        "init_params": {"model_y": GradientBoostingRegressor(), "model_t": GradientBoostingRegressor(),"featurizer":PolynomialFeatures(degree=2, include_bias=False)},
        "fit_params": {"inference": "statsmodels"},
    },
)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income

```python
est = lineardml_estimate._estimator_object
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test).flatten()
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0].flatten(),
    te_pred_interval[1].flatten(),
    alpha=0.2,
    label="90% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

From the plot above, it's clear to see that the true treatment effect is a **nonlinear** function of income, with elasticity around -1.75 when income is smaller than 1 and a small negative value when income is larger than 1. The model fits a quadratic treatment effect, which is not a great fit. But it still captures the overall trend: the elasticity is negative and people are less sensitive to the price change if they have higher income.

```python
# Get the final coefficient and intercept summary
est.summary(feat_name=X.columns)
```

Coefficient Results

      	     point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  income	        2.314	      0.089	 26.112	    0.0	    2.169	    2.46

  income^2	     -0.396	      0.031	 -12.907	   0.0	   -0.446	   -0.345

Intercept Results

      	      point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  intercept	     -2.957	      0.053	 -55.622	   0.0	   -3.045	    -2.87

`LinearDML` estimator can also return the summary of the coefficients and intercept for the final model, including point estimates, p-values and confidence intervals. From the table above, we notice that $income$ has positive effect and ${income}^2$ has negative effect, and both of them are statistically significant.

## Nonparametric Heterogeneity
Since we already know the true treatment effect function is nonlinear, let us fit another model using `ForestDML`, which assumes a fully **nonparametric estimation of the treatment effect**.

```python
forestdml_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.econml.dml.ForestDML",
    control_value=0,
    treatment_value=1,
    target_units=X_test_data.values,
    method_params={
        "init_params": {"model_y": GradientBoostingRegressor(), "model_t": GradientBoostingRegressor()},
        "fit_params": {"inference": "blb"},
    },
)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income

```python
est = forestdml_estimate._estimator_object
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test).flatten()
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0].flatten(),
    te_pred_interval[1].flatten(),
    alpha=0.2,
    label="90% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

We notice that this model fits much better than the `LinearDML`, the 90% confidence interval correctly covers the true treatment effect estimate and captures the variation when income is around 1. Overall, the model shows that people with low income are much more sensitive to the price changes than higher income people.

# Test Estimate Robustness with DoWhy

### Add Random Common Cause

How robust are our estimates to adding another confounder? We use DoWhy to test this!

```python
res_random = model.refute_estimate(
    identified_estimand, forestdml_estimate, method_name="random_common_cause"
)
print(res_random)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership+w_random | income

    Refute: Add a Random Common Cause
    Estimated effect:-0.9581697809424333
    New effect:-0.9506022074081668

### Add Unobserved Common Cause

How robust are our estimates to unobserved confounders? Since we assume the model is under unconfoundedness, adding an unobserved confounder might bias the estimates. We use DoWhy to test this!

```python
res_unobserved = model.refute_estimate(
    identified_estimand,
    forestdml_estimate,
    method_name="add_unobserved_common_cause",
    confounders_effect_on_treatment="linear",
    confounders_effect_on_outcome="linear",
    effect_strength_on_treatment=0.1,
    effect_strength_on_outcome=0.1,
)
print(res_unobserved)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income

    Refute: Add an Unobserved Common Cause
    Estimated effect:-0.9581697809424333
    New effect:0.19006356925218587

### Replace Treatment with a Random (Placebo) Variable

What happens our estimates if we replace the treatment variable with noise? Ideally, the average effect would be wildly different than our original estimate. We use DoWhy to investigate!

```python
res_placebo = model.refute_estimate(
    identified_estimand, forestdml_estimate,
    method_name="placebo_treatment_refuter", placebo_type="permute",
    num_simulations=3
)
print(res_placebo)
```

    INFO:dowhy.causal_refuters.placebo_treatment_refuter:Refutation over 3 simulated datasets of permute treatment
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    WARNING:dowhy.causal_refuters.placebo_treatment_refuter:We assume a Normal Distribution as the sample has less than 100 examples.
                     Note: The underlying distribution may not be Normal. We assume that it approaches normal with the increase in sample size.

    Refute: Use a Placebo Treatment
    Estimated effect:-0.9581697809424333
    New effect:-0.010447750440949649
    p value:0.19591286135361208

### Remove a Random Subset of the Data

Do we recover similar estimates on subsets of the data? This speaks to the ability of our chosen estimator to generalize well. We use DoWhy to investigate this!

```python
res_subset = model.refute_estimate(
    identified_estimand, forestdml_estimate,
    method_name="data_subset_refuter", subset_fraction=0.8,
    num_simulations=3)
print(res_subset)
```

    INFO:dowhy.causal_refuters.data_subset_refuter:Refutation over 0.8 simulated datasets of size 8000.0 each
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+income+songs_purchased+avg_hours+is_US+age+days_visited+friends_count+account_age+has_membership | income
    WARNING:dowhy.causal_refuters.data_subset_refuter:We assume a Normal Distribution as the sample has less than 100 examples.
                     Note: The underlying distribution may not be Normal. We assume that it approaches normal with the increase in sample size.

    Refute: Use a subset of data
    Estimated effect:-0.9581697809424333
    New effect:-0.9590948839172434
    p value:0.4093392013624114

# Understand Treatment Effects with EconML
EconML includes interpretability tools to better understand treatment effects. Treatment effects can be complex, but oftentimes we are interested in simple rules that can differentiate between users who respond positively, users who remain neutral and users who respond negatively to the proposed changes.

The EconML `SingleTreeCateInterpreter` provides interperetability by training a single decision tree on the treatment effects outputted by the any of the EconML estimators. In the figure below we can see in dark red users respond strongly to the discount and the in white users respond lightly to the discount.

```python
intrp = SingleTreeCateInterpreter(include_model_uncertainty=True, max_depth=2, min_samples_leaf=10)
intrp.interpret(est, X_test)
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, fontsize=12)
```

# Make Policy Decision with EconML
We want to make policy decisions to maximum the **revenue** instead of the demand. In this scenario,

\begin{align}
Rev & = Y \cdot T \\
    & = \exp^{log(Y)} \cdot T\\
    & = \exp^{(\theta(X) \cdot log(T) + f(X,W) + \epsilon)} \cdot T \\
    & = \exp^{(f(X,W) + \epsilon)} \cdot T^{(\theta(X)+1)}
\end{align}

With the decrease of price, revenue will increase only if $\theta(X)+1<0$. Thus, we set `sample_treatment_cast=-1` here to learn **what kinds of customers we should give a small discount to maximum the revenue**.

The EconML library includes policy interpretability tools such as `SingleTreePolicyInterpreter` that take in a treatment cost and the treatment effects to learn simple rules about which customers to target profitably. In the figure below we can see the model recommends to give discount for people with income less than $0.985$ and give original price for the others.

```python
intrp = SingleTreePolicyInterpreter(risk_level=0.05, max_depth=2, min_samples_leaf=1, min_impurity_decrease=0.001)
intrp.interpret(est, X_test, sample_treatment_costs=-1, treatment_names=["Discount", "No-Discount"])
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, fontsize=12)
```

Now, let us compare our policy with other baseline policies! Our model says which customers to give a small discount to, and for this experiment, we will set a discount level of 10% for those users. Because the model is misspecified we would not expect good results with large discounts. Here, because we know the ground truth, we can evaluate the value of this policy.

```python
# define function to compute revenue
def revenue_fn(data, discount_level1, discount_level2, baseline_T, policy):
    policy_price = baseline_T * (1 - discount_level1) * policy + baseline_T * (1 - discount_level2) * (1 - policy)
    demand = demand_fn(data, policy_price)
    rev = demand * policy_price
    return rev
```

```python
policy_dic = {}
# our policy above
policy = intrp.treat(X)
policy_dic["Our Policy"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, policy))

## previous strategy
policy_dic["Previous Strategy"] = np.mean(train_data["price"] * train_data["demand"])

## give everyone discount
policy_dic["Give Everyone Discount"] = np.mean(revenue_fn(train_data, 0.1, 0, 1, np.ones(len(X))))

## don't give discount
policy_dic["Give No One Discount"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, np.ones(len(X))))

## follow our policy, but give -10% discount for the group doesn't recommend to give discount
policy_dic["Our Policy + Give Negative Discount for No-Discount Group"] = np.mean(revenue_fn(train_data, -0.1, 0.1, 1, policy))

## give everyone -10% discount
policy_dic["Give Everyone Negative Discount"] = np.mean(revenue_fn(train_data, -0.1, 0, 1, np.ones(len(X))))
```

```python
# get policy summary table
res = pd.DataFrame.from_dict(policy_dic, orient="index", columns=["Revenue"])
res["Rank"] = res["Revenue"].rank(ascending=False)
res
```

      Revenue
      Rank

      Our Policy
      14.686241
      2.0

      Previous Strategy
      14.349342
      4.0

      Give Everyone Discount
      13.774469
      6.0

      Give No One Discount
      14.294606
      5.0

      Our Policy + Give Negative Discount for No-Discount Group
      15.564411
      1.0

      Give Everyone Negative Discount
      14.612670
      3.0

**We beat the baseline policies!** Our policy gets the highest revenue except for the one raising the price for the No-Discount group. That means our currently baseline price is low, but the way we segment the user does help increase the revenue!

# Conclusions

In this notebook, we have demonstrated the power of using EconML and DoWhy to:

* Estimate the treatment effect correctly even the model is misspecified
* Test causal assumptions and investigate the robustness of the resulting estimates
* Interpret the resulting individual-level treatment effects
* Make the policy decision beats the previous and baseline policies

To learn more about what EconML can do for you, visit our [website](https://aka.ms/econml), our [GitHub page](https://github.com/microsoft/EconML) or our [docummentation](https://econml.azurewebsites.net/).

To learn more about what DoWhy can do for you, visit the [GitHub page](https://github.com/microsoft/dowhy) or [documentation](https://microsoft.github.io/dowhy/index.html).

notebooks/CustomerScenarios/Case Study - Customer Segmentation at An Online Media Company - EconML + DoWhy.ipynb


### 2.7.1 Pure fluid solutions
Whilst the variables required for each pure-fluid solubility model are different, we take a very similar approach to calculating fluid properties for each. In general pure-fluid models provide analytical expressions for the concentration of a volatile species dissolved in a magma as a function of either its partial-pressure or fugacity in the vapour phase. To provide a homogeneous framework, each model must have a fugacity model associated with it. Where a solubility model uses partial-pressure in place of fugacity, the fugacity model is set to being an ideal gas.

In general, an analytical expression does not exist for the inverse calculation, i.e., calculating the saturation pressure given the concentration of dissolved volatile species. The calculation must then be solved numerically. Numerical methods obtain the answer by making guesses for the saturation pressure, checking whether the predicted concentration of the dissolved volatile species matches the measured value, and refining the guesses as appropriate. When the saturation pressure is found, this equality holds:

\begin{equation}
 V^L_\textrm{measured} - V^L_\textrm{predicted}(P,...) = 0
\end{equation}

When the guessed pressure is too high the left hand side of the equation will be negative, and if the guessed pressure is too low it will be positive, We therefore require an algorithm to find the root of the equation. To this end we employ the scipy.optimize.root_scalar() method. The advantage of using a generic scipy algorithm, in preference to creating a bespoke routine, is that it affords us flexibility in how the problem is solved, and the algorithm is underlain by robust, well-tested, efficient code.

### 2.7.2 Finding saturation pressure of mixed-fluid
Describe how scipy.optimize is used here

### 2.7.3 Finding the equilibrium fluid composition
When calculating the dissolved volatile concentrations in a progressively degassing magma, we break the calculation into steps, each with a small decrement in pressure. As pressure decreases the mass of fluid in equilibrium with the magma increases and its composition will change. At each stage we must, therefore, solve for the mass and composition of the fluid in equilibrium with the magma, given the total mass of volatiles in the system. This calculation is implemented in MixedFluid.calculate_equilibrium_fluid_comp(), which in turn is called by MixedFluid.calculate_degassing_path(). Here we summarise the methodology for the calculation.

Mass balance in the system can be expressed in terms of the mole fractions of H$_2$O ($X_{\textrm{H}_2 \textrm{O}}$) and CO$_2$ ($X_{\textrm{CO}_2}$) in the silicate-liquid ($L$), the H$_2$O-CO$_2$O fluid ($F$), and the system total ($T$), and the fraction of fluid present ($f$).

\begin{equation}
X^F_{\textrm{CO}_2} + X^F_{\textrm{H}_2 \textrm{O}} = 1
\end{equation}

\begin{equation}
(1-f)X^L_{\textrm{CO}_2} + f X^F_{\textrm{CO}_2} = X^T_{\textrm{CO}_2}
\end{equation}

\begin{equation}
(1-f)X^L_{\textrm{H}_2 \textrm{O}} + f X^F_{\textrm{H}_2 \textrm{O}} = X^T_{\textrm{H}_2 \textrm{O}}
\end{equation}

The fraction of fluid ($f$) can be eliminated by substitution, obtaining Equation --NUMBER-- of --CITATION--:

\begin{equation}
\frac{X^T_{\textrm{CO}_2}-X^L_{\textrm{CO}_2}}{X^F_{\textrm{CO}_2}-X^L_{\textrm{CO}_2}} - \frac{X^T_{\textrm{H}_2\textrm{O}}-X^L_{\textrm{H}_2\textrm{O}}}{X^F_{\textrm{H}_2\textrm{O}}-X^L_{\textrm{H}_2\textrm{O}}} = 0
\end{equation}

In principle, this equation allows us to numerically solve for mass balance; however, the equation is valid only for both $X^F_{\textrm{CO}_2} \neq X^L_{\textrm{CO}_2}$ and $X^F_{\textrm{H}_2 \textrm{O}} \neq X^L_{\textrm{H}_2 \textrm{O}}$. When $X^F_{\textrm{CO}_2} \rightarrow X^L_{\textrm{CO}_2}$ or $X^F_{\textrm{H}_2 \textrm{O}} \rightarrow X^L_{\textrm{H}_2 \textrm{O}}$ the equation tends to $\pm \infty$, which the scipy.optimize.root_scalar() routine will mistake for a root of the equation. The regions over which the expression is not infinite can be found; however we find a more reliable approach when using the scipy.optimize library is to use the equation in a different form.

Rearranging the CO$_2$ mass-balance expression allows the fraction of fluid ($f$) to be calculated:

\begin{equation}
f = \frac{X^T_{\textrm{CO}_2}-X^L_{\textrm{CO}_2}}{X^F_{\textrm{CO}_2}-X^L_{\textrm{CO}_2}}
\end{equation}

This is valid for $X^F_{\textrm{CO}_2} \neq X^L_{\textrm{CO}_2}$. It is much more unlikely that a solution will exist near $X^F_{\textrm{CO}_2} = X^L_{\textrm{CO}_2}$ than near $X^F_{\textrm{H}_2 \textrm{O}} = X^L_{\textrm{H}_2 \textrm{O}}$. By not directly eliminating $f$ by dividing through by both ($X^F_{\textrm{CO}_2} - X^L_{\textrm{CO}_2}$) and ($X^F_{\textrm{H}_2 \textrm{O}} - X^L_{\textrm{H}_2 \textrm{O}}$)  we significantly increase the reliability of using this expression with scipy.optimize.root_scalar(). In this expression, the only known value is $X^T_{\textrm{CO}_2}$. However, if a guess is provided for $X^F_{\textrm{CO}_2}$, the partial pressure of CO$_2$ can be calculated and the corresponding value for $X^L_{\textrm{CO}_2}$ obtained. In general the guess for $X^F_{\textrm{CO}_2}$ will not satisfy mass balance when the same parameters are applied to both CO$_2$ and H$_2$O.

Since $X^F_{\textrm{H}_2 \textrm{O}} = 1 - X^F_{\textrm{CO}_2}$, when we provide a guess for $X^F_{\textrm{CO}_2}$ we also fix $X^F_{\textrm{H}_2 \textrm{O}}$. This in turn fixes the partial pressure of H$_2$O, allowing us to calculate $X^L_{\textrm{H}_2 \textrm{O}}$ consistent with that guess. We can then test whether our guess for $X^F_{\textrm{CO}_2}$ satisfies mass balance by checking for the equality:

\begin{equation}
(1-f)X^L_{\textrm{H}_2 \textrm{O}} + f X^F_{\textrm{H}_2 \textrm{O}} - X^T_{\textrm{H}_2 \textrm{O}} = 0
\end{equation}

The value of $X^F_{\textrm{CO}_2}$ that uniquely satisfies mass balance can then be found numerically by solving this equation using scipy.optimize.root_scalar().

```python

```