# Start-to-Finish Example: `GiRaFFE_NRPy` 1D tests

### Authors: Terrence Pierre Jacques

### Adapted from [Start-to-Finish Example: Head-On Black Hole Collision](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb)

## This module compiles and runs code tests for all 1D initial data options available in GiRaFFE-NRPy+, evolving one-dimensional GRFFE waves.

### NRPy+ Source Code for this module:

* Main python module for all 1D initial data: [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_1D_tests.py](../../edit/in_progress/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_1D_tests.py) __Options:__
    1. [Fast Wave](Tutorial-GiRaFFEfood_NRPy_1D_tests-fast_wave.ipynb)
    1. [Alfven Wave](Tutorial-GiRaFFEfood_NRPy_1D_alfven_wave.ipynb)
    1. [Degenerate Alfven Wave](Tutorial-GiRaFFEfood_NRPy_1D_tests-degen_Alfven_wave.ipynb)
    1. [Three Alfven Waves](Tutorial-GiRaFFEfood_NRPy_1D_tests-three_waves.ipynb)
    1. [FFE Breakdown](Tutorial-GiRaFFEfood_NRPy_1D_tests-FFE_breakdown.ipynb)

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Set core NRPy+ parameters for numerical grids
1. [Step 2](#grffe): Output C code for GRFFE evolution
    1. [Step 2.a](#mol): Output macros for Method of Lines timestepping
1. [Step 3](#gf_id): Import `GiRaFFEfood_NRPy` initial data modules
1. [Step 4](#cparams): Output C codes needed for declaring and setting Cparameters; also set `free_parameters.h`
1. [Step 5](#mainc): `GiRaFFE_NRPy_standalone.c`: The Main C Code
1. [Step 6](#compileexec): Compile and execute C codes
1. [Step 7](#plots): Data Visualization
1. [Step 8](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Set up core functions and parameters for solving  GRFFE equations \[Back to [top](#toc)\]
$$\label{setup}$$

```python
import os, sys           # Standard Python modules for multiplatform OS-level functions
# First, we'll add the parent directory to the list of directories Python will check for modules.
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Import needed Python modules
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm   # NRPy+: Reference metric support
import sympy as sp                # SymPy: The Python computer algebra package upon which NRPy+ depends

#Step 0: Set the spatial dimension parameter to 3.
par.set_parval_from_str("grid::DIM", 3)
DIM = par.parval_from_str("grid::DIM")

# TINYDOUBLE = par.Cparameters("REAL", "TINYDOUBLE", 1e-100)

# Choices are: Spherical, SinhSpherical, SinhSphericalv2, Cylindrical, SinhCylindrical,
#              SymTP, SinhSymTP
dst_basis = "Cartesian"

# Set coordinate system to dst_basis
par.set_parval_from_str("reference_metric::CoordSystem",dst_basis)
rfm.reference_metric()
```

# Step 3: Import `GiRaFFEfood_NRPy` initial data modules \[Back to [top](#toc)\]
$$\label{gf_id}$$

With the preliminaries out of the way, we will write the C functions to set up initial data. There are two categories of initial data that must be set: the spacetime metric variables, and the GRFFE plasma variables. We will set up the spacetime first, namely the Minkowski spacetime.

Now, we will write out the initials data function for the GRFFE variables.

```python
ID_opts = ["AlfvenWave", "ThreeAlfvenWaves", "DegenAlfvenWave", "FastWave", "FFEBD"]

# for initial_data in ID_opts:
initial_data = "FFEBD"

if initial_data=="AlfvenWave":
    import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_1D_tests as gid
    gid.GiRaFFEfood_NRPy_1D_tests(stagger = True)
    desc = "Generate Alfven wave 1D initial data for GiRaFFEfood_NRPy."
elif initial_data=="ThreeAlfvenWaves":
    import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_1D_tests_three_waves as gid
    gid.GiRaFFEfood_NRPy_1D_tests_three_waves(stagger = True)
    desc = "Generate three Alfven wave 1D initial data for GiRaFFEfood_NRPy."
elif initial_data=="DegenAlfvenWave":
    import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_1D_tests_degen_Alfven_wave as gid
    gid.GiRaFFEfood_NRPy_1D_tests_degen_Alfven_wave(stagger = True)
    desc = "Generate degenerate Alfven wave 1D initial data for GiRaFFEfood_NRPy."
elif initial_data=="FastWave":
    import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_1D_tests_fast_wave as gid
    gid.GiRaFFEfood_NRPy_1D_tests_fast_wave(stagger = True)
    desc = "Generate fast wave 1D initial data for GiRaFFEfood_NRPy."
elif initial_data=="FFEBD":
    import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_1D_tests_FFE_breakdown as gid
    gid.GiRaFFEfood_NRPy_1D_tests_FFE_breakdown(stagger = True)
    desc = "Generate FFE breakdown 1D initial data for GiRaFFEfood_NRPy."

```

We define Jacobians relative to the center of the destination grid, at a point $x^j_{\rm dst}=$(`xx0,xx1,xx2`)${}_{\rm dst}$ on the destination grid:
$$
{\rm Jac\_dUCart\_dDdstUD[i][j]} = \frac{\partial x^i_{\rm Cart}}{\partial x^j_{\rm dst}},
$$

via exact differentiation (courtesy SymPy), and the inverse Jacobian
$$
{\rm Jac\_dUdst\_dDCartUD[i][j]} = \frac{\partial x^i_{\rm dst}}{\partial x^j_{\rm Cart}},
$$

using NRPy+'s `generic_matrix_inverter3x3()` function. In terms of these, the transformation of BSSN tensors from Cartesian to the destination grid's `"reference_metric::CoordSystem"` coordinates may be written:

$$
B^i_{\rm dst} = \frac{\partial x^i_{\rm dst}}{\partial x^\ell_{\rm Cart}} B^\ell_{\rm Cart},
$$

while for lowered indices we have

$$
A^{\rm dst}_{i} =
\frac{\partial x^\ell_{\rm Cart}}{\partial x^i_{\rm dst}} A^{\rm Cart}_{\ell}\\
$$

```python
# Step 3: Transform BSSN tensors in Cartesian basis to destination grid basis, using center of dest. grid as origin

# Step 3.a: Next construct Jacobian and inverse Jacobian matrices:
Jac_dUCart_dDrfmUD,Jac_dUrfm_dDCartUD = rfm.compute_Jacobian_and_inverseJacobian_tofrom_Cartesian()

# Step 3.b: Convert basis of all BSSN *vectors* from Cartesian to destination basis
BU_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUrfm_dDCartUD, gid.BU)

ValenciavU_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUrfm_dDCartUD, gid.ValenciavU)

# Note that the below the function should really be "...basis_transform_vectorUDfrom_Cartesian_to_rfmbasis.."
AD_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUCart_dDrfmUD, gid.AD)
```

```python
print("Initial data type = "+ initial_data)
for i in range(DIM):
    print(gid.ValenciavU[i] - ValenciavU_dst[i])
    print(gid.BU[i] - BU_dst[i])
    print(gid.AD[i] - AD_dst[i])
```

    Initial data type = FFEBD
    0
    0
    0
    0
    0
    0
    0
    0
    0

```python
# Choices are: Spherical, SinhSpherical, SinhSphericalv2, Cylindrical, SinhCylindrical,
#              SymTP, SinhSymTP
dst_basis = "SymTP"

# Set coordinate system to dst_basis
par.set_parval_from_str("reference_metric::CoordSystem",dst_basis)
rfm.reference_metric()

# Step 3: Transform BSSN tensors in Cartesian basis to destination grid basis, using center of dest. grid as origin

# Step 3.a: Next construct Jacobian and inverse Jacobian matrices:
Jac_dUCart_dDrfmUD,Jac_dUrfm_dDCartUD = rfm.compute_Jacobian_and_inverseJacobian_tofrom_Cartesian()

# Step 3.b: Convert basis of all BSSN *vectors* from Cartesian to destination basis
BU_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUrfm_dDCartUD, gid.BU)

ValenciavU_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUrfm_dDCartUD, gid.ValenciavU)

# Note that the below the function should really be "...basis_transform_vectorUDfrom_Cartesian_to_rfmbasis.."
AD_dst = rfm.basis_transform_vectorU_from_Cartesian_to_rfmbasis(Jac_dUCart_dDrfmUD, gid.AD)
```

```python
import GiRaFFEfood_NRPy.BasisTransform as BT
BT.basis_transform(dst_basis, gid.AD, gid.ValenciavU, gid.BU)

def consistency_check(quantity1,quantity2,string):
    if quantity1-quantity2==0:
        print(string+" is in agreement!")
    else:
        print(string+" does not agree!")
        sys.exit(1)

print("Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:")

for i in range(3):
    consistency_check(ValenciavU_dst[i],BT.ValenciavU_dst[i],"ValenciavU"+str(i))
    consistency_check(AD_dst[i],BT.AD_dst[i],"AD"+str(i))
    consistency_check(BU_dst[i],BT.BU_dst[i],"BU"+str(i))
```

    Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:
    ValenciavU0 is in agreement!
    AD0 is in agreement!
    BU0 is in agreement!
    ValenciavU1 is in agreement!
    AD1 is in agreement!
    BU1 is in agreement!
    ValenciavU2 is in agreement!
    AD2 is in agreement!
    BU2 is in agreement!

# Step 8: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Start_to_Finish-GiRaFFE_NRPy-1D_tests-staggered.pdf](Tutorial-Start_to_Finish-GiRaFFE_NRPy-1D_tests-staggered.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-Start_to_Finish-GiRaFFE_NRPy-1D_tests-staggered",location_of_template_file=os.path.join(".."))
```

    Created Tutorial-Start_to_Finish-GiRaFFE_NRPy-1D_tests-staggered.tex, and
        compiled LaTeX file to PDF file Tutorial-Start_to_Finish-
        GiRaFFE_NRPy-1D_tests-staggered.pdf


# Batch train models

# Purpose
Train a batch of models on the same dataset, same train/test and cross validation

# Methodology
* Run the [03.1_train_model.ipynb](03.1_train_model.ipynb) for each model.
* model, model score etc. is logged with [mlflow](https://mlflow.org/).

# Setup

```python
# %load imports.py
from typing import no_type_check
%matplotlib inline
%load_ext autoreload
%autoreload 2

import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sympy as sp

```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
import papermill as pm

import sys
from src.logger import logger
import src.mlflow_utils
```

## Batch run

```python
def create_output_path(parameters, artifact_dir = 'artifacts'):

    if not os.path.exists(artifact_dir):
        os.mkdir(artifact_dir)

    rolling_str = 'rolling' if parameters['rolling_cv'] else 'random'
    output_name = create_run_name(parameters=parameters)
    output_path = os.path.join(artifact_dir,f'{output_name}.ipynb')
    return output_path

def create_run_name(parameters):

    rolling_str = 'rolling' if parameters['rolling_cv'] else 'random'
    name = f'{parameters["model_name"]}_{rolling_str}_{parameters["test_size"]}'
    return name

```

```python
artifact_dir = 'artifacts'

parameters = {
    'experiment' : 'rolling_fit',
    'model_name' : 'polynomial',
    'data_path' : '../../data/processed/data_extended.csv',
    'n_splits' : 6,
    'n_repeats' :1,
    'rolling':True,
    'test_size':0.2,
    'input_path':r'pipeline/03.1_train_model.ipynb',
}

models = {

    'polynomial' : {
        'input_path':r'pipeline/03.1_train_model.ipynb',
    },

    'SVR' : {
        'input_path':r'pipeline/03.3_train_SVR.ipynb',
    },

    'XGBoost' : {
        'input_path':r'pipeline/03.2_train_xgboost.ipynb',
    },

    'ridge' : {
        'input_path':r'pipeline/03.4_train_ridge.ipynb',
    },

    'lasso' : {
        'input_path':r'pipeline/03.5_train_lasso.ipynb',
    },

}

df_parameters = pd.DataFrame(columns=parameters.keys())

for model_name, model in models.items():

    for test_size in [0.1,0.2,0.3]:

        for rolling_cv in [True,False]:

            s = pd.Series(parameters)

            if not rolling:
                s['experiment'] = 'random_fit'

            s['rolling_cv']=rolling_cv
            s['rolling_test']=True
            s['test_size'] = test_size

            s.update(model)
            s['model_name'] = model_name
            s['run_name'] = create_run_name(parameters=s)

            s['output_path'] = create_output_path(parameters=s, artifact_dir = artifact_dir)

            df_parameters = df_parameters.append(s, ignore_index=True)

df_parameters
```

      experiment
      model_name
      data_path
      n_splits
      n_repeats
      rolling
      test_size
      input_path
      output_path
      rolling_cv
      rolling_test
      run_name

      0
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_rolling_0.1.ipynb
      1.0
      1.0
      polynomial_rolling_0.1

      1
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_random_0.1.ipynb
      0.0
      1.0
      polynomial_random_0.1

      2
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_rolling_0.2.ipynb
      1.0
      1.0
      polynomial_rolling_0.2

      3
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_random_0.2.ipynb
      0.0
      1.0
      polynomial_random_0.2

      4
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_rolling_0.3.ipynb
      1.0
      1.0
      polynomial_rolling_0.3

      5
      random_fit
      polynomial
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.1_train_model.ipynb
      artifacts\polynomial_random_0.3.ipynb
      0.0
      1.0
      polynomial_random_0.3

      6
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_rolling_0.1.ipynb
      1.0
      1.0
      SVR_rolling_0.1

      7
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_random_0.1.ipynb
      0.0
      1.0
      SVR_random_0.1

      8
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_rolling_0.2.ipynb
      1.0
      1.0
      SVR_rolling_0.2

      9
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_random_0.2.ipynb
      0.0
      1.0
      SVR_random_0.2

      10
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_rolling_0.3.ipynb
      1.0
      1.0
      SVR_rolling_0.3

      11
      random_fit
      SVR
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.3_train_SVR.ipynb
      artifacts\SVR_random_0.3.ipynb
      0.0
      1.0
      SVR_random_0.3

      12
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_rolling_0.1.ipynb
      1.0
      1.0
      XGBoost_rolling_0.1

      13
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_random_0.1.ipynb
      0.0
      1.0
      XGBoost_random_0.1

      14
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_rolling_0.2.ipynb
      1.0
      1.0
      XGBoost_rolling_0.2

      15
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_random_0.2.ipynb
      0.0
      1.0
      XGBoost_random_0.2

      16
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_rolling_0.3.ipynb
      1.0
      1.0
      XGBoost_rolling_0.3

      17
      random_fit
      XGBoost
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.2_train_xgboost.ipynb
      artifacts\XGBoost_random_0.3.ipynb
      0.0
      1.0
      XGBoost_random_0.3

      18
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_rolling_0.1.ipynb
      1.0
      1.0
      ridge_rolling_0.1

      19
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_random_0.1.ipynb
      0.0
      1.0
      ridge_random_0.1

      20
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_rolling_0.2.ipynb
      1.0
      1.0
      ridge_rolling_0.2

      21
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_random_0.2.ipynb
      0.0
      1.0
      ridge_random_0.2

      22
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_rolling_0.3.ipynb
      1.0
      1.0
      ridge_rolling_0.3

      23
      random_fit
      ridge
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.4_train_ridge.ipynb
      artifacts\ridge_random_0.3.ipynb
      0.0
      1.0
      ridge_random_0.3

      24
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_rolling_0.1.ipynb
      1.0
      1.0
      lasso_rolling_0.1

      25
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.1
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_random_0.1.ipynb
      0.0
      1.0
      lasso_random_0.1

      26
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_rolling_0.2.ipynb
      1.0
      1.0
      lasso_rolling_0.2

      27
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.2
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_random_0.2.ipynb
      0.0
      1.0
      lasso_random_0.2

      28
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_rolling_0.3.ipynb
      1.0
      1.0
      lasso_rolling_0.3

      29
      random_fit
      lasso
      ../../data/processed/data_extended.csv
      6
      1
      True
      0.3
      pipeline/03.5_train_lasso.ipynb
      artifacts\lasso_random_0.3.ipynb
      0.0
      1.0
      lasso_random_0.3

```python
assert df_parameters['run_name'].is_unique
```

```python
for run_name, parameters in df_parameters.iterrows():

    logger.info(f'Running notebook for: {parameters["run_name"]}')
    output_path = parameters['output_path']

    if os.path.exists(output_path):
        if os.path.getsize(output_path) > 1000:
            logger.info('Already run, skipping...')
            continue

    try:
        pm.execute_notebook(input_path=parameters['input_path'],
                        output_path=output_path,
                        parameters={'run_params':dict(parameters)},
                        cwd=artifact_dir);
    except Exception as e:
        logger.error(e)
        continue
```

    2021-06-14 19:40:25,468 - Running notebook for: polynomial_rolling_0.1

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:40:40,693 - Running notebook for: polynomial_random_0.1

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:40:56,935 - Running notebook for: polynomial_rolling_0.2

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:41:12,113 - Running notebook for: polynomial_random_0.2

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:41:30,679 - Running notebook for: polynomial_rolling_0.3

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:41:50,327 - Running notebook for: polynomial_random_0.3

    Executing:   0%|          | 0/26 [00:00<?, ?cell/s]

    2021-06-14 19:42:05,533 - Running notebook for: SVR_rolling_0.1

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:42:38,555 - Running notebook for: SVR_random_0.1

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:43:23,536 - Running notebook for: SVR_rolling_0.2

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:43:53,873 - Running notebook for: SVR_random_0.2

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:44:33,196 - Running notebook for: SVR_rolling_0.3

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:44:59,452 - Running notebook for: SVR_random_0.3

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 19:45:31,709 - Running notebook for: XGBoost_rolling_0.1

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 20:03:45,048 - Running notebook for: XGBoost_random_0.1

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 20:32:13,530 - Running notebook for: XGBoost_rolling_0.2

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 20:48:19,313 - Running notebook for: XGBoost_random_0.2

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 21:13:47,959 - Running notebook for: XGBoost_rolling_0.3

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 21:28:01,837 - Running notebook for: XGBoost_random_0.3

    Executing:   0%|          | 0/24 [00:00<?, ?cell/s]

    2021-06-14 21:50:35,369 - Running notebook for: ridge_rolling_0.1

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:51:01,735 - Running notebook for: ridge_random_0.1

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:51:31,132 - Running notebook for: ridge_rolling_0.2

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:51:56,458 - Running notebook for: ridge_random_0.2

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:52:25,732 - Running notebook for: ridge_rolling_0.3

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:52:50,073 - Running notebook for: ridge_random_0.3

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:53:17,432 - Running notebook for: lasso_rolling_0.1

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:53:44,490 - Running notebook for: lasso_random_0.1

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:54:16,316 - Running notebook for: lasso_rolling_0.2

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:54:43,663 - Running notebook for: lasso_random_0.2

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:55:13,662 - Running notebook for: lasso_rolling_0.3

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

    2021-06-14 21:55:38,916 - Running notebook for: lasso_random_0.3

    Executing:   0%|          | 0/25 [00:00<?, ?cell/s]

```python
from mlflow.tracking import MlflowClient
import mlflow
```

```python
print(mlflow.get_registry_uri())
```

```python
import mlflow
run = mlflow.get_run(run_id='d44831137a6e40f9b1a938cf61676a92')
```

```python
run.to_dictionary()
```

```python
run.data.metrics
```

```python

```


# Scenario F - Dataset Real (Tablets NIR)

In this scenario the the probabilistic model is tested on real-world datasets. The model parameters differ per dataset, and are set by the user. The probabilistic model is formulated as follows:

\begin{equation}
\large y = f(x) = \sum\limits_{m=1}^M \big[A_m \cdot f_{pseudo-Voigt}(x)\big] + a_0 + a_1 \cdot x + \epsilon
\end{equation}

where:

\begin{equation}
\large f_{pseudo-Voigt}(x) = \eta \cdot \frac{\sigma_m^2}{(x-\mu_m)^2 + \sigma_m^2} + (1 - \eta) \cdot e^{-\frac{(x-\mu_m)^2}{2\cdot\sigma_m^2}}
\end{equation}

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pymc3 as pm
import arviz as az

#az.style.use('arviz-darkgrid')

print('Running on PyMC3 v{}'.format(pm.__version__))
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

    Running on PyMC3 v3.8

## Import local modules

```python
import os
import sys
sys.path.append('../../modules')
import datagen as dg
import models as mdl
import results as res
import figures as fig
import settings as cnf
from data import Data
```

## Local configuration

```python
# output for results and images
out_path      = './output_data_real_tablets'
file_basename = out_path + '/scenario_data_real_tablets'

conf = {}

# scenario name
conf['scenario'] = 'dataset real'

# initialization method for sampler
conf['init_mode'] = 'adapt_diag'

# probabilistic model (priors) (normal/lognormal/eta_rv)
conf['prior_model'] = 'normal'

# provide peak positions to the model as testvalues ('yes'/'no')
conf['peak_info'] = 'yes'

# model mode ('train'/eval')
conf['model_mode'] = 'eval'

# number of cores to run sampling chains on
conf['ncores'] = 2

# number of samples per chain
conf['nsamples'] = 2000
```

```python
# if the output dir does not exist, create it
if not os.path.exists(out_path):
    os.makedirs(out_path)

conf
```

    {'scenario': 'dataset real',
     'init_mode': 'adapt_diag',
     'prior_model': 'normal',
     'peak_info': 'yes',
     'model_mode': 'eval',
     'ncores': 2,
     'nsamples': 2000}

## Save configuration

```python
cnf.save(out_path, conf)
```

# Import data and plot

```python
# dataset filelocation
dataset_filename = '../../../data/tablets/NIR_tablets.csv'

# dataset object init
data_obj = Data(dataset_filename, shuffle='yes')
data_obj.data.head(5)
```

      7398.337
      7406.051
      7413.766
      7421.481
      7429.195
      7436.91
      7444.624
      7452.339
      7460.053
      7467.768
      ...
      10461.02
      10468.73
      10476.45
      10484.16
      10491.88
      10499.59
      10507.3
      content (% w/w)
      label
      scale

      262
      2.136078
      2.060586
      1.989330
      1.921227
      1.854084
      1.789294
      1.734358
      1.688767
      1.644500
      1.600998
      ...
      0.402949
      0.400739
      0.401894
      0.401219
      0.395639
      0.392095
      0.393905
      8.199922
      4
      1

      161
      2.078315
      2.004214
      1.934648
      1.864546
      1.797509
      1.738770
      1.686449
      1.637614
      1.592041
      1.550425
      ...
      0.263836
      0.262712
      0.264048
      0.261855
      0.254674
      0.251174
      0.254293
      8.141403
      3
      0

      264
      2.215751
      2.144743
      2.073390
      1.998496
      1.933801
      1.879729
      1.826225
      1.774429
      1.729536
      1.690336
      ...
      0.492507
      0.491604
      0.492943
      0.489631
      0.482625
      0.480078
      0.484371
      8.134533
      4
      1

      240
      2.369617
      2.324822
      2.254126
      2.184280
      2.124235
      2.056269
      1.987441
      1.938404
      1.900987
      1.858757
      ...
      0.523157
      0.520280
      0.519620
      0.516307
      0.510819
      0.508662
      0.511183
      8.320064
      4
      0

      233
      2.307317
      2.217612
      2.145420
      2.078144
      2.011093
      1.955767
      1.905825
      1.853925
      1.805600
      1.762719
      ...
      0.456231
      0.453186
      0.454264
      0.453350
      0.447606
      0.442439
      0.441739
      8.330028
      4
      0

5 rows × 407 columns

```python
# select single class for inference
sclass = 1
df_cls = data_obj.data.loc[(data_obj.data['label'] == sclass)]
```

```python
# drop non-data columns
df = df_cls.drop(columns=['label','scale','content (% w/w)'])
```

```python
print("total number of spectra      : {0}".format(len(data_obj.data)))
print("total number of x-values     : {0}".format(len(np.array(df.columns.to_list(), dtype='float32'))))
print("total number of classes      : {0}".format(data_obj.classes))
print("selected class for inference : {0}".format(sclass))
print("number of selected spectra   : {0}".format(len(df)))
print("\nclass distribution:")
print(data_obj.data[data_obj.label_column].value_counts())
```

    total number of spectra      : 310
    total number of x-values     : 404
    total number of classes      : 4
    selected class for inference : 1
    number of selected spectra   : 70

    class distribution:
    4    80
    3    80
    2    80
    1    70
    Name: label, dtype: int64

```python
# number of samples to run inference on
nsamples = 15

# set peak information for spectrum
peaks = [7300.0, 7850.0, 8200.0, 8830.0, 9950.0]

# baseline profile
base_shape = 'offset'

# plot dataset (nsamples spectra, classes mixed)
fig.plot_datasets_real(df, peaks, nsamples, savefig='yes', fname=file_basename)
```

```python
# compare samples of different classes
df_lst = []
df_lst.append(df)
df_lst.append(data_obj.data.loc[(data_obj.data['label'] == 2)].drop(columns=['label','scale','content (% w/w)']))
df_lst.append(data_obj.data.loc[(data_obj.data['label'] == 3)].drop(columns=['label','scale','content (% w/w)']))
df_lst.append(data_obj.data.loc[(data_obj.data['label'] == 4)].drop(columns=['label','scale','content (% w/w)']))

# plot dataset (nsamples spectra, classes mixed)
fig.plot_datasets_real_compare(df_lst, nsamples, savefig='yes', fname=file_basename + "_cmp")
```

# Initialize models and run inference

```python
ldata  = []
lpeaks = []

# add dataframes and peakinfo to list (for multiple classes or data per class)
ldata.append(df)
lpeaks.append(peaks)
```

```python
# convert pandas data to numpy arrays
x_val = np.array(df.columns, dtype='float32')

# store dataset y-values in list
cols = ldata[0].columns
y_val = [ldata[i][cols].values[:nsamples] for i in range(len(ldata))]
```

```python
# initialize models and run inference
models = []
traces = []

for i in range(len(ldata)):
    if conf['peak_info'] == 'yes':
        plist = np.array(lpeaks[0], dtype='float32').flatten()
        plist.sort()
        model_g = mdl.model_pvoigt(xvalues=x_val, observations=y_val[i], npeaks=len(peaks),
                              mu_peaks=plist, pmodel=conf['prior_model'], baseline=base_shape)
    else:
        model_g = mdl.model_pvoigt(xvalues=x_val, observations=y_val[i], npeaks=len(peaks),
                                              pmodel=conf['prior_model'], baseline=base_shape)

    models.append(model_g)

    with model_g:
        if conf['model_mode'] == 'train':
            print("running inference on dataset #{0}/{1}".format(i+1,len(ldata)))
            trace_g = pm.sample(conf['nsamples'], init=conf['init_mode'], cores=conf['ncores'])
            traces.append(trace_g)
            # save inference results
            pm.backends.text.dump(out_path + '/traces_%02d' % (i+1), trace_g)
        else:
            # load traces from disk
            print("loading dataset #{0}/{1}".format(i+1,len(ldata)))
            trace_g = pm.backends.text.load(out_path + '/traces_%02d' % (i+1))
            traces.append(trace_g)
```

    loading dataset #1/1

# Model visualization

```python
# display first model
pm.model_to_graphviz(models[0])
```

```python
# save model figure as image
img = pm.model_to_graphviz(models[0])
img.render(filename=file_basename + '_model', format='png');
```

# Collect results and save

```python
# posterior predictive traces
ppc = [pm.sample_posterior_predictive(traces[i], samples=500, model=models[i]) for i in range(len(traces))]
```

    /home/johan/VirtualEnv/ppsda/lib/python3.6/site-packages/pymc3/sampling.py:1247: UserWarning: samples parameter is smaller than nchains times ndraws, some draws and/or chains may not be represented in the returned posterior predictive sample
      "samples parameter is smaller than nchains times ndraws, some draws "
    100%|██████████| 500/500 [00:34<00:00, 14.44it/s]

```python
# various plots to inspect the inference results
varnames = mdl.get_varnames(traces[0])
#az.summary(traces[1], varnames)
```

```python
if conf['model_mode'] == 'train':
    # collect the results and display
    df = res.get_results_summary(traces, ppc, y_val, varnames)
else:
    # load results from disk
    df = pd.read_csv(file_basename + '.csv')
    df.index += 1
df
```

      r_hat
      mcse
      ess
      bfmi
      r2
      waic
      epsilon

      1
      1.457097
      15.672419
      19.967742
      0.485896
      0.987058
      -16487.937439
      0.062277

```python
if conf['model_mode'] == 'train':
    # save results to .csv
    df.to_csv(file_basename + '.csv', index=False)
```

```python
cnf.close(out_path)
```

# Plot posterior (single dataset)

```python
# dataset to plot
n_dataset = 0

# trace number in matrix
n_trace = 0

#fig.plot_posterior_single(x_val, ldata[n_dataset], ppc[n_trace], 10,
#                            file_basename, hpd_color='yellow', peak_pos=peaks)
fig.plot_posterior_single(x_val, ldata[n_dataset], ppc[n_trace], 10,
                             file_basename, hpd_color='orange', peak_pos=peaks, no_data='yes')
```

```python
az.plot_posterior(traces[n_trace], ['mu', 'amp', 'sigma', 'epsilon'], credible_interval=0.95);
```

```python
az.plot_posterior(traces[n_trace], ['a0'], credible_interval=0.95);
```

```python

```


 UNIVERSIDAD NACIONAL DE INGENIERIA

 Inteligencia Artificial

 Proyecto Final Parte 01 - Extracción de características

  Cristian Lazo : mecatronico.lazo@gmail.com

 Frank Solis : so_ni_mf@hotmail.com

 Diego Quispe  : dquispec@uni.pe

 Docente    : Paul Antonio Cardenas Lizana

--------------------------------

**Resumen  :**  Los usuarios de una línea telefónica son propensos a portar a otro operador, por ende es necesario saber previamente si esto va a suceder para que la empresa pueda emplear medidas para que esto no suceda. El proyecto presente tiene dos objetivos principales: la clasificación usuarios que portan o no y la interpretación del modelo para identificar las razones especificas de la portabilidad de los clientes.

```

```

--------------------------

# 1. INDICE

* [1. INDICE](#1)
* [2. DATASET](#2)
* [3. MÉTRICA](#3)
* [4. IMPORTANDO LIBRERIAS](#4)
* [5. CARGAR DATA](#5)
* [6. DATA EXPLORATION and FEATURES CREATION](#6)
* [7. PROBANDO PCA PARA REDUCIR LOS FEATURES](#7)
* [8. USANDO LOS FEATURES HALLADOS EN EL ARCHIVO PROYECTO_FINAL_02](#8)
* [9. CONTINUAR ARCHIVO PROYECTO_FINAL_02](#9)

# 2. DATASET

Consta de 6 archivos

- TRAIN_TARGET : 2 variables
- TRAIN_FUENTE_PLANTAPOST : 10 variables
- TRAIN_FUENTE_EQUIPOS : 18 variables
- TRAIN_FUENTE_DATOS : 10 variables
- TRAIN_FUENTE_VOZ : 29 variables
- TRAIN_FUENTE_APLICACIONES : 14 variables
- TRAIN_FUENTE_CONSULTA_PREV_PORTA : 3 variables

# 3. METRICA

La métrica de evaluación usaremos es el AUC, para entenderlo primero haremos unas definiciones:

### CURVA ROC

Una **curva ROC (curva de característica operativa del recepto)** es un gráfico que muestra el rendimiento de un modelo de clasificación en todos los umbrales de clasificación. Esta curva representa dos parámetros:
- Tasa de verdaderos positivos
- Tasas de falsos positivos

**Tasa de verdaderos positivos (TPR)** es sinónimo de exhaustividad y, por lo tanto, se define de la siguiente manera:

\begin{equation}
TPR = \frac{VP}{VP+FN}
\end{equation}

**Tasa de falsos positivos (FPR)**  se define de la siguiente manera:

\begin{equation}
FPR = \frac{FP}{FP+VN}
\end{equation}

Una curva ROC representa TPR frente a FPR en diferentes umbrales de clasificación. Reducir el umbral de clasificación clasifica más elementos como positivos, por lo que aumentarán tanto los falsos positivos como los verdaderos positivos. En la siguiente figura, se muestra una curva ROC típica.
Para calcular los puntos en una curva ROC, podríamos evaluar un modelo de regresión logística muchas veces con diferentes umbrales de clasificación, pero esto es ineficiente. Afortunadamente, existe un algoritmo eficiente basado en ordenamiento que puede brindarnos esta información, denominado AUC.

```
from PIL import Image
Image.open('/home/liiarpi/Downloads/DATA/MOVISTAR/PAPER/imagenes/roc.png')
```

### AUC : Área bajo la curva ROC

**AUC** significa "área bajo la curva ROC". Esto significa que el AUC mide toda el área bidimensional por debajo de la curva ROC completa (piensa en un cálculo integral) de (0,0) a (1,1).

```
Image.open('/home/liiarpi/Downloads/DATA/MOVISTAR/PAPER/imagenes/auc.png')
```

El AUC proporciona una medición agregada del rendimiento en todos los umbrales de clasificación posibles. Una forma de interpretar el AUC es como la probabilidad de que el modelo clasifique un ejemplo positivo aleatorio más alto que un ejemplo negativo aleatorio. Observa, a modo de ilustración, los siguientes ejemplos, que están ordenados de izquierda a derecha en orden ascendente con respecto a las predicciones de regresión logística:

El AUC representa la probabilidad de que un ejemplo aleatorio positivo (verde) se posicione a la derecha de un ejemplo aleatorio negativo (rojo).

El AUC oscila en valor del 0 al 1. Un modelo cuyas predicciones son un 100% incorrectas tiene un AUC de 0.0; otro cuyas predicciones son un 100% correctas tiene un AUC de 1.0.

```
Image.open('/home/liiarpi/Downloads/DATA/MOVISTAR/PAPER/imagenes/cuadri.png')

```

**Razones por la cual usamos AUC como métrica :**

- El AUC es **invariable con respecto a la escala.** Mide qué tan bien se clasifican las predicciones, en lugar de sus valores absolutos.
- El AUC es **invariable con respecto al umbral de clasificación.** Mide la calidad de las predicciones del modelo, sin tener en cuenta qué umbral de clasificación se elige.

# 4. IMPORTAR LIBRERIAS

```
from librerias3 import *
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as pltd
import seaborn as sns
import gc
from sklearn.model_selection import train_test_split
import lightgbm as lgb
import warnings
warnings.filterwarnings('ignore')
```

# 5. CARGAR DATA

```
train_path='/home/liiarpi/Downloads/DATA/MOVISTAR/DATA/TRAIN/'
target=pd.read_csv(train_path+'TRAIN_TARGET.txt',sep='|')
tfa   =pd.read_csv(train_path+'TRAIN_FUENTE_APLICACIONES.txt',sep='|',parse_dates=['PERIODO'])
tfc   =pd.read_csv(train_path+'TRAIN_FUENTE_CONSULTA_PREV_PORTA.txt',sep='|',parse_dates=['PERIODO'])
tfd   =pd.read_csv(train_path+'TRAIN_FUENTE_DATOS.txt',sep='|',parse_dates=['PERIODO'])
tfe   =pd.read_csv(train_path+'TRAIN_FUENTE_EQUIPOS.txt',sep='|',parse_dates=['PERIODO'])
tfv   =pd.read_csv(train_path+'TRAIN_FUENTE_VOZ.txt',sep='|',parse_dates=['PERIODO'])
tpp   =pd.read_csv(train_path+'TRAIN_PLANTAPOST.txt',sep='|',parse_dates=['PERIODO'])
```

```
target.head()
```

      ID_CLIENTE
      TARGET

      0
      M4+Yb9VBZlrmolusfk43+6zj9IEiONcbxv8Vo4mDskU=
      0

      1
      7CZN3fc0TkgCt99e2wXhHws0ZdjbidJ13wHbBCWo93s=
      0

      2
      Gato2h7aHbSHatI1jDDmHEgQsm+L0kVqt59jfZLp6vQ=
      0

      3
      a7ydviDHC4D3hefkfTJF9UP3sIJ55tM+Y7OW40bIlKc=
      0

      4
      HY8WUIZd4Bo8uLtyTtFowF+JM3UbqcHEyUaZvOkmp/w=
      0

```
tfc.head()
```

      PERIODO
      ID_CLIENTE
      CANT_CONSULTAS_PREV

      0
      2018-12-01
      Q0Sqxt9/+fI5C96Oj0qJZ92xheja1/7MUmzwG9BEJvU=
      1

      1
      2018-11-01
      yaomlj6mDsksD3XNE1++B9kjjPmR5IbU/YeZB6/xIt8=
      1

      2
      2018-11-01
      4IMVMkE/j892RjRR74zo+H1/vzj+TzyvDsvDqnMoSd8=
      1

      3
      2018-10-01
      7tzg/k08GC863JggrB3E2X2aJUqV0KbaRIDqZnDezUs=
      1

      4
      2018-10-01
      Wg7CVsL20twXVpeGqtjkCXsJYJC+fIIJyPvdmphPQeE=
      1

```
names=tfa.columns.values
found_index = np.in1d(names, ['ID_CLIENTE','PERIODO']).nonzero()[0]
names = np.delete(names, found_index)+'_mean'
names
```

    array(['N_DIAS_NAVEGACION_mean', 'APP_1_mean', 'APP_2_mean', 'APP_3_mean',
           'APP_4_mean', 'APP_5_mean', 'APP_6_mean', 'APP_7_mean',
           'APP_8_mean', 'APP_9_mean', 'APP_10_mean', 'MB_TOTAL_APP_mean'],
          dtype=object)

# 6. Data Exploration and Feature Creation

```
def procesar_2(names,df):
    x= df[names]
    names=x.columns.values
    for i in range(0,x.shape[1]):
        namecolumn=names[i]
        x['alta_'+namecolumn]=(x[namecolumn]>(x[namecolumn].mean()+x[namecolumn].std()))
        x['media_'+namecolumn]=(x[namecolumn]<=(x[namecolumn].mean()+x[namecolumn].std()))
        x['media_'+namecolumn]=x['media_'+namecolumn]&(x[namecolumn]>=(x[namecolumn].mean()-x[namecolumn].std()))
        x['baja_'+namecolumn]=(x[namecolumn]<=(x[namecolumn].mean()-x[namecolumn].std()))
        x=x.drop([namecolumn],axis=1)
    x=x.astype('float')
    return x
```

```
def agg_numeric(df, group_var):
    # Remove id variables other than grouping variable
    for col in df:
        if col != group_var and 'ID_CLIENTE' in col:
            df = df.drop(columns = col)

    group_ids = df[group_var]
    numeric_df = df.select_dtypes('number')
    numeric_df[group_var] = group_ids

    # Group by the specified variable and calculate the statistics
    agg = numeric_df.groupby(group_var).agg(['count', 'mean', 'max', 'min', 'sum']).reset_index()

    # Need to create new column names
    columns = [group_var]

    # Iterate through the variables names
    for var in agg.columns.levels[0]:
        # Skip the grouping variable
        if var != group_var:
            # Iterate through the stat names
            for stat in agg.columns.levels[1][:-1]:
                # Make a new column name for the variable and stat
                columns.append('%s_%s' % (var, stat))

    agg.columns = columns
    print('NUEVOS FEATURES CREADOS')
    return agg
def agg_numeric2(df, group_var):
    # Remove id variables other than grouping variable
    for col in df:
        if col != group_var and 'ID_CLIENTE' in col:
            df = df.drop(columns = col)

    group_ids = df[group_var]
    numeric_df = df.select_dtypes('number')
    numeric_df[group_var] = group_ids

    # Group by the specified variable and calculate the statistics
    agg = numeric_df.groupby(group_var).agg(['mean']).reset_index()

    # Need to create new column names
    columns = [group_var]

    # Iterate through the variables names
    for var in agg.columns.levels[0]:
        # Skip the grouping variable
        if var != group_var:
            # Iterate through the stat names
            for stat in agg.columns.levels[1][:-1]:
                # Make a new column name for the variable and stat
                columns.append('%s_%s' % (var, stat))
    agg.columns = columns
    print('NUEVOS FEATURES CREADOS')
    return agg
```

```
def procesar_1(path,name):
    df=pd.read_csv(path+name,sep='|', parse_dates=['PERIODO'])
    print(df.shape)
    #names=df.columns.values
    #found_index = np.in1d(names, ['ID_CLIENTE','PERIODO']).nonzero()[0]
    #names = np.delete(names, found_index)+'_mean'
    #names2=df.columns.values
    #found_index2 = np.in1d(names2, ['PERIODO']).nonzero()[0]
    #names2 = np.delete(names2, found_index2)

    #df_date = agg_numeric2(df[['ID_CLIENTE','PERIODO']], group_var = 'ID_CLIENTE')

    date_attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear', 'Is_month_end',
    'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']
    for n in date_attr:
        a = getattr(df['PERIODO'].dt, n.lower())
        #a[a==np.NaN]=0
        a[(1-np.isnan(a))>0.0001]=0
        a[a==np.Inf]=0
        a[a==-np.Inf]=0
        df['PERIODO_' + n]=a.astype(float)
        # 2. Agregamos una columan con una representación numérica de la fecha
        df['PERIODO_elapsed'] = df['PERIODO'].astype(np.int64) // 10 ** 9

    # 3. Eliminamos la variable date
    df.drop('PERIODO', axis=1, inplace=True)

    names3=df.columns.values
    found_index3 = np.in1d(names3, ['ID_CLIENTE']).nonzero()[0]
    names3 = np.delete(names3, found_index3)+'_mean'

    #VARIABLES OBJECT FLOAT
    for n,col in df.items():
        if not pd.api.types.is_numeric_dtype(col) and n != 'ID_CLIENTE':
            df[n] = col.astype('category')

    for n,col in df.items()  :
        if pd.api.types.is_categorical_dtype(col) and n != 'ID_CLIENTE':
            df[n] = col.cat.codes+1

    names = df.columns.values
    aa=df.isna().sum() / len(df) # % de data faltante
    for i in range (0,aa.shape[0]):
        if(aa[i]>0):
            print('DATA FALTANTE -',names[i])
            df[str(names[i])]=df[str(names[i])].fillna(df[str(names[i])].mean())

    df = agg_numeric(df, group_var = 'ID_CLIENTE')

    #df = agg_numeric(df[names2], group_var = 'ID_CLIENTE')
    #df = pd.merge(df,df_date, on='ID_CLIENTE', how='left')
    #print(df.columns.values)
    #print(names3)
    df2 =procesar_2(names3,df)

    df = pd.concat([df,df2], axis=1)

    print(df.shape)

    return df
```

```
a=[2,3,np.NaN,np.Inf]
a[a==np.NaN]=0
a

```

    [0, 3, nan, inf]

```
train_path='/home/liiarpi/Downloads/DATA/MOVISTAR/DATA/TRAIN/'
df=procesar_1(train_path,'TRAIN_FUENTE_CONSULTA_PREV_PORTA.txt')

```

    (9249, 3)
    NUEVOS FEATURES CREADOS
    (8524, 113)

```
train_path='/home/liiarpi/Downloads/DATA/MOVISTAR/DATA/TRAIN/'
df=procesar_1(train_path,'TRAIN_FUENTE_VOZ.txt')
```

    (587474, 29)
    DATA FALTANTE - AIRTIME_IN_TOT
    DATA FALTANTE - AIRTIME_IN_ON
    DATA FALTANTE - AIRTIME_IN_OFF
    DATA FALTANTE - CALLS_IN_TOT
    DATA FALTANTE - CALLS_IN_ON
    DATA FALTANTE - CALLS_IN_OFF
    DATA FALTANTE - DAYS_IN_VOICE_TOT
    DATA FALTANTE - DAYS_IN_VOICE_ON
    DATA FALTANTE - DAYS_IN_VOICE_OFF
    DATA FALTANTE - AIRTIME_OUT_TOT
    DATA FALTANTE - AIRTIME_OUT_ON
    DATA FALTANTE - AIRTIME_OUT_OFF
    DATA FALTANTE - CALLS_OUT_TOT
    DATA FALTANTE - CALLS_OUT_ON
    DATA FALTANTE - CALLS_OUT_OFF
    DATA FALTANTE - DEST_VOICE
    DATA FALTANTE - DEST_VOICE_ON
    DATA FALTANTE - DEST_VOICE_OFF
    DATA FALTANTE - DAYS_OUT_VOICE_TOT
    DATA FALTANTE - DAYS_OUT_VOICE_ON
    DATA FALTANTE - DAYS_OUT_VOICE_OFF
    DATA FALTANTE - CONT_TOT
    DATA FALTANTE - CONT_ON
    DATA FALTANTE - CONT_OFF
    DATA FALTANTE - TOP_CONT_5
    DATA FALTANTE - TOP_CONT_ON_5
    DATA FALTANTE - TOP_CONT_OFF_5
    DATA FALTANTE - PERIODO_Year
    DATA FALTANTE - PERIODO_Month
    DATA FALTANTE - PERIODO_Week
    DATA FALTANTE - PERIODO_Day
    DATA FALTANTE - PERIODO_Dayofweek
    DATA FALTANTE - PERIODO_Dayofyear
    NUEVOS FEATURES CREADOS
    (210000, 321)

```
path='/home/liiarpi/Downloads/DATA/MOVISTAR'
pathtrain=path+'/DATA/TRAIN/'
pathtest =path+'/DATA/TEST/'
pathguardartrain=path+'/NEW_DATA3/NEW_TRAIN/'
pathguardartest =path+'/NEW_DATA3/NEW_TEST/'

def transformar(pathorigen,path,name):
    df=procesar_1(pathorigen,name+'.txt')
    names = df.columns.values
    df.to_csv(path+name+'.csv',index=None)

```

```
transformar(pathtrain,pathguardartrain,'TRAIN_FUENTE_APLICACIONES')
transformar(pathtrain,pathguardartrain,'TRAIN_FUENTE_CONSULTA_PREV_PORTA')
transformar(pathtrain,pathguardartrain,'TRAIN_FUENTE_DATOS')
transformar(pathtrain,pathguardartrain,'TRAIN_FUENTE_EQUIPOS')
transformar(pathtrain,pathguardartrain,'TRAIN_FUENTE_VOZ')
transformar(pathtrain,pathguardartrain,'TRAIN_PLANTAPOST')
```

    (513548, 14)
    NUEVOS FEATURES CREADOS
    (180063, 201)
    (9249, 3)
    NUEVOS FEATURES CREADOS
    (8524, 113)
    (629997, 10)
    DATA FALTANTE - INDICADOR_DATOS_1
    DATA FALTANTE - INDICADOR_DATOS_2
    NUEVOS FEATURES CREADOS
    (209999, 169)
    (189225, 18)
    NUEVOS FEATURES CREADOS
    (189225, 233)
    (587474, 29)
    DATA FALTANTE - AIRTIME_IN_TOT
    DATA FALTANTE - AIRTIME_IN_ON
    DATA FALTANTE - AIRTIME_IN_OFF
    DATA FALTANTE - CALLS_IN_TOT
    DATA FALTANTE - CALLS_IN_ON
    DATA FALTANTE - CALLS_IN_OFF
    DATA FALTANTE - DAYS_IN_VOICE_TOT
    DATA FALTANTE - DAYS_IN_VOICE_ON
    DATA FALTANTE - DAYS_IN_VOICE_OFF
    DATA FALTANTE - AIRTIME_OUT_TOT
    DATA FALTANTE - AIRTIME_OUT_ON
    DATA FALTANTE - AIRTIME_OUT_OFF
    DATA FALTANTE - CALLS_OUT_TOT
    DATA FALTANTE - CALLS_OUT_ON
    DATA FALTANTE - CALLS_OUT_OFF
    DATA FALTANTE - DEST_VOICE
    DATA FALTANTE - DEST_VOICE_ON
    DATA FALTANTE - DEST_VOICE_OFF
    DATA FALTANTE - DAYS_OUT_VOICE_TOT
    DATA FALTANTE - DAYS_OUT_VOICE_ON
    DATA FALTANTE - DAYS_OUT_VOICE_OFF
    DATA FALTANTE - CONT_TOT
    DATA FALTANTE - CONT_ON
    DATA FALTANTE - CONT_OFF
    DATA FALTANTE - TOP_CONT_5
    DATA FALTANTE - TOP_CONT_ON_5
    DATA FALTANTE - TOP_CONT_OFF_5
    DATA FALTANTE - PERIODO_Year
    DATA FALTANTE - PERIODO_Month
    DATA FALTANTE - PERIODO_Week
    DATA FALTANTE - PERIODO_Day
    DATA FALTANTE - PERIODO_Dayofweek
    DATA FALTANTE - PERIODO_Dayofyear
    NUEVOS FEATURES CREADOS
    (210000, 321)
    (630000, 10)
    DATA FALTANTE - VALORRENTAMENSUAL
    DATA FALTANTE - ANTIGUEDAD_POSTPAGO
    DATA FALTANTE - SUB_TOTAL_FACT
    DATA FALTANTE - EDAD
    DATA FALTANTE - DIAS_QUEDANPERMANENCIA
    NUEVOS FEATURES CREADOS
    (210000, 169)

```
transformar(pathtest,pathguardartest,'TEST_FUENTE_APLICACIONES')
transformar(pathtest,pathguardartest,'TEST_FUENTE_CONSULTA_PREV_PORTA')
transformar(pathtest,pathguardartest,'TEST_FUENTE_DATOS')
transformar(pathtest,pathguardartest,'TEST_FUENTE_EQUIPOS')
transformar(pathtest,pathguardartest,'TEST_FUENTE_VOZ')
transformar(pathtest,pathguardartest,'TEST_PLANTAPOST')
```

    (171983, 14)
    NUEVOS FEATURES CREADOS
    (60309, 201)
    (3583, 3)
    NUEVOS FEATURES CREADOS
    (3305, 113)
    (210000, 10)
    DATA FALTANTE - INDICADOR_DATOS_1
    DATA FALTANTE - INDICADOR_DATOS_2
    NUEVOS FEATURES CREADOS
    (70000, 169)
    (62969, 18)
    NUEVOS FEATURES CREADOS
    (62969, 233)
    (194139, 29)
    DATA FALTANTE - AIRTIME_IN_TOT
    DATA FALTANTE - AIRTIME_IN_ON
    DATA FALTANTE - AIRTIME_IN_OFF
    DATA FALTANTE - CALLS_IN_TOT
    DATA FALTANTE - CALLS_IN_ON
    DATA FALTANTE - CALLS_IN_OFF
    DATA FALTANTE - DAYS_IN_VOICE_TOT
    DATA FALTANTE - DAYS_IN_VOICE_ON
    DATA FALTANTE - DAYS_IN_VOICE_OFF
    DATA FALTANTE - AIRTIME_OUT_TOT
    DATA FALTANTE - AIRTIME_OUT_ON
    DATA FALTANTE - AIRTIME_OUT_OFF
    DATA FALTANTE - CALLS_OUT_TOT
    DATA FALTANTE - CALLS_OUT_ON
    DATA FALTANTE - CALLS_OUT_OFF
    DATA FALTANTE - DEST_VOICE
    DATA FALTANTE - DEST_VOICE_ON
    DATA FALTANTE - DEST_VOICE_OFF
    DATA FALTANTE - DAYS_OUT_VOICE_TOT
    DATA FALTANTE - DAYS_OUT_VOICE_ON
    DATA FALTANTE - DAYS_OUT_VOICE_OFF
    DATA FALTANTE - CONT_TOT
    DATA FALTANTE - CONT_ON
    DATA FALTANTE - CONT_OFF
    DATA FALTANTE - TOP_CONT_5
    DATA FALTANTE - TOP_CONT_ON_5
    DATA FALTANTE - TOP_CONT_OFF_5
    DATA FALTANTE - PERIODO_Year
    DATA FALTANTE - PERIODO_Month
    DATA FALTANTE - PERIODO_Week
    DATA FALTANTE - PERIODO_Day
    DATA FALTANTE - PERIODO_Dayofweek
    DATA FALTANTE - PERIODO_Dayofyear
    NUEVOS FEATURES CREADOS
    (70000, 321)
    (210000, 10)
    DATA FALTANTE - VALORRENTAMENSUAL
    DATA FALTANTE - ANTIGUEDAD_POSTPAGO
    DATA FALTANTE - SUB_TOTAL_FACT
    DATA FALTANTE - EDAD
    DATA FALTANTE - DIAS_QUEDANPERMANENCIA
    NUEVOS FEATURES CREADOS
    (70000, 169)

```

```

```
train_path='/home/liiarpi/Downloads/DATA/MOVISTAR/NEW_DATA3/NEW_TRAIN/'
target=pd.read_csv('/home/liiarpi/Downloads/DATA/MOVISTAR/DATA/TRAIN/TRAIN_TARGET.txt',sep='|')
tfa=pd.read_csv(train_path+'TRAIN_FUENTE_APLICACIONES.csv')
tfc=pd.read_csv(train_path+'TRAIN_FUENTE_CONSULTA_PREV_PORTA.csv')
tfd=pd.read_csv(train_path+'TRAIN_FUENTE_DATOS.csv')
tfe=pd.read_csv(train_path+'TRAIN_FUENTE_EQUIPOS.csv')
tfv=pd.read_csv(train_path+'TRAIN_FUENTE_VOZ.csv')
tpp=pd.read_csv(train_path+'TRAIN_PLANTAPOST.csv')
```

```
TrainData = pd.merge(target, tfa, how='left')
TrainData = pd.merge(TrainData, tfc, on='ID_CLIENTE', how='left')
TrainData = pd.merge(TrainData, tfd, on='ID_CLIENTE', how='left')
TrainData = pd.merge(TrainData, tfe, on='ID_CLIENTE', how='left')
TrainData = pd.merge(TrainData, tfv, on='ID_CLIENTE', how='left')
TrainData = pd.merge(TrainData, tpp, on='ID_CLIENTE', how='left')

```

```
train=pd.read_csv('Data_tratada_Train.csv',na_values=['NaN','nan',-1])
```

```
print('tamaño = ', train.shape)
# Presentamos un ejemplo (de 5 datos) del dataset
train.head()
```

    tamaño =  (210000, 578)

      ID_CLIENTE
      TARGET
      N_DIAS_NAVEGACION_count
      N_DIAS_NAVEGACION_mean
      N_DIAS_NAVEGACION_max
      N_DIAS_NAVEGACION_min
      N_DIAS_NAVEGACION_sum
      APP_1_count
      APP_1_mean
      APP_1_max
      ...
      baja_SUB_TOTAL_FACT_mean
      alta_EDAD_mean
      media_EDAD_mean
      baja_EDAD_mean
      alta_DIAS_QUEDANPERMANENCIA_mean
      media_DIAS_QUEDANPERMANENCIA_mean
      baja_DIAS_QUEDANPERMANENCIA_mean
      alta_DEPARTAMENTO_mean
      media_DEPARTAMENTO_mean
      baja_DEPARTAMENTO_mean

      0
      M4+Yb9VBZlrmolusfk43+6zj9IEiONcbxv8Vo4mDskU=
      0
      2.0
      22.500000
      24.0
      21.0
      45.0
      2.0
      1720.460000
      1920.41
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

      1
      7CZN3fc0TkgCt99e2wXhHws0ZdjbidJ13wHbBCWo93s=
      0
      3.0
      29.666667
      31.0
      28.0
      89.0
      3.0
      372.153333
      517.42
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

      2
      Gato2h7aHbSHatI1jDDmHEgQsm+L0kVqt59jfZLp6vQ=
      0
      3.0
      30.000000
      31.0
      28.0
      90.0
      3.0
      257.596667
      427.18
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0

      3
      a7ydviDHC4D3hefkfTJF9UP3sIJ55tM+Y7OW40bIlKc=
      0
      3.0
      28.000000
      31.0
      26.0
      84.0
      3.0
      781.600000
      1042.77
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      1.0
      0.0
      0.0

      4
      HY8WUIZd4Bo8uLtyTtFowF+JM3UbqcHEyUaZvOkmp/w=
      0
      3.0
      29.000000
      31.0
      27.0
      87.0
      3.0
      933.283333
      1230.82
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0

5 rows × 578 columns

```
X=train.drop(columns=['ID_CLIENTE', 'TARGET'])
y=train['TARGET']
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=7)
```

```
#X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=7)
train_data=lgb.Dataset(X_train, y_train)
test_data=lgb.Dataset(X_test, y_test, reference=train_data)
```

```
params = {
    'task': 'train',
    'boosting_type': 'gbdt',
    'objective': 'binary',
    'metric': {'auc'},
    'num_leaves': 20,
    'num_boost_round': 5000,
    'min_data_in_leaf': 50,
    'max_depth': -1,
    'verbose': -1,
    'n_jobs': -1,
    'learning_rate': 0.035
}
```

```
model = lgb.train(params,train_data,valid_sets=test_data,early_stopping_rounds=80)
```

    [1]	valid_0's auc: 0.618285
    Training until validation scores don't improve for 80 rounds.
    [2]	valid_0's auc: 0.623808
    [3]	valid_0's auc: 0.626038
    [4]	valid_0's auc: 0.626204
    [5]	valid_0's auc: 0.62651
    [6]	valid_0's auc: 0.627204
    [7]	valid_0's auc: 0.626664
    [8]	valid_0's auc: 0.626893
    [9]	valid_0's auc: 0.627229
    [10]	valid_0's auc: 0.627513
    [11]	valid_0's auc: 0.627644
    [12]	valid_0's auc: 0.627571
    [13]	valid_0's auc: 0.627896
    [14]	valid_0's auc: 0.628177
    [15]	valid_0's auc: 0.628492
    [16]	valid_0's auc: 0.628649
    [17]	valid_0's auc: 0.629023
    [18]	valid_0's auc: 0.62928
    [19]	valid_0's auc: 0.629413
    [20]	valid_0's auc: 0.629706
    [21]	valid_0's auc: 0.629788
    [22]	valid_0's auc: 0.629955
    [23]	valid_0's auc: 0.629956
    [24]	valid_0's auc: 0.630142
    [25]	valid_0's auc: 0.630223
    [26]	valid_0's auc: 0.630383
    [27]	valid_0's auc: 0.630474
    [28]	valid_0's auc: 0.630586
    [29]	valid_0's auc: 0.630575
    [30]	valid_0's auc: 0.630611
    [31]	valid_0's auc: 0.630523
    [32]	valid_0's auc: 0.630594
    [33]	valid_0's auc: 0.630476
    [34]	valid_0's auc: 0.630559
    [35]	valid_0's auc: 0.630627
    [36]	valid_0's auc: 0.630757
    [37]	valid_0's auc: 0.630747
    [38]	valid_0's auc: 0.630775
    [39]	valid_0's auc: 0.630759
    [40]	valid_0's auc: 0.63086
    [41]	valid_0's auc: 0.630907
    [42]	valid_0's auc: 0.630927
    [43]	valid_0's auc: 0.630937
    [44]	valid_0's auc: 0.630976
    [45]	valid_0's auc: 0.631045
    [46]	valid_0's auc: 0.631058
    [47]	valid_0's auc: 0.631028
    [48]	valid_0's auc: 0.630945
    [49]	valid_0's auc: 0.63092
    [50]	valid_0's auc: 0.630967
    [51]	valid_0's auc: 0.63102
    [52]	valid_0's auc: 0.631018
    [53]	valid_0's auc: 0.631077
    [54]	valid_0's auc: 0.631074
    [55]	valid_0's auc: 0.631124
    [56]	valid_0's auc: 0.631105
    [57]	valid_0's auc: 0.631137
    [58]	valid_0's auc: 0.631176
    [59]	valid_0's auc: 0.631205
    [60]	valid_0's auc: 0.631187
    [61]	valid_0's auc: 0.631157
    [62]	valid_0's auc: 0.63114
    [63]	valid_0's auc: 0.631193
    [64]	valid_0's auc: 0.631214
    [65]	valid_0's auc: 0.631231
    [66]	valid_0's auc: 0.63117
    [67]	valid_0's auc: 0.631165
    [68]	valid_0's auc: 0.63113
    [69]	valid_0's auc: 0.631135
    [70]	valid_0's auc: 0.631177
    [71]	valid_0's auc: 0.631123
    [72]	valid_0's auc: 0.631101
    [73]	valid_0's auc: 0.631137
    [74]	valid_0's auc: 0.631149
    [75]	valid_0's auc: 0.631202
    [76]	valid_0's auc: 0.631272
    [77]	valid_0's auc: 0.631296
    [78]	valid_0's auc: 0.631295
    [79]	valid_0's auc: 0.63133
    [80]	valid_0's auc: 0.63136
    [81]	valid_0's auc: 0.631324
    [82]	valid_0's auc: 0.631298
    [83]	valid_0's auc: 0.631338
    [84]	valid_0's auc: 0.631361
    [85]	valid_0's auc: 0.631325
    [86]	valid_0's auc: 0.631308
    [87]	valid_0's auc: 0.631311
    [88]	valid_0's auc: 0.631293
    [89]	valid_0's auc: 0.631289
    [90]	valid_0's auc: 0.631291
    [91]	valid_0's auc: 0.631332
    [92]	valid_0's auc: 0.631387
    [93]	valid_0's auc: 0.631406
    [94]	valid_0's auc: 0.631426
    [95]	valid_0's auc: 0.631444
    [96]	valid_0's auc: 0.631448
    [97]	valid_0's auc: 0.631467
    [98]	valid_0's auc: 0.631443
    [99]	valid_0's auc: 0.63142
    [100]	valid_0's auc: 0.631466
    [101]	valid_0's auc: 0.631534
    [102]	valid_0's auc: 0.631536
    [103]	valid_0's auc: 0.631579
    [104]	valid_0's auc: 0.631558
    [105]	valid_0's auc: 0.631558
    [106]	valid_0's auc: 0.63162
    [107]	valid_0's auc: 0.631678
    [108]	valid_0's auc: 0.631644
    [109]	valid_0's auc: 0.631664
    [110]	valid_0's auc: 0.631685
    [111]	valid_0's auc: 0.631674
    [112]	valid_0's auc: 0.631634
    [113]	valid_0's auc: 0.631619
    [114]	valid_0's auc: 0.631614
    [115]	valid_0's auc: 0.631646
    [116]	valid_0's auc: 0.631596
    [117]	valid_0's auc: 0.631577
    [118]	valid_0's auc: 0.631586
    [119]	valid_0's auc: 0.631562
    [120]	valid_0's auc: 0.631553
    [121]	valid_0's auc: 0.631535
    [122]	valid_0's auc: 0.631559
    [123]	valid_0's auc: 0.631534
    [124]	valid_0's auc: 0.631516
    [125]	valid_0's auc: 0.631528
    [126]	valid_0's auc: 0.631533
    [127]	valid_0's auc: 0.63153
    [128]	valid_0's auc: 0.631457
    [129]	valid_0's auc: 0.631504
    [130]	valid_0's auc: 0.631548
    [131]	valid_0's auc: 0.631563
    [132]	valid_0's auc: 0.631597
    [133]	valid_0's auc: 0.631615
    [134]	valid_0's auc: 0.63161
    [135]	valid_0's auc: 0.63163
    [136]	valid_0's auc: 0.631636
    [137]	valid_0's auc: 0.631645
    [138]	valid_0's auc: 0.631694
    [139]	valid_0's auc: 0.631656
    [140]	valid_0's auc: 0.631689
    [141]	valid_0's auc: 0.631699
    [142]	valid_0's auc: 0.631719
    [143]	valid_0's auc: 0.631699
    [144]	valid_0's auc: 0.631731
    [145]	valid_0's auc: 0.631768
    [146]	valid_0's auc: 0.631793
    [147]	valid_0's auc: 0.631842
    [148]	valid_0's auc: 0.631841
    [149]	valid_0's auc: 0.631811
    [150]	valid_0's auc: 0.631788
    [151]	valid_0's auc: 0.631827
    [152]	valid_0's auc: 0.631851
    [153]	valid_0's auc: 0.631857
    [154]	valid_0's auc: 0.631826
    [155]	valid_0's auc: 0.631838
    [156]	valid_0's auc: 0.631929
    [157]	valid_0's auc: 0.631915
    [158]	valid_0's auc: 0.631917
    [159]	valid_0's auc: 0.63191
    [160]	valid_0's auc: 0.63192
    [161]	valid_0's auc: 0.631869
    [162]	valid_0's auc: 0.63185
    [163]	valid_0's auc: 0.631848
    [164]	valid_0's auc: 0.631891
    [165]	valid_0's auc: 0.631916
    [166]	valid_0's auc: 0.631942
    [167]	valid_0's auc: 0.631955
    [168]	valid_0's auc: 0.631977
    [169]	valid_0's auc: 0.632012
    [170]	valid_0's auc: 0.632086
    [171]	valid_0's auc: 0.631994
    [172]	valid_0's auc: 0.63201
    [173]	valid_0's auc: 0.632015
    [174]	valid_0's auc: 0.632041
    [175]	valid_0's auc: 0.631996
    [176]	valid_0's auc: 0.631965
    [177]	valid_0's auc: 0.631977
    [178]	valid_0's auc: 0.631894
    [179]	valid_0's auc: 0.631881
    [180]	valid_0's auc: 0.631874
    [181]	valid_0's auc: 0.631872
    [182]	valid_0's auc: 0.631861
    [183]	valid_0's auc: 0.631842
    [184]	valid_0's auc: 0.631845
    [185]	valid_0's auc: 0.631858
    [186]	valid_0's auc: 0.631853
    [187]	valid_0's auc: 0.631859
    [188]	valid_0's auc: 0.631776
    [189]	valid_0's auc: 0.631817
    [190]	valid_0's auc: 0.631795
    [191]	valid_0's auc: 0.631777
    [192]	valid_0's auc: 0.631795
    [193]	valid_0's auc: 0.631764
    [194]	valid_0's auc: 0.631737
    [195]	valid_0's auc: 0.63167
    [196]	valid_0's auc: 0.631649
    [197]	valid_0's auc: 0.631653
    [198]	valid_0's auc: 0.631643
    [199]	valid_0's auc: 0.631617
    [200]	valid_0's auc: 0.631655
    [201]	valid_0's auc: 0.63166
    [202]	valid_0's auc: 0.631625
    [203]	valid_0's auc: 0.631694
    [204]	valid_0's auc: 0.631674
    [205]	valid_0's auc: 0.631611
    [206]	valid_0's auc: 0.631589
    [207]	valid_0's auc: 0.631639
    [208]	valid_0's auc: 0.631678
    [209]	valid_0's auc: 0.631716
    [210]	valid_0's auc: 0.631691
    [211]	valid_0's auc: 0.631687
    [212]	valid_0's auc: 0.63165
    [213]	valid_0's auc: 0.631681
    [214]	valid_0's auc: 0.631637
    [215]	valid_0's auc: 0.631623
    [216]	valid_0's auc: 0.631591
    [217]	valid_0's auc: 0.631556
    [218]	valid_0's auc: 0.631645
    [219]	valid_0's auc: 0.631605
    [220]	valid_0's auc: 0.631612
    [221]	valid_0's auc: 0.631553
    [222]	valid_0's auc: 0.631543
    [223]	valid_0's auc: 0.631523
    [224]	valid_0's auc: 0.631519
    [225]	valid_0's auc: 0.631543
    [226]	valid_0's auc: 0.631508
    [227]	valid_0's auc: 0.631524
    [228]	valid_0's auc: 0.63149
    [229]	valid_0's auc: 0.631503
    [230]	valid_0's auc: 0.63148
    [231]	valid_0's auc: 0.63148
    [232]	valid_0's auc: 0.631485
    [233]	valid_0's auc: 0.631475
    [234]	valid_0's auc: 0.631419
    [235]	valid_0's auc: 0.631412
    [236]	valid_0's auc: 0.631386
    [237]	valid_0's auc: 0.631401
    [238]	valid_0's auc: 0.631399
    [239]	valid_0's auc: 0.631349
    [240]	valid_0's auc: 0.631327
    [241]	valid_0's auc: 0.631355
    [242]	valid_0's auc: 0.631305
    [243]	valid_0's auc: 0.631331
    [244]	valid_0's auc: 0.631323
    [245]	valid_0's auc: 0.631348
    [246]	valid_0's auc: 0.631324
    [247]	valid_0's auc: 0.631255
    [248]	valid_0's auc: 0.631278
    [249]	valid_0's auc: 0.631303
    [250]	valid_0's auc: 0.631352
    Early stopping, best iteration is:
    [170]	valid_0's auc: 0.632086

```
a=lgb.plot_importance(model,importance_type='split',height=0.8,figsize=(18,80),grid=False,ignore_zero=False)
```

# 7.PROBANDO PCA PARA REDUCIR LOS FEATURES

```
from sklearn.decomposition import PCA
from sklearn import metrics

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
def pcaa(a,xx,yy):
    print("PCA de %d elementos"%a)
    pca =PCA(n_components=a)
    pca.fit(xx)
    xx=pca.transform(xx)
    print('pca listo')
    print('USANDO RANDOMFOREST CLASSIFIER')
    xx_train, xx_val, yy_train, yy_val = train_test_split(xx, yy, test_size=0.2, random_state=42)
    mm = RandomForestClassifier(20, n_jobs=-1, oob_score=True,min_samples_split=10)#,max_features=10)
    mm.fit(xx_train,yy_train)
    y_train_pred = mm.predict(xx_train)
    y_val_pred = mm.predict(xx_val)
    print(f'Scores:')
    print(f'Train      = {metrics.accuracy_score(yy_train, y_train_pred):.4}')
    print(f'Validation = {metrics.accuracy_score(yy_val, y_val_pred):.4}')
    if hasattr(mm, 'oob_score_'): print(f'OOB        = {mm.oob_score_:.4}')
    print("AUC Train ",roc_auc_score(yy_train, y_train_pred))
    print("AUC Test ",roc_auc_score(yy_val, y_val_pred))
    print('USANDO SVM')
    '''
    from sklearn.svm import SVC
    model_svm = SVC(gamma='auto')
    model_svm.fit(xx_train,yy_train)
    y_train_pred = model_svm.predict(xx_train)
    y_val_pred = model_svm.predict(xx_val)
    print("Accuracy Training Support Vector Machines:",metrics.accuracy_score(yy_train, y_train_pred))
    print("Accuracy Validation Support Vector Machines:",metrics.accuracy_score(yy_val, y_val_pred))
    print("AUC Train ",roc_auc_score(yy_train, y_train_pred))
    print("AUC Test ",roc_auc_score(yy_val, y_val_pred))'''

    print('USANDO LOGISTIC')
    from sklearn.linear_model import LogisticRegression
    model_logistic = LogisticRegression(random_state=0, solver='lbfgs')
    model_logistic.fit(xx_train, yy_train)
    y_train_pred = model_logistic.predict(xx_train)
    y_val_pred = model_logistic.predict(xx_val)
    print("Accuracy Training Logistic:",metrics.accuracy_score(yy_train, y_train_pred))
    print("Accuracy Validation Logistic:",metrics.accuracy_score(yy_val, y_val_pred))
    print("AUC Train ",roc_auc_score(yy_train, y_train_pred))
    print("AUC Test ",roc_auc_score(yy_val, y_val_pred))

    return mm,pca
```

```
pcaa(10,X,y)
```

    PCA de 10 elementos
    pca listo
    USANDO RANDOMFOREST CLASSIFIER
    Scores:
    Train      = 0.9365
    Validation = 0.9
    OOB        = 0.8956
    AUC Train  0.6826190476190477
    AUC Test  0.5001984126984127
    USANDO SVM
    USANDO LOGISTIC
    Accuracy Training Logistic: 0.8999940476190477
    Accuracy Validation Logistic: 0.8999761904761905
    AUC Train  0.49999669312169315
    AUC Test  0.4999867724867725

    (RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                 max_depth=None, max_features='auto', max_leaf_nodes=None,
                 min_impurity_decrease=0.0, min_impurity_split=None,
                 min_samples_leaf=1, min_samples_split=10,
                 min_weight_fraction_leaf=0.0, n_estimators=20, n_jobs=-1,
                 oob_score=True, random_state=None, verbose=0, warm_start=False),
     PCA(copy=True, iterated_power='auto', n_components=10, random_state=None,
       svd_solver='auto', tol=0.0, whiten=False))

```
modelo,pca_modelo=pcaa(20,X,y)
```

    PCA de 20 elementos
    pca listo
    USANDO RANDOMFOREST CLASSIFIER
    Scores:
    Train      = 0.9405
    Validation = 0.8998
    OOB        = 0.8949
    AUC Train  0.7023776455026455
    AUC Test  0.5000925925925925
    USANDO SVM
    USANDO LOGISTIC
    Accuracy Training Logistic: 0.8999880952380952
    Accuracy Validation Logistic: 0.8999761904761905
    AUC Train  0.49999338624338624
    AUC Test  0.4999867724867725

```
pcaa(15,X,y)
```

    PCA de 15 elementos
    pca listo
    USANDO RANDOMFOREST CLASSIFIER
    Scores:
    Train      = 0.9364
    Validation = 0.8999
    OOB        = 0.8962
    AUC Train  0.6819642857142857
    AUC Test  0.49992063492063493
    USANDO SVM
    USANDO LOGISTIC
    Accuracy Training Logistic: 0.8793630952380952
    Accuracy Validation Logistic: 0.8794761904761905
    AUC Train  0.5106779100529102
    AUC Test  0.5091269841269841

    (RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                 max_depth=None, max_features='auto', max_leaf_nodes=None,
                 min_impurity_decrease=0.0, min_impurity_split=None,
                 min_samples_leaf=1, min_samples_split=10,
                 min_weight_fraction_leaf=0.0, n_estimators=20, n_jobs=-1,
                 oob_score=True, random_state=None, verbose=0, warm_start=False),
     PCA(copy=True, iterated_power='auto', n_components=15, random_state=None,
       svd_solver='auto', tol=0.0, whiten=False))

### No mejoramos el modelo de lightgbm

```
test_path='/home/liiarpi/Downloads/DATA/MOVISTAR/NEW_DATA3/NEW_TEST/'
sfa=pd.read_csv(test_path+'TEST_FUENTE_APLICACIONES.csv')
sfc=pd.read_csv(test_path+'TEST_FUENTE_CONSULTA_PREV_PORTA.csv')
sfd=pd.read_csv(test_path+'TEST_FUENTE_DATOS.csv')
sfe=pd.read_csv(test_path+'TEST_FUENTE_EQUIPOS.csv')
sfv=pd.read_csv(test_path+'TEST_FUENTE_VOZ.csv')
spp=pd.read_csv(test_path+'TEST_PLANTAPOST.csv')
envio=pd.read_csv('/home/liiarpi/Downloads/DATA/MOVISTAR/DATA/TEST/TEST_ENVIO.txt',sep='|')
```

```
sfa=sfa.reindex(tfa.columns,axis=1)
sfc=sfc.reindex(tfc.columns,axis=1)
sfd=sfd.reindex(tfd.columns,axis=1)
sfe=sfe.reindex(tfe.columns,axis=1)
sfv=sfv.reindex(tfv.columns,axis=1)
spp=spp.reindex(tpp.columns,axis=1)
```

```
envio=pd.merge(envio,sfa,on='ID_CLIENTE',how='left')
envio=pd.merge(envio,sfc,on='ID_CLIENTE',how='left')
envio=pd.merge(envio,sfd,on='ID_CLIENTE',how='left')
envio=pd.merge(envio,sfe,on='ID_CLIENTE',how='left')
envio=pd.merge(envio,sfv,on='ID_CLIENTE',how='left')
envio=pd.merge(envio,spp,on='ID_CLIENTE',how='left')
```

```
envio['TARGET']=model.predict(envio.drop(columns='ID_CLIENTE'),num_iteration=model.best_iteration)
```

```
envio = envio.loc[:,['ID_CLIENTE','TARGET']]
```

```
envio.to_csv('/home/liiarpi/Downloads/DATA/MOVISTAR/submissions/sub08_DATA3.csv',index=None, sep=',')
```

```
envio['ID_CLIENTE']=envio['ID_CLIENTE'].astype(str)
envio['TARGET']=envio['TARGET'].astype('float64')
```

```
pd.read_csv('/home/liiarpi/Downloads/DATA/MOVISTAR/submissions/sub08_DATA3.csv').info()

```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 70000 entries, 0 to 69999
    Data columns (total 2 columns):
    ID_CLIENTE    70000 non-null object
    TARGET        70000 non-null float64
    dtypes: float64(1), object(1)
    memory usage: 1.1+ MB

###  Probando los resultados en la página oficial obtenemos un 0.67375 en Private Score
### y obtenemos un 0.66571 en Public Score

```
from PIL import Image
Image.open('1.jpg')
```

# 8. USANDO LOS FEATURES HALLADOS EN EL ARCHIVO PROYECTO_FINAL_02

```
test=pd.read_csv('Data_tratada_test.csv',na_values=['NaN','nan',-1])

```

```
features=['ID_CLIENTE','AIRTIME_IN_TOT_count', 'AIRTIME_OUT_ON_count', 'APP_10_count',
       'APP_1_count', 'APP_2_count', 'APP_4_count', 'APP_5_count',
       'APP_6_count', 'APP_7_count', 'APP_8_count', 'CALLS_IN_TOT_count',
       'CALLS_OUT_ON_count', 'CALLS_OUT_TOT_count', 'CONT_OFF_count',
       'CONT_ON_count', 'CONT_TOT_count', 'DAYS_IN_VOICE_OFF_sum',
       'DAYS_IN_VOICE_TOT_count', 'DAYS_OUT_VOICE_OFF_sum',
       'DEST_VOICE_ON_count', 'DEST_VOICE_count', 'MB_TOTAL_APP_count',
       'N_DIAS_NAVEGACION_count', 'N_DIAS_NAVEGACION_max',
       'N_DIAS_NAVEGACION_sum', 'TOP_CONT_5_count', 'TOP_CONT_5_max',
       'TOP_CONT_5_sum', 'TOP_CONT_OFF_5_count', 'TOP_CONT_OFF_5_max',
       'TOP_CONT_OFF_5_mean', 'TOP_CONT_OFF_5_sum', 'TOP_CONT_ON_5_count',
       'TRAF_DATOS_mean', 'TRAF_DATOS_sum', 'alta_CALLS_IN_OFF_mean',
       'alta_CALLS_OUT_OFF_mean', 'alta_DAYS_IN_VOICE_OFF_mean',
       'alta_DAYS_OUT_VOICE_OFF_mean', 'alta_DEST_VOICE_OFF_mean',
       'alta_TOP_CONT_OFF_5_mean', 'baja_DAYS_IN_VOICE_OFF_mean',
       'baja_DAYS_OUT_VOICE_OFF_mean', 'baja_PANTALLA_mean',
       'baja_TEC_CHIP_mean', 'media_APP_10_mean', 'media_APP_6_mean',
       'media_APP_7_mean', 'media_APP_8_mean', 'media_APP_9_mean',
       'media_N_DIAS_NAVEGACION_mean']
```

```
test=test[features]
```

```
envio2=pd.read_csv('/home/cristianl/Desktop/MOVISTAR/PAPER2/DATA/TEST/TEST_ENVIO.txt',sep='|')
envio2.head(2)

```

      ID_CLIENTE

      0
      oaxnYAu2ZFI2ncdPQpJFdFxhoWCa6BBAyIgKRcKELJ4=

      1
      0uf2S8ucPs02X3uVr2iIcfnPnCfbfWkGZnlZpNdpR/c=

```
envio2=pd.merge(envio2,test,on='ID_CLIENTE',how='left')
```

```
envio2.head(2)
```

      ID_CLIENTE
      AIRTIME_IN_TOT_count
      AIRTIME_OUT_ON_count
      APP_10_count
      APP_1_count
      APP_2_count
      APP_4_count
      APP_5_count
      APP_6_count
      APP_7_count
      ...
      baja_DAYS_IN_VOICE_OFF_mean
      baja_DAYS_OUT_VOICE_OFF_mean
      baja_PANTALLA_mean
      baja_TEC_CHIP_mean
      media_APP_10_mean
      media_APP_6_mean
      media_APP_7_mean
      media_APP_8_mean
      media_APP_9_mean
      media_N_DIAS_NAVEGACION_mean

      0
      oaxnYAu2ZFI2ncdPQpJFdFxhoWCa6BBAyIgKRcKELJ4=
      3
      3
      3.0
      3.0
      3.0
      3.0
      3.0
      3.0
      3.0
      ...
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      1.0
      1.0
      1.0

      1
      0uf2S8ucPs02X3uVr2iIcfnPnCfbfWkGZnlZpNdpR/c=
      3
      3
      3.0
      3.0
      3.0
      3.0
      3.0
      3.0
      3.0
      ...
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      1.0

2 rows × 52 columns

```
for n,col in envio2.items():
    if  n != 'ID_CLIENTE':
        print(n)
        envio2[n][(1-np.isnan(envio2[n]))>0.0001]=0
        a=np.copy(envio2[n])
        envio2[n][np.isnan(a)]=0
        envio2[n][a==np.Inf]=0
        envio2[n][a==-np.Inf]=0
```

    AIRTIME_IN_TOT_count
    AIRTIME_OUT_ON_count
    APP_10_count
    APP_1_count
    APP_2_count
    APP_4_count
    APP_5_count
    APP_6_count
    APP_7_count
    APP_8_count
    CALLS_IN_TOT_count
    CALLS_OUT_ON_count
    CALLS_OUT_TOT_count
    CONT_OFF_count
    CONT_ON_count
    CONT_TOT_count
    DAYS_IN_VOICE_OFF_sum
    DAYS_IN_VOICE_TOT_count
    DAYS_OUT_VOICE_OFF_sum
    DEST_VOICE_ON_count
    DEST_VOICE_count
    MB_TOTAL_APP_count
    N_DIAS_NAVEGACION_count
    N_DIAS_NAVEGACION_max
    N_DIAS_NAVEGACION_sum
    TOP_CONT_5_count
    TOP_CONT_5_max
    TOP_CONT_5_sum
    TOP_CONT_OFF_5_count
    TOP_CONT_OFF_5_max
    TOP_CONT_OFF_5_mean
    TOP_CONT_OFF_5_sum
    TOP_CONT_ON_5_count
    TRAF_DATOS_mean
    TRAF_DATOS_sum
    alta_CALLS_IN_OFF_mean
    alta_CALLS_OUT_OFF_mean
    alta_DAYS_IN_VOICE_OFF_mean
    alta_DAYS_OUT_VOICE_OFF_mean
    alta_DEST_VOICE_OFF_mean
    alta_TOP_CONT_OFF_5_mean
    baja_DAYS_IN_VOICE_OFF_mean
    baja_DAYS_OUT_VOICE_OFF_mean
    baja_PANTALLA_mean
    baja_TEC_CHIP_mean
    media_APP_10_mean
    media_APP_6_mean
    media_APP_7_mean
    media_APP_8_mean
    media_APP_9_mean
    media_N_DIAS_NAVEGACION_mean

```
envio2.head()
```

      ID_CLIENTE
      AIRTIME_IN_TOT_count
      AIRTIME_OUT_ON_count
      APP_10_count
      APP_1_count
      APP_2_count
      APP_4_count
      APP_5_count
      APP_6_count
      APP_7_count
      ...
      baja_DAYS_IN_VOICE_OFF_mean
      baja_DAYS_OUT_VOICE_OFF_mean
      baja_PANTALLA_mean
      baja_TEC_CHIP_mean
      media_APP_10_mean
      media_APP_6_mean
      media_APP_7_mean
      media_APP_8_mean
      media_APP_9_mean
      media_N_DIAS_NAVEGACION_mean

      0
      oaxnYAu2ZFI2ncdPQpJFdFxhoWCa6BBAyIgKRcKELJ4=
      0
      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      1
      0uf2S8ucPs02X3uVr2iIcfnPnCfbfWkGZnlZpNdpR/c=
      0
      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      neSeIS1r7rrJhkWxhJjm8km68p3/uStOqx9fu9MNeIA=
      0
      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      WKHM28G6WZiStJaohTxQI6zqlYWpHebYZhpinSLR9Bc=
      0
      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      3EpIkwtqjUAcHDP3KS9iOFxeoupAt/PhPX6WubaTwcY=
      0
      0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

5 rows × 52 columns

```
envio3=pca_modelo.transform(envio2.drop(columns='ID_CLIENTE'))

```

```
envio2['TARGET']=modelo.predict(envio3)
```

```
envio2 = envio2.loc[:,['ID_CLIENTE','TARGET']]
```

```
envio2.to_csv('/home/cristianl/Desktop/MOVISTAR/PAPER3/subm/subfinal.csv',index=None, sep=',')
```

###  Probando los resultados en la página oficial obtenemos un 0.68032 en Private Score
### y obtenemos un 0.6733 en Public Score a  que nos ubicaría en el segundo puesto

```
Image.open('2.jpg')
```

```

```