#### TELEPERFORMANCE (TP) - Prueba Técnica - Científico de Datos  (Por: Andrés Felipe Escallón Portilla,  22-24 / Feb / 2021)

# CASO DE CONSULTORIA

###### **Requerimiento:**

Una de las operaciones de cobranzas de la compañía quiere generar estrategias diferenciadas para el proceso de gestión de recuperación de cartera de clientes de acuerdo con el riesgo de no pago de la primera factura.

La estrategia se divide en 3 grupos de intervención:

1. Alto riesgo: Llamarlos al 5 día de mora.
2. Medio riesgo: Enviar mensaje de texto al 5 día de mora.
3. Bajo riesgo: Enviar mensaje de texto al día 15 de mora.

Los costos por cada tipo de contacto son los siguientes:

- Llamada asesor de cobranza 1700 pesos
- Mensaje de texto 40 pesos

#### **Instrucciones**

1. Muestre un análisis descriptivo y/o diagnóstico inicial de la información insumo para el modelo.
2. Construya un modelo estadístico que calcule la probabilidad de que un cliente no pague la primera factura. Explique por qué escogió las variables con las que va a trabajar y si debió hacer modificaciones de estas.
3. Defina los puntos de corte que determinen a que grupo de estrategia pertenece cada cliente.
4. Describa el perfil de los clientes con un alto riesgo de no pago.
5. ¿Qué sugerencias haría usted al equipo de cobranzas de acuerdo con el análisis de la información del modelo?
6. Explique el modelo y sustente su validez estadística, así como los puntos de corte, la cantidad de clientes que pertenecen a cada estrategia, los perfiles de riesgo y sus sugerencias y conclusiones.
7. Adjunte la base de datos con la probabilidad de riesgo de cada cliente.

Todos los puntos anteriores deben evidenciarse en un notebook de Python o un Markdown de R que se deben compartir a través de GitHub.

# Solución del Caso:
Responderé a este caso en Español aclarando que lo ideal hubiese sido hacerlo en Inglés (algunos comentarios si están en Inglés).

1. **Muestre un análisis descriptivo y/o diagnóstico inicial de la información insumo para el modelo:**

#### Análisis de Datos Exploratorio (EDA):

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```python
!pip install -r '/content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt'
```

    Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 1)) (4.6.3)
    Requirement already satisfied: boto3 in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 2)) (1.17.14)
    Requirement already satisfied: branca in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 3)) (0.4.2)
    Requirement already satisfied: folium in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 4)) (0.8.3)
    Requirement already satisfied: geopandas in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (0.8.2)
    Requirement already satisfied: graphviz in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 6)) (0.10.1)
    Requirement already satisfied: ipython in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (5.5.0)
    Requirement already satisfied: ipywidgets in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (7.6.3)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 9)) (3.2.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 10)) (1.19.5)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 11)) (1.1.5)
    Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 12)) (7.0.0)
    Requirement already satisfied: pingouin in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.3.10)
    Requirement already satisfied: pydotplus in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 14)) (2.0.2)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 15)) (2.23.0)
    Requirement already satisfied: s3transfer in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 16)) (0.3.4)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 17)) (0.22.2.post1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 18)) (1.4.1)
    Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 19)) (0.11.1)
    Requirement already satisfied: shapely in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 20)) (1.7.1)
    Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 21)) (0.10.2)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 22)) (1.1.1)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 23)) (4.41.1)
    Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 24)) (1.24.3)
    Requirement already satisfied: wordcloud in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 25)) (1.5.0)
    Requirement already satisfied: xlrd in /usr/local/lib/python3.7/dist-packages (from -r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 26)) (1.1.0)
    Requirement already satisfied: botocore<1.21.0,>=1.20.14 in /usr/local/lib/python3.7/dist-packages (from boto3->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 2)) (1.20.14)
    Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from boto3->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 2)) (0.10.0)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from branca->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 3)) (2.11.3)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from folium->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 4)) (1.15.0)
    Requirement already satisfied: pyproj>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (3.0.0.post1)
    Requirement already satisfied: fiona in /usr/local/lib/python3.7/dist-packages (from geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (1.8.18)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (0.8.1)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (4.8.0)
    Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (2.6.1)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (53.0.0)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (1.0.18)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (0.7.5)
    Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (4.3.3)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (4.4.2)
    Requirement already satisfied: jupyterlab-widgets>=1.0.0; python_version >= "3.6" in /usr/local/lib/python3.7/dist-packages (from ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (1.0.0)
    Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (5.1.2)
    Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (3.5.1)
    Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (4.10.1)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 9)) (2.8.1)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 9)) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 9)) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 9)) (2.4.7)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 11)) (2018.9)
    Requirement already satisfied: outdated in /usr/local/lib/python3.7/dist-packages (from pingouin->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.2.0)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pingouin->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.8.8)
    Requirement already satisfied: pandas-flavor>=0.1.2 in /usr/local/lib/python3.7/dist-packages (from pingouin->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.2.0)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 15)) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 15)) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 15)) (2020.12.5)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 17)) (1.0.1)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 21)) (0.5.1)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 22)) (1.2.1)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->branca->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 3)) (1.1.1)
    Requirement already satisfied: cligj>=0.5 in /usr/local/lib/python3.7/dist-packages (from fiona->geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (0.7.1)
    Requirement already satisfied: attrs>=17 in /usr/local/lib/python3.7/dist-packages (from fiona->geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (20.3.0)
    Requirement already satisfied: click<8,>=4.0 in /usr/local/lib/python3.7/dist-packages (from fiona->geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (7.1.2)
    Requirement already satisfied: click-plugins>=1.0 in /usr/local/lib/python3.7/dist-packages (from fiona->geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (1.1.1)
    Requirement already satisfied: munch in /usr/local/lib/python3.7/dist-packages (from fiona->geopandas->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 5)) (2.5.0)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect; sys_platform != "win32"->ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (0.7.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (0.2.5)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from traitlets>=4.2->ipython->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 7)) (0.2.0)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (2.6.0)
    Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (4.7.1)
    Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (5.3.1)
    Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (5.1.1)
    Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (5.3.5)
    Requirement already satisfied: littleutils in /usr/local/lib/python3.7/dist-packages (from outdated->pingouin->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.2.2)
    Requirement already satisfied: xarray in /usr/local/lib/python3.7/dist-packages (from pandas-flavor>=0.1.2->pingouin->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 13)) (0.15.1)
    Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (1.5.0)
    Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.9.2)
    Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (5.6.1)
    Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (22.0.3)
    Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (3.3.0)
    Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.4.4)
    Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (1.4.3)
    Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.3)
    Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.6.0)
    Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.8.4)
    Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (0.5.1)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->-r /content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/requirements.txt (line 8)) (20.9)

```python
# Importing all the required packages

import pandas              as pd # The gold standard of Python data analysis, to create and manipulate tables of data
import numpy               as np # The Python module for processing arrays which/Pandas is based on
import seaborn             as sns; sns.set() # A package to make Matplotlib visualizations more aesthetic
import branca
import geopandas

import matplotlib.pyplot   as plt # The gold standard of Python data visualization, but can be complex to use
from matplotlib import cm
from matplotlib.colors import ListedColormap, LinearSegmentedColormap
from matplotlib.patches import Patch
from matplotlib.widgets import Slider, Button, RadioButtons

import statsmodels.api as sm
import statsmodels.formula.api as sfm
from statsmodels.formula.api import ols

import scipy
from scipy import stats
from scipy import interp
from scipy.optimize import fsolve
from scipy.stats import chi2_contingency, ttest_ind, norm # A module for Python machine learning--we'll stick to T-Tests here

import sklearn
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import StratifiedKFold
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler, MaxAbsScaler, RobustScaler, StandardScaler
from sklearn.tree import export_graphviz
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_curve, auc, accuracy_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from statsmodels.formula.api import ols

from IPython.display import display
from IPython.display import display_html
from IPython.display import Image, SVG

import folium # package for making maps, please make sure to use a version older than 1.0.0.
from folium.plugins import TimeSliderChoropleth
# from time_slider_choropleth import TimeSliderChoropleth

import json
import requests
from bs4 import BeautifulSoup
import os
import pydotplus
from io import StringIO
from sympy import var, plot_implicit, Eq
from graphviz import Source
from wordcloud import WordCloud # A package that will allow us to make a wordcloud

# when executing, the plot will be done
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = (8,5)
import warnings
warnings.filterwarnings('ignore')
# ignore log(0) and divide by 0 warning
np.seterr(divide='ignore');
```

```python
#leyendo los archivos de la base de datos (guardados previamente como csv) y asignándolos a dataframes:
df_var=pd.read_csv('/content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/data/base_de_datos_prueba_tecnica_csv_vars.csv')
df=pd.read_csv('/content/drive/MyDrive/Teleperformance_PruebaTecnica_CientificoDatos/data/base_de_datos_prueba_tecnica_csv_db.csv')
```

```python
#visualizando los dataframes:
#Para una base de datos, las variables y su descripción es fundamental
df_var
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>VARIABLE</th>
      <th>DESCRIPCIÓN</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>REGIONAL</td>
      <td>regional de instalación del servicio</td>
    </tr>
    <tr>
      <th>1</th>
      <td>DEPARTAMENTO</td>
      <td>Departamento de instalación del servicio</td>
    </tr>
    <tr>
      <th>2</th>
      <td>TECNOL</td>
      <td>Tipo de tecnología del servicio</td>
    </tr>
    <tr>
      <th>3</th>
      <td>GERENCIA</td>
      <td>Gerencia de venta del servicio</td>
    </tr>
    <tr>
      <th>4</th>
      <td>CANAL_HOMOLOGADO_MILLICON</td>
      <td>Canal de venta del servicio</td>
    </tr>
    <tr>
      <th>5</th>
      <td>tipo_fuerza_venta</td>
      <td>Tipo de fuerza o equipo que realizó la venta</td>
    </tr>
    <tr>
      <th>6</th>
      <td>estrato</td>
      <td>Estrato donde fue instalado el servicio</td>
    </tr>
    <tr>
      <th>7</th>
      <td>antiguedad_meses</td>
      <td>Antigüedad del cliente al momento de solicitar...</td>
    </tr>
    <tr>
      <th>8</th>
      <td>productos</td>
      <td>Productos solicitados TV= televisión, TO=Telef...</td>
    </tr>
    <tr>
      <th>9</th>
      <td>portafolio</td>
      <td>cantidad de productos adquiridos</td>
    </tr>
    <tr>
      <th>10</th>
      <td>no_serv_tecnicos</td>
      <td>Servicios técnicos solicitados en los últimos ...</td>
    </tr>
    <tr>
      <th>11</th>
      <td>fallo</td>
      <td>Tipo de fallo por elq ue solicitó el servicio ...</td>
    </tr>
    <tr>
      <th>12</th>
      <td>asesoria_factura</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>13</th>
      <td>pedidos_peticiones</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>14</th>
      <td>reagendamiento</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>15</th>
      <td>asesoria_servicios</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>16</th>
      <td>retencion</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>17</th>
      <td>Otras</td>
      <td>Si la variable asume el valor de 1 es que el c...</td>
    </tr>
    <tr>
      <th>18</th>
      <td>quejas_fraude</td>
      <td>El cliente presentó una queja de posible fraud...</td>
    </tr>
    <tr>
      <th>19</th>
      <td>traslado</td>
      <td>El cliente solicitó un traslado del servicio</td>
    </tr>
    <tr>
      <th>20</th>
      <td>Incumplimiento_pago</td>
      <td>Variable objetivo, el 1 indica que el cliente ...</td>
    </tr>
    <tr>
      <th>21</th>
      <td>cliente_id</td>
      <td>Identificador del cliente</td>
    </tr>
  </tbody>
</table>
</div>

```python
#visualizando la info completa de la DESCRIPCIÓN:
descripcion=[]
for row in range(len(df_var)):
    descripcion.append(df_var['DESCRIPCIÓN'][row])
descripcion
```

    ['regional de instalación del servicio',
     'Departamento de instalación del servicio',
     'Tipo de tecnología del servicio',
     'Gerencia de venta del servicio',
     'Canal de venta del servicio',
     'Tipo de fuerza o equipo que realizó la venta',
     'Estrato donde fue instalado el servicio',
     'Antigüedad del cliente al momento de solicitar el servicio',
     'Productos solicitados TV= televisión, TO=Telefonía, BA = Internet',
     'cantidad de productos adquiridos',
     'Servicios técnicos solicitados en los últimos 3 meses',
     'Tipo de fallo por elq ue solicitó el servicio técnico',
     'Si la variable asume el valor de 1 es que el cliente llamo a pedir una asesoría en factura',
     'Si la variable asume el valor de 1 es que el cliente llamo a hacer una petición queja o reclamo',
     'Si la variable asume el valor de 1 es que el cliente llamo a reagendar un servicio',
     'Si la variable asume el valor de 1 es que el cliente llamo a pedir una asesoría en el uso del servicio',
     'Si la variable asume el valor de 1 es que el cliente llamo a cancelar voluntariamente el servicio',
     'Si la variable asume el valor de 1 es que el cliente llamo por otros motivos',
     'El cliente presentó una queja de posible fraude por suplantación de identidad',
     'El cliente solicitó un traslado del servicio',
     'Variable objetivo, el 1 indica que el cliente no pago la primera factura',
     'Identificador del cliente']

```python
#Esta es la base de datos como tal con la cual se va a trabajar:
pd.options.display.max_columns = None # para visualizar todas las columnas (variables) de interés
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>REGIONAL</th>
      <th>DEPARTAMENTO</th>
      <th>TECNOL</th>
      <th>GERENCIA</th>
      <th>CANAL_HOMOLOGADO_MILLICON</th>
      <th>tipo_fuerza_venta</th>
      <th>estrato</th>
      <th>antiguedad_meses</th>
      <th>productos</th>
      <th>portafolio</th>
      <th>no_serv_tecnicos</th>
      <th>fallo</th>
      <th>asesoria_factura</th>
      <th>pedidos_peticiones</th>
      <th>reagendamiento</th>
      <th>asesoria_servicios</th>
      <th>retencion</th>
      <th>Otras</th>
      <th>quejas_fraude</th>
      <th>traslado</th>
      <th>Incumplimiento_pago</th>
      <th>cliente_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>19.0</td>
      <td>TV+BA</td>
      <td>Duo</td>
      <td>1.0</td>
      <td>No funciona línea telefónica</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004369760</td>
    </tr>
    <tr>
      <th>1</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>19.0</td>
      <td>TV+BA</td>
      <td>Duo</td>
      <td>1.0</td>
      <td>No funciona línea telefónica</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004369760</td>
    </tr>
    <tr>
      <th>2</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>1</td>
      <td>1.0</td>
      <td>TV</td>
      <td>Individual</td>
      <td>1.0</td>
      <td>No navega</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004371304</td>
    </tr>
    <tr>
      <th>3</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>IN BOUND</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>7.0</td>
      <td>TO+TV+BA</td>
      <td>Trio</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>1004382867</td>
    </tr>
    <tr>
      <th>4</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>IN BOUND</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>7.0</td>
      <td>TO+TV+BA</td>
      <td>Trio</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>1004382867</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>19937</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>REDCO</td>
      <td>CONTACT</td>
      <td>IN BOUND</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>NaN</td>
      <td>TO+BA</td>
      <td>Duo</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>98589757</td>
    </tr>
    <tr>
      <th>19938</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>REDCO</td>
      <td>CONTACT</td>
      <td>IN BOUND</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>NaN</td>
      <td>TO+BA</td>
      <td>Duo</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>98589757</td>
    </tr>
    <tr>
      <th>19939</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>REDCO</td>
      <td>TIENDAS</td>
      <td>TIENDAS</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>89.0</td>
      <td>BA</td>
      <td>Individual</td>
      <td>1.0</td>
      <td>Sin señal</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>98590638</td>
    </tr>
    <tr>
      <th>19940</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>36.0</td>
      <td>TV</td>
      <td>Individual</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>98591843</td>
    </tr>
    <tr>
      <th>19941</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>REDCO</td>
      <td>CONTACT</td>
      <td>OUT BOUND</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>203.0</td>
      <td>TV</td>
      <td>Individual</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>98594003</td>
    </tr>
  </tbody>
</table>
<p>19942 rows × 22 columns</p>
</div>

```python
#Panorama general de los datos:
df.shape # (num_filas, num_cols)
```

    (19942, 22)

```python
list(df.columns)
```

    ['REGIONAL',
     'DEPARTAMENTO',
     'TECNOL',
     'GERENCIA',
     'CANAL_HOMOLOGADO_MILLICON',
     'tipo_fuerza_venta',
     'estrato',
     'antiguedad_meses',
     'productos',
     'portafolio',
     'no_serv_tecnicos',
     'fallo',
     'asesoria_factura',
     'pedidos_peticiones',
     'reagendamiento',
     'asesoria_servicios',
     'retencion',
     'Otras',
     'quejas_fraude',
     'traslado',
     'Incumplimiento_pago',
     'cliente_id']

¡Efectivamente son 22 columnas (variables) que ya se han mostrado anteriormente con su descripción, recordando que la **Variable objetivo** es `Incumplimiento_pago` (donde el 1 indica que el cliente no pago la primera factura)!

```python
# Data types and amount of no-null values in dataset
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19942 entries, 0 to 19941
    Data columns (total 22 columns):
     #   Column                     Non-Null Count  Dtype
    ---  ------                     --------------  -----
     0   REGIONAL                   19942 non-null  object
     1   DEPARTAMENTO               19942 non-null  object
     2   TECNOL                     19942 non-null  object
     3   GERENCIA                   19942 non-null  object
     4   CANAL_HOMOLOGADO_MILLICON  19942 non-null  object
     5   tipo_fuerza_venta          19942 non-null  object
     6   estrato                    19942 non-null  object
     7   antiguedad_meses           18264 non-null  float64
     8   productos                  19942 non-null  object
     9   portafolio                 19942 non-null  object
     10  no_serv_tecnicos           6909 non-null   float64
     11  fallo                      6909 non-null   object
     12  asesoria_factura           19942 non-null  int64
     13  pedidos_peticiones         19942 non-null  int64
     14  reagendamiento             19942 non-null  int64
     15  asesoria_servicios         19942 non-null  int64
     16  retencion                  19942 non-null  int64
     17  Otras                      19942 non-null  int64
     18  quejas_fraude              19942 non-null  int64
     19  traslado                   19942 non-null  int64
     20  Incumplimiento_pago        19942 non-null  int64
     21  cliente_id                 19942 non-null  int64
    dtypes: float64(2), int64(10), object(10)
    memory usage: 3.3+ MB

Los registros de las variables numericas como `antiguedad_meses` y `no_serv_tecnicos` que no tienen información deben ser reemplazados por NaN

**(The most prudent option which will not alter subsequent summary statistics calculations and not skew the distribution of the non-missing data would be to replace all missing values with a standard NaN)**

Sin embargo, los registros sin información de la variable categórica `fallo` pueden reemplazarse por SE ("Sin Especificar"), similarmente a como lo tiene ya establecido la variable categórica `estrato`

```python
# Another way to see null values per column
df.isnull().sum()
```

    REGIONAL                         0
    DEPARTAMENTO                     0
    TECNOL                           0
    GERENCIA                         0
    CANAL_HOMOLOGADO_MILLICON        0
    tipo_fuerza_venta                0
    estrato                          0
    antiguedad_meses              1678
    productos                        0
    portafolio                       0
    no_serv_tecnicos             13033
    fallo                        13033
    asesoria_factura                 0
    pedidos_peticiones               0
    reagendamiento                   0
    asesoria_servicios               0
    retencion                        0
    Otras                            0
    quejas_fraude                    0
    traslado                         0
    Incumplimiento_pago              0
    cliente_id                       0
    dtype: int64

```python
# Information about numerical columns (descriptive statistics)
df.describe()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>antiguedad_meses</th>
      <th>no_serv_tecnicos</th>
      <th>asesoria_factura</th>
      <th>pedidos_peticiones</th>
      <th>reagendamiento</th>
      <th>asesoria_servicios</th>
      <th>retencion</th>
      <th>Otras</th>
      <th>quejas_fraude</th>
      <th>traslado</th>
      <th>Incumplimiento_pago</th>
      <th>cliente_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>18264.000000</td>
      <td>6909.0</td>
      <td>19942.000000</td>
      <td>19942.00000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>19942.000000</td>
      <td>1.994200e+04</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>43.136060</td>
      <td>1.0</td>
      <td>0.056564</td>
      <td>0.05185</td>
      <td>0.022515</td>
      <td>0.025975</td>
      <td>0.013690</td>
      <td>0.006168</td>
      <td>0.025524</td>
      <td>0.048240</td>
      <td>0.143165</td>
      <td>4.124402e+08</td>
    </tr>
    <tr>
      <th>std</th>
      <td>60.379264</td>
      <td>0.0</td>
      <td>0.231013</td>
      <td>0.22173</td>
      <td>0.148356</td>
      <td>0.159066</td>
      <td>0.116202</td>
      <td>0.078295</td>
      <td>0.157714</td>
      <td>0.214278</td>
      <td>0.350250</td>
      <td>4.993182e+08</td>
    </tr>
    <tr>
      <th>min</th>
      <td>0.000000</td>
      <td>1.0</td>
      <td>0.000000</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>3.978700e+04</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>4.000000</td>
      <td>1.0</td>
      <td>0.000000</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>3.225958e+07</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>16.000000</td>
      <td>1.0</td>
      <td>0.000000</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>6.697511e+07</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>52.000000</td>
      <td>1.0</td>
      <td>0.000000</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>1.036623e+09</td>
    </tr>
    <tr>
      <th>max</th>
      <td>337.000000</td>
      <td>1.0</td>
      <td>1.000000</td>
      <td>1.00000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>8.160082e+09</td>
    </tr>
  </tbody>
</table>
</div>

De la anterior tabla se concluye que **aproximadamente el 14% (exactamnente 14.3165%)** de los clientes (**aprox 2855 de 19942 clientes en total**) está incumpliendo el pago (no ha pagado la primera factura)

```python
0.143165*19942
```

    2854.9964299999997

```python
# Information about categorical columns
df.describe(include = ['O'])
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>REGIONAL</th>
      <th>DEPARTAMENTO</th>
      <th>TECNOL</th>
      <th>GERENCIA</th>
      <th>CANAL_HOMOLOGADO_MILLICON</th>
      <th>tipo_fuerza_venta</th>
      <th>estrato</th>
      <th>productos</th>
      <th>portafolio</th>
      <th>fallo</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>19942</td>
      <td>6909</td>
    </tr>
    <tr>
      <th>unique</th>
      <td>6</td>
      <td>22</td>
      <td>4</td>
      <td>6</td>
      <td>11</td>
      <td>2</td>
      <td>7</td>
      <td>7</td>
      <td>3</td>
      <td>20</td>
    </tr>
    <tr>
      <th>top</th>
      <td>NOROCCIDENTE</td>
      <td>ANTIOQUIA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>2</td>
      <td>TO+TV+BA</td>
      <td>Trio</td>
      <td>No navega</td>
    </tr>
    <tr>
      <th>freq</th>
      <td>11640</td>
      <td>11655</td>
      <td>15628</td>
      <td>10075</td>
      <td>4768</td>
      <td>18893</td>
      <td>6413</td>
      <td>7933</td>
      <td>7933</td>
      <td>2845</td>
    </tr>
  </tbody>
</table>
</div>

```python
df['fallo'].value_counts() #SE= Sin Especificar
```

    No navega                         2845
    Sin señal                         1307
    Servicio intermitente             1238
    No funciona línea telefónica       573
    Mala calidad en la imagen          206
    Navegacion Lenta                   184
    Problemas Control Remoto           160
    Falla Masiva                        76
    Línea con ruido                     70
    Problemas wifi                      67
    Solicitud configuracion tecni       63
    Fallas en audio/subtitulos          30
    Problemas paquetes adicionales      23
    No salen llamadas a X destino       20
    Problemas con el portal             16
    Problemas Servicios Especiales       9
    Ingresa llamada de otra línea        9
    Mala calidad en la voz               5
    Problemas de correo remoto           5
    Retención Clientes                   3
    Name: fallo, dtype: int64

El fallo reportado mas común es "No navega".

```python
#Contemos por ejemplo los fallos asociados a un tipo de tecnología por departamento:
df.groupby(["TECNOL","fallo","DEPARTAMENTO"])["DEPARTAMENTO"].count().reset_index(name="count").sort_values(by="count", ascending = False).reset_index(drop=True)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>TECNOL</th>
      <th>fallo</th>
      <th>DEPARTAMENTO</th>
      <th>count</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>HFC</td>
      <td>No navega</td>
      <td>ANTIOQUIA</td>
      <td>1222</td>
    </tr>
    <tr>
      <th>1</th>
      <td>HFC</td>
      <td>Sin señal</td>
      <td>ANTIOQUIA</td>
      <td>694</td>
    </tr>
    <tr>
      <th>2</th>
      <td>REDCO</td>
      <td>No navega</td>
      <td>ANTIOQUIA</td>
      <td>625</td>
    </tr>
    <tr>
      <th>3</th>
      <td>HFC</td>
      <td>Servicio intermitente</td>
      <td>ANTIOQUIA</td>
      <td>569</td>
    </tr>
    <tr>
      <th>4</th>
      <td>HFC</td>
      <td>No funciona línea telefónica</td>
      <td>ANTIOQUIA</td>
      <td>353</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>184</th>
      <td>HFC</td>
      <td>Problemas con el portal</td>
      <td>SANTANDER</td>
      <td>1</td>
    </tr>
    <tr>
      <th>185</th>
      <td>GPON</td>
      <td>Línea con ruido</td>
      <td>ANTIOQUIA</td>
      <td>1</td>
    </tr>
    <tr>
      <th>186</th>
      <td>HFC</td>
      <td>Problemas Control Remoto</td>
      <td>ATLANTICO</td>
      <td>1</td>
    </tr>
    <tr>
      <th>187</th>
      <td>GPON</td>
      <td>Falla Masiva</td>
      <td>ANTIOQUIA</td>
      <td>1</td>
    </tr>
    <tr>
      <th>188</th>
      <td>REDCO</td>
      <td>Mala calidad en la voz</td>
      <td>ANTIOQUIA</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
<p>189 rows × 4 columns</p>
</div>

```python
#Generemos una tabla de contingencia entre las variables categoricas DEPARTAMENTO y fallo (permite una visualización de la distribución geográfica de los fallos):
pd.crosstab(df["DEPARTAMENTO"],df["fallo"])
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>fallo</th>
      <th>Falla Masiva</th>
      <th>Fallas en audio/subtitulos</th>
      <th>Ingresa llamada de otra línea</th>
      <th>Línea con ruido</th>
      <th>Mala calidad en la imagen</th>
      <th>Mala calidad en la voz</th>
      <th>Navegacion Lenta</th>
      <th>No funciona línea telefónica</th>
      <th>No navega</th>
      <th>No salen llamadas a X destino</th>
      <th>Problemas Control Remoto</th>
      <th>Problemas Servicios Especiales</th>
      <th>Problemas con el portal</th>
      <th>Problemas de correo remoto</th>
      <th>Problemas paquetes adicionales</th>
      <th>Problemas wifi</th>
      <th>Retención Clientes</th>
      <th>Servicio intermitente</th>
      <th>Sin señal</th>
      <th>Solicitud configuracion tecni</th>
    </tr>
    <tr>
      <th>DEPARTAMENTO</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>ANTIOQUIA</th>
      <td>44</td>
      <td>14</td>
      <td>9</td>
      <td>56</td>
      <td>141</td>
      <td>5</td>
      <td>124</td>
      <td>476</td>
      <td>1862</td>
      <td>9</td>
      <td>123</td>
      <td>7</td>
      <td>6</td>
      <td>0</td>
      <td>16</td>
      <td>35</td>
      <td>3</td>
      <td>800</td>
      <td>935</td>
      <td>49</td>
    </tr>
    <tr>
      <th>ATLANTICO</th>
      <td>3</td>
      <td>3</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>5</td>
      <td>1</td>
      <td>60</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>19</td>
      <td>16</td>
      <td>0</td>
    </tr>
    <tr>
      <th>ATLÁNTICO</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
      <td>0</td>
      <td>7</td>
      <td>46</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>24</td>
      <td>13</td>
      <td>1</td>
    </tr>
    <tr>
      <th>BOLIVAR</th>
      <td>6</td>
      <td>2</td>
      <td>0</td>
      <td>2</td>
      <td>6</td>
      <td>0</td>
      <td>4</td>
      <td>17</td>
      <td>129</td>
      <td>2</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
      <td>48</td>
      <td>53</td>
      <td>5</td>
    </tr>
    <tr>
      <th>CALDAS</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>CAUCA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>5</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>6</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
    </tr>
    <tr>
      <th>CESAR</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
      <td>6</td>
      <td>3</td>
      <td>31</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>9</td>
      <td>18</td>
      <td>0</td>
    </tr>
    <tr>
      <th>CORDOBA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>9</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>5</td>
      <td>7</td>
      <td>0</td>
    </tr>
    <tr>
      <th>CUNDINAMARCA</th>
      <td>12</td>
      <td>0</td>
      <td>0</td>
      <td>8</td>
      <td>13</td>
      <td>0</td>
      <td>11</td>
      <td>24</td>
      <td>333</td>
      <td>2</td>
      <td>10</td>
      <td>0</td>
      <td>4</td>
      <td>0</td>
      <td>3</td>
      <td>5</td>
      <td>0</td>
      <td>125</td>
      <td>102</td>
      <td>6</td>
    </tr>
    <tr>
      <th>HUILA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>LA GUAJIRA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>MAGDALENA</th>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
      <td>4</td>
      <td>0</td>
      <td>3</td>
      <td>2</td>
      <td>14</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>17</td>
      <td>16</td>
      <td>0</td>
    </tr>
    <tr>
      <th>META</th>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>6</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
      <td>3</td>
      <td>7</td>
      <td>0</td>
    </tr>
    <tr>
      <th>NORTE DE SANTANDER</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>9</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>5</td>
      <td>10</td>
      <td>0</td>
    </tr>
    <tr>
      <th>RISARALDA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>SANTANDER</th>
      <td>7</td>
      <td>5</td>
      <td>0</td>
      <td>3</td>
      <td>14</td>
      <td>0</td>
      <td>7</td>
      <td>17</td>
      <td>168</td>
      <td>2</td>
      <td>11</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>5</td>
      <td>0</td>
      <td>85</td>
      <td>49</td>
      <td>2</td>
    </tr>
    <tr>
      <th>SUCRE</th>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>3</td>
      <td>11</td>
      <td>3</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>2</td>
      <td>2</td>
      <td>0</td>
    </tr>
    <tr>
      <th>TOLIMA</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>VALLE DEL CAUCA</th>
      <td>3</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
      <td>20</td>
      <td>0</td>
      <td>16</td>
      <td>17</td>
      <td>166</td>
      <td>2</td>
      <td>7</td>
      <td>2</td>
      <td>4</td>
      <td>3</td>
      <td>3</td>
      <td>10</td>
      <td>0</td>
      <td>96</td>
      <td>73</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>

```python
#numerical variables
num_vars=list(df.describe().columns)
num_vars
```

    ['antiguedad_meses',
     'no_serv_tecnicos',
     'asesoria_factura',
     'pedidos_peticiones',
     'reagendamiento',
     'asesoria_servicios',
     'retencion',
     'Otras',
     'quejas_fraude',
     'traslado',
     'Incumplimiento_pago',
     'cliente_id']

```python
#de las num_vars hay que remover cliente_id que no aporta al análisis pues es solo un dato para etiquetar a cada cliente con un número identificador:
num_vars=num_vars.copy()[:-1]
print(num_vars)
```

    ['antiguedad_meses', 'no_serv_tecnicos', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado', 'Incumplimiento_pago']

```python
#categorical variables
cat_vars=list(df.describe(include = ['O']).columns)
cat_vars
```

    ['REGIONAL',
     'DEPARTAMENTO',
     'TECNOL',
     'GERENCIA',
     'CANAL_HOMOLOGADO_MILLICON',
     'tipo_fuerza_venta',
     'estrato',
     'productos',
     'portafolio',
     'fallo']

```python
# Unique values per column excluding null
df.nunique()
```

    REGIONAL                         6
    DEPARTAMENTO                    22
    TECNOL                           4
    GERENCIA                         6
    CANAL_HOMOLOGADO_MILLICON       11
    tipo_fuerza_venta                2
    estrato                          7
    antiguedad_meses               232
    productos                        7
    portafolio                       3
    no_serv_tecnicos                 1
    fallo                           20
    asesoria_factura                 2
    pedidos_peticiones               2
    reagendamiento                   2
    asesoria_servicios               2
    retencion                        2
    Otras                            2
    quejas_fraude                    2
    traslado                         2
    Incumplimiento_pago              2
    cliente_id                   12106
    dtype: int64

```python
# Unique values per column including null
df.nunique(dropna=False)
```

    REGIONAL                         6
    DEPARTAMENTO                    22
    TECNOL                           4
    GERENCIA                         6
    CANAL_HOMOLOGADO_MILLICON       11
    tipo_fuerza_venta                2
    estrato                          7
    antiguedad_meses               233
    productos                        7
    portafolio                       3
    no_serv_tecnicos                 2
    fallo                           21
    asesoria_factura                 2
    pedidos_peticiones               2
    reagendamiento                   2
    asesoria_servicios               2
    retencion                        2
    Otras                            2
    quejas_fraude                    2
    traslado                         2
    Incumplimiento_pago              2
    cliente_id                   12106
    dtype: int64

### Generando algunas graficas estadísticas de algunas variables independientes (X) frente a la variable objetico o de interés (Y):

```python
#plotting the histogram of the numerical variables:
sns.histplot(df[num_vars[10]])
plt.title('Histograma de la variable objetivo (Inclumplimiento_pago: binaria 0 no, 1 si)')
plt.show()
```

Con el anterior histograma se comprueba que el **14.3165% de los clientes (aprox 2855 de 19942 clientes en total** está incumpliendo el pago (no ha pagado la primera factura)

Now that we have looked at the variables of interest in isolation, it makes sense to look at them in relation to `Inclumplimiento_pago`:

```python
#Inspecting Inclumplimiento_pago against another variable of interest (e.g antiguedad_meses):
plt.figure(figsize=(50, 10))
sns.boxplot(x = num_vars[0], y=num_vars[10], data = df)
title_string = "Boxplot of " + num_vars[10] + " vs. " + num_vars[0]
plt.ylabel(num_vars[10])
plt.title(title_string)
```

```python
# We can look out for trends using a line plot
plt.figure(figsize=(15, 8))
ax = sns.lineplot(
    x=num_vars[0],
    y=num_vars[10],
    data=df,
)
```

```python
plt.figure(figsize=(15, 8))
ax = sns.lineplot(
    x=cat_vars[6],
    y=num_vars[10],
    data=df,
)
```

```python
plt.figure(figsize=(15, 8))
ax = sns.lineplot(
    x=cat_vars[8],
    y=num_vars[10],
    data=df,
)
```

## Pre-processing our data

Now that we have an idea of what our dataset consists of, let's transform it so that we can display phase. The types of activities we may engage in during **pre-processing** include:

1. **Deleting columns**
2. **Enriching (or Transforming)** a data set, adding newly calculated columns in the indices
3. **Filtering** a subset of the rows or columns of a dataset according to some criteria
4. **Indexing** a dataset
5. **Aggregating** data
6. **Sorting** the rows of a data set according to some criteria
7. **Merging** the data
8. **Pivoting** so that data that was originally laid out vertically is laid out horizontally (increasing the number of columns) or vice versa (increasing the number of rows)

among others.

## What is data transformation?

Many times in real life, you will be working with imperfect datasets with quality issues. **Data transformation** is the process of modifying a dataset in appropriate ways in order to eliminate these quality issues. Some of these activities include:

- Splitting columns
- Converting dates to `datetime` objects, which are far more easily manipulable using `pandas` libraries
- Encoding categorical variables
- Dealing with and replacing null or missing values
- Creating unique identifiers

The `pandas` library has many functions which can help with this task. In addition, you will also be using some other standard libraries like `String`, `base64`, and `sklearn`.

```python
#Let's create a copy of our dataframe before we start changing it so we can refer back to the original values if necessary.
df_orig = df.copy()
df_orig.head(1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>REGIONAL</th>
      <th>DEPARTAMENTO</th>
      <th>TECNOL</th>
      <th>GERENCIA</th>
      <th>CANAL_HOMOLOGADO_MILLICON</th>
      <th>tipo_fuerza_venta</th>
      <th>estrato</th>
      <th>antiguedad_meses</th>
      <th>productos</th>
      <th>portafolio</th>
      <th>no_serv_tecnicos</th>
      <th>fallo</th>
      <th>asesoria_factura</th>
      <th>pedidos_peticiones</th>
      <th>reagendamiento</th>
      <th>asesoria_servicios</th>
      <th>retencion</th>
      <th>Otras</th>
      <th>quejas_fraude</th>
      <th>traslado</th>
      <th>Incumplimiento_pago</th>
      <th>cliente_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>19.0</td>
      <td>TV+BA</td>
      <td>Duo</td>
      <td>1.0</td>
      <td>No funciona línea telefónica</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004369760</td>
    </tr>
  </tbody>
</table>
</div>

Los registros de las variables numericas como `antiguedad_meses` y `no_serv_tecnicos` que no tienen información deben ser reemplazados por NaN

**(The most prudent option which will not alter subsequent summary statistics calculations and not skew the distribution of the non-missing data would be to replace all missing values with a standard NaN)**

Sin embargo, los registros sin información de la variable categórica `fallo` pueden reemplazarse por SE ("Sin Especificar"), similarmente a como lo tiene ya establecido la variable categórica `estrato`

```python
df['antiguedad_meses'].fillna(np.nan, inplace=True)
```

```python
df['no_serv_tecnicos'].fillna(np.nan, inplace=True)
```

```python
df['fallo'].fillna('SE', inplace=True) #SE=Sin Especificar (similar a cuando no hay info de Estrato)
```

```python
df.isnull().sum()
```

    REGIONAL                         0
    DEPARTAMENTO                     0
    TECNOL                           0
    GERENCIA                         0
    CANAL_HOMOLOGADO_MILLICON        0
    tipo_fuerza_venta                0
    estrato                          0
    antiguedad_meses              1678
    productos                        0
    portafolio                       0
    no_serv_tecnicos             13033
    fallo                            0
    asesoria_factura                 0
    pedidos_peticiones               0
    reagendamiento                   0
    asesoria_servicios               0
    retencion                        0
    Otras                            0
    quejas_fraude                    0
    traslado                         0
    Incumplimiento_pago              0
    cliente_id                       0
    dtype: int64

Una vez finalizado el tratamiento de valores faltantes y/o nulos, procedemos con los siguientes pasos del EDA:

Exploremos las **correlaciones** de las variables numericas con la variable objetivo (num_vars[10]=`Incumplimiento_pago`):

```python
# Create a correlation matrix
corr    = df[num_vars].corr()
pos_cor = corr[num_vars[10]] >0
neg_cor = corr[num_vars[10]] <0
corr[num_vars[10]][pos_cor].sort_values(ascending = False)
#This prints out the coefficients that are positively correlated with Incumplimiento_pago:
```

    Incumplimiento_pago    1.000000
    Otras                  0.011687
    quejas_fraude          0.010103
    Name: Incumplimiento_pago, dtype: float64

```python
corr[num_vars[10]][neg_cor].sort_values(ascending = False)
#This prints out the coefficients that are negatively correlated with Incumplimiento_pago:
```

    pedidos_peticiones   -0.000021
    traslado             -0.006498
    reagendamiento       -0.008957
    retencion            -0.012425
    asesoria_servicios   -0.019946
    asesoria_factura     -0.021996
    antiguedad_meses     -0.150507
    Name: Incumplimiento_pago, dtype: float64

De las resultados anteriores se concluye que:

- Hay **mas** incumplimiento de pago cuando los clientes llaman por otros motivos y hay quejas de fraude
- Existe **menos** incumplimiento de pago a medida que aumenta la antiguedad en meses

```python
# subdividiendo la columna producto en tres para trabajar los servicios de forma diferenciada:
df[['productoTO','productoTV','productoBA']] =  df.productos.str.split('+',expand=True,)
```

```python
df[['productoTO','productoTV','productoBA']]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>productoTO</th>
      <th>productoTV</th>
      <th>productoBA</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>TV</td>
      <td>BA</td>
      <td>None</td>
    </tr>
    <tr>
      <th>1</th>
      <td>TV</td>
      <td>BA</td>
      <td>None</td>
    </tr>
    <tr>
      <th>2</th>
      <td>TV</td>
      <td>None</td>
      <td>None</td>
    </tr>
    <tr>
      <th>3</th>
      <td>TO</td>
      <td>TV</td>
      <td>BA</td>
    </tr>
    <tr>
      <th>4</th>
      <td>TO</td>
      <td>TV</td>
      <td>BA</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>19937</th>
      <td>TO</td>
      <td>BA</td>
      <td>None</td>
    </tr>
    <tr>
      <th>19938</th>
      <td>TO</td>
      <td>BA</td>
      <td>None</td>
    </tr>
    <tr>
      <th>19939</th>
      <td>BA</td>
      <td>None</td>
      <td>None</td>
    </tr>
    <tr>
      <th>19940</th>
      <td>TV</td>
      <td>None</td>
      <td>None</td>
    </tr>
    <tr>
      <th>19941</th>
      <td>TV</td>
      <td>None</td>
      <td>None</td>
    </tr>
  </tbody>
</table>
<p>19942 rows × 3 columns</p>
</div>

```python
df['productoTO'][19939]
```

    'BA'

```python
df.loc[19939,'productoTO']='valor'
```

```python
df['productoTO'][19939]
```

    'valor'

Con lo anterior, se procede a hacer un proceso similar a una **codificación one-hot** de las tres variables categóricas (`productoTO`,`productoTV`,`productoBA`) que se extrajeron de la columna `productos`. Solo queda repartir el valor adecuado en la correspondiente columna (TO, TV, BA) y asignar para cada columna ya organizada, el valor de 1 en caso de que haya producto y 0 cuando no haya:

```python
#usando unas columnas auxiliares en primera instancia:
df['O']=''
df['V']=''
df['A']=''

```

```python
#organizando los productos así: TO, TV, BA (tipo one-hot-encoding)
cols=['productoTO','productoTV','productoBA']

for row in range(0, len(df)):
  for col in cols:
      if df[col][row]=='TO':
        df.loc[row,'O']=1
      elif df[col][row]=='TV':
        df.loc[row,'V']=1
      elif df[col][row]=='BA':
        df.loc[row,'A']=1

```

```python
df[['O','V','A']]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>O</th>
      <th>V</th>
      <th>A</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td></td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td></td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>19937</th>
      <td>1</td>
      <td></td>
      <td>1</td>
    </tr>
    <tr>
      <th>19938</th>
      <td>1</td>
      <td></td>
      <td>1</td>
    </tr>
    <tr>
      <th>19939</th>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <th>19940</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
    <tr>
      <th>19941</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
  </tbody>
</table>
<p>19942 rows × 3 columns</p>
</div>

```python
df[['productoTO','productoTV','productoBA']] = df[['O','V','A']]
df[['productoTO','productoTV','productoBA']]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>productoTO</th>
      <th>productoTV</th>
      <th>productoBA</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td></td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td></td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>19937</th>
      <td>1</td>
      <td></td>
      <td>1</td>
    </tr>
    <tr>
      <th>19938</th>
      <td>1</td>
      <td></td>
      <td>1</td>
    </tr>
    <tr>
      <th>19939</th>
      <td></td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <th>19940</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
    <tr>
      <th>19941</th>
      <td></td>
      <td>1</td>
      <td></td>
    </tr>
  </tbody>
</table>
<p>19942 rows × 3 columns</p>
</div>

```python
df['productoTO'][0]
```

    ''

```python
#ya tenemos los valores de 1, ahora falta poner 0 donde esté vacío (''):
cols=['productoTO','productoTV','productoBA']
for row in range(0, len(df)):
  for col in cols:
      if df[col][row]=='':
        df.loc[row,col]=0

```

```python
df[['productoTO','productoTV','productoBA']] #esta es la versión final tipo one-hot-encoding
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>productoTO</th>
      <th>productoTV</th>
      <th>productoBA</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>19937</th>
      <td>1</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>19938</th>
      <td>1</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>19939</th>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>19940</th>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>19941</th>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<p>19942 rows × 3 columns</p>
</div>

```python
df.head(1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>REGIONAL</th>
      <th>DEPARTAMENTO</th>
      <th>TECNOL</th>
      <th>GERENCIA</th>
      <th>CANAL_HOMOLOGADO_MILLICON</th>
      <th>tipo_fuerza_venta</th>
      <th>estrato</th>
      <th>antiguedad_meses</th>
      <th>productos</th>
      <th>portafolio</th>
      <th>no_serv_tecnicos</th>
      <th>fallo</th>
      <th>asesoria_factura</th>
      <th>pedidos_peticiones</th>
      <th>reagendamiento</th>
      <th>asesoria_servicios</th>
      <th>retencion</th>
      <th>Otras</th>
      <th>quejas_fraude</th>
      <th>traslado</th>
      <th>Incumplimiento_pago</th>
      <th>cliente_id</th>
      <th>productoTO</th>
      <th>productoTV</th>
      <th>productoBA</th>
      <th>O</th>
      <th>V</th>
      <th>A</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>19.0</td>
      <td>TV+BA</td>
      <td>Duo</td>
      <td>1.0</td>
      <td>No funciona línea telefónica</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004369760</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td></td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.columns
```

    Index(['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA',
           'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato',
           'antiguedad_meses', 'productos', 'portafolio', 'no_serv_tecnicos',
           'fallo', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento',
           'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado',
           'Incumplimiento_pago', 'cliente_id', 'productoTO', 'productoTV',
           'productoBA', 'O', 'V', 'A'],
          dtype='object')

```python
new_cols=['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA',
       'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato',
       'antiguedad_meses', 'productos', 'portafolio', 'no_serv_tecnicos',
       'fallo', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento',
       'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado',
       'Incumplimiento_pago', 'cliente_id', 'productoTO', 'productoTV',
       'productoBA']
```

```python
df=df[new_cols].copy() #finalmente nos quedamos con las columnas necesarias para el posterior análisis sin tener en cuenta las auxiliares (repetidas)
df.head(1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>REGIONAL</th>
      <th>DEPARTAMENTO</th>
      <th>TECNOL</th>
      <th>GERENCIA</th>
      <th>CANAL_HOMOLOGADO_MILLICON</th>
      <th>tipo_fuerza_venta</th>
      <th>estrato</th>
      <th>antiguedad_meses</th>
      <th>productos</th>
      <th>portafolio</th>
      <th>no_serv_tecnicos</th>
      <th>fallo</th>
      <th>asesoria_factura</th>
      <th>pedidos_peticiones</th>
      <th>reagendamiento</th>
      <th>asesoria_servicios</th>
      <th>retencion</th>
      <th>Otras</th>
      <th>quejas_fraude</th>
      <th>traslado</th>
      <th>Incumplimiento_pago</th>
      <th>cliente_id</th>
      <th>productoTO</th>
      <th>productoTV</th>
      <th>productoBA</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>COSTA</td>
      <td>MAGDALENA</td>
      <td>HFC</td>
      <td>CONTACT</td>
      <td>SAC</td>
      <td>INDIRECTA</td>
      <td>3</td>
      <td>19.0</td>
      <td>TV+BA</td>
      <td>Duo</td>
      <td>1.0</td>
      <td>No funciona línea telefónica</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1004369760</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>

```python
print(num_vars)
```

    ['antiguedad_meses', 'no_serv_tecnicos', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado', 'Incumplimiento_pago']

```python
print(cat_vars)
```

    ['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA', 'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato', 'productos', 'portafolio', 'fallo']

```python
#redefiniendo las nuevas vars categoricas (no considerar productos como un todo sino las tres tipo one-hot encoding)
new_cat_vars=['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA', 'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato', 'portafolio', 'fallo', 'productoTO',	'productoTV',	'productoBA']
print(new_cat_vars)
```

    ['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA', 'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato', 'portafolio', 'fallo', 'productoTO', 'productoTV', 'productoBA']

# Observación:

Para mi caso personal, por motivos de tiempo limitado y dado que el tratamiento de la información se está haciendo con las herramientas proporcionadas por Python (pandas, numpy, etc.) en este cuaderno de Jupyter y usando los recursos de Google Collab, no es necesario que la base de datos quede por ejemplo con todas sus variables y valores en minúsculas, sin tildes, sin caracterés especiales, etc, ya que no se va a usar SQL (tampoco se está trabajando en este caso con procesamiento de texto - si dispongo de tiempo, aunque lo dudo, trabajaré en NLP con Tweets de Teleperformance). Sin embargo,  es deseable e importante que la base de datos quede de forma apropiada para poner los modelos de ML en producción con el fin de que consuma info de la base de datos y finalmente se muestren los resultados en un Front End específico como Dash o  PowerBI, entre otros.

2. **Construya un modelo estadístico que calcule la probabilidad de que un cliente no pague la primera factura. Explique por qué escogió las variables con las que va a trabajar y si debió hacer modificaciones de estas.**

Teniendo en cuenta los resultados del análisis anterior (punto 1), procederé a construir el modelo estadístico que será **Regresión Logística**.

Para referencia, en estos dos trabajos hay información interesante y pernitente para este caso:

[1]https://bibdigital.epn.edu.ec/bitstream/15000/9194/3/CD-6105.pdf

[2]https://repository.eafit.edu.co/bitstream/handle/10784/12870/Adriana_SalamancaArias_JohnAlejandro_BenitezUrrea_2018.pdf?sequence=2

Según [2], *"los modelos logísticos son apropiados para medir la probabilidad de incumplimiento que enfrentan las empresas del sector real, al tener en cuenta su versatilidad para determinar rangos múltiples de la variable dependiente de manera ordenada, porque trabaja con distribución probabilística que permite que con poca información se puedan obtener resultados interesantes con respecto a la probabilidad de incumplimiento"*

```python
df.index
```

    RangeIndex(start=0, stop=19942, step=1)

```python
100*(13033/19942)

```

    65.35452813158159

```python
#para facilitar el análisis, no consideraré la columna no_serv_tecnicos debido a la poca información que tiene (solo el 35% puesto que el 65% son NULL) reemplazando los NaN por su valor promedio en 'antiguedad_meses'
df['antiguedad_meses'].fillna(df['antiguedad_meses'].mean(), inplace=True)
df['no_serv_tecnicos'].fillna(df['no_serv_tecnicos'].mean(), inplace=True) #lo hago solo por llenarla de la misma manera pero esa columna no se considerará
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19942 entries, 0 to 19941
    Data columns (total 25 columns):
     #   Column                     Non-Null Count  Dtype
    ---  ------                     --------------  -----
     0   REGIONAL                   19942 non-null  object
     1   DEPARTAMENTO               19942 non-null  object
     2   TECNOL                     19942 non-null  object
     3   GERENCIA                   19942 non-null  object
     4   CANAL_HOMOLOGADO_MILLICON  19942 non-null  object
     5   tipo_fuerza_venta          19942 non-null  object
     6   estrato                    19942 non-null  object
     7   antiguedad_meses           19942 non-null  float64
     8   productos                  19942 non-null  object
     9   portafolio                 19942 non-null  object
     10  no_serv_tecnicos           19942 non-null  float64
     11  fallo                      19942 non-null  object
     12  asesoria_factura           19942 non-null  int64
     13  pedidos_peticiones         19942 non-null  int64
     14  reagendamiento             19942 non-null  int64
     15  asesoria_servicios         19942 non-null  int64
     16  retencion                  19942 non-null  int64
     17  Otras                      19942 non-null  int64
     18  quejas_fraude              19942 non-null  int64
     19  traslado                   19942 non-null  int64
     20  Incumplimiento_pago        19942 non-null  int64
     21  cliente_id                 19942 non-null  int64
     22  productoTO                 19942 non-null  object
     23  productoTV                 19942 non-null  object
     24  productoBA                 19942 non-null  object
    dtypes: float64(2), int64(10), object(13)
    memory usage: 3.8+ MB

```python
df['estrato'].head(20)
```

    0      3
    1      3
    2      1
    3      3
    4      3
    5      3
    6     SE
    7     SE
    8      3
    9     SE
    10    SE
    11    SE
    12    SE
    13    SE
    14    SE
    15     4
    16     4
    17     4
    18     2
    19    SE
    Name: estrato, dtype: object

```python
#similarmente en 'estrato' cambio 'SE' por 0 (para luego hacerlo por la parte entera de su valor promedio):
for i in df.index:
  if df['estrato'][i]=='SE':
      df['estrato'][i]=0
```

```python
df['estrato'].head(20)
```

    0     3
    1     3
    2     1
    3     3
    4     3
    5     3
    6     0
    7     0
    8     3
    9     0
    10    0
    11    0
    12    0
    13    0
    14    0
    15    4
    16    4
    17    4
    18    2
    19    0
    Name: estrato, dtype: object

```python
df['estrato']=pd.to_numeric(df['estrato'])
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 19942 entries, 0 to 19941
    Data columns (total 25 columns):
     #   Column                     Non-Null Count  Dtype
    ---  ------                     --------------  -----
     0   REGIONAL                   19942 non-null  object
     1   DEPARTAMENTO               19942 non-null  object
     2   TECNOL                     19942 non-null  object
     3   GERENCIA                   19942 non-null  object
     4   CANAL_HOMOLOGADO_MILLICON  19942 non-null  object
     5   tipo_fuerza_venta          19942 non-null  object
     6   estrato                    19942 non-null  int64
     7   antiguedad_meses           19942 non-null  float64
     8   productos                  19942 non-null  object
     9   portafolio                 19942 non-null  object
     10  no_serv_tecnicos           19942 non-null  float64
     11  fallo                      19942 non-null  object
     12  asesoria_factura           19942 non-null  int64
     13  pedidos_peticiones         19942 non-null  int64
     14  reagendamiento             19942 non-null  int64
     15  asesoria_servicios         19942 non-null  int64
     16  retencion                  19942 non-null  int64
     17  Otras                      19942 non-null  int64
     18  quejas_fraude              19942 non-null  int64
     19  traslado                   19942 non-null  int64
     20  Incumplimiento_pago        19942 non-null  int64
     21  cliente_id                 19942 non-null  int64
     22  productoTO                 19942 non-null  object
     23  productoTV                 19942 non-null  object
     24  productoBA                 19942 non-null  object
    dtypes: float64(2), int64(11), object(12)
    memory usage: 3.8+ MB

```python
int(df['estrato'].mean())
```

    2

```python
#similarmente en 'estrato' cambio 'SE' por la parte entera de su valor promedio:
for i in df.index:
  if df['estrato'][i]==0:
      df['estrato'][i]=int(df['estrato'].mean())

df['estrato'].head(20)
```

    0     3
    1     3
    2     1
    3     3
    4     3
    5     3
    6     2
    7     2
    8     3
    9     2
    10    2
    11    2
    12    2
    13    2
    14    2
    15    4
    16    4
    17    4
    18    2
    19    2
    Name: estrato, dtype: int64

```python
#recordemos el conteo de la var objetivo:
sns.countplot(x='Incumplimiento_pago', data = df)
plt.title("Incumplimiento_pago")
```

```python
#Relación: Incumplimiento_pago (Y) vs antiguedad_meses(X)
sns.jointplot(df.antiguedad_meses, df.Incumplimiento_pago, kind="hex")
```

```python
print(num_vars)
```

    ['antiguedad_meses', 'no_serv_tecnicos', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado', 'Incumplimiento_pago']

```python
#Y='Incumplimiento_pago' (var DEPENDIENTE)

#redefiniendo las var INDEPENDIENTES verdaderamente numericas (porque las binarias serían categóricas) y que serán insumo para el modelo:
num_vars_def=['antiguedad_meses', 'estrato'] #SOLO QUEDAN 2
```

```python
print(new_cat_vars)
```

    ['REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA', 'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'estrato', 'portafolio', 'fallo', 'productoTO', 'productoTV', 'productoBA']

NOTA:

Por simplicidad, las siguientes var cat no se consideran relevantes para el modelo:
'REGIONAL', 'DEPARTAMENTO', 'TECNOL', 'GERENCIA', 'CANAL_HOMOLOGADO_MILLICON', 'tipo_fuerza_venta', 'portafolio' (está correlacionada con 'productos' que tampoco se considera pues se separó en tres var cat binarias que si se van a considerar en su reemplazo)

```python
#en consecuencia, redefiniendo las var verdaderamente categóricas y que serán insumo para el modelo:
cat_vars_def=['productoTO', 'productoTV', 'productoBA', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado']#QUEDAN 11
```

Veamos la matriz de correlación entre las var numericas definitivas (covariables) y la variable objetivo:

```python
num_vars_def=['antiguedad_meses', 'estrato']
```

```python
cols=num_vars_def.copy()
cols.append('Incumplimiento_pago')
print(cols)
```

    ['antiguedad_meses', 'estrato', 'Incumplimiento_pago']

```python
#Visualize the correlation matrix across all numerical features by using the sns.heatmap() command:

#compute correlation matrix

Data=df[cols].copy()

df_correlations = Data.corr()

#mask the upper half for visualization purposes
mask = np.zeros_like(df_correlations, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Draw the heatmap with the mask and correct aspect ratio
plt.figure(figsize= (10,10))

cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(df_correlations,mask=mask,  vmax=1, vmin=-1, cmap=cmap,
            center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5});
```

Construyendo el modelo predictivo:

...iniciando con un modelo de regresión logística estándard...

Using the `LogisticRegression()` function from `scikit-learn`, let me write a function named `fit_logistic_regression(X,y)` that fits a logistic regression on the array of covariates `X` and associated response variable `y`.

```python
from sklearn.linear_model import LogisticRegression
def fit_logistic_regression(X,y):
    """
    fit a logistic regression with feature matrix X and binary output y
    """
    clf = LogisticRegression(solver='lbfgs', tol=10**-4,
                             fit_intercept=True,
                             multi_class='multinomial').fit(X,y)
    return clf
```

Let me create a basic [logistic regression model](https://towardsdatascience.com/logistic-regression-detailed-overview-46c4da4303bc) for predicting `Incumplimiento_pago` with only one feature: `antiguedad_meses`.  I will call this model `model1`, using a 70/30 train-test split of the data.

```python
# we will use a 70%/30% split for training/validation
Data=df.copy()
n_total = len(Data)
n_train = int(0.7*n_total)

X, y = Data[["antiguedad_meses"]], Data.Incumplimiento_pago
X_train, y_train = X[:n_train], y[:n_train]
X_test, y_test = X[n_train:], y[n_train:]
```

```python
model1 = fit_logistic_regression(X_train, y_train)  # fit a logistic regression
y_test_pred = model1.predict_proba(X_test)[:,1]     # make probabilistic predictions on test set
```

Plotting the [ROC curve](https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5) of `model1` and finding the area under the curve:

```python
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import StratifiedKFold
```

```python
fpr, tpr, _ = roc_curve(y_test, y_test_pred)  #compute FPR/TPR
auc_baseline = auc(fpr, tpr) # compute AUC

plt.plot(fpr, tpr, "b-", label="AUC(basline)={:2.2f}".format(auc_baseline))
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.legend(fontsize=15)
plt.plot([0,1], [0,1], "r--")
plt.title("ROC curve -- Baseline Model");
```

Of course this should not be the final model. This is because I have not explored the contribution from other variables, which in addition to containing valuable information could also be confounding the perceived effect of `antiguedad_meses` on the response variable `Incumplimiento_pago`. This under-exploitation of information is called [**underfitting**](https://towardsdatascience.com/what-are-overfitting-and-underfitting-in-machine-learning-a96b30864690).

On the other hand:

Let's instead put all the variables available in the model, so that we are maximally leveraging our available info. This is also a bad idea. If we *blindly* use all of the variables in our model fitting, a phenomenon called [**overfitting**](https://towardsdatascience.com/what-are-overfitting-and-underfitting-in-machine-learning-a96b30864690) occurs. This is when a statistical model "fits" too closely to a particular set of data, which may well be noisy and exhibit randomness and therefore fail to predict future, different observations reliably.

In most cases, you will be working with datasets with many features that each have their own distribution. Generally, a large amount of time is spent on feature selection with many models being trained during this time. It is extremely rare that you simply plug all the features in and tune it once to get the optimal model.

There are many different techniques associated with feature selection and a comprehensive look into all of them is outside the scope of this case. For simplicity, I will demonstrate model training and testing on single-feature models and then directly move into multi-feature models to show the numerous possible cases you may encounter.

In reality, I would apply cross-validation on numerous subsets of features based on domain knowledge of the dataset to see which set of features truly optimizes the model I am trying to create.

[**Cross-validation**](https://towardsdatascience.com/why-and-how-to-cross-validate-a-model-d6424b45261f) is a set of techniques for assessing how well the results of a model will generalize to an out-of-sample dataset; i.e. in practice or production. It is chiefly used to flag overfitting.

```python
skf = StratifiedKFold(n_splits=5)
for k, (train_index, test_index) in enumerate( skf.split(X, y) ):
    plt.plot(train_index, [k+1 for _ in train_index], ".")
plt.ylim(0,6)
plt.ylabel("FOLD")
plt.title("CROSS VALIDATION FOLDS")
```

The following code defines a function `compute_AUC(X, y, train_index, test_index)` that computes the AUC of a model trained on "train_index" and tested in "test_index".

```python
def compute_AUC(X, y, train_index, test_index):
    """
    feature/output: X, y
    dataset split: train_index, test_index
    """
    X_train, y_train = X.iloc[train_index], y.iloc[train_index]
    X_test, y_test = X.iloc[test_index], y.iloc[test_index]

    clf = fit_logistic_regression(X_train, y_train)
    default_proba_test = clf.predict_proba(X_test)[:,1]
    fpr, tpr, _ = roc_curve(y_test, default_proba_test)
    auc_score = auc(fpr, tpr)
    return auc_score, fpr, tpr
```

With the help of the `compute_AUC` function defined above, let me write a function `cross_validation_AUC(X,y,nfold)` that carries out a 10-fold cross-validation and returns a list which contains the area under the curve for each fold of the cross-validation:

```python
def cross_validation_AUC(X,y, nfold=10):
    """
    use a n-fold cross-validation for computing AUC estimates
    """
    skf = StratifiedKFold(n_splits=nfold)  #create a cross-validation splitting
    auc_list = [] #this list will contain the AUC estimates associated with each fold
    for k, (train_index, test_index) in enumerate( skf.split(X, y) ):
        auc_score, _, _ = compute_AUC(X, y, train_index, test_index)
        auc_list.append(auc_score)
    return auc_list
```

I will now estimate and compare, through cross-validation analysis, the performance of all the "simple models" that only use one numerical features as input.

```python
print(num_vars_def)
```

    ['antiguedad_meses', 'estrato']

Let's compute cross-validation estimates of the AUC for each single-feature model:

```python
model_perf = pd.DataFrame({}) #this data-frame will contain the AUC estimates
for key in num_vars_def:
    X_full, y_full = Data[[key]], Data.Incumplimiento_pago
    auc_list = cross_validation_AUC(X_full, y_full, nfold=10)
    model_perf["SIMPLE:" + key] = auc_list
```

Let me construct a [boxplot](https://towardsdatascience.com/understanding-boxplots-5e2df7bcbd51) which shows the distribution of cross-validation scores of each variable (remember, each variable has 10 total scores):

```python
def plot_boxplot_ordered(df_model):
    """
    display a list of boxplot, ordered by the media values
    """
    df = df_model[df_model.median().sort_values().index]
    sns.boxplot(x="variable", y="value", data=pd.melt(df), showfliers=False)
    plt.xticks(rotation=90)
```

```python
plt.figure(figsize= (10,5))
plot_boxplot_ordered(model_perf)
plt.xlabel("Predictive Model with a Single Predictive Feature")
plt.ylabel("AUC")
```

According to what have been done so far, from the above picture I can conclude:

- The feature that has the highest predictive power is `antiguedad_meses`
- The feature that has the lowest predictive power is `estrato`

Let me consider the model that consists of using *all* the numerical features (and none of the categorical features). Carrying out a 10-fold cross-validation analysis to determine whether this model has better predictive performance than the best single-feature model. Using the boxplot method again as I did before:

```python
X_full, y_full = Data[num_vars_def], Data.Incumplimiento_pago
auc_list = cross_validation_AUC(X_full, y_full)
model_perf["ALL_NUMERICAL"] = auc_list
model_perf
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>SIMPLE:antiguedad_meses</th>
      <th>SIMPLE:estrato</th>
      <th>ALL_NUMERICAL</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.625454</td>
      <td>0.429581</td>
      <td>0.612515</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.592383</td>
      <td>0.462943</td>
      <td>0.580514</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.559453</td>
      <td>0.495811</td>
      <td>0.547094</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.674187</td>
      <td>0.497961</td>
      <td>0.669534</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.690632</td>
      <td>0.521252</td>
      <td>0.694829</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.581867</td>
      <td>0.494196</td>
      <td>0.575197</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.659101</td>
      <td>0.552888</td>
      <td>0.650529</td>
    </tr>
    <tr>
      <th>7</th>
      <td>0.671121</td>
      <td>0.533505</td>
      <td>0.672244</td>
    </tr>
    <tr>
      <th>8</th>
      <td>0.680397</td>
      <td>0.504706</td>
      <td>0.677434</td>
    </tr>
    <tr>
      <th>9</th>
      <td>0.642052</td>
      <td>0.509290</td>
      <td>0.644907</td>
    </tr>
  </tbody>
</table>
</div>

```python
plt.figure(figsize= (10,5))
plot_boxplot_ordered(model_perf)
plt.xlabel("Predictive Model with a Single Predictive Feature")
plt.ylabel("AUC")
```

I see that the combined model does perform better than the best single-feature model. Thus, I will move forward with it for the rest of this case. Note, however, that best practice would entail iteratively adding features to the best single-feature model until I reach a point where there is no significant improvement, as opposed to throwing all the features in at once. Let me advise and consider to take this more cautious approach when building models (it is a matter of time)

## Incorporating categorical variables:

```python
cat_vars_def # se podría decir que estas vars ya están codificadas tipo one-hot para irse añadiendo al modelo numérico anterior
```

    ['productoTO',
     'productoTV',
     'productoBA',
     'asesoria_factura',
     'pedidos_peticiones',
     'reagendamiento',
     'asesoria_servicios',
     'retencion',
     'Otras',
     'quejas_fraude',
     'traslado']

```python
Data.value_counts()
sns.countplot(x='productoTO', data = Data)
plt.xticks(rotation=90)
```

Let me investigate whether the categorical variable `productoTO` brings any predictive value when added to the current best model (remember again that the  encoding scheme of one-hot type is already there:

```python
plt.figure(figsize= (20,5))

df_TO_incump = Data[["Incumplimiento_pago", "productoTO"]].groupby("productoTO").mean()
df_TO_incump = df_TO_incump.sort_values(by="Incumplimiento_pago",axis=0, ascending=False)
sns.barplot(x=df_TO_incump.index[:50],
            y=df_TO_incump["Incumplimiento_pago"][:50].values,
            orient="v")
plt.xticks(rotation=90)

plt.ylabel("Probabilidad de Incumplimiento_pago")
plt.title("Incumplimiento_pago por productoTO", fontsize=20, verticalalignment='bottom')
```

```python
print(num_vars_def)
```

    ['antiguedad_meses', 'estrato']

```python
new_cols=['antiguedad_meses', 'estrato','productoTO','productoTV','productoBA']

X_full_productos, y_full = Data[new_cols], Data.Incumplimiento_pago
auc_list = cross_validation_AUC(X_full_productos, y_full)
model_perf["ALL_NUMERICAL_WITH_productos"] = auc_list
model_perf
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>SIMPLE:antiguedad_meses</th>
      <th>SIMPLE:estrato</th>
      <th>ALL_NUMERICAL</th>
      <th>ALL_NUMERICAL_WITH_productos</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.625454</td>
      <td>0.429581</td>
      <td>0.612515</td>
      <td>0.694596</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.592383</td>
      <td>0.462943</td>
      <td>0.580514</td>
      <td>0.627549</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.559453</td>
      <td>0.495811</td>
      <td>0.547094</td>
      <td>0.587298</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.674187</td>
      <td>0.497961</td>
      <td>0.669534</td>
      <td>0.706750</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.690632</td>
      <td>0.521252</td>
      <td>0.694829</td>
      <td>0.727204</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.581867</td>
      <td>0.494196</td>
      <td>0.575197</td>
      <td>0.706316</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.659101</td>
      <td>0.552888</td>
      <td>0.650529</td>
      <td>0.745785</td>
    </tr>
    <tr>
      <th>7</th>
      <td>0.671121</td>
      <td>0.533505</td>
      <td>0.672244</td>
      <td>0.728415</td>
    </tr>
    <tr>
      <th>8</th>
      <td>0.680397</td>
      <td>0.504706</td>
      <td>0.677434</td>
      <td>0.778226</td>
    </tr>
    <tr>
      <th>9</th>
      <td>0.642052</td>
      <td>0.509290</td>
      <td>0.644907</td>
      <td>0.724648</td>
    </tr>
  </tbody>
</table>
</div>

```python
plt.figure(figsize= (10,5))
plot_boxplot_ordered(model_perf)
plt.xlabel("Predictive Model with a Single Predictive Feature")
plt.ylabel("AUC")
```

The difference appears significant as the boxplot for the updated model is almost completely non-overlapping with that of the previous model.

#To finish:

Let me use the rest of the "cat_vars" (they are indeed numerical: binary):

```python
print(cat_vars_def)
```

    ['productoTO', 'productoTV', 'productoBA', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado']

```python
pqrs=['asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado']
```

```python
new_cols_def=['antiguedad_meses', 'estrato','productoTO','productoTV','productoBA', 'asesoria_factura', 'pedidos_peticiones', 'reagendamiento', 'asesoria_servicios', 'retencion', 'Otras', 'quejas_fraude', 'traslado']

X_full_productos_pqrs, y_full = Data[new_cols_def], Data.Incumplimiento_pago
auc_list = cross_validation_AUC(X_full_productos_pqrs, y_full)
model_perf["ALL_NUMERICAL_WITH_productos_pqrs"] = auc_list
model_perf
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>SIMPLE:antiguedad_meses</th>
      <th>SIMPLE:estrato</th>
      <th>ALL_NUMERICAL</th>
      <th>ALL_NUMERICAL_WITH_productos</th>
      <th>ALL_NUMERICAL_WITH_productos_pqrs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.625454</td>
      <td>0.429581</td>
      <td>0.612515</td>
      <td>0.694596</td>
      <td>0.695431</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.592383</td>
      <td>0.462943</td>
      <td>0.580514</td>
      <td>0.627549</td>
      <td>0.624117</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.559453</td>
      <td>0.495811</td>
      <td>0.547094</td>
      <td>0.587298</td>
      <td>0.588349</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.674187</td>
      <td>0.497961</td>
      <td>0.669534</td>
      <td>0.706750</td>
      <td>0.706378</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.690632</td>
      <td>0.521252</td>
      <td>0.694829</td>
      <td>0.727204</td>
      <td>0.722031</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.581867</td>
      <td>0.494196</td>
      <td>0.575197</td>
      <td>0.706316</td>
      <td>0.706624</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.659101</td>
      <td>0.552888</td>
      <td>0.650529</td>
      <td>0.745785</td>
      <td>0.745685</td>
    </tr>
    <tr>
      <th>7</th>
      <td>0.671121</td>
      <td>0.533505</td>
      <td>0.672244</td>
      <td>0.728415</td>
      <td>0.723830</td>
    </tr>
    <tr>
      <th>8</th>
      <td>0.680397</td>
      <td>0.504706</td>
      <td>0.677434</td>
      <td>0.778226</td>
      <td>0.781072</td>
    </tr>
    <tr>
      <th>9</th>
      <td>0.642052</td>
      <td>0.509290</td>
      <td>0.644907</td>
      <td>0.724648</td>
      <td>0.730572</td>
    </tr>
  </tbody>
</table>
</div>

```python
plt.figure(figsize= (10,5))
plot_boxplot_ordered(model_perf)
plt.xlabel("Predictive Model with a Single Predictive Feature")
plt.ylabel("AUC")
```

## Conclusions

Once I started building models, I started with very simple logistic regressions approaches – these baseline models were useful for quickly evaluating the predictive power of each individual variable. Next, I employed cross-validation approaches for building more complex models, often exploiting the interactions between the different features. Since the dataset contains a large number of covariates, using cross-validation was revealed to be crucial for avoiding overfitting, choosing the correct number of features and ultimately choosing an appropriate model that balanced complexity with accuracy.

Cross-validation is a robust and flexible technique for evaluating the predictive performance of statistical models. It is especially useful in big data settings where the number of features is large compared to the number of observations. When used appropriately, cross-validation is a powerful method for choosing a model with the correct complexity and best predictive performance. Remember that logistic regression is only one of many classification algorithms and the principles behind cross-validation are not limited to this case alone.

*Quiero finalmente comentar que 2 días para mi no son suficientes para terminar completa y exitosamente este caso. Con mas tiempo, seguro lo terminaría, aunque voy a seguir trabajando en el caso apesar de que el tiempo para entregarlo ya se terminó. Muchas gracias por la oprtunidad y también deseo seguir trabajando en el caso opcional donde tenía pensado trabajar con NLP (Tweets-TP)*

**Observación:**

Todo lo trabajado aquí está referenciado al *programa de Ciencia de Datos (DS4A-Colombia 3.0) ofertado por el Ministerio TIC de Colombia en convenio con la Compañía Correlation One.*

Todo el material puede ser encontrado aquí:

https://drive.google.com/drive/folders/1mGiM3lWtdkszSIrv-wpJjftZ_2qWbMNt?usp=sharing