# Analysis for the Telco company about their customers who churn

The Telco company have many customers whi churn every month, so the executives of the company asked us to conduct an analysis to determine what causes they are losing customers. Their marketing and customer service team are really interested by the output of the analysis. they provided a dataset of 7000 clients

```
#import libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from matplotlib.colors import ListedColormap
%matplotlib inline
```

```
df_Client = pd.read_csv("Telco_Customer_Churn.csv")
df_Client.head()
```

      customerID
      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      ...
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      7590-VHVEG
      Female
      0
      Yes
      No
      1
      No
      No phone service
      DSL
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      29.85
      29.85
      No

      1
      5575-GNVDE
      Male
      0
      No
      No
      34
      Yes
      No
      DSL
      Yes
      ...
      Yes
      No
      No
      No
      One year
      No
      Mailed check
      56.95
      1889.5
      No

      2
      3668-QPYBK
      Male
      0
      No
      No
      2
      Yes
      No
      DSL
      Yes
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Mailed check
      53.85
      108.15
      Yes

      3
      7795-CFOCW
      Male
      0
      No
      No
      45
      No
      No phone service
      DSL
      Yes
      ...
      Yes
      Yes
      No
      No
      One year
      No
      Bank transfer (automatic)
      42.30
      1840.75
      No

      4
      9237-HQITU
      Female
      0
      No
      No
      2
      Yes
      No
      Fiber optic
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      70.70
      151.65
      Yes

5 rows × 21 columns

### Analysis of the dataset

##### Variable Target : Churn

```
df_Client['Churn']
```

    0        No
    1        No
    2       Yes
    3        No
    4       Yes
           ...
    7038     No
    7039     No
    7040     No
    7041    Yes
    7042     No
    Name: Churn, Length: 7043, dtype: object

#### shape of our dataset

```
df_Client.shape
```

    (7043, 21)

#### Data types of our dataset

```
df_Client.dtypes
```

    customerID           object
    gender               object
    SeniorCitizen         int64
    Partner              object
    Dependents           object
    tenure                int64
    PhoneService         object
    MultipleLines        object
    InternetService      object
    OnlineSecurity       object
    OnlineBackup         object
    DeviceProtection     object
    TechSupport          object
    StreamingTV          object
    StreamingMovies      object
    Contract             object
    PaperlessBilling     object
    PaymentMethod        object
    MonthlyCharges      float64
    TotalCharges         object
    Churn                object
    dtype: object

#### Viualisation of the data types of our dataset

```
df_Client.dtypes.value_counts().plot.pie()
```

#### let's convert the dtype of the feature TotalCharges to float

```
df_Client["TotalCharges"] = pd.to_numeric(df_Client["TotalCharges"],errors='coerce')
df_Client["TotalCharges"].dtype
```

    dtype('float64')

### Analysis of the dataset to identify the missing values

```
missing_values = df_Client.isnull()
missing_values.head(5)
```

      customerID
      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      ...
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      1
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      2
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      3
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      4
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

5 rows × 21 columns

```
for column in missing_values.columns.values.tolist():
    print(column)
    print (missing_values[column].value_counts())
    print("")
```

    customerID
    False    7043
    Name: customerID, dtype: int64

    gender
    False    7043
    Name: gender, dtype: int64

    SeniorCitizen
    False    7043
    Name: SeniorCitizen, dtype: int64

    Partner
    False    7043
    Name: Partner, dtype: int64

    Dependents
    False    7043
    Name: Dependents, dtype: int64

    tenure
    False    7043
    Name: tenure, dtype: int64

    PhoneService
    False    7043
    Name: PhoneService, dtype: int64

    MultipleLines
    False    7043
    Name: MultipleLines, dtype: int64

    InternetService
    False    7043
    Name: InternetService, dtype: int64

    OnlineSecurity
    False    7043
    Name: OnlineSecurity, dtype: int64

    OnlineBackup
    False    7043
    Name: OnlineBackup, dtype: int64

    DeviceProtection
    False    7043
    Name: DeviceProtection, dtype: int64

    TechSupport
    False    7043
    Name: TechSupport, dtype: int64

    StreamingTV
    False    7043
    Name: StreamingTV, dtype: int64

    StreamingMovies
    False    7043
    Name: StreamingMovies, dtype: int64

    Contract
    False    7043
    Name: Contract, dtype: int64

    PaperlessBilling
    False    7043
    Name: PaperlessBilling, dtype: int64

    PaymentMethod
    False    7043
    Name: PaymentMethod, dtype: int64

    MonthlyCharges
    False    7043
    Name: MonthlyCharges, dtype: int64

    TotalCharges
    False    7043
    Name: TotalCharges, dtype: int64

    Churn
    False    7043
    Name: Churn, dtype: int64

#### from the previous command we can see there's no missing values but let's now visualize the missing values if exist

```
plt.figure(figsize=(10,5))
sns.heatmap(df_Client.isnull())
```

#### now we can be sure that the dataset doesn't contain missing values

#### Total of the customers who churn this month

```
churn_counts = df_Client['Churn'].value_counts().to_frame()
churn_counts.rename(columns={'Churn': 'value_counts'}, inplace=True)
churn_counts.index.name = 'Churn'
churn_counts.head(10)
```

      value_counts

      Churn

      No
      5174

      Yes
      1869

#### In Percent

```
df_Client['Churn'].value_counts(normalize="True")
```

    No     0.73463
    Yes    0.26537
    Name: Churn, dtype: float64

#### 1869 customers churn this month which represents 26,54% of the dataset

```
size = df_Client["Churn"].value_counts(sort=True)
color =["yellow","green"]
explode=(0.02,0.02)
plt.pie(size,colors=color, explode=explode,autopct='%1.1f%%',startangle=320)
plt.title("Visualization of Churn in percentage")
plt.show()
```

## what impact the churn has on the Company?

we saw above how many customers left the company, let's now see how much money the company lost

```
df_churn = df_Client[['Churn','MonthlyCharges']].groupby(['Churn']).MonthlyCharges.sum().to_frame().reset_index()
df_churn
```

      Churn
      MonthlyCharges

      0
      No
      316985.75

      1
      Yes
      139130.85

#### In percentage

```
df_churn = df_Client[['Churn','MonthlyCharges']].groupby(['Churn']).MonthlyCharges.sum().to_frame()/df_Client['MonthlyCharges'].sum()
df_churn
```

      MonthlyCharges

      Churn

      No
      0.694966

      Yes
      0.305034

##### the company has lost 139.130,85$ which represents 31% of their monthly charges.

Now that we have determined how many churn the company and its impact. Now we are going to explain churn by some features of our dataset

Explain churn by the below categories:

 Customer demographics like age and gender

Services used

 Billing information

#### Churn explain by Customer demographic

```
demographic = df_Client[["gender","SeniorCitizen","Partner","Dependents","Churn"]]
demographic
```

      gender
      SeniorCitizen
      Partner
      Dependents
      Churn

      0
      Female
      0
      Yes
      No
      No

      1
      Male
      0
      No
      No
      No

      2
      Male
      0
      No
      No
      Yes

      3
      Male
      0
      No
      No
      No

      4
      Female
      0
      No
      No
      Yes

      ...
      ...
      ...
      ...
      ...
      ...

      7038
      Male
      0
      Yes
      Yes
      No

      7039
      Female
      0
      Yes
      Yes
      No

      7040
      Female
      0
      Yes
      Yes
      No

      7041
      Male
      1
      Yes
      No
      Yes

      7042
      Male
      0
      No
      No
      No

7043 rows × 5 columns

```
demographic[["gender","SeniorCitizen","Partner","Dependents","Churn"]].groupby(["gender","SeniorCitizen","Partner","Dependents","Churn"]).size().to_frame().rename(columns ={0:"count"}).reset_index()
```

      gender
      SeniorCitizen
      Partner
      Dependents
      Churn
      count

      0
      Female
      0
      No
      No
      No
      909

      1
      Female
      0
      No
      No
      Yes
      429

      2
      Female
      0
      No
      Yes
      No
      110

      3
      Female
      0
      No
      Yes
      Yes
      32

      4
      Female
      0
      Yes
      No
      No
      482

      5
      Female
      0
      Yes
      No
      Yes
      114

      6
      Female
      0
      Yes
      Yes
      No
      720

      7
      Female
      0
      Yes
      Yes
      Yes
      124

      8
      Female
      1
      No
      No
      No
      159

      9
      Female
      1
      No
      No
      Yes
      158

      10
      Female
      1
      No
      Yes
      No
      2

      11
      Female
      1
      No
      Yes
      Yes
      1

      12
      Female
      1
      Yes
      No
      No
      136

      13
      Female
      1
      Yes
      No
      Yes
      73

      14
      Female
      1
      Yes
      Yes
      No
      31

      15
      Female
      1
      Yes
      Yes
      Yes
      8

      16
      Male
      0
      No
      No
      No
      963

      17
      Male
      0
      No
      No
      Yes
      418

      18
      Male
      0
      No
      Yes
      No
      168

      19
      Male
      0
      No
      Yes
      Yes
      43

      20
      Male
      0
      Yes
      No
      No
      439

      21
      Male
      0
      Yes
      No
      Yes
      128

      22
      Male
      0
      Yes
      Yes
      No
      717

      23
      Male
      0
      Yes
      Yes
      Yes
      105

      24
      Male
      1
      No
      No
      No
      126

      25
      Male
      1
      No
      No
      Yes
      118

      26
      Male
      1
      No
      Yes
      No
      4

      27
      Male
      1
      No
      Yes
      Yes
      1

      28
      Male
      1
      Yes
      No
      No
      176

      29
      Male
      1
      Yes
      No
      Yes
      105

      30
      Male
      1
      Yes
      Yes
      No
      32

      31
      Male
      1
      Yes
      Yes
      Yes
      12

```
fig,axes =plt.subplots(2,2,figsize =(20,15))
plt.subplots_adjust(hspace = 0.5)
sns.set(style="darkgrid")
ax1 = sns.countplot(x="gender", data=demographic,palette="Set2", hue="Churn" ,ax=axes[0,0])
ax2  = sns.countplot(x="SeniorCitizen" , data=demographic,palette="Set2", hue="Churn", ax=axes[0,1])
ax3  = sns.countplot(x="Partner" , data=demographic,palette="Set2", hue="Churn",ax=axes[1,0])
ax4  = sns.countplot(x="Dependents" , data=demographic,palette="Set2", hue="Churn",ax=axes[1,1])

ax1.set_title("Gender")
ax2.set_title("SeniorCitizen")
ax3.set_title("Partner")
ax4.set_title("Dependents")

```

by the charts above we can see that gender hasn't an impact on churn but we can notice that customers who are not SeniorCitizen churn more than the others. the third figure shows that customers without partner tend to churn the most and about the last figure we can say that customers without dependents churn.

#### Churn explain by Service Used

```
ServiceUsed = df_Client[["PhoneService","MultipleLines","InternetService","OnlineSecurity","DeviceProtection","TechSupport","StreamingTV","StreamingMovies","Churn"]]
ServiceUsed
```

      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Churn

      0
      0
      0
      1
      0
      0
      0
      0
      0
      0

      1
      1
      0
      1
      1
      1
      0
      0
      0
      0

      2
      1
      0
      1
      1
      0
      0
      0
      0
      1

      3
      0
      0
      1
      1
      1
      1
      0
      0
      0

      4
      1
      0
      2
      0
      0
      0
      0
      0
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      7038
      1
      1
      1
      1
      1
      1
      1
      1
      0

      7039
      1
      1
      2
      0
      1
      0
      1
      1
      0

      7040
      0
      0
      1
      1
      0
      0
      0
      0
      0

      7041
      1
      1
      2
      0
      0
      0
      0
      0
      1

      7042
      1
      0
      2
      1
      1
      1
      1
      1
      0

7043 rows × 9 columns

```
fig,axes =plt.subplots(2,2,figsize =(20,15))
plt.subplots_adjust(hspace = 0.5)
sns.set(style="darkgrid")
ax5 = sns.countplot(x="PhoneService", data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[0,0])
ax5.set_title("PhoneService")

ax6  = sns.countplot(x="MultipleLines" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[0,1])
ax6.set_title("MultiplineLines")

ax7  = sns.countplot(x="InternetService" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[1,0])
ax7.set_title("InternetService")

ax8  = sns.countplot(x="OnlineSecurity" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[1,1])
ax8.set_title("OnlineSecurity")

fig,axes =plt.subplots(2,2,figsize =(20,15))
plt.subplots_adjust(hspace = 0.5)
sns.set(style="darkgrid")
ax9  = sns.countplot(x="DeviceProtection" , data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[0,0])
ax9.set_title("DeviceProtection")

ax10  = sns.countplot(x="TechSupport" , data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[0,1])
ax10.set_title("TechSupport")

ax11  = sns.countplot(x="StreamingTV" , data=ServiceUsed,palette="Set2",hue="Churn" ,ax=axes[1,0])
ax11.set_title("StreamingTV")

ax12  = sns.countplot(x="StreamingMovies" , data=ServiceUsed,palette="Set2", hue ="Churn" ,ax=axes[1,1])
ax12.set_title("StreamingMovies")

```

#### Churn explain by Billing Information

```
billing_information = df_Client[["Contract","PaperlessBilling","PaymentMethod","Churn"]]
billing_information
```

      Contract
      PaperlessBilling
      PaymentMethod
      Churn

      0
      Month-to-month
      Yes
      Electronic check
      No

      1
      One year
      No
      Mailed check
      No

      2
      Month-to-month
      Yes
      Mailed check
      Yes

      3
      One year
      No
      Bank transfer (automatic)
      No

      4
      Month-to-month
      Yes
      Electronic check
      Yes

      ...
      ...
      ...
      ...
      ...

      7038
      One year
      Yes
      Mailed check
      No

      7039
      One year
      Yes
      Credit card (automatic)
      No

      7040
      Month-to-month
      Yes
      Electronic check
      No

      7041
      Month-to-month
      Yes
      Mailed check
      Yes

      7042
      Two year
      Yes
      Bank transfer (automatic)
      No

7043 rows × 4 columns

```
billing_information[["Contract","PaperlessBilling","PaymentMethod","Churn"]].groupby(["Contract","PaperlessBilling","PaymentMethod","Churn"]).size().to_frame().rename(columns ={0:"count"}).reset_index()
```

      Contract
      PaperlessBilling
      PaymentMethod
      Churn
      count

      0
      Month-to-month
      No
      Bank transfer (automatic)
      No
      151

      1
      Month-to-month
      No
      Bank transfer (automatic)
      Yes
      55

      2
      Month-to-month
      No
      Credit card (automatic)
      No
      116

      3
      Month-to-month
      No
      Credit card (automatic)
      Yes
      45

      4
      Month-to-month
      No
      Electronic check
      No
      265

      5
      Month-to-month
      No
      Electronic check
      Yes
      188

      6
      Month-to-month
      No
      Mailed check
      No
      351

      7
      Month-to-month
      No
      Mailed check
      Yes
      118

      8
      Month-to-month
      Yes
      Bank transfer (automatic)
      No
      237

      9
      Month-to-month
      Yes
      Bank transfer (automatic)
      Yes
      146

      10
      Month-to-month
      Yes
      Credit card (automatic)
      No
      249

      11
      Month-to-month
      Yes
      Credit card (automatic)
      Yes
      133

      12
      Month-to-month
      Yes
      Electronic check
      No
      591

      13
      Month-to-month
      Yes
      Electronic check
      Yes
      806

      14
      Month-to-month
      Yes
      Mailed check
      No
      260

      15
      Month-to-month
      Yes
      Mailed check
      Yes
      164

      16
      One year
      No
      Bank transfer (automatic)
      No
      160

      17
      One year
      No
      Bank transfer (automatic)
      Yes
      11

      18
      One year
      No
      Credit card (automatic)
      No
      161

      19
      One year
      No
      Credit card (automatic)
      Yes
      13

      20
      One year
      No
      Electronic check
      No
      100

      21
      One year
      No
      Electronic check
      Yes
      14

      22
      One year
      No
      Mailed check
      No
      204

      23
      One year
      No
      Mailed check
      Yes
      10

      24
      One year
      Yes
      Bank transfer (automatic)
      No
      193

      25
      One year
      Yes
      Bank transfer (automatic)
      Yes
      27

      26
      One year
      Yes
      Credit card (automatic)
      No
      196

      27
      One year
      Yes
      Credit card (automatic)
      Yes
      28

      28
      One year
      Yes
      Electronic check
      No
      183

      29
      One year
      Yes
      Electronic check
      Yes
      50

      30
      One year
      Yes
      Mailed check
      No
      110

      31
      One year
      Yes
      Mailed check
      Yes
      13

      32
      Two year
      No
      Bank transfer (automatic)
      No
      270

      33
      Two year
      No
      Bank transfer (automatic)
      Yes
      6

      34
      Two year
      No
      Credit card (automatic)
      No
      299

      35
      Two year
      No
      Credit card (automatic)
      Yes
      6

      36
      Two year
      No
      Electronic check
      No
      54

      37
      Two year
      No
      Electronic check
      Yes
      2

      38
      Two year
      No
      Mailed check
      No
      272

      39
      Two year
      No
      Mailed check
      Yes
      1

      40
      Two year
      Yes
      Bank transfer (automatic)
      No
      275

      41
      Two year
      Yes
      Bank transfer (automatic)
      Yes
      13

      42
      Two year
      Yes
      Credit card (automatic)
      No
      269

      43
      Two year
      Yes
      Credit card (automatic)
      Yes
      7

      44
      Two year
      Yes
      Electronic check
      No
      101

      45
      Two year
      Yes
      Electronic check
      Yes
      11

      46
      Two year
      Yes
      Mailed check
      No
      107

      47
      Two year
      Yes
      Mailed check
      Yes
      2

```
fig,axes =plt.subplots(1,3,figsize =(25,5))
plt.subplots_adjust(wspace = 0.2)
sns.set(style="darkgrid")
ax1 = sns.countplot(x="Contract", data=billing_information,palette="Set1", hue="Churn" ,ax=axes[0])
ax2  = sns.countplot(x="PaperlessBilling", data=billing_information,palette="Set1", hue="Churn", ax=axes[1])
ax3  = sns.countplot(x="PaymentMethod", data=billing_information,palette="Set1", hue="Churn",ax=axes[2])

ax1.set_title("Contract")
ax2.set_title("PaperlessBilling")
ax3.set_title("PaymentMethod")
```

 the figures obtained with Billing Information let us to say that Customer with a month-to-month contract, Customers who are paperlessbilling and those who pay by electronic Check are those who churn.

### What services are typically purchased by customers who churned?
### Are Any services especially helpful in retaining customers?

```
fig,axes =plt.subplots(2,2,figsize =(20,15))
plt.subplots_adjust(hspace = 0.5)
sns.set(style="darkgrid")
ax13 = sns.countplot(x="PhoneService", data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[0,0])
ax13.set_title("PhoneService")

ax14  = sns.countplot(x="MultipleLines" , data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[0,1])
ax14.set_title("MultiplineLines")

ax15  = sns.countplot(x="InternetService" , data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[1,0])
ax15.set_title("InternetService")

ax16  = sns.countplot(x="StreamingTV" , data=ServiceUsed,palette="Set2",hue ="Churn" ,ax=axes[1,1])
ax16.set_title("StreamingTV")
```

Phone service,Multiple Lines, Fiber optic as Internet service and Streaming TV. These services are typically used by Churned customers

```
fig,axes =plt.subplots(2,2,figsize =(20,15))
plt.subplots_adjust(hspace = 0.5)
sns.set(style="darkgrid")
ax9  = sns.countplot(x="DeviceProtection" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[0,0])
ax9.set_title("DeviceProtection")

ax10  = sns.countplot(x="TechSupport" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[0,1])
ax10.set_title("TechSupport")

ax8  = sns.countplot(x="OnlineSecurity" , data=ServiceUsed,palette="Set1",hue ="Churn" ,ax=axes[1,0])
ax8.set_title("OnlineSecurity")

ax12  = sns.countplot(x="StreamingMovies" , data=ServiceUsed,palette="Set1", hue ="Churn" ,ax=axes[1,1])
ax12.set_title("StreamingMovies")

```

the services like tech support, Online Security, Streaming movies, device protection tend to be the best services that might be helpful in retaining customers

### Let turn our categorical values into numerical values in order to use our dataset for the classifier

```
cat_val = ['gender','Partner','Dependents','PhoneService','MultipleLines','InternetService','OnlineSecurity','DeviceProtection','TechSupport','StreamingTV','StreamingMovies','Contract','PaperlessBilling','PaymentMethod']
for col in cat_val:
    print(col,df_Client[col].unique())
```

    gender ['Female' 'Male']
    Partner ['Yes' 'No']
    Dependents ['No' 'Yes']
    PhoneService ['No' 'Yes']
    MultipleLines ['No phone service' 'No' 'Yes']
    InternetService ['DSL' 'Fiber optic' 'No']
    OnlineSecurity ['No' 'Yes' 'No internet service']
    DeviceProtection ['No' 'Yes' 'No internet service']
    TechSupport ['No' 'Yes' 'No internet service']
    StreamingTV ['No' 'Yes' 'No internet service']
    StreamingMovies ['No' 'Yes' 'No internet service']
    Contract ['Month-to-month' 'One year' 'Two year']
    PaperlessBilling ['Yes' 'No']
    PaymentMethod ['Electronic check' 'Mailed check' 'Bank transfer (automatic)'
     'Credit card (automatic)']

```
cat_val0 = ['gender']

def clean(data):
    rep = data.replace('Female', '1')
    rep = rep.replace('Male', '0')
    return rep

df_Client[cat_val0] = df_Client[cat_val0].applymap(clean)
```

```
cat_val1 = ['MultipleLines']

def clean1(data):
    rep = data.replace('No phone service', 'No')
    return rep

df_Client[cat_val1] = df_Client[cat_val1].applymap(clean1)
```

```
cat_val2 = ['OnlineSecurity','OnlineBackup','DeviceProtection','TechSupport','StreamingTV','StreamingMovies']

def clean2(data):
    rep = data.replace('No internet service', 'No')
    return rep

df_Client[cat_val2] = df_Client[cat_val2].applymap(clean2)
```

```
cat_val3 = ['Partner','Dependents','PhoneService','MultipleLines','OnlineSecurity','OnlineBackup','DeviceProtection','TechSupport','StreamingTV','StreamingMovies','PaperlessBilling','Churn']
def clean_up(data):
    rep = data.replace('No','0')
    rep = rep.replace('Yes','1')
    return rep

df_Client[cat_val3] = df_Client[cat_val3].applymap(clean_up)
```

```
cat_val2_ = ['OnlineSecurity','DeviceProtection','TechSupport','StreamingTV','StreamingMovies']

def clean2_(data):
    rep = data.replace('0 service', '0')
    return rep

df_Client[cat_val2_] = df_Client[cat_val2_].applymap(clean2_)
```

```
cat_val4 = ['InternetService']
def clean4(data):
    rep = data.replace('No','0')
    rep = rep.replace('DSL','1')
    rep = rep.replace('Fiber optic','2')
    return rep

df_Client[cat_val4] = df_Client[cat_val4].applymap(clean4)
```

```
cat_val5 = ['Contract']
def clean5(data):
    rep = data.replace('Month-to-month','1')
    rep = rep.replace('One year','2')
    rep = rep.replace('Two year','3')
    return rep

df_Client[cat_val5] = df_Client[cat_val5].applymap(clean5)
```

```
cat_val6 = ['PaymentMethod']
def clean6(data):
    rep = data.replace('Electronic check','1')
    rep = rep.replace('Mailed check','2')
    rep = rep.replace('Bank transfer (automatic)','3')
    rep = rep.replace('Credit card (automatic)','4')
    return rep

df_Client[cat_val6] = df_Client[cat_val6].applymap(clean6)
```

```
cat_val_ = ['OnlineBackup']

def clean_(data):
    rep = data.replace('0 internet service', '0')
    return rep

df_Client[cat_val_] = df_Client[cat_val_].applymap(clean_)
```

```
cat_val = ['gender','Partner','Dependents','PhoneService','MultipleLines','InternetService','OnlineSecurity','OnlineBackup','DeviceProtection','TechSupport','StreamingTV','StreamingMovies','Contract','PaperlessBilling','Churn','PaymentMethod']
for col in cat_val:
    print(col,df_Client[col].unique())
```

    gender ['1' '0']
    Partner ['1' '0']
    Dependents ['0' '1']
    PhoneService ['0' '1']
    MultipleLines ['0' '1']
    InternetService ['1' '2' '0']
    OnlineSecurity ['0' '1']
    OnlineBackup ['1' '0']
    DeviceProtection ['0' '1']
    TechSupport ['0' '1']
    StreamingTV ['0' '1']
    StreamingMovies ['0' '1']
    Contract ['1' '2' '3']
    PaperlessBilling ['1' '0']
    Churn ['0' '1']
    PaymentMethod ['1' '2' '3' '4']

```
df_Client.drop(columns=['customerID'], inplace = True)
```

```
df_Client.head()
```

      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      OnlineBackup
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      1
      0
      1
      0
      1
      0
      0
      1
      0
      1
      0
      0
      0
      0
      1
      1
      1
      29.85
      29.85
      0

      1
      0
      0
      0
      0
      34
      1
      0
      1
      1
      0
      1
      0
      0
      0
      2
      0
      2
      56.95
      1889.5
      0

      2
      0
      0
      0
      0
      2
      1
      0
      1
      1
      1
      0
      0
      0
      0
      1
      1
      2
      53.85
      108.15
      1

      3
      0
      0
      0
      0
      45
      0
      0
      1
      1
      0
      1
      1
      0
      0
      2
      0
      3
      42.30
      1840.75
      0

      4
      1
      0
      0
      0
      2
      1
      0
      2
      0
      0
      0
      0
      0
      0
      1
      1
      1
      70.70
      151.65
      1

```
for column in df_Client:
    df_Client[column] = pd.to_numeric(df_Client[column],errors='coerce')
```

```
df_Client.dtypes
```

    gender                int64
    SeniorCitizen         int64
    Partner               int64
    Dependents            int64
    tenure                int64
    PhoneService          int64
    MultipleLines         int64
    InternetService       int64
    OnlineSecurity        int64
    OnlineBackup          int64
    DeviceProtection      int64
    TechSupport           int64
    StreamingTV           int64
    StreamingMovies       int64
    Contract              int64
    PaperlessBilling      int64
    PaymentMethod         int64
    MonthlyCharges      float64
    TotalCharges        float64
    Churn                 int64
    dtype: object

#### Now after turning our categorical values into numeric ones we have a dataset that can be used for prediction

### Let's see if there is any strong correlation between the variables

```
df_Client.corr()
```

      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      OnlineBackup
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      gender
      1.000000
      0.001874
      0.001808
      -0.010517
      -0.005106
      0.006488
      0.008414
      0.010380
      0.017021
      0.013773
      0.002105
      0.009212
      0.008393
      0.010487
      -0.000126
      0.011754
      0.005209
      0.014569
      -0.000048
      0.008612

      SeniorCitizen
      0.001874
      1.000000
      0.016479
      -0.211185
      0.016567
      0.008576
      0.142948
      0.259390
      -0.038653
      0.066572
      0.059428
      -0.060625
      0.105378
      0.120176
      -0.142554
      0.156530
      -0.093704
      0.220173
      0.102411
      0.150889

      Partner
      0.001808
      0.016479
      1.000000
      0.452676
      0.379697
      0.017706
      0.142057
      -0.000132
      0.143106
      0.141498
      0.153786
      0.119999
      0.124666
      0.117412
      0.294806
      -0.014877
      0.133115
      0.096848
      0.319072
      -0.150448

      Dependents
      -0.010517
      -0.211185
      0.452676
      1.000000
      0.159712
      -0.001762
      -0.024526
      -0.179631
      0.080972
      0.023671
      0.013963
      0.063268
      -0.016558
      -0.039741
      0.243187
      -0.111377
      0.123844
      -0.113890
      0.064653
      -0.164221

      tenure
      -0.005106
      0.016567
      0.379697
      0.159712
      1.000000
      0.008448
      0.331941
      0.033230
      0.327203
      0.360277
      0.360653
      0.324221
      0.279756
      0.286111
      0.671607
      0.006152
      0.340305
      0.247900
      0.825880
      -0.352229

      PhoneService
      0.006488
      0.008576
      0.017706
      -0.001762
      0.008448
      1.000000
      0.279690
      0.093720
      -0.092893
      -0.052312
      -0.071227
      -0.096340
      -0.022574
      -0.032959
      0.002247
      0.016505
      -0.004070
      0.247398
      0.113008
      0.011942

      MultipleLines
      0.008414
      0.142948
      0.142057
      -0.024526
      0.331941
      0.279690
      1.000000
      0.344684
      0.098108
      0.202237
      0.201137
      0.100571
      0.257152
      0.258751
      0.107114
      0.163530
      0.035851
      0.490434
      0.469042
      0.040102

      InternetService
      0.010380
      0.259390
      -0.000132
      -0.179631
      0.033230
      0.093720
      0.344684
      1.000000
      0.156799
      0.307420
      0.313603
      0.164833
      0.429707
      0.427264
      -0.290189
      0.378108
      -0.178137
      0.905491
      0.428212
      0.316846

      OnlineSecurity
      0.017021
      -0.038653
      0.143106
      0.080972
      0.327203
      -0.092893
      0.098108
      0.156799
      1.000000
      0.283832
      0.275438
      0.354931
      0.176207
      0.187398
      0.245530
      -0.003636
      0.163367
      0.296594
      0.412619
      -0.171226

      OnlineBackup
      0.013773
      0.066572
      0.141498
      0.023671
      0.360277
      -0.052312
      0.202237
      0.307420
      0.283832
      1.000000
      0.303546
      0.294233
      0.282106
      0.274501
      0.155085
      0.126735
      0.096550
      0.441780
      0.510100
      -0.082255

      DeviceProtection
      0.002105
      0.059428
      0.153786
      0.013963
      0.360653
      -0.071227
      0.201137
      0.313603
      0.275438
      0.303546
      1.000000
      0.333313
      0.390874
      0.402111
      0.219310
      0.103797
      0.111241
      0.482692
      0.522881
      -0.066160

      TechSupport
      0.009212
      -0.060625
      0.119999
      0.063268
      0.324221
      -0.096340
      0.100571
      0.164833
      0.354931
      0.294233
      0.333313
      1.000000
      0.278070
      0.279358
      0.293691
      0.037880
      0.167701
      0.338304
      0.432868
      -0.164674

      StreamingTV
      0.008393
      0.105378
      0.124666
      -0.016558
      0.279756
      -0.022574
      0.257152
      0.429707
      0.176207
      0.282106
      0.390874
      0.278070
      1.000000
      0.533094
      0.103944
      0.223841
      -0.013826
      0.629603
      0.515709
      0.063228

      StreamingMovies
      0.010487
      0.120176
      0.117412
      -0.039741
      0.286111
      -0.032959
      0.258751
      0.427264
      0.187398
      0.274501
      0.402111
      0.279358
      0.533094
      1.000000
      0.107520
      0.211716
      -0.004390
      0.627429
      0.519867
      0.061382

      Contract
      -0.000126
      -0.142554
      0.294806
      0.243187
      0.671607
      0.002247
      0.107114
      -0.290189
      0.245530
      0.155085
      0.219310
      0.293691
      0.103944
      0.107520
      1.000000
      -0.176733
      0.358913
      -0.074195
      0.450306
      -0.396713

      PaperlessBilling
      0.011754
      0.156530
      -0.014877
      -0.111377
      0.006152
      0.016505
      0.163530
      0.378108
      -0.003636
      0.126735
      0.103797
      0.037880
      0.223841
      0.211716
      -0.176733
      1.000000
      -0.101480
      0.352150
      0.157830
      0.191825

      PaymentMethod
      0.005209
      -0.093704
      0.133115
      0.123844
      0.340305
      -0.004070
      0.035851
      -0.178137
      0.163367
      0.096550
      0.111241
      0.167701
      -0.013826
      -0.004390
      0.358913
      -0.101480
      1.000000
      -0.074353
      0.222694
      -0.262818

      MonthlyCharges
      0.014569
      0.220173
      0.096848
      -0.113890
      0.247900
      0.247398
      0.490434
      0.905491
      0.296594
      0.441780
      0.482692
      0.338304
      0.629603
      0.627429
      -0.074195
      0.352150
      -0.074353
      1.000000
      0.651065
      0.193356

      TotalCharges
      -0.000048
      0.102411
      0.319072
      0.064653
      0.825880
      0.113008
      0.469042
      0.428212
      0.412619
      0.510100
      0.522881
      0.432868
      0.515709
      0.519867
      0.450306
      0.157830
      0.222694
      0.651065
      1.000000
      -0.199484

      Churn
      0.008612
      0.150889
      -0.150448
      -0.164221
      -0.352229
      0.011942
      0.040102
      0.316846
      -0.171226
      -0.082255
      -0.066160
      -0.164674
      0.063228
      0.061382
      -0.396713
      0.191825
      -0.262818
      0.193356
      -0.199484
      1.000000

```
figure, ax = plt.subplots( figsize =(25,10))
sns.heatmap (df_Client.corr(), center = 0, cmap ='BrBG', annot = True)
```

From the table and the heatmap above, we notice there is a big correlation between TotalCharges and Tenure
which is correct because the longer someone is with the company the higher the TotalCharges will be. And also
there's a big correlation between the columns MonthlyCharges and Internet Service.

### Now let's split the dataset so we can have our training and testing part in order to build the model we want.

let's first write a function to clean our dataset if there's any Nan or Infinity values

```

def clean_dataset(df):
    assert isinstance(df, pd.DataFrame), "df needs to be a pd.DataFrame"
    df.dropna(inplace=True)
    indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(1)
    return df[indices_to_keep].astype(np.float64)
```

```
clean_dataset(df_Client)
```

      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      OnlineBackup
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      1.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      29.85
      29.85
      0.0

      1
      0.0
      0.0
      0.0
      0.0
      34.0
      1.0
      0.0
      1.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      2.0
      0.0
      2.0
      56.95
      1889.50
      0.0

      2
      0.0
      0.0
      0.0
      0.0
      2.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      2.0
      53.85
      108.15
      1.0

      3
      0.0
      0.0
      0.0
      0.0
      45.0
      0.0
      0.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      2.0
      0.0
      3.0
      42.30
      1840.75
      0.0

      4
      1.0
      0.0
      0.0
      0.0
      2.0
      1.0
      0.0
      2.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      70.70
      151.65
      1.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      7038
      0.0
      0.0
      1.0
      1.0
      24.0
      1.0
      1.0
      1.0
      1.0
      0.0
      1.0
      1.0
      1.0
      1.0
      2.0
      1.0
      2.0
      84.80
      1990.50
      0.0

      7039
      1.0
      0.0
      1.0
      1.0
      72.0
      1.0
      1.0
      2.0
      0.0
      1.0
      1.0
      0.0
      1.0
      1.0
      2.0
      1.0
      4.0
      103.20
      7362.90
      0.0

      7040
      1.0
      0.0
      1.0
      1.0
      11.0
      0.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      29.60
      346.45
      0.0

      7041
      0.0
      1.0
      1.0
      0.0
      4.0
      1.0
      1.0
      2.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      2.0
      74.40
      306.60
      1.0

      7042
      0.0
      0.0
      0.0
      0.0
      66.0
      1.0
      0.0
      2.0
      1.0
      0.0
      1.0
      1.0
      1.0
      1.0
      3.0
      1.0
      3.0
      105.65
      6844.50
      0.0

7032 rows × 20 columns

 Dataset cleaned, 11 rows are dropped

 let's split the dataset, X will store the independents variables while y will keep the dependent one

```
X = df_Client.drop(columns=['Churn'])
y = df_Client['Churn']
```

 training the model

```
X_train, X_test, y_train, y_test = train_test_split (X, y, test_size=0.30, random_state=0)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)
```

    Train set: (4922, 19) (4922,)
    Test set: (2110, 19) (2110,)

 Putting our datasets on the same scale

```
sc = StandardScaler()
X_train = sc.fit_transform (X_train)
X_test = sc.transform (X_test)
X_test
```

    array([[-0.99392338, -0.43362195, -0.96564464, ..., -1.13494209,
            -1.47409441, -0.99678627],
           [-0.99392338, -0.43362195, -0.96564464, ...,  1.46529917,
            -1.47576141, -0.86022192],
           [ 1.00611377, -0.43362195, -0.96564464, ...,  1.46529917,
            -1.5024333 , -0.9971612 ],
           ...,
           [-0.99392338,  2.3061563 ,  1.03557764, ...,  0.59855208,
             0.55797071,  0.23590985],
           [ 1.00611377, -0.43362195,  1.03557764, ...,  0.59855208,
            -1.38574376, -0.28992463],
           [ 1.00611377, -0.43362195,  1.03557764, ...,  0.59855208,
            -1.48576337, -0.4341641 ]])

 Let's build our model using a classification method such as Logistic Regression

```
classifier = LogisticRegression (random_state = 0, solver ="liblinear")
classifier.fit(X_train, y_train)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=0, solver='liblinear', tol=0.0001, verbose=0,
                       warm_start=False)

### Prediction

```
y_predict = classifier.predict(X_test)
```

### Evaluation

```
df_predict = pd.DataFrame({'y_predict': y_predict[:]})
df_predict.head()
```

      y_predict

      0
      0

      1
      0

      2
      0

      3
      1

      4
      1

```
df_y_test = pd.DataFrame({'y_test': y_test[:]})
df_y_test.head()
```

      y_test

      5561
      0

      5814
      0

      2645
      0

      3983
      1

      6438
      1

### let's compare our prediction to our test set

```
val= dict(predict =list(df_predict.y_predict),test=list(df_y_test.y_test))
```

```
df = pd.DataFrame(val)
df.head()
```

      predict
      test

      0
      0
      0

      1
      0
      0

      2
      0
      0

      3
      1
      1

      4
      1
      1

Evaluation of the model using Jaccard Index

```
from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(y_test, y_predict)
```

    C:\Users\jake brown\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:635: DeprecationWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.
      'and multiclass classification tasks.', DeprecationWarning)

    0.8004739336492891

Using Jaccard Index our model is accurate at 80%

## Customer Profile

#### let' s split the dataset between the clients who churned and those who didn't

```
churn_yes = df_Client[df_Client["Churn"] == 1]
churn_no = df_Client[df_Client["Churn"] == 0]
```

```
churn_yes.head()
```

      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      OnlineBackup
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      2
      0
      0
      0
      0
      2
      1
      0
      1
      1
      1
      0
      0
      0
      0
      1
      1
      2
      53.85
      108.15
      1

      4
      1
      0
      0
      0
      2
      1
      0
      2
      0
      0
      0
      0
      0
      0
      1
      1
      1
      70.70
      151.65
      1

      5
      1
      0
      0
      0
      8
      1
      1
      2
      0
      0
      1
      0
      1
      1
      1
      1
      1
      99.65
      820.50
      1

      8
      1
      0
      1
      0
      28
      1
      1
      2
      0
      0
      1
      1
      1
      1
      1
      1
      1
      104.80
      3046.05
      1

      13
      0
      0
      0
      0
      49
      1
      1
      2
      0
      1
      1
      0
      1
      1
      1
      1
      3
      103.70
      5036.30
      1

```
churn_yes[churn_yes.gender == 0].groupby(['gender','SeniorCitizen']).size().to_frame().rename(columns={0:'count'}).apply(lambda r : r/r.sum())
```

      count

      gender
      SeniorCitizen

      0
      0
      0.746237

      1
      0.253763

```
churn_yes[churn_yes.gender == 0].groupby(['gender','PhoneService']).size().to_frame().rename(columns={0:'count'}).apply(lambda r : r/r.sum())
```

      count

      gender
      PhoneService

      0
      0
      0.096774

      1
      0.903226

```
churn_yes[churn_yes.gender == 1].groupby(['gender','SeniorCitizen']).size().to_frame().rename(columns={0:'count'}).apply(lambda r : r/r.sum())
```

      count

      gender
      SeniorCitizen

      1
      0
      0.744409

      1
      0.255591

```
churn_yes[churn_yes.gender == 1].groupby(['gender','SeniorCitizen']).size().to_frame().rename(columns={0:'count'}).apply(lambda r : r/r.sum())
```

```
female =dict(theta=['PhoneService_yes','PhoneService_no','SeniorCitizen_yes','SeniorCitizen_no'],r=[0.255591,0.744409,0.255591,0.744409])
df_female = pd.DataFrame(female)
df_female
```

      theta
      r

      0
      PhoneService_yes
      0.255591

      1
      PhoneService_no
      0.744409

      2
      SeniorCitizen_yes
      0.255591

      3
      SeniorCitizen_no
      0.744409

```
df_Client1 = pd.read_csv("Telco_Customer_Churn.csv")
df_Client1.head()
```

      customerID
      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      ...
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      7590-VHVEG
      Female
      0
      Yes
      No
      1
      No
      No phone service
      DSL
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      29.85
      29.85
      No

      1
      5575-GNVDE
      Male
      0
      No
      No
      34
      Yes
      No
      DSL
      Yes
      ...
      Yes
      No
      No
      No
      One year
      No
      Mailed check
      56.95
      1889.5
      No

      2
      3668-QPYBK
      Male
      0
      No
      No
      2
      Yes
      No
      DSL
      Yes
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Mailed check
      53.85
      108.15
      Yes

      3
      7795-CFOCW
      Male
      0
      No
      No
      45
      No
      No phone service
      DSL
      Yes
      ...
      Yes
      Yes
      No
      No
      One year
      No
      Bank transfer (automatic)
      42.30
      1840.75
      No

      4
      9237-HQITU
      Female
      0
      No
      No
      2
      Yes
      No
      Fiber optic
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      70.70
      151.65
      Yes

5 rows × 21 columns

```
df_Client_cpy = df_Client1.copy()
df_Client_cpy.head()
```

      customerID
      gender
      SeniorCitizen
      Partner
      Dependents
      tenure
      PhoneService
      MultipleLines
      InternetService
      OnlineSecurity
      ...
      DeviceProtection
      TechSupport
      StreamingTV
      StreamingMovies
      Contract
      PaperlessBilling
      PaymentMethod
      MonthlyCharges
      TotalCharges
      Churn

      0
      7590-VHVEG
      Female
      0
      Yes
      No
      1
      No
      No phone service
      DSL
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      29.85
      29.85
      No

      1
      5575-GNVDE
      Male
      0
      No
      No
      34
      Yes
      No
      DSL
      Yes
      ...
      Yes
      No
      No
      No
      One year
      No
      Mailed check
      56.95
      1889.5
      No

      2
      3668-QPYBK
      Male
      0
      No
      No
      2
      Yes
      No
      DSL
      Yes
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Mailed check
      53.85
      108.15
      Yes

      3
      7795-CFOCW
      Male
      0
      No
      No
      45
      No
      No phone service
      DSL
      Yes
      ...
      Yes
      Yes
      No
      No
      One year
      No
      Bank transfer (automatic)
      42.30
      1840.75
      No

      4
      9237-HQITU
      Female
      0
      No
      No
      2
      Yes
      No
      Fiber optic
      No
      ...
      No
      No
      No
      No
      Month-to-month
      Yes
      Electronic check
      70.70
      151.65
      Yes

5 rows × 21 columns

```
col = ['SeniorCitizen','Partner','Dependents','PhoneService','MultipleLines','InternetService']

df1 = pd.get_dummies(df_Client_cpy[col])
```

```
df1['gender'] = df_Client_cpy['gender']
df1['Churn'] = df_Client_cpy['Churn']
```

```
df1 =df1.groupby(['Churn','gender']).sum()
columns = list(df1.columns)
df1 =df1.reset_index()
df2 =df1[df1.Churn == 'Yes']
df2
```

      Churn
      gender
      SeniorCitizen
      Partner_No
      Partner_Yes
      Dependents_No
      Dependents_Yes
      PhoneService_No
      PhoneService_Yes
      MultipleLines_No
      MultipleLines_No phone service
      MultipleLines_Yes
      InternetService_DSL
      InternetService_Fiber optic
      InternetService_No

      2
      Yes
      Female
      240
      620.0
      319.0
      774.0
      165.0
      80.0
      859.0
      443.0
      80.0
      416.0
      219.0
      664.0
      56.0

      3
      Yes
      Male
      236
      580.0
      350.0
      769.0
      161.0
      90.0
      840.0
      406.0
      90.0
      434.0
      240.0
      633.0
      57.0

```
values = list(df2[df2.gender =='Female'][columns].values[0])
values

```

    [240.0,
     620.0,
     319.0,
     774.0,
     165.0,
     80.0,
     859.0,
     443.0,
     80.0,
     416.0,
     219.0,
     664.0,
     56.0]

```
import plotly.express as px
import pandas as pd

fig = px.line_polar(df2, r=values, theta=columns, line_close=True)
fig.show()
```

#### This radar chart above is a customer profile according to the gender women

```
values_ = list(df2[df2.gender =='Male'][columns].values[0])
```

```
import plotly.express as px
import pandas as pd

fig = px.line_polar(df2, r=values_, theta=columns, line_close=True)
fig.show()
```

#### This radar chart above is a customer profile according to the gender women

## BONUS

Knowing that the company cannot have new customers how long it will take to lose all the customers and which demographics will they lose first.

 To calculate how long it will take for the company to lose all the customers we used sympy and library from python.

#### quick explanation of the library sympy
SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.

```
from sympy.solvers import solve
from sympy import symbols, Eq, solve
```

```
Churn_rate = 0.2654
population = 5163
end_population = 0
```

```
x = symbols('x')
```

```
fx_population = Eq(population*(1-Churn_rate)**x)
sol = solve(fx_population,x)
sol
```

    []

```
curr_pop = 5163
day_from_start = 0
while curr_pop>=1:
    curr_pop = curr_pop - (curr_pop*(Churn_rate))
    day_from_start +=1
print (day_from_start)
```

    28

@presented by Max-Havel JACOB


# Assignment 3

**Submission deadline: last lab session before or on Thursday, 02.04.2020**

**Points: 12 + 4 bonus points**

## Submission instructions
**To ease grading, please do not remove the TODO lines, put your code below them!**

The class is held remotely. To sumbmit your solutions please save the notebook to your Google Drive, then:
1. Rename it it to: Assignment3_Surname_FirstName
2. Rerun the whole notebook `Runtime -> Restart and run all`
3. Make a pinned revision `File->Save and pin revision`
4. Share the notebook and paste the link into an appropriate SKOS assignment

We will use the commenting system and video conferences to check and discuss the solutions.

As always, please submit corrections using GitHub's Pull Requests.

```
%matplotlib inline
```

```
import os
import time

import numpy as np
import matplotlib.pyplot as plt

from tqdm.auto import tqdm

import torch
from torch import nn
import torch.nn.functional as F
from torchvision import datasets, models, transforms
import torchvision
from torch.utils.data import Dataset, DataLoader

from IPython.display import clear_output
```

# Helper code

The code in this section handles data loading, result validation and plotting.

```
def compute_error_rate(model, data_loader, device='cpu'):
    """Evaluate model on all samples from the data loader.
    """
    # Put the model in eval mode, and move to the evaluation device.
    model.eval()
    model.to(device)
    if isinstance(data_loader, InMemDataLoader):
        data_loader.to(device)

    num_errs = 0.0
    num_examples = 0
    # we don't need gradient during eval!
    with torch.no_grad():
        for x, y in data_loader:
            x = x.to(device)
            y = y.to(device)
            outputs = model.forward(x)
            _, predictions = outputs.data.max(dim=1)
            num_errs += (predictions != y.data).sum().item()
            num_examples += x.size(0)
    return num_errs / num_examples

def plot_history(history):
    """Helper to plot the trainig progress over time."""
    plt.figure(figsize=(16, 4))
    plt.subplot(1,2,1)
    train_loss = np.array(history['train_losses'])
    plt.semilogy(np.arange(train_loss.shape[0]), train_loss, label='batch train loss')
    plt.legend()

    plt.subplot(1,2,2)
    train_errs = np.array(history['train_errs'])
    plt.plot(np.arange(train_errs.shape[0]), train_errs, label='batch train error rate')
    val_errs = np.array(history['val_errs'])
    plt.plot(val_errs[:,0], val_errs[:,1], label='validation error rate', color='r')
    plt.ylim(0, 0.20)
    plt.legend()
```

## Data preparation

We will download the MNIST data using `torchvision`, then use a fast in-memory loader.

```
class InMemDataLoader(object):
    """
    A data loader that keeps all data in CPU or GPU memory.
    """
    __initialized = False
    def __init__(self, dataset, batch_size=1, shuffle=False, sampler=None,
                 batch_sampler=None, drop_last=False):
        """A torch dataloader that fetches data from memory."""
        batches = []
        for i in tqdm(range(len(dataset))):
            batch = [torch.tensor(t) for t in dataset[i]]
            batches.append(batch)
        tensors = [
            torch.stack(ts) for ts in zip(*batches)
        ]
        dataset = torch.utils.data.TensorDataset(*tensors)
        self.dataset = dataset
        self.batch_size = batch_size
        self.drop_last = drop_last

        if batch_sampler is not None:
            if batch_size > 1 or shuffle or sampler is not None or drop_last:
                raise ValueError('batch_sampler option is mutually exclusive '
                                 'with batch_size, shuffle, sampler, and '
                                 'drop_last')
            self.batch_size = None
            self.drop_last = None

        if sampler is not None and shuffle:
            raise ValueError('sampler option is mutually exclusive with '
                             'shuffle')

        if batch_sampler is None:
            if sampler is None:
                if shuffle:
                    sampler = torch.utils.data.RandomSampler(dataset)
                else:
                    sampler = torch.utils.data.SequentialSampler(dataset)
            batch_sampler = torch.utils.data.BatchSampler(sampler, batch_size, drop_last)

        self.sampler = sampler
        self.batch_sampler = batch_sampler
        self.__initialized = True

    def __setattr__(self, attr, val):
        if self.__initialized and attr in ('batch_size', 'sampler', 'drop_last'):
            raise ValueError('{} attribute should not be set after {} is '
                             'initialized'.format(attr, self.__class__.__name__))

        super(InMemDataLoader, self).__setattr__(attr, val)

    def __iter__(self):
        for batch_indices in self.batch_sampler:
            yield self.dataset[batch_indices]

    def __len__(self):
        return len(self.batch_sampler)

    def to(self, device):
        self.dataset.tensors = tuple(t.to(device) for t in self.dataset.tensors)
        return self
```

```
# Load the data

batch_size = 128
data_path = './data'

transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.1307,), (0.3081,)),
    ])

_test = torchvision.datasets.MNIST(
    data_path, train=False, download=True, transform=transform)

# Load training data, split into train and valid sets
_train = torchvision.datasets.MNIST(
    data_path, train=True, download=True, transform=transform)
_train.data = _train.data[:50000]
_train.targets = _train.targets[:50000]

_valid = torchvision.datasets.MNIST(
    data_path, train=True, download=True, transform=transform)
_valid.data = _valid.data[50000:]
_valid.targets = _valid.targets[50000:]

mnist_loaders = {
    'train': InMemDataLoader(
        _train, batch_size=batch_size, shuffle=True),
    'valid': InMemDataLoader(
        _valid, batch_size=batch_size, shuffle=False),
    'test': InMemDataLoader(
        _test, batch_size=batch_size, shuffle=False)}
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw
    Processing...
    Done!

    HBox(children=(IntProgress(value=0, max=50000), HTML(value='')))

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:11: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      # This is added back by InteractiveShellApp.init_path()

    HBox(children=(IntProgress(value=0, max=10000), HTML(value='')))

    HBox(children=(IntProgress(value=0, max=10000), HTML(value='')))

# SGD implementation

We provide below a scaffolding for SGD. You will need to fill the TODOs whilel solving the assignments.

```
def SGD(model, data_loaders,
        alpha=1e-4, epsilon=0.0, decay=0.0,
        num_epochs=1, max_num_epochs=np.nan, patience_expansion=1.5,
        log_every=100, device='cpu', nesterov=True):

    # Put the model in train mode, and move to the evaluation device.
    model.train()
    model.to(device)
    for data_loader in data_loaders.values():
        if isinstance(data_loader, InMemDataLoader):
            data_loader.to(device)

    #
    # TODO: Initialize momentum variables
    # Hint: You need one velocity matrix for each parameter
    #
    velocities = [torch.zeros(p.shape, device=device) for p in model.parameters()]
    #
    iter_ = 0
    epoch = 0
    best_params = None
    best_val_err = np.inf
    history = {'train_losses': [], 'train_errs': [], 'val_errs': []}
    print('Training the model!')
    print('Interrupt at any time to evaluate the best validation model so far.')
    try:
        tstart = time.time()
        siter = iter_
        while epoch < num_epochs:
            model.train()
            epoch += 1
            if epoch > max_num_epochs:
                print('Maximum epochs')
                break
            #
            # TODO: You can implement learning rate control here (it is updated
            # once per epoch), or below in the loop over minibatches.
            #
            '''
            Typical values might be reducing the learning rate by a half
            every 5 epochs, or by 0.1 every 20 epochs. These numbers depend
            heavily on the type of problem and the model.
            One heuristic you may see in practice is to watch the validation
            error while training with a fixed learning rate,
            and reduce the learning rate by a constant (e.g. 0.5)
            whenever the validation error stops improving.
            '''
            if epoch % 5 == 0:
              alpha /= 2

            # if alpha < 0.0001:
            #   alpha += np.random.normal(0.001, 0.001)

            # if epoch % 20 == 0:
            #   alpha -= 0.1

            for x, y in data_loaders['train']:
                x = x.to(device)
                y = y.to(device)
                iter_ += 1
                # This calls the `forward` function: https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_module.html
                out = model(x)
                loss = model.loss(out, y)
                loss.backward()
                _, predictions = out.max(dim=1)
                batch_err_rate = (predictions != y).sum().item() / out.size(0)

                history['train_losses'].append(loss.item())
                history['train_errs'].append(batch_err_rate)

                # disable gradient computations - we do not want torch to
                # backpropagate through the gradient application!
                with torch.no_grad():
                    for (name, p), v in zip(model.named_parameters(), velocities):
                        if 'weight' in name:
                            #
                            # TODO: Implement weight decay (L2 regularization
                            # on weights by changing the gradients
                            p.grad += 2 * decay * p

                        #
                        # TODO: Implement a learning rate schedule
                        # Hint: You can use the iteration or epoch counters
                        # alpha = TODO
                        # Done above

                        #
                        # TODO: If needed, implement here a momentum schedule
                        # epsilon = TODO
                        #

                        # NESTEROV MOMENTUM DECAY
                        # TODO: Implement velocity updates for momentum
                        # lease make sure to modify the contents of v, not the v pointer!!!
                        #

                        v_prev = v[...]
                        v[...] = epsilon * v[...] - alpha * p.grad

                        #
                        # TODO: Set a more sensible learning rule here,
                        #       using your learning rate schedule and momentum
                        #

                        # working on reference p
                        if nesterov:
                          p += - epsilon * v_prev + (1 + epsilon) * v[...]
                        else:
                          p += v[...]

                        # Zero gradients for the next iteration
                        p.grad.zero_()

                if iter_ % log_every == 0:
                    num_iter = iter_ - siter + 1
                    print("Minibatch {0: >6}  | loss {1: >5.2f} | err rate {2: >5.2f}%, steps/s {3: >5.2f}" \
                          .format(iter_, loss.item(), batch_err_rate * 100.0, num_iter / (time.time() - tstart)))
                    tstart = time.time()

            val_err_rate = compute_error_rate(model, data_loaders['valid'], device)
            history['val_errs'].append((iter_, val_err_rate))

            if val_err_rate < best_val_err:
                # Adjust num of epochs
                num_epochs = int(np.maximum(num_epochs, epoch * patience_expansion + 1))
                best_epoch = epoch
                best_val_err = val_err_rate
                best_params = [p.detach().cpu() for p in model.parameters()]
            clear_output(True)
            m = "After epoch {0: >2} | valid err rate: {1: >5.2f}% | doing {2: >3} epochs" \
                .format(epoch, val_err_rate * 100.0, num_epochs)
            print('{0}\n{1}\n{0}'.format('-' * len(m), m))

    except KeyboardInterrupt:
        pass

    if best_params is not None:
        print("\nLoading best params on validation set (epoch %d)\n" %(best_epoch))
        with torch.no_grad():
            for param, best_param in zip(model.parameters(), best_params):
                param[...] = best_param
    plot_history(history)
```

```
class Model(nn.Module):
    def __init__(self, *args, **kwargs):
        super(Model, self).__init__()
        self.layers = nn.Sequential(*args, **kwargs)

    def forward(self, X):
        X = X.view(X.size(0), -1)
        return self.layers.forward(X)

    def loss(self, Out, Targets):
        return F.cross_entropy(Out, Targets)
```

# Problem 1: Stochastic Gradient Descent [3p]
Implement the following additions to the SGD code provided above:
  1. **[1p]** momentum
  2. **[1p]** learning rate schedule
  3. **[1p]** weight decay, in which we additionally minimize for each weight matrix (but typically not the bias) the sum of its elements squared. One way to implement it is to use function `model.named_parameters` and select all parameters whose names contain "`weight`" rather than "`bias`".

# Problem 2: Tuning the Network for MNIST [4p]

Tune the following network to reach **validation error rate below 1.9%**.
This should result in a **test error rate below 2%**. To
tune the network you will need to:
1. Choose the number of layers (more than 1, less than 5);
2. Choose the number of neurons in each layer (more than 100,
    less than 5000);
3. Pick proper weight initialization;
4. Pick proper learning rate schedule (need to decay over time,
    good range to check on MNIST is about 1e-2 ... 1e-1 at the beginning and
    half of that after 10000 batches);
5. Pick a momentum constant (probably a constant one will be OK).

Please note: there are many hyperparameter settings that give the desired answer, some may require tuning all hyperparameters, some only a few.

```
%%time

model = Model(
    nn.Linear(28*28, 1000),
    # nn.Tanh(),
    nn.ReLU(),
    nn.Linear(1000, 10)
    # no SoftMax here
    )

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.5)
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders, alpha=0.01, epsilon=0.99, max_num_epochs=50,
    device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

# Problem 3: Convolutional Network [2p]

Use convolutional and max-pooling layers (`Conv2d`, `Max_pool2d` or their functional variants) and (without dropout) get a test error rate below 1.5%.

>torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')

Parameters

* in_channels (int) – Number of channels in the input image
* out_channels (int) – Number of channels produced by the convolution
* kernel_size (int or tuple) – Size of the convolving kernel
* stride (int or tuple, optional) – Stride of the convolution. (Default: 1)
* padding (int or tuple, optional) – Zero-padding added to both sides of the input (Default: 0)
* padding_mode (string, optional) – zeros
* dilation (int or tuple, optional) – Spacing between kernel elements. (Default: 1)
* groups (int, optional) – Number of blocked connections from input to output channels. (Default: 1)
* bias (bool, optional) – If True, adds a learnable bias to the output (Default: True)

So the in_channels in the beginning is 3 for images with 3 channels (colored images). For images black and white it should be 1. Some satellite images should have 4.

[Convolution animation](https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md)

```
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        h1 = 16
        h2 = 32
        kernel = (5, 5)
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=h1, kernel_size=kernel,
                      stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d((2, 2)),
            nn.Conv2d(in_channels=h1, out_channels=h2, kernel_size=kernel,
                      stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d((2, 2))
            )
        self.linear = nn.Sequential(
            nn.Linear(h2*kernel[0]*kernel[1], 500),
            nn.ReLU(),
            nn.Linear(500, 10)
            )

    def forward(self, X):
        out = self.conv(X)
        out = out.reshape(out.size(0), -1)
        out = self.linear(out)
        return out

    def loss(self, Out, Targets):
        return F.cross_entropy(Out, Targets)
```

```
%%time

model = ConvNet()

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.01)
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders, alpha=0.02, epsilon=0.99, max_num_epochs=50,
    device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

```
for x, y in mnist_loaders['train']:
  print(x.shape)
  print(y.shape)

  tmp = x[0].cpu().numpy()
  tmp = tmp.squeeze()
  print(tmp.shape)
  plt.imshow(tmp)
  break
```

# Problem 4: Dropout [2p]

Learn about dropout:

- implement a **dropout** layer
- or use `nn.Dropout` (then the exercise is worth 1.5 points)

and try to train a
network getting below 1.5% test error rates with dropout, but no convolutions, or below 1% when dropout is used jointly with convolutions!

Remember to turn off dropout during testing, using `model.train()` and `model.eval()`!

Hint: Use [torch.nn.functional.dropout](http://pytorch.org/docs/master/nn.html#torch.nn.functional.dropout).

Details: http://arxiv.org/pdf/1207.0580.pdf.

```
%%time

model = Model(
    nn.Linear(28*28, 2000),
    nn.Dropout(0.5), # % 20% dropout
    nn.ReLU(),
    nn.Linear(2000, 1000),
    nn.Dropout(0.2), # % 20% dropout
    nn.ReLU(),
    nn.Linear(1000, 10)
    )

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            # p.normal_(0, 0.01)
            nn.init.kaiming_normal_(p, mode='fan_out')  # let's try out kaiming
                                                        # initialization
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders, alpha=0.005, epsilon=0.99, patience_expansion=2.0,
    max_num_epochs=100, device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

# Problem 5: Data Augmentation [1p]

Apply data augmentation methods (e.g. rotations, noise, crops) when training networks on MNIST, to significantly reduce test error rate for your network. You can use functions from the [torchvision.transforms](http://pytorch.org/docs/master/torchvision/transforms.html) module.

Please note: when using random transfromations during training, use a data loader that re-computes them for each training minibatch. The `InMemoryDataloader` assumes that the transformations are deterministic and applies them only once.

# 5.0 Data Augmentation Playground

```
mean, std = 0.13101533792088266, 0.3085401559245608

transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize([0.5,],[0.5,])])

transform2 = transforms.Compose([transforms.RandomResizedCrop(28),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1),
                                 transforms.ToTensor(),
                                 transforms.Normalize([0.5,],[0.5,])])

transform3 = transforms.Compose([transforms.RandomAffine(degrees=20, translate=(0.1,0.1), scale=(0.9, 1.1)),
                                      transforms.ColorJitter(brightness=0.2, contrast=0.2),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean=[mean],std=[std])
                                      ])
```

```
fashion_mnist = torchvision.datasets.FashionMNIST(root='./data', train=False,
                                                  download=True, transform=transform)

fashion_mnist2 = torchvision.datasets.FashionMNIST(root='./data', train=False,
                                                  download=True, transform=transform2)

fashion_mnist3 = torchvision.datasets.FashionMNIST(root='./data', train=False,
                                                  download=True, transform=transform3)

loader = torch.utils.data.DataLoader(fashion_mnist, batch_size=64, shuffle=True)
loader2 = torch.utils.data.DataLoader(fashion_mnist2, batch_size=64, shuffle=True)
loader3 = torch.utils.data.DataLoader(fashion_mnist3, batch_size=64, shuffle=True)
```

```
def show_transformed(images):
  grid = torchvision.utils.make_grid(images, nrow = 20, padding = 2)
  plt.figure(figsize = (18, 18))
  plt.imshow(np.transpose(grid, (1, 2, 0)))
```

```
show_transformed(list(loader)[0][0])
```

```
show_transformed(list(loader2)[0][0])
```

```
show_transformed(list(loader3)[0][0])
```

# 5.1 Apply on MNIST

```
%%time
# Load the data
batch_size = 128
data_path = './data'

transform_train = transforms.Compose([transforms.RandomAffine(degrees=20, translate=(0.1,0.1), scale=(0.9, 1.1)),
                                      transforms.ColorJitter(brightness=0.2, contrast=0.2),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean=[mean],std=[std])
                                      ])

transform_val = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                                                transforms.Normalize(mean=[mean],std=[std])
                                                ])

_test = torchvision.datasets.MNIST(
    data_path, train=False, download=True, transform=transform_val)

# Load training data, split into train and valid sets
_train = torchvision.datasets.MNIST(
    data_path, train=True, download=True, transform=transform_train)
_train.data = _train.data[:50000]
_train.targets = _train.targets[:50000]

_valid = torchvision.datasets.MNIST(
    data_path, train=True, download=True, transform=transform_val)
_valid.data = _valid.data[50000:]
_valid.targets = _valid.targets[50000:]

mnist_loaders_augmented = {
    'train': InMemDataLoader(
        _train, batch_size=batch_size, shuffle=True),
    'valid': InMemDataLoader(
        _valid, batch_size=batch_size, shuffle=False),
    'test': InMemDataLoader(
        _test, batch_size=batch_size, shuffle=False)}
```

    HBox(children=(IntProgress(value=0, max=50000), HTML(value='')))

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:11: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      # This is added back by InteractiveShellApp.init_path()

    HBox(children=(IntProgress(value=0, max=10000), HTML(value='')))

    HBox(children=(IntProgress(value=0, max=10000), HTML(value='')))

    CPU times: user 23 s, sys: 322 ms, total: 23.3 s
    Wall time: 23.3 s

```
%%time

model = ConvNet()

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.01)
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders_augmented, alpha=0.02, epsilon=0.99, max_num_epochs=50,
    device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders_augmented['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

# 5.2 Non deterministic dataloader (Significant improvement)
Previously I transformed data once using in memory MNIST loader and it was deterministic. <br>
Now using dataloader let's try to make every batch random.

```
mnist_dataloader = {
    'train': DataLoader(
        _train, batch_size=batch_size, shuffle=True, num_workers=4),
    'valid': DataLoader(
        _valid, batch_size=batch_size, shuffle=False, num_workers=4),
    'test': DataLoader(
        _test, batch_size=batch_size, shuffle=False, num_workers=4)
    }
```

```
%%time

model = ConvNet()

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.01)
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_dataloader, alpha=0.02, epsilon=0.99, max_num_epochs=50,
    device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_dataloader['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

# Problem 6: Batch Normalization [1p bonus]

[Batch Normalization](https://arxiv.org/abs/1502.03167) helps training neural networks because it [normalizes layer activation magnitudes](https://papers.nips.cc/paper/7515-how-does-batch-normalization-help-optimization.pdf). It typically allows to train networks faster and/or with higher learning rates, lessens the importance
of initialization and might eliminate the need for Dropout.

Implement Batch Normalization and compare with regular training of MNIST models.

Remember to use the batch statistics during model training and to use an average of training batch statistics during model evaluation. For details please consult the paper.

https://www.coursera.org/learn/deep-neural-network/lecture/4ptp2/normalizing-activations-in-a-network

1. Normalize output of a layer before or after activation function?
  - usually before

2. Why can't I use Kaiming weights initialization?
  - 'mode=fan_out' preserves the magnitudes in the backwards pass.
  - so we could change them in forward pass ??
  - how does it work ...

3. On test use exponentially weighted average of $\mu$ and $\sigma$ calculated from the training set

\begin{align}
  &  z_1, ... , z_m \\
  \mu & = \frac{1}{m} * \sum_{i}{z_i} \\
  {\sigma^2} & = \frac{1}{m} * \sum_{i}{(z_i - \mu)^2} \\
  z_{i}{norm} & = \frac{z_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \\
  z_{i}{hat} & = \gamma * z_{i}{norm}  + \beta \\
\end{align}

 $ {\gamma} $  and  $ {\beta} $ are learnable parameters <br>
 If  $ {\gamma} = {\sqrt{\sigma^2 + \epsilon}}  $  and  $ {\beta} = \mu $, then $ z_{i}{hat} = z_i $

### 6.1 Batch Norm vs Dropout

1. Dropout: 1.46% / 1.49%
2. Batch Norm: 1.33% / 1.41%

```
%%time

model = Model(
    nn.Linear(28*28, 2000),
    nn.BatchNorm1d(num_features=2000),
    nn.ReLU(),
    nn.Linear(2000, 1000),
    nn.BatchNorm1d(num_features=1000),
    nn.ReLU(),
    nn.Linear(1000, 10)
    )

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.01)
            # nn.init.kaiming_normal_(p, mode='fan_in')  # cannot use kaiming here
                                                         # Fan in and fan out can not be computed for tensor with fewer than 2 dimensions
                                                         # Why ???
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders, alpha=0.005, epsilon=0.99, patience_expansion=2.0,
    max_num_epochs=100, device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))

```

### 6.2 Batch Norm vs CNN
1. CNN  ---> 0.79% / 0.83%
2. CNN + BN ---> 0.56% / 0.6%

```
class ConvNetBN(nn.Module):
    def __init__(self):
        super(ConvNetBN, self).__init__()
        h1 = 16
        h2 = 32
        kernel = (5, 5)
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=h1, kernel_size=kernel,
                      stride=1, padding=1),
            # nn.BatchNorm2d(num_features=h1),
            nn.ReLU(),
            nn.MaxPool2d((2, 2)),
            nn.Conv2d(in_channels=h1, out_channels=h2, kernel_size=kernel,
                      stride=1, padding=1),
            nn.BatchNorm2d(num_features=h2),
            nn.ReLU(),
            nn.MaxPool2d((2, 2))
            )
        self.linear = nn.Sequential(
            nn.Linear(h2*kernel[0]*kernel[1], 500),
            nn.BatchNorm1d(500),
            nn.ReLU(),
            nn.Linear(500, 10)
            )

    def forward(self, X):
        out = self.conv(X)
        out = out.reshape(out.size(0), -1)
        out = self.linear(out)
        return out

    def loss(self, Out, Targets):
        return F.cross_entropy(Out, Targets)
```

```
%%time

model = ConvNetBN()

with torch.no_grad():
    # Initialize parameters
    for name, p in model.named_parameters():
        if 'weight' in name:
            p.normal_(0, 0.01)
        elif 'bias' in name:
            p.zero_()
        else:
            raise ValueError('Unknown parameter name "%s"' % name)

# On GPU enabled devices set device='cuda' else set device='cpu'
t_start = time.time()
SGD(model, mnist_loaders, alpha=0.01, epsilon=0.99, max_num_epochs=50,
    device='cuda', nesterov=True, decay=0.0005)

test_err_rate = compute_error_rate(model, mnist_loaders['test'])
m = (f"Test error rate: {test_err_rate * 100.0:.3f}%, "
     f"training took {time.time() - t_start:.0f}s.")

print('{0}\n{1}\n{0}'.format('-' * len(m), m))
```

# Problem 7: Norm Constraints [1p bonus]

Implement norm constraints, i.e. instead of weight decay, that tries to set
all weights to small values, apply a limit on the total
norm of connections incoming to a neuron. In our case, this
corresponds to clipping the norm of *rows* of weight
matrices. An easy way of implementing it is to make a gradient
step, then look at the norm of rows and scale down those that are
over the threshold (this technique is called "projected gradient descent").

Please consult the Dropout paper (http://arxiv.org/pdf/1207.0580.pdf) for details.

# Problem 8: Polyak Averaging [1p bonus]

Implement Polyak averaging. For each parameter $\theta$
keep a separate, exponentially decayed average of the past values
$$
\bar{\theta}_n = \alpha_p\bar{\theta}_{n-1} + (1-\alpha_p)\theta_n.
$$
Use that average when evaluating the model on the test set.
Validate the approach by training a model on the MNIST dataset.

# Problem 9: Hyperparameter tuner [1p bonus]

Implement a hyper-parameter tuner able to optimize the learing rate schedule, number of neurons and similar hyperparameters. For start, use random search (please see http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf and especially Fig 1. for intuitions on why random search is better than grid search). It may be a good idea to use a fixed maximum number of epochs (or training time) for each optimization trial to prevent selecting hyperparameters that yield slowly converging solutions. A good result will be a set of hyperparameters that reach on MNIST solutions with test errors less than $1.3\%$ in no more than 50 epochs.

# Problem 10: Other tricks [1-many bonus pounts]

The neural network literature is ful of trickss for training neural networks. Find some and implement them. Please note: the number of points depends on the hardness of the extension you want to implement. If in doubt, consult the TA beforehand

```

```