# 线性判别分析(Fisher's Linear Discriminant, FLD)

## 符号定义

|符号|定义|
|:-:|:-:|
|$\pmb{x}$|样本点|
|$N$|数据总数|
|$X$|样本集合|
|$l$|总类别数|
|$\pmb{m}$|均值向量|
|$y$|每一个样本点对应的类别|
|$\pmb{w}$|投影向量|
|$C$|协方差矩阵|
|$S$|散度矩阵|
|$k$|降维后维度|
|$\sigma$|每一个类降维后的标准差|
|$\mu$|每一个类降维后的均值向量|

## PCA的缺陷

PCA得到的降维总是向着最大化方差的方向进行投影，这一策略往往是有效的。直观上来说，向着最大化方差的方向投影显然能最大程度上离散原始的数据点，即使得降维后的数据尽量可分。

但是在一些极端情况下，这一策略可能会失效。对于细粒度数据，其类间差距小，而类内差距大，若依然向着最大化方差的方向进行降维，显然同类别的数据由于类内差距大而分离，不同类别的数据由于类间差距小而降维到相似的坐标。对于细粒度的数据，显然，同类别的数据降维后聚合到一起，不同类别的数据相互分离才是有效的降维结果

FLD正是这样一种能依靠已有数据进行监督降维的方法。类似于PCA，FLD也是一种线性的降维方法，因此其可以视为原数据点向某些方向进行投影

## 应用于二维二分类的FLD

对于分类问题，往往希望降维后同类数据聚合到一起，而不同类的数据分离。

这一描述显然涉及到两个关键问题
* 同类数据聚合到一起：每一类的方差尽可能小
* 不同类数据的距离尽可能大：类间平均距离大

那么FLD的目标就是：最小化类内方差，最大化类间间距（每一个类的期望的距离）

首先进行定义

* 正样本集合
$$
\begin{equation}
    X_1 = \{ \pmb{x_i}|1\leq i \leq N,\ y_i=1\}
\end{equation}
$$

* 负样本集合
$$
\begin{equation}
    X_2 = \{ \pmb{x_i}|1\leq i \leq N,\ y_i=2\}
\end{equation}
$$

* 对应的均值向量
$$
\begin{equation}
    \pmb{m_1} = \frac{1}{N_1}\sum_{\pmb{x}\in X_1}\pmb{x}
\end{equation}
$$
$$
\begin{equation}
    \pmb{m_2} = \frac{1}{N_2}\sum_{\pmb{x}\in X_2}\pmb{x}
\end{equation}
$$

* 对应的协方差矩阵
$$
\begin{equation}
    C_1 = \frac{1}{N_1}\sum_{\pmb{x} \in X_1}(\pmb{x}-\pmb{m_1})(\pmb{x}-\pmb{m_1})^T
\end{equation}
$$
$$
\begin{equation}
    C_2 = \frac{1}{N_2}\sum_{\pmb{x} \in X_2}(\pmb{x}-\pmb{m_2})(\pmb{x}-\pmb{m_2})^T
\end{equation}
$$

* 对应的散度矩阵
$$
\begin{equation}
    S_1 = \sum_{\pmb{x} \in X_1}(\pmb{x}-\pmb{m_1})(\pmb{x}-\pmb{m_1})^T = N_1C_1
\end{equation}
$$
$$
\begin{equation}
    S_2 = \sum_{\pmb{x} \in X_2}(\pmb{x}-\pmb{m_2})(\pmb{x}-\pmb{m_2})^T= N_2C_2
\end{equation}
$$

* 对应的均值降维结果
$$
\begin{equation}
    \mu_1 = \pmb{m_1}^T\pmb{w}
\end{equation}
$$
$$
\begin{equation}
    \mu_2 = \pmb{m_2}^T\pmb{w}
\end{equation}
$$

* 对应的降维后标准差
$$
\begin{equation}
    \begin{split}
    \sigma_1
    &= \sqrt{\frac{1}{N_1}\sum_{\pmb{x}\in X_1}(\pmb{x}^T\pmb{w}-\pmb{m_1}^T\pmb{w})^2} \\
    &= \sqrt{\pmb{w}^T C_1 \pmb{w}} \\
    \end{split}
\end{equation}
$$
$$
\begin{equation}
    \begin{split}
    \sigma_2
    &= \sqrt{\frac{1}{N_2}\sum_{\pmb{x}\in X_2}(\pmb{x}^T\pmb{w}-\pmb{m_2}^T\pmb{w})^2} \\
    &= \sqrt{\pmb{w}^T C_2 \pmb{w}} \\
    \end{split}
\end{equation}
$$

因此FLD的目标：最小化类内方差，最大化类间间距（每一个类的期望的距离）可以被翻译为如下的几种形式：

$$
\begin{equation}
    \frac{|\mu_1-\mu_2|}{\sigma_1 + \sigma_2}
\end{equation}
$$
$$
\begin{equation}
    \frac{|\mu_1-\mu_2|}{\sqrt{\sigma_1^2 + \sigma_2^2}}
\end{equation}
$$

以及
$$
\begin{equation}
    \frac{(\mu_1-\mu_2)^2}{\sigma1_1^2 + \sigma_2^2}
\end{equation}
$$

式-13相较于上述的式-11和式-12，其显然要便于后续的推导（既没有绝对值，也没有开方）

将式-5、式-6、式-7、式-8、式-9和式-10带入式-13可得

$$
\begin{equation}
    \begin{split}
    \mathcal{L}
    &= \frac{(\mu_1-\mu_2)^2}{\sigma1_1^2 + \sigma_2^2} \\
    &= \frac{\pmb{w}^T(\pmb{m_1}-\pmb{m_2})(\pmb{m_1}-\pmb{m_2})^T\pmb{w}}{\pmb{w}^T(C_1+C_2)\pmb{w}}
    \end{split}
\end{equation}
$$

若将上式中的协方差矩阵换为散度矩阵可以得到最为原始的FLD形式

$$
\begin{equation}
    \mathcal{L} = \frac{\pmb{w}^T(\pmb{m_1}-\pmb{m_2})(\pmb{m_1}-\pmb{m_2})^T\pmb{w}}{\pmb{w}^T(S_1+S_2)\pmb{w}}
\end{equation}
$$

显然，替换为散度矩阵后并不影响上述损失的最优解

进一步将式-17简写为如下的形式
$$
\begin{equation}
    \mathcal{L} = \frac{\pmb{w}^TS_B\pmb{w}}{\pmb{w}^TS_W\pmb{w}}
\end{equation}
$$
其中$S_B$为类间散度矩阵，$S_W$为类内散度矩阵
$$
\begin{equation}
    S_B = (\pmb{m_1}-\pmb{m_2})(\pmb{m_1}-\pmb{m_2})^T
\end{equation}
$$
$$
\begin{equation}
    S_W = S_1+S_2
\end{equation}
$$

有了上述定义后，显然优化目标是$\max\mathcal{L}$
对$\pmb{w}$求偏导可得

$$
\begin{equation}
    \frac{\partial{\mathcal{L}}}{\partial{\pmb{w}}} = \frac{2((\pmb{w}^TS_W\pmb{w})S_B\pmb{w}-(\pmb{w}^TS_B\pmb{w})S_W\pmb{w})}{(\pmb{w}^TS_W\pmb{w})^2} = 0
\end{equation}
$$

可得：

$$
\begin{equation}
    S_B\pmb{w} = \frac{\pmb{w}^TS_B\pmb{w}}{\pmb{w}^TS_W\pmb{w}}S_W\pmb{w}
\end{equation}
$$

显然$\pmb{w}$为上式的一个广义特征向量，而$\mathcal{L} = \frac{\pmb{w}^TS_B\pmb{w}}{\pmb{w}^TS_W\pmb{w}}$是上式的一个广义特征值，考虑到优化目标是$\max{\mathcal{L}}$，因此$\pmb{w}$的最优解为式-22最大广义特征值对应的特征向量

## 应用于二分类的FLD
上述分析同样适用于多于二维的数据点，类似于PCA中的分析，仅需要提取式-22前k个最大广义特征值对应的特征向量就可以得到k维子空间下的降维结果

## 应用于多分类的FLD
对于多分类，其基本原则没有改变：依然是尽可能增大降维后的类间距离并减小类内方差，但是不同的是类间散度以及类内散度的计算方法。

对于类内散度，其定义扩展为子集的散度矩阵之和
$$
\begin{equation}
    S_W = \sum_{i=1}^lS_i = \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m_i})(\pmb{x}-\pmb{m_i})^T
\end{equation}
$$

对于类间散度，若依然依照二分类的定义，则会计算$l\times l$个矩阵并求和，这样显然效率低下
$$
\begin{equation}
    S_B = \sum_{i=1}^l\sum_{j=1}^l(\pmb{m_i}-\pmb{m_j})(\pmb{m_i}-\pmb{m_j})^T
\end{equation}
$$

考虑到在不考虑类别的情况下，总散度矩阵$S_T$为
$$
\begin{equation}
    S_T = \sum_{i=1}^N(\pmb{x_i}-\pmb{m})(\pmb{x_i}-\pmb{m})^T
\end{equation}
$$
$$
\begin{equation}
    \pmb{m} = \frac{1}{N}\sum_{i=1}^N\pmb{x_i}
\end{equation}
$$

上述的总散度矩阵$S_T$可化简为如下形式
$$
\begin{split}
    \begin{split}
    S_T
    &= \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m})(\pmb{x}-\pmb{m})^T \\
    &= \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m_i}+\pmb{m_i}-\pmb{m})(\pmb{x}-\pmb{m_i}+\pmb{m_i}-\pmb{m})^T \\
    &= \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m_i})(\pmb{x}-\pmb{m_i})^T + \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{m_i}-\pmb{m})(\pmb{m_i}-\pmb{m})^T + \sum_{i=1}^l \sum_{\pmb{x} \in X_i}((\pmb{x}-\pmb{m_i})(\pmb{m_i}-\pmb{m})^T+(\pmb{m_i}-\pmb{m})(\pmb{x}-\pmb{m_i})^T) \\
    &= \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m_i})(\pmb{x}-\pmb{m_i})^T + \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{m_i}-\pmb{m})(\pmb{m_i}-\pmb{m})^T \\
    &= S_W + \sum_{i=1}^l N_i(\pmb{m_i}-\pmb{m})(\pmb{m_i}-\pmb{m})^T
    \end{split}
\end{split}
$$
因此总散度矩阵减去类内散度矩阵为
$$
\begin{equation}
    S_T - S_W = \sum_{i=1}^l N_i(\pmb{m_i}-\pmb{m})(\pmb{m_i}-\pmb{m})^T
\end{equation}
$$
上式表示的是每一个类的均值向量之间带权重的散度矩阵，在FLD中类间散度就被定义为上式

FLD中使用的类间散度矩阵为
$$
\begin{equation}
    S_B = \sum_{i=1}^lN_i(\pmb{m_i}-\pmb{m})(\pmb{m_i}-\pmb{m})^T
\end{equation}
$$
上式考虑到了不同类的数据数量的差异，对于不同类的数据量存在显著差异的情况，这一处理显然是有利的。

## 算法流程
* 定义数据$X=\{\pmb{x_1}, \pmb{x_2}, \cdots, \pmb{x_N}\}$、每一个数据点对应的类别$Y=\{y_1, y_2, \cdots, y_N\}$以及一个新的维度$k, k<d$
* 计算总散度矩阵$S_T$
$$
\begin{equation}
    S_T = \sum_{i=1}^N(\pmb{x_i}-\pmb{m})(\pmb{x_i}-\pmb{m})^T
\end{equation}
$$
* 计算类内散度矩阵$S_W$
$$
\begin{equation}
    S_W = \sum_{i=1}^lS_i = \sum_{i=1}^l \sum_{\pmb{x} \in X_i}(\pmb{x}-\pmb{m_i})(\pmb{x}-\pmb{m_i})^T
\end{equation}
$$
* 计算类间散度矩阵$S_B$
$$
\begin{equation}
    S_B = S_T - S_W
\end{equation}
$$
* 计算下式的广义特征向量以及特征值
$$
\begin{equation}
    S_B\pmb{w} = \frac{\pmb{w}^TS_B\pmb{w}}{\pmb{w}^TS_W\pmb{w}}S_W\pmb{w}
\end{equation}
$$
* 则任意$\pmb{x}\in\mathcal{R}^d$其近似为
$$
\begin{equation}
    \hat{\pmb{x_i}} = \bar{\pmb{x}} + (\pmb{\xi_1}^T(\pmb{x_i}-\bar{\pmb{x}}))\pmb{\xi_1} + (\pmb{\xi_2}^T(\pmb{x_i}-\bar{\pmb{x}}))\pmb{\xi_2} + \cdots + (\pmb{\xi_k}^T(\pmb{x_i}-\bar{\pmb{x}}))\pmb{\xi_k}
\end{equation}
$$
其对应的坐标为
$$
\begin{equation}
   (\pmb{\xi_1}^T(\pmb{x_i}-\bar{\pmb{x}}), \pmb{\xi_2}^T(\pmb{x_i}-\bar{\pmb{x}}), \cdots, \pmb{\xi_k}^T(\pmb{x_i}-\bar{\pmb{x}}))^T
\end{equation}
$$

```python
import numpy as np
import scipy
from sklearn.decomposition import PCA
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
```

```python
# 数据数量
sample_num = 1000
# 方差
std_ = 0.35

x_list, y_list = make_blobs(n_samples=sample_num, centers=2,
                            cluster_std=std_, random_state=1, center_box=(-2, 2))

transformation = [[0.60834549, -0.63667341], [-0.40887718, 0.85253229]]
x_list = np.dot(x_list, transformation)
```

```python
class MyFLD(object):

    def __init__(self, input_dim, reduction_dim):

        self.input_dim = input_dim
        self.reduction_dim = reduction_dim

        self.fisher_matrix = None
        self.mean_mat = list()

        assert self.reduction_dim <= self.input_dim

    def fit(self, x_data, y_data):

        input_data = np.array(x_data)
        y_data = np.array(y_data).reshape(-1)
        assert input_data.shape[1] == self.input_dim and y_data.shape[0] == x_data.shape[0]

        # cal S_T
        scatter_t = np.cov(input_data.T) * input_data.shape[0]

        # cal S_W
        scatter_w = 0
        y_key_list = np.unique(y_data)
        for key_ in y_key_list:
            temp_input_data = input_data[np.argwhere(y_data == key_).reshape(-1), :].copy()

            scatter_w += np.cov(temp_input_data.T) * temp_input_data.shape[0]

            self.mean_mat.append(np.mean(temp_input_data, axis=0))
        self.mean_mat = np.array(self.mean_mat)

        # cal S_B
        scatter_b = scatter_t - scatter_w

        # cal top k
        _, topk_e_vecs_mat = self._topk_eigh(scatter_b, scatter_w, self.reduction_dim)
        self.fisher_matrix = topk_e_vecs_mat.T

    def transform(self, x):
        input_vec = np.array(x)
        input_vec = np.reshape(x, (self.input_dim))
        return np.matmul(self.fisher_matrix, input_vec)

    def fit_transform(self, x_data, y_data):

        # fit
        self.fit(x_data, y_data)

        reduction_vecs = list()

        for input_vec in x_data:
            reduction_vecs.append(self.transform(input_vec))

        return np.array(reduction_vecs)

    def _topk_eigh(self, mat1, mat2, k):
        e_vals, e_vecs = scipy.linalg.eigh(mat1, mat2)

        sorted_ind = np.argsort(e_vals)

        topk_e_vals = e_vals[sorted_ind[:-1-k:-1]]
        topk_e_vecs = e_vecs[:, sorted_ind[:-1-k:-1]]
        return topk_e_vals, topk_e_vecs
```

```python
# FLD
my_fld = MyFLD(2, 1)
my_fld_pred_x = my_fld.fit_transform(x_list, y_list)
show_fld_x = my_fld_pred_x * my_fld.fisher_matrix + my_fld.mean_mat[np.array(y_list).reshape(-1)]

# sklearn pca
sklearn_pca = PCA(n_components=1)
sklearn_pca_pred_x = sklearn_pca.fit_transform(x_list)
show_pca_x = sklearn_pca_pred_x * sklearn_pca.components_[0] + sklearn_pca.mean_

plt.figure(figsize=(20, 20))

plt.subplot(2, 2, 1)
plt.title("PCA", fontsize=20)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
pos_x_list = x_list[np.argwhere(y_list == 0).reshape(-1), :].copy()
neg_x_list = x_list[np.argwhere(y_list == 1).reshape(-1), :].copy()
plt.scatter(pos_x_list[:, 0], pos_x_list[:, 1], label="Class 1")
plt.scatter(neg_x_list[:, 0], neg_x_list[:, 1], label="Class 2")
plt.scatter(show_pca_x[:, 0], show_pca_x[:, 1], c="#cc3536", label="PCA Result")
plt.legend(fontsize=15)

plt.subplot(2, 2, 2)
plt.title("FLD", fontsize=20)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
pos_x_list = x_list[np.argwhere(y_list == 0).reshape(-1), :].copy()
neg_x_list = x_list[np.argwhere(y_list == 1).reshape(-1), :].copy()
plt.scatter(pos_x_list[:, 0], pos_x_list[:, 1], label="Class 1")
plt.scatter(neg_x_list[:, 0], neg_x_list[:, 1], label="Class 2")
plt.scatter(show_fld_x[:, 0], show_fld_x[:, 1], c="#cc3536", label="FLD Result")
plt.legend(fontsize=15)

plt.subplot(2, 2, 3)
plt.title("PCA", fontsize=20)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)
binwidth = 0.03
pos_x_list = sklearn_pca_pred_x.reshape(-1)[np.argwhere(y_list == 0).reshape(-1)].copy()
plt.hist(pos_x_list, bins=np.arange(np.min(pos_x_list), np.max(pos_x_list), binwidth), label="Class 1")
neg_x_list = sklearn_pca_pred_x.reshape(-1)[np.argwhere(y_list == 1).reshape(-1)].copy()
plt.hist(neg_x_list, bins=np.arange(np.min(neg_x_list), np.max(neg_x_list), binwidth), label="Class 2")
plt.legend(fontsize=15)

plt.subplot(2, 2, 4)
plt.title("FLD", fontsize=20)
plt.xticks(fontsize=20)
plt.yticks(fontsize=20)

binwidth = 0.005
pos_x_list = my_fld_pred_x.reshape(-1)[np.argwhere(y_list == 0).reshape(-1)].copy()
plt.hist(pos_x_list, bins=np.arange(np.min(pos_x_list), np.max(pos_x_list), binwidth), label="Class 1")
neg_x_list = my_fld_pred_x.reshape(-1)[np.argwhere(y_list == 1).reshape(-1)].copy()
plt.hist(neg_x_list, bins=np.arange(np.min(neg_x_list), np.max(neg_x_list), binwidth), label="Class 2")
plt.legend(fontsize=15)

plt.show()
```

上图非常明显，PCA降维后两类混到了一起，而FLD降维后虽然类内数据点的可分性下降了，但是类间的可分性得到了保持


# Geodemographic Segmentation Model

Dataset: https://www.superdatascience.com/training/

Author: Filipa C. S. Rodrigues (filipacsrodrigues@gmail.com)

```python
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
from __future__ import division
import pandas
import csv
import numpy as np
import statsmodels.api as sm
import statsmodels.discrete.discrete_model as smdis
import statsmodels.stats.outliers_influence as outliers
import matplotlib.pyplot as plt
```

    /usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

```python
df = pandas.DataFrame.from_csv('Churn-Modelling.csv', index_col=None)
df[:5]
```

      RowNumber
      CustomerId
      Surname
      CreditScore
      Geography
      Gender
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary
      Exited

      0
      1
      15634602
      Hargrave
      619
      France
      Female
      42
      2
      0.00
      1
      1
      1
      101348.88
      1

      1
      2
      15647311
      Hill
      608
      Spain
      Female
      41
      1
      83807.86
      1
      0
      1
      112542.58
      0

      2
      3
      15619304
      Onio
      502
      France
      Female
      42
      8
      159660.80
      3
      1
      0
      113931.57
      1

      3
      4
      15701354
      Boni
      699
      France
      Female
      39
      1
      0.00
      2
      0
      0
      93826.63
      0

      4
      5
      15737888
      Mitchell
      850
      Spain
      Female
      43
      2
      125510.82
      1
      1
      1
      79084.10
      0

#### Variables

- Exited: dependent variable (y) -->  is binary
- Gender, Geography: categorical independent variable (x)
- All the remaining variables: numeric independent variable (x)

```python
df_y = df['Exited']
df_x = df.drop(['Exited'], axis = 1)
```

Create dummy variables for the categorical variables:

```python
dummy = pandas.get_dummies(df_x['Gender'])
df_x = dummy.join(df_x)
dummy = pandas.get_dummies(df_x['Geography'])
df_x = dummy.join(df_x)
df_x[:5]
```

      France
      Germany
      Spain
      Female
      Male
      RowNumber
      CustomerId
      Surname
      CreditScore
      Geography
      Gender
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      0
      1
      0
      1
      15634602
      Hargrave
      619
      France
      Female
      42
      2
      0.00
      1
      1
      1
      101348.88

      1
      0
      0
      1
      1
      0
      2
      15647311
      Hill
      608
      Spain
      Female
      41
      1
      83807.86
      1
      0
      1
      112542.58

      2
      1
      0
      0
      1
      0
      3
      15619304
      Onio
      502
      France
      Female
      42
      8
      159660.80
      3
      1
      0
      113931.57

      3
      1
      0
      0
      1
      0
      4
      15701354
      Boni
      699
      France
      Female
      39
      1
      0.00
      2
      0
      0
      93826.63

      4
      0
      0
      1
      1
      0
      5
      15737888
      Mitchell
      850
      Spain
      Female
      43
      2
      125510.82
      1
      1
      1
      79084.10

Only one dummy variable should be used to avoid the "dummy variable trap":

```python
df_x = df_x.drop(['Gender', 'Male', 'France', 'Geography'], axis =1)
df_x[:5]
```

      Germany
      Spain
      Female
      RowNumber
      CustomerId
      Surname
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      0
      0
      1
      1
      15634602
      Hargrave
      619
      42
      2
      0.00
      1
      1
      1
      101348.88

      1
      0
      1
      1
      2
      15647311
      Hill
      608
      41
      1
      83807.86
      1
      0
      1
      112542.58

      2
      0
      0
      1
      3
      15619304
      Onio
      502
      42
      8
      159660.80
      3
      1
      0
      113931.57

      3
      0
      0
      1
      4
      15701354
      Boni
      699
      39
      1
      0.00
      2
      0
      0
      93826.63

      4
      0
      1
      1
      5
      15737888
      Mitchell
      850
      43
      2
      125510.82
      1
      1
      1
      79084.10

Add a constant \begin{align} b_0 \end{align} to the model:

```python
df_x = sm.add_constant(df_x)
df_x[:2]
```

      const
      Germany
      Spain
      Female
      RowNumber
      CustomerId
      Surname
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      0
      1
      1
      15634602
      Hargrave
      619
      42
      2
      0.00
      1
      1
      1
      101348.88

      1
      1
      0
      1
      1
      2
      15647311
      Hill
      608
      41
      1
      83807.86
      1
      0
      1
      112542.58

Exclude the variables that should not affect the model:

```python
df_x = df_x.drop(['RowNumber', 'CustomerId', 'Surname'], axis = 1)
df_x[:2]
```

      const
      Germany
      Spain
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      0
      1
      619
      42
      2
      0.00
      1
      1
      1
      101348.88

      1
      1
      0
      1
      1
      608
      41
      1
      83807.86
      1
      0
      1
      112542.58

Create a model with all the remaining variables:

```python
model1 = smdis.Logit(df_y, df_x).fit()
print model1.summary()
print "\n_____P-values____"
p = model1.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model1.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428068
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9988
    Method:                           MLE   Df Model:                           11
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1532
    Time:                        13:04:11   Log-Likelihood:                -4280.7
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ===================================================================================
                          coef    std err          z      P>|z|      [95.0% Conf. Int.]
    -----------------------------------------------------------------------------------
    const              -3.9208      0.245    -15.980      0.000        -4.402    -3.440
    Germany             0.7747      0.068     11.448      0.000         0.642     0.907
    Spain               0.0352      0.071      0.499      0.618        -0.103     0.174
    Female              0.5285      0.054      9.699      0.000         0.422     0.635
    CreditScore        -0.0007      0.000     -2.384      0.017        -0.001    -0.000
    Age                 0.0727      0.003     28.230      0.000         0.068     0.078
    Tenure             -0.0159      0.009     -1.705      0.088        -0.034     0.002
    Balance          2.637e-06   5.14e-07      5.128      0.000      1.63e-06  3.64e-06
    NumOfProducts      -0.1015      0.047     -2.154      0.031        -0.194    -0.009
    HasCrCard          -0.0447      0.059     -0.753      0.452        -0.161     0.072
    IsActiveMember     -1.0754      0.058    -18.643      0.000        -1.189    -0.962
    EstimatedSalary  4.807e-07   4.74e-07      1.015      0.310     -4.48e-07  1.41e-06
    ===================================================================================

    _____P-values____
    const               1.759879e-57
    Germany             2.413604e-30
    Spain               6.180842e-01
    Female              3.044696e-22
    CreditScore         1.712765e-02
    Age                2.521111e-175
    Tenure              8.821318e-02
    Balance             2.922724e-07
    NumOfProducts       3.124760e-02
    HasCrCard           4.515135e-01
    IsActiveMember      1.434412e-77
    EstimatedSalary     3.101752e-01
    dtype: float64

    _____Highest p-value____

     Spain    0.618084
    dtype: float64

    ____Confusion Matrix___
    [[ 7666.   297.]
     [ 1600.   437.]]

    Number of cases correctly predicted: 8103.0 (81.03 %)

Note:

__"Pseudo R-squ."__ -  pseudo R-squared.
Logistic regression does not have an equivalent to the R-squared that is found in OLS regression; however, many people have tried to come up with one.  There are a wide variety of pseudo-R-square statistics.  Because this statistic does not mean what R-square means in OLS regression (the proportion of variance explained by the predictors), we suggest interpreting this statistic with great caution.        (source: http://www.ats.ucla.edu/stat/stata/output/stata_logistic.htm)

__Backward Elimination__

New model without the variable with the highest p-value: "Spain"

```python
df_x1 = df_x.drop(['Spain'], axis = 1)
df_x1[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1
      1
      101348.88

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      0
      1
      112542.58

```python
model2 = smdis.Logit(df_y, df_x1).fit()
print model2.summary()
print "\n_____P-values____"
p = model2.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model2.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428080
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9989
    Method:                           MLE   Df Model:                           10
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1531
    Time:                        13:04:12   Log-Likelihood:                -4280.8
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ===================================================================================
                          coef    std err          z      P>|z|      [95.0% Conf. Int.]
    -----------------------------------------------------------------------------------
    const              -3.9110      0.245    -15.994      0.000        -4.390    -3.432
    Germany             0.7629      0.063     12.041      0.000         0.639     0.887
    Female              0.5283      0.054      9.697      0.000         0.422     0.635
    CreditScore        -0.0007      0.000     -2.378      0.017        -0.001    -0.000
    Age                 0.0727      0.003     28.238      0.000         0.068     0.078
    Tenure             -0.0160      0.009     -1.708      0.088        -0.034     0.002
    Balance          2.637e-06   5.14e-07      5.129      0.000      1.63e-06  3.65e-06
    NumOfProducts      -0.1013      0.047     -2.149      0.032        -0.194    -0.009
    HasCrCard          -0.0449      0.059     -0.757      0.449        -0.161     0.071
    IsActiveMember     -1.0752      0.058    -18.640      0.000        -1.188    -0.962
    EstimatedSalary  4.813e-07   4.74e-07      1.016      0.310     -4.47e-07  1.41e-06
    ===================================================================================

    _____P-values____
    const               1.405607e-57
    Germany             2.162265e-33
    Female              3.114487e-22
    CreditScore         1.739401e-02
    Age                1.996715e-175
    Tenure              8.764548e-02
    Balance             2.913057e-07
    NumOfProducts       3.161546e-02
    HasCrCard           4.489327e-01
    IsActiveMember      1.531167e-77
    EstimatedSalary     3.095145e-01
    dtype: float64

    _____Highest p-value____

     HasCrCard    0.448933
    dtype: float64

    ____Confusion Matrix___
    [[ 7665.   298.]
     [ 1602.   435.]]

    Number of cases correctly predicted: 8100.0 (81.0 %)

New model without the variable with the highest p-value: "HasCrCard"

```python
df_x2 = df_x1.drop(['HasCrCard'], axis = 1)
df_x2[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1
      101348.88

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1
      112542.58

```python
model3 = smdis.Logit(df_y, df_x2).fit()
print model3.summary()
print "\n_____P-values____"
p = model3.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model3.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428109
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9990
    Method:                           MLE   Df Model:                            9
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1531
    Time:                        13:04:12   Log-Likelihood:                -4281.1
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ===================================================================================
                          coef    std err          z      P>|z|      [95.0% Conf. Int.]
    -----------------------------------------------------------------------------------
    const              -3.9444      0.241    -16.395      0.000        -4.416    -3.473
    Germany             0.7619      0.063     12.028      0.000         0.638     0.886
    Female              0.5285      0.054      9.700      0.000         0.422     0.635
    CreditScore        -0.0007      0.000     -2.369      0.018        -0.001    -0.000
    Age                 0.0727      0.003     28.243      0.000         0.068     0.078
    Tenure             -0.0162      0.009     -1.727      0.084        -0.034     0.002
    Balance          2.645e-06   5.14e-07      5.146      0.000      1.64e-06  3.65e-06
    NumOfProducts      -0.1013      0.047     -2.150      0.032        -0.194    -0.009
    IsActiveMember     -1.0744      0.058    -18.631      0.000        -1.187    -0.961
    EstimatedSalary  4.818e-07   4.74e-07      1.017      0.309     -4.47e-07  1.41e-06
    ===================================================================================

    _____P-values____
    const               2.068294e-60
    Germany             2.545658e-33
    Female              3.024897e-22
    CreditScore         1.782382e-02
    Age                1.731605e-175
    Tenure              8.415226e-02
    Balance             2.660186e-07
    NumOfProducts       3.152337e-02
    IsActiveMember      1.809198e-77
    EstimatedSalary     3.090842e-01
    dtype: float64

    _____Highest p-value____

     EstimatedSalary    0.309084
    dtype: float64

    ____Confusion Matrix___
    [[ 7673.   290.]
     [ 1599.   438.]]

    Number of cases correctly predicted: 8111.0 (81.11 %)

New model without the variable with the highest p-value: "EstimatedSalary"

```python
df_x3 = df_x2.drop(['EstimatedSalary'], axis = 1)
df_x3[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1

```python
model4 = smdis.Logit(df_y, df_x3).fit()
print model4.summary()
print "\n_____P-values____"
p = model4.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model4.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428161
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9991
    Method:                           MLE   Df Model:                            8
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1530
    Time:                        13:04:12   Log-Likelihood:                -4281.6
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==================================================================================
                         coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ----------------------------------------------------------------------------------
    const             -3.8959      0.236    -16.528      0.000        -4.358    -3.434
    Germany            0.7621      0.063     12.031      0.000         0.638     0.886
    Female             0.5290      0.054      9.710      0.000         0.422     0.636
    CreditScore       -0.0007      0.000     -2.378      0.017        -0.001    -0.000
    Age                0.0727      0.003     28.238      0.000         0.068     0.078
    Tenure            -0.0160      0.009     -1.710      0.087        -0.034     0.002
    Balance         2.653e-06   5.14e-07      5.162      0.000      1.65e-06  3.66e-06
    NumOfProducts     -0.1005      0.047     -2.132      0.033        -0.193    -0.008
    IsActiveMember    -1.0751      0.058    -18.644      0.000        -1.188    -0.962
    ==================================================================================

    _____P-values____
    const              2.308408e-61
    Germany            2.434718e-33
    Female             2.744698e-22
    CreditScore        1.741371e-02
    Age               2.006986e-175
    Tenure             8.734004e-02
    Balance            2.440782e-07
    NumOfProducts      3.297139e-02
    IsActiveMember     1.407482e-77
    dtype: float64

    _____Highest p-value____

     Tenure    0.08734
    dtype: float64

    ____Confusion Matrix___
    [[ 7676.   287.]
     [ 1598.   439.]]

    Number of cases correctly predicted: 8115.0 (81.15 %)

New model without the variable with the highest p-value: "Tenure"

```python
df_x4 = df_x3.drop(['Tenure'], axis = 1)
df_x4[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Balance
      NumOfProducts
      IsActiveMember

      0
      1
      0
      1
      619
      42
      0.00
      1
      1

      1
      1
      0
      1
      608
      41
      83807.86
      1
      1

```python
model5 = smdis.Logit(df_y, df_x4).fit()
print model5.summary()
print "\n_____P-values____"
p = model5.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model5.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428307
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9992
    Method:                           MLE   Df Model:                            7
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1527
    Time:                        13:04:13   Log-Likelihood:                -4283.1
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==================================================================================
                         coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ----------------------------------------------------------------------------------
    const             -3.9760      0.231    -17.200      0.000        -4.429    -3.523
    Germany            0.7608      0.063     12.014      0.000         0.637     0.885
    Female             0.5306      0.054      9.741      0.000         0.424     0.637
    CreditScore       -0.0007      0.000     -2.377      0.017        -0.001    -0.000
    Age                0.0727      0.003     28.237      0.000         0.068     0.078
    Balance         2.652e-06   5.14e-07      5.160      0.000      1.64e-06  3.66e-06
    NumOfProducts     -0.1010      0.047     -2.144      0.032        -0.193    -0.009
    IsActiveMember    -1.0718      0.058    -18.603      0.000        -1.185    -0.959
    ==================================================================================

    _____P-values____
    const              2.658461e-66
    Germany            3.005769e-33
    Female             2.006779e-22
    CreditScore        1.745446e-02
    Age               2.037936e-175
    Balance            2.463940e-07
    NumOfProducts      3.199990e-02
    IsActiveMember     3.044455e-77
    dtype: float64

    _____Highest p-value____

     NumOfProducts    0.032
    dtype: float64

    ____Confusion Matrix___
    [[ 7676.   287.]
     [ 1601.   436.]]

    Number of cases correctly predicted: 8112.0 (81.12 %)

Since the Accuracy decreases, the variable "Tenure" should be used in the model.
Now, the variable with the highest p-value is "NumOfProducts" with 0.032, which is above of the defined threshold - 0.05.

```python
df_x3[:5]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1

      2
      1
      0
      1
      502
      42
      8
      159660.80
      3
      0

      3
      1
      0
      1
      699
      39
      1
      0.00
      2
      0

      4
      1
      0
      1
      850
      43
      2
      125510.82
      1
      1

#### Transforming Independent Variables

Most common transformations:

1. $$\sqrt{x}$$
2. $$x^2$$
3. $$ln(x)$$

Transform "Balance" variable with ln(x):

Original value: Balance (in 1000$):

        Bal2 = Bal1 + 1unit -->  Bal2 = Bal1 + 1000$

        Scenario1: Bal1 = 1000$
            --> Bal2 = 1000$ + 1000$ = 2000$
        Scenario2: Bal1 = 10000$
            --> Bal2 = 10000$ + 1000$ = 11000$

Log10(Balance + 1):

        log10(Bal2) = log10(Bal1) + 1unit --> Bal2 = Bal1*10

        Scenario1: Bal1 = 1000$
            --> Bal2 = 1000*10 = 10000
    Scenario2: Bal1 = 10000$
            ---> Bal2 = 10000*10 = 100000

Using the original value, means that if someone that starts with a balance of 1000$ and has an increase of 1000$, and someone that starts with a balance of 10000$ and has the same amount of increasing, are two completly different things.

Using the ln transformation, it has the same affect on any person regardless of their starting point. It always a 10 times increase so a unit increase has a consistent increase in the balance variable which is 10 times.

So regardless of who we're segmenting we can say that the effect of a one unit increase in the new transformed variable is consistent throughout our population and that is much more powerful because that does not restrict the logistic regression.

```python
df_x4 = df_x3
df_x4['Balance_log'] = df_x4['Balance'].map(lambda x: np.log10(x+1))
df_x4[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember
      Balance_log

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1
      0.00000

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1
      4.92329

```python
model6 = smdis.Logit(df_y, df_x4).fit()
print model6.summary()
print "\n_____P-values____"
p = model6.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model6.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428134
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9990
    Method:                           MLE   Df Model:                            9
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1530
    Time:                        13:04:14   Log-Likelihood:                -4281.3
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==================================================================================
                         coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ----------------------------------------------------------------------------------
    const             -3.9147      0.237    -16.505      0.000        -4.380    -3.450
    Germany            0.7509      0.065     11.532      0.000         0.623     0.879
    Female             0.5286      0.054      9.701      0.000         0.422     0.635
    CreditScore       -0.0007      0.000     -2.387      0.017        -0.001    -0.000
    Age                0.0727      0.003     28.233      0.000         0.068     0.078
    Tenure            -0.0160      0.009     -1.706      0.088        -0.034     0.002
    Balance         1.871e-06   1.19e-06      1.568      0.117     -4.68e-07  4.21e-06
    NumOfProducts     -0.0957      0.048     -2.013      0.044        -0.189    -0.003
    IsActiveMember    -1.0754      0.058    -18.650      0.000        -1.188    -0.962
    Balance_log        0.0235      0.032      0.728      0.467        -0.040     0.087
    ==================================================================================

    _____P-values____
    const              3.364882e-61
    Germany            9.111212e-31
    Female             2.971672e-22
    CreditScore        1.697954e-02
    Age               2.287651e-175
    Tenure             8.794792e-02
    Balance            1.169019e-01
    NumOfProducts      4.409600e-02
    IsActiveMember     1.262368e-77
    Balance_log        4.668374e-01
    dtype: float64

    _____Highest p-value____

     Balance_log    0.466837
    dtype: float64

    ____Confusion Matrix___
    [[ 7681.   282.]
     [ 1597.   440.]]

    Number of cases correctly predicted: 8121.0 (81.21 %)

The accuracy is improved!

#### Derived Variables

Sometimes it is useful too account for some effects, for example in our case, the balance and age might be correlated, because the older a person is the more whealth he/she can accumulate.

So a new variable can be created:

$$WealthAccumlation = \frac{Balance}{Age}$$

```python
df_x5 = df_x4
df_x5['WealthAccumulation'] = df_x5['Balance']/ df_x5['Age']
df_x5[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember
      Balance_log
      WealthAccumulation

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1
      0.00000
      0.000000

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1
      4.92329
      2044.094146

```python
model7 = smdis.Logit(df_y, df_x5).fit()
print model7.summary()
print "\n_____P-values____"
p = model7.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model7.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.427323
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9989
    Method:                           MLE   Df Model:                           10
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1546
    Time:                        13:04:15   Log-Likelihood:                -4273.2
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ======================================================================================
                             coef    std err          z      P>|z|      [95.0% Conf. Int.]
    --------------------------------------------------------------------------------------
    const                 -3.4253      0.266    -12.896      0.000        -3.946    -2.905
    Germany                0.7534      0.066     11.483      0.000         0.625     0.882
    Female                 0.5289      0.055      9.693      0.000         0.422     0.636
    CreditScore           -0.0007      0.000     -2.326      0.020        -0.001    -0.000
    Age                    0.0611      0.004     15.903      0.000         0.054     0.069
    Tenure                -0.0165      0.009     -1.762      0.078        -0.035     0.002
    Balance             8.362e-06   2.02e-06      4.143      0.000      4.41e-06  1.23e-05
    NumOfProducts         -0.1011      0.048     -2.119      0.034        -0.195    -0.008
    IsActiveMember        -1.0601      0.058    -18.349      0.000        -1.173    -0.947
    Balance_log            0.0213      0.032      0.657      0.511        -0.042     0.085
    WealthAccumulation    -0.0003   6.51e-05     -3.954      0.000        -0.000    -0.000
    ======================================================================================

    _____P-values____
    const                 4.756747e-38
    Germany               1.599880e-30
    Female                3.244594e-22
    CreditScore           2.000443e-02
    Age                   6.030631e-57
    Tenure                7.801725e-02
    Balance               3.428919e-05
    NumOfProducts         3.411316e-02
    IsActiveMember        3.376037e-75
    Balance_log           5.111832e-01
    WealthAccumulation    7.673375e-05
    dtype: float64

    _____Highest p-value____

     Balance_log    0.511183
    dtype: float64

    ____Confusion Matrix___
    [[ 7662.   301.]
     [ 1586.   451.]]

    Number of cases correctly predicted: 8113.0 (81.13 %)

This new variable had a negative effect in the model accuracy.

However, one should have in consideration that might be some collinearity effects because wealth accumulation includes the variables balance and age that are already in the model. So basically this means that this new variable might be somehow correlated with the other two and including all of them in the model might cause some damage, so on of them should be excluded.

#### Multicollinearity

Multicollinearity is a phenomenon in which two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a substantial degree of accuracy.

The "Variance Inflation Factors - VIF" can be used to measure the degree of multicollinearity.

Minimum possible value = 1.0
Values > 5 may indicate a collinearity problem.

$$VIF(j) = \frac{1}{( 1 - R(j)^2)}$$

where R(j) is the multiple correlarion coefficient between variable j and the other independent variables

```python
i = 0
for column in df_x5.columns:

    print column + " %s" % outliers.variance_inflation_factor(df_x5.values, i)
    i += 1
```

    const 91.4773722803
    Germany 1.27101224421
    Female 1.00321582613
    CreditScore 1.00104450767
    Age 2.10487620512
    Tenure 1.00161073156
    Balance 20.8895673667
    NumOfProducts 1.15312466504
    IsActiveMember 1.01384972933
    Balance_log 8.72881202924
    WealthAccumulation 14.5598595824

Rerun without the "Balance_log" variable:

```python
df_x6 = df_x5.drop('Balance_log', axis = 1)
```

```python
model8 = smdis.Logit(df_y, df_x6).fit()
print model8.summary()
print "\n_____P-values____"
p = model8.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model8.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.427344
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9990
    Method:                           MLE   Df Model:                            9
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1546
    Time:                        13:04:15   Log-Likelihood:                -4273.4
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ======================================================================================
                             coef    std err          z      P>|z|      [95.0% Conf. Int.]
    --------------------------------------------------------------------------------------
    const                 -3.4089      0.264    -12.904      0.000        -3.927    -2.891
    Germany                0.7636      0.064     11.967      0.000         0.639     0.889
    Female                 0.5292      0.055      9.699      0.000         0.422     0.636
    CreditScore           -0.0007      0.000     -2.318      0.020        -0.001    -0.000
    Age                    0.0612      0.004     15.938      0.000         0.054     0.069
    Tenure                -0.0165      0.009     -1.766      0.077        -0.035     0.002
    Balance             9.078e-06    1.7e-06      5.351      0.000      5.75e-06  1.24e-05
    NumOfProducts         -0.1054      0.047     -2.229      0.026        -0.198    -0.013
    IsActiveMember        -1.0598      0.058    -18.343      0.000        -1.173    -0.947
    WealthAccumulation    -0.0003   6.49e-05     -3.965      0.000        -0.000    -0.000
    ======================================================================================

    _____P-values____
    const                 4.282278e-38
    Germany               5.318191e-33
    Female                3.042840e-22
    CreditScore           2.047417e-02
    Age                   3.455661e-57
    Tenure                7.744433e-02
    Balance               8.770661e-08
    NumOfProducts         2.578202e-02
    IsActiveMember        3.744959e-75
    WealthAccumulation    7.325518e-05
    dtype: float64

    _____Highest p-value____

     Tenure    0.077444
    dtype: float64

    ____Confusion Matrix___
    [[ 7672.   291.]
     [ 1586.   451.]]

    Number of cases correctly predicted: 8123.0 (81.23 %)

```python
i = 0

for column in df_x6.columns:

    print column + " %s" % outliers.variance_inflation_factor(df_x6.values, i)
    i += 1
```

    const 90.6320775047
    Germany 1.2161676569
    Female 1.00302785174
    CreditScore 1.00097995156
    Age 2.10477505278
    Tenure 1.00153406128
    Balance 14.0245653407
    NumOfProducts 1.12330226029
    IsActiveMember 1.01353234553
    WealthAccumulation 14.5598562679

Create new variable log for wealth accumulation:

```python
df_x7 = df_x5
df_x7['WealthAccumulation_log'] = df_x7['WealthAccumulation'].map(lambda x: np.log10(x + 1))
df_x7[:2]
```

      const
      Germany
      Female
      CreditScore
      Age
      Tenure
      Balance
      NumOfProducts
      IsActiveMember
      Balance_log
      WealthAccumulation
      WealthAccumulation_log

      0
      1
      0
      1
      619
      42
      2
      0.00
      1
      1
      0.00000
      0.000000
      0.000000

      1
      1
      0
      1
      608
      41
      1
      83807.86
      1
      1
      4.92329
      2044.094146
      3.310713

```python
df_x7 = df_x7.drop(['Balance', 'WealthAccumulation'], axis = 1)
```

```python
model9 = smdis.Logit(df_y, df_x7).fit()
print model9.summary()
print "\n_____P-values____"
p = model9.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model9.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.427891
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9990
    Method:                           MLE   Df Model:                            9
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1535
    Time:                        13:04:16   Log-Likelihood:                -4278.9
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==========================================================================================
                                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ------------------------------------------------------------------------------------------
    const                     -3.5735      0.267    -13.362      0.000        -4.098    -3.049
    Germany                    0.7516      0.065     11.492      0.000         0.623     0.880
    Female                     0.5263      0.055      9.655      0.000         0.420     0.633
    CreditScore               -0.0007      0.000     -2.386      0.017        -0.001    -0.000
    Age                        0.0647      0.004     16.631      0.000         0.057     0.072
    Tenure                    -0.0162      0.009     -1.729      0.084        -0.035     0.002
    NumOfProducts             -0.0983      0.048     -2.063      0.039        -0.192    -0.005
    IsActiveMember            -1.0687      0.058    -18.516      0.000        -1.182    -0.956
    Balance_log                0.8699      0.297      2.932      0.003         0.288     1.451
    WealthAccumulation_log    -1.1762      0.435     -2.702      0.007        -2.029    -0.323
    ==========================================================================================

    _____P-values____
    const                     1.007546e-40
    Germany                   1.441579e-30
    Female                    4.661027e-22
    CreditScore               1.704065e-02
    Age                       4.171785e-62
    Tenure                    8.382654e-02
    NumOfProducts             3.907377e-02
    IsActiveMember            1.520894e-76
    Balance_log               3.373142e-03
    WealthAccumulation_log    6.887931e-03
    dtype: float64

    _____Highest p-value____

     Tenure    0.083827
    dtype: float64

    ____Confusion Matrix___
    [[ 7685.   278.]
     [ 1597.   440.]]

    Number of cases correctly predicted: 8125.0 (81.25 %)

```python
i = 0

for column in df_x7.columns:

    print column + " %s" % outliers.variance_inflation_factor(df_x7.values, i)
    i += 1
```

    const 93.5608074531
    Germany 1.26986359735
    Female 1.00282038022
    CreditScore 1.00102945704
    Age 2.26479838945
    Tenure 1.00165459162
    NumOfProducts 1.15243000098
    IsActiveMember 1.01168357579
    Balance_log 705.940971113
    WealthAccumulation_log 704.739971273

"Balance_log" and "WealthAccumulation_log" have too large values which means that these two variables are basically the same thing. So one of these variables must be excluded!

```python
df_x8 = df_x7.drop(['Balance_log'], axis = 1)
```

```python
model10 = smdis.Logit(df_y, df_x8).fit()
print model10.summary()
print "\n_____P-values____"
p = model10.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model10.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428322
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9991
    Method:                           MLE   Df Model:                            8
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1527
    Time:                        13:04:18   Log-Likelihood:                -4283.2
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==========================================================================================
                                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ------------------------------------------------------------------------------------------
    const                     -3.9325      0.239    -16.467      0.000        -4.401    -3.464
    Germany                    0.7515      0.065     11.550      0.000         0.624     0.879
    Female                     0.5266      0.054      9.671      0.000         0.420     0.633
    CreditScore               -0.0007      0.000     -2.409      0.016        -0.001    -0.000
    Age                        0.0733      0.003     28.398      0.000         0.068     0.078
    Tenure                    -0.0159      0.009     -1.697      0.090        -0.034     0.002
    NumOfProducts             -0.0967      0.048     -2.034      0.042        -0.190    -0.004
    IsActiveMember            -1.0763      0.058    -18.672      0.000        -1.189    -0.963
    WealthAccumulation_log     0.0985      0.020      4.812      0.000         0.058     0.139
    ==========================================================================================

    _____P-values____
    const                      6.349678e-61
    Germany                    7.379902e-31
    Female                     4.015783e-22
    CreditScore                1.598070e-02
    Age                       2.132441e-177
    Tenure                     8.978088e-02
    NumOfProducts              4.195258e-02
    IsActiveMember             8.443410e-78
    WealthAccumulation_log     1.490621e-06
    dtype: float64

    _____Highest p-value____

     Tenure    0.089781
    dtype: float64

    ____Confusion Matrix___
    [[ 7688.   275.]
     [ 1597.   440.]]

    Number of cases correctly predicted: 8128.0 (81.28 %)

Correlation between variables:

```python
df_x5[['Age', 'Balance_log', 'WealthAccumulation_log', 'WealthAccumulation']].corr()
```

      Age
      Balance_log
      WealthAccumulation_log
      WealthAccumulation

      Age
      1.000000
      0.034530
      -0.007524
      -0.246293

      Balance_log
      0.034530
      1.000000
      0.998404
      0.865141

      WealthAccumulation_log
      -0.007524
      0.998404
      1.000000
      0.888872

      WealthAccumulation
      -0.246293
      0.865141
      0.888872
      1.000000

As one can observe, the "WealthAccumulation_log" and "Balance_log" variables are highly correlated - they are basically the same which is very bad for the model.

Thumb rule: anything 0.9 is very high correlation. Correlations above 0.5 should be addressed.

```python
df_final = df_x5.drop(['Balance', 'WealthAccumulation', 'WealthAccumulation_log'], axis = 1)
```

```python
model11 = smdis.Logit(df_y, df_final).fit()
print model11.summary()
print "\n_____P-values____"
p = model11.pvalues
print p
print "\n_____Highest p-value____"
print "\n %s \n" % p[p ==max(p)]
print "\n____Confusion Matrix___"
cm = model11.pred_table()
print cm
print "\nNumber of cases correctly predicted: %s (%s %%)" % (cm[0][0] + cm[1][1], (cm[0][0] + cm[1][1])*100/np.sum(cm))
```

    Optimization terminated successfully.
             Current function value: 0.428257
             Iterations 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                 Exited   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9991
    Method:                           MLE   Df Model:                            8
    Date:                Wed, 28 Dec 2016   Pseudo R-squ.:                  0.1528
    Time:                        13:04:19   Log-Likelihood:                -4282.6
    converged:                       True   LL-Null:                       -5054.9
                                            LLR p-value:                     0.000
    ==================================================================================
                         coef    std err          z      P>|z|      [95.0% Conf. Int.]
    ----------------------------------------------------------------------------------
    const             -3.9126      0.237    -16.497      0.000        -4.377    -3.448
    Germany            0.7476      0.065     11.492      0.000         0.620     0.875
    Female             0.5267      0.054      9.672      0.000         0.420     0.633
    CreditScore       -0.0007      0.000     -2.408      0.016        -0.001    -0.000
    Age                0.0727      0.003     28.221      0.000         0.068     0.078
    Tenure            -0.0159      0.009     -1.699      0.089        -0.034     0.002
    NumOfProducts     -0.0950      0.048     -1.999      0.046        -0.188    -0.002
    IsActiveMember    -1.0758      0.058    -18.662      0.000        -1.189    -0.963
    Balance_log        0.0690      0.014      4.945      0.000         0.042     0.096
    ==================================================================================

    _____P-values____
    const              3.835044e-61
    Germany            1.441009e-30
    Female             3.970390e-22
    CreditScore        1.604464e-02
    Age               3.239286e-175
    Tenure             8.932416e-02
    NumOfProducts      4.562540e-02
    IsActiveMember     1.011855e-77
    Balance_log        7.619575e-07
    dtype: float64

    _____Highest p-value____

     Tenure    0.089324
    dtype: float64

    ____Confusion Matrix___
    [[ 7687.   276.]
     [ 1597.   440.]]

    Number of cases correctly predicted: 8127.0 (81.27 %)

```python
i = 0

for column in df_final.columns:

    print column + " %s" % outliers.variance_inflation_factor(df_final.values, i)
    i += 1
```

    const 76.3936634172
    Germany 1.26934389528
    Female 1.00281465894
    CreditScore 1.00102204693
    Age 1.01171547993
    Tenure 1.0014367308
    NumOfProducts 1.15145597018
    IsActiveMember 1.01028026511
    Balance_log 1.42007014313

Now, none of the variables presents multicollinearity.

#### CAP - Cumulative Accuracy Profile Curve

```python
df_cap = df[['RowNumber', 'Exited']]
df_cap['phat'] = model11.predict()
```

```python
df_cap.sort('phat', ascending=False)[:5]
```

      RowNumber
      Exited
      phat

      4815
      4816
      0
      0.930669

      3531
      3532
      1
      0.928311

      9587
      9588
      0
      0.923303

      7499
      7500
      1
      0.906178

      9555
      9556
      1
      0.904457

```python
Total_Exited = sum(df_cap['Exited'])
print "Total Exited: %s" % Total_Exited
Total_Records = len(df)
print "Total number of clients: %s" % Total_Records
Exit_Ratio = Total_Exited/Total_Records
print "Ratio of exits: %s " % Exit_Ratio
```

    Total Exited: 2037
    Total number of clients: 10000
    Ratio of exits: 0.2037

First, I select the clients that have a higher probability.

```python
df_cap['total_selected'] = range(1,len(df_cap)+1)
df_cap['total_selected_per'] = df_cap['total_selected'].map(lambda x: round(x*100/Total_Records, 1))
df_cap['random_selection'] = df_cap['total_selected'].map(lambda x: x*Exit_Ratio)
df_cap['random_selection_per'] = df_cap['total_selected'].map(lambda x: round(x*100/Total_Records, 1))
```

```python
df_cap = df_cap.sort('phat', ascending=False)
```

```python
df_cap['model_select'] = df_cap.Exited.cumsum()
df_cap['model_select_per'] = df_cap['model_select'].map(lambda x: round(x*100/Total_Exited, 1))
```

```python
df_cap[:5]
```

      RowNumber
      Exited
      phat
      total_selected
      total_selected_per
      random_selection
      random_selection_per
      model_select
      model_select_per

      4815
      4816
      0
      0.930669
      1
      0.0
      0.2037
      0.0
      0
      0.0

      3531
      3532
      1
      0.928311
      2
      0.0
      0.4074
      0.0
      1
      0.0

      9587
      9588
      0
      0.923303
      3
      0.0
      0.6111
      0.0
      1
      0.0

      7499
      7500
      1
      0.906178
      4
      0.0
      0.8148
      0.0
      2
      0.1

      9555
      9556
      1
      0.904457
      5
      0.1
      1.0185
      0.1
      3
      0.1

```python
df_cap = df_cap.sort('RowNumber')
plt.plot(df_cap['total_selected_per'], df_cap['random_selection_per'], label = 'random model' )
plt.plot(df_cap['total_selected_per'], df_cap['model_select_per'], label = 'Good Model' )
plt.plot([50, 50], [80, 0], 'r--', lw=1, label = 'x = 50%, y = X')
plt.plot([0, 50], [80, 80], 'r--', lw=1)
plt.plot([10, 100], [99, 99], 'y-', lw=3, label = 'perfect model')
plt.plot([0, 10], [0, 99], 'y-', lw=3)
plt.legend(bbox_to_anchor=(1, 1), loc='upper left', ncol=1)
plt.xlabel('Total Contacted')
plt.ylabel('Total Purchased')
plt.show()
```

The closer the green line is to the yellow line, the better the model. The closer to the blue line, the worse.

__Accuracy Ratio__:
    - Take the area under the perfect model (yellow line) to the random model (blue line) -
$$a_p$$
    - Take the area under the good model (green line) to the random model (blue line) -

$$a_r$$
    - So the accuracy ratio:

$$AR = \frac{a_r}{a_p}$$

that is between 0 and 1, and the closer the ratio is to 1 the better.

__Rule of thumb:__

Another way of evaluating the model is to look to the red line, x = 50%, and y = X:

If:

    90% < X < 100% -> Too Good (overfitting?)
    80% < X < 90% -> Very Good
    70% < X < 80% -> Good
    60% < X < 70% -> Poor
    X < 60% -> Rubbish

#### Test model with test data

```python
df_test = pandas.DataFrame.from_csv('Churn-Modelling-Test-Data.csv', index_col=None)
df_test[:5]
```

      RowNumber
      CustomerId
      Surname
      CreditScore
      Geography
      Gender
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary
      Exited

      0
      10001
      15798485
      Copley
      565
      France
      Male
      31
      1
      0.00
      1
      0
      1
      20443.08
      0

      1
      10002
      15588959
      T'ang
      569
      France
      Male
      34
      4
      0.00
      1
      0
      1
      4045.90
      0

      2
      10003
      15624896
      Ku
      669
      France
      Female
      20
      7
      0.00
      2
      1
      0
      128838.67
      0

      3
      10004
      15639629
      McConnan
      694
      France
      Male
      39
      4
      173255.48
      1
      1
      1
      81293.10
      0

      4
      10005
      15638852
      Ts'ui
      504
      Spain
      Male
      28
      10
      109291.36
      1
      1
      1
      187593.15
      0

```python
df_test = sm.add_constant(df_test)
```

```python
df_test_y = df_test['Exited']
df_test = df_test.drop(['Exited'], axis = 1)
```

```python
dummy = pandas.get_dummies(df_test['Gender'])
df_test= dummy.join(df_test)
dummy = pandas.get_dummies(df_test['Geography'])
df_test = dummy.join(df_test)
df_test[:5]
```

      France
      Germany
      Spain
      Female
      Male
      const
      RowNumber
      CustomerId
      Surname
      CreditScore
      Geography
      Gender
      Age
      Tenure
      Balance
      NumOfProducts
      HasCrCard
      IsActiveMember
      EstimatedSalary

      0
      1
      0
      0
      0
      1
      1
      10001
      15798485
      Copley
      565
      France
      Male
      31
      1
      0.00
      1
      0
      1
      20443.08

      1
      1
      0
      0
      0
      1
      1
      10002
      15588959
      T'ang
      569
      France
      Male
      34
      4
      0.00
      1
      0
      1
      4045.90

      2
      1
      0
      0
      1
      0
      1
      10003
      15624896
      Ku
      669
      France
      Female
      20
      7
      0.00
      2
      1
      0
      128838.67

      3
      1
      0
      0
      0
      1
      1
      10004
      15639629
      McConnan
      694
      France
      Male
      39
      4
      173255.48
      1
      1
      1
      81293.10

      4
      0
      0
      1
      0
      1
      1
      10005
      15638852
      Ts'ui
      504
      Spain
      Male
      28
      10
      109291.36
      1
      1
      1
      187593.15

```python
df_test['Balance_log'] = df_test['Balance'].map(lambda x: np.log10(x+1))
```

Check the parameters included in the final model (model11) and rerun the model with the same parameters but now including the test data:

```python
model11.params
```

    const            -3.912576
    Germany           0.747595
    Female            0.526721
    CreditScore      -0.000675
    Age               0.072655
    Tenure           -0.015879
    NumOfProducts    -0.095020
    IsActiveMember   -1.075776
    Balance_log       0.069026
    dtype: float64

```python
df_test = df_test[['Germany', 'Female', 'CreditScore', 'Age', 'Tenure', 'NumOfProducts', 'IsActiveMember', 'Balance_log', 'const']]
```

```python
df_total = df_final.append(df_test)
```

```python
model_final = smdis.Logit(df_y, df_total[:10000]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.428257
             Iterations 6

```python
y_pred = model_final.predict(df_total[10000:])
y_pred[:5]
```

    array([ 0.03811657,  0.04476137,  0.06383845,  0.08165338,  0.03915974])

Let's construct the CAP curve:

```python
df_pred = pandas.DataFrame()
df_pred['phat'] = y_pred
df_pred['Exited'] = df_test_y
df_pred = df_pred.sort( 'phat', ascending=False)
df_pred[:5]
```

      phat
      Exited

      957
      0.877614
      1

      249
      0.857731
      1

      429
      0.832754
      1

      209
      0.826484
      1

      693
      0.826381
      0

```python
Total_Exited = sum(df_pred['Exited'])
print "Total Exited: %s" % Total_Exited
Total_Records = len(df_pred)
print "Total number of clients: %s" % Total_Records
Exit_Ratio = Total_Exited/Total_Records
print "Ratio of exits: %s" % Exit_Ratio
```

    Total Exited: 260
    Total number of clients: 1000
    Ratio of exits: 0.26

```python
df_pred['total_selected'] = range(1,len(df_pred)+1)
df_pred['total_selected_per'] = df_pred['total_selected'].map(lambda x: round(x*100/Total_Records, 1))
df_pred['random_selection'] = df_pred['total_selected'].map(lambda x: x*Exit_Ratio)
df_pred['random_selection_per'] = df_pred['total_selected'].map(lambda x: round(x*100/Total_Records, 1))
```

```python
df_pred = df_pred.sort('phat', ascending=False)
```

```python
df_pred['model_select'] = df_pred.Exited.cumsum()
df_pred['model_select_per'] = df_pred['model_select'].map(lambda x: round(x*100/Total_Exited, 1))
```

```python
df_pred[:5]
```

      phat
      Exited
      total_selected
      total_selected_per
      random_selection
      random_selection_per
      model_select
      model_select_per

      957
      0.877614
      1
      1
      0.1
      0.26
      0.1
      1
      0.4

      249
      0.857731
      1
      2
      0.2
      0.52
      0.2
      2
      0.8

      429
      0.832754
      1
      3
      0.3
      0.78
      0.3
      3
      1.2

      209
      0.826484
      1
      4
      0.4
      1.04
      0.4
      4
      1.5

      693
      0.826381
      0
      5
      0.5
      1.30
      0.5
      4
      1.5

```python
plt.plot(df_pred['total_selected_per'], df_pred['random_selection_per'], label = 'random model' )
plt.plot(df_pred['total_selected_per'], df_pred['model_select_per'], label = 'Good Model' )
plt.plot([50, 50], [76, 0], 'r--', lw=1, label = 'x = 50%, y = X')
plt.plot([0, 50], [76, 76], 'r--', lw=1)
plt.plot([10, 100], [99, 99], 'y-', lw=3, label = 'perfect model')
plt.plot([0, 10], [0, 99], 'y-', lw=3)
plt.legend(bbox_to_anchor=(1, 1), loc='upper left', ncol=1)
plt.xlabel('Total Contacted')
plt.ylabel('Total Purchased')
plt.show()
```

__Conclusions__:

- The model is performing above 80%, so a bit worse than the training data (a drop of about 3%).
- The concentration of data is about 10 times less than the training data which explains the ruggedness of the curve.

### Parameters Interpretation

Odds ratio:

$$odds = \frac{p}{1-p}$$

Multiple logistic regression:

$$ln(\frac{p}{1-p}) = b_0 + b_0x_1 + ... + b_nx_n$$

So we can write:

$$ln(odds) = b_0 + b_0x_1 + ... + b_nx_n$$

$$<=> odds = e^{b_0 + b_0x_1 + ... + b_nx_n}$$

$$<=> odds = e^{b_0} + e^{b_0x_1} + ... + e^{b_nx_n}$$

which means that, if we increase an unit to the $$x_1$$ variable,  for example:

$$e^{b_1x_1} -> e^{b_1(x_1 + 1)} = e^{b_1x_1}e^{b_1}$$

which means that, increading an independente variable, e.g. $$x_1$$, by 1 unit, will increase the odds by a multiplicative factor of $$e^{b_i}$$

So let's transform the coefficients of our model into odds ratios:

```python
odds_ratio = np.exp(model_final.params)
odds_ratio
```

    Age               1.075360
    Balance_log       1.071464
    CreditScore       0.999325
    Female            1.693371
    Germany           2.111916
    IsActiveMember    0.341033
    NumOfProducts     0.909355
    Tenure            0.984246
    const             0.019989
    dtype: float64

- "Germany" has the greater odds ratio, which means that being in Germany has the highest impact for people leaving the bank - most important independent variable that influences the model. If we change from not Germany to Germany, the odds ratio increases by 2.1.
- The next one is gender. One we go from a male customer to a female customer with all else held constant, the odds ratio of a female custormer leaving is 1.7 times grater that a Male customer leaving. So basically the odds ratio increases by 70%.
- Being an "ActiveMember" has the oppositive effect because the value is less than 1. This means that when we go from a member who is not active to a member that is active, the odds ratio gets multiplied by 0.34. So basically it drops by 66% which is also great in terms of retention.
- Anything above 0.8 and bellow 1.2 has not a great impact and can be ignored.


```python
from sympy import *
init_printing()
```

```python
K,L,r,w,p,T = symbols('K L r w p T',
                  real=True,
                  positive=True,
                  finite=True)

production = T * K * L
cost = r*K + w*L**2
profit = p * production - cost
profit
```

```python
DK = profit.diff(K)
DL = profit.diff(L)

DK, DL
```

```python
A = Matrix([
    [0, T*p],
    [T*p, -2*w]
])

b = Matrix([
    [r],
    [0]
])

A, b
```

```python
d = A.det()
d
```

Assume that $p >0$.

```python
Ainv = A.inv()
Ainv
```

```python
sol = solve([DK, DL], [K, L], dict=True)
sol
```

```python
Hessian = Matrix([
    [DK.diff(K), DK.diff(L)],
    [DL.diff(K), DL.diff(L)]
])
Hessian
```

```python
# What is the rank of the matrix A?
A.rank()
```

```python

```


## The governing equations

\begin{align}
  \partial_t \mathbf u
      = &- (\mathbf{u}.\nabla) \mathbf{u} - c^2 \nabla h - f\mathbf{e_z} \times
\mathbf u  \\
  \partial_t h
      =& - \nabla. (h \mathbf u)
\end{align}

Equations may also be written in the \textit{rotational form} as:

\begin{equation}
 \partial_t \mathbf u
 = - \nabla |\mathbf u|^2/2 - c^2 \nabla h - \zeta \times \mathbf u
\end{equation}
where, $\zeta$
represents absolute vorticity, i.e. the sum of vorticity and system rotation.

## Linearised governing equations

### without beta effect
  \begin{align}
    \partial_t \mathbf u =& - c^2 \nabla \eta - f\mathbf{e_z} \times \mathbf u  \\
    \partial_t \eta =& - \nabla.  \mathbf u
  \end{align}

Taking curl and divergence of the linearized equation for momentum  gives the following evolution equations with change of variables:

  \begin{align}
    \partial_t \zeta =& - f \delta\\
    \partial_t \delta =&  f \zeta - c^2 \nabla^2 \eta \\
    \partial_t \eta =& - \delta
  \end{align}

as an eigenvalue problem:
\begin{align}
 i\omega
   \begin{Bmatrix}
       \hat{\zeta} \\ \hat{\delta} \\c\kappa\hat{\eta}
  \end{Bmatrix}
  = i
  \begin{bmatrix}
     0 & if & 0 \\
   -if &  0 & -ic\kappa \\
     0 & ic\kappa & 0
  \end{bmatrix}
  \begin{Bmatrix}
       \hat{\zeta} \\ \hat{\delta} \\c\kappa\hat{\eta}
  \end{Bmatrix}
\end{align}
where, $\kappa = |\mathbf{k}|$.

### with beta effect
  \begin{align}
    \partial_t u =& - c^2 \partial_x \eta + (f+\beta y)v  \\
    \partial_t v =& - c^2 \partial_y \eta - (f+\beta y)u  \\
    \partial_t \eta =& - \nabla.  \mathbf u
  \end{align}

#### IMPORTANT: must be derived from spherical coordinates!
But instead if we use the nabla operator from the cartesian coordinates, and take curl of the coriolis term:

$$ [\nabla \times ({(f + \beta y)\hat{e}_z \times \mathbf{u}})].\hat{e}_z $$

$$ = [(f + \beta y)\hat e_z \delta - \mathbf{u} (\nabla . \hat e_z  (f + \beta y)) +(\mathbf{u}.\nabla)\hat e_z(f + \beta y) - ((f + \beta y) \hat e_z . \nabla) \mathbf{u}].\hat{e}_z $$

We remove:
* Term ... because we assume the vertical velocity \pa
$w << (u, v)$.

Taking curl and divergence of the linearized equation for momentum  gives the following evolution equations with change of variables:
$$
  \begin{align}
    \partial_t \zeta =& - f \delta  - v \beta \\
    \partial_t \delta =&  f \zeta + u \beta - c^2 \nabla^2 \eta \\
    \partial_t \eta =& - \delta
  \end{align}
$$

Apply helmholtz decomposition,

$\mathbf u = - (\nabla \times e_z \psi) + \nabla \chi$

\begin{align}
        u &= -\psi_y + \chi_x \\
        v &= \psi_x + \chi_y \\
        \zeta &= \nabla^2 \psi \\
        \delta &= \nabla^2 \chi
\end{align}

as an eigenvalue problem:
\begin{align}
 &i\omega
   \begin{Bmatrix}
       \hat{\psi} \\ \hat{\chi} \\ c\hat{\eta}/\kappa
  \end{Bmatrix}\\
 =&
  \begin{bmatrix}
     -i\beta k_x/\kappa^2 & f - i\beta k_y/\kappa^2 & 0 \\
     -f-i\beta k_y/\kappa^2 &  +i\beta k_x/\kappa^2 & c \kappa \\
     0 & -c\kappa & 0
  \end{bmatrix}
  \begin{Bmatrix}
       \hat{\psi} \\ \hat{\chi} \\ c\hat{\eta}/\kappa
  \end{Bmatrix}
\end{align}

\begin{align}
 =& i
  \begin{bmatrix}
     -\beta k_x/\kappa^2 & -if - \beta k_y/\kappa^2 & 0 \\
     if-\beta k_y/\kappa^2 &  \beta k_x/\kappa^2 & -ic \kappa \\
     0 & ic\kappa & 0
  \end{bmatrix}
  \begin{Bmatrix}
       \hat{\psi} \\ \hat{\chi} \\ c\hat{\eta}/\kappa
  \end{Bmatrix}
\end{align}

```python
# coding: utf-8
from sympy import *
from sympy.abc import psi, chi, eta, omega

init_printing()
f, ck, c = symbols('f ck c', real=True)
sigma, kappa, beta = symbols(r'\sigma \kappa \beta', real=True)
kx, ky, kh =  symbols('k_x k_y k_h', real=True)
u,v =  symbols('u v')

def subsigma(M):
    M = M.subs(f**2 + ck**2,  sigma**2)
    M = M.subs(f**2 + c**2*kappa**2, sigma**2)
    return M

def simplifysigma(M):
    M = M.subs(sigma, sqrt(f**2 + ck**2) )
    M = simplify(M)
    return subsigma(M)

def subkappa(M):
    M = M.subs(kx**2 + ky**2,  kappa**2)
    M = M.subs(kh*kh.conjugate(), kappa**2)
    M = M.subs(kx*kx.conjugate() + ky*ky.conjugate(), kappa**2)
    return M

def simplifykappa(M):
    M = M.subs(kappa, sqrt(kx**2 + ky**2))
    M = M.subs(kh, sqrt(kx**2 + ky**2))
    M = simplify(M)
    return subkappa(M)

def subck(M):
    return M.subs(ck,c*kappa)
```

```python
A = Matrix(([-beta*kx/kappa**2, -1j*f - beta*ky/kappa**2, 0],
            [1j*f - beta*ky/kappa**2, beta*kx/kappa**2, -1j*c*kappa],
            [0,1j*c*kappa,0]))

```

```python
A
```

```python
Ah = A.conjugate().transpose()
Ah - A
```

```python
A.det()
```

```python
A.conjugate().transpose()
```

```python
charpoly = A.charpoly(omega)
```

```python
chareqn = Eq(subsigma(simplify(charpoly.expand())),0)
```

```python
chareqn
```

```python
solve(chareqn, omega)
```

```python
L = A.eigenvals()
X = A.eigenvects()
```

```python
L
```

```python
A
```

```python
XX = A.diagonalize()[0]
XX = subsigma(XX)
sig = sqrt(f**2 + ck**2)

NORM = eye(3)
NORM[0,0] = f/sig
NORM[1,1] = ck/(sig*sqrt(2))
NORM[2,2] = ck/(sig*sqrt(2))
XN = subsigma(XX*NORM)

#CONJ = eye(3)
#CONJ[1,1] = -1
#XNCT = (CONJ*XN).transpose()
XNCT = XN.conjugate().transpose()
#XNCT = subsigma(XNCT)

#------------Check-----------#
ID = simplifysigma(XN*XNCT)
print 'Identity matrix:'
```

```python
ID
```

```python
print 'Norm. Eigenvector matrix Xn:'
```

```python
XN
```

```python
W = Matrix( [psi, -phi, c*eta/kappa] ) * kappa ** 2
U = Matrix( [u,v,eta] )
P = Matrix(( [-1j*ky, 1j*kx, 0],[1j*kx,1j*ky,0],[0,0,kappa] ))

Pinv = simplifykappa(P.inv())
XNCTinv = simplifysigma(XNCT.inv())

Q = Pinv*(XNCTinv)
Q = simplifysigma(Q)
Q = simplifykappa(Q)
Q = subck(Q)
```

```python
print 'inversion:'; Pinv
```

```python
U,Q,W
```

```python
NORM = eye(3)
NORM[2,2] = c
U = NORM*U
#Q = NORM*Q
#W = NORM*W
```

```python
U, Q, W
```

```python
#RHS BB*
N = XNCT*W
B = expand(subck(N / kappa))
B = simplifysigma(B)
```

```python
collect_list = [psi*psi.conjugate(), eta*eta.conjugate(), phi*phi.conjugate(),
                psi*phi.conjugate(), psi*eta.conjugate(),
                phi*psi.conjugate(), phi*eta.conjugate(),
                eta*psi.conjugate(), eta*phi.conjugate()]
```

```python
BB2 = subck(simplifykappa(expand(B.conjugate().dot(B))))
collect(BB2,collect_list)
```

```python
kappa**2 * psi*psi.conjugate() + kappa**2 * phi*phi.conjugate() + c**2 * eta*eta.conjugate()
```

```python
#LHS uu*
U.transpose().conjugate()*U
```

```python
simplifykappa(Q.transpose()*Q)
```

```python
#U=[Q]W
QW = Q*W
```

```python
#collect(subck(simplifykappa(expand(QW.transpose().conjugate()*QW))), collect_list)
QW2 = QW.conjugate().dot(QW)
QW2 = simplifysigma(subck((simplifykappa(expand(QW.conjugate().dot(QW))))))
QW2 = collect(QW2,collect_list)
QW2
```

```python
simplifykappa(collect( U.conjugate().dot((P.conjugate().transpose()*P)*U) ,
        [u*u.conjugate(), v*v.conjugate(), eta*eta.conjugate()]))
```

```python
subkappa(P.conjugate().transpose()*P)
```

```python
P*P.conjugate().transpose()
```

```python
P
```

```python
XNCT*XN
```

```python
simplifysigma(_)
```

```python
P
```

```python
P*U
```

```python
Q
```

```python
Q*kappa*abs(sigma)
```

```python
XNCT*P/kappa
```

```python
subck(XNCT)*W/kappa
```

```python
Pinv,XNCTinv
```

```python
Q
```

```python
W,P,U
```

```python
P*U
```

```python
Q = kappa*Pinv*(XNCTinv)
Q = simplifysigma(Q)
Q = simplifykappa(Q)
Q = subck(Q)
Q*W
```

```python
simplify(Q*B)
```

```python
Q
```

```python

```


## The inviscid Burgers' Equation

#### Problem Setup

Then $u_0(x) := u(x,0) = x$

Using the backward Euler scheme, the equation can be re-written as:

We set $u_n = \mu_{n-1}$, where $\mu_{n-1}$ is the mean of $u_{n-1}$, to deal with the non-linearity:

Consider $u_n$ to be a Gaussian process.

$u_n \sim \mathcal{GP}(0, k_{uu}(x_i, x_j; \theta, l))$

And the linear operator:

$\mathcal{L}_x^c = \cdot + \tau c \mu_{n-1} \frac{d}{dx}\cdot$

so that

$\mathcal{L}_x^c u_n = u_{n-1}$

Problem at hand: Estimate $c$. Using 20 data points, we will be able to estimate c to be 0.9994.

For the sake of simplicity, take $u := u_n$ and $f := u_{n-1}$.

#### Step 1: Simulate data

Take data points at $t = 0$ for $(u_{n-1})$ and $t = \tau$ for $(u_n)$, where $\tau$ is the time step.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
import warnings
import time
```

$x \in [0, 1], \; t \in \{0, \tau \}$

```python
tau = 0.001
def get_simulated_data(tau, n=20):
    x = np.random.rand(n)
    y_u = x/(1+tau)
    y_f = x
    return (x, y_u, y_f)

(x, y_u, y_f) = get_simulated_data(tau)
```

```python
def show_1(x,y_u,y_f):
    f, (ax1, ax2) = plt.subplots(ncols=2, nrows=1, sharey=True, figsize=(10,3))
    f.suptitle("Data for u(x) in red and f(x) in blue")
    ax1.plot(x, y_u, 'ro')
    ax1.set(xlabel= r"x", ylabel= "u(x)")
    ax2.plot(x, y_f, 'bo')
    ax2.set(xlabel= r"x", ylabel= "f(x)");
```

```python
show_1(x,y_u,y_f)
```

#### Step 2: Evaluate kernels

1) $k_{uu}(x_i, x_j; \theta, l) = \theta exp(-\frac{1}{2l}(x_i-x_j)^2)$, where $\theta, l > 0$

```python
x_i, x_j, theta, l, c = sp.symbols('x_i x_j theta l c')
mu = np.mean(y_f)
kuu_sym = theta*sp.exp(-1/(2*l)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta, l), kuu_sym, "numpy")
def kuu(x, theta, l):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta, l)
    return k
```

2) $k_{ff}(x_i,x_j;\theta, l, c)
= \mathcal{L}_{x_i}^c \mathcal{L}_{x_j}^c k_{uu}(x_i, x_j; \theta, l) \\
= k_{uu} + \tau c \mu_{n-1} \frac{d}{dx_i}k_{uu} + \tau c \mu_{n-1} \frac{d}{dx_j}k_{uu} + \tau^2 c^2 \mu_{n-1}^2 \frac{d^2}{d x_i x_j} k_{uu}$

```python
kff_sym = kuu_sym \
        + tau*c*mu*(sp.diff(kuu_sym, x_i) + sp.diff(kuu_sym, x_j)) \
        + tau**2*c**2*mu**2*sp.diff(kuu_sym, x_i, x_j)
kff_fn = sp.lambdify((x_i, x_j, theta, l, c), kff_sym, "numpy")
def kff(x, theta, l, c):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, l, c)
    return k
```

3) $k_{fu}(x_i,x_j; \theta, l, c)
= \mathcal{L}_{x_i}^c k_{uu}(x_i, x_j; \theta, l) \\
= k_{uu} + \tau \mu_{n-1} c \frac{d}{dx_i}k_{uu}$

```python
kfu_sym = kuu_sym + tau*mu*c*sp.diff(kuu_sym, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, l, c), kfu_sym, "numpy")
def kfu(x, theta, l, c):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], theta, l, c)
    return k
```

4) $k_{uf}(x_i,x_j;\theta, l, c)$ is given by the transpose of $k_{fu}(x_i,x_j;\theta, l, c)$

```python
def kuf(x, theta, l, c):
    return kfu(x,theta, l, c).T
```

#### Steps 3 and 4: Compute NLML and optimize the hyperparameters

```python
def nlml(params, x, y1, y2, s):
    theta_exp = np.exp(params[0])
    l_exp = np.exp(params[1])
    K = np.block([
        [kuu(x, theta_exp, l_exp) + s*np.identity(x.size), kuf(x, theta_exp, l_exp, params[2])],
        [kfu(x, theta_exp, l_exp, params[2]), kff(x, theta_exp, l_exp, params[2]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
m = minimize(nlml, np.random.rand(3), args=(x, y_u, y_f, 1e-7), method=\
             "Nelder-Mead", options = {'maxiter' : 1000})
```

```python
m.x[2]  # This is our inferred value for c
```

    0.9994300669651587

```python
m
```

     final_simplex: (array([[6.72217068, 7.81870533, 0.99943007],
           [6.72217757, 7.81870686, 0.99942905],
           [6.72224191, 7.81878793, 0.9994286 ],
           [6.72221172, 7.81875754, 0.99942915]]), array([-293.59378882, -293.59377864, -293.59377797, -293.59377179]))
               fun: -293.59378881947436
           message: 'Optimization terminated successfully.'
              nfev: 225
               nit: 116
            status: 0
           success: True
                 x: array([6.72217068, 7.81870533, 0.99943007])

#### Step 5: Analysis w.r.t. the number of data points (up to 25):

In this section we want to analyze the error of our algorithm using two different ways and plot its time complexity.

```python
res = np.zeros((5,25))
timing = np.zeros((5,25))
warnings.filterwarnings("ignore")
for k in range(5):
    for n in range(25):
        start_time = time.time()
        (x, y_u, y_f) = get_simulated_data(tau, n)
        m = minimize(nlml, np.random.rand(3), args=(x, y_u, y_f, 1e-7), method="Nelder-Mead")
        res[k][n] = m.x[2]
        timing[k][n] = time.time() - start_time
```

**1. Plotting the error in our estimate for c:**

The error is given by $| c_{estimate} - c_{true} |$.

```python
lin = np.linspace(5, res.shape[1] - 1, res.shape[1] - 5)
ones = np.ones(res.shape[1])

def show_2(lin, ones, res):
plt.figure(figsize=(5,3))
to_del = np.linspace(0, 4, 5)
for i in range(res.shape[0]):
    a = np.delete(np.abs(res[i,:] - ones), to_del)
    plt.plot(lin, a)
    plt.ylabel('Error')
    plt.xlabel('Number of data points')
    plt.show()
```

We have altogether ran the algorithm five times, each time increasing the number of data points:

```python
# The rows of res consist of the predicted values for c with an increasing
# number of samples. The number of rows correspond to the amount of reruns
# and thereby resulting plots.

%matplotlib inline
import matplotlib.ticker as ticker

# Setting global figsize to (5,3)
import matplotlib
matplotlib.rcParams["figure.figsize"] = (5,3)

lin = np.linspace(5, res.shape[1] - 1, res.shape[1] - 5)
ones = np.ones(res.shape[1])

def show_2(lin, ones, res):

    to_del = np.linspace(0, 4, 5)

    # Forcing integer labels for the x-axis.
    ax = plt.figure().gca()
    ax.xaxis.set_major_locator(ticker.MaxNLocator(integer=True))

    for i in range(res.shape[0]):
        a = np.delete(np.abs(res[i,:] - ones), to_del)
        plt.plot(lin, a)
        plt.suptitle('Error in our estimate of c')
        plt.ylabel('Error')
        plt.xlabel('Number of data points')

    est = np.repeat(0.5, len(lin))
    plt.plot(lin, est, color='blue', linestyle='dashed')
    plt.show();
```

```python
show_2(lin, ones, res)
```

We see that for n sufficiently large (in this case $n \geq 5$), we can assume the error to be bounded by 0.5. <br>
It seems to be difficult to (even roughly) describe the limiting behavior of the error w.r.t. the number of data points.

**2. Plotting the error between the solution and the approximative solution:**

```python
lin = np.linspace(5, res.shape[1] - 1, res.shape[1] - 5)
ones = np.ones(res.shape[1])
to_del = np.linspace(0, 4, 5)

f, (ax1, ax2) = plt.subplots(ncols=2, nrows=2, figsize=(13,7))

a0 = np.delete(np.abs(np.divide(ones, res[0,:]) - ones), to_del)
ax1[0].plot(lin, a0)
ax1[0].set(xlabel= r"Number of data points", ylabel= "Error")

a1 = np.delete(np.abs(np.divide(ones, res[1,:]) - ones), to_del)
ax1[1].plot(lin, a1)
ax1[1].set(xlabel= r"Number of data points", ylabel= "Error")

a2 = np.delete(np.abs(np.divide(ones, res[2,:]) - ones), to_del)
ax2[0].plot(lin, a2)
ax2[0].set(xlabel= r"Number of data points", ylabel= "Error")

a3 = np.delete(np.abs(np.divide(ones, res[3,:]) - ones), to_del)
ax2[1].plot(lin, a3)
ax2[1].set(xlabel= r"Number of data points", ylabel= "Error");
```

```python
lin = np.linspace(5, res.shape[1] - 1, res.shape[1] - 5)
ones = np.ones(res.shape[1])

def show_3(lin, ones, res):
    plt.figure(figsize=(5,3))

    to_del = np.linspace(0, 4, 5)

    ax = plt.figure().gca()
    ax.xaxis.set_major_locator(ticker.MaxNLocator(integer=True))

    plt.suptitle('Relative error in our estimated solution')
    plt.ylabel('Error')
    plt.xlabel('Number of data points')

    for i in range(res.shape[0]):
        a_i = np.delete(np.abs(np.divide(ones, res[i,:]) - ones), to_del)
        plt.plot(lin, a_i)

```

```python
show_2(lin, ones, res)
```

**3. Plotting the execution time:**

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

for i in range(timing.shape[0]):
    plt.plot(lin, timing[i,:])
    plt.ylabel('Execution time in seconds')
    plt.xlabel('Number of data points')
    plt.show()
```

All in one plot (the blue dashed line follows $f(x) = 0.01x^{2.2}$):

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

def show_3(lin, timing):
    plt.figure(figsize=(5,3))

    for i in range(timing.shape[0]):
        plt.plot(lin, timing[i,:])
        plt.suptitle('Execution time of our algorithm')
        plt.ylabel('Seconds')
        plt.xlabel('Number of data points')

    est = 0.01*(lin)**(2.2)
    plt.plot(lin, est, color='blue', linestyle='dashed')
    plt.show()
```

```python
show_3(lin, timing)
```

The time complexity seems to be around $\mathcal{O}(n^{2.2})$ (blue-dashed line).