 Mathematics in Machine Learning | Politecnico di Torino

Machine learning algorithms for predicting
 default of Credit Card clients

Prof. Francesco Vaccarino

Prof. Mauro Gasparini

Student: Roberto Franceschi (S276243)

---

### Contents:
1. [Introduction](#intro)
2. [Exploratory Data Analysis](#eda)
    - [Feature Analysis](#feat)
    - [Data cleaning](#data_cleaning)
    - [Correlation](#corr)
    - [Box plot](#boxplot)
3. [Data Preprocessing](#data_preproc)
    - [One-hot encoding for categorical variables](#one_hot)
    - [Feature Scaling](#feat_scaling)
    - [Train-test split](#train-test)
    - [Dimensionality Reduction](#dim_red)
        - [Feature selection](#feat_sel)
        - [Principal Component Analysis](#pca)
    - [Oversampling and Undersampling](#over_under)
4. [Classification algorithms](#classification)
    - [Evaluation Methods](#eval_methods)
    - [Evaluation Metrics](#eval_metrics)
    - [Models](#models)
        - [Logistic Regression](#log_reg)
        - [Random Forests](#random_forest)
        - [Support Vector Machines](#svm)
5. [Results](#results)
6. [Conclusions](#conclusions)

---

## 1. Introduction

The aim of this study is to exploit some supervised machine learning algorithms to identify the key drivers that determine the likelihood of credit card default, underlining the mathematical aspects behind the methods used. Credit card default happens when you have become severely delinquent on your credit card payments. In order to increase market share, card-issuing banks in Taiwan over-issued cash and credit cards to unqualified applicants. At the same time, most cardholders, irrespective of their repayment ability, the overused credit card for consumption and accumulated heavy credit and debts

The goal is to build an automated model for both identifying the key factors, and predicting a credit card default based on the information about the client and historical transactions. The general concepts of the supervised machine learning paradigm are later reported, together with a detailed explanation of all techniques and algorithms used to build the models. In particular, Logistic Regression, Random Forest and Support Vector Machines algorithms have been applied.

### 1.1 Environment
The analysis has been fully conducted with Python language, exploiting several machine learning and statistical frameworks available such as `scikit-learn`, `numpy`, `pandas`, `imblearn` together with other data visualization libraries (`matplotlib` and `seaborn`).

Some code snippets have been reported for a better understanding of each step. Moreover, the full code implementation of this analysis is publicly available at [github.com/robertofranceschi/default-credit-card-prediction](https://github.com/robertofranceschi/default-credit-card-prediction).

## 2. Exploratory Data Analysis (EDA)

### 2.1 Feature Analysis

The dataset used in this study is the *Default of credit card clients* from the UCI machine learning repository, available at the following [link](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients).
 It consists of **30000 observations** that represent distinct credit card clients. Each observation has **24 attributes** that contain information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card clients in Taiwan from April 2005 to September 2005.

The first group of variables contains information about the **client personal information**:

1. `ID`: ID of each client, categorical variable
2. `LIMIT_BAL`: Amount of given credit in NT dollars (includes individual and family/supplementary credit)
3. `SEX`: Gender, categorical variable (1=male, 2=female)
4. `EDUCATION`: level of education, categorical variable (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)
5. `MARRIAGE`: Marital status, categorical variable (1=married, 2=single, 3=others)
6. `AGE`: Age in years, numerical variable

The following attributes contains information about the **delay of the past payment** referred to a specific month:

7. `PAY_0`: Repayment status in September 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, … 8=payment delay for eight months, 9=payment delay for nine months and above)
8. `PAY_2`: Repayment status in August 2005 (same scale as before)
9. `PAY_3`: Repayment status in July 2005 (same scale as before)
10. `PAY_4`: Repayment status in June 2005 (same scale as before)
11. `PAY_5`: Repayment status in May 2005 (same scale as before)
12. `PAY_6`: Repayment status in April 2005 (same scale as before)

Other variables instead consider the information related to the **amount of bill statement** (i.e. a monthly report that credit card companies issue to credit card holders in a specific month):

13. `BILL_AMT1`: Amount of bill statement in September, 2005 (NT dollar)
14. `BILL_AMT2`: Amount of bill statement in August, 2005 (NT dollar)
15. `BILL_AMT3`: Amount of bill statement in July, 2005 (NT dollar)
16. `BILL_AMT4`: Amount of bill statement in June, 2005 (NT dollar)
17. `BILL_AMT5`: Amount of bill statement in May, 2005 (NT dollar)
18. `BILL_AMT6`: Amount of bill statement in April, 2005 (NT dollar)

The following variables instead consider the **amount of previous payment** in a specific month:

19. `PAY_AMT1`: Amount of previous payment in September, 2005 (NT dollar)
20. `PAY_AMT2`: Amount of previous payment in August, 2005 (NT dollar)
21. `PAY_AMT3`: Amount of previous payment in July, 2005 (NT dollar)
22. `PAY_AMT4`: Amount of previous payment in June, 2005 (NT dollar)
23. `PAY_AMT5`: Amount of previous payment in May, 2005 (NT dollar)
24. `PAY_AMT6`: Amount of previous payment in April, 2005 (NT dollar)

The last variable is the one to be predicted:

25. `default.payment.next.month`: indicate whether the credit card holders are defaulters or non-defaulters (1=yes, 0=no)

The main aim of the data is to discriminate clients that are predicted to credit card default the next month, according to the `default.payment.next.month` column which is set to “0” for non-defaulters and “1” for defaulters. Thus, it is a ***binary classification problem*** on a relatively *unbalanced dataset*, as shown in the following figure.

 Class distribution

As can be seen, a number of 6.636 out of 30.000 (or 22.1%) of clients will default next month (i.e. has category 1).

In order to get a first look at how data is presented, some observations of the dataset are shown in the table below. No missing feature is recorded for any of the 30.000 samples.
 The data appears already encoded and cleaned. However, by better inspecting some samples and statistics there are some observations to point out.

```python
# read input
data = pd.read_csv(INPUT_PATH)
data.head()
```

      ID
      LIMIT_BAL
      SEX
      EDUCATION
      MARRIAGE
      AGE
      PAY_0
      PAY_2
      PAY_3
      PAY_4
      ...
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      default.payment.next.month

      0
      1
      20000.0
      2
      2
      1
      24
      2
      2
      -1
      -1
      ...
      0.0
      0.0
      0.0
      0.0
      689.0
      0.0
      0.0
      0.0
      0.0
      1

      1
      2
      120000.0
      2
      2
      2
      26
      -1
      2
      0
      0
      ...
      3272.0
      3455.0
      3261.0
      0.0
      1000.0
      1000.0
      1000.0
      0.0
      2000.0
      1

      2
      3
      90000.0
      2
      2
      2
      34
      0
      0
      0
      0
      ...
      14331.0
      14948.0
      15549.0
      1518.0
      1500.0
      1000.0
      1000.0
      1000.0
      5000.0
      0

      3
      4
      50000.0
      2
      2
      1
      37
      0
      0
      0
      0
      ...
      28314.0
      28959.0
      29547.0
      2000.0
      2019.0
      1200.0
      1100.0
      1069.0
      1000.0
      0

      4
      5
      50000.0
      1
      2
      1
      57
      -1
      0
      -1
      0
      ...
      20940.0
      19146.0
      19131.0
      2000.0
      36681.0
      10000.0
      9000.0
      689.0
      679.0
      0

5 rows × 25 columns

```python
# Summary Statistics
data.describe()
```

      ID
      LIMIT_BAL
      SEX
      EDUCATION
      MARRIAGE
      AGE
      PAY_0
      PAY_2
      PAY_3
      PAY_4
      ...
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      default.payment.next.month

      count
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      ...
      30000.00
      30000.00
      30000.00
      30000.00
      3.00e+04
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00

      mean
      15000.50
      167484.32
      1.60
      1.85
      1.55
      35.49
      -0.02
      -0.13
      -0.17
      -0.22
      ...
      43262.95
      40311.40
      38871.76
      5663.58
      5.92e+03
      5225.68
      4826.08
      4799.39
      5215.50
      0.22

      std
      8660.40
      129747.66
      0.49
      0.79
      0.52
      9.22
      1.12
      1.20
      1.20
      1.17
      ...
      64332.86
      60797.16
      59554.11
      16563.28
      2.30e+04
      17606.96
      15666.16
      15278.31
      17777.47
      0.42

      min
      1.00
      10000.00
      1.00
      0.00
      0.00
      21.00
      -2.00
      -2.00
      -2.00
      -2.00
      ...
      -170000.00
      -81334.00
      -339603.00
      0.00
      0.00e+00
      0.00
      0.00
      0.00
      0.00
      0.00

      25%
      7500.75
      50000.00
      1.00
      1.00
      1.00
      28.00
      -1.00
      -1.00
      -1.00
      -1.00
      ...
      2326.75
      1763.00
      1256.00
      1000.00
      8.33e+02
      390.00
      296.00
      252.50
      117.75
      0.00

      50%
      15000.50
      140000.00
      2.00
      2.00
      2.00
      34.00
      0.00
      0.00
      0.00
      0.00
      ...
      19052.00
      18104.50
      17071.00
      2100.00
      2.01e+03
      1800.00
      1500.00
      1500.00
      1500.00
      0.00

      75%
      22500.25
      240000.00
      2.00
      2.00
      2.00
      41.00
      0.00
      0.00
      0.00
      0.00
      ...
      54506.00
      50190.50
      49198.25
      5006.00
      5.00e+03
      4505.00
      4013.25
      4031.50
      4000.00
      0.00

      max
      30000.00
      1000000.00
      2.00
      6.00
      3.00
      79.00
      8.00
      8.00
      8.00
      8.00
      ...
      891586.00
      927171.00
      961664.00
      873552.00
      1.68e+06
      896040.00
      621000.00
      426529.00
      528666.00
      1.00

8 rows × 25 columns

For example, the following criticities can be observed:
- the column name `PAY_0` should be renamed in `PAY_1`;
- about `EDUCATION` and `MARRIAGE` there are some undocumented categories;
- the columns `PAY_n` have a minimum in -2 (undocumented) and a maximum equal to 8. So, it is likely that these attributes need to be rescaled.

### 2.2 Data Cleaning

The presence of errors in the dataset can be addressed in two ways:
1. deleting of the rows associated with an error (paying attention to the number of lines with respect to the total number of observations);
2. with a correction of the wrong attribute.

In our case the first method is applied to the categorical variables, by safely removing the unknown categories (since the total number of anomalous values is 399):
- Regarding the attribute `EDUCATION` there are three categories not listed in the description of the dataset provided by the UCI website that corresponds to 0, 5, and 6.
- While for `MARRIAGE` we can notice the presence of category 0 that does not correspond to any categories previously described.

while the second approach is applied to the `PAY_n` features, correcting them by adding 1 to each attribute and grouping together the resulting values of '0' and '-1' (since 0 does not correspond to any value previously described).

```python
data['MARRIAGE'].value_counts()
```

    2    15964
    1    13659
    3      323
    0       54
    Name: MARRIAGE, dtype: int64

```python
data['EDUCATION'].value_counts()
```

    2    14030
    1    10585
    3     4917
    5      280
    4      123
    6       51
    0       14
    Name: EDUCATION, dtype: int64

```python
# Payment delay description
data[['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']].describe()
```

      PAY_1
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6

      count
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00

      mean
      -0.01
      -0.13
      -0.16
      -0.22
      -0.26
      -0.29

      std
      1.12
      1.20
      1.20
      1.17
      1.14
      1.15

      min
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00

      25%
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00

      50%
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      75%
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      max
      8.00
      8.00
      8.00
      8.00
      8.00
      8.00

```python
# Amount of given credit in NT dollars
data.LIMIT_BAL.describe()
```

    count      29601.00
    mean      167550.54
    std       129944.02
    min        10000.00
    25%        50000.00
    50%       140000.00
    75%       240000.00
    max      1000000.00
    Name: LIMIT_BAL, dtype: float64

The very high value of standard deviation has been further investigated. As can be seen, most of defaults are for credit limits 0-100,000 (and density for this interval is larger for defaults than for non-defaults).

 Density plot of amount of given credit (LIMIT_BAL)

### 2.3 Correlation between features
Another relevant point which could affect the classification performances is the correlation among features: the presence of strongly correlated features may lead to a decline in the performances of some classification algorithms which assume that the predictors are all independent. Another benefit from spotting correlation among features is that the same information may be encoded with less attributes, and this could lead to simpler final models. Indeed, some methods suffer from high dimensional datasets (especially distance based ones with a few number of samples), so reducing the dimensions of the feature vectors can make it more trustable and stable, if the discarded dimensions don’t affect significantly the total original information.

The *Pearson’s correlation coefficient* is a statistic that measures the linear correlation between a pair of random variables. In particular, given the random vector $(X, Y)$ :

\begin{equation}
\rho _{X,Y} :=\ \frac{Cov( X,\ Y)}{\sigma _{X} \sigma _{Y}} \ \in \ [ -1,\ 1] \ \subset \ \mathbb{R}
\end{equation}

where $Cov( X,\ Y)$ is the Covariance and $\sigma _{X}$ and $\sigma _{Y}$ are respectively the standard deviations of $X$ and $Y$.
 $\rho _{X,Y}$ has a value bounded between $-1$ and $+1$.

The higher in absolute value, the stronger the linear correlation is, meaning that on average for each large (w.r.t. the mean of $X$) value  observed, a corresponding large value (w.r.t. the mean of $Y$) is also observed.

Note that the Pearson’s correlation coefficient is only able to capture linear trends, hence it might lead to a value of 0 for strongly non-linearly correlated variables (e.g. quadratic trend).

In practice, the underlying probability distribution is not known but only a sample of realizations of the random vector $(X,Y)$ is available, so the statistic is computed by inferring the unbiased covariance and standard deviations from the dataset for each pair of attributes.

The following representation shows the correlation matrix between features: since the way the matrix is constructed make it
symmetric and because of the large number of features, only the heatmap of the lower diagonal matrix (the diagonal itself is
excluded) is reported for a better visualization.

 Correlation matrix by means of the Pearson’s coefficient for all feature pairs.

As shown in the correlation matrix above, some features show high correlations with each other. In particular, there exist an high positive correlation among the `BILL_AMTn` features, for example:

- BILL_AMT1 and BILL_AMT2 have $\rho = 0.95$
- BILL_AMT2 and BILL_AMT3 have $\rho = 0.93$
- BILL_AMT4 and BILL_AMT5 have $\rho = 0.94$

To give a further proof of the linear dipendence of these variables, their interactions plots are shown:

 Distribution of correlated features (scatter plot)

The charts confirm what expected, the features in the same graph shows a linear trend as the Pearson coefficient suggested,
indicating they encode pretty similar information.

For completness, also charts of non-strongly correlated features are reported. In particular:
- AGE and LIMIT_BAL $\rho = 0.14$
- AGE and BILL_AMT1 $\rho = 0.055$
- PAY_AMT1 and BILL_AMT1 $\rho = 0.099$

The following charts are obtained:

 Distribution of uncorrelated features

In order to remove correlated features some dimensionality reduction techinques will be performed during the Data Preprocessing phase (point [3.](#data_preproc)).

### 2.4 Boxplot

Finally, the boxplots for each of the numerical features are plotted in order to have a further look at the distribution and to eventually spot outliers (i.e. “bad” data that either has been misscomputed or is strongly exceptional wrt expected range).

A boxplot represents a 5-number summary of data:

- 1st quartile: the twenty-fifth percentile;
- median (or 2nd quartile): the middle value of a dataset;
- 3rd quartile: the seventy-fifth percentile;
- minimum: the lowest data point excluding any outliers;
- maximum: the largest data point excluding any outliers.

In the boxplot we define the Interquartile range (IQR) as is the distance between the upper and lower quartiles:

\begin{equation}
IQR = Q_3 - Q_1 = q(0.75) - q(0.25)
\end{equation}

 Boxplot description

Furthermore, through them it is possible to detect outliers presence (i.e. the single points), with the following rule: every sample located beyond $Q_1 − 1.5IQR$ and $Q_3 + 1.5IQR$ is considered an outlier.

*Note that for the purpose a normalization was applied on all the attributes to make them have the same scale, otherwise it would have been impossible to compare them.*

### Min-Max scaling for numerical variables

As mentioned above, input variables may have different units so different scales and magnitude; for this reason before drawing a boxplot, a `MinMaxScaler()` is applied in order to scale the features between a range $(0, 1)$. The basic idea behind this rescaling technique is that for every feature, the minimum value of that feature gets transformed into a 0, the maximum value gets transformed into a 1, and every other value gets transformed into a decimal between 0 and 1.

The transformation is given by the following formula:

\begin{equation}
X_{scaled} = \frac{(X - X_{min})}{(X_{max} - X_{min})}
\end{equation}

where $X_{min}$ is the minimum value on the column and $X_{max}$ is the maximum value on the column.

This transformation is applied on numerical features only as the categorical variables has been already transformed into one-hot vectors, that rescale the categorical variable in the range $(0,1)$.

### Standardization

Another possible scaling technique is referred to as **standardization**, which transforms each value $x$ of feature $X$ as follows, independently for each column:

\begin{equation}
x^{\prime} =\frac{x-\mu _{X}}{\sigma _{X}}
\end{equation}

where $\mu_{X}$ is the sample mean and $\sigma_{X}$ is the sample standard deviation of feature $X$. This way, we force each attribute to have a new empirical mean value of 0 and variance 1.

*Note that the statistics (e.g. mean and standard deviation for the Standard Scaler) are computed on the training set only, then both datasets are transformed according to them. This is very important as it avoids leaking any information from the test set into the training process, which would be formally incorrect by definition.*

```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler() # or MinMaxScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
```

 Boxplots of numerical features scaled according min-max normalization (left) and standardization (right)

According to the method previously described from the boxplot we can identifty many outliers. It is possible to change the multiplicative coefficient to decide the threshold to overcome to be considered an outlier, but still dubling the coefficent the possible outliers are more than 4000.

Due to the lack of domain knowledge and the very high number of identifiable outliers, no samples have been discarded as outliers.

## 3. Data Preprocessing

### 3.1 One-hot encoding for categorical variables

An integer encoding may impose some ordinal relationship between categorical variables that does not exist, for this reason a **one-hot encoding** is performed.
Categorical variable such as `SEX`, `MARRIAGE` and `EDUCATION` are turned into one-hot variables in order to remove any orders that in this case have no meaning.

A one hot encoding is a representation of categorical variables as binary vectors. This first requires that the categorical values be mapped to integer values. Then, each integer value is represented as a binary vector that is all zero values except the index of the integer, which is marked with a 1.

```python
# Set 'category' type to categorical attributes
for att in ['SEX', 'EDUCATION', 'MARRIAGE']:
  data[att] = data[att].astype('category')

# one-hot encoding
data=pd.concat([pd.get_dummies(data['SEX'], prefix='SEX'),
                pd.get_dummies(data['EDUCATION'], prefix='EDUCATION'),
                pd.get_dummies(data['MARRIAGE'], prefix='MARRIAGE'),
                data],axis=1)
# drop original columns
data.drop(['EDUCATION'],axis=1, inplace=True)
data.drop(['SEX'],axis=1, inplace=True)
data.drop(['MARRIAGE'],axis=1, inplace=True)
# print samples
data.head()
```

      SEX_1
      SEX_2
      EDUCATION_1
      EDUCATION_2
      EDUCATION_3
      EDUCATION_4
      MARRIAGE_1
      MARRIAGE_2
      MARRIAGE_3
      LIMIT_BAL
      ...
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      Default

      0
      0
      1
      0
      1
      0
      0
      1
      0
      0
      0.01
      ...
      0.16
      0.08
      0.26
      0.00e+00
      4.09e-04
      0.00e+00
      0.00e+00
      0.00e+00
      0.00e+00
      1

      1
      0
      1
      0
      1
      0
      0
      0
      1
      0
      0.11
      ...
      0.16
      0.08
      0.26
      0.00e+00
      5.94e-04
      1.12e-03
      1.61e-03
      0.00e+00
      3.78e-03
      1

      2
      0
      1
      0
      1
      0
      0
      0
      1
      0
      0.08
      ...
      0.17
      0.10
      0.27
      1.74e-03
      8.91e-04
      1.12e-03
      1.61e-03
      2.34e-03
      9.46e-03
      0

      3
      0
      1
      0
      1
      0
      0
      1
      0
      0
      0.04
      ...
      0.19
      0.11
      0.28
      2.29e-03
      1.20e-03
      1.34e-03
      1.77e-03
      2.51e-03
      1.89e-03
      0

      4
      1
      0
      0
      1
      0
      0
      1
      0
      0
      0.04
      ...
      0.18
      0.10
      0.28
      2.29e-03
      2.18e-02
      1.12e-02
      1.45e-02
      1.62e-03
      1.28e-03
      0

5 rows × 30 columns

### 3.2 Feature Scaling

As previously explained, a `MinMaxScaler()` has been already applied to all the numerical features.

### 3.3 Train-test split
The dataset is divided in training set and test set, with the proportion 3:1. Given the imbalance fashion in the original dataset a **stratified sampling** strategy has been applied during the split, so that the proportion of defaulters/non-defaulters clients in each of the final sets would approximately be the same as the initial one. This approach is desirable every time we have a highly *unbalanced dataset* as in our case.

The final split obtained is distributed as follows:

|              | Non-defaulter | Defaulters |    Total   |
|--------------|:-------------:|:----------:|:----------:|
| **Training set** |    17246         | 4954 | 22200 |
| **Test set**   |    5750           | 1651  | 7401 |

*Note that the split has been randomly generated but kept the same for all the following methods, in order to to make evaluations statistically meaningful and provide consistent results among the different classifiers.*
```python
random_state = 42
```

### 3.4 Dimensionality reduction

As already mentioned above, generally speaking it is not always a good idea to propagate in the algorithms all the samples attributes. The main reasons for which the dimensionality reduction is required are the following:
- high dimensional data may negatively affect the algorithms perfomances (i.e. high computational costs);
- high dimensionality might lead to poor generalization;
- can be used for interpretability of the data (e.g. for illustraton purposes).

Moreover different machine learning algorithms suffer from the ***curse of dimensionality*** problem: if there are more features than observations the risk of massively overfitting the model becomes really high. Then, too many dimensions causes every observation in the dataset to appear equidistant from all the other, and this is effectively a problem when used distance-based algorithms (such as K-Nearest Neighbors), because if the distances are all approximately equal, then the observation appear equally alike (as well as equally different), making the algorithm perfomances meaningless.

### Feature Selection
On top of feature transformation, the data preprocessing phase often includes a feature selection step, where the attributes from the given dataset are carefully analysed and finally selected to feed the machine learning algorithm.
 As previously seen in the correlation matrix and with scatter plots, some features in this dataset are strongly linearly correlated, so it is not meaningful to keep all of them because they may contain redundant information. In this way, features with a Pearson coefficient $\rho \geqslant 0.92$ with other predictors should be discarded.

In particular, at this step the following features are removed: `BILL_AMT2`, `BILL_AMT3`, `BILL_AMT4`, `BILL_AMT5`, `BILL_AMT6`.

### Principal Component Analysis (PCA)
Another way to obtain a lower dimensional dataset when dealing with multicollinearity among features, is by applying a Principal Component Analysis (PCA) to our data.

PCA is an **unsupervised learning** technique that performs a linear transformation (by means of an orthogonal matrix) of the original space such that the new basis found has dimensions (features) that are sorted from largest to smallest possible variance.

The intuition behind this approach lies in the fact that if the direction of maximum variance in the original space is not directly captured by the features in the dataset, then that direction might be used to construct a new feature that has a larger variance, hence probably encoding more information. Therefore, the following minimization problem is performed for finding the direction of maximum variance:

\begin{equation}
X\in \mathbb{R}^{n\times d} ,\ dataset\ centered\ in\ zero\\
\Sigma :=\frac{X^{T} X}{n-1} \ ,\ sample\ covariance\ matrix\\
\\
find\ \ \overrightarrow{z_{1}} :=a_{1}\overrightarrow{e_{1}} \ +...+a_{d}\overrightarrow{e_{d}}\\
\\ \\
\\ s.t.:\\
\\
\ \overrightarrow{z_{1}} =\underset{\vec{z}_{1}}{argmax} \ \ \vec{z}_{1}^{T} \ \Sigma \ \ \vec{z}_{1} ,\ subject\ to:\ \Vert \vec{z}_{1}\Vert =1
\end{equation}

Recursively, all other dimensions are then computed in the same way, additionally forcing them to be orthogonal to the previous dimensions found.
The new features $z_{i}$ are denoted as **principal components** (PCs).

The above optimization problem can be easily proved to be equivalent to computing the eigendecomposition of $\Sigma$ and selecting its eigenvectors as the principal components, which is the way PCA is executed in practice.
In particular, being $\Sigma$ a symmetric positive semidefinite matrix, it is always possible to diagonalize it by means of an orthogonal matrix $P$ (eigenvectors of $\Sigma$), and obtain the resulting similar matrix $\Lambda$:

\begin{equation}
\Lambda =\ \begin{pmatrix}
							\lambda _{1} & 0 & \cdots & 0\\
							0 & \lambda _{2} & \ddots & \vdots \\
							\vdots & \ddots & \ddots & 0\\
							0 & \cdots & 0 & \lambda _{d}
							\end{pmatrix} \ ,\ \lambda _{1} \geqslant \lambda _{2} \geqslant ...\geqslant \lambda _{d} \geqslant 0,\ \lambda _{i} \in \mathbb{R}^{+}
\end{equation}

where $\lambda_{i}$ are the eigenvalues of $\Sigma$, or equivalently the variances of the new features found. Note that, since $\Sigma$ and $\Lambda$ are similar matrices, they have the same trace, meaning that the initial total variance among the features of the dataset is not changing but it is just getting redistributed on new axes (which is expected as all we are doing is just a rigid rotation of the space).
In particular, we can compute the variance explained by each new principal component with respect to the total variance of the dataset.

Finally note that $\Lambda$ is now the covariance matrix in the new basis found, and since it’s diagonal all new features result to be linearly uncorrelated. We can then select only a subset of the first principal components in order to reduce the initial dimensionality of the dataset with minimal information loss.

In this study, PCA has been performed on the Credit card dataset to deal with the multicollinearity problem and reduce the number of dimensions. The following figure shows how the variance has been redistributed on the new features extracted.

 Explained variance ratio of each principle component, together with the
cumulative variance explained as more dimensions are considered.

The graph above is relative to the **proportion of explained variance**: the green line represents the proportion of variance explained by each principal component. While the orange line instead is the cumulative amount of proportion of variance explained (i.e. the sum of the single variance explained by the principal components on the left).

This graph is useful to take decision regarding the number of components to keep. The following table report some interesting values for the number of components.

| Number of PCs | Cumulative Variance Explained |
|:-------------:|:-------------------------------:|
|       4       |               89.7 %             |
|       6       |               95.3 %             |
| 8             | 97.1 %                           |
| 12            | 99.0 %                          |

For example, the first 12 PCs are able to capture almost the whole variance (99%) of the data points and for this reason we decided to keep these number.

*Note that PCA is applied based only on the training data in order to avoid any leaking the information of test data.*

```python
from sklearn.decomposition import PCA

pca = PCA(n_components=len(X_train.columns))
pca.fit(X_train) # not on the whole dataset

```

```python
n_pc = 12
pca = PCA(n_components=n_pc)
pca.fit(X_train_val)
#   X_train_val.index = pd.RangeIndex(start=0, stop=len(X_train_val), step=1)
X_6d_train_val = pd.DataFrame(pca.transform(X_train_val))
X_6d_test = pd.DataFrame(pca.transform(X_test))
X_6d_train_val.columns = ['PC' + str(i) for i in range(1, n_pc+1) ]
X_6d_test.columns = ['PC' + str(i) for i in range(1, n_pc+1) ]
X_6d_train_val.head()
```

      PC1
      PC2
      PC3
      PC4
      PC5
      PC6
      PC7
      PC8
      PC9
      PC10
      PC11
      PC12

      0
      1.15
      0.53
      0.42
      -0.20
      0.05
      -0.06
      -0.03
      8.00e-03
      -1.02e-01
      -0.12
      0.13
      -9.53e-03

      1
      0.17
      -0.67
      -0.82
      -0.04
      0.04
      0.37
      -0.10
      4.24e-02
      -4.01e-03
      -0.07
      -0.01
      -8.76e-03

      2
      -0.87
      -0.29
      0.56
      0.85
      -0.28
      -0.02
      -0.27
      1.47e-01
      2.03e-01
      -0.08
      -0.11
      -2.49e-02

      3
      -0.52
      -0.38
      0.93
      -0.31
      0.71
      -0.10
      0.09
      -7.04e-02
      2.04e-02
      -0.09
      -0.05
      3.32e-03

      4
      -1.07
      -0.25
      -0.30
      -0.23
      0.24
      0.39
      -0.03
      -9.61e-03
      3.43e-02
      -0.20
      0.11
      -1.48e-04

### 3.5 Oversampling and undersampling

When dealing with significantly **unbalanced dataset** in the target label, it becomes harder for most machine learning algorithms to efficiently learn all classes. The training process might be indeed biased towards a certain class if the dataset distribution is poorly balanced.

In the specific case of the credit card clients, only about 22.1% of the data are labelled as defaulters (y=1).

|              | Number of rows | Percentage |
|--------------|:-------------:|:----------:|
| Non-defaulters (class=0) |    17246         | 77.68 % |
| Defaulters (class=1) |    4954         |  22.32 % |

While the obvious and most desirable solution would be to collect more real data, *oversampling* and *undersampling* are techniques that may still come in handy in these situations.
 For both techniques there is a naïve approach that is the ***random oversampling (undersampling)*** where training data is incremented (decremented) with multiple copies of the samples, until the same proportion is obtained on the minority (majority) classes.

###  Oversampling: Synthetic Minority Over-sampling Technique (SMOTE)
With SMOTE algorithm the minority class is augmented artificially, by constructing new synthetic samples randomly positioned in between one point and its k-neighbors. In other words, given a limited set of data points that belong to the class that we wish to augment, we trace high-dimensional lines connecting the data points and we draw new samples from such lines.

 Example of SMOTE application on a small dataset

Finally, note that a synthetic sample should never be used as a test sample, since it has technically never appeared in the real world and hence not formally belonging to the target distribution. Also, the stratified test set is believed to be the correct representation of the real world, and augmenting a part of it would lead to a misleading evaluation.

```python
from imblearn.over_sampling import SMOTE

clf = Classifier(**best_config)
#oversampling
oversample = SMOTE()
X_train, y_train = oversample.fit_resample(X_train, y_train)

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

accuracy = accuracy_score(y_test, y_pred)
```

### Undersampling: Cluster Centroids
Cluster Centroids make use of K-means algorithm to perform undersampling. After finding the cluster centroids on the majority class points, the algorithm selects the instances belonging to the cluster (labelled with the majority class), which are furthest from the cluster centroid in feature space. These data points are considered as the most unimportant instance. On the contrary, the instance belonging to the majority class, that is nearest to the cluster centroid in feature space, is considered to be the most important instance. In this way, instances belonging to the majority class are removed on the basis of their importance.

 Example of Cluster Centroids application on a trivial dataset

In particular, given two class which has respectively $N$ and $M$ data points with $N<M$, the algoritm trains a K-Means on points labelled with the majority class label, with $k = N$, the cardinality of the set of minority data points.

Then for each cluster, it replace the data points of the majority class with a new point whose coordinates correspond to the cluster centroid's coordinates. So we undersample majority class by forming clusters and replacing it with cluster centroids.

## 4. Classification algorithms

In this section a description of the algorithms used is presented. For each of them, different hyperparameters are tuned in order to find the ones performing best, and, once found, the model trained with them is used to predict class labels on test data combining different data preprocessing techniques presented above in order to spot some performances differences.
The best configuration is selected by comparing the different metrics, principally based on the *f1-score* because accuracy on unbalance dataset may return an high values even if the minority class is not correcly classified.

Each algorithm is trained on dataset with different preprocessing techniques combined:
- Principal Component Analysis (PCA)
- PCA + SMOTE (oversampling)
- PCA + Cluster Centroids (undersampling)

The algorithms considered are:
- Logistic Regression
- Random forest (and Decision Trees)
- SVM (both Linear and RBF)

### 4.1 Evaluation methods
Even though learning theory guarantees to reach a probably approximately correct model, in practice it is often needed to perform an actual evaluation of the model to check how well it’s performing. This is generally done by splitting the original dataset into two parts, a training set and a test set, so that the model can be trained on the training samples only and be evaluated on previously unseen data. This technique is referred to as ***Holdout***.

Instead, ***Cross Validation*** is a resampling method that allows to repeteadly drawing samples from a training set, in order to have a better estimate of the evaluation score used.

With the cross-validation technique we aim to verify if the model is able predict labels for data points that it hasn't seen so far.
The complete dataset is divided into $k$ subsets (folds):
- $k-1$ folds will be used to train the model, all togheter compose the *training set*;
- one fold composes the *validation set*, on which we evaluate the performance of the model.

This operation is repeated $k$ times in order to reduce the variability, and at every round the validation subset changes. At the end, the $k$ estimates of model's predictive performance are averaged.

 Example of Cross Validation (k=8) where the red and blue square is used respectively as training and validation set

The special case where $k$ is equal to $m$, the number of examples, is called *leave-one-out*. This procedures gives a very good estimate of the true error but, on the other side, it computationally expensive.

A combination of holdout and cross validation can be used when also dealing with a validation set (*k-fold cross validation*). So after having divided the dataset in a stratify way into training and test set, the training part is again splitted into training and validation set (even in this case with `stratify=True`).

### 4.2 Evaluation metrics
In a classification scenario, the model can be evaluated by computing different metrics. In order to better understand these metrics could be useful to get some fundamentals:

|            | Predicted (1) | Predicted (0) |
|:----------:|:-------------:|:-------------:|
| **Actual (1)** |       TP      |       FN      |
| **Actual (0)** |       FP      |       TN      |

- *True Positive (TP)*: samples for which the prediction is positive and the true class is positive
- *False Positive (FP)*: samples for which the prediction is positive but the true class is negative
- *True Negative (TN)*: samples for which the prediction is negative and the true class is negative
- *False Negative (FN)*: samples for which the prediction is negative but the true class is positive

Some of the most popular metrics are:

- ***Accuracy***: ratio of correct predictions over the total number of data points classified

\begin{equation}
Accuracy = \frac{\#\ correctly\ classified\ samples}{total\ number\ of\ samples\ tested}=\frac{TP+TN}{TP+FP+TN+FN}
\end{equation}

- ***Precision***: measures the fraction of correct classified instances among the ones classified as positive. Precision is an appropriate measure to use when the aim is to minimize *false positives*.

\begin{equation}
Precision(c) = \frac{\#\ samples\ correctly\ assigned\ to\ class\ c}{\#\ of\ samples\ assigned\ to\ class\ c}=\frac{TP}{TP+FP}
\end{equation}

- ***Recall***:  it measures how many of the actual positives a model capture through labelling it as True Positive.It is an appropriate score when the aim is to minimize false negatives

\begin{equation}
Recall(c) = \frac{\#\ samples\ correctly\ assigned\ to\ class\ c}{\#\ of\ samples\ actually\ belonging\ to\ c}=\frac{TP}{TP+FN}
\end{equation}

- ***F1-score***: is the harmonic mean of the precision and recall.

\begin{equation}
F1 score(c) = \frac{2*precision( c) *recall( c)}{precision( c) +recall( c)}
\end{equation}

The last three measures are *class-specific*, and a global metric can be obtained by averaging their values across all classes in the dataset. In particular, class-specific metrics work better when dealing with significantly unbalanced dataset, as they take into account how every class is being learned. Thus, the accuracy measure alone might be misleading in these cases, since may return an high score even if the minority class is not correcly classified.

In this analysis we focus our attention in detecting which customer may be defaults clients, and the positive class captures the attention of the classifier. As already mentioned, in the following section the F1-score will be used to find the best configuration for each model.

## 4.3 Models

### 4.3.1 Logistic Regression
Logistic Regression is a parametric, discriminative binary classification algorithm. The name is given after the fact that the algorithm can be interpreted as part of the *Generalized Linear Model*, where the response variables are distributed according to a Bernoulli distribution. In particular, the model assumes the predictors to be linked to the mean of the response variables ($p_i$) as:

\begin{array}{l}
							observation\ ( x_{i} ,\ y_{i}) \ ,\ y_{i} \ realization\ of\ Y_{i} \sim Bernoulli( p_{i}( x_{i}))\\
							\\
							log\left(\frac{p_{i}}{1-p_{i}}\right) =w^{T} x_{i} \ \Longleftrightarrow \ p_{i} =\frac{1}{1+e^{-w^{T} x_{i}}} \ ,\ for\ all\ i
							\end{array}

 Logits function

The algorithm then tries to find the MLE (or MAP if a regularization term is added) of the mean of the response variables, by acting on *w*, assuming i.i.d. samples. Unlike ordinary least squares the optimization problem does not have a closed-form solution, but it’s instead solved through numerical methods.

\begin{equation}
\underset{w}{\max} \ \prod ^{n}_{i} \ p_{i}( x_{i} |w)^{y_{i}} \ *\ ( 1-p_{i}( x_{i} |w))^{1-y_{i}}
\end{equation}

A ***regularization term*** is often added to prevent the coefficients to reach a too high value which would overfit the training data. The hyperparameter $C$ is applied on the new penalty term and inversely regulates the strength of the regularization in the objective function.

```python
# C values tuned:
params = {'C': [0.0001, 0.001, 0.01, 0.1, 1, 10]}

#Best configuration found C=0.01
#Data prepocessing: PCA+SMOTE
```

 Tuning of hyperparameter C on the training set both with SMOTE (left) and Cluster Centroids (right)
 algorithms. Note that scores reported refers to the positive class.

As a result, the output of a Logistic Regression model is the probability of the input sample to be of class 1, hence a confidence measure is also returned when predictions are performed. On top of this, the coefficients returned by the algorithm may give interpretable insights on what attributes are contributing the most to a higher output value, and viceversa.

 Coefficients found by Logistic Regression. The PC5 feature empirically
gives the highest contribute towards being predicted as defaulters

### 4.3.2 Decision Tree
Decision Trees are the most intuitive and interpretable machine learning model, which predict the target label by learning simple decision rules inferred from the data.
At each step, a Decision Tree picks the feature that best divides the space into different labels, by means of the GINI impurity measure:
\begin{equation}
GINI( t) =\sum ^{C}_{i=1} p_{t}( k)( 1-p_{t}( k)) =1-\sum ^{C}_{i=1} p_{t}( k)^{2}
\end{equation}

where $p_{t}(k)$ is the frequency of class $k$ appearing in node $t$, with $C$ the total number of classes. The lower the measure, the less impure the node is.

A full tree is then constructed on the training data and will be used at classification time for predicting the target label. To avoid overfitting, decision trees are often pruned before reaching their full growth, and predictions are made according to a majority vote on the samples of each leaf.
Note how trees cannot handle missing values, but do not suffer from feature scaling since each attribute is treated independently.

```python
#Best configuration found
#Data prepocessing: PCA+SMOTE
params = {'criterion': 'gini', 'max_depth': 5, 'min_impurity_decrease': 0.0, 'splitter': 'best'}
```

Decision trees are however considered weak learners in the majority of cases, and ensemble techniques such as *bagging* (Random forests) or *boosting* are in practice applied on them in order to reach higher performances and more robust solutions.

### 4.3.3 Random Forest
Random Forests are an ensemble method made out of multiple decision trees. Each individual tree in the random forest makes a class prediction and the class with the most votes is assigned to the sample taken into account. The intuition behind this method is that a large number of relatively **uncorrelated** models (trees) operating as a committee will outperform any of the individual constituent model. Uncorrelation is the key of Random Forests better performances, and this is ensured through two methods:
1. Bagging (boostrap aggregation)  to promote variance between each internal model, each individual tree is trained on a sample of the same size of the training dataset randomly chosen with replacement;
2. In addition, trees are further decorrelated by choosing only a subset of features at each split of each tree (usually $\sqrt{\#features}$).

Overall, each tree is then trained on a sample of the same size of the training dataset chosen with replacement, and their splits consider different features every time. This way, even if the decision trees are somewhat overfitting the data and are unstable (high variance), the overall prediction will be statistically more robust (low variance) because of Central Limit Theorem. The prediction is indeed performed by picking the majority class out of all trees.

Hyperparameter tuning on the number of trees and pruning factors for each tree has been performed. In particular, note how the predictions tend to be better as the ensemble model grows horizontally in size:

 Tuning of hyperparameter n_estimator is reported both for SMOTE (left) and Cluster Centroids (right) algorithms.

- The choice of the imbalancing technique affect the performance of the Random forest. Indeed, the oversampling technique is preferable.
- Also, the hyperparameter that regulates the number of estimators seems to be irrelevant over the performance of the algorithms.

```python
#Best configuration found
#Data prepocessing: PCA+SMOTE
params = {'criterion': 'gini', 'max_features': 'sqrt', 'n_estimators': 100}
```

Finally, tree-based model can be used as an alternative method for feature selection. They indeed provide a feature importance measure by means of how much the gini index has been affected on the various splits with the given feature.
Compared to decision tree, random forests guarantee robustness and are less prone to overfitting.

 Feature importance computed by the random forest on each of the attributes.

### 4.3.4 SVM
Support vector machine (SVM) is a parametric linear classification algorithm that aims at separating two classes through a hyperplane in the data dimension. Once the hyperplane has been set, the prediction rule is simply based on whether the test point lays on one side or the other one of it. If more than one hyperplanes fit the purpose, the one with the higher margin (distance) between support vector (the closest points to the hyperplane, one for each class) is selected.

 Maximum-margin hyperplane and margins for an SVM trained with samples from two classes.
 Samples on the margin are called the support vectors.

More formally, it tries to define $\vec{w} ,b$ s.t. the hyperplane $w^{T}x+b=0$ is the “maximum-margin hyperplane”, i.e. such that the distance between the hyperplane and the nearest point $\overrightarrow{x_{i}}$ from either group is maximized. If we impose that all points must satisfy $|w^{T}x+b|\geqslant1$ relatively to their label, then the optimization problem becomes equivalent to finding the smallest $\|w\|$:

#### Primal Optimization Problem (Hard Margin)

\begin{array}{l}
\underset{w,b}{min} \ \tfrac{1}{2}\Vert w\Vert ^{2} \ \\
\\
s.t.\ y_{i}\left( w^{T} x_{i} +b\right) \geqslant 1,\ \forall i
\end{array}

The decision function on a binary label $\{-1,1\}$ is then applied as $x\mapsto sign\left( w^{T} x+b\right)$. Geometrically, the best hyperplane is completely determined by the points $\overrightarrow{x_{i}}$ the lie right on the boundary of the margin $|w^{T}x_{i}+b|=1$. These points are called ***support vectors***.

This model is referred to as **Hard-margin SVM**, where data is supposed to be linearly separable otherwise no feasible solution would be found.

To extend the model on classes that are not perfectly separable by means of a hyperplane, the hinge loss function is introduced, which penalizes points falling in between the margin:

\begin{equation}
L_{hinge} :=\ \max\left( 0,\ 1-y_{i}\left( w^{T} x_{i} +b\right)\right)
\end{equation}

The previous opt. problem is then generalized to the **soft margin** version, which is equivalent to the ERM of the hinge loss with an L2 regularization term:

#### Primal Optimization Problem (Soft Margin)

\begin{array}{l}
\underset{w,b,\xi }{min} \ \ \tfrac{1}{2}\Vert w\Vert ^{2} \ +C\sum ^{n}_{i} \xi _{i}\\
\\
s.t.:\\
y_{i}\left( w^{T} x_{i} +b\right) \geqslant 1-\xi _{i} \ ,\ \forall i\\
\xi _{i} \geqslant 0\ ,\ \forall i\\
\end{array}

Where $𝝃$ is a slack variable which is introduced to in order to soften the misclassification constraint, allowing the model to make a certain number of mistakes and letting the margin to remain as wide as possible. The amount of misclassification allowed
is controlled by the hyperparameter $C$ which regulates the strength of the Hinge Loss term introduced: a higher $C$ will lead to a stronger minimization of the hinge loss (forcing training points to be more correctly classified), while a lower $C$ imposes a harder regularization (allowing more missclassifications) leading to a larger margin.
The Lagrangian Dual Problem is as follows:

#### Dual Optimization Problem (Soft Margin)

\begin{array}{l}
\underset{\alpha }{max} \ \ \sum ^{n}_{i} \alpha _{i} -\tfrac{1}{2}\sum _{i,j} \alpha _{i} \alpha _{j} y_{i} y_{j}\left( x^{T}_{i} x_{j}\right)\\
\\
s.t.\ \\
\sum \alpha _{i} y_{i} =0\ \ \land \ 0\leqslant \alpha _{i} \leqslant C,\ \forall i
\end{array}

By solving the dual problem (e.g. through Quadratic Programming) we find $w=\sum ^{n}_{i} \alpha _{i} y_{i} x_{i}$ as a linear combination of the training data. In particular, only the points that are in between (or exactly on) the margin will have an $\alpha_{i}\neq0$, i.e. only the so called support vectors are affecting the decision function. The latter can be then also denoted as $sign\left(\sum \alpha _{i} y_{i}\left( x^{T}_{i} x\right)+b\right)$, with $x$ a generic test observation.
Note how in the soft margin version the number of support vectors found is generally much higher.

Starting from these assumptions, the SVM model can still be generalized to multiclass classification and **non-linearly separable classes**. Indeed, as for the second case, classes may follow a non-linear pattern and a hyperplane may not be the best fit for explaining our data.
However, the given dataset might still be linearly separable when mapped onto a higher-dimensional space by a non-linear function phi, such that the training points are transformed as $x_{i} \in \mathbb{R}^{d} \mapsto \varphi ( x_{i}) \in \mathbb{R}^{d^{\prime}} ,\ d^{\prime}>d$.

While it is possible to compute the SVM algorithm in a higher space through a feature map , a different trick is instead practically used which exploits a nice property of support vector machines.
Indeed, note that the dual optimization problem and the newly defined decision function solely depend on the scalar product between the various points at all times. Whenever this property is satisfied then no matter what the higher-dimensional space is, all that it’s needed is a function able to compute the scalar products in the new transformed space.

We then introduce the notion of a Kernel function $K$, defined as:

\begin{equation}
K\ :\ \mathcal{X} \times \mathcal{X}\rightarrow \ \mathbb{R}\\
( x,\ x^{\prime} ) \mapsto K( x,\ x^{\prime} ):=\varphi ( x)^{T} \varphi ( x^{\prime} )
\end{equation}

for some feature map $\phi$. Then, we can simply apply different kernel functions to the SVM algorithm without directly knowing what the feature transformation is.
The power of this approach relies on the Mercer’s Theorem, which gives a sufficient condition for a function to be a Kernel for some unknown $\varphi ( \cdot )$.

Common Kernel functions used are:

- Linear Kernel: $K( x,x^{\prime} ) =x^{T} x^{\prime}$
- Gaussian RBF Kernel: $K(x,x^{\prime} )\ =\ \exp (-\gamma \| x-x^{\prime} \| ^{2} ),\ \gamma >0$
- Polynomial Kernel: $K( x,x^{\prime} ) \ =\ \left( x^{T} x^{\prime} +c\right)^{d}$

In particular, for the Gaussian RBF Kernel the decision function is computing a similarity measure between the test point and the support vectors (weigthed by $\alpha_i$ and $y_i$), which is inversely proportional to the euclidean distance of the two, leading to an output label that is determined by the closest support vectors. The new hyperparameter gamma can be tuned to regulate how many support vectors are significantly affecting the decision function (usually set to auto: $1/n\_features$ or scale: $1/( n\_features*X.var())$).

The decision function then becomes $sign\left(\sum \alpha _{i} y_{i}K(x_{i},x)+b\right)$.
Finally note that while it is convenient to use a kernel function, the hyperplane in higher space is not actually computed when the kernel trick is applied, i.e. the coefficients of the vector $w$ are unknown.

In this study, both the linear and the gaussian RBF Kernels have been tried. The regularization parameter $C$ has been hypertuned to find the best performing model on the training set.

```python
#Best configuration found
#polynomial kernel
params = {'C': 100, 'kernel': 'poly'}
#kernel RBF
params = {'C': 1, 'kernel': 'rbf'}
```

|                              	| Precision 	| Recall 	| F1-score 	|
|------------------------------	|:---------:	|:------:	|:--------:	|
| SVM POLY + SMOTE             	|    0.64   	|  0.28  	|   0.39   	|
| SVM RBF + SMOTE              	|    0.54   	|  0.52  	|   0.53   	|
| SVM POLY + Cluster Centroids 	|    0.68   	|  0.18  	|   0.29   	|
| SVM RBF + Cluster Centroids  	|    0.58   	|  0.46  	|   0.51   	|

 Scores on validation for the best configuration found with both kernels and both preprocessing techniques

- the score obtained are quite similar indipendently from the preprocessing technique (SMOTE is a little bit better);
- with a polinomial kernel in general the recall on positive class is lower than all the other cases;
- the gaussian version of the SVM outperforms the polinomial one.

In conclusion, also in this case the results should be improved in order to get more accurate prediction on default credit card clients.

## 5. Results

The following barplot displays a summary of results in the training-validation phase where all the algorithms are trained with their best hyperparameters (i.e. the ones that maximize the f1-score on positive class), with both techniques presented to overcome class imbalancing problem.

 Comparison of F1-score with different algorithms

#### Comments

- Logistic Regression and Support Vector Machines maintain the same performance regardless the imbalancing techniques used, while this is not true for Random Forest, the reason might be that a decision tree to perform well needs lot of data but at the same time undersampling technique reduce the amount of data.
- Overall, oversampling slightly outperforms undersampling;
- The positive class is the most difficult to classify and none of the models chosen seems to be not able to get the complexity of the problem.
- Empirically has been observed that in general undersampling require more time with respect to oversampling as preprocessing step, this is likely since the first one compute a k-means over all data to find centroids.

All the above mentioned algorithms have been tried out and hypertuned on the training set with a k-fold cross validation technique. The best performing configurations of hyperparameters on the training set have been chosen for rebuilding the models and evaluating them on the test set.

Finally, the table below the results of all the classifiers (in their best configuration) on validation and test set are compared. As we can see, the score obtained on the test set are pretty close to the one obtained in validation.

| Algorithm      | Validation Accuracy | Validation F1-score | Test Accuracy | Test F1-score |
|----------------|:-------------------:|:-------------------:|:-------------:|:-------------:|
| LR + PCA       |         0.75        |         0.45        |      0.73     |      0.44     |
| LR + PCA + OS  |         0.81        |         0.50        |      0.81     |      0.51     |
| LR + PCA + US  |         0.77        |         0.51        |      0.77     |      0.51     |
| RF + PCA       |         0.74        |         0.47        |      0.74     |      0.47     |
| RF + PCA + OS  |         0.75        |         0.46        |      0.75     |      0.48     |
| RF + PCA + US  |         0.47        |         0.38        |      0.47     |      0.37     |
| SVM + PCA      |         0.77        |         0.52        |      0.78     |      0.52     |
| SVM + PCA + OS |         0.78        |         **0.53**    |      0.78     |      0.52     |
| SVM + PCA + US |         0.77        |         0.51        |      0.77     |      0.51     |

 LR: Logistic Regression, RF: Random Forest, SVM: Support Vector Machines,
 OS: Oversampling (SMOTE), US: Undersampling (Cluster Centroids)

*Note that the scores obtained on the test should never be treated as additional information to change how the training is performed, but only as a final evaluation of the model.*

## 6. Conclusions
In this study different supervised learning algorithms have been inspected and presented with their mathematical details, and finally used on the UCI dataset to build a classification model that is able to predict if a credit card clients will default in the next month. Data preprocessing makes algorithms perform slightly better than when trained with original data: in particular, PCA results are approximately the same, but the computational cost has been lowered. Oversampling and undersampling techniques has been combined with PCA to assess the dataset imbalance problem. Oversampling as mentioned performed slightly better w.r.t. the undersampling, this is likely because the model is trained on a large amount of data. However, all the models implemented achieved comparable results in terms of accuracy.

### References

[1] Default of credit card clients Data Set: UCI dataset [link](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients)

[2] Yeh, I. C., & Lien, C. H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473-2480. [link](https://bradzzz.gitbooks.io/ga-seattle-dsi/content/dsi/dsi_05_classification_databases/2.1-lesson/assets/datasets/DefaultCreditCardClients_yeh_2009.pdf)

[3] Understanding Machine Learning: From Theory to Algorithms, S. Shalev-Shwartz, S. Ben-David, 2014 [link](https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf)


# MXNet for Collaborative Deep Learning in Recommender Systems
In this tutorial, we build on MXNet to implement the Collaborative Deep Learning (CDL) [1] model for recommender systems.

## Brief Introduction of CDL

In CDL, a probabilistic stacked denoising autoencoder (pSDAE) is connected to a matrix factorization (MF) component. Model training will alternate between pSDAE and MF. In each epoch, a pSDAE with a reconstruction target at the end and a regression target in the bottleneck will be udpated before updating the latent factors U and V in the regularized MF.

Below is the graphical model for CDL. The part in the red rectangle is pSDAE and the rest is the MF component regularized by pSDAE. Essentially, the updating will alternate between pSDAE (updating $W^+$) and the MF component (updating $u$ and $v$).
### Some Notation:
- $x_0$: the input vectors to pSDAE (corrupted data, e.g., randomly deleting some entries of the input)
- $x_c$: the reconstruction target vectors (the uncorrupted data)
- $x_{L/2}$: the output vectors of pSDAE's middle layer (bottleneck layer)
- $X_0$: the matrix consists of vectors $x_0$
- $X_c$: the matrix consists of vectors $x_c$
- $W^+$: weights and biases of pSDAE
- $v$: latent item vectors
- $u$: latent user vectors
- $R$: rating matrix ('1' if the article is in the user's library and '0' otherwise)
- $I$: number of users
- $J$: number of items
- $\lambda_u$, $\lambda_v$, $\lambda_w$, $\lambda_n$: hyperparameters

Below we show a special case of CDL (from a neural network perspective), where it degenerates to simultaneously training two neural networks overlaid together with a common input layer (the corrupted input) but different output layers. This might be a lot easier to understand for people not familiar with graphical models.

The objective function (which we use in this implementation) for this special case is:
\begin{align}
\mathscr{L}=&-\frac{\lambda_u}{2}\sum\limits_i \|u_i\|_2^2
-\frac{\lambda_w}{2}\sum\limits_l(\|W_l\|_F^2+\|b_l\|_2^2)\nonumber \\
&-\frac{\lambda_v}{2}\sum\limits_j\|v_j-f_e(X_{0,j*},W^+)^T\|_2^2 \nonumber \\
&-\frac{\lambda_n}{2}\sum\limits_j\|f_r(X_{0,j*},W^+)-X_{c,j*}\|_2^2 \nonumber \\
&-\sum\limits_{i,j}\frac{C_{ij}}{2}(R_{ij}-u_i^Tv_j)^2,
\end{align}
where the encoder function $f_e(\cdot,W^+)$ takes the corrupted content vector $X_{0,j*}$ of item $j$ as input and computes the encoding of the item, and the function $f_r(\cdot,W^+)$ also takes $X_{0,j*}$ as input, computes the encoding and then the reconstructed content vector of item $j$. Here $\lambda_w$, $\lambda_n$, $\lambda_u$, and $\lambda_v$ are hyperparameters and $C_{ij}$ is a confidence parameter ($C_{ij} = a$ if $R_{ij}=1$ and $C_{ij}=b$ otherwise). For example, if the number of layers $L=6$, $f_e(X_{0,j*},W^+)$ is the output of the third layer while $f_r(X_{0,j*},W^+)$ is the output of the sixth layer.

To learn CDL, we have to implement the block coordinate descent (BCD) update using numpy/mshadow and call this BCD procedure after each epoch of pSDAE. Besides the MF part, another difference between CDL and conventional deep learning models is that pSDAE has a fixed target at the end and a dynamic target (the latent item factors V) in the bottleneck layer. It might need some hacking to make this work.

[1] H. Wang, N. Wang, and D. Yeung. Collaborative deep learning for recommender systems. In KDD, 2015.

## Implementing CDL in MXNet for Recommender Systerms

```python
import mxnet as mx
import numpy as np
import logging
import data
from math import sqrt
from autoencoder import AutoEncoderModel
import os
```

### Setting Hyperparameters
- lambda_u: regularization coefficent for user latent matrix U
- lambda_v: regularization coefficent for item latent matrix V
- K: number of latent factors
- is_dummy: whether to use a dummy dataset for demo
- num_iter: number of iterations (minibatches) to train (a epoch in the used dataset takes about 68 iterations)
- batch_size: minibatch size
- dir_save: directory to save training results
- lv: lambda_v/lambda_n in CDL; this controls the trade-off between reconstruction error in pSDAE and recommendation accuracy during training

```python
lambda_u = 1 # lambda_u in CDL
lambda_v = 10 # lambda_v in CDL
K = 50
p = 1
is_dummy = False
num_iter = 100 # about 68 iterations/epoch, the recommendation results at the end need 100 epochs
batch_size = 256

np.random.seed(1234) # set seed
lv = 1e-2 # lambda_v/lambda_n in CDL
dir_save = 'cdl%d' % p
```

Create the directory and the log file.

```python
if not os.path.isdir(dir_save):
    os.system('mkdir %s' % dir_save)
fp = open(dir_save+'/cdl.log','w')
print 'p%d: lambda_v/lambda_u/ratio/K: %f/%f/%f/%d' % (p,lambda_v,lambda_u,lv,K)
fp.write('p%d: lambda_v/lambda_u/ratio/K: %f/%f/%f/%d\n' % \
        (p,lambda_v,lambda_u,lv,K))
fp.close()
```

    p1: lambda_v/lambda_u/ratio/K: 10.000000/1.000000/0.010000/50

### Loading Data
Here we load the text information (as input to pSDAE) in the file mult.dat and the rating matrix (as input for the MF part) in the file cf-train-1-users.dat. Code for loading the data are packed in data.py.

We use the CiteULike dataset here. The input text is bag-of-words vectors normalized to [0,1]. Some details:
- task: recommend articles to users
- number of users: 5551
- number of items: 16980
- number of ratings for training: ~169800
- number of terms: 8000

```python
# download data
import os
data_url='https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/cdl'
for filename in ('mult.dat', 'cf-train-1-users.dat', 'cf-test-1-users.dat', 'raw-data.csv'):
    if not os.path.exists(filename):
        os.system("wget %s/%s" % (data_url, filename))
# read data
X = data.get_mult()
R = data.read_user()
```

### Network Definition
Here we deine the logging level and construct the network. As mentioned before, pSDAE has multiple targets, we used `mx.symbol.Group` to group both loss. The codes snippets are shown as following, refer to [autoencode.py](autoencoder.py) for more details.

```python
fe_loss = mx.symbol.LinearRegressionOutput(
            data=self.lambda_v_rt*self.encoder,
            label=self.lambda_v_rt*self.V)
fr_loss = mx.symbol.LinearRegressionOutput(
            data=self.decoder, label=self.data)
loss = mx.symbol.Group([fe_loss, fr_loss])
```

```python
logging.basicConfig(level=logging.INFO)
cdl_model = AutoEncoderModel(mx.cpu(2), [X.shape[1],100,K],
    pt_dropout=0.2, internal_act='relu', output_act='relu')
```

### Initializing Variables
Here we initialize several variables. V is the latent item matrix and lambda_v_rt is an ndarray with entries equal to sqrt(lv). We need this lambda_v_rt to hack the trade-off between two targets in pSDAE.

```python
train_X = X
V = np.random.rand(train_X.shape[0],K)/10
lambda_v_rt = np.ones((train_X.shape[0],K))*sqrt(lv)
```

### Training the CDL
Train the whole CDL (joint training of pSDAE and the connected MF). We use SGD for pSDAE and BCD for the MF part. U is the user latent matrix, V is the item latent matrix, theta is the output of pSDAE's middle layer, and BCD_loss equals to rating_loss+reg_loss_for_U+reg_loss_for_V. For demostration we train for only 100 iterations (about 1.5 epochs) here. The shown recommendations in later parts are results after 100 epochs.

The function finetune below will call the function 'solve' in the solver.py, where the customized training loop resides. In the training loop, we call the following code after each epoch of pSDAE to update U and V using BCD. The BCD updating procedure is wrapped up in the function BCD_one. Note that after each epoch, we upate U and V for only one iteration.

```python
theta = model.extract_feature(sym[0], args, auxs,
    data_iter, X.shape[0], xpu).values()[0]
# update U, V and get BCD loss
U, V, BCD_loss = BCD_one(R, U, V, theta,
    lambda_u, lambda_v, dir_save, True)
# get recon' loss
Y = model.extract_feature(sym[1], args, auxs,
    data_iter, X.shape[0], xpu).values()[0]
Recon_loss = lambda_v/np.square(lambda_v_rt_old[0,0])*np.sum(np.square(Y-X))/2.0
lambda_v_rt[:] = lambda_v_rt_old[:] # back to normal lambda_v_rt
data_iter = mx.io.NDArrayIter({'data': X, 'V': V, 'lambda_v_rt':
    lambda_v_rt},
    batch_size=batch_size, shuffle=False,
    last_batch_handle='pad')
data_iter.reset()
batch = data_iter.next()
```

```python
U, V, theta, BCD_loss = cdl_model.finetune(train_X, R, V, lambda_v_rt, lambda_u,
        lambda_v, dir_save, batch_size,
        num_iter, 'sgd', l_rate=0.1, decay=0.0,
        lr_scheduler=mx.misc.FactorScheduler(20000,0.1))
```

    INFO:root:Fine tuning...
    INFO:root:Iter:0 metric:0.001668

    Epoch 1 - tr_err/bcd_err/rec_err: 53641376.1/27755.0/53613621.1

### Saving Models and Parameters
Save the network (pSDAE) parameters, latent matrices, and middle-layer output.

```python
cdl_model.save(dir_save+'/cdl_pt.arg')
np.savetxt(dir_save+'/final-U.dat.demo',U,fmt='%.5f',comments='')
np.savetxt(dir_save+'/final-V.dat.demo',V,fmt='%.5f',comments='')
np.savetxt(dir_save+'/final-theta.dat.demo',theta,fmt='%.5f',comments='')
```

### Computing Training Error
The training loss consists of the loss in pSDAE and that in MF.

```python
Recon_loss = lambda_v/lv*cdl_model.eval(train_X,V,lambda_v_rt)
print "Training error: %.3f" % (BCD_loss+Recon_loss)
fp = open(dir_save+'/cdl.log','a')
fp.write("Training error: %.3f\n" % (BCD_loss+Recon_loss))
fp.close()
```

    Training error: 53629559.864

### Generating Recommendations
Load the latent matrices (U and V), compute the predicted ratings R=UV^T, and generate recommendation lists for each user. There 5551 users in the dataset.

```python
import numpy as np
from data import read_user
def cal_rec(p,cut):
    R_true = read_user('cf-test-1-users.dat')
    dir_save = 'cdl'+str(p)
    #U = np.mat(np.loadtxt(dir_save+'/final-U.dat'))
    #V = np.mat(np.loadtxt(dir_save+'/final-V.dat'))
    R = U*V.T
    num_u = R.shape[0]
    num_hit = 0
    fp = open(dir_save+'/rec-list.dat','w')
    for i in range(num_u):
        if i!=0 and i%100==0:
            print 'User '+str(i)
        l_score = R[i,:].A1.tolist()
        pl = sorted(enumerate(l_score),key=lambda d:d[1],reverse=True)
        l_rec = list(zip(*pl)[0])[:cut]
        s_rec = set(l_rec)
        s_true = set(np.where(R_true[i,:]>0)[1].A1)
        cnt_hit = len(s_rec.intersection(s_true))
        fp.write('%d:' % cnt_hit)
        fp.write(' '.join(map(str,l_rec)))
        fp.write('\n')
    fp.close()

cal_rec(1,8)
```

    User 100
    User 200
    User 300
    User 400
    User 500
    User 600
    User 700
    User 800
    User 900
    User 1000
    User 1100
    User 1200
    User 1300
    User 1400
    User 1500
    User 1600
    User 1700
    User 1800
    User 1900
    User 2000
    User 2100
    User 2200
    User 2300
    User 2400
    User 2500
    User 2600
    User 2700
    User 2800
    User 2900
    User 3000
    User 3100
    User 3200
    User 3300
    User 3400
    User 3500
    User 3600
    User 3700
    User 3800
    User 3900
    User 4000
    User 4100
    User 4200
    User 4300
    User 4400
    User 4500
    User 4600
    User 4700
    User 4800
    User 4900
    User 5000
    User 5100
    User 5200
    User 5300
    User 5400
    User 5500

### Show Recommendations
Load the article titles (raw-data.csv), ratings (cf-train-1-users.dat and cf-test-1-users.dat), and recommendation lists (rec-list.dat).

```python
import csv
from data import read_user
import numpy as np
p = 1
# read predicted results
dir_save = 'cdl%d' % p
csvReader = csv.reader(open('raw-data.csv','rb'))
d_id_title = dict()
for i,row in enumerate(csvReader):
    if i==0:
        continue
    d_id_title[i-1] = row[3]
R_test = read_user('cf-test-1-users.dat')
R_train = read_user('cf-train-1-users.dat')
fp = open(dir_save+'/rec-list.dat')
lines = fp.readlines()
```

Show the titles of articles in the training set and titles of recommended articles. Correctly recommended articles are marked by asterisks.

```python
user_id = 3
s_test = set(np.where(R_test[user_id,:]>0)[1].A1)
l_train = np.where(R_train[user_id,:]>0)[1].A1.tolist()
l_pred = map(int,lines[user_id].strip().split(':')[1].split(' '))
print '##########  User '+str(user_id)+'  ##########\n'
print '#####  Articles in the Training Sets  #####'
for i in l_train:
    print d_id_title[i]
print '\n#####  Articles Recommended (Correct Ones Marked by Asterisks)  #####'
for i in l_pred:
    if i in s_test:
        print '* '+d_id_title[i]
    else:
        print d_id_title[i]
fp.close()
```

    ##########  User 3  ##########

    #####  Articles in the Training Sets  #####
    Formal Ontology and Information Systems
    Ontologies: a silver bullet for knowledge management and electronic commerce
    Business Process Execution Language for Web Services version 1.1
    Unraveling the Web services web: an introduction to SOAP, WSDL, and UDDI
    A cookbook for using the model-view controller user interface paradigm in Smalltalk-80
    Object-oriented application frameworks
    Data integration: a theoretical perspective
    Web services: been there, done that?
    Sweetening Ontologies with DOLCE
    Naive Geography

    #####  Articles Recommended (Correct Ones Marked by Asterisks)  #####
    The Wisdom of Crowds
    Nexus: Small Worlds and the Groundbreaking Theory of Networks
    VisANT: an integrative framework for networks in systems biology
    The Essays of Warren Buffett : Lessons for Corporate America
    What's new in pediatric orthopaedics.
    Doing with Understanding: Lessons from Research on Problem- and Project-Based Learning
    Predictably Irrational: The Hidden Forces That Shape Our Decisions
    The Predictably Irrational CD: The Hidden Forces That Shape Our Decisions


# Exact Wald `GiRaFFEfood` Initial Data for `GiRaFFE`

## Author: Zach Etienne & Patrick Nelson
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

### NRPy+ Source Code for this module: [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py](../edit/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py)

**Notebook Status:**  Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation1). The initial data has validated against the original `GiRaFFE`, as documented [here](Tutorial-Start_to_Finish_UnitTest-GiRaFFEfood_NRPy.ipynb).

## Introduction:
With the `GiRaFFE` evolution thorn constructed, we now need to "feed" our giraffe with initial data to evolve. There are several different choices of initial data we can use here; here, we will only be implementing the "Exact Wald" initial data, given by Table 3 in [the original paper](https://arxiv.org/pdf/1704.00599.pdf):
\begin{align}
A_{\phi} &= \frac{C_0}{2} r^2 \sin^2 \theta \\
E_{\phi} &= 2 M C_0 \left( 1+ \frac {2M}{r} \right)^{-1/2} \sin^2 \theta \\
\end{align}
(the unspecified components are set to 0). Here, $C_0$ is a constant that we will set to $1$ in our simulations. Now, to use this initial data scheme, we need to transform the above into the quantities actually tracked by `GiRaFFE` and HydroBase: $A_i$, $B^i$, $\tilde{S}_i$, $v^i$, and $\Phi$. Of these quantities, $\texttt{GiRaFFEfood}$ will only set $A_i$, $v^i$, and $\Phi=0$, then call a separate function to calculate  $\tilde{S}_i$; `GiRaFFE` itself will call a function to set $B^i$ before the time-evolution begins. This can be done with eqs. 16 and 18, here given in that same order:
\begin{align}
v^i &= \alpha \frac{\epsilon^{ijk} E_j B_k}{B^2} -\beta^i \\
B^i &= \frac{[ijk]}{\sqrt{\gamma}} \partial_j A_k \\
\end{align}
In the simulations, $B^i$ will be calculated numerically from $A_i$; however, it will be useful to analytically calculate $B^i$ to use calculating the initial $v^i$.

# Table of Contents:
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Import core NRPy+ modules and set NRPy+ parameters
1. [Step 2](#set_aphi_ephi): Set the vectors $A_{\phi}$ and $E_{\phi}$ in Spherical coordinates
1. [Step 3](#jacobian): Use the Jacobian matrix to transform the vectors to Cartesian coordinates
1. [Step 4](#vi): Calculate $v^i$ from $A_i$ and $E_i$
    1. [Step 4.a](#code_validation1): Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy` NRPy+ Module
1. [Step 5](#si): Build the expression for $\tilde{S}_i$
    1. [Step 5.a](#code_validation2): Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy` NRPy+ Module
1. [Step 6](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Import core NRPy+ modules and set NRPy+ parameters \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Here, we will import the NRPy+ core modules and set the reference metric to Cartesian, set commonly used NRPy+ parameters, and set C parameters that will be set from outside the code eventually generated from these expressions. We will also set up a parameter to determine what initial data is set up, although it won't do much yet.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0.a: Import the NRPy+ core modules and set the reference metric to Cartesian
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support

import reference_metric as rfm
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()

# Step 1a: Set commonly used parameters.
thismodule = "GiRaFFEfood_NRPy"

# KerrSchild_radial_shift = par.Cparameters("REAL",thismodule,"KerrSchild_radial_shift",0.4) # Default value for ExactWald
KerrSchild_radial_shift = sp.symbols("KerrSchild_radial_shift")
M = sp.symbols("M") # Black hole mass
```

# Step 2: Set the vectors $A_{\phi}$ and $E_{\phi}$ in Spherical coordinates \[Back to [top](#toc)\]
$$\label{set_aphi_ephi}$$

We will first build the fundamental vectors $A_i$ and $E_i$ in spherical coordinates (see [Table 3](https://arxiv.org/pdf/1704.00599.pdf)). Note that we use reference_metric.py to set $r$ and $\theta$ in terms of Cartesian coordinates; this will save us a step later when we convert to Cartesian coordinates. Since $C_0 = 1$,
\begin{align}
A_{\phi} &= \frac{1}{2} r^2 \sin^2 \theta \\
E_{\phi} &= 2 M \left( 1+ \frac {2M}{r} \right)^{-1/2} \sin^2 \theta. \\
\end{align}
While we have $E_i$ set as a variable in NRPy+, note that the final C code won't store these values.

```python
# Step 2: Set the vectors A and E in Spherical coordinates

r     = rfm.xxSph[0] + KerrSchild_radial_shift # We are setting the data up in Shifted Kerr-Schild coordinates
theta = rfm.xxSph[1]

# Initialize all components of A and E in the *spherical basis* to zero
ASphD = ixp.zerorank1()
ESphD = ixp.zerorank1()
ASphD[2] = (r * r * sp.sin(theta)**2)/2
ESphD[2] = 2 * M * sp.sin(theta)**2 / sp.sqrt(1+2*M/r)
```

# Step 3: Use the Jacobian matrix to transform the vectors to Cartesian coordinates \[Back to [top](#toc)\]
$$\label{jacobian}$$

Now, we will use the coordinate transformation definitions provided by reference_metric.py to build the Jacobian

$$
\frac{\partial x_{\rm Sph}^j}{\partial x_{\rm Cart}^i},
$$

where $x_{\rm Sph}^j \in \{r,\theta,\phi\}$ and $x_{\rm Cart}^i \in \{x,y,z\}$. We would normally compute its inverse, but since none of the quantities we need to transform have upper indices, it is not necessary. Then, since both $A_i$ and $E_i$ have one lower index, both will need to be multiplied by the Jacobian:

\begin{align}
A_i^{\rm Cart} &= A_j^{\rm Sph} \frac{\partial x_{\rm Sph}^j}{\partial x_{\rm Cart}^i} \\
{\rm and\ }E_i^{\rm Cart} &= E_j^{\rm Sph} \frac{\partial x_{\rm Sph}^j}{\partial x_{\rm Cart}^i}.
\end{align}

```python
# Step 3: Use the Jacobian matrix to transform the vectors to Cartesian coordinates.
drrefmetric__dx_0UDmatrix = sp.Matrix([[sp.diff(rfm.xxSph[0],rfm.xx[0]), sp.diff(rfm.xxSph[0],rfm.xx[1]), sp.diff(rfm.xxSph[0],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[1],rfm.xx[0]), sp.diff(rfm.xxSph[1],rfm.xx[1]), sp.diff(rfm.xxSph[1],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[2],rfm.xx[0]), sp.diff(rfm.xxSph[2],rfm.xx[1]), sp.diff(rfm.xxSph[2],rfm.xx[2])]])
#dx__drrefmetric_0UDmatrix = drrefmetric__dx_0UDmatrix.inv() # We don't actually need this in this case.

AD = ixp.zerorank1()
ED = ixp.zerorank1()

for i in range(3):
    for j in range(3):
        AD[i] = drrefmetric__dx_0UDmatrix[(j,i)]*ASphD[j]
        ED[i] = drrefmetric__dx_0UDmatrix[(j,i)]*ESphD[j]

#Step 4: Declare the basic spacetime quantities
alpha   = sp.symbols("alpha",real=True)
betaU   = ixp.declarerank1("betaU",DIM=3)
gammaDD = ixp.declarerank2("gammaDD", "sym01",DIM=3)

import GRHD.equations as GRHD
GRHD.compute_sqrtgammaDET(gammaDD)
```

# Step 4: Calculate $v^i$ from $A_i$ and $E_i$ \[Back to [top](#toc)\]
$$\label{vi}$$

We will now find the magnetic field using equation 18 in [the original paper](https://arxiv.org/pdf/1704.00599.pdf) $$B^i = \frac{[ijk]}{\sqrt{\gamma}} \partial_j A_k. $$ We will need the metric quantites: the lapse $\alpha$, the shift $\beta^i$, and the three-metric $\gamma_{ij}$. We will also need the Levi-Civita symbol.

```python
# Step 4: Calculate v^i from A_i and E_i
# Step 4a: Calculate the magnetic field B^i

GRHD.compute_sqrtgammaDET(gammaDD)
LeviCivitaTensorUUU = ixp.LeviCivitaTensorUUU_dim3_rank3(GRHD.sqrtgammaDET)

# For the initial data, we can analytically take the derivatives of A_i
ADdD = ixp.zerorank2()
for i in range(3):
    for j in range(3):
        ADdD[i][j] = sp.simplify(sp.diff(AD[i],rfm.xxCart[j]))

BU = ixp.zerorank1()
for i in range(3):
    for j in range(3):
        for k in range(3):
            BU[i] += LeviCivitaTensorUUU[i][j][k] * ADdD[k][j]
```

We will now build the initial velocity using equation 152 in [this paper,](https://arxiv.org/pdf/1310.3274v2.pdf) cited in the original `GiRaFFE` code: $$ v^i = \alpha \frac{\epsilon^{ijk} E_j B_k}{B^2} -\beta^i. $$
However, our code needs the Valencia 3-velocity while this expression is for the drift velocity. So, we will need to transform it to the Valencia 3-velocity using the rule $\bar{v}^i = \frac{1}{\alpha} \left(v^i +\beta^i \right)$.
Thus, $$\bar{v}^i = \frac{\epsilon^{ijk} E_j B_k}{B^2}$$

```python
# Step 4b: Calculate B^2 and B_i
# B^2 is an inner product defined in the usual way:
B2 = sp.sympify(0)
for i in range(3):
    for j in range(3):
        B2 += gammaDD[i][j] * BU[i] * BU[j]

# Lower the index on B^i
BD = ixp.zerorank1()
for i in range(3):
    for j in range(3):
        BD[i] += gammaDD[i][j] * BU[j]

# Step 4c: Calculate the Valencia 3-velocity
ValenciavU = ixp.zerorank1()
for i in range(3):
    for j in range(3):
        for k in range(3):
            ValenciavU[i] += LeviCivitaTensorUUU[i][j][k]*ED[j]*BD[k]/B2
```

## Step 4.a: Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy` NRPy+ module  \[Back to [top](#toc)\]
$$\label{code_validation1}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the `GiRaFFE` Exact Wald initial data equations  we intend to use between
1. this tutorial and
2. the NRPy+ [GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Exact_Wald](../edit/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py) module.

```python
import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Exact_Wald as gfho
gfho.GiRaFFEfood_NRPy_Exact_Wald(gammaDD,M,KerrSchild_radial_shift)

def consistency_check(quantity1,quantity2,string):
    if quantity1-quantity2==0:
        print(string+" is in agreement!")
    else:
        print(string+" does not agree!")
        sys.exit(1)

print("Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:")

for i in range(3):
    consistency_check(ValenciavU[i],gfho.ValenciavU[i],"ValenciavU"+str(i))
    consistency_check(AD[i],gfho.AD[i],"AD"+str(i))
```

    Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:
    ValenciavU0 is in agreement!
    AD0 is in agreement!
    ValenciavU1 is in agreement!
    AD1 is in agreement!
    ValenciavU2 is in agreement!
    AD2 is in agreement!

# Step 6: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFEfood_NRPy.pdf](Tutorial-GiRaFFEfood_NRPy.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-GiRaFFEfood_NRPy_Exact_Wald")
```

    Created Tutorial-GiRaFFEfood_NRPy_Exact_Wald.tex, and compiled LaTeX file
        to PDF file Tutorial-GiRaFFEfood_NRPy_Exact_Wald.pdf


# Tutorial: Small Angle Neutron Scattering

Small Angle Neutron Scattering (SANS) is a powerful reciprocal space technique that can be used to investigate magnetic structures on mesoscopic length scales. In SANS the atomic structure generally has a minimal impact hence the sample can be approximated by a continuous magnetisation vector field \[[Mühlbauer 2019](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.015004)\].

The differential scattering cross section $d\Sigma/d\Omega$ can be used as a function of the scattering vector ${\bf q}$ to predict the scattering patterns produced.

## SANS Reference frame

In `mag2exp` the experimental SANS reference frame is the same as that defined to be congruent with the sample reference frame. The neutron polarisation direction can then be defined relative to the sample reference frame along with the desired scattering plane.

There are common scattering geometries defined in SANS, namely with a magnetic field applied to the incoming neutron beam in either a perpendicular or and parallel geometry to polarise the neutrons. The `mag2exp` package gives a more general set up with the ability to have the polarisation pointing in any arbitrary direction. We can recreate the standard scattering geometries

**Perpendicular geometry**: the magnetic field is applied along the $z$ direction while the incoming neutron beam propagates along the $x$ direction. This is the same as `polarisation=(0, 0, 1)` and cutting the scattering plane along `x=0`.

**Parallel geometry**: both the magnetic field and the incoming neutron beam are along the $z$ direction. This is the same as `polarisation=(0, 0, 1)` and cutting the scattering plane along `z=0`.

## The micromagnetic simulation

A micromagnetic simulation can be set up using Ubermag to obtain a 3-Dimentional magntic structure with periodic boundary conditions in the $xy$ plane.

```python
# NBVAL_IGNORE_OUTPUT
import oommfc as oc
import discretisedfield as df
import micromagneticmodel as mm
import numpy as np
import ubermagutil.units as uu
np.random.seed(1)

region = df.Region(p1=(-160e-9, -160e-9, 0), p2=(160e-9, 160e-9, 20e-9))
mesh = df.Mesh(region=region, cell=(5e-9, 5e-9, 5e-9), bc='xyz')

system = mm.System(name='Box2')

system.energy = (mm.Exchange(A=1.6e-11)
                 + mm.DMI(D=4e-3, crystalclass='T')
                 + mm.Zeeman(H=(0, 0, 2e5)))

Ms = 1.1e6

def m_fun(pos):
    return 2 * np.random.rand(3) - 1

# create system with above geometry and initial magnetisation
system.m = df.Field(mesh, dim=3, value=m_fun, norm=Ms)
system.m.plane('z').mpl()
```

Relax the system and plot its magnetisation.

```python
# NBVAL_IGNORE_OUTPUT
# minimize the energy
md = oc.MinDriver()
md.drive(system)

# Plot relaxed configuration: vectors in z-plane
system.m.plane('z').mpl()
```

```python
system.m.plane('x').mpl()
```

Now we have a magnetisation texture we can compute the SANS scattering cross sections.

## Computing SANS Cross-sections

We can use the `mag2exp` package to calculate the cross sections.
The scattering can be calculated with the use of the magnetic interaction vector  ${\bf Q}$ \[[Mühlbauer 2019](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.015004)\] where

\begin{equation}
{\bf Q} = \hat{\bf q} \times \left[ \widetilde{\bf M} \times \hat{\bf q} \right],
\end{equation}
where $\hat{\bf q}$ is the unit scattering vector and $\widetilde{\bf M}$ is the Fourier transform of the magnetisation.

The scattering vector is defined as
\begin{equation}
{\bf q} = {\bf k}_1 - {\bf k}_0
\end{equation}
where ${\bf k}_0$ and ${\bf k}_1$ are the incident and scattered wavevectors respectively.

From this the magnetic contribution to the cross sections can be calculated as

\begin{equation}
\frac{d\sum}{d\Omega} \sim |{\bf Q} \cdot {\bf \sigma}|^2,
\end{equation}
where ${\bf \sigma}$ is the Pauli vector
\begin{equation}
{\bf \sigma} = \begin{bmatrix} \sigma_x \\ \sigma_y \\ \sigma_z \end{bmatrix},
\end{equation}
and
\begin{align}
\sigma_x &= \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}, \\
\sigma_y &= \begin{pmatrix} 0 & -i \\ i & 0 \end{pmatrix}, \\
\sigma_z &= \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}.
\end{align}

For a polarisation in an arbitrary direction the Pauli vector can be rotated.

This takes the form
\begin{equation}
\frac{d\sum}{d\Omega} = \begin{pmatrix}
\frac{d\sum^{++}}{d\Omega} & \frac{d\sum^{-+}}{d\Omega}\\
\frac{d\sum^{+-}}{d\Omega} & \frac{d\sum^{--}}{d\Omega}
\end{pmatrix}
\end{equation}

These then be combined in order to get the half and unpolarised cross sections.

\begin{align}
\frac{d\sum^{+}}{d\Omega} &= \frac{d\sum^{++}}{d\Omega} + \frac{d\sum^{+-}}{d\Omega} \\
\frac{d\sum^{-}}{d\Omega} &= \frac{d\sum^{--}}{d\Omega} + \frac{d\sum^{-+}}{d\Omega} \\
\frac{d\sum}{d\Omega} &= \frac{1}{2} \left( \frac{d\sum^{+}}{d\Omega} + \frac{d\sum^{-}}{d\Omega} \right) \\
\frac{d\sum}{d\Omega} &= \frac{1}{2} \left( \frac{d\sum^{++}}{d\Omega} + \frac{d\sum^{+-}}{d\Omega} +
\frac{d\sum^{--}}{d\Omega} + \frac{d\sum^{-+}}{d\Omega} \right)
\end{align}

For example the unpolarised cross section calculated with the polarisation along the $z$ direction.

```python
import mag2exp
```

```python
cross_section = mag2exp.sans.cross_section(system.m, method='unpol', polarisation=(0, 0, 1))
```

The `mag2exp.sans.cross_section` function produces a three dimentional cross section in scattering space. A plane of this cross section can then be cut to obtain the relevant scattering plane. For example the `z=0` scattering plane is used here, this is equivalent to the incoming neutron beam being parallel to the $z$ direction.

As the cross section is a `discretisedfield` object the built in plotting functions can be used to view them.

NOTE: The values of the axis in Fourier space are frequency NOT angular frequency so DO NOT include a factor of $2\pi$. i.e. $|{\bf k}| = \frac{1}{\lambda} \neq \frac{2 \pi}{\lambda}$, where $\bf k$ is the wave vector and $\lambda$ is the wavelength.

```python
# NBVAL_IGNORE_OUTPUT
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16',
                                    colorbar_label=r'Intensity')
```

Due to the high intensity of some areas a linear colour map make some features more difficult to see. We can use `matplotlib.color` to change the colour bar to a logarithmic scale. This
reveals the higher order low intensity diffraction peaks.

```python
# NBVAL_IGNORE_OUTPUT
import matplotlib.colors as colors
cross_section.plane(z=0).mpl.scalar(cmap='gray',
                                    colorbar_label=r'Intensity',
                                    norm=colors.LogNorm(vmin=1e-3,vmax=cross_section.real.array.max()))
```

It is also possible to just plot a selected region of the cross section by

```python
# NBVAL_IGNORE_OUTPUT
sans_region = df.Region(p1=(-40e6, -40e6, 0), p2=(40e6, 40e6, 0.5))
cross_section.plane(z=0)[sans_region].mpl.scalar(cmap='gray', interpolation='spline16',
                                                 colorbar_label=r'Intensity')
```

For this in the parallel scattering geometry the spin-flip cross sections are

```python
# NBVAL_IGNORE_OUTPUT
cross_section = mag2exp.sans.cross_section(system.m, method='pn', polarisation=(0, 0, 1))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
cross_section = mag2exp.sans.cross_section(system.m, method='np', polarisation=(0, 0, 1))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
```

and non-spin-flip cross sections are

```python
# NBVAL_IGNORE_OUTPUT
cross_section = mag2exp.sans.cross_section(system.m, method='pp', polarisation=(0, 0, 1))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
cross_section = mag2exp.sans.cross_section(system.m, method='nn', polarisation=(0, 0, 1))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
```

We can also look at the scattering geometry where the beam and polarisation are no longer parallel.

```python
# NBVAL_IGNORE_OUTPUT
cross_section = mag2exp.sans.cross_section(system.m, method='unpol', polarisation=(1, 0, 0))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16',
                                    colorbar_label=r'Intensity')
```

```python
cross_section = mag2exp.sans.cross_section(system.m, method='pp', polarisation=(1, 0, 0))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
cross_section = mag2exp.sans.cross_section(system.m, method='nn', polarisation=(1, 0, 0))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
```

```python
cross_section = mag2exp.sans.cross_section(system.m, method='pn', polarisation=(1, 0, 0))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
cross_section = mag2exp.sans.cross_section(system.m, method='np', polarisation=(1, 0, 0))
cross_section.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
```

It is possible to examine the chiral function $-2i\chi$, which gives an indication of the asymmetry of the scattering.

```python
chiral = mag2exp.sans.chiral_function(system.m, polarisation=(1, 0, 0))
chiral.plane(z=0).mpl.scalar(cmap='gray', interpolation='spline16', colorbar_label=r'Intensity (arb.)')
```