# Assignement for course ML2

Students: Tristan STAMPFLER and Thomas CARTA

## Table of contents

### 1) Exploration of the dataset
### 2) Data cleaning
### 3) Feature engineering
### 4) Feature selection
### 5) Metric definition
### 6) Various prediction models
### 7) Conclusion

## 1) Exploration of the dataset

Preliminary exploration of the dataset is necessary to determine which field we can use.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
df = pd.read_csv("data/CreditTraining.csv")
```

```python
df.head()
```

      Id_Customer
      Y
      Customer_Type
      BirthDate
      Customer_Open_Date
      P_Client
      Educational_Level
      Marital_Status
      Number_Of_Dependant
      Years_At_Residence
      Net_Annual_Income
      Years_At_Business
      Prod_Sub_Category
      Prod_Decision_Date
      Source
      Type_Of_Residence
      Nb_Of_Products
      Prod_Closed_Date
      Prod_Category

      0
      7440
      0
      Non Existing Client
      07/08/1977
      13/02/2012
      NP_Client
      University
      Married
      3.0
      1
      36
      1.0
      C
      14/02/2012
      Sales
      Owned
      1
      NaN
      B

      1
      573
      0
      Existing Client
      13/06/1974
      04/02/2009
      P_Client
      University
      Married
      0.0
      12
      18
      2.0
      C
      30/06/2011
      Sales
      Parents
      1
      NaN
      G

      2
      9194
      0
      Non Existing Client
      07/11/1973
      03/04/2012
      NP_Client
      University
      Married
      2.0
      10
      36
      1.0
      C
      04/04/2012
      Sales
      Owned
      1
      NaN
      B

      3
      3016
      1
      Existing Client
      08/07/1982
      25/08/2011
      NP_Client
      University
      Married
      3.0
      3
      36
      1.0
      C
      07/09/2011
      Sales
      New rent
      1
      31/12/2012
      L

      4
      6524
      0
      Non Existing Client
      18/08/1953
      10/01/2012
      NP_Client
      University
      Married
      2.0
      1
      36
      1.0
      C
      11/01/2012
      Sales
      Owned
      1
      NaN
      D

```python
df.describe()
```

      Id_Customer
      Y
      Number_Of_Dependant
      Years_At_Residence
      Years_At_Business
      Nb_Of_Products

      count
      5380.000000
      5380.000000
      5378.000000
      5380.000000
      5378.000000
      5380.000000

      mean
      4784.535688
      0.073048
      1.058572
      12.626022
      4.266270
      1.089033

      std
      2781.436262
      0.260240
      1.339001
      9.972164
      7.225926
      0.297587

      min
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000

      25%
      2368.500000
      0.000000
      0.000000
      4.000000
      1.000000
      1.000000

      50%
      4762.500000
      0.000000
      0.000000
      10.000000
      1.000000
      1.000000

      75%
      7180.250000
      0.000000
      2.000000
      18.000000
      4.000000
      1.000000

      max
      9605.000000
      1.000000
      20.000000
      70.000000
      98.000000
      3.000000

### Checking for NaNs

```python
df.isna().sum()
```

    Id_Customer               0
    Y                         0
    Customer_Type             0
    BirthDate                 0
    Customer_Open_Date        0
    P_Client                  0
    Educational_Level         0
    Marital_Status            0
    Number_Of_Dependant       2
    Years_At_Residence        0
    Net_Annual_Income         2
    Years_At_Business         2
    Prod_Sub_Category         0
    Prod_Decision_Date        0
    Source                    0
    Type_Of_Residence         0
    Nb_Of_Products            0
    Prod_Closed_Date       4206
    Prod_Category             0
    dtype: int64

We can see have a lot of missing values for the field Prod_Closed_Date, and some for Number_Of_Dependant, Net_Annual_Income and Years_At_Business

### Checking if Id_Customer is an unique key

```python
print(df[['Id_Customer']].nunique() == len(df))
```

    Id_Customer    True
    dtype: bool

Yes, Id_Customer is a unique key for this dataset

### Practising on marital

```python
df_marital = df[['Id_Customer','Y','Marital_Status']]
df_marital['Marital_Status'].value_counts()
```

    Married      4206
    Single       1046
    Widowed        64
    Divorced       63
    Separated       1
    Name: Marital_Status, dtype: int64

```python
marital_grp = df_marital.groupby('Marital_Status')
marital_grp['Y'].value_counts(normalize=True)
```

    Marital_Status  Y
    Divorced        0    0.920635
                    1    0.079365
    Married         0    0.930813
                    1    0.069187
    Separated       0    1.000000
    Single          0    0.911090
                    1    0.088910
    Widowed         0    0.937500
                    1    0.062500
    Name: Y, dtype: float64

```python
marital_grp.agg({'Y':'mean',
                 'Id_Customer':'count'})
```

      Y
      Id_Customer

      Marital_Status

      Divorced
      0.079365
      63

      Married
      0.069187
      4206

      Separated
      0.000000
      1

      Single
      0.088910
      1046

      Widowed
      0.062500
      64

## 2) Data cleaning

We need to handle NaNs and select the columns we will use for our approaches

```python
df.isna().sum()
```

    Id_Customer               0
    Y                         0
    Customer_Type             0
    BirthDate                 0
    Customer_Open_Date        0
    P_Client                  0
    Educational_Level         0
    Marital_Status            0
    Number_Of_Dependant       2
    Years_At_Residence        0
    Net_Annual_Income         2
    Years_At_Business         2
    Prod_Sub_Category         0
    Prod_Decision_Date        0
    Source                    0
    Type_Of_Residence         0
    Nb_Of_Products            0
    Prod_Closed_Date       4206
    Prod_Category             0
    dtype: int64

### Removing NaNs

We will remove NaNs in 2 ways. For Prod_Closed_Date there are too much values missing so we will just get rid of this field. For Number_Of_Dependant, Net_Annual_Income and Years_At_Business we just have a few values missing. Assuming the clients are iid, we can remove the rows with missing values.

```python
df = df.drop("Prod_Closed_Date", axis=1)
df = df.dropna()
```

```python
df = df.reset_index(drop=True)
df.isna().sum()
df.head()
```

      Id_Customer
      Y
      Customer_Type
      BirthDate
      Customer_Open_Date
      P_Client
      Educational_Level
      Marital_Status
      Number_Of_Dependant
      Years_At_Residence
      Net_Annual_Income
      Years_At_Business
      Prod_Sub_Category
      Prod_Decision_Date
      Source
      Type_Of_Residence
      Nb_Of_Products
      Prod_Category

      0
      7440
      0
      Non Existing Client
      07/08/1977
      13/02/2012
      NP_Client
      University
      Married
      3.0
      1
      36
      1.0
      C
      14/02/2012
      Sales
      Owned
      1
      B

      1
      573
      0
      Existing Client
      13/06/1974
      04/02/2009
      P_Client
      University
      Married
      0.0
      12
      18
      2.0
      C
      30/06/2011
      Sales
      Parents
      1
      G

      2
      9194
      0
      Non Existing Client
      07/11/1973
      03/04/2012
      NP_Client
      University
      Married
      2.0
      10
      36
      1.0
      C
      04/04/2012
      Sales
      Owned
      1
      B

      3
      3016
      1
      Existing Client
      08/07/1982
      25/08/2011
      NP_Client
      University
      Married
      3.0
      3
      36
      1.0
      C
      07/09/2011
      Sales
      New rent
      1
      L

      4
      6524
      0
      Non Existing Client
      18/08/1953
      10/01/2012
      NP_Client
      University
      Married
      2.0
      1
      36
      1.0
      C
      11/01/2012
      Sales
      Owned
      1
      D

## Replace some object type with float

```python
df['Net_Annual_Income'] = df['Net_Annual_Income'].str.replace(',','.').astype(float)
```

## 3) Feature engineering

We create some features from the fields we have in our dataset.

We compute here for instance the age of the client. We then get rid of obsolete fields.

```python
df['Age'] = (pd.Timestamp('2020-01-01 00:00:00.000000') - pd.to_datetime(df['BirthDate'])).dt.days / 365.25
df = df.drop('BirthDate', axis=1)
```

## 4) Feature selection

We select all features

```python
df.head()
```

      Id_Customer
      Y
      Customer_Type
      Customer_Open_Date
      P_Client
      Educational_Level
      Marital_Status
      Number_Of_Dependant
      Years_At_Residence
      Net_Annual_Income
      Years_At_Business
      Prod_Sub_Category
      Prod_Decision_Date
      Source
      Type_Of_Residence
      Nb_Of_Products
      Prod_Category
      Age

      0
      7440
      0
      Non Existing Client
      13/02/2012
      NP_Client
      University
      Married
      3.0
      1
      36.0
      1.0
      C
      14/02/2012
      Sales
      Owned
      1
      B
      42.483231

      1
      573
      0
      Existing Client
      04/02/2009
      P_Client
      University
      Married
      0.0
      12
      18.0
      2.0
      C
      30/06/2011
      Sales
      Parents
      1
      G
      45.552361

      2
      9194
      0
      Non Existing Client
      03/04/2012
      NP_Client
      University
      Married
      2.0
      10
      36.0
      1.0
      C
      04/04/2012
      Sales
      Owned
      1
      B
      46.475017

      3
      3016
      1
      Existing Client
      25/08/2011
      NP_Client
      University
      Married
      3.0
      3
      36.0
      1.0
      C
      07/09/2011
      Sales
      New rent
      1
      L
      37.401780

      4
      6524
      0
      Non Existing Client
      10/01/2012
      NP_Client
      University
      Married
      2.0
      1
      36.0
      1.0
      C
      11/01/2012
      Sales
      Owned
      1
      D
      66.370979

```python
df.dtypes
```

    Id_Customer              int64
    Y                        int64
    Customer_Type           object
    Customer_Open_Date      object
    P_Client                object
    Educational_Level       object
    Marital_Status          object
    Number_Of_Dependant    float64
    Years_At_Residence       int64
    Net_Annual_Income      float64
    Years_At_Business      float64
    Prod_Sub_Category       object
    Prod_Decision_Date      object
    Source                  object
    Type_Of_Residence       object
    Nb_Of_Products           int64
    Prod_Category           object
    Age                    float64
    dtype: object

## 5) Metric definition

```python
from IPython.display import Latex
```

```python
df['Y'].mean()
```

    0.07294380349832527

We need a consistent metric for our models to optimize. Since the dataset is heavily imbalanced (about 7.3% of 1 and 92.7% of 0) the accuracy is not by itself so relevant. Indeed a predictor that always predict 0 could get a 92.7% accuracy on this dataset). Furthermore, misclassifying a credit default as 0 is far more costly than misclassifying a credit that will be repaid as 1. But for the bank to earn some money we cannot reject all credit applications either.

We choose a target metric that maximizes the bank's profit per capita. We claim that this metric is:

\begin{equation}
\text{metric} = (1-\alpha) i \times \text{true_negative_rate} + \alpha \times \text{true_positive_rate} - \alpha
\end{equation}

With $i$ the annual interest rate and $\alpha$ the proportion credit default among all credits.

__Demonstration__:

We firstly assume that the rows in the dataset are realisations of iid distributed variables reprensenting the population. We then assume that the bank only lends money for a one year period with no monthly payment and always lend the same amount. So after one year, either the bank gets back its money plus $i$ of interests or it faces a credit default and lost all of its money.

So if the bank uses a prediction algorithm to decide wether or not to give a credit to a customer and follows strictly the algorithm predictions, its total profit (normalized by the amount of the credit) on the population is:
\begin{equation}
\text{total profit} = i* TN - FN
\end{equation}

Where we note $n$ the total number of individuals, $TP$ the number of true positives, $FP$ the number of false positives, $TN$ the number of true negatives and $FN$ the number of false negatives. According to our modelisation we have the following properties:

\begin{equation}
n = TP + FP + TN + FN \; \text{(1)} \text{,}\;\;\;\;\; TP + FN = \alpha n \; \text{(2)} \;\;\; \text{and}\;\;\; TN + FP = (1-\alpha)n \; \text{(3)}
\end{equation}

We can next rewrite the total profit using (1): $\text{total profit} = i*TN - n + TN + FP + TP$.

If we set our metric to be the profit per capita (including all individuals) we get:

\begin{equation}
\text{metric} = \text{profit per capita} \\
= \frac{\text{total profit}}{n} \\
= i*\frac{TN}{n} - 1 + \frac{TN + FP}{n} + \frac{TP}{n} \\
= (1-\alpha)i*\frac{TN}{TN+FP} -1 + (1-\alpha)\frac{TN+FP}{TN+FP} + \alpha\frac{TP}{TP+FN} \;\;\; \text{using (2) and (3)} \\
= (1-\alpha)i*\text{true negative rate} + \alpha*\text{true positive rate} - \alpha
\end{equation}
Which is exactly the metric stated above.

__End of demonstration__

For the rest of the exercise, we set $i=5\%$ for the annual interest rate and we estimate $\alpha$ with the dataset.

```python
from sklearn.metrics import confusion_matrix
```

```python
def metric(y, y_pred, i=0.05):
    tn, fp, fn, tp = confusion_matrix(y, y_pred).ravel()
    n = y_pred.size
    alpha = (tp+fn)/n
    metric = (1-alpha)*i*(tn/(tn+fp))+alpha*(tp/(tp+fn)) - alpha
    return metric
```

## 6) Various prediction models

### a) Logistic regression on quantitative variables

We try out Logistic regression on the quantitative variables

```python
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
```

We split the data between a train set and a test. We use the train set to train the algorithm and to tune hyperparameters, and we use the test set to evaluate the algorithm performance.

```python
quant_cols = ['Number_Of_Dependant','Years_At_Residence','Net_Annual_Income','Years_At_Business','Nb_Of_Products','Age']
X = df.loc[:,quant_cols]
Y = df.loc[:,'Y']
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
clf = LogisticRegression()
clf.fit(X_train, y_train)
```

    C:\Users\Thomas\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, zero_division=False)
recall = recall_score(y_test, y_pred, zero_division=False)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

As we can notice here, Y_pred is all zeros. The logistic regression has learned to predict only the majority class: 0. As the metric shows, with such prediction algorithm the bank is loosing money.

We can fix it with weighting the classes in the loss.

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)).

```python
clf_balanced = LogisticRegression(class_weight='balanced')
clf_balanced.fit(X_train, y_train)
```

    C:\Users\Thomas\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight='balanced', dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
y_pred = clf_balanced.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

    metric:-0.007999999999999993
    accuracy:0.5144186046511627, 1-mean_Y:0.9265116279069767, precision:0.08442776735459662, recall:0.569620253164557

What we lost in accuracy we gained in recall, more credit default are being detected by the logistic regression. From now on we will use balanced loss for logistic regression. But the bank is still loosing money, we have to improve our model.

### b) Logistic regression on qualitative variables

We use one-hot encoder to encode class variables so we can use them as well for the logistic regression. We will use OneHotEncoder from sklearn.preprocessing for this task.

```python
from sklearn.preprocessing import OneHotEncoder
```

#### Demo with marital status

```python
ohe = OneHotEncoder(sparse=False, handle_unknown='ignore')
X = ohe.fit_transform(df[['Marital_Status']])
X
```

    array([[0., 1., 0., 0., 0.],
           [0., 1., 0., 0., 0.],
           [0., 1., 0., 0., 0.],
           ...,
           [1., 0., 0., 0., 0.],
           [0., 1., 0., 0., 0.],
           [0., 1., 0., 0., 0.]])

```python
ohe.get_feature_names()
```

    array(['x0_Divorced', 'x0_Married', 'x0_Separated', 'x0_Single',
           'x0_Widowed'], dtype=object)

```python
clf = LogisticRegression(class_weight='balanced')
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
clf.fit(X_train, y_train)
```

    C:\Users\Thomas\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight='balanced', dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

    metric:-0.017186046511627903
    accuracy:0.7637209302325582, 1-mean_Y:0.9265116279069767, precision:0.09490740740740741, recall:0.25949367088607594

#### We can encode all strings columns

```python
df.dtypes
```

    Id_Customer              int64
    Y                        int64
    Customer_Type           object
    Customer_Open_Date      object
    P_Client                object
    Educational_Level       object
    Marital_Status          object
    Number_Of_Dependant    float64
    Years_At_Residence       int64
    Net_Annual_Income      float64
    Years_At_Business      float64
    Prod_Sub_Category       object
    Prod_Decision_Date      object
    Source                  object
    Type_Of_Residence       object
    Nb_Of_Products           int64
    Prod_Category           object
    Age                    float64
    dtype: object

```python
string_cols = ['Customer_Type', 'P_Client', 'Educational_Level', 'Marital_Status', 'Prod_Sub_Category', 'Source', 'Type_Of_Residence', 'Prod_Category']
X = ohe.fit_transform(df[string_cols])
X.shape
```

    (5374, 36)

```python
clf = LogisticRegression(class_weight='balanced')
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
clf.fit(X_train, y_train)
```

    C:\Users\Thomas\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight='balanced', dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

    metric:-0.004860465116279064
    accuracy:0.7893023255813953, 1-mean_Y:0.9265116279069767, precision:0.15456674473067916, recall:0.4177215189873418

### c) Logistic regression on all variables

We can concatenate the encoding of class variables to quantitative variables

```python
X_quantitative = df.loc[:,quant_cols]
X_class = ohe.fit_transform(df[string_cols])
print(X_quantitative.shape, X_class.shape)
X = np.concatenate((X_quantitative,X_class), axis=1)
print(X.shape)
```

    (5374, 6) (5374, 36)
    (5374, 42)

```python
clf = LogisticRegression(class_weight='balanced')
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
clf.fit(X_train, y_train)
```

    C:\Users\Thomas\Anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight='balanced', dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='warn', n_jobs=None, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

    metric:-0.003534883720930221
    accuracy:0.6920930232558139, 1-mean_Y:0.9265116279069767, precision:0.12048192771084337, recall:0.5063291139240507

### c-bis) Adjusting the threshold for the decision to achieve optimal metric

We can plot the ROC curve thanks to adjusments made to the threshold for decision. Indeed logistic regression gives a number between 0 and 1 representing the probability of the sample beeing positive. We can then play on the decision boundary to increase our True Positive Rate or to increase our True Negative Rate.

```python
from sklearn.metrics import roc_curve
```

```python
y_pred_score = clf.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, y_pred_score)
```

```python
plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve')
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('1 - True Negative Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()
```

Note: $\;\; \text{False Positive Rate} = 1 - \text{True Negative Rate}$

Finally we can look for the best metric across all thresholds (but we select it only using the train set):

```python
y_pred_score = clf.predict_proba(X_train)[:,1]
fpr, tpr, thresholds = roc_curve(y_train, y_pred_score)
```

```python
metric_best = -1
threshold_best = 0
for threshold in thresholds:
    y_pred =  (clf.predict_proba(X_train)[:,1] >= threshold).astype(int)
    metric_score = metric(y_train, y_pred)
    if metric_score >= metric_best:
        metric_best = metric_score
        threshold_best = threshold

print(f'metric_best_train:{metric_best}, threshold_best:{threshold_best}')
```

    metric_best_train:0.01065446650124069, threshold_best:0.43739613642560293

__Conclusion for Logistic Regression__:

The optimal threshold to reject a credit application is about 0.36, which is lower than the default threshold of 0.5. Indeed, as stated in the beginning, misclassiying a credit default is more costly compared to misclasfying a credit that will be repaid. Our best logistic regression algorithm has the following metrics scores.

```python
y_pred = (clf.predict_proba(X_test)[:,1] >= threshold_best).astype(int)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')
```

    metric:0.0035581395348837225
    accuracy:0.5246511627906977, 1-mean_Y:0.9265116279069767, precision:0.10511882998171847, recall:0.7278481012658228

As we can notice, an high recall is necessary (despite a low precision) to maximise the bank's profits. This comes with a fairly low accuracy because the bank has to be very cautious of credit default. In a further study, we could vary the interest rate $i$, which was set to $5\%$ here, and observe how the metrics for the best algorithm would repond to it.

 As $i$ increases, we would probably see the recall going down and the accuracy and the precision go up, as paid-back credits become more lucrative.

### d) Tree classification

```python
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from xgboost import plot_importance
import matplotlib.pyplot as plt

```

We first compute a tree with general parameters then print the importance of features

```python
X = np.concatenate((X_quantitative,X_class), axis=1)
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=42)
y_train = y_train.to_numpy()
y_test = y_test.to_numpy()

model = XGBClassifier(learning_rate = 0.05, n_estimators=300, max_depth=8)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')

# plot feature importance
plot_importance(model)
#to_graphviz(model)
plt.show()
```

The quantitave figures (f0 to f5) are the most important features, the age is overall the most important one. The important disparity among the weights of the features entices us to elastic net penalisation $J(\beta)=(1-\alpha) || \beta ||^{2} +\alpha || \beta||_{1}$ to get rid of not important features.

```python
X = df.loc[:,quant_cols]
X = X.loc[:,['Age','Net_Annual_Income','Years_At_Residence','Years_At_Business']]

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=42)
y_train = y_train.to_numpy()
y_test = y_test.to_numpy()

model = XGBClassifier(learning_rate = 0.05, n_estimators=300, max_depth=10, reg_alpha=0.5)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}')

# plot feature importance
plot_importance(model)

plt.show()
```

If we exclude the threshold for the classification we have 4 parameters to optimize:

the learning rate $lr \in [0.01, 0.09]$,

the number of estimators $  \in [\![500;1500]\!]  $,

the coefficient $\alpha$ for the L1 regularisation in the elastic net $ \in ]0,1[$,

the depth of the tree $ \in [\![4;8]\!]   $

The vast state space to explore prevent us to use exhaustive search of the best hyperparameters, we use stochastic research instead refering  to the paper  [Random Search for Hyper-Parameter Optimization]( https://pdfs.semanticscholar.org/8e28/30fb6ba9ea201d4ec5d2d800ab9a9bf9677c.pdf)

```python
ar=np.arange(0.01,0.09,0.0005)

X = np.concatenate((X_quantitative,X_class), axis=1)
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
y_train = y_train.to_numpy()
y_test = y_test.to_numpy()

scale_pos_weight=y_train.size/y_train.sum()

m_lr = -1
m_n_estimators = -1
m_max_depth = -1
m_L1 = -1
max_metric=0

thresholds_tree = np.arange(10**(-4),10**(-2),10**(-5))
print('done')
n_step=50
for j in range(n_step):

    #paramètres
    lr = np.random.choice(ar)
    n_estim = np.random.randint(500,1500)
    m_depth = np.random.randint(4,8)
    L1 = np.random.choice([0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9])

    model = XGBClassifier(learning_rate = lr, n_estimators=n_estim, max_depth=m_depth, reg_alpha=L1,scale_pos_weight=scale_pos_weight,objective='multi:softprob',num_class=2)
    model.fit(X_train, y_train)

    y_pred = model.predict_proba(X_test)

    metric_best = -1
    threshold_best = 0
    for threshold in thresholds_tree:

        y_pred_tuning =  (y_pred[:,1] >= threshold).astype(int)
        metric_score = metric(y_test, y_pred_tuning)

        if metric_score >= metric_best:
            metric_best = metric_score
            threshold_best = threshold

    print("\r {:.2%} loading...".format(j/n_step), end="")
    if max_metric < metric_best:
        max_metric=metric_best
        m_lr = lr
        m_n_estimators =  n_estim
        m_max_depth = m_depth
        m_L1 = L1
        print(j)
        print(f'lr:{m_lr}, n_estimators:{m_n_estimators}, max_depth:{m_max_depth}, reg L1:{m_L1}, metric:{max_metric},threshold:{threshold_best}')

```

    done
     0.00% loading...0
    lr:0.022500000000000013, n_estimators:1495, max_depth:5, reg L1:0.9, metric:0.00448837209302326,threshold:0.00677
     2.00% loading...1
    lr:0.04050000000000003, n_estimators:1057, max_depth:6, reg L1:0.1, metric:0.00595348837209303,threshold:0.0047799999999999995
     30.00% loading...15
    lr:0.02100000000000001, n_estimators:1212, max_depth:7, reg L1:0.2, metric:0.006558139534883725,threshold:0.00509
     98.00% loading...

If we test the select parameters on the test set we obtain:

```python
print(f'lr:{m_lr}, n_estimators:{m_n_estimators}, max_depth:{m_max_depth}, reg L1:{m_L1}, metric on the train set:{max_metric},threshold:{threshold_best}')
model = XGBClassifier(learning_rate = m_lr, n_estimators=m_n_estimators, max_depth=m_max_depth, reg_alpha=m_L1,scale_pos_weight=100,objective='multi:softprob',num_class=2)
model.fit(X_train, y_train)

y_pred = model.predict_proba(X_test)
y_pred = (y_pred[:,1]>= threshold_best).astype(int)

metric_final = metric(y_test, y_pred)
print(f'metric on the test set:{metric_final}')

accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(Y)}, precision:{precision}, recall:{recall}')
```

    lr:0.02100000000000001, n_estimators:1212, max_depth:7, reg L1:0.2, metric on the train set:0.006558139534883725,threshold:0.00584
    metric on the test set:0.004837209302325576
    accuracy:0.37348837209302327, 1-mean_Y:0.9270561965016747, precision:0.09252912954078135, recall:0.8544303797468354

As in the logistic regression the chosen metric has for consequence to give a high importance to the recall at the expense of the precision

### Deep Learning

```python
import tensorflow as tf
from tensorflow import keras
```

We build our neural network using the sigmoid function for activation and  dense layers with dropout to reduce overfiting, we also use a normalization layer to have a continuum of values between zero and one and thus easily separate the results with a threshold.

```python
model_deep = keras.Sequential([
    keras.layers.Dense(128, activation='sigmoid'),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(128, activation='sigmoid'),
    keras.layers.Dropout(0.5),
    keras.layers.BatchNormalization(),
    keras.layers.Dense(2, activation= 'sigmoid')
])

```

```python
model_deep.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy',])

```

```python
model_deep.fit(X_train,y_train, epochs=30)
```

    Train on 3224 samples
    Epoch 1/30
    3224/3224 [==============================] - 1s 332us/sample - loss: 0.6388 - accuracy: 0.6616
    Epoch 2/30
    3224/3224 [==============================] - 0s 97us/sample - loss: 0.4905 - accuracy: 0.8824
    Epoch 3/30
    3224/3224 [==============================] - 0s 99us/sample - loss: 0.4175 - accuracy: 0.9225
    Epoch 4/30
    3224/3224 [==============================] - 0s 100us/sample - loss: 0.3966 - accuracy: 0.9271
    Epoch 5/30
    3224/3224 [==============================] - 0s 97us/sample - loss: 0.3918 - accuracy: 0.9274
    Epoch 6/30
    3224/3224 [==============================] - 0s 97us/sample - loss: 0.3899 - accuracy: 0.9274
    Epoch 7/30
    3224/3224 [==============================] - 0s 100us/sample - loss: 0.3884 - accuracy: 0.9274
    Epoch 8/30
    3224/3224 [==============================] - 0s 97us/sample - loss: 0.3874 - accuracy: 0.9274
    Epoch 9/30
    3224/3224 [==============================] - 0s 99us/sample - loss: 0.3873 - accuracy: 0.9274
    Epoch 10/30
    3224/3224 [==============================] - 0s 99us/sample - loss: 0.3869 - accuracy: 0.9274
    Epoch 11/30
    3224/3224 [==============================] - 0s 96us/sample - loss: 0.3866 - accuracy: 0.9274
    Epoch 12/30
    3224/3224 [==============================] - 0s 97us/sample - loss: 0.3864 - accuracy: 0.9274
    Epoch 13/30
    3224/3224 [==============================] - 0s 113us/sample - loss: 0.3864 - accuracy: 0.9274
    Epoch 14/30
    3224/3224 [==============================] - 0s 112us/sample - loss: 0.3863 - accuracy: 0.9274
    Epoch 15/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3862 - accuracy: 0.9274
    Epoch 16/30
    3224/3224 [==============================] - 0s 94us/sample - loss: 0.3862 - accuracy: 0.9274
    Epoch 17/30
    3224/3224 [==============================] - 0s 96us/sample - loss: 0.3861 - accuracy: 0.9274
    Epoch 18/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3861 - accuracy: 0.9274
    Epoch 19/30
    3224/3224 [==============================] - 0s 94us/sample - loss: 0.3861 - accuracy: 0.9274
    Epoch 20/30
    3224/3224 [==============================] - 0s 96us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 21/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 22/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 23/30
    3224/3224 [==============================] - 0s 96us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 24/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 25/30
    3224/3224 [==============================] - 0s 95us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 26/30
    3224/3224 [==============================] - 0s 118us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 27/30
    3224/3224 [==============================] - 0s 122us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 28/30
    3224/3224 [==============================] - 0s 111us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 29/30
    3224/3224 [==============================] - 0s 117us/sample - loss: 0.3860 - accuracy: 0.9274
    Epoch 30/30
    3224/3224 [==============================] - 0s 103us/sample - loss: 0.3859 - accuracy: 0.9274

    <tensorflow.python.keras.callbacks.History at 0x16c7b995fc8>

We then selection the best threshold

```python
y_pred = model_deep.predict(X_train)
y_pred_tuning = y_pred

amax = np.amax(y_pred_tuning[:,1])
amin = np.amin(y_pred_tuning[:,1])
#print(amin)
#print(amax)

thresholds_deep = np.arange(amin,amax,10**(-7))

metric_best = -1
threshold_best = 0

for threshold in thresholds_deep:
    y_pred_tuning = y_pred
    y_pred_tuning =  (y_pred_tuning[:,1] >= threshold).astype(int)
    metric_score = metric(y_train, y_pred_tuning)

    if metric_score >= metric_best:
        metric_best = metric_score
        threshold_best = threshold

print(f'metric on the train set with best_threshold:{metric_best}, threshold_best:{threshold_best}')
```

    metric on the train set with best_threshold:0.0041098014888337475, threshold_best:2.537273865878827e-05

```python
y_pred = model_deep.predict(X_test)
y_pred =  (y_pred[:,1] >= threshold_best).astype(int)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
metric_score = metric(y_test, y_pred)
print(f'metric on the test set:{metric_score}')
print(f'accuracy:{accuracy}, 1-mean_Y:{1-np.average(y_test)}, precision:{precision}, recall:{recall}, threshold_best:{threshold_best}')
```

    metric on the test set:0.0024883720930232583
    accuracy:0.22930232558139535, 1-mean_Y:0.9265116279069767, precision:0.08151870463428253, recall:0.9240506329113924, threshold_best:2.537273865878827e-05

Once again the recall is really high

## 7) Conclusion

After preprocessing the data we tried various models and evaluated how each one performed on optimising the metric we defined for detecting credit default. After comparing them, we can conclude that our best model is the classification using tree with a metric score of 0.0048 (profit per capita/credit application) on the test set.

Looking at other standard ML metrics for this best model, we have a precision of 0.0925 and a recall of 0.8544 The high recall and low precision highlight the necessity of finding the credit default as they are far more costly to the bank.


# Qiskit Tutorials

Welcome Qiskitters.

These tutorials aim to explain how to use Qiskit. We assume you have installed Qiskit; if not, please look at [qiskit.org](http://www.qiskit.org) or the install [documentation](https://qiskit.org/documentation/install.html).

The focus of these notebooks is not on learning quantum computing.  Instead we will focus on how to use Qiskit, and will go into details only when needed. For those interested in learning about quantum computing we recommend the [Qiskit Textbook](https://qiskit.org/textbook) that we and the community have put together, or the [Qiskit documentation](https://qiskit.org/documentation).

## Circuits

This section gives you the tools to make your first circuits, execute them, and view the data.

1. [Circuit basics](tutorials/circuits/01_circuit_basics.ipynb) - How to use the Qiskit quantum circuit.

2. [Plotting data in Qiskit](tutorials/circuits/2_plotting_data_in_qiskit.ipynb) -  Illustrates the different ways of plotting data in Qiskit.

3. [Summary of quantum operations](tutorials/circuits/3_summary_of_quantum_operations.ipynb) - List of quantum operations (gates, reset, measurements) in Qiskit Terra

## Advanced Circuits

1. [Advanced circuits](tutorials/circuits_advanced/01_advanced_circuits.ipynb) - Circuit building tools added including registerless declarations, composite gate updates and parameterized circuits.

2. [Operators overview](tutorials/circuits_advanced/02_operators_overview.ipynb) - Gives a summary of the features and uses of the Operator class.

3. [Advanced circuit visualization](tutorials/circuits_advanced/03_advanced_circuit_visualization.ipynb) - Details on drawing your quantum circuits.

4. [Transpiler passes and passmanager](tutorials/circuits_advanced/04_transpiler_passes_and_passmanager.ipynb) - How to use the transpiler passes, passmanger, and extend the transpiler with a new pass.

## Pulse

1. [Building pulse schedules](tutorials/circuits_advanced/06_building_pulse_schedules.ipynb) - Building schedules of pulses.

2. [Pulse Scheduler](tutorials/circuits_advanced/07_pulse_scheduler.ipynb) - Scheduling pulse.

3. [Getting system information](tutorials/circuits_advanced/08_gathering_system_information.ipynb) - Obtaining system information.

4. [Pulse simulation](tutorials/circuits_advanced/09_pulse_simulator_duffing_model.ipynb) - Simulate a Duffing oscillator using the pulse simulator.

##  High-Performance Simulators

To really speed up development of quantum computers, we need better simulators with the ability to model realistic noise processes that occur during computation on actual devices. Qiskit provides a high-performance simulator framework called `Aer` for studying quantum computing algorithms and applications in the noisy intermediate-scale quantum regime.

1. [Simulators](tutorials/simulators/1_aer_provider.ipynb) - Gives a summary of the Qiskit Aer provider containing the Qasm, statevector, and unitary simulator.

2. [Device noise simulation](tutorials/simulators/2_device_noise_simulation.ipynb) - Shows how to use the Qiskit Aer noise module to automatically generate a basic noise model for simulating hardware backends.

3. [Building noise models](tutorials/simulators/3_building_noise_models.ipynb) - Shows how to use Qiskit Aer noise module to construct custom noise models for noisy simulations

4. [Custom gate noise](tutorials/simulators/4_custom_gate_noise.ipynb) - Shows to implement simulations using custom noisy gates.

5. [Noise transformations](tutorials/simulators/5_noise_transformation.ipynb) - Noise approximation utility functions to construct approximate Clifford noise models out of a general noise model

6. [Extended stabilizer tutorial](tutorials/simulators/6_extended_stabilizer_tutorial.ipynb) - Gives an overview of the *extended stabilizer* Qasm Simulator method

7. [Matrix Product State simulator](tutorials/simulators/7_matrix_product_state_method.ipynb) - Gives an overview of the *matrix product state* Simulator method

##  Quantum Device Noise Analysis

This includes better characterization of errors, improving gates, and computing in the presence of noise. Qiskit `ignis` is meant for those who want to design quantum error correction codes, or who wish to study ways to characterize errors through methods such as tomography and randomized benchmarking, or even to find a better way for using gates by exploring dynamical decoupling and optimal control.

1. [Hamiltonian and gate characterizations](tutorials/noise/1_hamiltonian_and_gate_characterization.ipynb) - Sequences to measure ZZ rates between qubits and to measure rotation and angle errors in the gates.

2. [Relaxation and decoherence](tutorials/noise/2_relaxation_and_decoherence.ipynb) - How to measure coherence times on the real quantum hardware

3. [Measurement error mitigation](tutorials/noise/3_measurement_error_mitigation.ipynb) - How to peform calibration experiments for measurement errors and fed those calibrations into a "filter" that can be utilized to mitigate errors in subsequent experiments.

4. [Randomized benchmarking](tutorials/noise/4_randomized_benchmarking.ipynb) - Randomized benchmarking (RB) is a technique used to measure the average gate error by measuring the outcomes of random Clifford circuits. This is used internally to report gate errors on our systems.

5. [Quantum volume](tutorials/noise/5_quantum_volume.ipynb) - How to run quantum volume measurements on the quantum hardware.

6. [Repetition Code](tutorials/noise/6_repetition_code.ipynb) - How to run a simple error correction code, known as the repetition code. This can be used to characterize bit flip errors in the hardware.

7. [Accreditation](tutorials/noise/7_accreditation.ipynb) - protocol devised to characterize the reliability of noisy quantum devices.

```python
import qiskit.tools.jupyter
%qiskit_copyright
```

This code is a part of Qiskit

© Copyright IBM 2017, 2022.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
 of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals.

```python
pip install qiskit
```

    Collecting qiskit
      Using cached qiskit-0.34.2.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.2
      Downloading qiskit_terra-0.19.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)
         |████████████████████████████████| 6.5 MB 3.5 MB/s eta 0:00:01
    Collecting qiskit-aer==0.10.3
      Downloading qiskit_aer-0.10.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
         |████████████████████████████████| 18.0 MB 11.2 MB/s eta 0:00:01
    Collecting qiskit-ibmq-provider==0.18.3
      Using cached qiskit_ibmq_provider-0.18.3-py3-none-any.whl (238 kB)
    Collecting qiskit-ignis==0.7.0
      Using cached qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
    Requirement already satisfied: numpy>=1.16.3 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-aer==0.10.3->qiskit) (1.20.1)
    Requirement already satisfied: scipy>=1.0 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-aer==0.10.3->qiskit) (1.6.2)
    Requirement already satisfied: requests>=2.19 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.25.1)
    Requirement already satisfied: python-dateutil>=2.8.0 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.1)
    Collecting websocket-client>=1.0.1
      Using cached websocket_client-1.3.1-py3-none-any.whl (54 kB)
    Collecting requests-ntlm>=1.1.0
      Using cached requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Requirement already satisfied: urllib3>=1.21.1 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.26.4)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.11.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 19.1 MB/s eta 0:00:01
    Requirement already satisfied: setuptools>=40.1.0 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-ignis==0.7.0->qiskit) (52.0.0.post20210125)
    Requirement already satisfied: sympy>=1.3 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-terra==0.19.2->qiskit) (1.8)
    Collecting python-constraint>=1.4
      Using cached python-constraint-1.4.0.tar.bz2 (18 kB)
    Collecting dill>=0.3
      Using cached dill-0.3.4-py2.py3-none-any.whl (86 kB)
    Requirement already satisfied: psutil>=5 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-terra==0.19.2->qiskit) (5.8.0)
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (929 kB)
         |████████████████████████████████| 929 kB 15.5 MB/s eta 0:00:01
    Requirement already satisfied: ply>=3.10 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from qiskit-terra==0.19.2->qiskit) (3.11)
    Collecting symengine>=0.8
      Downloading symengine-0.9.2-cp38-cp38-manylinux2010_x86_64.whl (37.5 MB)
         |████████████████████████████████| 37.5 MB 20.5 MB/s eta 0:00:01
    Collecting stevedore>=3.0.0
      Using cached stevedore-3.5.0-py3-none-any.whl (49 kB)
    Requirement already satisfied: six>=1.5 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: chardet<5,>=3.0.2 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (4.0.0)
    Requirement already satisfied: idna<3,>=2.5 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2020.12.5)
    Requirement already satisfied: cryptography>=1.3 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (3.4.7)
    Collecting ntlm-auth>=1.0.2
      Using cached ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Requirement already satisfied: cffi>=1.12 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.14.5)
    Requirement already satisfied: pycparser in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.20)
    Collecting pbr!=2.1.0,>=2.0.0
      Using cached pbr-5.8.1-py2.py3-none-any.whl (113 kB)
    Requirement already satisfied: mpmath>=0.19 in /home/dsmishler/anaconda3/lib/python3.8/site-packages (from sympy>=1.3->qiskit-terra==0.19.2->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.34.2-py3-none-any.whl size=11792 sha256=c39badb3b0928def2628a9dac0acc060a9f28936042334e0589bee4834bc4a4c
      Stored in directory: /home/dsmishler/.cache/pip/wheels/fc/65/25/bf94d3c84cabe3eca2af16a7e8fcc0d39a0910e9497245a1e4
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24079 sha256=e782789a0c00a0a6ce83f411d90d1e244450609583627506130e145402c19324
      Stored in directory: /home/dsmishler/.cache/pip/wheels/86/ba/5c/4e9115777de42c6a2e1ca77ef7c9d0d479254c5080341b55c5
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, retworkx, python-constraint, ntlm-auth, dill, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
    Successfully installed dill-0.3.4 ntlm-auth-1.5.0 pbr-5.8.1 python-constraint-1.4.0 qiskit-0.34.2 qiskit-aer-0.10.3 qiskit-ibmq-provider-0.18.3 qiskit-ignis-0.7.0 qiskit-terra-0.19.2 requests-ntlm-1.1.0 retworkx-0.11.0 stevedore-3.5.0 symengine-0.9.2 tweedledum-1.1.1 websocket-client-1.3.1
    Note: you may need to restart the kernel to use updated packages.

```python

```


## Formal Grammar Example: Barsalou Perceptual Symbol Grammar

To demonstrate the formal grammar and language framework I developed, I created a formal grammar that implements a weird version of Barsalou's Percetpual Symbol Systems. Each sentence is a "simulation", with each simulation having exactly one "simulator", which in turn could have many categories and instances that make it up. Not that that's exactly what he was saying, but it's a fun attempt to formalize his system from his 1999 BBS paper.

```python
from IPython.display import display, Latex

from grammar import Grammar, ProductionRule

# build a hierarchy of production rules
production_rules = [
    ProductionRule('simulation', ['simulator', 'category', 'instance'], n_outputs='all'),
        ProductionRule('simulator', ['driving', 'flying', 'seeing red', 'seeing blue', 'seeing green'], n_outputs='one'),
        ProductionRule('category', ['car', 'airplane', 'bird', 'grass', 'bed'], n_outputs='many'),
        ProductionRule('instance', ['2004 pontiac grand am', '2016 subaru crosstrack', '2010 chevy silverado',
                                    'Boeing 747', 'queen bed with yellow sheets', 'cardinal', 'sky',
                                    'cheatgrass', 'kentucky bluegrass'], n_outputs='many')
]

# instantiate the grammar based on the rules
perceptual_symbol_grammar = Grammar(production_rules)
# use the print_latex() method of the Grammar class to print the formal grammar
display(Latex(perceptual_symbol_grammar.print_latex()))
```

\begin{equation}
\begin{array}{ll}
    ( & \\
       & S = \{\textrm{ simulation }\}, \\
       & N = \{ \textrm{category}, \textrm{simulation}, \textrm{instance}, \textrm{simulator} \}, \\
       & \Sigma = \{ \textrm{queen bed with yellow sheets}, \textrm{2016 subaru crosstrack}, \textrm{car}, \textrm{seeing blue}, \textrm{cheatgrass}, \textrm{2004 pontiac grand am}, \textrm{flying}, \textrm{seeing green}, \textrm{bed}, \textrm{Boeing 747}, \textrm{driving}, \textrm{seeing red}, \textrm{grass}, \textrm{2010 chevy silverado}, \textrm{sky}, \textrm{cardinal}, \textrm{bird}, \textrm{airplane}, \textrm{kentucky bluegrass} \}, \\
       &P = \{ \\
       & \begin{array}{ll}
            & simulation\rightarrow\textrm{simulator}~|~\textrm{category}~|~\textrm{instance} \\
& simulator\rightarrow\textrm{driving}~|~\textrm{flying}~|~\textrm{seeing red}~|~\textrm{seeing blue}~|~\textrm{seeing green} \\
& category\rightarrow\textrm{car}~|~\textrm{airplane}~|~\textrm{bird}~|~\textrm{grass}~|~\textrm{bed} \\
& instance\rightarrow\textrm{2004 pontiac grand am}~|~\textrm{2016 subaru crosstrack}~|~\textrm{2010 chevy silverado}~|~\textrm{Boeing 747}~|~\textrm{queen bed with yellow sheets}~|~\textrm{cardinal}~|~\textrm{sky}~|~\textrm{cheatgrass}~|~\textrm{kentucky bluegrass} \\
       \end{array} \\
     & \} \\

    ) &
\end{array}
\end{equation}

```python
# now let's create a language that will produce sentences based on the grammar
from grammar import Language

l = Language(perceptual_symbol_grammar)

prods = l.productions()
```

```python
# You can run this as many times as you like and get a new production
for i in range(10):
    p = next(prods)
    print(next(prods) + '\n')
```

    seeing green bird car airplane grass 2010 chevy silverado cheatgrass sky Boeing 747

    flying airplane 2004 pontiac grand am Boeing 747 2010 chevy silverado queen bed with yellow sheets kentucky bluegrass cheatgrass sky 2016 subaru crosstrack

    seeing green grass airplane bird car bed sky cardinal 2016 subaru crosstrack kentucky bluegrass cheatgrass 2004 pontiac grand am Boeing 747 2010 chevy silverado queen bed with yellow sheets

    driving car bird bed airplane grass 2010 chevy silverado Boeing 747 2016 subaru crosstrack

    seeing red bird car kentucky bluegrass 2010 chevy silverado Boeing 747 cardinal cheatgrass sky 2016 subaru crosstrack

    seeing blue airplane car bed bird queen bed with yellow sheets 2016 subaru crosstrack Boeing 747

    seeing red bird airplane 2010 chevy silverado

    flying car 2010 chevy silverado queen bed with yellow sheets kentucky bluegrass cheatgrass cardinal 2004 pontiac grand am sky 2016 subaru crosstrack Boeing 747

    driving bird car queen bed with yellow sheets Boeing 747 sky 2004 pontiac grand am 2016 subaru crosstrack cardinal 2010 chevy silverado

    seeing blue airplane grass Boeing 747 2010 chevy silverado queen bed with yellow sheets 2016 subaru crosstrack


\title{Some extensions for Jupyter/IPython notebook}
\author{J.-F Bercher \par \par Professor at ESIEE Paris}
\maketitle

**Foreword -- **

This notebook was used during a talk at [PyParis 2017](http://pyparis.org/). In order to fully reproduce what was done there,

1. you should have a working installation of the  [jupyter_contrib](https://github.com/ipython-contrib/jupyter_contrib_nbextensions) notebook extensions. See [here](https://github.com/ipython-contrib/jupyter_contrib_nbextensions) for installation instructions. Using pip, it should be as easy as
```
pip install https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/master
```
and then
```
jupyter contrib nbextension install --user
```
(or `--system` to install system wide). The, you should enable at least `highlighter`, `latex_envs`

2. You need to have a copy of the following repo and a demo nbextension `toggleCase` installed: do this by
    - install the demo nbextension by
    ```
    jupyter nbextension install https://rawgit.com/jfbercher/small_nbextensions/master/toggleCase.zip --user
    ```
    - download https://github.com/jfbercher/PyParis2017/raw/master/JupNbextensions.zip, unzip, change to PyParis2017 directory and \underline{run jupyter notebook from that directory} (this preserve links) -- alternatively, you can just git clone this repo.

3. Displaying correcly the present notebook needs the [jupyter_latex_envs](https://pypi.python.org/pypi/jupyter_latex_envs) nbextension to be installed and enabled; is it not a strong requirement though.

- Unfortunately, during the talk, the demo for `python-markdown` failed. This was due to the fact that this extension needs the notebook to be *trusted* (by pressing a menu button on the upper right)

```javascript
%%javascript
// Load useful extensions
Jupyter.utils.load_extensions('highlighter/highlighter')
Jupyter.utils.load_extensions('latex_envs/latex_envs')
```

    <IPython.core.display.Javascript object>

# Objective and agenda

- Present and describe some Jupyter/IPython notebook extensions by the author and others of the [jupyter_contrib_nbextensions](https://github.com/ipython-contrib/jupyter_contrib_nbextensions) group which are useful for

    - core development and data analysis,
    - authoring; publication of research reports and papers,
    - education/classroom animation.
- [*jupyter_contrib_nbextensions is also a forum to experiment new ideas and have fun*]
- Give some hints on ways to develop an nbextension
- call for ideas and contributors for the `jupyter_contrib_nbextensions` group

\begin{comment}

- open source: many contributors to find in the commit history
- will preferentially describe extensions I have initiated or contributed to, but also other ones
- even if I say "I" or "my", there are often other contribs (code by others/suggestions/patches,etc )
\end{comment}

\begin{comment}

- Wonderful introduction by Sylvain Corlay

    - --> be involved in open source and start small! Almost only small things here
    - --> Almost everybody have alreay used Jupyter... don't need to advocate

\end{comment}

# A small recall: what is the Jupyter notebook

(formerly the IPython notebook)

- a browser application that allows to create and share live documents which embed text, code and multimedia.
- use as an IDE with integrated documentation capabilities,
- for reproductible research
- as an authoring system,
- demos or full experimentations for teaching.

In these three areas (and more), the basic capabilities of the notebook can be extended through notebook extensions.

## Markdown cells
Can mix [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet), with
- *emphasis*,
- **strong emphasis**,
- lists
- support for images, $\LaTeX$ equations, e.g.
\begin{equation}
\sum_{n=0}^{N-1} x[n] e^{j2\pi f n}
\end{equation}
etc...

## Code cells

```python
import numpy as np
def random_letters(size=1, maxletter=52):
    import string
    return [string.ascii_letters[k]
            for k in np.random.randint( 0, maxletter, size=size)]

z = random_letters(12, 26)
print(z)
```

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
plt.plot(np.random.randn(256))
```

\begin{textboxa}The basic capabilities of the notebook can be extended through notebook extensions. These can modify the UI or behavior of the notebook, such as adding buttons to your toolbar, new keyboard shortcuts, etc. They can also add new functionalities.
\end{textboxa}

# Example of a small toy extension

*from [https://mindtrove.info/4-ways-to-extend-jupyter-notebook/#nb-extensions](https://mindtrove.info/4-ways-to-extend-jupyter-notebook/#nb-extensions)*

Jupyter Notebook extensions (nbextensions) are javaScript modules.

To write a new extension, you must implement your logic in a JavaScript file conforming to the **AMD specification** so that Jupyter can load it using RequireJS. You should define and **export a load_ipython_extension function** in your module so that Jupyter can invoke it after initializing its own components.

Within that function, you are free to
- manipulate the DOM of the page,
- invoke Jupyter JavaScript APIs,
- listen for Jupyter events,
- load other modules, and so on.

## First extension

The extension features a menu button and a keyboard shortcut. It displays an alert dialog. You can test it with the "T" button on the toolbar.

[Source: demo1.js](http://localhost:8888/nbextensions/toggleCase/demo1.js)

The line below loads the extension we devised.
Since the javascript is stored silently in a `<script>` tag in the output. The extension will be reloaded automatically when reloading the notebook.

To avoid this, you will need to clear the output (or all outputs) of the relevant cells before reloading, by the `Cell/Curent Output/Clear` menu; then save and reload.

```javascript
%%javascript
Jupyter.utils.load_extensions('toggleCase/demo1')
```

    <IPython.core.display.Javascript object>

## Organizing

- button and shorcut definitions can be devoted to separated functions. A initialisation step can be added.

[Source: demo2.js](http://localhost:8888/nbextensions/toggleCase/demo2.js)

```javascript
%%javascript
Jupyter.utils.load_extensions('toggleCase/demo2')
```

    <IPython.core.display.Javascript object>

## Something to do

- A function is added to do something with the extension. When a text is selected and the button pushed, or the shortcut pressed, then the case of that text is toogled.

[Source: demo3.js](http://localhost:8888/nbextensions/toggleCase/demo3.js)

```javascript
%%javascript
Jupyter.utils.load_extensions('toggleCase/demo3')
```

    <IPython.core.display.Javascript object>

## Loading a configuration

- It is possible to store parameters in system, and even *edit* them.
- The great utility here is the [nbextensions_configurator](https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator) by @jcb91  [Application](http://localhost:8888/nbextensions/) -- This is a *server extension*.
     - the parameters are specified via a *yaml* description file
     - they can be inspected, selected, edited via a UI - Here, we edit the keyboard shortcut

[Source: demo4.js](http://localhost:8888/nbextensions/toggleCase/demo4.js)

[demo4.yaml](https://raw.githubusercontent.com/jfbercher/small_nbextensions/master/toggleCase/toggleCase.yaml)

```javascript
%%javascript
Jupyter.utils.load_extensions('toggleCase/demo4')
```

## Ensuring correct loading by waiting for the notebook or kernel to be fully available.

Indeed, it is possible that the extension is loaded **before** the notebook is fully loaded.
But sometimes the extension needs the notebook to be fully ready, or the kernel to be started.

- This check is not necessary here, though, but an example is provided.

Also a css can be loaded in the notebook with styles added for the nbextension.

[Source: toggleCase](http://localhost:8888/nbextensions/toggleCase/toggleCase.js)

```javascript
%%javascript
Jupyter.utils.load_extensions('toggleCase/toggleCase')
```

    <IPython.core.display.Javascript object>

## Summary
So, we see that the structure of a nbextension is

DIRECTORY _____  main.js               // main script
            |__  extension_name.yaml   // extension description and parameters (for @jcb91's nbextension configurator)
            |__  main.css              // optional css
            |__  other.js              // other js file that can be loaded by the main one

**Loading (one shot/one notebook)** : `Jupyter.utils.load_extensions('DIRECTORY/main.js')`

**Loading (permanently/any notebook)** : `jupyter nbextension enable DIRECTORY/main.js`

**Loading via the nbextension_configurator**: use the UI in `Edit/nbextensions Config` or directly at [http://localhost:8888/nbextensions](http://localhost:8888/nbextensions) (adapt the port if your server runs on a different port, maybe on a different address if you use JupyterHub)

# A tour of some nbextensions

We will have a tour of some nbextensions, pointing out
- history, contributors
- new functionalities
- some implementation details (principles and Jupyter API)
- specific highlights

1. classroom
2. usability/authoring
3. coding

## Classroom/Education

  - classroom e.g: [exercise](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/exercise/readme.html), [exercise2](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/exercise2/readme.html) (include exercises with hidden solutions), [highlighter](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/highlighter/readme.html) (select, highlight, annotate text), [snippets menu](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/snippets_menu/readme.html)..

### exercise and exercise2

- Exercise is an extension by @juhasch Juergen Hasch
- Exercise 2 is a fork by @jfbercher with a different layout -- but the underlying logic is the same

demo [here](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/exercise/readme.md)
or
[here](http://localhost:8888/notebooks/Example%20exercise2.ipynb)

**Internals** exercise and exercise2 add respectively a solution and solution2 metadata to solution cells, with for value the current state hidden/shown of the solution.
- For exercise, a div with the plus/minus character is prepended to the solution heading cell.
- For exercise2, a flex-wrap style is added to the solution heading cell and a checkbox widget, with some css styling, is appended to the cell.

☞ At startup, a loop over all cells tests if these metadata are present and restore the widgets, and the visibility to  the last recorded state.
[source](http://localhost:8888/nbextensions/exercise/main.js)

☞ Jupyter API: `cell = IPython.notebook.get_selected_cell();` `IPython.notebook.select_next();`
The cell object has a metadata attribute (another object) --> `cell.metadata.solution = shown`

### Highlighter

**Use case:** With IPython notebooks distributed to students, in class, let them highlight what they feel important.

demo [here](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/highlighter/README.md)

[Source](http://localhost:8888/nbextensions/highlighter/highlighter.js)

**Internals**

☞ API: `var selectedText = window.getSelection().toString();`
`cell_text = cell_text.replace(identifiedText,highlight(identifiedText,scheme))`
(command mode)

☞ Jupyter/codemirror API: `var cm = cell.code_mirror;  var selectedText = cm.getSelection()`
`cm.replaceSelection(highlight(selectedText,scheme))`
(edit mode)

☞ A fuzzy search is implemented to align the rendered text (command mode) to the underlying source text

**Highlights can be exported to html/LaTeX**

- see [this documentation](http://localhost:8888/nbextensions/highlighter/export_highlights.html)
- Conversion is achieved using `nbconvert`. This app. supports pre and postprocessing.

The pre-post script is [here](http://localhost:8888/files/pp_highlighter.py).

The key for conversion resides in subclassing
```
preprocess_cell(self, cell, resources, index)
```
and `postprocess(self, input)`

### Snippets menu

That's a great extension by @moble (with contribs from @jcb91)

[documentation](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/snippets_menu/readme.md) and demo

## Usability/authoring

 - notebook usage and publication: [toc2](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/toc2/README.html) (table of contents/navigation menu),  [jupyter_highlight_selected_word](https://github.com/jcb91/jupyter_highlight_selected_word),[collapsible headings](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html), [python markdown](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/python-markdown/readme.html),  [nbTranslate](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/nbTranslate/README.html) (multilanguage support), [latex_envs](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/latex_envs/README.html) (use LaTeX environments within the notebook -- and export them)

### The toc2 extension

This is a fork of  @minrk's toc extension, with many additional capabilities. It is a quite big extension (essentially) maintained by @jfbercher.

[documentation](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/toc2/README.md) and demo

Sources :
- [main](http://localhost:8888/nbextensions/toc2/main.js);
- [toc2](http://localhost:8888/nbextensions/toc2/toc2.js)

**Internals**

Many things, but let us point out

- Configuration is stored in part in system (`notebook.json`) and in each edited notebooks. Thus config can be modified per notebook and states stored (and restored)
- Additional css is generated on the fly from user parameters (as entered in configurator UI, eg colors)
- Structure for waiting full availabilty of the notebook before running extension code.

**Exporting** the table of contents

Since the toc is generated and updated on the fly, with many options and functionalities, the idea was to
- *embed in the html output the very same javascript that is used in the liveNotebook*
   - [jinja2 template](http://localhost:8888/files/toc2.tpl)
   - [exporter](http://localhost:8888/files/toc2.py) (python file) --> only set the template name
   - entry point set in [setup.py](http://localhost:8888/files/setup.py) recognized by `nbconvert`, as in
   ```
     'nbconvert.exporters': [
                'html_toc = jupyter_contrib_nbextensions.nbconvert_support.toc2:TocExporter',  # noqa: E501
   ...
    ```

Export can be done as follows:

```jupyter nbconvert FILE.ipynb --template toc2```

An exporter is also available. It is possible to export to html with toc by

```jupyter nbconvert --to html_toc FILE.ipynb```

If you also use latex_envs, you can embed both functionalities while exporting with

```jupyter nbconvert --to html_with_toclenvs FILE.ipynb```

See the example just below:

```bash
%%bash
jupyter nbconvert --to html_toc JupExtensions_talk.ipynb
xdg-open JupExtensions_talk.html
```

### Python markdown

An extension by @juhasch which enables to include variable values in the markdown code.

**!** -- **Needs the notebook to be *trusted***

```python
import numpy as np
np.random.seed(55)
a = np.random.randn(3)
a
```

    array([-1.62373111, -0.10178393, -1.8097911 ])

```javascript
%%javascript
Jupyter.utils.load_extensions('python-markdown/main')
```

```python
x=3
```

`variable x is {{ x }}`
variable x is {{ x }}

This is a markdown **text cell** with the value of a variable, {{a[1]}} inserted in it.

### latex_envs

The `latex_envs` extension enables

- to use many familiar $\LaTeX$ structures in the notebook;
- export all that (in $\LaTeX$ and html)
- and enables to copy and paste from $\LaTeX$ documents,

[documentation](http://localhost:8888/notebooks/JFB/JupNbextensions_talk/latex_env_doc.ipynb) and demo.

**Internals/Highlights**

Many things, but let us point out

- `latex_envs` menu
- toolbar
- read user configuration/extension from json files
- store configuration in notebook's metadata

**Exporting**

- template `latex_envs.tpl`
- an entry point

```
jupyter nbconvert --to html_with_lenvs FILE.ipynb
```
It should be noted that the rendering is done exactly in the same way as in the livenotebook. Actually, it is the very same javascript which is run in the html file, as for the `toc2` nbextension.

[The exporter](http://localhost:8888/edit/latex_envs.py)

\begin{theorem}\label{theo:one}
This is a first theorem
\end{theorem}

\begin{theorem}\label{theo:two}
And a second one
\end{theorem}

\begin{exercise}\label{exo:exo1}
And an exercise
\end{exercise}

With a reference to theo:one \ref{theo:one} and another to theo:two \ref{theo:two}

### nbTranslate

[documentation](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/nbTranslate/README.md) and demo

Source:

- [main](http://localhost:8888/nbextensions/nbTranslate/main.js);
- [nbTranslate](http://localhost:8888/nbextensions/nbTranslate/nbTranslate.js)

```javascript
%%javascript
Jupyter.utils.load_extensions('nbTranslate/main')
```

    <IPython.core.display.Javascript object>

A small text with an equation in it, an why not, a
definition
\begin{definition}
Something on $x_k$ for $k>0$
\end{definition}

Mala tekst jednadžba u njemu, što je zašto ne, a
definicija
\begin{definition}
Nešto o $x_k$ za $k>0$
\end{definition}

Un petit texte avec une équation, et pourquoi pas, un
définition
\begin{definition}
Quelque chose sur $x_k$ pour $k>0$
\end{definition}

**Internals/Highlights**

Many things, but let us point out

- `Langs` menu
- configuration toolbar
- Request to a web service (google translate)
- configuration stored in notebook's metadata (langs in notebook)

**Exporting**

It is possible to extract one language from the multilanguage notebook.
An exporter with an entry-point selectLanguage is provided that converts the notebook into another one as follows
```
jupyter nbconvert --to selectLanguage --NotebookLangExporter.language=lang  FILE.ipynb
```

### Some other nice extensions

By @jcb91

- [jupyter_highlight_selected_word](https://github.com/jcb91/jupyter_highlight_selected_word),
- [collapsible headings](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/collapsible_headings/readme.html),

## Code developpement

- code developpement: [code_prettify](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/code_prettify/README_code_prettify.html) (code formatter via yapf or auto PEP8), [2to3](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/code_prettify/README_2to3.html)  (python converter),
[hinterland](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/hinterland/README.html),
[runtools](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/runtools/readme.html), [codefolding](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/codefolding/readme.html), [varinspector](http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/varInspector/README.html) (variable inspector),

### Code_prettify / 2to3 / autopep8

**A bit of history**

Originally, a request was sent on `jupyter_contrib_nbextensions` for using YAPF for Python code formatting.
This was done using a simple strategy [yapf_ext](http://localhost:8888/files/README_yapf_ext.md) / [source](http://localhost:8888/files/yapf_ext.js)

- read the cell content
- transform it using a call to the python kernel

This was rapidly extended to recognize the running kernel and reformat accordingly (using ` formatR` and `jsbeautify`) under the name [code_prettify](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/code_prettify/README_code_prettify.md).

Another request on `jupyter_contrib_nbextensions` was sent for a 2to3 conversion. The architecture for the extension is the same. With @jcb91, we developped a general library for that: [KerneExecOnCells library](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/code_prettify/README.md). A [2to3](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/code_prettify/README_2to3.md) plugin was defined, as well as a [autopep8](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/code_prettify/README_autopep8.md) reformatter plugin.

[Source for code_prettify as a plugin](http://localhost:8888/nbextensions/code_prettify/code_prettify.js)

### Some other nice extensions

- [hinterland](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/hinterland/README.md) by @jcb91
- [runtools](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/runtools/readme.md) by @juhasch
- [codefolding](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/codefolding/readme.md),
- [codefolding](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/codefolding/readme.md)
- [ExecuteTime](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/execute_time/readme.md)

```python
# demo for hinterland

```

### Variable inspector

A request on `jupyter_contrib_nbextensions` asked for a variable inspector like in some commercial packages as well as in RStudio. A small googling provided [some hints](https://github.com/jupyter-widgets/ipywidgets/blob/ffa094e061c899292036049b00ff93e46e8b4691/docs/source/examples/Variable%20Inspector.ipynb) from ipywidgets.

Then the extension was built using lot of code from toc2 and the same approach as in `code-prettify`, that is call the kernel with some provided code and catch the result.

[varInspector documentation](http://localhost:8888/nbextensions/nbextensions_configurator/rendermd/nbextensions/varInspector/README.md)

[source code](http://localhost:8888/nbextensions/varInspector/main.js);

[kernel code](http://localhost:8888/nbextensions/varInspector/var_list.py);

**events**
```
 // event: on cell execution, update the list of variables
            events.on('execute.CodeCell', varRefresh);
            events.on('varRefresh', varRefresh);

 // on variable deletion
 Jupyter.notebook.events.trigger('varRefresh')
```

```python
w = "A long string "
```

# Miscellaneous

- **Building a nbextension package** Actually a python package, embedding the javascript files, the templates, and defining entry points. See [jupyter_highlight_selected_word](https://github.com/jcb91/jupyter_highlight_selected_word) or [jupyter_latex_envs](https://github.com/jfbercher/jupyter_latex_envs) for examples.
- **Installing the [jupyter_contrib_nbextensions](https://github.com/ipython-contrib/jupyter_contrib_nbextensions)** See instructions on the repo.
- **This document** will be available shortly, say before next week, on  https://github.com/jfbercher