# PC lab: classification

## 1 Introduction

In a binary classification setting, we are interested in assigning an observation 𝐱 to one of two possible classes, denoted by 𝑦. For example, maybe we would like to tell if a patient has a particular disease (y = 1) or not (y = 0), given certain symptoms 𝐱, in other words, we want to model $P(y|\mathbf{x})$, also called the *class posterior* or the *class-membership probability*. Two main approaches exist to model this probability: either directly using *discriminative models* or using *generative models*, which first model the joint probability $P(\mathbf{x},y)$ and obtain the class posterior using Bayes' rule.

### 1.1 Logistic regression

**Logistic regression** (LR) is the most fundamental example of a **discriminative** model. Just like linear regression, LR is a linear model. However, LR does not model the mean of a continuous outcome, but the logarithm of the [odds](https://en.wikipedia.org/wiki/Odds) of the probability $P(X)$:

\begin{equation}
log \frac{P(y|\mathbf{x})}{1-P(y|\mathbf{x})} = w_{0}x_{0} + w_{1}x_{1} + ... + w_{p}x_{p} = \mathbf{w^Tx}
\end{equation}

In practice,  we are really interested in the probability $P(y|\mathbf{x})$ and not in the odds of $P(y|\mathbf{x})$. To obtain this from the previous equation, we apply the *logistic function* (otherwise called a *sigmoid* function):

\begin{equation}
\phi(z) = \frac{1}{1 + e^{-z}} = \frac{e^{z}}{1+e^{z}}
\end{equation}

Applying this function on both sides of the LR equation, we get a model for $P(y|\mathbf{x})$ as follows:

\begin{equation}
P(y|\mathbf{x}) = \phi(\mathbf{w^Tx}) = \frac{1}{1 + e^{-\mathbf{w^Tx}}}
\end{equation}

If we want to classify a data point $\mathbf{x}$, we can calculate the probability of it being class $y$ with the above equation. A final classification is then made by assigning it to the class if the probability $P(y|\mathbf{x})$ exceeds a certain threshold. A typical threshold is 0.5.

 THOUGHT EXERCISE: What would happen to our predictions when we would choose a lower threshold, let's say 0.2? How would this affect the accuracy of our predictions? Can you think of a situation where we would want to do this?

#### 1.1.1 OPTIONAL EXTRA: Training a LR model: cross-entropy loss

Now that we know the logistic regression model to predict the probability of belonging to a certain class, all that remains is the question of how to find the weights of the model on a given set of training data. As always, this is the problem of minimizing a loss function to find an optimal set of weights. Where we used the mean squared error (MSE) for linear regression, we will use the **cross-entropy** loss function for LR. Minimizing the binary cross-entropy loss is equivalent to minimizing the negative log-likelihood of the data under a binomial distribution:

\begin{equation}
l_{log}^{i} = \frac{1}{n}\sum\limits_{i=1}^{n}-y_{i}log(P(y_i|\mathbf{x}_i))-(1-y_i)log(1-P(y_i|\mathbf{x}_i))
\end{equation}

Where $y_i$ is the class of data point $i$ and $p(\mathbf{x}_i)$ is the class-membership probability predicted by logistic regression for the observation $\mathbf{x}_i$. If we look at the cross-entropy loss **for a single data point** $l_{log}^{i}$, we can break it down in two parts:

\begin{equation}
l_{log}^{i} =
\begin{cases}
   -log(P(y_i|\mathbf{x}_i)) & \text{if} \ y_i = 1\\
   -log(1-P(y_i|\mathbf{x}_i)) & \text{if} \ y_i = 0
\end{cases}
\end{equation}

Or even more generally, the cross-entropy can be written as the log probability of predicting the correct class:

\begin{equation}
l_{log}^{i} = -log(P(\mathrm{correct} y_i|\mathbf{x}_i))
\end{equation}

From this, it should be clear that the cross-entropy loss will be larger for smaller values of $P(y_i|\mathbf{x}_i))$ if $y_i = 1$, and vice versa. If we visualize the cross-entropy loss for these two cases:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-white')

p = np.arange(0.01,0.99,0.01) # Generate a range of predicted probabilities between zero and 1
l_0 = -np.log(p)  # cross-entropy loss if y = 1
l_1 = -np.log(1-p)

# Plot them
fig, ax = plt.subplots(figsize=(10,6))
ax.scatter(p,l_0, marker='.')
ax.scatter(p, l_1, marker='.')
ax.set_xlabel('Predicted class-membership probability $P$')
ax.set_ylabel('Cross-entropy loss')
ax.legend(['Cross-entropy loss of a sample when y is 1',
           'Cross-entropy loss of a sample when y is 0'])
plt.show()
```

#### 1.1.2 OPTIONAL EXTRA: Training a LR model: gradient descent

For linear regression, the solutions to the normal equations provide a convenient analytical solution to obtain the optimal set of model weights $\mathbf{w}$ on a set of training data. There is no such solution to find the optimal weights for a logistic regression model, so instead an optimization algorithm such as **gradient descent** is used to train a LR model.

Gradient descent is an iterative optimization algorithm that searches for the optimum of an objective function by making small changes to a set of optimization variables. Gradient descent (and its more complex variants) are widely used in machine learning to find the optimal set of model weights that minimize a certain loss function. Especially when there is no analytical solution for the weights available like for linear regression.

Generally, gradient descent uses the **gradient of the loss function with respect to the model weights** to perform updates to those weights in each iteration. At iteration $k+1$, the algoritm computes the gradient of a loss function $J(\mathbf{w})$ evaluated in the training data. Then, it performs an update to the current parameter values that is relative to the gradient multiplied with the learning rate $\gamma$, which is a constant:

\begin{equation}
\mathbf{w}_{k+1} = \mathbf{w}_{k} - \gamma\nabla{J(\mathbf{w}_{k})}
\end{equation}

Initially, the weights are often initialized with random draws from some distribution. The algorithm continues to do updates, until it converges or until some stopping criterion is reached.

In order to perform gradient descent to find the weights of a logistic regression model, we need to compute the gradient of the loss function with respect to the model parameters. Recall that, for a single data point, the cross-entropy loss function was as follows:

\begin{equation}
l_{log}^{i}(\mathbf{w}) = -y_{i}log(P(y_i|\mathbf{x}_i))-(1-y_i)log(1-P(y_i|\mathbf{x}_i)))
\end{equation}

Where $P(y_i|\mathbf{x}_i))$ is nothing else than the weighted sum of the inputs squashed through the sigmoid function:

\begin{equation}
P(y_i|\mathbf{x}_i)) = \phi(w_{0}x_{0i} + w_{1}x_{1i} + ... + w_{p}x_{pi})
\end{equation}

Before going on, let's first calculate the partial derivative of the sigmoid function:

\begin{equation}
\frac{\partial}{\partial z} \phi(z) = \frac{\partial}{\partial z} \frac{1}{1+e^{-z}} = \frac{e^{-z}}{(1+e^{-z})^2}
\end{equation}

We can rewrite this as follows:

\begin{equation}
\frac{e^{-z}}{(1+e^{-z})^2} = \frac{1 +e^{-z} -1}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}} \Big( 1 - \frac{1}{1+e^{-z}}\Big) = \phi(z)(1 - \phi(z))
\end{equation}

With this result and by applying the chain rule, we can compute the partial derivative of the loss function with respect to the weight $w_j$. We will use the symbol $z$ to denote the weighted sum of the features (i.e., the input for the logistic function) and drop the subscript $i$ for clarity:

\begin{equation}
\frac{\partial l_{log}(\mathbf{w})}{\partial w_j} = \frac{\partial}{\partial w_j} \Big(-ylog(\phi(z))-(1-y)log(1-\phi(z)) \Big) \\ = \Big(  \frac{-y}{\phi(z)} + \frac{1-y}{1-\phi(z)}  \Big)\frac{\partial}{\partial w_j}\phi(z) \\ = \Big(  \frac{-y}{\phi(z)} + \frac{1-y}{1-\phi(z)}  \Big) \phi(z)(1-\phi(z))\frac{\partial}{\partial w_j}z
\end{equation}

Since $z = w_{0}x_{0} + w_{1}x_{1} + ... + w_{p}x_{p}$, $\frac{\partial}{\partial w_j}z$ is nothing more than $x_j$, so we can rewrite the above as:

\begin{equation}
\frac{\partial l_{log}(\mathbf{w})}{\partial w_j} = \Big( -y(1-\phi(z) + (1-y)\phi(z))\Big)x_j \\ = \big(  -y + \phi(z) \big)x_j = \big( \phi(z) - y \big)x_j
\end{equation}

With this partial derivative of the loss w.r.t $w_j$, we can write the update rule of the gradient descent algorithm for the $j^{th}$ weight:

\begin{equation}
w_{j,k+1} = w_{j,k} - \gamma(\phi(z_k)-y)x_{j}
\end{equation}

In other words, the algorithm will each time perform an update to the weight $w_{j}$ that is in proportion to the difference between the predicted probability of class membership in the previous iteration and the actual class. Makes sense! The entire gradient is simply the vector that contains the partial derivatives with respect to the entire weight vector $\mathbf{w}$, and in reality gradient descent acts on $\mathbf{w}$ and not on an individual weight $w_j$. Also, the gradient is typically not calculated for one data point, but evaluated over the entire training data set (unless the training set is prohibitively large, in that case mini-batch gradient descent is used).

In practice, software packages such as [scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegressionscikit-learn) do this optimization under the hood, so there is no need to implement it manually each time we want to use logistic regression.

### 1.2 Linear discriminant analysis

In contrast to the (discriminative) logistic regression model, linear discriminative analysis (LDA) is a **generative** model. Generative models are called as they are because they specify the class-conditional densities $P(\mathbf{x}|y)$, which allows to generate new observations $\mathbf{x}$ for a class $y$: for a given class $y$, we can sample from the distribution $P(\mathbf{x}|y)$.

The main assumption in LDA is that the data in each class is assumed to follow a multivariate Gaussian distribution. Furthermore, the assumption is made that the (co)variance within and between predictor variables is the same for all the classes. As such, only a limited number of parameters have to be estimated from the data to fully specify the class-conditional distributions: in the multivariate case, we need to estimate the class-specific mean vectors $\mathbf{\mu_{k}}$ and the common covariance matrix $\Sigma$. In the case of just one predictor (feature), this simplifies to estimating the mean of $x$ within each class and the variance $\sigma^2$. The priors $p(y=k)$ are simply estimated from the data by calculating the proportion of training instances in each class.

Once the class-conditional densities $P(\mathbf{x}|y)$ are modeled using LDA, predictions à la logistic regression $P(y|\mathbf{x})$ can be obtained by using Bayes' rule:

\begin{equation}
P(y=k|\mathbf{x}) = \frac{P(\mathbf{x}|y=k)P(y)}{P(\mathbf{x})} = \frac{P(\mathbf{x}|y=k)P(y=k)}{\sum_{l=1}^{K}P(\mathbf{x}|y=l)P(y=l)}
\end{equation}

where all probabilities $P(\mathbf{x}|y=l)$ is obtained by the LDA model and all probabilities $P(y=l)$ are simply observed from the class frequency in the dataset (e.g.: 70% of the training dataset samples belongs to class $l$ so: $P(y=l) = 0.70$)

## 1.3 Classification metrics

As you may imagine, the classification accuracy (the percentage of samples predicted correctly), may not always be the most appropriate metric to evaluate the performance of a classifier. In particular, the following two scenarios seem to require a different approach:

* In some applications, we want to avoid false negatives at the cost of more false positives. For instance, we want to make sure that we do not falsely predict that a patient is healthy when in reality he has a disease. When we allow for more false positives, the classification accuracy will go down.
* When the data is very imbalanced, a *null classifier* that always predicts the same class will achieve a high classification accuracy.

A first step to get some more insight in the performance of a classifier is to create a confusion matrix, which gives an overview of the kind of mistakes we have made during classification. A confusion matrix might look like this:

The concepts of *specificity* and *sensitivity* are better suited in the previous example scenarios. These are *class-specific* metrics: the sensitivity is the fraction of (true) positives that is correctly identified (as positive). The specificity is the fraction of (true negatives) that is correctly classified (as negative). A classifier might predict nearly every sample as positive, hence obtaining a high sensitivity but a very low specificity. One way to change this is to change the threshold for classification: by selecting a higher threshold, we will predict more samples as negatives, hence the sensitivity will go down because we will now predict some positives as being negative. However, the specificity will improve, because we now only predict samples as positive if the model outputs a higher probability for it being positive.

The apparent trade-off between sensitivity and specificity is summarized by the receiver operating characteristic (ROC) curve. A ROC curve is a plot of the true positive rate (sensitivity) versus the false positive rate (one minus the specificity) for various thresholds. It typically looks like this:

It illustrates the trade-off between sensitivity and specificity as we vary the threshold. A good measure to evaluate a classifier is the **area under the ROC curve (AUROC or AUC)**. A classifier that does random guesses has an AUC of 0.5 (dotted line in the plot), so if we design a classifier for a certain problem we want it to achieve an AUC higher than 0.5.

Check this [blog](https://datascienceplus.com/interpretation-of-the-auc/) to read a bit more considering the interpretation of the AUROC.

## 2 Exercises

### 2.1 Breast cancer dataset

In this set of exercises, we will use the [Breast Cancer Wisconsin (Diagnostic) Data Set](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29). The dataset contains information on the disease status of 569 breast cancer patients: they were either diagnosed with a malign (status M) or with a benign (status B) tumor.

For each patient, the dataset also contains 30 features that represent statistics of the cell nuclei present in images taken after [fine needle aspirate tissue samples](https://en.wikipedia.org/wiki/Fine-needle_aspiration). These 30 features are the mean, standard deviation and the maximum of 10 measurements on the cell nuclei:
- radius
- texture
- perimeter
- area
- smoothness
- compactness
- concavity
- concave points
- symmetry
- fractal dimension

**Based on these feature of the cell nuclei, we would like to predict whether a patient has a malign or a benign breast cancer tumor.** Let's download and read in the data:

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/wdbc.data
```

    --2021-10-05 14:17:50--  https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/wdbc.data
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 124103 (121K) [text/plain]
    Saving to: ‘wdbc.data.1’

    wdbc.data.1         100%[===================>] 121.19K  --.-KB/s    in 0.01s

    2021-10-05 14:17:50 (8.50 MB/s) - ‘wdbc.data.1’ saved [124103/124103]

```python
import pandas as pd
import numpy as np
data = pd.read_csv('./wdbc.data', header=None, index_col=0, names=['Patient ID', 'status'] + list(np.arange(1,31,1)))
status = data['status']
data.head()
```

      status
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30

      Patient ID

      842302
      M
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      ...
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890

      842517
      M
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      ...
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902

      84300903
      M
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      ...
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758

      84348301
      M
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      ...
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300

      84358402
      M
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      ...
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678

5 rows × 31 columns

First, let's look at the distribution of the disease status:

```python
pd.value_counts(data['status']).plot(kind='bar');
```

There are about 350 benign cases and roughly 200 malign cases. This is a fairly balanced dataset.

THOUGHT EXERCISE: Suppose that the dataset was unbalanced, with 525 B cases and only 25 M cases. Would you still use accuracy to evaluate the model?

In these exercises, we will use the [scikit-learn](https://scikit-learn.org/stable/) implementations of [LR](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) and [LDA](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html). In order to fit models, we will first need to encode the labels 'Benign' (B) and 'Malign' (M) to a binary encoder:

```python
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder().fit(status)
encoder.classes_  # 'B' will become class 0, 'M' will become class 1
```

    array(['B', 'M'], dtype=object)

```python
y = encoder.transform(status)
x = data.drop('status', axis=1).values # Drop the disease status from the dataframe, convert to numpy array
y[:100]
```

    array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1,
           1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1,
           0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1,
           0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1])

EXERCISE: Using scikit-learn, split the data in a 80% training and a 20% test set. Fit a logistic regression model and evaluate training and testing accuracy.

Use [this method](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) for train-test splitting and [this implementation](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to perform logistic regression. Accuracy can be computed by using the score() method of each model.

```python
# ** solution **
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.20)

LRmodel = LogisticRegression()
LRmodel.fit(X_train, y_train)

print(LRmodel.score(X_train, y_train))
print(LRmodel.score(X_test, y_test))

# ** solution **
```

    0.9626373626373627
    0.9035087719298246

    /home/gaetan/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      n_iter_i = _check_optimize_result(

EXERCISE: Now do the same with LDA. Do you split your dataset again in a new random training and testing split? Think about fair evaluation/comparison between the two models.

Use [this implementation](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html) for LDA.

```python
# ** solution **
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

LDAmodel = LinearDiscriminantAnalysis()
LDAmodel.fit(X_train, y_train)

print(LDAmodel.score(X_train, y_train))
print(LDAmodel.score(X_test, y_test))
# ** solution **
```

    0.9692307692307692
    0.9298245614035088

Depending on your data split, LDA or LR will perform better. The model that we prefer is hence determined by a random data splitting, a behavior that we do not want. One way to avoid this is to perform [cross-validation](https://machinelearningmastery.com/k-fold-cross-validation/), we will return on this topic later in the course.

EXERCISE: Use your preferred (LDA or LR) model to predict the class probabilities $P(y|\mathbf{x})$ and the classes for the training data. Use the predict_proba() method to generate the predicted probabilities. Take note of the output shape of predict_proba(), what do the columns mean? Use the code below to plot the two against each other. Which data points are most likely to be misclassified?

```python
# ** solution **
predicted_class_probabilities = LDAmodel.predict_proba(X_train)[:,1]
predicted_classes = LDAmodel.predict(X_train)

#** solution **
```

```python
import matplotlib.pyplot as plt

misclassified = predicted_classes !=  y_train

colors = ['#b2182b' if wrong else '#2166ac' for wrong in misclassified ]
fig, ax = plt.subplots(figsize=(8,6))
test_jitter = np.random.normal(scale=0.1, size=len(predicted_class_probabilities))
ax.scatter(test_jitter, predicted_class_probabilities,
           marker='.', s=25, color=colors, alpha=0.75)
ax.get_xaxis().set_ticks([]);
ax.set_ylabel('Predicted class probabilies').set_fontsize(20)
ax.set_title('Correctly classified samples in blue', size=24)
plt.show()
```

Clearly, the misclassified points are those points where the predicted probability of class membership is rather close to 0.5.

EXERCISE: Use the predicted class probabilities from the previous exercise to compute the ROC AUC score on the test dataset. Then, use the code below to plot the ROC curves of both models.

```python
# plot the roc curves
from sklearn.metrics import roc_curve, roc_auc_score

# Use test data!
#########
predicted_class_probabilities_LDA = LDAmodel.predict_proba(X_test)[:,1]
predicted_class_probabilities_LR = LRmodel.predict_proba(X_test)[:,1]

AUC_LDA = roc_auc_score(y_test, predicted_class_probabilities_LDA)
AUC_LR = roc_auc_score(y_test, predicted_class_probabilities_LR)

##############
```

```python
roc_LDA = roc_curve(y_test, predicted_class_probabilities_LDA)
roc_LR = roc_curve(y_test, predicted_class_probabilities_LR)

fig, ax = plt.subplots(figsize=(15,10))
ax.plot([0, 1], [0, 1], '--')
ax.plot(roc_LR[0], roc_LR[1])
ax.plot(roc_LDA[0], roc_LDA[1])
ax.set_xlabel('1-specificity').set_fontsize(18)
ax.set_ylabel('sensitivity').set_fontsize(18)

ax.legend(['Random classifier - AUC 0.5',
           'LR - test data AUC {}'.format(AUC_LR),
           'LDA - test data AUC {}'.format(AUC_LDA)],
         fontsize='large')
plt.show()
```

 EXERCISE:
We have succesfully implemented a pipeline going from data exploration to model construction and evaluation (including comparison) for the wisconsin breast cancer dataset. For the end of the PC lab, we introduce a new dataset: [Job change analytics](https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists). Implement the data exploration and model construction pipeline for this dataset by reusing code from previous exercises. The goal of this exercise is to show that every dataset has its own unique set of characteristics and challenges.

Some information about the dataset: A company which is active in Big Data and Data Science wants to hire data scientists among people who successfully pass some courses which conduct by the company. Many people signup for their training. Company wants to know which of these candidates really want to work for the company after training or looking for a new employment because it helps to reduce the cost and time as well as the quality of training or planning the courses and categorization of candidates. Information related to demographics, education, experience are in hands from candidates signup and enrollment. **Hence, the input features are details about persons who took data science courses, and the output of interest (target) is whether those persons are actually looking for a new job.**

More information [here](https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists).

- What do you do with missing values in the dataset?: You can throw samples with at least one missing feature value out, or you can [impute](https://scikit-learn.org/stable/modules/impute.html) those missing values.
- What about class imbalance?
- Many features are categorical, how will you encode these?

Since there is no single right way to answer the above questions, we will not provide a solution to this exercise. Experiment at free will! If you have any questions, you can always mail the teaching assistant.

Below is some preliminary data exploration. Use this to further explore the data and make design choices.

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/HR_analytics.csv
```

    --2021-10-05 14:17:56--  https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/HR_analytics.csv
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
    HTTP request sent, awaiting response... 404 Not Found
    2021-10-05 14:17:57 ERROR 404: Not Found.

```python
data = pd.read_csv('HR_analytics.csv')
data.shape
```

    (19158, 14)

```python
data.head()
```

      enrollee_id
      city
      city_development_index
      gender
      relevent_experience
      enrolled_university
      education_level
      major_discipline
      experience
      company_size
      company_type
      last_new_job
      training_hours
      target

      0
      8949
      city_103
      0.920
      Male
      Has relevent experience
      no_enrollment
      Graduate
      STEM
      >20
      NaN
      NaN
      1
      36
      1.0

      1
      29725
      city_40
      0.776
      Male
      No relevent experience
      no_enrollment
      Graduate
      STEM
      15
      50-99
      Pvt Ltd
      >4
      47
      0.0

      2
      11561
      city_21
      0.624
      NaN
      No relevent experience
      Full time course
      Graduate
      STEM
      5
      NaN
      NaN
      never
      83
      0.0

      3
      33241
      city_115
      0.789
      NaN
      No relevent experience
      NaN
      Graduate
      Business Degree
      <1
      NaN
      Pvt Ltd
      never
      52
      1.0

      4
      666
      city_162
      0.767
      Male
      Has relevent experience
      no_enrollment
      Masters
      STEM
      >20
      50-99
      Funded Startup
      4
      8
      0.0

```python
# number of NaN (missing values) per feature:
pd.isnull(data).sum(0)
```

    enrollee_id                  0
    city                         0
    city_development_index       0
    gender                    4508
    relevent_experience          0
    enrolled_university        386
    education_level            460
    major_discipline          2813
    experience                  65
    company_size              5938
    company_type              6140
    last_new_job               423
    training_hours               0
    target                       0
    dtype: int64

```python
# number of unique values per feature:
for k in data.columns:
    print(k, len(np.unique(data[k][~pd.isnull(data[k])])))
```

    enrollee_id 19158
    city 123
    city_development_index 93
    gender 3
    relevent_experience 2
    enrolled_university 3
    education_level 5
    major_discipline 6
    experience 22
    company_size 8
    company_type 6
    last_new_job 6
    training_hours 241
    target 2

```python
# Look at what the unique values are for a specific feature:
feature = 'enrolled_university'
np.unique(data[feature][~pd.isnull(data[feature])])
```

    array(['Full time course', 'Part time course', 'no_enrollment'],
          dtype=object)

```python
#################### YOUR OWN CODE HERE #######################
```

### 2.2 OPTIONAL EXTRA: Multiclass classification

Some classifiers inherently support multiclass classification as part of their design, for example, in LDA, the class-conditional density $P(\mathbf{x}|y)$ is modeled for every class $y$ and can be converted to predictions using Bayes' rule. For logistic regression, a natural extension to the multi-class case is [multinomial logistic regression](https://en.wikipedia.org/wiki/Multinomial_logistic_regression). Check out the [scikit-learn page on multiclass prediction](https://scikit-learn.org/stable/modules/multiclass.html) for more information. In principle, any classifier can be made multiclass using following methods:

#### One-versus-one classification

One-versus-one classification is another approach to a multiclass classification problem. For a K-class problem, the strategy consists of training $\frac{K(K-1)}{2}$ classifiers. Each of these classifiers much learn to distinguish to classes. One the classifiers are trained, a voting scheme is applied to make a prediction for an unseen data point: each classifier has to decide between two possible classes. The final predicted class is that class that gets the largest number of votes.

#### One-versus-all classification

In one-versus-all (OvA) classification (also called one-versus-rest), a single classifier is trained per class, with the samples of that class as positive samples and all other samples as negatives. The strategy proceeds as follows for a K-class classification problem:

**Inputs:**
* a classification algorithm L (learner)
* feature matrix $\mathbf{X}$
* label vector y where $y_i \in {1,...,K}$

**Procedure:**
for each k in {1,...,K}:
* construct a new label vector z where $z_i$ is 1 if $y_i$ = k and 0 otherwise
* train L on $\mathbf{X}$ to obtain a classifier $f_k$. The classifier should return class probabilities and not hard labels.

**Returns**
A list of trained classifiers $f_k$ for each  k in {1,...,K}

To make predictions for a new sample $\mathbf{x}$, the $k$ classifiers are applied to $\mathbf{x}$ and the final predicted label is the label that is predicted with the highest confidence (probability):

$\hat{y} = \underset{k \in {1,...,K}}{\mathrm{argmax}} \, f_k(\mathbf{x})$

Let's simulate a toy dataset with three classes and two features, and split it in training and test data:

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split

X, y = make_blobs(n_samples=1000, centers= [[-2.5, 0], [0, 1], [3.5, -1]], random_state=42)

#train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

```python
# Make the plot
fig, ax = plt.subplots(figsize=(15,10))
colors=['#66c2a5', '#fc8d62', '#8da0cb']
for i, color in enumerate(colors):
    idx_train = np.where(y_train==i)
    idx_test = np.where(y_test==i)
    plt.scatter(X_train[idx_train,0], X_train[idx_train,1], c=color, edgecolor='black', s=30)
    plt.scatter(X_test[idx_test,0], X_test[idx_test, 1],c='white', edgecolor=color, s=70)

ax.legend(['Class 1 - train',
           'Class 1 - test',
           'Class 2 - train',
           'Class 2 - test',
           'Class 3 - train',
           'Class 3 - test']);

ax.set_xlabel('Feature 1');
ax.set_ylabel('Feature 2');
ax.set_title('Toy dataset for multiclass classification').set_fontsize(20);
```

OPTIONAL EXTRA EXERCISE: Look again at the documentation of LDA and Logistic Regression. What options do you need to specify to train these models on a multi-class problem? Train a model on this toy dataset

```python
LRmodel = LogisticRegression(multi_class='ovr')
LRmodel.fit(X_train, y_train)

predicted_classes = LRmodel.predict(X_test)
predicted_probas = LRmodel.predict_proba(X_test)

classification_accuracy = np.round(np.mean(y_test == predicted_classes)*100,2)

# Visualize the predictions

fig, ax = plt.subplots(figsize=(15,10))
colors=['#66c2a5', '#fc8d62', '#8da0cb']

for i, color in enumerate(colors):
    idx_train = np.where(y_train==i)
    idx_test = np.where(y_test==i)

    plt.scatter(X_train[idx_train,0], X_train[idx_train,1], c=color, edgecolor='black', s=30)
    plt.scatter(X_test[idx_test,0], X_test[idx_test, 1],c='white', edgecolor=color, s=70)

ax.legend(['Class 1 - train',
           'Class 1 - test',
           'Class 2 - train',
           'Class 2 - test',
           'Class 3 - train',
           'Class 3 - test']);

# add predictions
for i, color in enumerate(colors):
    idx_predicted = np.where(predicted_classes==i)
    plt.scatter(X_test[idx_predicted,0], X_test[idx_predicted,1], c=color, marker='s', s=2)

ax.set_xlabel('Feature 1');
ax.set_ylabel('Feature 2');
ax.set_title('Toy dataset for multiclass classification - classification accuracy: {}%'.format(classification_accuracy)).set_fontsize(20);
```

### 2.3 OPTIONAL EXTRA: Data preprocessing for text inputs (example with DNA)

Since machine learning models expect numbers as inputs, not words, we need to do some preprocessing first to make them compatible with ML models. A very common feature representation for text sequences is called the **Bag of Words**. It consists of listing all the possible tokens that might occur in your text. A token might be one word, or a combination of two or multiple words. The feature representation is then a matrix with the count of the tokens for each text instance. The simplest transformation is to just look at the counts of individual tokens: the 1-grams. However, it might be interesting to count combinations of two or more words as well. Consider the following example, where we have 2 example input sample sentences:

```python
from sklearn.feature_extraction.text import CountVectorizer

data = ['this is real life',
       'is this real life']
print('data:\n', data[0], '\n', data[1])
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1,2))

analyze = vectorizer.build_analyzer()
print('vocabulary:\n', analyze(data[0]))

features = vectorizer.fit_transform(data)
print('final feature representation:\n', features.toarray())
```

    data:
     this is real life
     is this real life
    vocabulary:
     ['this', 'is', 'real', 'life', 'this is', 'is real', 'real life']
    final feature representation:
     [[1 1 0 1 1 1 1 1 0]
     [1 0 1 1 1 1 1 0 1]]

Note that by default, the vectorizer returns the features as a sparse array: in most applications, each row of the feature matrix will contain a lot of zeros. For this exercise, you can convert it to a regular numpy array with the ```toarray()``` method.

If we restrict ourselves to look at 1-grams, both sentences above are transformed into the exact same feature representation. However, suppose we want to do sentiment analysis and classify whether a sentence contains a question or not. Then it might be useful to look at bigrams as well, as above

```python
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1,1))
analyze = vectorizer.build_analyzer()
analyze(data[0])
features = vectorizer.fit_transform(data)
print('Only 1-grams:')
print('final feature representation:\n', features.toarray())
```

    Only 1-grams:
    final feature representation:
     [[1 1 1 1]
     [1 1 1 1]]

When we consider bigrams, the feature representation for both sentences is no longer exactly the same. Maybe the features that represent 'this is' and 'is this' could be helpful here.

Still the bag-of-words features completely ignores where every n-gram was in the sentence. Another way to take this into account is to assign $v$ dummy features for every location in the sentence, with $v$ being the vocabulary size (number of possible different words). Every first, second, third and fourth word get assigned 1 feature for every possible word in that location (out of 4 possible words in the vocabulary). This is called [one-hot encoding](https://en.wikipedia.org/wiki/One-hot#Natural_language_processing)
This requires the sentences to be all of the same length, for example, 4 words as in the 2 sentences above.

```python
data_splitwords = [sentence.split(' ') for sentence in data]
print('data splitted by words:')
print(data_splitwords)
vocab = list(set([word for sentence in data_splitwords for word in sentence]))
print('vocabulary:')
print(vocab)
from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown='ignore',
                    categories=[vocab]*len(data_splitwords[0]))

features = enc.fit_transform(data_splitwords)
print('resulting feature representation:')
print(features.toarray())
```

    data splitted by words:
    [['this', 'is', 'real', 'life'], ['is', 'this', 'real', 'life']]
    vocabulary:
    ['real', 'life', 'is', 'this']
    resulting feature representation:
    [[0. 0. 0. 1. 0. 0. 1. 0. 1. 0. 0. 0. 0. 1. 0. 0.]
     [0. 0. 1. 0. 0. 0. 0. 1. 1. 0. 0. 0. 0. 1. 0. 0.]]

If it is not immediately clear what the features mean in this example: the first four features represent which word is the first word in each sentence. The first and sentence has as first word 'this', this is the last word in the vocabulary, so of the first four features, the last is given by $1$. The second sentence has 'is' as the first word, which is the second word in the vocabulary, for this reason, the second column of the first four features is given by $1$ for this sentence. Both sentences have the same last two words 'real life', it can be seen that the last 8 features for both sentences are identical.

Armed with these encoding schemes, we can tackle a real problem. **DNA sequences** as inputs are peculiar because they are essentially text: following an alphabet containing only A, T, C or G. We will use data from a published [study](https://bmcbiol.biomedcentral.com/articles/10.1186/1741-7007-12-4) about sigma factors in E. coli. Some biological background information for those interested:

> The binding region of the transcription unit,  commonly called the promoter region, is known to play a key role  in the transcription rate of downstream genes. In Eubacteria, the sigma factor ($\sigma$) binds with the RNA  polymerase subunit ($\alpha\beta\beta'\omega$) to create RNA polymerase. Being part of the RNA polymerase holoenzyme, the sigma element acts as the connection between RNA polymerase and DNA. Depending on the sigma factor bound to the transcription unit, different binding regions of the enzyme with the DNA are observed. The data represents specific DNA regions of the *E. coli* genome to which one or more sigma factors bind.

In less-biological-jargon words: certain molecules called sigma factors control activity of genes, they do so by binding to the DNA close to the gene. If we can predict for an input DNA sequence if such a molecule will bind, we gain an understanding in the biological function of those genes and molecules.

The dataset contains 4724 DNA sequences. For each sequence, we have information on five target variables, each representing 5 sigma factors. Each time, the outcome is binary, indicating whether the sigma factor binds to the sequence or not. This is a **multi-label** problem, where an input can belong to any (multiple at once) of the classes, as opposed to only one possible class for each sample in multi-class. For simplicity in this exercise, we will focus only on one class: RPOD.

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/promotor_list_exp_growth.csv
promotor_data = pd.read_csv('./promotor_list_exp_growth.csv', index_col=0)
promotor_data.head(20)

X = np.array(promotor_data.PROBE_SEQUENCE)
y = promotor_data['RPOD'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=23)
```

    --2021-09-23 11:30:59--  https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/promotor_list_exp_growth.csv
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 315440 (308K) [text/plain]
    Saving to: ‘promotor_list_exp_growth.csv.1’

    promotor_list_exp_g 100%[===================>] 308.05K  --.-KB/s    in 0.1s

    2021-09-23 11:31:00 (3.06 MB/s) - ‘promotor_list_exp_growth.csv.1’ saved [315440/315440]

 OPTIONAL EXTRA EXERCISE:
Use one of the encoding schemes to preprocess the DNA data. Construct a model and evaluate performance on the test set. Play around with different preprocessing values, and keep testing performances. What types of preprocessing lead to good performance for this dataset? In what cases are we overfitting? Think about fair evaluation: if we keep testing out configurations and evaluating on the test set, is our final test performance estimate fair? We can solve this by keeping an additional data split separate: training, validation and testing. Then we test different hyperparameters (different ways of preprocessing) on the validation set. We don't touch the test set until we have our final model. This is the only fair way of evaluating. For more information, see this [link](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).

```python
# N-GRAM features:

vectorizer = CountVectorizer(analyzer='char', ngram_range=(1,5))

X_train1 = vectorizer.fit_transform(X_train).toarray()
X_test1 = vectorizer.transform(X_test).toarray()

LDAmodel = LinearDiscriminantAnalysis()
LDAmodel.fit(X_train1, y_train)

y_pred = LDAmodel.predict(X_test1)
y_pred_train = LDAmodel.predict(X_train1)

print(roc_auc_score(y_test, y_pred), roc_auc_score(y_train, y_pred_train))

# LOCATION features:

X_train2 = [list(x) for x in X_train]
X_test2 = [list(x) for x in X_test]

enc = OneHotEncoder(categories=[['A', 'T', 'C', 'G']]*len(X_train2[0]))

X_train2 = enc.fit_transform(X_train2).toarray()
X_test2 = enc.fit_transform(X_test2).toarray()

LDAmodel = LinearDiscriminantAnalysis()
LDAmodel.fit(X_train2, y_train)

y_pred = LDAmodel.predict(X_test2)
y_pred_train = LDAmodel.predict(X_train2)

print(roc_auc_score(y_test, y_pred), roc_auc_score(y_train, y_pred_train))
```

    0.5784724527049452 0.8140549048447562
    0.6786010620643878 0.6831724991479974

```python

```

```python

```


# Qiskit Aqua: Generating Random Variates

## Introduction

While classical computers use only pseudo-random routines, quantum computers
can generate true random variates.
For example, the measurement of a quantum superposition is intrinsically random,
as suggested by Born's rule.
Consequently, some of the
best random-number generators are based on such quantum-mechanical effects.
Further, with a logarithmic amount of random bits, quantum computers can produce
linearly many more bits, which is known as
randomness expansion protocols.

In practical applications, one wishes to use random variates of well-known
distributions, rather than random bits.
In this notebook, we illustrate ways of generating random variates of several popular
distributions on IBM Q.

## Random Bits and the Bernoulli distribution

It is clear that there are many options for generating random bits (i.e., Bernoulli-distributed scalars, taking values either 0 or 1). Starting from a simple circuit such as a Hadamard gate followed by measurement, one can progress to vectors of Bernoulli-distributed elements. By addition of such random variates, we could get binomial distributions. By multiplication we could get geometric distributions, although perhaps leading to a circuit depth that may be impractical at the moment, though.

Let us start by importing the basic modules and creating a quantum circuit for generating random bits:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sys, math, time
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)

from qiskit import BasicAer
from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister, execute

# In this example we use 'qasm_simulator' backend.
glo_backend = BasicAer.get_backend("qasm_simulator")
```

In the next step we create a quantum circuit, which will be used for generation:

```python
# Number of qubits utilised simultaneously.
glo_num_qubits = 5

def create_circuit(num_target_qubits: int) -> QuantumCircuit:
    """
    Creates and returns quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :return: quantum curcuit.
    """
    assert isinstance(num_target_qubits, int) and num_target_qubits > 0
    q = QuantumRegister(num_target_qubits)
    c = ClassicalRegister(num_target_qubits)
    circuit = QuantumCircuit(q, c)
    circuit.h(q)
    circuit.barrier()
    circuit.measure(q, c)
    return circuit

# Create and plot generating quantum circuit.
circuit = create_circuit(glo_num_qubits)
#print(circuit)
circuit.draw(output='mpl')
```

## Uniformly-distributed scalars and vectors

It is clear that there are many options for approximating uniformly-distributed scalars by the choice of an integer from a finite range uniformly at random, e.g., by a binary-code construction from the Bernoulli-distributed vectors. In the following snippet, we generate random bits, which we then convert using the binary-code construction, up to the machine precision of a classical computer.

```python
def uniform_rand_float64(circuit: QuantumCircuit, num_target_qubits: int,
                         size: int, vmin: float, vmax: float) -> np.ndarray:
    """
    Generates a vector of random float64 values in the range [vmin, vmax].
    :param circuit: quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :param size: length of the vector.
    :param vmin: lower bound.
    :param vmax: upper bound.
    :return: vector of random values.
    """
    assert sys.maxsize == np.iinfo(np.int64).max    # sizeof(int) == 64 bits
    assert isinstance(size, int) and size > 0
    assert isinstance(vmin, float) and isinstance(vmax, float) and vmin <= vmax
    nbits = 7 * 8                                   # nbits > mantissa of float64
    bit_str_len = (nbits * size + num_target_qubits - 1) // num_target_qubits
    job = execute(circuit, glo_backend, shots=bit_str_len, memory=True)
    bit_str = ''.join(job.result().get_memory())
    scale = float(vmax - vmin) / float(2**nbits - 1)
    return np.array([vmin + scale * float(int(bit_str[i:i+nbits], 2))
                     for i in range(0, nbits * size, nbits)], dtype=np.float64)

def uniform_rand_int64(circuit: QuantumCircuit, num_target_qubits: int,
                       size: int, vmin: int, vmax: int) -> np.ndarray:
    """
    Generates a vector of random int64 values in the range [vmin, vmax].
    :param circuit: quantum circuit for random variate generation.
    :param num_target_qubits: number of qubits to be used.
    :param size: length of the vector.
    :param vmin: lower bound.
    :param vmax: upper bound.
    :return: vector of random values.
    """
    assert sys.maxsize == np.iinfo(np.int64).max        # sizeof(int) == 64 bits
    assert isinstance(size, int) and size > 0
    assert isinstance(vmin, int) and isinstance(vmax, int) and vmin <= vmax
    assert abs(vmin) <= 2**52 and abs(vmax) <= 2**52    # 52 == mantissa of float64
    return np.rint(uniform_rand_float64(circuit, num_target_qubits,
                                        size, float(vmin), float(vmax))).astype(np.int64)
```

### Uniform distribution over floating point numbers.
In this example we draw a random vector of floating-point values uniformly distributed within some arbitrary selected interval:

```python
# Draw a sample from uniform distribution.
start_time = time.time()
sample = uniform_rand_float64(circuit, glo_num_qubits, size=54321, vmin=-7.67, vmax=19.52)
sampling_time = time.time() - start_time

# Print out some details.
print("Uniform distribution over floating point numbers:")
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:.4f}, max: {:.4f}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='b', alpha=0.75)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Uniform distribution over float64 numbers in [{:.2f} ... {:.2f}]".format(
            np.amin(sample), np.amax(sample)), size=12)
plt.grid(True)
# plt.savefig("uniform_distrib_float.png", bbox_inches="tight")
plt.show()
```

### Uniform distribution over integers.
Our next example is similar to the previous one, but here we generate a random vector of integers:

```python
# Draw a sample from uniform distribution.
start_time = time.time()
sample = uniform_rand_int64(circuit, glo_num_qubits, size=54321, vmin=37, vmax=841)
sampling_time = time.time() - start_time

# Print out some details.
print("Uniform distribution over bounded integer numbers:")
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:d}, max: {:d}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='g', alpha=0.75)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Uniform distribution over int64 numbers in [{:d} ... {:d}]".format(
            np.amin(sample), np.amax(sample)), size=12)
plt.grid(True)
# plt.savefig("uniform_distrib_int.png", bbox_inches="tight")
plt.show()
```

## Normal distribution

To generate random variates with a standard normal distribution using two independent
samples $u_1, u_2$ of the uniform distribution on the unit interval [0, 1], one can
consider the Box-Muller transform to obtain a 2-vector:

\begin{align}
\begin{bmatrix}
%R\cos(\Theta )=
{\sqrt {-2\ln u_{1}}}\cos(2\pi u_{2}) \\
% R\sin(\Theta )=
{\sqrt {-2\ln u_{1}}}\sin(2\pi u_{2})
\end{bmatrix},
\end{align}

wherein we have two independent samples of the standard normal distribution.
In IBM Q, this is implemented as follows:

```python
def normal_rand_float64(circuit: QuantumCircuit, num_target_qubits: int,
                        size: int, mu: float, sigma: float) -> np.ndarray:
    """
    Draws a sample vector from the normal distribution given the mean and standard
    deviation, using the Box-Muller method.
    """
    TINY = np.sqrt(np.finfo(np.float64).tiny)
    assert isinstance(size, int) and size > 0
    rand_vec = np.zeros((size,), dtype=np.float64)

    # Generate array of uniformly distributed samples, factor 1.5 longer that
    # actually needed.
    n = (3 * size) // 2
    x = np.reshape(uniform_rand_float64(circuit, num_target_qubits,
                                        2*n, 0.0, 1.0), (-1, 2))

    x1 = 0.0                # first sample in a pair
    c = 0                   # counter
    for d in range(size):
        r2 = 2.0
        while r2 >= 1.0 or r2 < TINY:
            # Regenerate array of uniformly distributed samples upon shortage.
            if c >= n:
                c = 0
                n = max(size // 10, 1)
                x = np.reshape(uniform_rand_float64(circuit, num_target_qubits,
                                                    2*n, 0.0, 1.0), (-1, 2))

            x1 = 2.0 * x[c, 0] - 1.0        # first sample in a pair
            x2 = 2.0 * x[c, 1] - 1.0        # second sample in a pair
            r2 = x1 * x1 + x2 * x2
            c += 1

        f = np.sqrt(np.abs(-2.0 * np.log(r2) / r2))
        rand_vec[d] = f * x1

    return (rand_vec * sigma + mu)
```

The following example demonstrates how to draw a random vector of normally distributed variates:

```python
# Mean and standard deviation.
mu = 2.4
sigma = 5.1

# Draw a sample from the normal distribution.
start_time = time.time()
sample = normal_rand_float64(circuit, glo_num_qubits, size=4321, mu=mu, sigma=sigma)
sampling_time = time.time() - start_time

# Print out some details.
print("Normal distribution (mu={:.3f}, sigma={:.3f}):".format(mu, sigma))
print("  sample type:", type(sample), ", element type:", sample.dtype, ", shape:", sample.shape)
print("  sample min: {:.4f}, max: {:.4f}".format(np.amin(sample), np.amax(sample)))
print("  sampling time: {:.2f} secs".format(sampling_time))

# Plotting the distribution.
x = np.linspace(mu - 4.0 * sigma, mu + 4.0 * sigma, 1000)
analyt = np.exp(-0.5 * ((x - mu) / sigma)**2) / (sigma * math.sqrt(2.0 * math.pi))
plt.hist(sample.ravel(),
         bins=min(int(np.ceil(np.sqrt(sample.size))), 100),
         density=True, facecolor='r', alpha=0.75)
plt.plot(x, analyt, '-b', lw=1)
plt.xlabel("value", size=12)
plt.ylabel("probability", size=12)
plt.title("Normal distribution: empirical vs analytic", size=12)
plt.grid(True)
# plt.savefig("normal_distrib.png", bbox_inches="tight")
plt.show()
```

There is a substantial amount of further work needed to either certify the quality of the source of random numbers (cf. NIST SP 800-90B, Recommendation for the Entropy Sources Used for Random Bit Generation) or to use random variates within quantum algorithms (cf. uncertainty_models within Qiskit Aqua).

```python
import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright
```

Version Information

Qiskit Software	Version

Qiskit	0.14.0

Terra	0.11.0

Aer	0.3.4

Ignis	0.2.0

Aqua	0.6.1

IBM Q Provider	0.4.4

System information

Python	3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)]

OS	Darwin

CPUs	4

Memory (Gb)	16.0

Thu Dec 12 10:17:05 2019 EST

This code is a part of Qiskit

© Copyright IBM 2017, 2019.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
 of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals.

```python

```


For the Ronbrock method, we need to solve a linear system of the form
$$
M_{ij}x_{j}=b_{i} \;,
$$
with M a square matrix (repeated indecies imply summation).

Such systems are soved by (among  other methods) the so-called LU factorization (or decomposition),
where you decompose $M_{ij}=L_{ik}U_{kj}$ with $L_{i, j>i}=0$, $L_{i, j=i}=1$, $U_{i,j<i}=0$.

That is if $M$ is $N \times N$ matrix, L,U are defined as
\begin{align}
    &L=\left( \begin{matrix}
    1 & 0 & 0 & 0 & \dots &0 & 0 \\
    L_{2,1} & 1 & 0 & 0 & \dots &0 & 0\\
    L_{3,1} & L_{3,2} & 1 & 0 & \dots &0 & 0 \\
    \vdots & \vdots & \vdots & \ddots & \dots & \vdots & \vdots \\
    L_{N-1, 1} & L_{N-1, 2} & L_{N-1, 3} & L_{N-1, 4} & \dots & 1 & 0 \\
    L_{N, 1} & L_{N, 2} & L_{N, 3} & L_{N, 4} & \dots & L_{N, N-1} & 1 \\
    \end{matrix}\right) \;, \\
    %
    &U=\left( \begin{matrix}
    U_{1,1} & U_{1,2} & U_{1,3} & U_{1,4} & \dots & U_{1,N-1} & U_{1,N} \\
    0 & U_{2,2} & U_{2,3} & U_{2,4} & \dots & U_{2,N-1} & U_{2,N}\\
    0 & 0 & U_{3,3} & U_{3,4} & \dots & U_{3,N-1} & U_{3,N} \\
    \vdots & \vdots & \vdots & \ddots & \dots & \vdots & \vdots \\
    0 & 0 & 0 &0 & \dots & U_{N-1,N-1} & U_{N-1,N} \\
    0 & 0 & 0& 0 & \dots & 0 &U_{N,N} \\
    \end{matrix}\right)
    %
\end{align}

Then we have in general $M_{i, j} = \sum_{k=1}^{i}L_{i,k}U_{k,j}$. Since
$L_{i, k \geq i}=0$ and $U_{k>j,j}=0$, the sum runs up to $j$ if $i \geq j$ and
$i$ if $i \leq j$  (for $i=j$ then both are correct). That is

$$
M_{i, j \geq i} = \sum_{k=1}^{i-1}L_{i,k}U_{k,j}+ U_{i,j} \Rightarrow
U_{i,j }=M_{i,j } - \sum_{k=1}^{i-1}L_{i,k}U_{k,j }\; , \;\;\; j \geq i \\[0.5cm]
M_{i, j \leq i} = \sum_{k=1}^{j-1}L_{i,k}U_{k,j} +L_{i,j}U_{j,j} \Rightarrow
L_{i,j}=\left( M_{i,j} - \sum_{k=1}^{j-1}L_{i,k}U_{k,j} \right) U_{jj}^{-1} , \;\;\; j \leq i
$$

Since $U$ and $L$ are triangular matrices, we can solve these two systems sequentially
$$
L_{i,k}y_{k}=b_{k} \\
U_{k,j}x_{j}=y_{k},
$$

since
$$
y_1 = b_{1} \\
L_{2,1}y_{1}+y_{2}=b_{2} \Rightarrow y_{2}=b_{2}-L_{2,1}y_{1} \\
\vdots \\
y_{i}=b_{i} - \sum_{j=1}^{i-1}L_{i,j}y_{j}
$$

and

$$
U_{N,N}x_{N}=y_{N} \Rightarrow x_{N}=y_{N}/U_{N,N}\\
U_{N-1,N}x_{N}+U_{N-1,N-1}x_{N-1}=y_{N-1} \Rightarrow x_{N-1}=\left(y_{N-1} -U_{N-1,N}x_{N} \right)/U_{N-1,N-1} \\
\vdots \\
x_{i}=\dfrac{y_{i} -\displaystyle\sum_{j=i+1}^{N} U_{i,j}x_{j}  }{U_{i,i}}
$$

Since $U_{i,i}$  appears to denominator, if the diagonal terms of $U$ are small (or god forbid they vanish), we would have a problem.

To solve this problem we do $LUP$ decomposition, where $L \; U=P \; M$ with $P$ a permutation matrix so that the diagonal of $U$ has the dominant components in each row.

Then solving $M x =b$ is equavalent to solving $\left( P \; M \right) x =P \; b$ with LU decomposition of
$P \; M$. That is x solves both systems (no need to permute x).

There is a clever way to make the docomposition faster. This is by initializing
$L=1_{N \times N}$ and $U=M$. Then we have the follwing algorithm for LU decomposition without
pivoting:

```bash

Input: M, N
#M: matrix
#N: size of M

#initialize U
U=M
#initialize L
L=Unit(N,N)

    for k in [2,...,N] do
        for i in [k,...,N] do
            L[i][k-1]=U[i][k-1]/U[k-1][k-1]
            for j in [k-1,...,N] do
                U[i][j]=U[i][j]-L[i][k-1]*U[k-1][j]
            done
        done
    done
```

I will not write the algorithm including pivoting, as the code in python will not be different.

```python
import numpy as np
```

```python
def ind_max(row,N):
    '''
    Find the index of the maximum of a list (row) of lentgth N.
    '''
    _in=0
    _max=row[0]
    i=0
    while i<N:#the end of the row should be included (convension in how I use LUP..)
        if row[i]>_max:
            _max=row[i]
            _in=i
        i+=1

    return _in

def row_swap(A,index_1,index_2,N):
    '''
        row_swap takes a N*N array and interchanges
        row index_1 with row index_2.
    '''
    for i in range(N):
        tmp=A[index_1][i]
        A[index_1][i]=A[index_2][i]
        A[index_2][i]=tmp
```

```python
#This is the same as in the main notebook, but here I use permutation matrix instead of a permutation vector.
#The actual algorithm does not change, as you don't realy care about the definition of the permutation matrix
#or vector.

def LUP(M,N,_tiny=1e-20):
    U=[  [ M[i][j] for j in range(N)] for i in range(N) ]
    L=[  [ 0 if i!=j else 1 for j in range(N)] for i in range(N) ]
    P=[  [ 0 if i!=j else 1 for j in range(N)] for i in range(N) ]

    for k in range(1,N):
        for i in range(k,N):
            #find the index of the maximum in column
            _col=[np.abs(U[_r][k-1]) for _r in range(k-1,N)]

            #find the index of the maximum of _col
            # notice that the length of _col is N-(k-1)
            len_col=N-(k-1)
            pivot=ind_max( _col ,len_col) + k - 1 #convert the index of _col (it has a length of len_col) to the index of  a row of U

            ##################################################
            #this was in LU_julia (instead of "<_tiny"  it had "== 0").
            #if you remove it, then you get a lot of infinities
            #it has to do with the fact that if U[pivot][k-1] <_tiny , then U[k-1][k-1] will be a zero,
            #L[i][k-1] explodes.
            #You are allowed to skip this i, then, because if U[pivot][k-1] <_tiny , then all U[i][k-1] are small!
            #Check that this is true by  uncommenting print(_col)
            if np.abs(U[pivot][k-1]) < _tiny  :
                #print(_col)
                break
            ###################################################
            #if the maximum is not at k-1, swap!
            if pivot != k-1 :
                # Permute rows k-1 and pivot in U
                row_swap(P,k-1,pivot,N)

                tmpU=[U[k-1][_r] for _r in range(k-1,N)]

                #print(U)
                for _r in range(k-1,N):
                    U[k-1][_r]=U[pivot][_r]
                #print(U)
                for _r in range(k-1,N):
                    U[pivot][_r]=tmpU[_r-(k-1)]#again we have to convert the index of tmpU
                #print(U)
                #print("=========================")
                tmpL=[L[k-1][_r] for _r in range(k-1)]
                #print(L)
                for _r in range(k-1):
                    L[k-1][_r]=L[pivot][_r]
                #print(L)
                for _r in range(k-1):
                    L[pivot][_r]=tmpL[_r]

                #print(L)
                #print("========================")

            L[i][k-1]=U[i][k-1]/U[k-1][k-1]

            for j in range(k-1,N):
                U[i][j]=U[i][j]-L[i][k-1]*U[k-1][j]

    return L,U,P

```

```python
def Dot(M,x,N):
    '''
    Product of N*N matrix M with vector x.
    '''
    c=[0 for i in range(N) ]

    for i in range(N):
        for j in range(N):
            c[i]+=M[i][j]*x[j]

    return c

def Sum(List,N):
    '''
    Calculates the sum of a List of size N
    '''
    s=0
    for i in range(N):
        s+=List[i]
    return s
```

```python
def Solve_LU(L,U,P,b,N):
    '''
    This solves P*M*x=P*b (x is also the solution to M*x=b)
    Input:
    L,U,P= LUP decomposition of M. with P*M=L*U
    b=the right hand side of the equation
    N=the number of equations
    '''
    b=Dot(P,b,N)
    d=[0 for i in range(N) ]
    x=[0 for i in range(N) ]

    d[0]=b[0]
    for i in range(1,N):
        d[i]=b[i]-Sum(  [L[i][j]*d[j] for j in range(i)],i )

    x[N-1]  = d[N-1]/U[N-1][N-1]
    for i in range(N-2,-1,-1):
        x[i]=(d[i]-Sum( [U[i][j]*x[j] for j in  range(i+1,N)],N-(i+1) ))/U[i][i]

    return x
```

## tests

```python

```

```python
#check if Solve_LU works

if True:

    NT=500#NT tests
    N=4#N*N matrices
    testSol=[0 for i in range(NT)]

    for i in range(NT):

        #M=np.random.randint(-3,3,size=[N,N])
        b=np.random.rand(N)*13.-6.5
        M=np.random.rand(N,N)*4-2
        L,U,P=LUP(M,N)
        x=Solve_LU(L,U,P,b,N)
        testSol[i]=np.array(Dot(M,x,N))-np.array(b)

    print(np.max(testSol))
```

    3.268496584496461e-13

```python
from scipy.linalg import lu_factor,lu_solve,lu

```

```python
#check LUP against numpy.
#in test I will have the maximum difference between my L,U with what np.lu returns,
#and the difference between my L*U-P*M. So, test should be an array with small numbers!

#even when I get difference with numpy it is not important, because the decomposition is still correct
#(no nan or inf)!!!!

#change to True to run tests
if True:

    NT=500#NT tests
    N=10#N*N matrices
    testL=[0 for i in range(NT)]
    testU=[0 for i in range(NT)]
    testM=[0 for i in range(NT)]

    for i in range(NT):

        #M=np.random.randint(-3,3,size=[N,N])
        M=np.random.rand(N,N)*4-2
        L,U,P=LUP(M,N)
        Ps,Ls,Us=lu(M)

        testU[i]=np.max(np.array(U)-Us)
        testL[i]=np.max(np.array(L)-Ls)

        testM[i]=np.max(np.dot(L,U)-np.dot( P,M) )
        if testL[i] > 1e-5:
            #print(np.array(L))
            #print(Ls)
            #print([U[_t][_t] for _t in range(N)])
            print(testM[i])
            pass

    print(np.max(testU) , np.max(testL) , np.max(testM))
```

    1.865174681370263e-14 6.38378239159465e-15 1.7763568394002505e-15

```python

```

```python

```


# Haverly's Pooling Problem

## Objective and Prerequisites

One of the new features of Gurobi 9.0 is the addition of a bilinear solver, which enables finding the optimal solution of non-convex quadratic programming  problems (i.e. QPs, QCQPs, MIQPs, and MIQCQPs). This notebook will show you how to use this feature by tackling the Haverly's Pooling Problem.

To fully understand the content of this notebook, the reader should be familiar with the following:

- Python.
- Gurobi Python interface.
- Knowledge of building mathematical optimization models.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Motivation

The Pooling Problem is a challenging problem in the petrochemical refining, wastewater treatment and mining industries. This problem can be regarded as a generalization of the minimum-cost flow problem and the blending problem. It is indeed important because of the significant savings it can generate, so it comes at no surprise that it has been studied extensively since Haverly pointed out the non-linear structure of this problem in 1978 [3].

---
## Problem Description

The Minimum-Cost Flow Problem (MCFP) seeks to find the cheapest way of sending a certain amount of flow from a set of source nodes to a set of target nodes, possibily via transshipment nodes, in a directed capacitated network. The Blending Problem is a type of MCFP with only source and target nodes, where raw materials with different attribute qualities are blended together to create end products in such a way that their attribute qualities are within tolerances.

The Pooling Problem combines features of both problems, as flow streams from different sources are mixed at intermediate pools and blended again at the target nodes. The non-linearity is in fact the direct result of considering pools, as the quality of a given attribute at a pool —defined as the weighted average of the qualities of the incoming streams— is an unknown quantity and thus needs to be captured by a decision variable. We refer to this problem as the Standard Pooling Problem when the network can be represented by a tripartite graph, i.e. three disjoint sets of nodes such that no nodes within the same set are adjacent. In a nutshell, it can be stated as follows: Given a list of source nodes with raw materials containing known attribute qualities, what is the cheapest way of mixing these materials at intermediate pools so as to meet the demand and tolerances at multiple target nodes? (Gupte et al., 2017) [2].

---
## Define Data

In this example, there are three sources of oil, each of which contains a different percentage of sulfur. Given the pooling layout, these materials are to be blended in such a way to create two separate oil products. These final products have requirements on the percentage of sulfur they can contain, as well as how much of the product can be produced.

Now, we define the data used in Haverly's 1978 paper:

```python
import numpy as np
import pandas as pd
from itertools import product
from gurobipy import *

attrs = {'sulfur'}

sources, cost, content = multidict({
    "A": [6, {'sulfur': 3}],
    "B": [16, {'sulfur': 1}],
    "C": [10, {'sulfur': 2}]
})

targets, price, demand, max_tol = multidict({
    "X": [9, 100, {'sulfur': 2.5}],
    "Y": [15, 200, {'sulfur': 1.5}]
})

pools = {"P"}

s2p = {("A", "P"),
       ("B", "P")}
# The function `product` deploys the Cartesian product of elements in sets A and B
p2t = set(product(pools, targets))
s2t = {("C", "X"),
       ("C", "Y")}
```

---
## Mathematical Formulation

Mathematical programming is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

- Sets and indices.
- Parameters.
- Decision variables.
- Objective function(s).
- Constraints.

A quadratic constraint that involves only products of disjoint pairs of variables is often called a bilinear constraint, and a model that contains bilinear constraints is often called a Bilinear Program. Bilinear constraints are a special case of non-convex quadratic constraints. This type of problems is typically solved using spatial Branch and Bound (sB&B). This algorithm explores the entire search space, so it provides a globally valid lower bound on the optimal objective value and —given enough time— it will find a globally optimal solution (subject to tolerances). The interested reader is referred to [references](#references) [1], [4] and [5].

We now present a Bilinear Program for the Standard Pooling Problem:

#### Sets and Indices

$G=(V,E)$: Directed graph.

$i,j \in V$: Set of nodes.

$(i,j) \in E \subset  V \times V$: Set of edges.

$N(i)^+ = \{j \in V \mid (i,j) \in E \}$: Set of successor nodes receiving outflow from node $i$.

$N(j)^- = \{i \in V \mid (i,j) \in E \}$: Set of predecessor nodes sending inflow to node $i$.

$k \in \text{Attrs}$: Set of attributes.

$s \in \text{Sources} \subset V$: Set of source nodes, i.e. $N(s)^-= \emptyset$.

$t \in \text{Targets} \subset V$: Set of target nodes, i.e. $N(t)^+= \emptyset$.

$p \in \text{Pools} = V \setminus (\text{Sources} \cup \text{Targets})$: Set of pools.

#### Parameters

$\text{Cost}_s \in \mathbb{R}^+$: Cost of acquiring one unit of raw material at source node $s$.

$\text{Content}_{s,k} \in \mathbb{R}^+$: Content of attribute $k$ in raw material at source node $s$.

$\text{Price}_t \in \mathbb{R}^+$: Price for selling one unit of final blend at target node $t$.

$\text{Demand}_t \in \mathbb{R}^+$: Minimum number of units required of final blend at target node $t$.

$\text{Max_tol}_{t,k} \in \mathbb{R}^+$: Maximum tolerance for attribute $k$ in final blend at target node $t$.

#### Decision Variables

$\text{flow}_{i,j} \in [0, \text{UB}_{i,j}]$: Flow from node $i$ to node $j$.

$\text{quality}_{p,k} \in \mathbb{R}^+$: Concentration of attribute $k$ at pool $p$.

#### Objective Function

- **Profit**: Maximize total profits.

\begin{equation}
\text{Max} \quad Z = \sum_{t \in \text{Targets}}{\sum_{i \in N(t)^-}{\text{Price}_t \cdot \text{flow}_{i,t}}} - \sum_{s \in \text{Sources}}{\sum_{j \in N(s)^+}{\text{Cost}_s \cdot \text{flow}_{s,j}}}
\tag{0}
\end{equation}

#### Constraints

- **Flow conservation**: Total inflow of pool $p$ must be equal to its toal outflow (nothing is stored in them).

\begin{equation}
\sum_{t \in N(p)^+}{\text{flow}_{p,t}} - \sum_{s \in N(p)^-}{\text{flow}_{s,p}} = 0 \quad \forall p \in \text{Pools}
\tag{1}
\end{equation}

- **Target demand**: Total inflow of target $t$ cannot exceed maximum demand.

\begin{equation}
\sum_{i \in N(t)^-}{\text{flow}_{i,t}} \leq \text{Demand}_t \quad \forall t \in \text{Targets}
\tag{2}
\end{equation}

- **Pool concentration**: Concentration of attribute $k$ at pool $p$ is expressed as the weighted average (linear blending) of the concentrations associated to the incoming flows (notice the bilinear terms on the right-hand side).

\begin{equation}
\sum_{s \in N(p)^-}{\text{Content}_{s,k} \cdot \text{flow}_{s,p}} = \text{quality}_{p,k} \cdot \sum_{t \in N(p)^+}{\text{flow}_{p,t}} \quad \forall (p,k) \in \text{Pools} \times \text{Attrs}
\tag{3}
\end{equation}

- **Target tolerances**: Concentration of attribute $k$ at target $t$ is also the result of linear blending, and must be within tolerances (notice the bilinear terms on the second expression of the left-hand side).

\begin{equation}
\sum_{s \in N(t)^- \cap \text{Sources}}{\text{Content}_{s,k} \cdot \text{flow}_{s,t}}+ \sum_{p \in N(t)^- \cap \text{Pools}}{\text{quality}_{p,k} \cdot \text{flow}_{p,t}} \leq \text{Max_tol}_{t,k} \cdot \sum_{i \in N(t)^-}{\text{flow}_{i,t}} \quad \forall (t,k) \in \text{Targets} \times \text{Attrs}
\tag{4}
\end{equation}

---

## Python Implementation

Solving Bilinear Programs with Gurobi is as easy as configuring the global parameter `nonConvex`. When setting this parameter to a value of 2, non-convex quadratic problems are solved by means of translating them into bilinear form and applying spatial Branch and Bound (sB&B).

```python
haverly = Model("Pooling")

# Set global parameters
haverly.params.nonConvex = 2

# Declare decision variables

# flow
ik = haverly.addVars(s2t, name="Source2Target")
ij = haverly.addVars(s2p, name="Source2Pool")
jk = haverly.addVars(p2t, name="Pool2Target")
# quality
prop = haverly.addVars(pools, attrs, name="Proportion")

# Deploy constraint sets

# 1. Flow conservation
haverly.addConstrs((ij.sum('*',j) == jk.sum(j,'*') for j in pools),
                     name="Flow_conservation")
# 2. Target demand
haverly.addConstrs((ik.sum('*',k) + jk.sum('*',k) <= demand[k] for k in targets),
                     name="Target_demand")
# 3. Pool concentration
haverly.addConstrs((quicksum(content[i][attr]*ij[i,j]
                               for i in sources if (i,j) in s2p)
                      == prop[j,attr]*jk.sum(j,'*') for j in pools for attr in attrs),
                     name="Pool_concentration")
# 4. Target (max) tolerances
haverly.addConstrs((quicksum(content[i][attr]*ik[i,k]
                               for i in sources if (i,k) in s2t)
                      + quicksum(prop[j,attr]*jk[j,k]
                                 for j in pools if (j,k) in p2t)
                      <= max_tol[k][attr]*(ik.sum('*',k) + jk.sum('*',k))
                      for k in targets for attr in max_tol[k].keys()),
                     name="Target_max_tolerances")

# Deploy Objective Function

# 0. Total profit
obj = quicksum(price[k]*(ik.sum('*',k) + jk.sum('*',k))
               for k in targets) \
- quicksum(cost[i]*(ij.sum(i,'*') + ik.sum(i,'*'))
           for i in sources)
haverly.setObjective(obj, GRB.MAXIMIZE)

# Find the optimal solution
haverly.optimize()
```

    Using license file /Users/orojuan/gurobi.lic
    Set parameter TokenServer to value Juans-MacBook-Pro-3.local
    Changed value of parameter nonConvex to 2
       Prev: -1  Min: -1  Max: 2  Default: -1
    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (mac64)
    Optimize a model with 3 rows, 7 columns and 8 nonzeros
    Model fingerprint: 0x777f4d01
    Model has 3 quadratic constraints
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      QMatrix range    [1e+00, 1e+00]
      QLMatrix range   [5e-01, 3e+00]
      Objective range  [1e+00, 2e+01]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+02, 2e+02]

    Continuous model is non-convex -- solving as a MIP.

    Found heuristic solution: objective -0.0000000
    Presolve time: 0.00s
    Presolved: 16 rows, 10 columns, 34 nonzeros
    Presolved model has 4 bilinear constraint(s)
    Variable types: 10 continuous, 0 integer (0 binary)

    Root relaxation: objective 2.100000e+03, 4 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 2100.00000    0    3   -0.00000 2100.00000      -     -    0s
         0     0 2100.00000    0    3   -0.00000 2100.00000      -     -    0s
         0     0 1937.83784    0    3   -0.00000 1937.83784      -     -    0s
         0     0 1921.35002    0    4   -0.00000 1921.35002      -     -    0s
         0     0 1921.23235    0    4   -0.00000 1921.23235      -     -    0s
         0     0 1921.17768    0    4   -0.00000 1921.17768      -     -    0s
         0     2 1921.17768    0    4   -0.00000 1921.17768      -     -    0s
    *    6     6               2     321.5204987 1882.20468   485%   3.0    0s
    *   12     4               3     346.2144981 1846.72927   433%   2.4    0s
    *   14     4               4     348.0276050 1846.72927   431%   3.1    0s
    *   22    10               5     355.0833492 1778.85855   401%   3.0    0s
    *   23     5               5     363.4227077 1778.85855   389%   2.9    0s
    *   24     5               6     363.6955948 1778.85855   389%   2.9    0s
    *   25     5               6     370.7666965 1778.85855   380%   2.8    0s
    *   26     5               7     372.0510296 1778.85855   378%   2.8    0s
    *   32     2               7     377.9877375 1746.46493   362%   2.5    0s
    *   34     2               8     380.1494347 1746.46493   359%   2.4    0s
    *   36     4               8     383.8453897 1746.46493   355%   2.5    0s
    *   40     5               9     387.9905769 1715.10036   342%   2.5    0s
    *   41     5               9     390.8479187 1715.10036   339%   2.4    0s
    *   45     5              10     395.5742065 1684.76584   326%   2.4    0s
    *   47     5              10     397.5729772 1684.76584   324%   2.3    0s
    *   52     7              12     399.9188266 1684.76584   321%   2.3    0s
    *   60     2              16     399.9999622 1655.46244   314%   2.2    0s
    *   70     2              18     400.0000031 1627.19132   307%   2.1    0s

    Cutting planes:
      RLT: 3

    Explored 105 nodes (257 simplex iterations) in 0.06 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 10: 400 400 399.919 ... 377.988

    Optimal solution found (tolerance 1.00e-04)
    Best objective 4.000000031377e+02, best bound 4.000000031377e+02, gap 0.0000%

### Analysis

Let's see the optimal flows found:

```python
def _print_table(rows, columns, variables):
    table = pd.DataFrame(columns=columns, index=rows, data=0.0)
    for row, col in variables.keys():
        value = variables[row, col].x
        if abs(value) > 1e-6:
            table.loc[row, col] = np.round(value, 1)
    print(table)

def print_solution():
    print("Flows from Sources to Targets")
    _print_table(sources.copy(), targets.copy(), ik)
    print("Flows from Pools to Targets")
    _print_table(pools.copy(), targets.copy(), jk)
    print("Flows from Sources to Pools")
    _print_table(sources.copy(), pools.copy(), ij)
```

```python
print_solution()
```

    Flows from Sources to Targets
         X      Y
    A  0.0    0.0
    B  0.0    0.0
    C  0.0  100.0
    Flows from Pools to Targets
         X      Y
    P  0.0  100.0
    Flows from Sources to Pools
           P
    A    0.0
    B  100.0
    C    0.0

---
## Changing the Data

We now consider how the optimal solution changes by modifying some parameters in the model.

### Increasing Demand

The maximum demand of product $\text{X}$ increased from 100 to 600:

```python
dem_x = haverly.getConstrByName("Target_demand[X]")
dem_x.setAttr("rhs", 600)
haverly.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (mac64)
    Optimize a model with 3 rows, 7 columns and 8 nonzeros
    Model fingerprint: 0xf2ff1488
    Model has 3 quadratic constraints
    Variable types: 7 continuous, 0 integer (0 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      QMatrix range    [1e+00, 1e+00]
      QLMatrix range   [5e-01, 3e+00]
      Objective range  [1e+00, 2e+01]
      Bounds range     [0e+00, 0e+00]
      RHS range        [2e+02, 6e+02]

    MIP start from previous solve produced solution with objective 324 (0.01s)
    MIP start from previous solve produced solution with objective 526.694 (0.02s)
    MIP start from previous solve produced solution with objective 599.488 (0.02s)
    MIP start from previous solve produced solution with objective 599.976 (0.02s)
    MIP start from previous solve produced solution with objective 600 (0.02s)
    MIP start from previous solve produced solution with objective 600 (0.03s)
    Loaded MIP start from previous solve with objective 600

    Presolve time: 0.00s
    Presolved: 16 rows, 10 columns, 34 nonzeros
    Presolved model has 4 bilinear constraint(s)
    Variable types: 10 continuous, 0 integer (0 binary)

    Root relaxation: objective 3.600000e+03, 4 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 3600.00000    0    3  600.00000 3600.00000   500%     -    0s
         0     0 3600.00000    0    3  600.00000 3600.00000   500%     -    0s
         0     0 3381.81818    0    4  600.00000 3381.81818   464%     -    0s
         0     0 3230.76924    0    3  600.00000 3230.76924   438%     -    0s
         0     0 3217.40285    0    4  600.00000 3217.40285   436%     -    0s
         0     0 3207.95528    0    4  600.00000 3207.95528   435%     -    0s
         0     2 3207.95528    0    4  600.00000 3207.95528   435%     -    0s
    *  126     5              54     600.0000064  600.70747  0.12%   3.5    0s

    Cutting planes:
      RLT: 3

    Explored 141 nodes (472 simplex iterations) in 0.06 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 6: 600 600 599.976 ... 324

    Optimal solution found (tolerance 1.00e-04)
    Best objective 6.000000063584e+02, best bound 6.000000063584e+02, gap 0.0000%

```python
print_solution()
```

    Flows from Sources to Targets
           X    Y
    A    0.0  0.0
    B    0.0  0.0
    C  300.0  0.0
    Flows from Pools to Targets
           X    Y
    P  300.0  0.0
    Flows from Sources to Pools
           P
    A  300.0
    B    0.0
    C    0.0

### Decreasing Cost

The price of extracting crude oil from source $\text{B}$ decreased from $\$16$ to $\$13$:

```python
dem_x.setAttr("rhs", 100) # reinstate the model to its initial state
ij["B", "P"].obj = -13 # the coefficient is negative since we're modifying a cost
haverly.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (mac64)
    Optimize a model with 3 rows, 7 columns and 8 nonzeros
    Model fingerprint: 0x94aace23
    Model has 3 quadratic constraints
    Variable types: 7 continuous, 0 integer (0 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      QMatrix range    [1e+00, 1e+00]
      QLMatrix range   [5e-01, 3e+00]
      Objective range  [1e+00, 2e+01]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+02, 2e+02]

    MIP start from previous solve produced solution with objective -14.0625 (0.01s)
    MIP start from previous solve produced solution with objective -0 (0.02s)
    MIP start from previous solve produced solution with objective 1.88255 (0.03s)
    MIP start from previous solve produced solution with objective 721.662 (0.03s)
    MIP start from previous solve produced solution with objective 726.254 (0.03s)
    Loaded MIP start from previous solve with objective 726.254

    Presolve time: 0.00s
    Presolved: 16 rows, 10 columns, 34 nonzeros
    Presolved model has 4 bilinear constraint(s)
    Variable types: 10 continuous, 0 integer (0 binary)

    Root relaxation: objective 2.100000e+03, 4 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 2100.00000    0    3  726.25404 2100.00000   189%     -    0s
         0     0 2100.00000    0    3  726.25404 2100.00000   189%     -    0s
         0     0 1935.80379    0    3  726.25404 1935.80379   167%     -    0s
         0     0 1919.29441    0    4  726.25404 1919.29441   164%     -    0s
         0     0 1918.94856    0    4  726.25404 1918.94856   164%     -    0s
         0     0 1918.78695    0    4  726.25404 1918.78695   164%     -    0s
         0     2 1918.78695    0    4  726.25404 1918.78695   164%     -    0s
    *    4     6               2     740.2106233 1900.53786   157%   2.8    0s
    *   10     4               3     750.0000000 1844.51703   146%   2.6    0s

    Cutting planes:
      RLT: 3

    Explored 49 nodes (143 simplex iterations) in 0.08 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 7: 750 740.211 726.254 ... -14.0625

    Optimal solution found (tolerance 1.00e-04)
    Best objective 7.500000000000e+02, best bound 7.500000000000e+02, gap 0.0000%

```python
print_solution()
```

    Flows from Sources to Targets
         X    Y
    A  0.0  0.0
    B  0.0  0.0
    C  0.0  0.0
    Flows from Pools to Targets
         X      Y
    P  0.0  200.0
    Flows from Sources to Pools
           P
    A   50.0
    B  150.0
    C    0.0

---
## Conclusion

This notebook showed how easy it is to solve Bilinear Programs using Gurobi.

The Haverly's pooling problem is indeed a simple instance of the Standard Pooling Problem, as it considers only one attribute. It was modeled using what the literature calls the P-Formulation, where the number of bilinear terms is proportional to the number of attributes. For intances with more specifications, it may be worth considering using alternative formulations —such as the Q-Formulation— to improve the performance of the optimization process. The Jupyter Notebook `Standard Pooling Problem` presents and compares both formulations with a more challenging scenario.

---

## References

1. Dombrowski, J. (2015, June 07). McCormick envelopes. Retrieved from https://optimization.mccormick.northwestern.edu/index.php/McCormick_envelopes
2. Gupte, A., Ahmed, S., Dey, S. S., & Cheon, M. S. (2017). Relaxations and discretizations for the pooling problem. Journal of Global Optimization, 67(3), 631-669.
3. Haverly, C. A. (1978). Studies of the behavior of recursion for the pooling problem. Acm sigmap bulletin, (25), 19-28.
4. Liberti, L. (2008). Introduction to global optimization. Ecole Polytechnique.
5. Zhuang E. (2015, June 06). Spatial branch and bound method. Retrieved from
https://optimization.mccormick.northwestern.edu/index.php/Spatial_branch_and_bound_method

Copyright © 2019 Gurobi Optimization, LLC