# Main KernelPyLogit Example

The purpose of this Jupyter notebook is to demonstrate the key functionalities of PyKernelLogit by means of a real case study. Take into account that some of the functionalities described here are inherited from the original PyLogit package, especially those related with the estimation of conditional logit models. This example is divided in several parts.
1. Loading and preprocessing the data. This involves converting a dataframe between 'wide' and 'long' format.
2. Estimating a linear Random Utility model using a conditional logit model.
    1. Definition of a conditional logit model.
    2. Estimation of the previous model.
    3. Evaluation of the estimated model.
    4. Prediction using the estimated model.
    5. Computation of the Willingness to Pay (WTP) indicator using the estimated model.
3. Estimating a non-parametric model using Kernel Logistic Regression (KLR).
    1. Definition of a KLR model.
    2. Estimation of the previous KLR model.
    3. Evaluation of the estimated model.
    4. Prediction using the estimated model.
    5. Computation of the Willingness to Pay (WTP) indicator using estimated KLR model.

The dataset selected for this example is the 'Mode' dataset from the 'mlogit' package for R programming language (https://cran.r-project.org/web/packages/mlogit/mlogit.pdf). This dataset was elaborated by a project team in 'Discrete choice methods with simulation' (Train, 2003) and contains data about the mode choice of $453$ commuters. Each commuter has four available travel modes for their trips to work in a choice set $C=\{ \hbox{}bus, car, carpool, rail  \}$. The time and cost of travel for each mode were determined for each commuter. The attributes of the alternative $i \in C$ for the commuter $n$  are $x_{in}=(x_{in,cost}, x_{in,time})$ .

The first step is to load the different packages that will be used in this example.

```python
from collections import OrderedDict    # For recording the model specification

import pandas as pd                    # For file input/output
import numpy as np                     # For vectorized math operations

import PyKernelLogit as pkl            # For the estimation of the MNL and KLR models.
```

## 1. Loading and preprocessing the data

### 1.1. Load the data
The first step to estimate a model using PyKernelLogit is loading the dataset using a pandas dataframe.

```python
# Load the Mode dataset from a .csv file. Nota that all the elements are delimited by a ',' character.
mode_choice = pd.read_csv("../data/ModeChoice.csv", sep=',')

# Show the first 5 rows of the data
mode_choice.head().T
```

      0
      1
      2
      3
      4

      choice
      car
      rail
      car
      car
      car

      cost.car
      1.50701
      6.057
      5.79468
      1.86914
      2.49895

      cost.carpool
      2.33561
      2.89692
      2.13745
      2.57243
      1.72201

      cost.bus
      1.80051
      2.23713
      2.57638
      1.90352
      2.686

      cost.rail
      2.35892
      1.85545
      2.74748
      2.26828
      2.97387

      time.car
      18.5032
      31.3111
      22.5474
      26.0903
      4.69914

      time.carpool
      26.3382
      34.257
      23.2552
      29.896
      12.4141

      time.bus
      20.8678
      67.1819
      63.3091
      19.7527
      43.092

      time.rail
      30.0335
      60.2931
      49.1716
      13.4727
      39.7433

### 1.2. Convert the dataset to long format

In order to estimate models using the PyKernelLogit library it is necessary to convert the data to "long" format.

Long format has 1 row per individual per available alternative, and wide format has 1 row per individual or observation. Long format is useful because it permits one to directly use matrix dot products to calculate the index, $V_{ij} = x_{ij} \beta$, for each individual $\left(i \right)$ for each alternative $\left(j \right)$. In situations where a dataset is provided in wide format (as in the case of the Mode dataset), it will be necessary to convert the data from wide format to long format.

To convert a dataset to long format, it is needed to specify:

The variables or columns that are specific to a given individual, regardless of what alternative is being considered.

The variables that vary across some or all alternatives, for a given individual (e.g. travel time).

The availability variables.

The unique observation id column.

The choice column.

The following cells of code identify perform the necessary conversions from the wide to the long dataframe.

```python
# Look at the columns of the Mode dataset
mode_choice.columns
```

    Index(['choice', 'cost.car', 'cost.carpool', 'cost.bus', 'cost.rail',
           'time.car', 'time.carpool', 'time.bus', 'time.rail'],
          dtype='object')

```python
# Convert choice from categorical variable to numeric one
mode_choice['choice'] = mode_choice['choice'].map({'car' : 1, 'carpool' : 2 , 'bus' : 3, 'rail' : 4})

# Create the list of individual specific variables
ind_variables = [] # There are no indivicual specific variables in this dataset

# Variables that vary across some or all alternatives, for a given individual
alt_varying_variables = {u'cost': dict([(1, 'cost.car'),
                                        (2, 'cost.carpool'),
                                        (3, 'cost.bus'),
                                        (4, 'cost.rail')]),
                         u'time': dict([(1, 'time.car'),
                                        (2, 'time.carpool'),
                                        (3, 'time.bus'),
                                        (4, 'time.rail')])
                        }

### Specify the availability variables
# Note that the keys of the dictionary are the alternative id's.
# The values are the columns denoting the availability for the
# given mode in the dataset. If the dataset does not contains any
# availability columns (such as in the Mode dataset), then the
# keyword 'available_for_all' can be used to specify that an alternative
# is availables for all individuals.
availability_variables = {1: 'available_for_all',
                          2: 'available_for_all',
                          3: 'available_for_all',
                          4: 'available_for_all'}

### Determine the columns for: alternative ids, the observation ids and the choice

# The 'custom_alt_id' is the name of a column to be created in the long format data
# It will identify the alternative associated with each row.
custom_alt_id = "mode_id"

# Create a custom id column. Note the +1 ensures the id's start at one.
obs_id_column = "custom_id"
mode_choice[obs_id_column] = np.arange(mode_choice.shape[0], dtype=int) + 1

# Create a variable recording the choice column
choice_column = "choice"
```

```python
# Perform the conversion to long format
long_mode_choice = pkl.convert_wide_to_long(mode_choice,
                                           ind_variables,
                                           alt_varying_variables,
                                           availability_variables,
                                           obs_id_column,
                                           choice_column,
                                           new_alt_id_name=custom_alt_id)
```

```python
# Look at the resulting long format dataframe
long_mode_choice.head(10).T
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9

      custom_id
      1.00000
      1.000000
      1.000000
      1.000000
      2.000000
      2.000000
      2.000000
      2.000000
      3.000000
      3.000000

      mode_id
      1.00000
      2.000000
      3.000000
      4.000000
      1.000000
      2.000000
      3.000000
      4.000000
      1.000000
      2.000000

      choice
      1.00000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      1.000000
      0.000000

      cost
      1.50701
      2.335612
      1.800512
      2.358920
      6.056998
      2.896919
      2.237128
      1.855451
      5.794677
      2.137454

      time
      18.50320
      26.338233
      20.867794
      30.033469
      31.311107
      34.256956
      67.181889
      60.293126
      22.547429
      23.255171

### 1.3. Divide the final dataset into a train and test subsets

In machine learning it is a typical practise to divide the data into two sets: the training set and the test set. As these names imply, the model is trained using the training set, and then it is tested using the test set. This is commonly used to obtain the accuracy of the classifier on the new cases (the observations in the test set), which is called generalization error. This value indicates how well the model performs on instances it has never seen before. PyKernelLogit allows to divide a long format dataframe into a train and test subsets.

```python
# Define the percentaje of total observations which are used for training.
train_size = 0.7

(train_set, test_set) = pkl.divide_long_train_test(long_mode_choice, train_size, obs_id_column)
```

## 2. Estimating a linear Random Utility model using a conditional logit model

Firstly, a linear Random Utility model using a conditional logit model will be estimated using the Mode dataset.

### 2.1. Definition of a conditional logit model

Before estimating a model, it is necessary to pre-compute all of the variables that are going to be used in the model. This is different from the functionalities of other package such as mlogit or statsmodels that use formula strings to create new variables. This is also somewhat different from Python Biogeme where new variables can be defined in the script but not actually created by the user before model estimation. PyKernelLogit (as it contains the same internal structure as PyLogit) does not perform variable creation. It only estimates models using variables that already exist.

For the Mode dataset no variable preprocessing was necessary. Moreover, it was decided not to scale any of the variables. No new variables have been defined either.

The first step is to create the model specification being used in this example:

$$
\begin{aligned}
V_{\textrm{Car}, n} &= \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Car}, n} + \\
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Car}, n} \\
        V_{\textrm{Carpool}, n} &= \textrm{ASC Carpool} + \\
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Carpool}, n} + \\
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Carpool}, n} \\
        V_{\textrm{Bus}, n} &= \textrm{ASC Bus} + \\
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Bus}, n} + \\
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Bus}, n} \\
        V_{\textrm{Rail}, n} &= \textrm{ASC Rail} + \\
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Rail}, n} + \\
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Rail}, n}
\end{aligned}
$$

Note that packages such as mlogit and statsmodels do not, by default, handle coefficients that vary over some alternatives but not all. PyKernelLogit does have this capability.

In order to specify a linear model in PyKernelLogit it is necessary to create two ordered dictionaries:

* Specification ordered dictionary. It contains the desired PyKernelLogit model specification. The dictionary keys should be variables within the long format dataframe (i.e. the column names from the dataframe), whereas, their associate value should be a list of integers, where the integers are the alternative ID's of the alternative whose utility specification the explanatory variable is entering. In other words, for each integer $i$ in the list, a $\beta_{ik}$ parameter is created, which multiplies the attribute $x_{ink}$ selected on the ordered dictionary key. It is also possible to use the same $\beta_{k}$ parameter for more than one alternative. In this case, a nested list should be added containing the alternative ID's of the alternatives that will share the common parameter.

* Variable names ordered dictionary. This order dictionary should contain the same keys that the Specification ordered dictionary. The value should be a list with the names for the $\beta_{k}$ parameters that will be estimated for each of the elements within the outer-most list on the Specification ordered dictionary.

The keys of both ordered dictionaries should be variables from the long format dataframe with the sole exception of the intercept key. The value associated with this key is a list with the alternative ID's for which an intercept will be added in the model specification.

```python
# Create an empty basic_specification and basic_names ordered dictionary
basic_specification = OrderedDict()
basic_names = OrderedDict()

# Model specification
basic_specification["intercept"] = [2, 3, 4]
basic_names["intercept"] = ['ASC CarPool',
                            'ASC Bus',
                            'ASC Rail']

basic_specification["cost"] = [[1, 2, 3, 4]]
basic_names["cost"] = ['Travel cost (dollars)']

basic_specification["time"] = [[1, 2, 3, 4]]
basic_names["time"] = ['Travel time (minutes)']
```

### 2.2. Estimation of the previous model

Once the model has been specified, it is necessary to estimate its parameters. PyKernelLogit allows to estimate the previous model using Maximum Likelihood Estimation (MLE) or Penalized Maximum Likelihood Estimation (PMLE). In this example, as the model is not complex, MLE is used.

```python
# Estimate the multinomial logit model (MNL)
linear_model = pkl.create_choice_model(data=train_set,
                                           alt_id_col=custom_alt_id,
                                           obs_id_col=obs_id_column,
                                           choice_col=choice_column,
                                           specification=basic_specification,
                                           model_type="MNL",
                                           names=basic_names)
```

The previous model can be estimated using the fit_mle() method. This method is designed to estimate Multinomial Logit Models and receives several parameters:
* init_vals. The initial values of $\boldsymbol{\beta}$ (RUM) or $\boldsymbol{\alpha}$ (KLR) to start the optimization process with. It should be a vector with one value for each utility coefficient being estimated.

* method. It should be a valid string that can be passed to scipy.optimize.minimize. Determines the optimization algorithm that is used for this problem. The default newton-cg which uses the Newton-CG algorithm.

* PMLE. It determines if a Penalized Maximum Likelihood Estimation should be executed. The value of the parameter determines the type of PMLE and the possible values are LASSO and RIDGE.

* PMLE_lambda. Lambda parameter for LASSO or ridge regression. It should be an int, float or long and determines the penalty parameter for the PMLE.

```python
# Fit the linear model using MLE
linear_model.fit_mle(np.zeros(5), method='BFGS')
```

    Log-likelihood at zero: -439.4553
    Initial Log-likelihood: -439.4553
    Estimation Time for Point Estimation: 0.01 seconds.
    Final log-likelihood: -249.6988

### 2.3. Evaluation of the estimated model

Once the model has been estimated, the next step is to asses the performance of the model.

In machine learning, there are several alternatives for assessing whether a model is good enough for the specific application for which the model is going to be used. The goodness of fit of a model is a statistic which descries how well a model, with its estimated parameters, performs compared with a model in which all its parameters are zero (which is usually equivalent to having no model at all, i.e. in the discrete choice context is equivalent to a model for which the probability of choosing an alternative is the same for all the alternatives). This comparison is made using the log-likelihood function, which is evaluated at both models, one with estimated parameters and other with all its parameters equal zero. The likelihood ratio index (also called McFadden R square) is defined as:
\begin{equation}
    \label{eq:ro-square}
    \rho^2 = 1-\frac{LL(\hat{\boldsymbol{\beta}})}{LL(\boldsymbol{0})}.
\end{equation}

This measure has been derived to mimic the $R^2$ used in linear regression, but in this case its interpretation is not similar at all. If the estimated model has the same log-likelihood as an equal probability model, then $\rho^2 = 0$. On the contrary, if the estimated model perfectly fits the data ($LL(\hat{\boldsymbol{\beta}}) = 0$), then $\rho^2 = 1$. This value, unlike $R^2$, cannot be interpreted, and it must only be used to compare between two models estimated using the same sample. For this reason, two models estimated on different samples cannot be compared via their likelihood ratio index values.

It presents another disadvantage because this measure is monotonic in the number of parameters of the model, which means that this index increase each time an additional variable is added to the model, even if it does not explain anything. There is a variant of the $\rho^2$ index which offer a corrected measure:
\begin{equation}
    \label{eq:pseudo-ro-bar-square}
    \overline{\rho}^{2} = 1-\frac{LL(\hat{\boldsymbol{\beta}})-\nu}{LL(0)},
\end{equation}
where $\nu$ is the number of parameters in the model.

When a model is estimated, PyKernelLogit allows to obtain a summary of the results by using the function get_statsmodels_summary. This function integrates both, $\rho^2$ and $\overline{\rho}^{2}$ goodness of fit measures, which are called here as Pseudo R-squ. and Pseudo R-bar-squ., respectively.

```python
# Obtain a summary of the results for the linear model
linear_model.get_statsmodels_summary()
```

Multinomial Logit Model Regression Results

  Dep. Variable:	         choice	            No. Observations:  	    317

  Model:	         Multinomial Logit Model	   Df Residuals:      	    312

  Method:	                  MLE	             Df Model:          	     5

  Date:	             Fri, 28 Feb 2020	       Pseudo R-squ.:     	   0.432

  Time:	                 14:15:35	           Pseudo R-bar-squ.: 	   0.420

  AIC:	                   509.398	           Log-Likelihood:    	 -249.699

  BIC:	                   528.192	           LL-Null:           	 -439.455

            	               coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  ASC CarPool	              -4.1890	     0.470	    -8.915	  0.000	    -5.110	    -3.268

  ASC Bus	                  -3.4403	     0.390	    -8.819	  0.000	    -4.205	    -2.676

  ASC Rail	                 -2.7398	     0.354	    -7.745	  0.000	    -3.433	    -2.046

  Travel cost (dollars)	    -0.7903	     0.112	    -7.077	  0.000	    -1.009	    -0.571

  Travel time (minutes)	    -0.0829	     0.009	    -9.051	  0.000	    -0.101	    -0.065

These goodness of fit measures cannot measure how the model predict on new data points, for which it has not yet seen the true value of the dependent variable (the alternative chosen). Therefore, some other model validation techniques are available on the PyKernelLogit package.

A much better way to evaluate the performance of a classifier is to look at the so-called confusion matrix. The idea is to count the number of instances of class A which are classified as class B. The confusion matrix can be computed as follows.

```python
# Compute the confunsion matrix using the test set
confusion_matrix = linear_model.confusion_matrix(test_set)

# The generated confusion matrix can be shown directly as a pandas dataframe
confusion_matrix
```

      1
      2
      3
      4

      1
      56
      3
      2
      3

      2
      3
      0
      0
      4

      3
      8
      0
      9
      9

      4
      11
      0
      5
      23

```python
# Another option is to use the heatmap method of the seaborn package to
# visualize the confusion matrix. This generates more elegant ilustrations.

%matplotlib inline

# Import seaborn and matplotlib
import seaborn as sns
import matplotlib.pyplot as plt

# Generate a new figure and set the figure size.
plt.figure(figsize=(10,5))

# Generate a heatmap for the confusion matrix using seaborn
ax = sns.heatmap(confusion_matrix, annot=True, linewidth=0.5, cmap='PuBu')

# labels, title and ticks
ax.set_xlabel('Predicted alternative')
ax.set_ylabel('True alternative')
ax.set_title('Confusion Matrix')

# Figures can be stored as images by means of ax.figure.savefig() method.
```

As it is shown, the confusion matrix gives lot of useful information, but sometimes it is preferred a more concise metric. An other interesting way of assessing the performance of a model is to look the accuracy of the positive predictions, which is called the precision.

This metric has an important drawback because if the model makes a single positive prediction and it is correct, then the precision will be 1/1 = 100\%. This results would not be useful since the classifier would ignore all the predictions except for the only one positive instance. For that reason, the precision metric is commonly used along with another metric called recall, or sensitivity. The recall is the ratio of positive instances that are correctly detected by the classifier.

Finally, there is a third metric called F-score, which combines the precision and the recall results to compute the score. The F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0. The $\beta$ parameter determines the weight of recall in the combined score. When $\beta < 1$, the F-score lends more weight to precision, while $\beta > 1$ favors recall. Moreover, $\beta \rightarrow 0$ considers only the precision and $\beta \rightarrow \infty$ only the recall.

This metrics can be applied directly to problems where only two alternatives are considered. For problems with more than two alternatives an averaging technique should be applied. PyKernelLogit offers two averaging techniques:
* Macro-averaging.
* Micro-averaging.

The precision, recall and F-score metrics can be computed for the previous model using the following command.

```python
# Compute the precision, recall and F1-score for the linear model using a
# Micro-averaging technique. The F1-score is the F-score for beta = 1.
linear_model.precision_recall_fscore(test_set, "micro", beta = 1)
```

    (0.6470588235294118, 0.6470588235294118, 0.6470588235294118)

### 2.4. Prediction using the estimated model

Once the model has been estimated, new instances can be predicted using the model. The next block of code shows how to create a pandas dataframe which contains a matrix with the probabilities predicted for each alternative for a given observation.

```python
# The different observations from the test_set are represented on each row.
# The columns represent each of the alternatives. The value from each cell
# is the probability that the alternative on the corresponding column was
# selected on that observation.
pd.DataFrame(data=linear_model.predict(test_set).reshape(-1, 4))
```

      0
      1
      2
      3

      0
      0.963694
      0.003965
      0.020135
      0.012206

      1
      0.838091
      0.005317
      0.044212
      0.112380

      2
      0.981976
      0.014513
      0.001126
      0.002385

      3
      0.590096
      0.086270
      0.042587
      0.281048

      4
      0.148308
      0.019157
      0.082536
      0.749999

      ...
      ...
      ...
      ...
      ...

      131
      0.216609
      0.103513
      0.087728
      0.592150

      132
      0.777684
      0.017586
      0.056032
      0.148698

      133
      0.781351
      0.117042
      0.066393
      0.035214

      134
      0.767810
      0.031292
      0.009875
      0.191023

      135
      0.880017
      0.053360
      0.050753
      0.015870

136 rows × 4 columns

### 2.5. Computation of the Willingness to Pay (WTP) indicator using the estimated model

There are many indicators that are particularly relevant in the context of discrete choice models, such as, Willingness to Pay (WTP) and the Value of Time (VOT). If the estimated model contains a cost or price variable, which indicates the total cost of the alternative for a given decision-maker, it is possible to analyze the trade-off between any other variable form the model and money. This trade-off reflects the **willingness of the decision-maker to pay** for a modification of one unit of another variable of the model.

Lets consider that $c_{in}$ is the cost of the alternative $i$ for a decision-maker $n$, and $x_{in}$ is the value of another variable. Now, let $V_{in}(c_{in}, x_{in})$ be the value of the utility function that is associated to an alternative $i$ for a decision-maker $n$. It can be considered a scenario where the variable under interest takes the value $x_{in} + \delta^{x}_{in}$, where $\delta^{x}_{in}$ is an increment over the value of the $x$ variable. It is denoted by $\delta^{c}_{in}$ the additional cost increment that would be necessary to achieve the same utility as if no increment were made over $x$ variable, that is
\begin{equation}
    \label{eq:WTP-same-utility}
    V_{i n}\left(c_{i n}+\delta_{i n}^{c}, x_{i n}+\delta_{i n}^{x}\right)=V_{i n}\left(c_{i n}, x_{i n}\right).
\end{equation}

If $x_{in}$ and $c_{in}$ appear linearly in the utility function, that is if
$V_{i n}\left(c_{i n}, x_{i n}\right)=\beta_{c} c_{i n}+\beta_{x} x_{i n}+\cdots$
Then, the WTP can be calculated directly using the coefficients ratio as:
\begin{equation}
    WTP = \frac{\delta_{i n}^{c}}{\delta_{i n}^{x}}=-\frac{\left(\partial V_{i n} / \partial x_{i n}\right)\left(c_{i n}, x_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, x_{i n}\right)}=-\frac{\beta_{x}}{\beta_{c}}
\end{equation}
where $\beta_{c}$ is the calculated coefficient for the cost.

The above equation is the willingness to pay for an increase of the value of
the variable $x_{in}$. Therefore, if this increase improves the utility of the alternative, then $\beta_{x}>0$.

Another concept is the **Value of Time (VOT)**, which is closely related with the previous one, as it reflects the price that a traveler is willing to pay to decrease the travel time in one unit. Hence, the formula for the VOT is:
\begin{equation}
    \mathrm{VOT}_{i n}=\delta_{i n}^{c} /\left(-\delta_{i n}^{t}\right)=\frac{\left(\partial V_{i n} / \partial t_{i n}\right)\left(c_{i n}, t_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, t_{i n}\right)}
\end{equation}

As $V$ is linear in the cost and time parameters, we have
\begin{equation}
    \mathrm{VOT}_{i n}=\delta_{i n}^{c} /\left(-\delta_{i n}^{t}\right)=\frac{\beta_{t}}{\beta_{c}}
\end{equation}

In this example, it is interesting to compute the **VOT** as the model contains the cost and time variables.

```python
# To compute the VOT for the linear model, we need to apply the previous equation.
# Take into account that travel time is in minutes, therefore beta_t = 1/minutes,
# which is equal as beta_t = 60 * 1/hours. Now, VOT is measured in dollars/hours.

(linear_model.coefs['Travel time (minutes)']*60) / linear_model.coefs['Travel cost (dollars)']
```

    6.293743644982097

## 3. Estimating a non-parametric model using Kernel Logistic Regression (KLR)

Now, a non-parametric model is estimated using KLR capabilities from PyKernelLogit.

### 3.1. Definition of a KLR model

The first step is to consider if the data should be preprocessed. It is a good practice to scale the data before a KLR model is applied. Skcikit-learn Python package can be used to scale the data.

```python
# Scale the data
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

scaler.fit(train_set[['cost', 'time']])

train_set_scaled = train_set.copy()
train_set_scaled[['cost', 'time']] = scaler.transform(train_set[['cost', 'time']])

test_set_scaled = test_set.copy()
test_set_scaled[['cost', 'time']] = scaler.transform(test_set[['cost', 'time']])
```

The following step to define a KLR model is to compute the kernel matrix, or Gram matrix, using the input dataframe. The kernel_matrix function returns a list composed of one kernel matrix $\mathbf{K}^{(i)}$ per alternative $i$. This function takes the following input values:
* X. The dataset in long format for which the kernel matrix is computed, i.e. $\mathbf{X}_{i}$.

* alt_id_col. The name for the alternative ID column in the long dataframe.

* obs_id_col. The column from the wide dataset that contains the ID of each observation.

* variables. A Python dictionary where each key represents each of the alternatives IDs and its associated value contains a list of characteristics from the long dataframe X to be used on that alternative.

* kernel_type. The type of kernel function to be used. The kernel_matrix function relies on the scikit-learn Python package for the kernel functions. The value passed on the kernel_type parameter specifies the scikit-learn kernel function which is going to be used to compute the kernel matrix.

* Z. The kernel_matrix function also allows the user to specify a reference matrix $\mathbf{Z}_{i}$ to be used when computing the kernel matrix. If no Z dataframe is specified, then the kernel matrix is computed taking as reference matrix the input matrix X.

* \*\*kernel\_kwargs. Keyworded variable length of arguments to be provided to the invoked scikit-learn kernel function.

```python
variables = {1: ['cost', 'time'],
             2: ['cost', 'time'],
             3: ['cost', 'time'],
             4: ['cost', 'time']}

K_long_format_train = pkl.long_format_with_kernel_matrix(train_set_scaled,
                                                         custom_alt_id,
                                                         obs_id_column,
                                                         choice_column,
                                                         variables,
                                                         kernel_type = "RBF")
```

The next step is to define the KLR model specification. Similarly to the linear model specification, it is necessary to create two ordered dictionaries:
* Specification ordered dictionary.
* Variable names ordered dictionary.

The PyKernelLogit function define_kernel_specification generates automatically the specification and variable names ordered dictionaries for the desired kernel model using the specification attributes provided through its input parameters:
* K_long_format. A long format dataframe that contains the kernel matrix for all the alternatives in the dataset. This dataframe should be obtained using the long_format_with_kernel_matrix function.

* alt_id_col. The name for the alternative ID column in the long dataframe.

* obs_id_col. The column from the wide dataset that contains the ID of each observation.

* specification. A previously defined specification ordered dictionary where the kernel logistic model specification will be inserted. It is recommended to pass an empty ordered dictionary.

* names. A previously defined variable names ordered dictionary where it will be inserted the names for the $\boldsymbol{\alpha}_i$ parameters to be estimated. It is recommended to pass an empty ordered dictionary.

* alpha_per_alternative. A boolean variable. If this this variable is True then one vector of parameters to estimate $\boldsymbol{\alpha}_i$ is used per alternative $i$. Otherwise, the same vector of parameters $\boldsymbol{\alpha}$ is used for all the alternatives. This generates a more restricted model, but this model will have much less parameters to be estimated. The default value is False.

* intercept. If this parameter is None (the default value), then no intercept parameter is added to the model. If this parameter contains an integer value, then an intercept parameter is added to the model. The intercept parameter must be the alternative ID of the alternative selected to be the reference alternative. No intercept is generated for the reference alternative because it is considered to be zero.

This method returns a tuple with 3 elements: the basic_specification and basic_names ordered dictionaries and the total number of variables to be estimated in the model.

The following code fragment is used to define the basic_specification and basic_names ordered dictionaries for a KLR model with the same $\boldsymbol{\alpha}$ vector of parameters for all the alternatives and an intercept parameter.

```python
[basic_specification, basic_names, total_vars] = pkl.define_kernel_specification(K_long_format_train,
                                                                                custom_alt_id,
                                                                                obs_id_column,
                                                                                specification=OrderedDict(),
                                                                                names=OrderedDict(),
                                                                                alpha_per_alternative=False,
                                                                                intercept=1)
```

### 3.2. Estimation of the previous KLR model

Once the KLR model has been specified, it is necessary to estimate its parameters in a similar way that the previous linear model.

```python
# Estimate the multinomial logit model (MNL)
kernel_model = pkl.create_choice_model(data=K_long_format_train,
                                       alt_id_col=custom_alt_id,
                                       obs_id_col=obs_id_column,
                                       choice_col=choice_column,
                                       specification=basic_specification,
                                       model_type="MNL",
                                       names=basic_names)
```

The previous model can be also estimated using the fit_mle() method. Now, instead of using Maximum Likelihood Estimation (MLE), it was decided to use Penalized Maximum Likelihood Estimation (PMLE) due to the high number of parameters. It was decided to use a Ridge regression with a penalty parameter $\lambda$ = 4. Moreover,  L-BFGS-B optimization method was used to estimate the parameters because, although it obtains slightly worse results than BFGS method, the parameters are estimated remarkably fast as it is designed to optimize a large number of parameters using a limited amount of memory.

```python
# Specify the initial values and method for the optimization.
kernel_model.fit_mle(np.zeros(total_vars), method="L-BFGS-B", PMLE="RIDGE", PMLE_lambda=4)
```

    Log-likelihood at zero: -439.4553
    Initial Log-likelihood: -439.4553
    Estimation Time for Point Estimation: 0.03 seconds.
    Final log-likelihood: -277.9502

    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/conditional_logit.py:376: UserWarning: NOTE: A penalized regression is being performed. The reported standard errors and robust standard errors ***WILL BE INCORRECT***.
      warnings.warn(_ridge_warning_msg)
    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/base_multinomial_cm_v2.py:1231: RuntimeWarning: invalid value encountered in sqrt
      self._store_inferential_results(np.sqrt(np.diag(self.cov)),
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in greater
      return (a < x) & (x < b)
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in less
      return (a < x) & (x < b)
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:1912: RuntimeWarning: invalid value encountered in less_equal
      cond2 = cond0 & (x <= _a)
    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/base_multinomial_cm_v2.py:1262: RuntimeWarning: invalid value encountered in sqrt
      self._store_inferential_results(np.sqrt(np.diag(self.robust_cov)),

### 3.3. Evaluation of the estimated model

Once the model has been estimated, the performance of the model can be assessed such as in the linear model. Therefore, get_statsmodels_summary() can be used to obtain the goodness of fit of the model and the estimated parameters.

```python
# Look at the estimation results
kernel_model.get_statsmodels_summary()
```

Multinomial Logit Model Regression Results

  Dep. Variable:	         choice	            No. Observations:  	    317

  Model:	         Multinomial Logit Model	   Df Residuals:      	    -3

  Method:	                  MLE	             Df Model:          	    320

  Date:	             Fri, 28 Feb 2020	       Pseudo R-squ.:     	   0.368

  Time:	                 14:15:37	           Pseudo R-bar-squ.: 	  -0.361

  AIC:	                  1,195.900	          Log-Likelihood:    	 -277.950

  BIC:	                  2,398.749	          LL-Null:           	 -439.455

      	         coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  ASC 2	        -0.7932	   843.246	    -0.001	  0.999	 -1653.526	  1651.939

  ASC 3	        -0.5352	   925.261	    -0.001	  1.000	 -1814.013	  1812.942

  ASC 4	        -0.0256	   827.401	  -3.1e-05	  1.000	 -1621.702	  1621.651

  alpha_1	      -0.1119	       nan	       nan	    nan	       nan	       nan

  alpha_2	      -0.0724	       nan	       nan	    nan	       nan	       nan

  alpha_3	       0.1261	       nan	       nan	    nan	       nan	       nan

  alpha_4	       0.0074	  1.33e+06	  5.59e-09	  1.000	 -2.61e+06	  2.61e+06

  alpha_5	       0.0525	       nan	       nan	    nan	       nan	       nan

  alpha_6	       0.0348	   9.4e+06	   3.7e-09	  1.000	 -1.84e+07	  1.84e+07

  alpha_7	      -0.0053	  7.93e+06	 -6.67e-10	  1.000	 -1.55e+07	  1.55e+07

  alpha_8	      -0.1580	       nan	       nan	    nan	       nan	       nan

  alpha_9	      -0.0290	   3.9e+06	 -7.44e-09	  1.000	 -7.64e+06	  7.64e+06

  alpha_10	     -0.2071	  5.64e+06	 -3.67e-08	  1.000	 -1.11e+07	  1.11e+07

  alpha_11	      0.0867	  5.19e+06	  1.67e-08	  1.000	 -1.02e+07	  1.02e+07

  alpha_12	      0.1213	  7.52e+05	  1.61e-07	  1.000	 -1.47e+06	  1.47e+06

  alpha_13	      0.0644	  8.12e+06	  7.94e-09	  1.000	 -1.59e+07	  1.59e+07

  alpha_14	      0.2021	  9.17e+05	   2.2e-07	  1.000	  -1.8e+06	   1.8e+06

  alpha_15	      0.0202	  9.52e+06	  2.12e-09	  1.000	 -1.87e+07	  1.87e+07

  alpha_16	     -0.0955	  9.07e+06	 -1.05e-08	  1.000	 -1.78e+07	  1.78e+07

  alpha_17	      0.0715	       nan	       nan	    nan	       nan	       nan

  alpha_18	     -0.1768	  6.45e+06	 -2.74e-08	  1.000	 -1.26e+07	  1.26e+07

  alpha_19	     -0.0726	       nan	       nan	    nan	       nan	       nan

  alpha_20	     -0.1114	  3.35e+06	 -3.33e-08	  1.000	 -6.56e+06	  6.56e+06

  alpha_21	      0.0443	  7.08e+06	  6.25e-09	  1.000	 -1.39e+07	  1.39e+07

  alpha_22	     -0.0446	   1.6e+06	 -2.79e-08	  1.000	 -3.14e+06	  3.14e+06

  alpha_23	     -0.0163	       nan	       nan	    nan	       nan	       nan

  alpha_24	      0.0758	  3.61e+06	   2.1e-08	  1.000	 -7.07e+06	  7.07e+06

  alpha_25	      0.0586	  3.68e+06	  1.59e-08	  1.000	 -7.22e+06	  7.22e+06

  alpha_26	     -0.0488	       nan	       nan	    nan	       nan	       nan

  alpha_27	     -0.1262	  4.75e+06	 -2.66e-08	  1.000	 -9.31e+06	  9.31e+06

  alpha_28	      0.0352	  4.83e+06	  7.29e-09	  1.000	 -9.47e+06	  9.47e+06

  alpha_29	     -0.0126	  7.15e+06	 -1.76e-09	  1.000	  -1.4e+07	   1.4e+07

  alpha_30	      0.1004	   7.1e+06	  1.42e-08	  1.000	 -1.39e+07	  1.39e+07

  alpha_31	      0.1157	  2.27e+06	  5.09e-08	  1.000	 -4.45e+06	  4.45e+06

  alpha_32	     -0.0167	       nan	       nan	    nan	       nan	       nan

  alpha_33	     -0.2332	  3.58e+06	 -6.51e-08	  1.000	 -7.01e+06	  7.01e+06

  alpha_34	     -0.1810	       nan	       nan	    nan	       nan	       nan

  alpha_35	      0.0023	  2.79e+06	  8.41e-10	  1.000	 -5.46e+06	  5.46e+06

  alpha_36	      0.0215	       nan	       nan	    nan	       nan	       nan

  alpha_37	      0.0613	       nan	       nan	    nan	       nan	       nan

  alpha_38	     -0.0230	   4.6e+06	 -4.99e-09	  1.000	 -9.02e+06	  9.02e+06

  alpha_39	      0.1023	  3.97e+06	  2.57e-08	  1.000	 -7.79e+06	  7.79e+06

  alpha_40	      0.2244	       nan	       nan	    nan	       nan	       nan

  alpha_41	      0.0752	       nan	       nan	    nan	       nan	       nan

  alpha_42	     -0.0097	       nan	       nan	    nan	       nan	       nan

  alpha_43	     -0.0938	       nan	       nan	    nan	       nan	       nan

  alpha_44	      0.1832	  3.83e+06	  4.79e-08	  1.000	  -7.5e+06	   7.5e+06

  alpha_45	     -0.2462	       nan	       nan	    nan	       nan	       nan

  alpha_46	      0.0980	  5.29e+06	  1.85e-08	  1.000	 -1.04e+07	  1.04e+07

  alpha_47	     -0.1189	       nan	       nan	    nan	       nan	       nan

  alpha_48	     -0.0588	       nan	       nan	    nan	       nan	       nan

  alpha_49	      0.0069	   5.3e+06	   1.3e-09	  1.000	 -1.04e+07	  1.04e+07

  alpha_50	     -0.0620	       nan	       nan	    nan	       nan	       nan

  alpha_51	     -0.0952	       nan	       nan	    nan	       nan	       nan

  alpha_52	     -0.3174	       nan	       nan	    nan	       nan	       nan

  alpha_53	     -0.1900	  2.59e+06	 -7.35e-08	  1.000	 -5.07e+06	  5.07e+06

  alpha_54	     -0.0395	  5.76e+06	 -6.84e-09	  1.000	 -1.13e+07	  1.13e+07

  alpha_55	     -0.0124	  1.43e+06	 -8.66e-09	  1.000	  -2.8e+06	   2.8e+06

  alpha_56	      0.0650	       nan	       nan	    nan	       nan	       nan

  alpha_57	      0.0311	       nan	       nan	    nan	       nan	       nan

  alpha_58	      0.0825	  1.88e+06	   4.4e-08	  1.000	 -3.67e+06	  3.67e+06

  alpha_59	     -0.1329	  3.39e+06	 -3.92e-08	  1.000	 -6.65e+06	  6.65e+06

  alpha_60	     -0.0867	       nan	       nan	    nan	       nan	       nan

  alpha_61	      0.0450	       nan	       nan	    nan	       nan	       nan

  alpha_62	     -0.0297	       nan	       nan	    nan	       nan	       nan

  alpha_63	      0.0498	       nan	       nan	    nan	       nan	       nan

  alpha_64	     -0.0822	       nan	       nan	    nan	       nan	       nan

  alpha_65	     -0.1865	  4.45e+06	 -4.19e-08	  1.000	 -8.73e+06	  8.73e+06

  alpha_66	      0.0331	  7.61e+06	  4.35e-09	  1.000	 -1.49e+07	  1.49e+07

  alpha_67	      0.0592	  5.14e+06	  1.15e-08	  1.000	 -1.01e+07	  1.01e+07

  alpha_68	      0.0697	       nan	       nan	    nan	       nan	       nan

  alpha_69	     -0.0273	  1.99e+06	 -1.37e-08	  1.000	  -3.9e+06	   3.9e+06

  alpha_70	      0.2125	       nan	       nan	    nan	       nan	       nan

  alpha_71	      0.0106	       nan	       nan	    nan	       nan	       nan

  alpha_72	     -0.0717	       nan	       nan	    nan	       nan	       nan

  alpha_73	     -0.0563	       nan	       nan	    nan	       nan	       nan

  alpha_74	     -0.0099	       nan	       nan	    nan	       nan	       nan

  alpha_75	     -0.0957	  2.48e+06	 -3.86e-08	  1.000	 -4.85e+06	  4.85e+06

  alpha_76	      0.1815	       nan	       nan	    nan	       nan	       nan

  alpha_77	      0.1622	       nan	       nan	    nan	       nan	       nan

  alpha_78	     -0.0225	       nan	       nan	    nan	       nan	       nan

  alpha_79	     -0.1379	       nan	       nan	    nan	       nan	       nan

  alpha_80	     -0.0905	       nan	       nan	    nan	       nan	       nan

  alpha_81	      0.0599	  3.21e+06	  1.87e-08	  1.000	 -6.28e+06	  6.28e+06

  alpha_82	     -0.1238	       nan	       nan	    nan	       nan	       nan

  alpha_83	     -0.0421	  3.89e+06	 -1.08e-08	  1.000	 -7.63e+06	  7.63e+06

  alpha_84	     -0.1254	       nan	       nan	    nan	       nan	       nan

  alpha_85	     -0.1435	   2.4e+06	 -5.98e-08	  1.000	 -4.71e+06	  4.71e+06

  alpha_86	      0.0636	       nan	       nan	    nan	       nan	       nan

  alpha_87	     -0.0943	  5.46e+06	 -1.73e-08	  1.000	 -1.07e+07	  1.07e+07

  alpha_88	     -0.0520	       nan	       nan	    nan	       nan	       nan

  alpha_89	      0.0431	       nan	       nan	    nan	       nan	       nan

  alpha_90	     -0.0021	  1.77e+06	 -1.21e-09	  1.000	 -3.47e+06	  3.47e+06

  alpha_91	     -0.1926	       nan	       nan	    nan	       nan	       nan

  alpha_92	      0.1309	  3.14e+06	  4.17e-08	  1.000	 -6.15e+06	  6.15e+06

  alpha_93	     -0.1069	  4.75e+06	 -2.25e-08	  1.000	 -9.31e+06	  9.31e+06

  alpha_94	     -0.0025	       nan	       nan	    nan	       nan	       nan

  alpha_95	     -0.0657	       nan	       nan	    nan	       nan	       nan

  alpha_96	      0.0782	  9.13e+06	  8.56e-09	  1.000	 -1.79e+07	  1.79e+07

  alpha_97	     -0.1390	       nan	       nan	    nan	       nan	       nan

  alpha_98	     -0.2913	  4.43e+06	 -6.57e-08	  1.000	 -8.69e+06	  8.69e+06

  alpha_99	     -0.1440	  4.52e+06	 -3.19e-08	  1.000	 -8.86e+06	  8.86e+06

  alpha_100	    -0.0950	  2.27e+06	 -4.18e-08	  1.000	 -4.46e+06	  4.46e+06

  alpha_101	    -0.1456	       nan	       nan	    nan	       nan	       nan

  alpha_102	     0.0483	  9.61e+06	  5.02e-09	  1.000	 -1.88e+07	  1.88e+07

  alpha_103	    -0.2813	  2.85e+06	 -9.88e-08	  1.000	 -5.58e+06	  5.58e+06

  alpha_104	    -0.1262	  3.07e+06	 -4.11e-08	  1.000	 -6.02e+06	  6.02e+06

  alpha_105	    -0.1589	  8.88e+05	 -1.79e-07	  1.000	 -1.74e+06	  1.74e+06

  alpha_106	     0.0296	  1.96e+06	  1.51e-08	  1.000	 -3.84e+06	  3.84e+06

  alpha_107	    -0.0162	  6.84e+06	 -2.37e-09	  1.000	 -1.34e+07	  1.34e+07

  alpha_108	     0.0208	   2.4e+06	  8.66e-09	  1.000	 -4.71e+06	  4.71e+06

  alpha_109	     0.1239	       nan	       nan	    nan	       nan	       nan

  alpha_110	     0.0235	       nan	       nan	    nan	       nan	       nan

  alpha_111	     0.0044	       nan	       nan	    nan	       nan	       nan

  alpha_112	    -0.2971	  8.99e+05	 -3.31e-07	  1.000	 -1.76e+06	  1.76e+06

  alpha_113	     0.2028	       nan	       nan	    nan	       nan	       nan

  alpha_114	    -0.0756	       nan	       nan	    nan	       nan	       nan

  alpha_115	     0.0083	       nan	       nan	    nan	       nan	       nan

  alpha_116	    -0.0351	       nan	       nan	    nan	       nan	       nan

  alpha_117	     0.0773	       nan	       nan	    nan	       nan	       nan

  alpha_118	     0.0768	       nan	       nan	    nan	       nan	       nan

  alpha_119	     0.0453	  5.07e+06	  8.94e-09	  1.000	 -9.93e+06	  9.93e+06

  alpha_120	     0.0896	       nan	       nan	    nan	       nan	       nan

  alpha_121	     0.0323	  2.24e+06	  1.44e-08	  1.000	 -4.39e+06	  4.39e+06

  alpha_122	     0.0729	       nan	       nan	    nan	       nan	       nan

  alpha_123	     0.1574	       nan	       nan	    nan	       nan	       nan

  alpha_124	     0.0126	       nan	       nan	    nan	       nan	       nan

  alpha_125	     0.0614	       nan	       nan	    nan	       nan	       nan

  alpha_126	    -0.3628	       nan	       nan	    nan	       nan	       nan

  alpha_127	    -0.1861	   2.8e+06	 -6.65e-08	  1.000	 -5.49e+06	  5.49e+06

  alpha_128	     0.1717	  3.29e+06	  5.22e-08	  1.000	 -6.45e+06	  6.45e+06

  alpha_129	    -0.0119	  5.28e+06	 -2.26e-09	  1.000	 -1.03e+07	  1.03e+07

  alpha_130	     0.1482	       nan	       nan	    nan	       nan	       nan

  alpha_131	     0.0230	       nan	       nan	    nan	       nan	       nan

  alpha_132	    -0.2162	  3.36e+06	 -6.43e-08	  1.000	 -6.59e+06	  6.59e+06

  alpha_133	     0.1891	       nan	       nan	    nan	       nan	       nan

  alpha_134	     0.0137	   2.4e+06	  5.72e-09	  1.000	  -4.7e+06	   4.7e+06

  alpha_135	     0.1030	  3.32e+06	   3.1e-08	  1.000	 -6.51e+06	  6.51e+06

  alpha_136	     0.1470	       nan	       nan	    nan	       nan	       nan

  alpha_137	     0.1630	       nan	       nan	    nan	       nan	       nan

  alpha_138	     0.0529	  1.87e+06	  2.83e-08	  1.000	 -3.67e+06	  3.67e+06

  alpha_139	    -0.1658	       nan	       nan	    nan	       nan	       nan

  alpha_140	     0.0811	  3.29e+06	  2.47e-08	  1.000	 -6.44e+06	  6.44e+06

  alpha_141	     0.2265	       nan	       nan	    nan	       nan	       nan

  alpha_142	    -0.0778	  6.18e+06	 -1.26e-08	  1.000	 -1.21e+07	  1.21e+07

  alpha_143	    -0.2297	       nan	       nan	    nan	       nan	       nan

  alpha_144	     0.0126	  2.17e+06	   5.8e-09	  1.000	 -4.25e+06	  4.25e+06

  alpha_145	    -0.1028	  1.54e+06	 -6.68e-08	  1.000	 -3.02e+06	  3.02e+06

  alpha_146	    -0.0012	  5.51e+06	  -2.1e-10	  1.000	 -1.08e+07	  1.08e+07

  alpha_147	    -0.1505	       nan	       nan	    nan	       nan	       nan

  alpha_148	     0.0560	  2.66e+06	   2.1e-08	  1.000	 -5.22e+06	  5.22e+06

  alpha_149	    -0.0485	       nan	       nan	    nan	       nan	       nan

  alpha_150	    -0.0721	       nan	       nan	    nan	       nan	       nan

  alpha_151	    -0.0257	  2.12e+06	 -1.21e-08	  1.000	 -4.16e+06	  4.16e+06

  alpha_152	     0.0594	  4.08e+06	  1.46e-08	  1.000	 -7.99e+06	  7.99e+06

  alpha_153	    -0.1511	  2.47e+06	 -6.13e-08	  1.000	 -4.83e+06	  4.83e+06

  alpha_154	    -0.1401	       nan	       nan	    nan	       nan	       nan

  alpha_155	     0.0431	  3.23e+06	  1.33e-08	  1.000	 -6.33e+06	  6.33e+06

  alpha_156	     0.0569	       nan	       nan	    nan	       nan	       nan

  alpha_157	    -0.0333	  1.64e+06	 -2.04e-08	  1.000	 -3.21e+06	  3.21e+06

  alpha_158	    -0.1185	       nan	       nan	    nan	       nan	       nan

  alpha_159	     0.0800	  4.28e+06	  1.87e-08	  1.000	 -8.38e+06	  8.38e+06

  alpha_160	     0.0015	       nan	       nan	    nan	       nan	       nan

  alpha_161	    -0.0234	  1.49e+06	 -1.57e-08	  1.000	 -2.92e+06	  2.92e+06

  alpha_162	     0.0708	       nan	       nan	    nan	       nan	       nan

  alpha_163	    -0.0849	       nan	       nan	    nan	       nan	       nan

  alpha_164	     0.0766	       nan	       nan	    nan	       nan	       nan

  alpha_165	    -0.1192	       nan	       nan	    nan	       nan	       nan

  alpha_166	    -0.0742	  3.94e+06	 -1.88e-08	  1.000	 -7.72e+06	  7.72e+06

  alpha_167	    -0.2277	       nan	       nan	    nan	       nan	       nan

  alpha_168	     0.1165	       nan	       nan	    nan	       nan	       nan

  alpha_169	     0.0760	       nan	       nan	    nan	       nan	       nan

  alpha_170	    -0.0497	  1.54e+06	 -3.24e-08	  1.000	 -3.01e+06	  3.01e+06

  alpha_171	     0.1372	  1.83e+06	  7.49e-08	  1.000	 -3.59e+06	  3.59e+06

  alpha_172	     0.0494	  1.38e+06	  3.57e-08	  1.000	 -2.71e+06	  2.71e+06

  alpha_173	     0.0543	       nan	       nan	    nan	       nan	       nan

  alpha_174	    -0.0303	       nan	       nan	    nan	       nan	       nan

  alpha_175	    -0.0426	       nan	       nan	    nan	       nan	       nan

  alpha_176	    -0.2416	       nan	       nan	    nan	       nan	       nan

  alpha_177	    -0.2136	       nan	       nan	    nan	       nan	       nan

  alpha_178	     0.0035	  4.27e+06	  8.16e-10	  1.000	 -8.37e+06	  8.37e+06

  alpha_179	    -0.0043	       nan	       nan	    nan	       nan	       nan

  alpha_180	    -0.2604	  1.97e+06	 -1.33e-07	  1.000	 -3.85e+06	  3.85e+06

  alpha_181	    -0.0043	       nan	       nan	    nan	       nan	       nan

  alpha_182	     0.1750	  1.94e+06	  9.02e-08	  1.000	 -3.81e+06	  3.81e+06

  alpha_183	    -0.1624	  5.57e+06	 -2.92e-08	  1.000	 -1.09e+07	  1.09e+07

  alpha_184	    -0.0081	  1.59e+06	 -5.08e-09	  1.000	 -3.11e+06	  3.11e+06

  alpha_185	     0.0542	  2.22e+06	  2.44e-08	  1.000	 -4.36e+06	  4.36e+06

  alpha_186	    -0.0353	       nan	       nan	    nan	       nan	       nan

  alpha_187	     0.0813	  2.06e+06	  3.94e-08	  1.000	 -4.04e+06	  4.04e+06

  alpha_188	     0.0463	       nan	       nan	    nan	       nan	       nan

  alpha_189	    -0.0821	       nan	       nan	    nan	       nan	       nan

  alpha_190	    -0.1220	       nan	       nan	    nan	       nan	       nan

  alpha_191	    -0.1454	  1.93e+06	 -7.55e-08	  1.000	 -3.77e+06	  3.77e+06

  alpha_192	     0.0037	  1.31e+06	  2.83e-09	  1.000	 -2.56e+06	  2.56e+06

  alpha_193	    -0.1975	   2.8e+06	 -7.06e-08	  1.000	 -5.48e+06	  5.48e+06

  alpha_194	    -0.2787	  2.29e+06	 -1.22e-07	  1.000	 -4.48e+06	  4.48e+06

  alpha_195	    -0.1792	  3.07e+06	 -5.84e-08	  1.000	 -6.01e+06	  6.01e+06

  alpha_196	    -0.0844	       nan	       nan	    nan	       nan	       nan

  alpha_197	     0.0059	  4.68e+06	  1.27e-09	  1.000	 -9.18e+06	  9.18e+06

  alpha_198	    -0.0317	   1.6e+06	 -1.99e-08	  1.000	 -3.13e+06	  3.13e+06

  alpha_199	     0.0823	  6.69e+06	  1.23e-08	  1.000	 -1.31e+07	  1.31e+07

  alpha_200	    -0.0092	       nan	       nan	    nan	       nan	       nan

  alpha_201	    -0.0512	  2.49e+06	 -2.06e-08	  1.000	 -4.88e+06	  4.88e+06

  alpha_202	    -0.2186	       nan	       nan	    nan	       nan	       nan

  alpha_203	     0.0616	       nan	       nan	    nan	       nan	       nan

  alpha_204	     0.1598	  5.78e+06	  2.77e-08	  1.000	 -1.13e+07	  1.13e+07

  alpha_205	    -0.0086	       nan	       nan	    nan	       nan	       nan

  alpha_206	    -0.2852	       nan	       nan	    nan	       nan	       nan

  alpha_207	     0.1323	       nan	       nan	    nan	       nan	       nan

  alpha_208	    -0.0135	  2.79e+06	 -4.84e-09	  1.000	 -5.48e+06	  5.48e+06

  alpha_209	    -0.2285	       nan	       nan	    nan	       nan	       nan

  alpha_210	     0.0053	  1.21e+06	  4.36e-09	  1.000	 -2.38e+06	  2.38e+06

  alpha_211	    -0.0127	       nan	       nan	    nan	       nan	       nan

  alpha_212	    -0.2706	  3.89e+06	 -6.95e-08	  1.000	 -7.63e+06	  7.63e+06

  alpha_213	     0.0254	       nan	       nan	    nan	       nan	       nan

  alpha_214	    -0.0553	       nan	       nan	    nan	       nan	       nan

  alpha_215	     0.1737	  2.69e+06	  6.46e-08	  1.000	 -5.27e+06	  5.27e+06

  alpha_216	     0.0488	       nan	       nan	    nan	       nan	       nan

  alpha_217	    -0.1708	  1.32e+06	 -1.29e-07	  1.000	 -2.58e+06	  2.58e+06

  alpha_218	    -0.0962	       nan	       nan	    nan	       nan	       nan

  alpha_219	    -0.1178	  3.46e+06	  -3.4e-08	  1.000	 -6.79e+06	  6.79e+06

  alpha_220	     0.0096	  3.62e+06	  2.64e-09	  1.000	  -7.1e+06	   7.1e+06

  alpha_221	    -0.0139	       nan	       nan	    nan	       nan	       nan

  alpha_222	     0.0855	  6.59e+05	   1.3e-07	  1.000	 -1.29e+06	  1.29e+06

  alpha_223	    -0.0232	       nan	       nan	    nan	       nan	       nan

  alpha_224	    -0.1281	       nan	       nan	    nan	       nan	       nan

  alpha_225	    -0.0588	       nan	       nan	    nan	       nan	       nan

  alpha_226	     0.1737	  2.16e+06	  8.03e-08	  1.000	 -4.24e+06	  4.24e+06

  alpha_227	    -0.1555	  2.96e+06	 -5.25e-08	  1.000	 -5.81e+06	  5.81e+06

  alpha_228	     0.0411	       nan	       nan	    nan	       nan	       nan

  alpha_229	     0.0263	  2.17e+06	  1.21e-08	  1.000	 -4.24e+06	  4.24e+06

  alpha_230	    -0.1260	       nan	       nan	    nan	       nan	       nan

  alpha_231	    -0.0563	       nan	       nan	    nan	       nan	       nan

  alpha_232	    -0.1642	       nan	       nan	    nan	       nan	       nan

  alpha_233	    -0.1848	  6.35e+06	 -2.91e-08	  1.000	 -1.25e+07	  1.25e+07

  alpha_234	    -0.0164	  1.67e+06	 -9.81e-09	  1.000	 -3.28e+06	  3.28e+06

  alpha_235	     0.1883	       nan	       nan	    nan	       nan	       nan

  alpha_236	    -0.1571	  1.31e+06	  -1.2e-07	  1.000	 -2.57e+06	  2.57e+06

  alpha_237	    -0.3025	       nan	       nan	    nan	       nan	       nan

  alpha_238	    -0.0345	  4.94e+06	 -6.98e-09	  1.000	 -9.69e+06	  9.69e+06

  alpha_239	     0.0552	       nan	       nan	    nan	       nan	       nan

  alpha_240	     0.0464	  3.44e+06	  1.35e-08	  1.000	 -6.74e+06	  6.74e+06

  alpha_241	    -0.1885	       nan	       nan	    nan	       nan	       nan

  alpha_242	     0.0461	       nan	       nan	    nan	       nan	       nan

  alpha_243	     0.0189	  4.42e+06	  4.27e-09	  1.000	 -8.67e+06	  8.67e+06

  alpha_244	     0.1661	  9.94e+05	  1.67e-07	  1.000	 -1.95e+06	  1.95e+06

  alpha_245	    -0.1414	  5.13e+05	 -2.76e-07	  1.000	 -1.01e+06	  1.01e+06

  alpha_246	     0.0408	       nan	       nan	    nan	       nan	       nan

  alpha_247	     0.0032	       nan	       nan	    nan	       nan	       nan

  alpha_248	    -0.2314	       nan	       nan	    nan	       nan	       nan

  alpha_249	    -0.2004	       nan	       nan	    nan	       nan	       nan

  alpha_250	    -0.0854	       nan	       nan	    nan	       nan	       nan

  alpha_251	    -0.0119	   1.2e+05	 -9.85e-08	  1.000	 -2.36e+05	  2.36e+05

  alpha_252	    -0.0351	       nan	       nan	    nan	       nan	       nan

  alpha_253	    -0.0369	  2.36e+06	 -1.56e-08	  1.000	 -4.63e+06	  4.63e+06

  alpha_254	     0.2275	       nan	       nan	    nan	       nan	       nan

  alpha_255	    -0.1291	  2.21e+06	 -5.84e-08	  1.000	 -4.33e+06	  4.33e+06

  alpha_256	     0.1254	       nan	       nan	    nan	       nan	       nan

  alpha_257	    -0.0045	  2.57e+06	 -1.75e-09	  1.000	 -5.03e+06	  5.03e+06

  alpha_258	     0.0640	       nan	       nan	    nan	       nan	       nan

  alpha_259	     0.0309	       nan	       nan	    nan	       nan	       nan

  alpha_260	     0.1127	  4.31e+06	  2.62e-08	  1.000	 -8.45e+06	  8.45e+06

  alpha_261	     0.1426	       nan	       nan	    nan	       nan	       nan

  alpha_262	     0.0249	  1.71e+06	  1.46e-08	  1.000	 -3.35e+06	  3.35e+06

  alpha_263	     0.1937	       nan	       nan	    nan	       nan	       nan

  alpha_264	    -0.0702	       nan	       nan	    nan	       nan	       nan

  alpha_265	     0.0252	       nan	       nan	    nan	       nan	       nan

  alpha_266	     0.0804	  2.88e+06	  2.79e-08	  1.000	 -5.65e+06	  5.65e+06

  alpha_267	    -0.0415	   2.5e+06	 -1.66e-08	  1.000	 -4.89e+06	  4.89e+06

  alpha_268	     0.0553	  3.26e+06	   1.7e-08	  1.000	 -6.39e+06	  6.39e+06

  alpha_269	     0.0414	       nan	       nan	    nan	       nan	       nan

  alpha_270	    -0.0161	       nan	       nan	    nan	       nan	       nan

  alpha_271	    -0.1356	  2.94e+06	  -4.6e-08	  1.000	 -5.77e+06	  5.77e+06

  alpha_272	     0.0215	  4.56e+06	  4.72e-09	  1.000	 -8.95e+06	  8.95e+06

  alpha_273	     0.0413	  5.42e+06	  7.61e-09	  1.000	 -1.06e+07	  1.06e+07

  alpha_274	     0.1034	       nan	       nan	    nan	       nan	       nan

  alpha_275	    -0.0340	  7.19e+06	 -4.73e-09	  1.000	 -1.41e+07	  1.41e+07

  alpha_276	    -0.1941	  2.78e+06	 -6.99e-08	  1.000	 -5.45e+06	  5.45e+06

  alpha_277	    -0.2046	       nan	       nan	    nan	       nan	       nan

  alpha_278	    -0.2574	  2.34e+06	  -1.1e-07	  1.000	 -4.58e+06	  4.58e+06

  alpha_279	    -0.0351	  4.55e+06	 -7.71e-09	  1.000	 -8.93e+06	  8.93e+06

  alpha_280	    -0.2097	       nan	       nan	    nan	       nan	       nan

  alpha_281	     0.0694	  6.12e+06	  1.13e-08	  1.000	  -1.2e+07	   1.2e+07

  alpha_282	    -0.2733	  1.79e+06	 -1.53e-07	  1.000	  -3.5e+06	   3.5e+06

  alpha_283	     0.0476	  2.02e+06	  2.36e-08	  1.000	 -3.95e+06	  3.95e+06

  alpha_284	     0.0117	       nan	       nan	    nan	       nan	       nan

  alpha_285	     0.0404	       nan	       nan	    nan	       nan	       nan

  alpha_286	     0.0497	   4.9e+06	  1.01e-08	  1.000	  -9.6e+06	   9.6e+06

  alpha_287	    -0.0898	       nan	       nan	    nan	       nan	       nan

  alpha_288	     0.1765	       nan	       nan	    nan	       nan	       nan

  alpha_289	    -0.2823	       nan	       nan	    nan	       nan	       nan

  alpha_290	     0.0345	       nan	       nan	    nan	       nan	       nan

  alpha_291	     0.0331	  2.21e+06	   1.5e-08	  1.000	 -4.33e+06	  4.33e+06

  alpha_292	    -0.0943	  5.16e+06	 -1.83e-08	  1.000	 -1.01e+07	  1.01e+07

  alpha_293	    -0.2573	       nan	       nan	    nan	       nan	       nan

  alpha_294	     0.0413	  7.09e+05	  5.83e-08	  1.000	 -1.39e+06	  1.39e+06

  alpha_295	    -0.0415	  2.93e+06	 -1.42e-08	  1.000	 -5.75e+06	  5.75e+06

  alpha_296	     0.0145	       nan	       nan	    nan	       nan	       nan

  alpha_297	    -0.0695	  2.04e+06	 -3.41e-08	  1.000	    -4e+06	     4e+06

  alpha_298	     0.1025	  2.09e+06	   4.9e-08	  1.000	  -4.1e+06	   4.1e+06

  alpha_299	    -0.0479	  2.39e+06	    -2e-08	  1.000	 -4.69e+06	  4.69e+06

  alpha_300	     0.0355	  4.62e+06	  7.68e-09	  1.000	 -9.06e+06	  9.06e+06

  alpha_301	    -0.0802	  5.25e+06	 -1.53e-08	  1.000	 -1.03e+07	  1.03e+07

  alpha_302	    -0.0184	  6.87e+05	 -2.68e-08	  1.000	 -1.35e+06	  1.35e+06

  alpha_303	     0.0760	       nan	       nan	    nan	       nan	       nan

  alpha_304	    -0.0437	       nan	       nan	    nan	       nan	       nan

  alpha_305	     0.0244	  2.52e+06	   9.7e-09	  1.000	 -4.93e+06	  4.93e+06

  alpha_306	     0.1834	       nan	       nan	    nan	       nan	       nan

  alpha_307	    -0.1459	       nan	       nan	    nan	       nan	       nan

  alpha_308	     0.0737	  1.36e+06	   5.4e-08	  1.000	 -2.67e+06	  2.67e+06

  alpha_309	     0.0103	   7.4e+06	  1.39e-09	  1.000	 -1.45e+07	  1.45e+07

  alpha_310	    -0.0891	  4.45e+06	    -2e-08	  1.000	 -8.72e+06	  8.72e+06

  alpha_311	    -0.1262	  2.99e+06	 -4.23e-08	  1.000	 -5.85e+06	  5.85e+06

  alpha_312	     0.0527	       nan	       nan	    nan	       nan	       nan

  alpha_313	     0.0296	  4.83e+06	  6.14e-09	  1.000	 -9.46e+06	  9.46e+06

  alpha_314	     0.0723	       nan	       nan	    nan	       nan	       nan

  alpha_315	    -0.0476	       nan	       nan	    nan	       nan	       nan

  alpha_316	    -0.2535	  2.85e+06	 -8.89e-08	  1.000	 -5.59e+06	  5.59e+06

  alpha_317	    -0.2573	       nan	       nan	    nan	       nan	       nan

It is more interesting to compute the confusion matrix and the precision, recall and F-score metric such as in the previous model. Nonetheless, the first step must be to compute the long format kernel matrix for the test set that is necessary to compute the previous metrics using KLR. This can be done using the following code.

```python
# Comute the long format kernel matrix for the test set.
K_long_format_test = pkl.long_format_with_kernel_matrix(test_set_scaled,
                                                        custom_alt_id,
                                                        obs_id_column,
                                                        choice_column,
                                                        variables,
                                                        kernel_type = "RBF",
                                                        Z = train_set_scaled,
                                                        length_scale = 1)
```

```python
# Compute the confunsion matrix using the test set
confusion_matrix = kernel_model.confusion_matrix(K_long_format_test)

# The generated confusion matrix can be shown directly as a pandas dataframe
confusion_matrix
```

      1
      2
      3
      4

      1
      57
      0
      0
      7

      2
      3
      0
      0
      4

      3
      7
      0
      11
      8

      4
      8
      0
      6
      25

```python
# Visualize the confusion matrix using seaborn.
# Generate a new figure and set the figure size.
plt.figure(figsize=(10,5))

# Generate a heatmap for the confusion matrix using seaborn
ax = sns.heatmap(confusion_matrix, annot=True, linewidth=0.5, cmap='PuBu')

# labels, title and ticks
ax.set_xlabel('Predicted alternative')
ax.set_ylabel('True alternative')
ax.set_title('Confusion Matrix')
```

```python
# Compute the precision, recall and F1-score for the linear model using a
# Micro-averaging technique. The F1-score is the F-score for beta = 1.
kernel_model.precision_recall_fscore(K_long_format_test, "micro", beta = 1)
```

    (0.6838235294117647, 0.6838235294117647, 0.6838235294117647)

### 3.4. Prediction using the estimated model

Again, once the model has been estimated, new instances can be predicted using the model.

```python
# The different observations from the test_set are represented on each row.
# The columns represent each of the alternatives. The value from each cell
# is the probability that the alternative on the corresponding column was
# selected on that observation.
pd.DataFrame(data=kernel_model.predict(K_long_format_test).reshape(-1, 4))
```

      0
      1
      2
      3

      0
      0.916604
      0.017250
      0.040786
      0.025359

      1
      0.739171
      0.024550
      0.082291
      0.153987

      2
      0.953754
      0.033085
      0.004249
      0.008912

      3
      0.314859
      0.203336
      0.105414
      0.376390

      4
      0.149059
      0.049907
      0.112112
      0.688923

      ...
      ...
      ...
      ...
      ...

      131
      0.217392
      0.137808
      0.116807
      0.527993

      132
      0.612135
      0.105613
      0.090062
      0.192190

      133
      0.764801
      0.114906
      0.084024
      0.036269

      134
      0.660301
      0.128328
      0.030257
      0.181114

      135
      0.828948
      0.071262
      0.072905
      0.026885

136 rows × 4 columns

### 3.5. Computation of the Willingness to Pay (WTP) indicator using the estimated KLR model

As shown previously, the equation to compute the WTP is:
\begin{equation}
    WTP = \frac{\delta_{i n}^{c}}{\delta_{i n}^{x}}=-\frac{\left(\partial V_{i n} / \partial x_{i n}\right)\left(c_{i n}, x_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, x_{i n}\right)}
\end{equation}
where $c_{in}$ is the cost of the alternative $i$ for a decision-maker $n$ and $x_{in}$ is the value of another variable from the model. The WTP can be computed directly using a close expression for the linear RUM (as shown in section 2.5, nonetheless, for the KLR models it is necessary to compute the gradient vector of the partial derivatives of the systematic utility function.

Since KLR is a non-linear model, the first step is to define the point where the WTP is going to be computed. Any observation from a long format dataframe can be selected as this WTP point and, therefore, the WTP for that decision-maker can be computed. Moreover, the analyst can use any point for the WTP calculation, but this point should be provided in a long format dataframe.

The following lines of code shows how to compute the WTP for any observation from the dataframe using KLR. The first step is to select the WTP point.

```python
# Compute the WTP of any observation from the dataframe
WTP_point = long_mode_choice.loc[long_mode_choice[obs_id_column] == 12]

WTP_point
```

      custom_id
      mode_id
      choice
      cost
      time

      44
      12
      1
      1
      1.864544
      17.530151

      45
      12
      2
      0
      0.568055
      26.731061

      46
      12
      3
      0
      2.167807
      55.347093

      47
      12
      4
      0
      1.978343
      32.651246

Next, the PyKernelLogit function calculate_WTP_kernel can be used to calculate the WTP indicador for an observation using KLR by numerically approximating the gradient vector of the systematic utility. This function takes following input arguments:
* WTP_cost_column. Column name with contains the cost or price of the alternative.

* WTP_x_column. Column name with contains the variable to be analyzed.

* alt. Indicates the alternative for which the WTP is computed.

* model. A MNDC_Model instance corresponding with the trained model.

* variables. A Python dictionary where each key represents each of the alternatives IDs and its associated value contains a list of characteristics from the long dataframe WTP_point and Z to be used on that alternative.

* WTP_point. A dataframe in long format with only one observation (1 row per each alternative of that observation) for which the Willingness to Pay is going to be computed.

* Z. A reference matrix $\mathbf{Z}_{i}$ to be used when computing the kernel matrix.

* kernel_type. The type of kernel function to be used.

* scaler. If none scaler was used to fit the kernel model, this parameter must be None. Otherwise, an instance of the scaler function must be provided here. The provided scaler must contain a transform function which scale the features of the dataset according to the scaler definition.

* epsilon. The $\varepsilon$ parameter. The increment of the coordinate vector used for determining the function gradient.

As it was described previously the Value of time (VOT) reflects the price that a traveler is willing to pay to decrease the travel time in one unit. In this example, it is interesting to compute the **VOT** as the model contains the cost and time variables.

```python
# Comuthe the VOT for a KLR model, i.e. the WTP using cost an time variables.
# Take into account that travel time is in minutes, therefore, the VOT must be
# multiplied by 60 to convert it to hours.
VOT = -60 * pkl.calculate_WTP_kernel(WTP_cost_column = 'cost',
                                    WTP_x_column = 'time',
                                    alt = 1,
                                    model = kernel_model,
                                    variables = variables,
                                    WTP_point = WTP_point,
                                    Z = train_set_scaled,
                                    kernel_type = "RBF",
                                    scaler = scaler,
                                    scaled_fetures=['cost', 'time'])

VOT
```

    7.617601618103002

The analyst can evalute the WTP also for an hypothetical observation which is not present at the dataframe, but this point should be provided in a long format dataframe. In this case, the obs_id_col, alt_id_col and choice_col must be created by the user manually. To illustrate this concept, the following code computes the WTP for an observation for which the value of each variable is the mean value of that variable in the Mode dataframe.

```python
# Define the WTP_points columns.
WTP_point_columns = [obs_id_column, custom_alt_id, choice_column, 'cost', 'time']
WTP_point = pd.DataFrame(columns=WTP_point_columns)

# Append to WTP_points dataframe the mean value for each dataframe variable
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 1, # Fixed to 1 in the first alternative. Necessary to create a long type dataframe
                               custom_alt_id: 1, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.car'].mean(),
                               'time': mode_choice['time.car'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 2, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.carpool'].mean(),
                               'time': mode_choice['time.carpool'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 3, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.bus'].mean(),
                               'time': mode_choice['time.bus'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 4, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.rail'].mean(),
                               'time': mode_choice['time.rail'].mean(),
                               }, ignore_index=True)

# Show the generated dataframe
WTP_point
```

      custom_id
      mode_id
      choice
      cost
      time

      0
      1.0
      1.0
      1.0
      4.873487
      37.044117

      1
      1.0
      2.0
      0.0
      1.686348
      39.771005

      2
      1.0
      3.0
      0.0
      2.035669
      39.923409

      3
      1.0
      4.0
      0.0
      2.211683
      39.504507

```python
# Comuthe the VOT for a KLR model, i.e. the WTP using cost an time variables.
VOT = -60 * pkl.calculate_WTP_kernel(WTP_cost_column = 'cost',
                                    WTP_x_column = 'time',
                                    alt = 1,
                                    model = kernel_model,
                                    variables = variables,
                                    WTP_point = WTP_point,
                                    Z = train_set_scaled,
                                    kernel_type = "RBF",
                                    scaler = scaler,
                                    scaled_fetures=['cost', 'time'])

VOT
```

    8.133675662298685

## About this Jupyter notebook

This Jupyter notebook was developed by José Ángel Martín Baos on September 2019 to demonstrate the key functionalities of the PyKernelLogit Python package (https://pypi.org/project/pykernellogit/). This package is based on the PyLogit Python package (https://pypi.org/project/pylogit/) developed by Timothy Brathwaite (Timothy Brathwaite and Joan L. Walker. Asymmetric, closed-form, finite-parameter models of multinomial choice. Journal of Choice Modelling. Volume 29, December 2018.). This notebook is based on the example notebook for the PyLogit package.

* José Ángel Martín Baos (https://joseangelmartinb.github.io/). Faculty of Computer Science. University of Castilla-La Mancha.


# NUME

Instructor: Keny **ORDAZ**

# Systems of Linear Algebraic Equations: Iterative Methods

$\newcommand{\vb}[1]{\boldsymbol{#1}}\newcommand{\RR}[1]{\mathbb{R}^{#1}}
\newcommand{\vt}[1]{\vb{#1}^\top}$

## Introduction

Iterative methods vs Direct methods of solution.

$$\vb{A} \vb{x} = \vb{b}$$

| Direct methods | Iterative (indirect) methods |
|:---------------|:------------------|
| Finite number of operations | Unknown number of iterations, slower |
| Best for dense and small $\vb{A}$ | Best for sparse and large $\vb{A}$|
| Always provide a solution | Not always converge to the solution |

- Iterative methods start with an initial guess of the solution $\vb{x}$ and then repeatedly improve the solution until the change in $\vb{x}$ becomes negligible.

- Because the required number of iterations can be large, the indirect methods are, in general, slower than their direct counterparts.

- Iterative methods do have the following two advantages that make them attractive for certain problems:

    1. It is feasible to store only the nonzero elements of the coefficient matrix. This makes it possible to deal with very large matrices that are sparse, but not necessarily banded. In many problems, there is no need to store the coefficient matrix at all.

    2. Iterative procedures are self-correcting, meaning that roundoff errors (or even arithmetic mistakes) in one iterative cycle are corrected in subsequent cycles.

- It can be shown that _convergence_ is **guaranteed only if the coefficient matrix is diagonally dominant**. The initial guess for $\vb{x}$ plays no role in determining whether convergence takes place: if the procedure converges for one starting vector, it would do so for any starting vector. The initial guess affects only the number of iterations that are required for convergence.

$\newcommand{\x}{\vb{x}}
\newcommand{\xT}{\vt{x}}
\newcommand{\b}{\vb{b}}
\newcommand{\A}{\vb{A}}
\newcommand{\bT}{\vt{b}}
\newcommand{\AT}{\vt{A}}$

```python
import numpy as np
import scipy
import sympy as sp
import matplotlib.pyplot as plt
```

```python
!ls book/*.py
```

    book/LUdecomp.py       book/example8_8.py     book/newtonRaphson2.py
    book/LUdecomp3.py      book/gaussElimin.py    book/plotPoly.py
    book/LUdecomp5.py      book/gaussNodes.py     book/polyFit.py
    book/LUpivot.py        book/gaussPivot.py     book/polyRoots.py
    book/bisection.py      book/gaussQuad.py      book/powell.py
    book/bulStoer.py       book/gaussQuad2.py     book/printSoln.py
    book/choleski.py       book/gaussSeidel.py    book/rational.py
    book/conjGrad.py       book/gerschgorin.py    book/ridder.py
    book/cubicSpline.py    book/goldSearch.py     book/romberg.py
    book/downhill.py       book/householder.py    book/rootsearch.py
    book/eigenvals3.py     book/inversePower.py   book/run_kut4.py
    book/error.py          book/inversePower3.py  book/run_kut5.py
    book/euler.py          book/inversePower5.py  book/sortJacobi.py
    book/evalPoly.py       book/jacobi.py         book/stdForm.py
    book/example8_1.py     book/lamRange.py       book/sturmSeq.py
    book/example8_3.py     book/linInterp.py      book/swap.py
    book/example8_4.py     book/midpoint.py       book/trapezoid.py
    book/example8_5.py     book/neville.py        book/triangleQuad.py
    book/example8_6.py     book/newtonPoly.py
    book/example8_7.py     book/newtonRaphson.py

$\A \x = \lambda \x$

```python
def build_householder_projector(v):
    #P = I - 2 * v v^T / v^T v
    return np.eye(len(v)) - 2 * np.outer(v, v) / np.dot(v, v)

v = np.array([1, 2, 3, 4, 5, 6])

P = build_householder_projector(v)
P
```

    array([[ 0.97802198, -0.04395604, -0.06593407, -0.08791209, -0.10989011,
            -0.13186813],
           [-0.04395604,  0.91208791, -0.13186813, -0.17582418, -0.21978022,
            -0.26373626],
           [-0.06593407, -0.13186813,  0.8021978 , -0.26373626, -0.32967033,
            -0.3956044 ],
           [-0.08791209, -0.17582418, -0.26373626,  0.64835165, -0.43956044,
            -0.52747253],
           [-0.10989011, -0.21978022, -0.32967033, -0.43956044,  0.45054945,
            -0.65934066],
           [-0.13186813, -0.26373626, -0.3956044 , -0.52747253, -0.65934066,
             0.20879121]])

```python
u = np.array([4, 2, 3, 0, 0, 0])
w = np.array([1, 0, 0, 0, 0, 3])
y = np.array([ 0, 0, 0, 1, 0, 0])

yp = P @ y
wp = P @ w
up = P @ u
up, wp, yp
```

    (array([ 3.62637363,  1.25274725,  1.87912088, -1.49450549, -1.86813187,
            -2.24175824]),
     array([ 0.58241758, -0.83516484, -1.25274725, -1.67032967, -2.08791209,
             0.49450549]),
     array([-0.08791209, -0.17582418, -0.26373626,  0.64835165, -0.43956044,
            -0.52747253]))

$$\begin{aligned}P &= I - 2 \frac{ v v^T }{ v^T v}\end{aligned}$$

Aplicamos el proyector:

$$\begin{aligned}w' &= P w \\
 &= \left[I - 2 \frac{ v v^T }{ v^T v}\right] w\\
 &= w - \left[2 \frac{ v v^T }{ v^T v}\right] w\\
  &= w - \frac{2  }{ v^T v} v v^T w \\
  &= w - 2\frac{v^T w  }{ v^T v} v  \\
  &= w - 2\frac{v \cdot w  }{ v \cdot v} v \\
  &= w - 2\frac{v \cdot w  }{ \|v\| \|v\|} v \\
  &= w - 2(\hat{v} \cdot w)  \hat{v}
\end{aligned}$$

donde:

- $w'$ es la imagen de $w$ bajo la proyección (transformación) $P$
- $\hat{v} = \frac{v}{ \|v\|}$ es el vector unitario (normalizado de $v$).
- $v \cdot v = \| v \|^2$

```python
def hh_projector(w, v):
    # P = I - 2 * v v^T / v^T v
    # w' = w - 2 * (np.dot(v, w)) * v
    return w - (2 * np.dot(v, w) / np.dot(v, v)) * v
```

```python
def hh_projector_normalized(w, v):
    # P = I - 2 * v v^T / v^T v
    # w' = w - 2 * (np.dot(v, w)) * v
    return w - 2 * (np.dot(v, w)) * v

"""
w - v ==> n ops
w . v ==> n-1 sums + n prods = 2 n - 1
2 op
a . v ==> n ops

2(2n - 1) + 2 + n + n = 4 n - 2 + 2 + 2n = 6n

n = 3 --> fu
"""
None
```

```python
ui = hh_projector(u, v)
wi = hh_projector(w, v)
yi = hh_projector(y, v)

# n = 3
# Forma matricial P @ w --> n*m sumas + n*m prods => n^2 + n^2 = 2 * n^2 ops = 2 * (3)^2 = 2 * 9 = 18 ops
# Forma funcional (operaciones vectoriales): 6 n; con n=3 --> 6(3) = 18 ops

# n = 6
# Forma matricial P @ w --> 2 * n^2 ops = 2 * (6)^2 = 2 * 36 = 72 ops
# Forma funcional (operaciones vectoriales): 6 n; con n=6 --> 6(6) = 36 ops

## O(n) vs O(n^2), 6 n  vs 2 n^2

ui, up, wi, wp, yi, yp
ui - up, wi - wp, yi - yp
```

    (array([ 4.44089210e-16,  0.00000000e+00, -4.44089210e-16,  2.22044605e-16,
             0.00000000e+00,  0.00000000e+00]),
     array([ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
             0.00000000e+00, -1.11022302e-16]),
     array([ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
            -5.55111512e-17,  0.00000000e+00]))

```python
for n in [2, 3, 6, 20, 50, 200]:
    print(f'{n:5} {6*n:8} {2*n**2:10}')
```

        2       12          8
        3       18         18
        6       36         72
       20      120        800
       50      300       5000
      200     1200      80000

Con $\vb{v}$ normalizado, i.e. $\hat{\vb{v}}$:

```python
vnorm = v / np.linalg.norm(v)

projec_v = lambda w : w - 2 * (np.dot(vnorm, w)) * vnorm
```

```python
uin = projec_v(u)
win = projec_v(w)
yin = projec_v(y)
uin - up, win - wp, yin - yp, uin == ui, win == wi, yin == yi, np.allclose(win, wi)
```

    (array([ 4.44089210e-16,  0.00000000e+00, -4.44089210e-16,  2.22044605e-16,
            -2.22044605e-16,  0.00000000e+00]),
     array([ 1.11022302e-16,  1.11022302e-16,  0.00000000e+00,  2.22044605e-16,
             0.00000000e+00, -1.11022302e-16]),
     array([ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
            -5.55111512e-17,  0.00000000e+00]),
     array([ True,  True,  True,  True, False,  True]),
     array([False, False,  True, False,  True,  True]),
     array([ True,  True,  True,  True,  True,  True]),
     True)

# Sparse Matrix

```python
from scipy.sparse import csr_matrix

# create a 2-D representation of the matrix
A = np.array([[1, 0, 0, 0, 0, 0], [0, 0, 2, 0, 0, 1],\
 [0, 0, 0, 2, 0, 0]])
print("Dense matrix representation: \n", A)

# convert to sparse matrix representation
S = csr_matrix(A)
print("Sparse matrix: \n",S)

# convert back to 2-D representation of the matrix
B = S.todense()
print("Dense matrix: \n", B)
```

    Dense matrix representation:
     [[1 0 0 0 0 0]
     [0 0 2 0 0 1]
     [0 0 0 2 0 0]]
    Sparse matrix:
       (0, 0)	1
      (1, 2)	2
      (1, 5)	1
      (2, 3)	2
    Dense matrix:
     [[1 0 0 0 0 0]
     [0 0 2 0 0 1]
     [0 0 0 2 0 0]]

```python

```

```python

```

```python
A = np.zeros((3, 6))
A[0, 0] = A[1, 5] = 1
A[1, 2] = A[2, 3] = 2
A
#A = np.array([[1 ,0 , 0, 0 , 0....]])

x = np.ones(6)
x[-1] = -5
# GAXPY  y = y + A x
A, x.reshape((-1,1)), A @ x # 3 * 6 prod y 3 * 5 sum = 33 ops =  m * (2n - 1) = 2 m*n - m
```

    (array([[1., 0., 0., 0., 0., 0.],
            [0., 0., 2., 0., 0., 1.],
            [0., 0., 0., 2., 0., 0.]]),
     array([[ 1.],
            [ 1.],
            [ 1.],
            [ 1.],
            [ 1.],
            [-5.]]),
     array([ 1., -3.,  2.]))

```python
# b0 = a00 * x0
# b1 = a12 * x2 + a15 * x5
# b2 = a23 * x3 = b2

# SPARSE: prods: 4, sum: 1 --> 5 ops
# DENSE: 2 * 3 * 6 = 36 ops

def function_example_sparse(x):
    # 5 ops vs  2mn - m = 36 - 6 = 30 ops matricial ? (6 p + 5s)*3 = 33
    assert len(x) == 6
    b = np.zeros(3)
    b[0] = S[0, 0] * x[0]
    b[1] = S[1, 2] * x[2] + S[1, 5] * x[5]
    b[2] = S[2, 3] * x[3]
    return b

```

```python
b_i = function_example_sparse(x)
b_i, b_i - A @ x
```

    (array([ 1., -3.,  2.]), array([0., 0., 0.]))

# Cases

```python
from scipy.sparse import coo_matrix, diags

a0 = np.array([[-2., 5, 9], [7, 1, 1], [-3, 7, -1]])
b0 = np.array([1, 6, -26.])

a1 = np.array([[12, -2., 3, 1],
               [-2, 15, 6, -3],
               [1, 6, 20, -4],
               [0, -3, 2, 9]])
b1 = np.array([0, 0, 20., 0])

a2 = np.array([[4, -1, 0, 0],
               [-1, 4, -1, 0],
               [0, -1, 4, -1],
               [0, 0, -1, 3]])
b2 = np.array([15.] + [10]*3)
w2 = 1.1
initial2 = b2 / np.diag(a2)

"""
a3 =
2, -1, 0
-1, 2, -1
0, -1, 1
b3 = 1, 1, 1

a4 =
3, 0, -1
0, 4, -2
-1, -2, 5
b4 = [4, 10, -10]
"""

row  = np.array([0, 5])
col  = np.array([0, 5])
data = np.array([-1., -1])
a5 = coo_matrix((data, (row, col)), shape=(6, 6)).toarray()
a5 += diags([1, 1, -2, 4, -2, 1, 1], [-5, -2, -1, 0, 1, 2,5], shape=(6, 6)).toarray()
b5 = [10, -8, 10, 10, -8, 10]

# ex 2.17

def sparse_exam_exer(a00, a01, a0n, bn, n):
    a = diags([a0n, a01, a00, a01, a0n], [-n+1, -1, 0, 1, n-1], shape=(n, n)).toarray()
    b = np.zeros(n)
    b[-1] = bn
    x = np.array([-n / 4 + i/2 for i in range(1, n + 1)])
    return a, b, x

n6 = 4
a6, b6, x6 = sparse_exam_exer(4, -1, 1, 100, n6)

ne17 = 20
ae17, be17, xe17 = sparse_exam_exer(2, -1, 1, 1, ne17)

"""
a1 = np.array([[]])
b2 = np.array([])

a1 = np.array([[]])
b2 = np.array([])

a1 = np.array([[]])
b2 = np.array([])
"""
initial2
a2, a0, a1, a5, a6, b6, ae17, be17, xe17, ae17 @ xe17 - be17
```

    (array([[ 4, -1,  0,  0],
            [-1,  4, -1,  0],
            [ 0, -1,  4, -1],
            [ 0,  0, -1,  3]]),
     array([[-2.,  5.,  9.],
            [ 7.,  1.,  1.],
            [-3.,  7., -1.]]),
     array([[12., -2.,  3.,  1.],
            [-2., 15.,  6., -3.],
            [ 1.,  6., 20., -4.],
            [ 0., -3.,  2.,  9.]]),
     array([[ 3., -2.,  1.,  0.,  0.,  1.],
            [-2.,  4., -2.,  1.,  0.,  0.],
            [ 1., -2.,  4., -2.,  1.,  0.],
            [ 0.,  1., -2.,  4., -2.,  1.],
            [ 0.,  0.,  1., -2.,  4., -2.],
            [ 1.,  0.,  0.,  1., -2.,  3.]]),
     array([[ 4., -1.,  0.,  1.],
            [-1.,  4., -1.,  0.],
            [ 0., -1.,  4., -1.],
            [ 1.,  0., -1.,  4.]]),
     array([  0.,   0.,   0., 100.]),
     array([[ 2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  1.],
            [-1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,  0.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2., -1.,
              0.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,  2.,
             -1.,  0.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -1.,
              2., -1.,  0.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
             -1.,  2., -1.,  0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0., -1.,  2., -1.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0., -1.,  2., -1.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0., -1.,  2., -1.,  0.],
            [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0., -1.,  2., -1.],
            [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
              0.,  0.,  0.,  0.,  0., -1.,  2.]]),
     array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 1.]),
     array([-4.5, -4. , -3.5, -3. , -2.5, -2. , -1.5, -1. , -0.5,  0. ,  0.5,
             1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ]),
     array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0.]))

### Matriz diagonal estrictamente dominante

- Por renglones

$${\displaystyle |a_{i,i}|>\sum _{j=1,j\neq i}^{n}|a_{i,j}|,\forall i\in \mathbb {N} :1\leq i\leq n}$$

- Por columnas

$${\displaystyle |a_{j,j}|>\sum _{i=1,i\neq j}^{n}|a_{i,j}|,\forall j\in \mathbb {N} :1\leq j\leq n}$$

# Resolución de sistemas lineales con métodos iterativos

```python
!cat book/jacobi.py
```

```python
!cat book/gaussSeidel.py
```

```python

```

```python
import numpy as np

def solve_sor(A, b, w, x):
    x = np.asarray(x)
    n = b.shape[0]
    xold = x.copy()
    convergence = False
    while not convergence:
        for i in range(n):
            s = 0.0
            for j in range(n):
                if j != i:
                    s += A[i,j]*x[j]
            suma = np.sum(A[i,:] @ x) - A[i,i]*x[i]
            #print(f'{suma=} \t\t {s=}\t{A[i,:] @ x}')
            x[i] = x[i] + w * ( (b[i] - s) / A[i, i] - x[i])
            # print(f'\t{x=}')
        # print(f'{x=}')
        convergence = np.allclose(A @ x, b, 1e-14) # or np.allclose(xold, x, 1e-14)
        xold = x.copy()
    return  x

def solve_jacobi_no(A, b, w, x):
    x = np.asarray(x)
    L, D, U = -np.tril(A, -1), np.diagonal(A), -np.triu(A, 1)
    assert (A == -L + np.diag(D) + -U).all()
    n = b.shape[0]
    xold = x.copy()
    convergence = False
    k = 0
    while not convergence:
        k += 1
        x = ((L + U )@x - b)
        print(f'1 {x=}')
        x = x / D
        print(f'2 {x=}')
        convergence = k > 10 or np.allclose(A @ x, b, 1e-14) # or np.allclose(xold, x, 1e-14)
        xold = x.copy()
    return  x

def solve_jacobi(A, b, w, x, max_iter=1000):
    x = np.asarray(x)
    L, D, U = -np.tril(A, -1), np.diagonal(A), -np.triu(A, 1)
    assert (A == -L + np.diag(D) + -U).all()
    n = b.shape[0]
    xold = x.copy()
    convergence = False
    k = 0
    while not convergence:
        k += 1
        for i in range(n):
            s = 0.0
            for j in range(n):
                if j != i:
                    s += A[i,j]*xold[j]
            suma = np.sum(A[i,:] @ x) - A[i,i]*x[i]
            #print(f'{suma=} \t\t {s=}\t{A[i,:] @ x}')
            x[i] = xold[i] + ( (b[i] - s) / A[i, i] - xold[i])
            # print(f'\t{x=}')
        # print(f'{x=}')
        convergence = k > max_iter or np.allclose(A @ x, b, 1e-14) # or np.allclose(xold, x, 1e-14)
        xold = x.copy()
    return  x

A = np.array([
    [4, -1, -6, 0.],
    [-5, -4, 10, 8],
    [0, 9, 4, -2],
    [1, 0, -7, 5]
    ])
b = np.array([2., 21, -12, -6])

E2_A = np.array([[10, -1, 2,0], [-1, 11, -1, 3], [2, -1, 10, -1], [3, -1, 8, 15]])
E2_b = np.array([6., 25, -11, 15])

for solver in [solve_sor]: #, solve_jacobi]:
    x = solver(A, b, 0.5, [0.0, 0, 0, 0])
    print(x, A@x - b)

y = solve_jacobi(E2_A, E2_b, 0. , np.zeros_like(E2_b))
print(f'sol: {y}\n{b - A @ y}')

```

    [ 3. -2.  2.  1.] [-6.99743374e-09  4.48017090e-09 -1.49637991e-09 -1.09573328e-09]
    sol: [ 0.98082192  1.88066971 -0.96377473  1.44322679]
    [ -5.82526636  31.51872146 -22.18447489 -20.943379  ]

# Conjugate Gradient Method

Find the vector $\vb{x}$ that minimizes the scalar function:

$$f(\vb{x}) = \frac{1}{2} \vt{x}\vb{A}\vb{x} - \vt{b}\vb{x}$$

where:

- $\vb{A}$ is symmetric ($\vb{A} = \vt{A}$) and positive definite ($\vt{x}\vb{A}\vb{x} > 0$)
- $f(\vb{x})$ is minimized when its gradient $\nabla f(\vb{x}) = \vb{A}\vb{x} - \vb{b} = \vb{0}$
- Minimization is equivalente to solving

$$\vb{A}\vb{x} = \vb{b}$$

## Conjugate Gradient Algorithm

- Choose $\vb{x}_0$, $\epsilon_\tau$ (error tolerance).
- $\vb{r}_0 \leftarrow \vb{b} - \A\x_0$
- $\vb{s}_0 \leftarrow \vb{r}_0$ (choose the direction of steepest descent), search direction vector
- For $i$ in $[0,1,2,\cdots]$ do:
    - $\alpha_k \leftarrow \frac{\vt{s}_k\vb{r}_k}{\vt{s}_k\A\vb{s}_k}$
    - $\x_{k+1} \leftarrow \x_k + \alpha_k \vb{s}_k$
    - $\vb{r}_{k+1} \leftarrow \vb{b} - \A\x_{k+1}$
    - if $\|\vb{r}_{k+1}\| \le \epsilon_\tau$ break
    - $\beta_k \leftarrow - \frac{\vt{r}_{k+1}\A\vb{s}_k}{\vt{s}_k\A\vb{s}_k}$
    - $\vb{s}_{k+1} \leftarrow \vb{r}_{k+1} + \beta_k \vb{s}_k$

c.f. Householder projector:

$$ p(w) = w - 2\frac{v^T w  }{ v^T v} v$$

Los vectores residuales son mutuamente ortogonales:

$$\vb{r}_i \cdot \vb{r}_j = 0, i\ne j$$

```python
# asignación: [=] a = b
# igualdad:    [==] a==b

# a <-- b
# a = b
```

```python
!cat book/conjGrad.py
```

```python
from book.conjGrad import conjGrad

A, b, x_sol = ae17.copy(), be17.copy(), xe17.copy()

def fAx(x):
    return A @ x

x_conjGrad, i_conjGrad = conjGrad(fAx, np.zeros_like(b), b)
```

```python
x_conjGrad, x_sol, i_conjGrad
```

    (array([-4.5, -4. , -3.5, -3. , -2.5, -2. , -1.5, -1. , -0.5,  0. ,  0.5,
             1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ]),
     array([-4.5, -4. , -3.5, -3. , -2.5, -2. , -1.5, -1. , -0.5,  0. ,  0.5,
             1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ]),
     9)

```python
x_conjGrad, i_conjGrad = conjGrad(fAx, np.ones_like(b), b, 1e-12)
x_conjGrad, x_sol, i_conjGrad
```

    (array([-4.50000000e+00, -4.00000000e+00, -3.50000000e+00, -3.00000000e+00,
            -2.50000000e+00, -2.00000000e+00, -1.50000000e+00, -1.00000000e+00,
            -5.00000000e-01, -2.22044605e-16,  5.00000000e-01,  1.00000000e+00,
             1.50000000e+00,  2.00000000e+00,  2.50000000e+00,  3.00000000e+00,
             3.50000000e+00,  4.00000000e+00,  4.50000000e+00,  5.00000000e+00]),
     array([-4.5, -4. , -3.5, -3. , -2.5, -2. , -1.5, -1. , -0.5,  0. ,  0.5,
             1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ]),
     9)

```python
import scipy.sparse.linalg

nb_iters = 0

def cg_callback(xk):
    global nb_iters
    nb_iters = nb_iters + 1
    print(f'xk: {xk}')

x_py, i_py = scipy.sparse.linalg.cg(A, b, np.zeros_like(b), callback=cg_callback)
x_py, i_py, nb_iters
```

    xk: [0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
     0.  0.5]
    xk: [-0.5  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.
      0.   0.   0.   0.   0.5  1. ]
    xk: [-1.  -0.5  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.
      0.   0.   0.   0.5  1.   1.5]
    xk: [-1.5 -1.  -0.5  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.
      0.   0.   0.5  1.   1.5  2. ]
    xk: [-2.  -1.5 -1.  -0.5  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.
      0.   0.5  1.   1.5  2.   2.5]
    xk: [-2.5 -2.  -1.5 -1.  -0.5  0.   0.   0.   0.   0.   0.   0.   0.   0.
      0.5  1.   1.5  2.   2.5  3. ]
    xk: [-3.  -2.5 -2.  -1.5 -1.  -0.5  0.   0.   0.   0.   0.   0.   0.   0.5
      1.   1.5  2.   2.5  3.   3.5]
    xk: [-3.5 -3.  -2.5 -2.  -1.5 -1.  -0.5  0.   0.   0.   0.   0.   0.5  1.
      1.5  2.   2.5  3.   3.5  4. ]
    xk: [-4.  -3.5 -3.  -2.5 -2.  -1.5 -1.  -0.5  0.   0.   0.   0.5  1.   1.5
      2.   2.5  3.   3.5  4.   4.5]
    xk: [-4.5 -4.  -3.5 -3.  -2.5 -2.  -1.5 -1.  -0.5  0.   0.5  1.   1.5  2.
      2.5  3.   3.5  4.   4.5  5. ]

    (array([-4.5, -4. , -3.5, -3. , -2.5, -2. , -1.5, -1. , -0.5,  0. ,  0.5,
             1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ]),
     0,
     10)

```python
%matplotlib widget
```

```python
import ipywidgets as widgets
```

```python
def three(x, y, z):
    return (x, y, z)

_ = widgets.interact(
    three,
    x=(0, 10, 1),
    y=True,
    z=['a', 'b', 'c']
)
```

    interactive(children=(IntSlider(value=5, description='x', max=10), Checkbox(value=True, description='y'), Drop…

```python
# set up plot
fig, ax = plt.subplots(figsize=(6, 4))
ax.set_ylim([-4, 4])
ax.grid(True)

# generate x values
x = np.linspace(0, 2 * np.pi, 100)

def my_sine(x, w, amp, phi):
    """
    Return a sine for x with angular frequeny w and amplitude amp.
    """
    return amp*np.sin(w * (x-phi))

@widgets.interact(w=(0, 10, 1), amp=(0, 4, .1), phi=(0, 2*np.pi+0.01, 0.01))
def update(w = 1.0, amp=1, phi=0):
    """Remove old lines from plot and plot new one"""
    [l.remove() for l in ax.lines]
    ax.plot(x, my_sine(x, w, amp, phi), color='C0')
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    interactive(children=(IntSlider(value=1, description='w', max=10), FloatSlider(value=1.0, description='amp', m…

```python

```

# NUME

Instructor: Keny **ORDAZ**

# Final Exam

$\newcommand{\vb}[1]{\boldsymbol{#1}}\newcommand{\RR}[1]{\mathbb{R}^{#1}}
\newcommand{\vt}[1]{\vb{#1}^\top}$

$\newcommand{\x}{\vb{x}}
\newcommand{\xT}{\vt{x}}
\newcommand{\b}{\vb{b}}
\newcommand{\A}{\vb{A}}
\newcommand{\bT}{\vt{b}}
\newcommand{\AT}{\vt{A}}$

## Instructions

- You have exactly 24 hours to submit your examination
- The results and the solving process are to be evaluated
- Clarity of code and good programming practices must be observed
- It must be presented as a **self-contained Jupyter Python Notebook** by e-mail.
- Take all the appropriate precautions to deliver the exam on time.
- Out-of-time deliveries will not be graded.
- Python libraries accepted: `numpy`, `scipy`, `sympy`, `matplotlib`.

```python

```