## Subsurface Data Analytics

### Bayesian Linear Regression for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Bayesian Linear Regression for Subsurface Modeling in Python

Here's a simple workflow, demonstration of Bayesian linear regression for subsurface modeling workflows. This should help you get started with building subsurface models with data analytics and machine learning. Here's some basic details about Bayesian linear regression.

#### Bayesian Updating

The prediction for Bayesian linear regression is distributed:

\begin{equation}
y \sim N(\beta^{T}X, \sigma^{2} I)
\end{equation}

We estimate the model parameter distributions through Bayesian updating for infering the model parameters from a prior and likelihood from training data.

\begin{equation}
p(\beta | y, X) = \frac{p(y,X| \beta) p(\beta)}{p(y,X)}
\end{equation}

In general for continuous features, we must use a sampling method, such as Markov chain Monte Carlo to sample the posterior.

#### Workflow Goals

Learn the basics of isotonic regression in Python for analysis, modeling and prediction of porosity from density. This includes:

* Basic Python workflows and data preparation

* Training / fitting a ridge regression model

* Checking the model and learning about the impact of hyperparameters

#### Objective

Content to support the PGE 383: Subsurface Machine Learning class.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

There are examples below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

Let's import the GeostatsPy package.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
from sklearn.isotonic import IsotonicRegression
from sklearn.model_selection import train_test_split        # train and test split
from sklearn.metrics import explained_variance_score
import seaborn as sns
seed = 73073
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

We will need to install a couple of packages that are not available in the standard Anaconda 3.7 build.

* pymc3 - streamlines Markov chain Monte Carlo Sampling

* arviz - exploratory analysis of Bayesian models

This can be accomplished with 'pip install [package_name]' in a terminal window.

```python
import warnings
warnings.filterwarnings('ignore')
import pymc3 as pm
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
#os.chdir("d:\PGE383")                                       # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'GrainSize_por.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 grain size ($mm$) and porosity (as a percentage) measures from the subsurface. We load the data file with the pandas 'read_csv' function into a data frame we called 'df'.  We preview each with the head function from Pandas DataFrames.

```python
#df = pd.read_csv("GrainSize_Por.csv")                    # read a .csv file in as a DataFrame
df = pd.read_csv("https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/GrainSize_Por.csv")
df.head()

```

      Grainsize
      Porosity

      0
      2.689284
      5.400000

      1
      6.683638
      10.647582

      2
      9.877106
      7.926271

      3
      12.239344
      11.278953

      4
      27.623312
      10.508753

#### Visualize the Data

Looks like a reasonable dataset for a linear regression model.

```python
X_data = df.iloc[:,0]
y_data = df.iloc[:,1]
plt.subplot(111)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data')
plt.title('Porosity from Grainsize with Training Data'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Least Squares Linear Regression

Let's calculate the ordinary least squares (OLS) linear regression model for comparison.

* for a non-informative prior our Bayesian linear regression model parameters will be centered on the OLS solution.

```python
OLS_slope, OLS_intercept, r_value, p_value, std_err = st.linregress(X_data,y_data)

print('The model parameters are, slope (b1) = ' + str(round(OLS_slope,2)) + ', and the intercept (b0) = ' + str(round(OLS_intercept,2)))
```

    The model parameters are, slope (b1) = 0.16, and the intercept (b0) = 8.12

```python
plt.subplot(111)
plt.plot(X_data, y_data, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.plot(X_data, OLS_intercept + OLS_slope*X_data, label='model', color = 'black')
plt.title('Porosity vs Grain Size')
plt.xlabel('Grain Size (mm)')
plt.ylabel('Porosity (%)')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

The following code was based on the work of Will Koehrsen [blog post](https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7) and example workflow in [Jupyter Notebook](https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian_lr/Bayesian%20Linear%20Regression%20Demonstration.ipynb).

https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian_lr/Bayesian%20Linear%20Regression%20Demonstration.ipynb

#### Bayesian Linear Regression

In the following code we take advantage of the pymc3 package's API to build a very readible and compact model.

The parts include:

* specify the prior models for $\beta_0$, $\beta_1$ and $\sigma^2$.

* specify the model estimates

* define the distribution of the predictions and the observated predictor features

* specify the specific form for the MCMC steps

* conduct MCMC sampling

Note the default MCMC method in pymc3 is the Hamiltonian Monte Carlo (HMC).

* HMC reduces correlation between succesive samples by including proposals for moves to distant states, $L$
* No U-Turn Sampler (NUTS) is an extension of HMC that auotmatically tunes L to avoid oscillation

```python
with pm.Model() as linear_all_data:
    # prior distributions, Naive - set to Gaussian with very large variance
    intercept = pm.Normal('Intercept', mu = 0, sd = 10.0)# model B0 - Intercept Prior
    slope = pm.Normal('slope', mu = 0, sd = 10.0)        # model B1 - Slope Prior
    sigma = pm.HalfNormal('sigma', sd = 10.0)            # homoscedastic error in standard deviation

    # specify the model, linear regression
    mean = intercept + slope * X_data                  # estimate of model

    # define the distribution of the predictions from the model and provide the sampled response features
    Y_obs = pm.Normal('Y_obs', mu = mean, sd = sigma, observed = y_data.values[:]) # Observed values

    # specify the sampler
    step = pm.NUTS(target_accept = 0.95) # use the No-U-Turn Sampler (NUTS) for Markov Chains Monte Carlo MCMC, addaptive step size

    # Posterior distribution
    linear_trace_all_data = pm.sample(draws = 500, tune = 500, step = step, random_seed = seed)
    #linear_trace_all_data = pm.sample(draws = 500, tune = 500, step = step, random_seed = seed,discard_tuned_samples = False)

```

    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [sigma, slope, Intercept]
    Sampling 4 chains, 0 divergences: 100%|██████████| 4000/4000 [00:03<00:00, 1166.87draws/s]

#### Summary Statistics for the Model Parameter Posterior Samples

It is useful to assess the mean, dispersion and credible intervals for the model parameters.

```python
def trace_mean(x):
    return pd.Series(np.average(x, 0), name='mean')

def trace_sd(x):
    return pd.Series(np.std(x, 0), name='sd')

def trace_975(x):
    return pd.Series(np.percentile(x, 97.5), name='97.5')

def trace_025(x):
    return pd.Series(np.percentile(x, 2.5), name='2.5')

# Get the expectation / average for the model parameters from sampling the posterior distribution
bayes_exp_intercept = pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean]).iloc[0,0]
bayes_exp_slope = pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean]).iloc[1,0]

pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean, trace_sd, trace_025, trace_975])
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat
      trace_mean
      trace_sd
      trace_025
      trace_975

      Intercept
      8.075
      0.956
      6.315
      9.863
      0.042
      0.030
      510.0
      500.0
      531.0
      685.0
      1.01
      8.075
      0.956
      6.143
      9.957

      slope
      0.159
      0.019
      0.124
      0.195
      0.001
      0.001
      498.0
      498.0
      505.0
      646.0
      1.01
      0.159
      0.019
      0.122
      0.196

      sigma
      2.180
      0.430
      1.500
      2.921
      0.016
      0.012
      711.0
      640.0
      782.0
      848.0
      1.00
      2.180
      0.430
      1.535
      3.172

```python
pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean])
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat
      trace_mean

      Intercept
      8.075
      0.956
      6.315
      9.863
      0.042
      0.030
      510.0
      500.0
      531.0
      685.0
      1.01
      8.075

      slope
      0.159
      0.019
      0.124
      0.195
      0.001
      0.001
      498.0
      498.0
      505.0
      646.0
      1.01
      0.159

      sigma
      2.180
      0.430
      1.500
      2.921
      0.016
      0.012
      711.0
      640.0
      782.0
      848.0
      1.00
      2.180

#### Diagnostic Plots

For each of the model parameters, let's look at the results for the 4 Markov chains.

* distributions / PDFs based on kernel density estimates (KDE) from the 1000 MCMC states

* trace of the samples over each state to observe for burn-in, stuck in local minimums / sufficient variation

```python
pm.traceplot(linear_trace_all_data)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Posterior Distributions

We can take our 1000 states from our 4 Markov chains as samples of the posterior distributions of our Bayesian linear regresion model parameters.

* once again the distributions are represented as PDFs from kernel density estimation

```python
pm.plot_posterior(linear_trace_all_data, credible_interval = 0.95)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Visualize the Credible Intervals for the Model Parameters

We can visualize the credible intervals for all of the model parameters

* one for each Markov chain

```python
pm.forestplot(linear_trace_all_data, credible_interval = 0.95)
plt.show()
```

#### Sampling the Bayesian Regression Model

We can sample from the resulting uncertainty distributions for the model parameters.

* we'll sample 1000 times and visualize the reuslting models

```python
plt.subplot(111)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data', zorder = 3)
pm.plot_posterior_predictive_glm(linear_trace_all_data, samples = 1000, eval=np.linspace(2, 100, 100), linewidth = 1, color = 'red', alpha = 0.02, label = 'Bayesian Posterior Models',lm = lambda x, sample: sample['Intercept'] + sample['slope'] * x, zorder = 1)
plt.plot(X_data, bayes_exp_intercept + bayes_exp_slope*X_data, label='OLS Model', color = 'black', zorder= 2)
plt.title('Porosity from Grainsize Bayesian Model'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Posterior Prediction

Let's pick a specific grain size and apply the above 1000 samples of the model to sample the Bayes posterior prediction.

* note we would still need to add in the noise term, $\sigma$, to the model samples

```python
grain_size = 40
nruns = linear_trace_all_data['slope'].shape[0]

plt.subplot(121)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data', zorder = 3)
pm.plot_posterior_predictive_glm(linear_trace_all_data, samples = 1000, eval=np.linspace(2, 100, 100), linewidth = 1, color = 'red', alpha = 0.02, label = 'Bayesian Posterior Models',lm = lambda x, sample: sample['Intercept'] + sample['slope'] * x, zorder = 1)

plt.plot(X_data, OLS_intercept + OLS_slope*X_data, label='model', color = 'black', zorder= 2)
plt.vlines(x = grain_size,ymin = 0, ymax = 30, label = 'OLS Prediction',colors = 'black', linestyles='--')
plt.title('Porosity from Grainsize Bayesian Model'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()

plt.subplot(122)
model_uncert = linear_trace_all_data['Intercept'] + linear_trace_all_data['slope'] * grain_size
sns.kdeplot(model_uncert, label = 'Bayes Model Uncertainty', color = 'blue')
bayes_prediction = linear_trace_all_data['Intercept'] + linear_trace_all_data['slope'] * grain_size + np.random.normal(loc=0,scale=linear_trace_all_data['sigma'],size=nruns)
sns.kdeplot(bayes_prediction, label = 'Bayes Posterior Prediction', color = 'black')
plt.vlines(x = OLS_intercept + OLS_slope * grain_size,ymin = 0, ymax = 2.5, label = 'OLS Prediction',colors = 'red', linestyles='--')
plt.xlim(0,30),plt.ylim(0,1.0)
plt.xlabel('Porosity (%)'), plt.ylabel('Density');
plt.title('Posterior Prediction for Grain Size = ' + str(grain_size));
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Comments

This was a basic demonstration of Bayesian linear regression. Motivation to work with Bayesian methods:

* work with the model parameter uncertainty distributions
* ingrate prior information.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


    Run in Google Colab

    View source on GitHub

# Multimodality

While preference data in the form of user-item interactions are the backbone of many recommender systems, such data tends to be sparse in nature. One way to address this sparsity is to look beyond the interaction data to the additional information associated with users or with items. The intuition is that items with similarity in "content profiles" would be correlated with similarity in preferences. Multimodality deals with how to model both preference data (one modality) and some content data either on user or item side (other modalities). In this tutorial, we see three forms of additional modalities, namely graph, text, and image, and investigate whether they add value to the the resulting recommendations.

## 1. Setup

```
!pip install --quiet cornac==1.6.1
```

```
import os
import sys
from collections import defaultdict

import numpy as np
import pandas as pd; pd.set_option("max_colwidth", 0);
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

import cornac
from cornac.utils import cache
from cornac.datasets import filmtrust, amazon_clothing
from cornac.eval_methods import RatioSplit
from cornac.models import PMF, SoRec, WMF, CTR, BPR, VBPR
from cornac.data import GraphModality, TextModality, ImageModality
from cornac.data.text import BaseTokenizer

%tensorflow_version 1.x
import tensorflow as tf

print(f"System version: {sys.version}")
print(f"Cornac version: {cornac.__version__}")
print(f"Tensorflow version: {tf.__version__}")

SEED = 42
VERBOSE = True
```

    System version: 3.6.9 (default, Apr 18 2020, 01:56:04)
    [GCC 8.4.0]
    Cornac version: 1.6.1
    Tensorflow version: 1.15.2

## 2. Graph Modality

In recommender systems, graph can be used to represent user social network or item contexts (e.g., co-views, co-purchases).  In this tutorial, we take the former as an example and discuss SoRec [2], a representative model for this class of algorithms.

### Social Recommendation (SoRec)

SoRec model is based on matrix factorization framework. The idea is fusing user-item rating matrix with the user’s social network.  In summary, the *user-item rating matrix* ($R$) and the *user-user graph adjacency matrix* ($G$) are factorized with shared users' latent factors.  The user latent vectors in $\mathbf{U}$ are shared to capture both user preferences as well as their social connections.  The rating prediction is obtained as $\hat{r}_{i,j} = \mathbf{u}_i^T \mathbf{v}_j$, similarly to PMF model.

To learn the model parameters, we minimize the following loss function:

$$ \mathcal{L}(\mathbf{U,V,Z}|\lambda,\lambda_C) = \frac{1}{2} \sum_{r_{i,j} \in \mathcal{R}} (r_{i,j} - \mathbf{u}_i^T \mathbf{v}_j)^2 + \frac{\lambda_C}{2} \sum_{g_{i,h} \in \mathcal{G}} (g_{i,h} - \mathbf{u}_i^T \mathbf{z}_h)^2 + \frac{\lambda}{2} \sum_{i=1}^{N} ||\mathbf{u}_i||^2 + \frac{\lambda}{2} \sum_{j=1}^{M} ||\mathbf{v}_j||^2 + \frac{\lambda}{2} \sum_{h=1}^{N} ||\mathbf{z}_h||^2 $$

where $\lambda_C$ is the relative importance of the social network factorization and $\lambda$ is the regularization weight.

Let's do a comparison between SoRec and its base model PMF on [FilmTrust dataset](http://konect.cc/networks/librec-filmtrust-trust/).

```
K = 20
sorec = SoRec(k=K, max_iter=50, learning_rate=0.001, verbose=VERBOSE, seed=SEED)
pmf = PMF(k=K, max_iter=50, learning_rate=0.001, lambda_reg=0.01, verbose=VERBOSE, seed=SEED)

ratings = filmtrust.load_feedback()
trust = filmtrust.load_trust()

user_graph_modality = GraphModality(data=trust)

ratio_split = RatioSplit(
    data=ratings,
    test_size=0.2,
    rating_threshold=2.5,
    exclude_unknowns=True,
    user_graph=user_graph_modality,
    verbose=VERBOSE,
    seed=SEED,
)

mae = cornac.metrics.MAE()

cornac.Experiment(eval_method=ratio_split, models=[sorec, pmf], metrics=[mae]).run()
```

    /usr/local/lib/python3.6/dist-packages/cornac/data/dataset.py:361: UserWarning: 3 duplicated observations are removed!
      warnings.warn("%d duplicated observations are removed!" % dup_count)

    TEST:
    ...
          |    MAE | Train (s) | Test (s)
    ----- + ------ + --------- + --------
    SoRec | 0.6644 |    0.8935 |   0.3607
    PMF   | 0.7018 |    0.7642 |   0.3115

From the experiment, we see that SoRec achieves lower (better) MAE score as compared to PMF.  This improvement should be explained by useful information from user social network captured inside the model predictions.  Let's take a look at user factors and see if there are connections to the social network.

### Interpreting Recommendation

Below is the variance of each user latent dimension.

```
var_df = pd.DataFrame({"Factor": np.arange(K), "Variance": np.var(sorec.U, axis=0)})
fig, ax = plt.subplots(figsize=(13, 5))
sns.barplot(x="Factor", y="Variance", data=var_df, palette="ch:.25", ax=ax);
```

Let's select two dimensions and see how users are distributed in that 2D space.  In addition, we can visualize connections between two users if they are linked in the social network.

```
TOP2F = (9, 19)
SAMPLE_SIZE = 200

rng = np.random.RandomState(SEED)
sample_inds = rng.choice(np.arange(sorec.U.shape[0]), size=SAMPLE_SIZE, replace=False)
sample_df = pd.DataFrame(data=sorec.U[sample_inds][:, TOP2F], columns=["x", "y"])
g = sns.lmplot(x="x", y="y", data=sample_df, height=11.0, fit_reg=False)
g.ax.set_title("Users in latent space with their social connections", fontsize=16)

adj_mat = sorec.train_set.user_graph.matrix
for i in range(len(sample_inds)):
  for j in range(len(sample_inds)):
    if j != i and adj_mat[sample_inds[i], sample_inds[j]]:
      sns.lineplot(x="x", y="y", data=sample_df.loc[[i, j]])
```

There are indeed some locality of user connections shown in the figure above. That suggests social network information is captured in the user latent factors and has some influence on the recommendations by the SoRec model.

## 3. Text Modality

Often times, we are interested in building a recommender system for textual items (e.g., news, scientific papers), or items associated with text (e.g., titles, descriptions, reviews).  Text is informative and descriptive, therefore, exploiting textual information for better recommendations is an important topic in recommender systems.  In this tutorial, we introduce CTR [3], a recommendation model that combines matrix factorization and probablistic topic modeling.

### Collaborative Topic Regression (CTR)

Under factorization framework, adoption prediction is in the form of $\hat{r}_{i,j} = \mathbf{u}_i^T \mathbf{v}_j $.  The intuition in CTR model is that two items with similar topics would behave similarly. Thus, item latent factors $\mathbf{v_j}$ is assumed to be drawn from a Normal distribution:

$$
\mathbf{v}_j \sim \mathcal{N}(\mathbf{\theta}_j, \lambda^{-1} \mathbf{I})
$$

where the mean $\mathbf{\theta}_j$ is a vector indicating topic proportions of the item $j$. It is equivalent to:

\begin{align}
\mathbf{v}_j &= \mathbf{\theta}_j + \mathbf{\epsilon}_j \\
\mathbf{\epsilon}_j &\sim \mathcal{N}(\mathbf{0}, \lambda^{-1} \mathbf{I})
\end{align}

Please refer to paper [3] for the generative process of CTR model.

CTR also extends matrix factorization, in which the base model is WMF under implicit feedback setting. The adoption $p_{i,j}$ and confidence $c_{i,j}$ are defined as follows:

\begin{equation}
p_{i,j} =
\begin{cases}
r_{i, j} &\mbox{if } r_{i,j} > 0 \\
0 & \mbox{otherwise}
\end{cases}
\end{equation}

\begin{equation}
c_{i,j} =
\begin{cases}
a & \mbox{if } r_{i,j} > 0 \\
b & \mbox{otherwise }
\end{cases}
\end{equation}

The learning of CTR model is done via minimizing the following negative log-likelihood:

$$ \mathcal{L}(\mathbf{U,V,\theta, \beta}|\lambda) = \frac{1}{2} \sum_{i,j} c_{i,j} (p_{i,j} - \mathbf{u}_i^T \mathbf{v}_j)^2 - \sum_{j}\sum_{n} \log \big( \sum_{k=1}^K \mathbf{\theta}_{j,k} \mathbf{\beta}_{k,w_{jn}} \big) + \frac{\lambda}{2} \sum_{i=1}^{N} ||\mathbf{u}_i||^2 + \frac{\lambda}{2} \sum_{j=1}^{M} (\mathbf{v}_j - \mathbf{\theta}_j)^T (\mathbf{v}_j - \mathbf{\theta}_j) $$

It is an iterative procedure of alternating between three steps:
- Optimize for user and item latent vectors, $\mathbf{u}_i$ and $\mathbf{v}_j$, based on the current topic proportions $\mathbf{\theta}_j$.
- Optimize for topic proportions $\mathbf{\theta}_j$ based on the current vectors $\mathbf{u}_i$ and $\mathbf{v}_j$ and topic words $\mathbf{\beta}_k$.
- Optimize for topic words $\mathbf{\beta}_k$ based on the current topic proportions $\mathbf{\theta}_i$.

Let's experiment with two models CTR and WMF on a dataset from Amazon Clothing category.  Using this dataset, CTR will learn topics from item description.

```
K = 20
ctr = CTR(k=K, max_iter=50, a=1.0, b=0.01, lambda_u=0.01, lambda_v=0.01, verbose=VERBOSE, seed=SEED)
wmf = WMF(k=K, max_iter=50, a=1.0, b=0.01, learning_rate=0.005, lambda_u=0.01, lambda_v=0.01,
          verbose=VERBOSE, seed=SEED)

ratings = amazon_clothing.load_feedback()
docs, item_ids = amazon_clothing.load_text()

item_text_modality = TextModality(
    corpus=docs,
    ids=item_ids,
    tokenizer=BaseTokenizer(sep=" ", stop_words="english"),
    max_vocab=8000,
    max_doc_freq=0.5,
)

ratio_split = RatioSplit(
    data=ratings,
    test_size=0.2,
    rating_threshold=4.0,
    exclude_unknowns=True,
    item_text=item_text_modality,
    verbose=VERBOSE,
    seed=SEED,
)

rec_50 = cornac.metrics.Recall(50)

cornac.Experiment(eval_method=ratio_split, models=[ctr, wmf], metrics=[rec_50]).run()
```

    TEST:
    ...
        | Recall@50 | Train (s) | Test (s)
    --- + --------- + --------- + --------
    CTR |    0.2239 |  123.8224 |   1.3055
    WMF |    0.1582 |   12.3231 |   1.3365

The results show that CTR model performs significantly better than WMF model in terms of Recall@50, which is due to the contribution of items' textual information.

### Interpreting Recommendation

As CTR is based on topic model, it has intuitive explainability both in terms of model parameters as well as its recommendations. First, we can see what the top words of each topic are, based on learned topic-word distributions $\mathbf{\beta}$:

```
vocab = ctr.train_set.item_text.vocab
topic_word_dist = ctr.model.beta.T[:, -ctr.train_set.item_text.max_vocab:]
top_word_inds = np.argsort(topic_word_dist, axis=1) + 4  # ingore 4 special tokens

topic_words = {}
topic_df = defaultdict(list)
print("WORD TOPICS:")
for t in range(len(topic_word_dist)):
  top_words = vocab.to_text(top_word_inds[t][-10:][::-1], sep=", ")
  topic_words[t+1] = top_words
  topic_df["Topic"].append(t + 1)
  topic_df["Top words"].append(top_words)
topic_df = pd.DataFrame(topic_df)
topic_df
```

    WORD TOPICS:

      Topic
      Top words

      0
      1
      belt, flatlock, case, size, better, choose, earrings, soft, warm, collar

      1
      2
      shirt, style, breathable, princess, youre, colors, quality, come, people, socks

      2
      3
      easy, collection, colors, rubber, water, quickly, stretch, lightweight, design, body

      3
      4
      activities, handle, helmet, star, way, flat, makes, day, half, tee

      4
      5
      fit, sweat, mesh, large, pocket, fleece, body, necklace, imported, travel

      5
      6
      wicks, material, dry, fit, panels, piece, insect, construction, durability, standard

      6
      7
      support, company, fabric, apparel, fun, sizes, hip, outdoor, fiber, features

      7
      8
      x, merino, sleeve, fun, items, water, weather, bracelets, comfortable, display

      8
      9
      great, performance, black, pockets, inches, make, away, design, jewelry, unique

      9
      10
      fits, sun, store, engineered, pocket, shield, conditions, mobility, hat, maximum

      10
      11
      costume, products, small, inside, classic, lightweight, comfort, white, ideal, designed

      11
      12
      bag, elastic, jacket, perfect, favorite, hat, d, box, look, feature

      12
      13
      s, use, sure, soft, comfort, easy, products, skin, displays, moisture

      13
      14
      size, comfortable, product, warm, need, ua, accessory, clean, footwear, silver

      14
      15
      high, storage, height, waistband, wide, way, chest, include, growth, new

      15
      16
      delivers, tray, nylon, sturdy, free, pirate, builtin, allows, fairy, wear

      16
      17
      cool, shoulder, provide, x, cold, prevents, womens, color, beautiful, years

      17
      18
      pockets, mask, durable, moisture, jewelry, features, new, offers, comfort, brushed

      18
      19
      straps, clothing, metal, way, microbes, shoes, silver, easily, toe, polyester

      19
      20
      complete, wear, fabric, features, display, measures, merino, technology, gift, want

As item vectors are close to its topic proportions, user vectors can also be used to explain what topics they are interested in.

```
UIDX = 123
TOPK = 5

item_id2idx = ctr.train_set.iid_map
item_idx2id = list(ctr.train_set.item_ids)

print(f"USER {UIDX} TOP-3 TOPICS:")
topic_df.loc[np.argsort(ctr.U[UIDX])[-3:][::-1]]
```

    USER 123 TOP-3 TOPICS:

      Topic
      Top words

      14
      15
      high, storage, height, waistband, wide, way, chest, include, growth, new

      13
      14
      size, comfortable, product, warm, need, ua, accessory, clean, footwear, silver

      12
      13
      s, use, sure, soft, comfort, easy, products, skin, displays, moisture

Those are the topics most relevant to her interests. What will be recommendations to this user?

```
recommendations, scores = ctr.rank(UIDX)
print(f"\nTOP {TOPK} RECOMMENDATIONS FOR USER {UIDX}:")
rec_df = defaultdict(list)
for i in recommendations[:TOPK]:
  rec_df["URL"].append(f"https://www.amazon.com/dp/{item_idx2id[i]}")
  rec_df["Description"].append(ctr.train_set.item_text.corpus[i])
pd.DataFrame(rec_df)
```

    TOP 5 RECOMMENDATIONS FOR USER 123:

      URL
      Description

      0
      https://www.amazon.com/dp/B002M62ES6
      Accessorize your Spider-man Costume with these Gloves. Fits most child sizes from 4-12.

      1
      https://www.amazon.com/dp/B002MKPKFG
      This glamorous dress is perfect for a night at the ball. The stretchy blue velvet bodice is elegantly detailed with a gathered white inlay, ruffled cap sleeves, sparkling silver trim and a white rose. The silver trimmed waist drapes outlined with ruffle capture that authentic Cinderella look.The dress is designed with an empire waist that sits 1.5 inches above the waist. They are machine washable and made for long repeated use, unlike many one-off Halloween costumes that are not made for everyday play.  Our stretchy accommodating fabrics will allow for a comfortable fit for each child within the respective age ranges and there are no itchy fabrics or trims.Each dress comes with an exclusive Princess Hairbow as part of the bundle set.

      2
      https://www.amazon.com/dp/B001NAHW2U
      LACY 14K GOLD on 925 Silver Charm BEAD compatible with European Bracelet

      3
      https://www.amazon.com/dp/B000HS6NEC
      Includes jacket, elastic pants, hat, belt, whistle, and gun holster. Child Size Small (4-6).

      4
      https://www.amazon.com/dp/B001EDF3OU
      The Child Indiana Jones Hat and Whip are two must have accessories for your child's Officially Licensed Indiana Jones costume. This Indiana Jones costume accessory contains the two hallmarks of Indy's style: his awesome whip and his trademark hat.

Interestingly, in the desctiption of top-5 recommendations, we see words relevant to the user's topics (e.g., *Accessorize, Gloves, Bracelet, Silver*). This would suggest that CTR model can capture user preferences via the notions of item topics.

---

Let's check out another user with *index = 321*. Similarly, we see the top-3 topics this user cares most about:

```
UIDX = 321

print(f"USER {UIDX} TOP-3 TOPICS:")
topic_df.loc[np.argsort(ctr.U[UIDX])[-3:][::-1]]
```

    USER 321 TOP-3 TOPICS:

      Topic
      Top words

      19
      20
      complete, wear, fabric, features, display, measures, merino, technology, gift, want

      2
      3
      easy, collection, colors, rubber, water, quickly, stretch, lightweight, design, body

      10
      11
      costume, products, small, inside, classic, lightweight, comfort, white, ideal, designed

The topics include clothing with certain characteristics (*easy, water, stretch, lightweight, comfort*).

Based on this, how would the model give recommendations to the user?

```
recommendations, scores = ctr.rank(UIDX)
print(f"\nTOP {TOPK} RECOMMENDATIONS FOR USER {UIDX}:")
rec_df = defaultdict(list)
for i in recommendations[:TOPK]:
  rec_df["URL"].append(f"https://www.amazon.com/dp/{item_idx2id[i]}")
  rec_df["Description"].append(ctr.train_set.item_text.corpus[i])
pd.DataFrame(rec_df)
```

    TOP 5 RECOMMENDATIONS FOR USER 321:

      URL
      Description

      0
      https://www.amazon.com/dp/B000MARQFA
      You demand a lot from your pants when you're on the move and the PROPPER Tactical Pant delivers. Nine pockets hold everything from your smartphone to your multi-tool. Rugged canvas fabric offers durability and strength, with DuPont Teflon fabric protector to repel stains and liquids. An action-stretch waistband adds comfort. And with our innovative wallet "pocket in a pocket" you'll appreciate not sitting on your wallet all day long. Come with a free matching belt (12.99 value).

      1
      https://www.amazon.com/dp/B001VIH648
      You demand a lot from your pants when you're on the move and the PROPPER Tactical Pant delivers. Nine pockets hold everything from your smartphone to your multi-tool. Rugged canvas fabric offers durability and strength, with DuPont Teflon fabric protector to repel stains and liquids. An action-stretch waistband adds comfort. And with our innovative wallet "pocket in a pocket" you'll appreciate not sitting on your wallet all day long. Come with a free matching belt (12.99 value).

      2
      https://www.amazon.com/dp/B002V44LTO
      Sleeves - Made From 50% nylon & 50% cotton. Chest & Back - Made from 40% Acrylic & 60% Cotton. Resists melting & dripping. Melting point is rated to 230 degrees Fahrenheit. Moisture wicking. Lightweight & breathable. Zippered sleeve storage pockets. Mock turtle neck. Anti abrasion padded elbow patches. Gusseted sleeves. Glint tape holder for IR tape. Loop on sleeve pockets for attaching name, rank and flag. Colors Available: Black, ACU Digital Camouflage, Olive Drab, Desert Digital Camouflage & Digitial Woodland Camouflage. Sizes to choose from: Small, Medium, Large, X-Large, 2X-Large & 3X-Large.

      3
      https://www.amazon.com/dp/B002FIVZD6
      TRU-SPEC is one of the leading suppliers of uniforms and personal equipment to military, law enforcement and public safety markets.&#xA0; Thousands of police, firefighters, EMTs and military personnel use and depend on our products daily. The name TRU-SPEC has become synonymous with quality, innovation and service. Gunnery Sgt. R. Lee Ermey, USMC (Ret.) known as "The Gunny" has become the official celebrity spokesperson for TRU-SPEC's line of tactically inspired apparel.

      4
      https://www.amazon.com/dp/B00062B1IC
      The PROPPER BDU trouser offers an authentic military cut in comfortable, durable Battle Rip fabric. Sewn to military specification, this button-fly BDU is fade, shrink and wrinkle resistant. It features felled seams and reinforced seat and knee for maximum durability. Six pockets with fused pocket flaps ensure a clean, professional look. With adjustable waist tabs and durable drawstring leg closures for a secure fit.

From the top-5 recommendations, we notice that those items are *military/outdoor/sports* clothing, based on their descriptions as well as the URL to Amazon website. That is also evident from the user's topics that we observe above.

## 4. Image Modality

In some contexts, item images are informative (e.g., fashion). With the existence of effective methods to learn image representation, using item images in recommender systems is gaining popularity. In this tutorial, we present VBPR [4], a recommendation model making use of item image features extracted from pre-trained Convolutional Neural Network (CNN).

### Visual Bayesian Personalized Ranking (VBPR)

VBPR, which is also based on matrix factorization, is an extension of BPR model.  The novelty of VBPR is on how item visual features incorporated into the matrix factorization framework.  The preference score user $i$ giving to item $j$ is predicted as follows:

$$
\hat{r}_{i,j} = \alpha + b_i + b_j + \mathbf{u}_i^T \mathbf{v}_j + \mathbf{p}_{u}^T(\mathbf{E} \times \mathbf{f}_j) + \mathbf{\Theta}^T \mathbf{f}_j
$$

where:
- $\alpha, b_i, b_j$ are global bias, user bias, and item bias, respectively
- $\mathbf{u}_i \in \mathbb{R}^K$ and $\mathbf{v}_j \in \mathbb{R}^K$ are user and item latent vectors, respectively
- $\mathbf{f}_j \in \mathbb{R}^D$ is the item image feature vector
- $\mathbf{p}_i \in \mathbb{R}^Q$ is user visual preference, and $(\mathbf{E} \times \mathbf{f}_j) \in \mathbb{R}^Q$ is item visual representation with $\mathbf{E} \in \mathbb{R}^{K \times D}$ is the projection from visual feature space into preference space
- $\mathbf{\Theta} \in \mathbb{R}^D$ is global visual bias vector

Learning parameters of VBPR model can be done, similarly to BPR, via minimizing the following negative log-likelihood:

$$ \mathcal{L}(\mathbf{U,V,b,E,\Theta, P}|\lambda) = \sum_{(j >_i l) \in \mathbf{S}} \ln (1 + \exp\{-(\hat{r}_{i,j} - \hat{r}_{i,l})\}) + \frac{\lambda}{2} \sum_{i=1}^{N} (||\mathbf{u}_i||^2 + ||\mathbf{p}_i||^2) + \frac{\lambda}{2} \sum_{j=1}^{M} (b_j + ||\mathbf{v}_j||^2) + \frac{\lambda}{2} ||\mathbf{\Theta}||^2 + \frac{\lambda}{2} ||\mathbf{E}||^2_2 $$

Noted that global bias $\alpha$ and user bias $b_i$ do not affect the ranking of items, thus they are redundant and removed from the model parameters.

Let's compare VBPR and BPR models with an experiment on Amazon Clothing dataset.

```
K = 10
vbpr = VBPR(k=K, k2=K, n_epochs=50, batch_size=100, learning_rate=0.001,
            lambda_w=1, lambda_b=0.01, lambda_e=0.0, use_gpu=True, verbose=VERBOSE, seed=SEED)
bpr = BPR(k=(K * 2), max_iter=50, learning_rate=0.001, lambda_reg=0.001, verbose=VERBOSE, seed=SEED)

ratings = amazon_clothing.load_feedback()
img_features, item_ids = amazon_clothing.load_image()

item_image_modality = ImageModality(features=img_features, ids=item_ids, normalized=True)

ratio_split = RatioSplit(
    data=ratings,
    test_size=0.2,
    rating_threshold=4.0,
    exclude_unknowns=True,
    item_image=item_image_modality,
    verbose=VERBOSE,
    seed=SEED,
)

auc = cornac.metrics.AUC()

cornac.Experiment(eval_method=ratio_split, models=[vbpr, bpr], metrics=[auc]).run()
```

    Optimization finished!

    TEST:
    ...
         |    AUC | Train (s) | Test (s)
    ---- + ------ + --------- + --------
    VBPR | 0.7053 |   51.0625 |   1.4523
    BPR  | 0.5992 |    0.1625 |   1.3823

The results show that VBPR obtains higher performance than BPR in terms of AUC. That can be attributed to the usage of item visual features. Let's try to get some ideas on how the visual features contributing to the model recommendations.

### Interpreting Recommendation

First, let's take a look at items rated by a particular user. Then we see how VBPR model recommends to the user.

```
UIDX = 111

item_id2idx = vbpr.train_set.iid_map
item_idx2id = list(vbpr.train_set.item_ids)
img_features = vbpr.train_set.item_image.features
item_desc = {iid: desc for desc, iid in zip(*amazon_clothing.load_text())}

print(f"RATED ITEMS BY USER {UIDX}:")
rated_items = np.nonzero(vbpr.train_set.matrix[UIDX].A)[1]
rated_df = defaultdict(list)
for i in rated_items:
  rated_df["URL"].append(f"https://www.amazon.com/dp/{item_idx2id[i]}")
  rated_df["Description"].append(item_desc[item_idx2id[i]])
pd.DataFrame(rated_df)
```

    RATED ITEMS BY USER 111:

      URL
      Description

      0
      https://www.amazon.com/dp/B00BXQF1XG
      Disguise Boy's Marvel Ultimate Spider-Man Classic Costume, 3T-4T Jumpsuit.

      1
      https://www.amazon.com/dp/B002M62ES6
      Accessorize your Spider-man Costume with these Gloves. Fits most child sizes from 4-12.

```
TOPK = 5

recommendations, scores = vbpr.rank(UIDX)

print(f"\nTOP {TOPK} RECOMMENDATIONS:")
rec_df = defaultdict(list)
for i in recommendations[:TOPK]:
  cf_contrib = vbpr.beta_item[i] + np.dot(vbpr.gamma_item[i], vbpr.gamma_user[UIDX])
  vis_contrib = vbpr.visual_bias[i] + np.dot(vbpr.theta_item[i], vbpr.theta_user[UIDX])
  rec_df["URL"].append(f"https://www.amazon.com/dp/{item_idx2id[i]}")
  rec_df["Score"].append(scores[i])
  rec_df["CF Contribution"].append(cf_contrib)
  rec_df["Visual Contribution"].append(vis_contrib)
  rec_df["Description"].append(item_desc[item_idx2id[i]])
pd.DataFrame(rec_df)
```

    TOP 5 RECOMMENDATIONS:

      URL
      Score
      CF Contribution
      Visual Contribution
      Description

      0
      https://www.amazon.com/dp/B002M62ES6
      7.457715
      1.116012
      6.341703
      Accessorize your Spider-man Costume with these Gloves. Fits most child sizes from 4-12.

      1
      https://www.amazon.com/dp/B007DLVLAW
      7.312136
      1.509102
      5.803034
      Patented elastic shoelace and fastening system. Engineered and designed to improve performance and keep laces locked and secure throughout the course of a race, game, practice, or workout.

      2
      https://www.amazon.com/dp/B00BHPRRKS
      6.589272
      -0.176793
      6.766065
      Jumpsuit with muscle torso and arms attached belt.

      3
      https://www.amazon.com/dp/B0000CNR1L
      6.477900
      0.324168
      6.153731
      No more digging around in deep toiletries bags for that pair of tweezers or spare safety pin you know is in there somewhere. Made of clear vinyl for quick viewing, the ten compartments on this unique organizer keep everything from dental floss to shaving cream readily retrievable. Organized in four zippered rows, the compartments vary in size and keep contents secure. When empty, the bag folds to a flat 10 by 14 inches. When filled, it can be laid inside a suitcase. The hook allows the bag to be hung inside a garment bag if desired, and then looped over a bathroom door or towel hook for accessibility.  Made of PVC microfiber, the bag extends to 28 by 14 inches when fully opened.--Ann Bieri

      4
      https://www.amazon.com/dp/B00BXQF1XG
      6.426818
      0.020040
      6.406779
      Disguise Boy's Marvel Ultimate Spider-Man Classic Costume, 3T-4T Jumpsuit.

The recommendations (1, 3, 5), with high visual contribution, are indeed visually similar to the two items rated by the user.  That gives us some intuition on how VPBR model makes use of the item visual features in the predictions.

## References

1.   Aggarwal, C. C. (2016). Recommender systems (Vol. 1). Cham: Springer International Publishing.
2.   Ma, H., Yang, H., Lyu, M. R., & King, I. (2008, October). Sorec: social recommendation using probabilistic matrix factorization. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 931-940).
3.   Wang, C., & Blei, D. M. (2011, August). Collaborative topic modeling for recommending scientific articles. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 448-456).
4.   He, R., & McAuley, J. (2016, February). VBPR: visual bayesian personalized ranking from implicit feedback. In Thirtieth AAAI Conference on Artificial Intelligence.
5.   Cornac - A Comparative Framework for Multimodal Recommender Systems (https://cornac.preferred.ai/)


```python
import numpy as np
import matplotlib.pyplot as plt
import math
from IPython.display import display, clear_output
import time
%matplotlib inline
%load_ext autoreload
%autoreload 2
%matplotlib notebook
```

### Policy Iteration

\begin{align}
V_{k+1}(s) &= \sum_a\pi(a|s)\sum_{s',r}p(s',r|s,a)(r+\gamma V_k(s'))\\
\pi'(s) &= \arg\max_a\sum_{s',r}p(s',r|s,a)(r+\gamma V_\pi(s'))
\end{align}

where $V(s)$ is the state-value function of state $s$, $k$ is the iteration of the for-loop, $a$ the action state took in the state $s$, $s'$ the next state, $r$ the reward led by taking the action $a$ in state $s$, $\pi$ the policy and $\gamma$ the discount.

- Policy Evaluation
$$\max_s|V_{k+1}(s)-V_k(s)|<\theta$$
- Policy Improvement
$$V_{\pi'}(s) \geq V_\pi(s)$$
The new policy $\pi'$ should be as good as, or better than, the current policy $\pi$.

```python
def GridLine(col,row):
    ax = plt.gca();
    # Major ticks
    ax.set_xticks(np.arange(0, col, 1));
    ax.set_yticks(np.arange(0, row, 1));
    # Labels for major ticks
    ax.set_xticklabels(np.arange(1, col+1, 1));
    ax.set_yticklabels(np.arange(1, row+1, 1));
    # Minor ticks
    ax.set_xticks(np.arange(-.5, col, 1), minor=True);
    ax.set_yticks(np.arange(-.5, row, 1), minor=True);
    # Gridlines based on minor ticks
    ax.grid(which='minor', color='w', linestyle='-', linewidth=2)
    return ax
```

```python
aUp = 0; aDown = 1; aLeft = 2; aRight = 3;
class World():
    def __init__(self,mode):
        self.mode = mode
    def getParamSet(self):
        if self.mode == 'small':
            paramSet = self.getsmall()
        elif self.mode == 'grid':
            paramSet = self.getgrid()
        elif self.mode == 'cliff':
            paramSet = self.getcliff()
        return paramSet
    def getsmall(self):
        paramSet = dict()
        paramSet['rowCount'] = 4; paramSet['colCount'] = 4;
        paramSet['obsSet'] = np.array([[1,1],[2,1],[1,2]])
        paramSet['startState'] = np.array([0,0])
        paramSet['goalState'] = np.array([3,3])
        paramSet['badSet'] = np.array([])
        paramSet['pGoodTrans'] = .8; paramSet['bias'] = .5
        paramSet['rStep']=-1; paramSet['rGoal']=10
        paramSet['rBad'] = -6; paramSet['gamma'] = .9
        return paramSet
    def getgrid(self):
        paramSet = dict()
        paramSet['rowCount'] = 9; paramSet['colCount'] = 12;
        paramSet['obsSet'] = np.array([[8,6],[7,6],[6,6],
                                      [5,6],[4,6],[3,6],
                                      [3,7],[3,8],[3,9]])
        paramSet['startState'] = np.array([0,1])
        paramSet['goalState'] = np.array([7,8])
        paramSet['badSet'] = np.array([[2,1]])
        paramSet['pGoodTrans'] = .7; paramSet['bias'] = .5
        paramSet['rStep']=-1; paramSet['rGoal']=10
        paramSet['rBad'] = -6; paramSet['gamma'] = .9
        return paramSet
    def getcliff(self):
        paramSet = dict()
        paramSet['rowCount'] = 5; paramSet['colCount'] = 10;
        paramSet['obsSet'] = np.array([[0,9],[1,9],[2,9],
                                      [3,9],[4,9]])
        paramSet['startState'] = np.array([4,0])
        paramSet['goalState'] = np.array([4,8])
        paramSet['badSet'] = np.array([[4,1],[4,2],[4,3],[4,4],
                                      [4,5],[4,6],[4,7]])
        paramSet['pGoodTrans'] = 1; paramSet['bias'] = 0
        paramSet['rStep']=-1; paramSet['rGoal']=10
        paramSet['rBad'] =-100; paramSet['gamma'] = .9
        return paramSet
```

```python
def getDir(a,ang):
    leftLUT = np.array([2,3,1,0])
    rightLUT = np.array([3,2,0,1])
    if ang == 0:
        Dir = a
    elif ang == -1:
        Dir = leftLUT[a]
    elif ang == 1:
        Dir = rightLUT[a]
    return Dir
```

```python
def getState(s,Dir,paramSet):
    cLUT = np.array([0,0,-1,1])
    rLUT = np.array([-1,1,0,0])
    rc = np.array(np.unravel_index(s,
                    (paramSet['rowCount'],paramSet['colCount'])))
    rc[0] = rc[0] + rLUT[Dir]
    rc[1] = rc[1] + cLUT[Dir]

    cond1 = np.any(rc<0)
    cond2 = np.any(rc[0]>paramSet['rowCount']-1)
    cond3 = np.any(rc[1]>paramSet['colCount']-1)
    cond4 = np.any(np.sum(np.abs(
        paramSet['obsSet']-
        np.tile(rc,(paramSet['obsSet'].shape[0],1))),1) ==0)

    if cond1 or cond2 or cond3 or cond4:
        sout = s
    else:
        sout = np.ravel_multi_index(rc,
                        (paramSet['rowCount'],paramSet['colCount']))
    return sout
```

```python
def initGridworld(paramSet):
    model = dict()
    model['stateCount'] = paramSet['colCount']*paramSet['rowCount']+1
    model['gamma'] = paramSet['gamma']
    model['startState'] = np.ravel_multi_index(
                            paramSet['startState'],
                            (paramSet['rowCount'],paramSet['colCount']))
    model['goalState'] = np.ravel_multi_index(
                            paramSet['goalState'],
                            (paramSet['rowCount'],paramSet['colCount']))
    model['R'] = np.tile(paramSet['rStep'], (model['stateCount'], 4))
    model['R'][model['stateCount']-1,:] = 0
    model['R'][model['goalState'],:] = paramSet['rGoal']
    for i in range(paramSet['badSet'].shape[0]):
        badState=np.ravel_multi_index(
                    paramSet['badSet'][i,:],
                    (paramSet['rowCount'],paramSet['colCount']))
        model['R'][badState,:] = paramSet['rBad']

    model['P'] = np.zeros((model['stateCount'],model['stateCount'],4))
    for a in range(4):
        for s in range(model['stateCount']):
            if s == model['stateCount']-1:
                model['P'][s,s,a] = 1
                continue

            rc = np.array(np.unravel_index(s,
                    (paramSet['rowCount'],paramSet['colCount'])))
            endSet=np.concatenate((paramSet['obsSet'],
                               paramSet['goalState'].reshape(-1,2)))
            cond=np.sum(np.abs(endSet-np.tile(rc,(endSet.shape[0],1))),1)
            if np.any(cond==0):
                model['P'][s,model['stateCount']-1,a]=1
            else:
                for ang in np.linspace(-1,1,3):
                    ang = int(ang)
                    Dir = getDir(a,ang)
                    sout = getState(s,Dir,paramSet)
                    if ang == 0:
                        p= paramSet['pGoodTrans']
                    elif ang == -1:
                        p=(1-paramSet['pGoodTrans'])*paramSet['bias']
                    elif ang == 1:
                        p=(1-paramSet['pGoodTrans'])*(1-paramSet['bias'])
                    model['P'][s,sout,a]=model['P'][s,sout,a]+p

    return model
```

```python
def plotVP(Vopt,Popt,paramSet):
    Vopt = Vopt[:-1]
    Popt = Popt[:-1]
    # value plot
    v = Vopt
    for rc in paramSet['obsSet']:
        tmp = np.ravel_multi_index(rc,
                    (paramSet['rowCount'],paramSet['colCount']))
        v[tmp] = -np.inf

    v = v.reshape((paramSet['rowCount'],paramSet['colCount']))
    plt.imshow(v)

    # policy plot directions
    thetaLUT = np.pi*np.array([.5,-.5,1,0])
    for s in range(paramSet['rowCount']*paramSet['colCount']):
        isbad = False
        for rc in paramSet['obsSet']:
            if s == np.ravel_multi_index(rc,
                    (paramSet['rowCount'],paramSet['colCount'])):
                isbad = True
                break
            if not isbad:
                rc = np.array(np.unravel_index(s,
                    (paramSet['rowCount'],paramSet['colCount'])))
                r = rc[0]; c = rc[1]
                ang = thetaLUT[Popt[s]]
                plt.plot(c,r,'ko')
                tmpx = np.array([c,c+.25*np.cos(ang)])
                tmpy = np.array([r,r-.25*np.sin(ang)])
                plt.plot(tmpx,tmpy,'k-')

    plt.plot(paramSet['goalState'][1],paramSet['goalState'][0],
             'r*',markersize=15)
    plt.plot(paramSet['startState'][1],paramSet['startState'][0],
            'rp',markersize=15)
```

### Policy Iteration

```python
def policyIteration(model,maxIter):
    v = np.zeros((model['stateCount'],1))
    pi= np.ones((model['stateCount'],1)).astype(int)

    for i in range(maxIter):
        v_ = np.zeros((model['stateCount'],1))

        for j in range(maxIter):
            for s in range(model['stateCount']):
                tmpw=np.dot(model['P'][s,:,pi[s]],
                        model['R'][s,pi[s]]+model['gamma']*v[:])
                v_[s] = np.sum(tmpw)

            if (v_-v).max()<.001:
                break
            else:
                v = v_

        for s in range(model['stateCount']):
            tmpa = np.zeros((4,1))
            for a in range(4):
                tmpa[a]=np.dot(model['P'][s,:,a],
                            model['R'][s,a]+model['gamma']*v[:])

            V = tmpa.max()
            choice = int(tmpa.argmax())
            tmpV = np.dot(model['P'][s,:,pi[s]],
                            model['R'][s,pi[s]]+model['gamma']*v[:])
            if V >= tmpV:
                pi[s] = choice

        pi = pi.astype(int)
    return v,pi
```

### Small world

```python
world = World('small')
paramSet = world.getParamSet()
model = initGridworld(paramSet)
v,pi = policyIteration(model,10000)
plotVP(v,pi,paramSet)
GridLine(paramSet['colCount'],paramSet['rowCount'])
```

### Grid world

```python
world = World('grid')
paramSet = world.getParamSet()
model = initGridworld(paramSet)
v,pi = policyIteration(model,10000)
plotVP(v,pi,paramSet)
GridLine(paramSet['colCount'],paramSet['rowCount'])
```

### Cliff world

```python
world = World('cliff')
paramSet = world.getParamSet()
model = initGridworld(paramSet)
v,pi = policyIteration(model,10000)
plotVP(v,pi,paramSet)
GridLine(paramSet['colCount'],paramSet['rowCount'])
```

```python

```