Table of Contents

1  Calibrated Recommendations

1.1  Preparation

1.2  Deep Dive Into Calibrated Recommendation

1.2.1  Calibration Metric

1.2.2  Generating Calibrated Recommendations

1.3  End Note

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(css_style='custom2.css', plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import os
import time
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.sparse import csr_matrix
from implicit.bpr import BayesianPersonalizedRanking
from implicit.evaluation import train_test_split, precision_at_k

%watermark -a 'Ethen' -d -t -v -p scipy,numpy,pandas,matplotlib,implicit
```

    Ethen 2018-10-17 10:09:55

    CPython 3.6.4
    IPython 6.4.0

    scipy 1.1.0
    numpy 1.14.1
    pandas 0.23.0
    matplotlib 2.2.2
    implicit 0.3.8

# Calibrated Recommendations

When a user has watched, say, 70% romance movies and 30% action movies in the past, then it is reasonable to expect the personalized list of recommended movies to be comprised of 70% romance and 30% action movies as well since we would like to cover the user's diverse set of interests. A recommendation that actually reflect most if not all of the user's interest is considered a **Calibrated Recommendation**. But the question is, does our recommendation exhibit this trait?

Recommendation algorithm provides personalized user experience past on the user's past historical interaction with the product/system/website. However, when serving the recommendation such as recommendation top 10 movies that we think the user might be interested in, a recommendation engine that is solely measured based on ranking metrics can easily generate recommendations that focus on the main area of interests, resulting the user's other area of interest to be under-represented, or worse, absent in the final recommendation.

To hit the notion home, using the example above, given a user that has watched 70% romance movies and 30% action movies, if we were to solely measure the metric based on precision, we can say we can achieve the best performance by predicting the majority genre, i.e. we will recommend 100% romance movies and we can expect the user to interact with those recommendations 70% of the time. On other other hand, if we were to recommend 70% romance movies and 30% action movies, then we would expect our recommendation to only be correct 0.7 * 0.7 + 0.3 * 0.3 = 58% of the time.

Throughout the rest of this notebook, we will take a look at if the phenomenon of crowding out user's sub-interest occurs with our recommendation, develop a quantitative metric to measure how severe this issue is and implement a post-preprocessing logic that is agnostic of the underlying recommendation algorithm we decided to use to ensure the recommendation becomes more calibrated.

## Preparation

We'll be using the publicly available movielens-20m dataset throughout this experiment. We can download it via the following [link](https://www.kaggle.com/grouplens/movielens-20m-dataset). There's multiple data under that folder, we can select download all to make things easier.

The algorithm that we will be using to generate the recommendation is Bayesian Personalized Ranking, which is a matrix factorization based collaborative filtering algorithm. The reader doesn't need to be acquainted with this model per se to continue with the rest of this notebook as the discussion is model-agnostic and we'll be explaining the syntax. That said, this [link](https://github.com/ethen8181/machine-learning#recsys--20161217) contains some resources on this algorithm if it is of interest.

Given the dataset and the algorithm the preparation step we'll be doing in the next few code chunks is this:
- The raw `rating.csv` contains user's rating for each movie. Here, we will focus on implicit data, and follow the usual procedure of simulating binary implicit feedback data (i.e. whether the user enjoyed the movie) by retaining only ratings of 4 stars and higher, while dropping lower ratings.
- The raw `movie.csv` contains each movies genre tag. We will also eliminate movies that had no genre information attached and create a mapping that stores each movies' genre distribution. In this dataset, each movie $i$ typically has several genres $g$ associated with it, thus we assign equal probabilities $p(g|i)$ to each genre such that $\sum_g p(g|i) = 1$ for each movie $i$. This genre distribution will play a strong role in determining whether our recommendation is well calibrated or not.

```python
data_dir = 'movielens-20m-dataset'

# we are working with movie data, but we'll name
# the movie as item to make it more generic to
# all use-cases
user_col = 'userId'
item_col = 'movieId'
value_col = 'rating'
time_col = 'timestamp'

rating_path = os.path.join(data_dir, 'rating.csv')
df_raw = pd.read_csv(rating_path)
print('dimension: ', df_raw.shape)
df_raw.head()
```

    dimension:  (20000263, 4)

      userId
      movieId
      rating
      timestamp

      0
      1
      2
      3.5
      2005-04-02 23:53:47

      1
      1
      29
      3.5
      2005-04-02 23:31:16

      2
      1
      32
      3.5
      2005-04-02 23:33:39

      3
      1
      47
      3.5
      2005-04-02 23:32:07

      4
      1
      50
      3.5
      2005-04-02 23:29:40

```python
title_col = 'title'
genre_col = 'genres'

item_info_path = os.path.join(data_dir, 'movie.csv')
df_item = pd.read_csv(item_info_path)
df_item = df_item[df_item[genre_col] != '(no genres listed)']
print('dimension: ', df_item.shape)
df_item.head()
```

    dimension:  (27032, 3)

      movieId
      title
      genres

      0
      1
      Toy Story (1995)
      Adventure|Animation|Children|Comedy|Fantasy

      1
      2
      Jumanji (1995)
      Adventure|Children|Fantasy

      2
      3
      Grumpier Old Men (1995)
      Comedy|Romance

      3
      4
      Waiting to Exhale (1995)
      Comedy|Drama|Romance

      4
      5
      Father of the Bride Part II (1995)
      Comedy

```python
class Item:
    """
    Data holder for our item.

    Parameters
    ----------
    id : int

    title : str

    genre : dict[str, float]
        The item/movie's genre distribution, where the key
        represents the genre and value corresponds to the
        ratio of that genre.

    score : float
        Score for the item, potentially generated by some
        recommendation algorithm.
    """
    def __init__(self, _id, title, genres, score=None):
        self.id = _id
        self.title = title
        self.score = score
        self.genres = genres

    def __repr__(self):
        return self.title

def create_item_mapping(df_item, item_col, title_col, genre_col):
    """Create a dictionary of item id to Item lookup."""
    item_mapping = {}
    for row in df_item.itertuples():
        item_id = getattr(row, item_col)
        item_title = getattr(row, title_col)
        item_genre = getattr(row, genre_col)

        splitted = item_genre.split('|')
        genre_ratio = 1. / len(splitted)
        item_genre = {genre: genre_ratio for genre in splitted}

        item = Item(item_id, item_title, item_genre)
        item_mapping[item_id] = item

    return item_mapping

item_mapping = create_item_mapping(df_item, item_col, title_col, genre_col)
item_mapping[1]
```

    Toy Story (1995)

```python
# convert to implicit feedback data and filter out
# movies that doesn't have any genre
df_rating = df_raw[df_raw[value_col] >= 4.0].copy()
df_rating = df_rating.merge(df_item, on=item_col)

for col in (user_col, item_col):
    df_rating[col] = df_rating[col].astype('category')

# the original id are converted to indices to create
# the sparse matrix, so we keep track of the mappings here
# e.g. a userId 1 will correspond to index 0 in our sparse matrix
index2user = df_rating[user_col].cat.categories
index2item = df_rating[item_col].cat.categories

print('dimension: ', df_rating.shape)
df_rating.head()
```

    dimension:  (9995306, 6)

      userId
      movieId
      rating
      timestamp
      title
      genres

      0
      1
      151
      4.0
      2004-09-10 03:08:54
      Rob Roy (1995)
      Action|Drama|Romance|War

      1
      48
      151
      4.0
      1996-12-16 04:36:03
      Rob Roy (1995)
      Action|Drama|Romance|War

      2
      58
      151
      5.0
      2006-04-03 10:12:03
      Rob Roy (1995)
      Action|Drama|Romance|War

      3
      67
      151
      5.0
      1999-12-11 17:20:46
      Rob Roy (1995)
      Action|Drama|Romance|War

      4
      91
      151
      4.0
      2005-04-11 07:01:45
      Rob Roy (1995)
      Action|Drama|Romance|War

Given this dataframe we will use the `userId`, `movieId` and `rating` to construct a sparse matrix, perform the random train/test split (we can split based on the time if preferred) and feed the training set into a collaborative filtering based algorithm to train the model, so we can generate item recommendations for users.

```python
def create_user_item_csr_matrix(data, user_col, item_col, value_col):
    rows = data[user_col].cat.codes
    cols = data[item_col].cat.codes
    values = data[value_col].astype(np.float32)
    return csr_matrix((values, (rows, cols)))

user_item = create_user_item_csr_matrix(df_rating, user_col, item_col, value_col)
user_item
```

    <138287x20660 sparse matrix of type '<class 'numpy.float32'>'
    	with 9995306 stored elements in Compressed Sparse Row format>

```python
np.random.seed(1234)
user_item_train, user_item_test = train_test_split(user_item, train_percentage=0.8)
user_item_train
```

    <138287x20660 sparse matrix of type '<class 'numpy.float32'>'
    	with 7995785 stored elements in Compressed Sparse Row format>

```python
user_item_test
```

    <138287x20660 sparse matrix of type '<class 'numpy.float32'>'
    	with 1999521 stored elements in Compressed Sparse Row format>

```python
# the model expects item-user sparse matrix,
# i.e. the rows represents item and the column
# represents users
np.random.seed(1234)
bpr = BayesianPersonalizedRanking(iterations=70)
bpr.fit(user_item_train.T.tocsr())
```

    100%|██████████| 70/70 [01:26<00:00,  1.16s/it, correct=89.30%, skipped=12.78%]

we will look at a precision_at_k metric just to make sure our recommender is reasonable, feel free to tune the model's hyperparameter to squeeze out performance, but that is not the focus here.

```python
precision = precision_at_k(bpr, user_item_train, user_item_test, K=10)
precision
```

    100%|██████████| 138287/138287 [02:07<00:00, 1081.09it/s]

    0.1572725960678889

## Deep Dive Into Calibrated Recommendation

We will take the first user as an example to see whether our recommendations are calibrated or not. Once we're familiar with the procedure for one user, we can repeat the process for all of the users if we'd like to.

Let's start of by defining the problem. We are given the distribution genres $g$ for each movie $i$, $p(g|i)$, what we are interested is whether $p(g|u)$ is similar to $q(g|u)$. Where:

- $p(g|u)$ is the distribution over genre $g$ of the set of movies $H$ played by user $u$ in the past.

\begin{align}
p(g|u) = \sum_{i \in H} p(g|i)
\end{align}

- $q(g|u)$ is the distribution over genre $g$ of the set of movies $I$ we recommended to user $u$.

\begin{align}
q(g|u) = \sum_{i \in I} p(g|i)
\end{align}

For these distributions, we can have a weighted version if we liked to get sophisticated. e.g. the $p(g|i)$ can be weighted by recency saying something like the item/movie interaction matters more if its a more recent interaction, indicating that item/movie's genre should also be weighted more, but let's not go there yet.

Let's first look at some code to generate these information.

```python
# look a the first user
user_id = 0

# find the index that the user interacted with,
# we can then map this to a list of Item, note that we need to first
# map the recommended index to the actual itemId/movieId first
interacted_ids = user_item_train[user_id].nonzero()[1]
interacted_items = [item_mapping[index2item[index]] for index in interacted_ids]
interacted_items[:10]
```

    [Rob Roy (1995),
     Clerks (1994),
     Interview with the Vampire: The Vampire Chronicles (1994),
     Star Wars: Episode IV - A New Hope (1977),
     Léon: The Professional (a.k.a. The Professional) (Léon) (1994),
     Pulp Fiction (1994),
     Shawshank Redemption, The (1994),
     Platoon (1986),
     E.T. the Extra-Terrestrial (1982),
     Star Wars: Episode V - The Empire Strikes Back (1980)]

For the same user, we can use the .recommend method to recommend the topn recommendation for him/her, note that we also passed in the original sparse matrix, and by default, the items/movies that the user has already played will be filtered from the list (controlled by a `filter_already_liked_items` argument, which defaults to `True`).

```python
# it returns the recommended index and their corresponding score
topn = 20
reco = bpr.recommend(user_id, user_item_train, N=topn)
reco[:10]
```

    [(7383, 3.7959714),
     (4015, 3.5959027),
     (3992, 3.423524),
     (5509, 3.3848886),
     (3805, 3.3572443),
     (1231, 3.3039956),
     (1307, 3.265023),
     (6094, 3.1746044),
     (6825, 3.1063333),
     (1290, 3.0932918)]

```python
# map the index to Item
reco_items = [item_mapping[index2item[index]] for index, _ in reco]
reco_items[:10]
```

    [Jason and the Argonauts (1963),
     Lost Boys, The (1987),
     Evil Dead, The (1981),
     Godzilla vs. Mothra (Mosura tai Gojira) (1964),
     Hellraiser (1987),
     Evil Dead II (Dead by Dawn) (1987),
     Dracula (Bram Stoker's Dracula) (1992),
     Ringu (Ring) (1998),
     Invasion of the Body Snatchers (1978),
     American Werewolf in London, An (1981)]

The next code chunk defines a function to obtain the genre distribution for a list of items. Given that we now have the list of interacted items and recommended items, we can pass it to the function to obtain the two genre distributions.

```python
def compute_genre_distr(items):
    """Compute the genre distribution for a given list of Items."""
    distr = {}
    for item in items:
        for genre, score in item.genres.items():
            genre_score = distr.get(genre, 0.)
            distr[genre] = genre_score + score

    # we normalize the summed up probability so it sums up to 1
    # and round it to three decimal places, adding more precision
    # doesn't add much value and clutters the output
    for item, genre_score in distr.items():
        normed_genre_score = round(genre_score / len(items), 3)
        distr[item] = normed_genre_score

    return distr
```

```python
# we can check that the probability does in fact add up to 1
# np.array(list(interacted_distr.values())).sum()
interacted_distr = compute_genre_distr(interacted_items)
interacted_distr
```

    {'Action': 0.113,
     'Drama': 0.109,
     'Romance': 0.034,
     'War': 0.015,
     'Comedy': 0.073,
     'Horror': 0.093,
     'Adventure': 0.178,
     'Sci-Fi': 0.06,
     'Crime': 0.038,
     'Thriller': 0.058,
     'Children': 0.019,
     'Fantasy': 0.163,
     'Western': 0.007,
     'Musical': 0.005,
     'Animation': 0.01,
     'Mystery': 0.016,
     'IMAX': 0.008}

```python
reco_distr = compute_genre_distr(reco_items)
reco_distr
```

    {'Action': 0.133,
     'Adventure': 0.121,
     'Fantasy': 0.117,
     'Comedy': 0.058,
     'Horror': 0.271,
     'Thriller': 0.133,
     'Sci-Fi': 0.108,
     'Romance': 0.013,
     'Mystery': 0.046}

```python
# change default style figure and font size
plt.rcParams['figure.figsize'] = 10, 8
plt.rcParams['font.size'] = 12

def distr_comparison_plot(interacted_distr, reco_distr, width=0.3):

    # the value will automatically be converted to a column with the
    # column name of '0'
    interacted = pd.DataFrame.from_dict(interacted_distr, orient='index')
    reco = pd.DataFrame.from_dict(reco_distr, orient='index')
    df = interacted.join(reco, how='outer', lsuffix='_interacted')

    n = df.shape[0]
    index = np.arange(n)
    plt.barh(index, df['0_interacted'], height=width, label='interacted distr')
    plt.barh(index + width, df['0'], height=width, label='reco distr')
    plt.yticks(index, df.index)
    plt.legend(bbox_to_anchor=(1, 0.5))
    plt.title('Genre Distribution between User Historical Interaction v.s. Recommendation')
    plt.ylabel('Genre')
    plt.show()

distr_comparison_plot(interacted_distr, reco_distr)
```

### Calibration Metric

Looking at the results above, we can see that according to $p(g|u)$, the user has interacted with genres such as War, Western, however, they are nowhere to be seen in the topn recommendation to the user, hence we can argue based on the output that our recommendation might not be that well calibrated to the user's past interaction.

To scale this type of comparison, we'll now define our calibration metric $C$. There are various methods to compare whether two distributions are similar to each other, and one popular choice is KL-divergence.

\begin{align}
C(p,q) = D_{KL}(p || q) = \sum_{g} p(g|u) \cdot \log \frac{p(g|u)}{\tilde{q}(g|u)}
\end{align}

The denominator in the formula should be $q(g|u)$, but given that the formula would be undefined if $q(g|u) = 0$ and $p(g|u) > 0$ for a genre $g$. We instead use:

\begin{align}
\tilde{q}(g|u) = (1 - \alpha) \cdot q(g|u) + \alpha \cdot p(g|u)
\end{align}

with a small $\alpha$ such as 0.01, so that $q(g|u) \approx \tilde{q}(g|u)$.

```python
def compute_kl_divergence(interacted_distr, reco_distr, alpha=0.01):
    """
    KL (p || q), the lower the better.

    alpha is not really a tuning parameter, it's just there to make the
    computation more numerically stable.
    """
    kl_div = 0.
    for genre, score in interacted_distr.items():
        reco_score = reco_distr.get(genre, 0.)
        reco_score = (1 - alpha) * reco_score + alpha * score
        kl_div += score * np.log2(score / reco_score)

    return kl_div

compute_kl_divergence(interacted_distr, reco_distr)
```

    1.3345197164266038

### Generating Calibrated Recommendations

Being able to compute the calibration metric between $p(g|u)$ and $q(g|u)$ is all well and good, but how can we generate a recommendation list that is more calibrated becomes the next important and interesting question.

Different recommendation algorithm's objective function might be completely different, thus instead of going to hard-route of incorporating it into the objective function right off the bat and spend two weeks writing the customized algorithm in an efficient manner, we will start with an alternative approach of re-ranking the predicted list of a recommender system in a post-processing step.

To determine the optimal set $I^*$ of $N$ recommended items, we'll be using maximum marginal relevance.

\begin{align}
I^* = \underset{I, |I|=N}{\text{argmax}} \; (1 - \lambda) \cdot s(I) - \lambda \cdot C(p, q(I))
\end{align}

Where

- $s(i)$ is the score of the items $i \in I$ predicted by the recommender system and $s(I) = \sum_{i \in I} s(i)$, i.e. the sum of all the items' score in the recommendation list.
- $\lambda \in [0, 1]$ is a tuning parameter that determines the trade-off between the score generated by the recommender and the calibration score, notice that since the calibration score is measured by KL-divergence, which is a metric that's the lower the better we use its negative in the maximization formula.

Finding the optimal set $I^*$ is a combinatorial optimization problem and can be a topic by itself. We won't do a deep dive into it, but instead leverage a popular greedy submodular optimization to solve this problem. The process is as follows:

- We start out with the empty set.
- Iteratively append one item $i$ at a time, and at step $n$, when we already have the set $I_{n-1}$ comprised of $n - 1$ items, the item $i$ that maximizes the objective function defined above for the set $I_{n-1} \cup {i}$ is added to obtain $I_n$
- Repeat the process the generate the full $I^*$ of size $N$.

From a theoretical standpoint, this procedure guarantees a solution that has a score of 0.63 of the optimal set.

With these information at hand, let's look at the implementation part:

```python
def generate_item_candidates(model, user_item, user_id, index2item, item_mapping,
                             filter_already_liked_items=True):
    """
    For a given user, generate the list of items that we can recommend, during this
    step, we will also attach the recommender's score to each item.
    """
    n_items = user_item.shape[1]

    # this is how implicit's matrix factorization generates
    # the scores for each item for a given user, modify this
    # part of the logic if we were to use a completely different
    # algorithm to generate the ranked items
    user_factor = model.user_factors[user_id]
    scores = model.item_factors.dot(user_factor)

    liked = set()
    if filter_already_liked_items:
        liked = set(user_item[user_id].indices)

    item_ids = set(np.arange(n_items))
    item_ids -= liked

    items = []
    for item_id in item_ids:
        item = item_mapping[index2item[item_id]]
        item.score = scores[item_id]
        items.append(item)

    return items
```

```python
items = generate_item_candidates(bpr, user_item_train, user_id, index2item, item_mapping)
print('number of item candidates:', len(items))
items[:5]
```

    number of item candidates: 20588

    [Toy Story (1995),
     Jumanji (1995),
     Grumpier Old Men (1995),
     Waiting to Exhale (1995),
     Father of the Bride Part II (1995)]

```python
def compute_utility(reco_items, interacted_distr, lmbda=0.5):
    """
    Our objective function for computing the utility score for
    the list of recommended items.

    lmbda : float, 0.0 ~ 1.0, default 0.5
        Lambda term controls the score and calibration tradeoff,
        the higher the lambda the higher the resulting recommendation
        will be calibrated. Lambda is keyword in Python, so it's
        lmbda instead ^^
    """
    reco_distr = compute_genre_distr(reco_items)
    kl_div = compute_kl_divergence(interacted_distr, reco_distr)

    total_score = 0.0
    for item in reco_items:
        total_score += item.score

    # kl divergence is the lower the better, while score is
    # the higher the better so remember to negate it in the calculation
    utility = (1 - lmbda) * total_score - lmbda * kl_div
    return utility
```

```python
def calib_recommend(items, interacted_distr, topn, lmbda=0.5):
    """
    start with an empty recommendation list,
    loop over the topn cardinality, during each iteration
    update the list with the item that maximizes the utility function.
    """
    calib_reco = []
    for _ in range(topn):
        max_utility = -np.inf
        for item in items:
            if item in calib_reco:
                continue

            utility = compute_utility(calib_reco + [item], interacted_distr, lmbda)
            if utility > max_utility:
                max_utility = utility
                best_item = item

        calib_reco.append(best_item)

    return calib_reco
```

```python
start = time.time()
calib_reco_items = calib_recommend(items, interacted_distr, topn, lmbda=0.99)
elapsed = time.time() - start
print('elapsed: ', elapsed)
calib_reco_items
```

    elapsed:  18.013550996780396

    [Aelita: The Queen of Mars (Aelita) (1924),
     Rubber (2010),
     Valiant (2005),
     Hellboy (2004),
     Tai Chi Hero (2012),
     Descent, The (2005),
     Jason and the Argonauts (1963),
     Happy New Year (2014),
     Godzilla vs. Mothra (Mosura tai Gojira) (1964),
     Picture of Dorian Gray, The (1945),
     Lost Boys, The (1987),
     Beastmaster, The (1982),
     Enemy Mine (1985),
     Hound of the Baskervilles, The (1959),
     Highlander (1986),
     Dracula (Bram Stoker's Dracula) (1992),
     Last Starfighter, The (1984),
     NeverEnding Story, The (1984),
     Dune (2000),
     Evil Dead II (Dead by Dawn) (1987)]

In the code chunk above, we turned the $\lambda$ knob extremely high to generate the most calibrated recommendation list possible. Let's now compare the calibrated recommendation (which only optimizes for score, $s$), the original recommendation and the user's interaction distribution.

```python
calib_reco_distr = compute_genre_distr(calib_reco_items)
calib_reco_kl_div = compute_kl_divergence(interacted_distr, calib_reco_distr)
reco_kl_div = compute_kl_divergence(interacted_distr, reco_distr)
print('\noriginal reco kl-divergence score:', reco_kl_div)
print('calibrated reco kl-divergence score:', calib_reco_kl_div)

distr_comparison_plot(interacted_distr, calib_reco_distr)
```

Printing out the genre distribution from the calibrated recommendation list shows that this list covers more genre and its distribution closely resembles the distribution of the user's past historical interaction and our quantitative calibration metric, KL-divergence also confirms this. i.e. the calibrated recommendation's KL-divergence is lower than the original recommendation's score.

Thankfully from the results above, it seems that the re-ranked recommendation list that aims to maximize calibration score does in fact generate a more calibrated list. But the question is at what cost? Does other ranking metrics that recommender system often optimize for drop? Let's take a look at precision_at_k. Here the number for `k` is the `topn` parameter that we've defined earlier. i.e. the number of recommendations to generate for the user.

```python
def precision(user_item, user_id, reco_items, index2item):
    indptr = user_item.indptr
    indices = user_item.indices

    reco_ids = {item.id for item in reco_items}
    likes = {index2item[indices[i]] for i in range(indptr[user_id], indptr[user_id + 1])}

    relevant = len(reco_ids & likes)
    total = min(len(reco_items), len(likes))
    return relevant / total
```

```python
reco_precision = precision(user_item_test, user_id, reco_items, index2item)
calib_reco_precision = precision(user_item_test, user_id, calib_reco_items, index2item)
print('original reco precision score:', reco_precision)
print('calibrated reco precision score:', calib_reco_precision)
```

    original reco precision score: 0.1875
    calibrated reco precision score: 0.125

Well ..., it's not a surprise that the calibrated recommendation list's precision score is a bit disappointing compared to the original recommendation. But let's see if we try a different value of $\lambda$, this time turning it down a bit to strike a balance between calibration and precision.

```python
start = time.time()
calib_reco_items = calib_recommend(items, interacted_distr, topn, lmbda=0.5)
elapsed = time.time() - start
print('elapsed: ', elapsed)
calib_reco_items
```

    elapsed:  15.620396137237549

    [Jason and the Argonauts (1963),
     Lost Boys, The (1987),
     Dune (2000),
     Godzilla vs. Mothra (Mosura tai Gojira) (1964),
     Dracula (Bram Stoker's Dracula) (1992),
     Evil Dead, The (1981),
     Evil Dead II (Dead by Dawn) (1987),
     Hellraiser (1987),
     Secret of NIMH, The (1982),
     Ringu (Ring) (1998),
     Beastmaster, The (1982),
     Highlander (1986),
     Flash Gordon (1980),
     American Werewolf in London, An (1981),
     Invasion of the Body Snatchers (1978),
     Road Warrior, The (Mad Max 2) (1981),
     Krull (1983),
     Phenomena (a.k.a. Creepers) (1985),
     Last Starfighter, The (1984),
     Serpent and the Rainbow, The (1988)]

```python
calib_reco_distr = compute_genre_distr(calib_reco_items)
calib_reco_kl_div = compute_kl_divergence(interacted_distr, calib_reco_distr)
calib_reco_precision = precision(user_item_test, user_id, calib_reco_items, index2item)
print('calibrated reco kl-divergence score:', calib_reco_kl_div)
print('calibrated reco precision score:', calib_reco_precision)
```

    calibrated reco kl-divergence score: 0.6256542329089019
    calibrated reco precision score: 0.1875

```python
calib_reco_distr = compute_genre_distr(calib_reco_items)
distr_comparison_plot(interacted_distr, calib_reco_distr)
```

Well, well, well. It turns out calibration can be improved considerably while accuracy is reduced only slightly if we find the sweet spot for the tuning parameter $\lambda$.

The following code chunk curates all the code to generate the calibrated recommendation, the original recommendation and compare it with the user's historical interaction in one place for ease of tracking the flow. This process is outlined for 1 user, feel free to modify the code to perform this comparison across all users and due to the randomness in the recommendation algorithm, the results might differ across runs, but the underlying trend should remain the same.

```python
topn = 20
user_id = 0
lmbda = 0.99

reco = bpr.recommend(user_id, user_item_train, N=topn)
reco_items = [item_mapping[index2item[index]] for index, _ in reco]
reco_distr = compute_genre_distr(reco_items)

interacted_ids = user_item_train[user_id].nonzero()[1]
interacted_items = [item_mapping[index2item[index]] for index in interacted_ids]
interacted_distr = compute_genre_distr(interacted_items)

items = generate_item_candidates(bpr, user_item_train, user_id, index2item, item_mapping)
calib_reco_items = calib_recommend(items, interacted_distr, topn, lmbda)
calib_reco_distr = compute_genre_distr(calib_reco_items)

calib_reco_kl_div = compute_kl_divergence(interacted_distr, calib_reco_distr)
calib_reco_precision = precision(user_item_test, user_id, calib_reco_items, index2item)
print('calibrated reco kl-divergence score:', calib_reco_kl_div)
print('calibrated reco precision score:', calib_reco_precision)
distr_comparison_plot(interacted_distr, calib_reco_distr)

reco_kl_div = compute_kl_divergence(interacted_distr, reco_distr)
reco_precision = precision(user_item_test, user_id, reco_items, index2item)
print('original reco kl-divergence score:', reco_kl_div)
print('original reco precision score:', reco_precision)
distr_comparison_plot(interacted_distr, reco_distr)
```

## End Note

We have calibrated our recommendation here based on movies' genre but the same idea can be applied if we wish to calibrate our recommendation based on some other traits that matters for the problem at hand.

The approach we took here is from a user-centric perspective, i.e. we are generating calibrated recommendations for each user, we can also tackle the problem from a item-centric perspective to see if items are recommended properly, e.g. if popular items just gets recommended a lot more times than other items.

Although we used Bayesian Personalized Ranking to generated recommendations for each user, the technique that we used here to generated the calibrated ranked items for each user is independent of the recommendation algorithm we use. We're only using the algorithm here to demonstrate the process, so feel free is try this out on a recommendation algorithm of your liking/interest.

The discussion here focused on generating calibrated recommendation anchored on a user's past interaction. So if a user played 70% romance movies and 30% action movies, then when we say our recommendation is well-calibrated, that means the top ranked movies for the user should also consists of 70% romance movies and 30% action movies. We can, however, expand this work to recommend diversified items/movies. In other words, instead of limiting ourselves to only work with romance and action movies for a user that has only played movies falling under these two genres, we recommend movies from genres that are outside the user's historical play list.

To elaborate, we were using $p(g|u)$ to represent the user's historical genre interaction, we can extend that distribution to $\tilde{p}(g|u)$, where:

\begin{align}
\tilde{p}(g|u) = \beta \cdot p_0(g) + (1 - \beta) \cdot p(g|u)
\end{align}

Here $p_0(g)$ denote a prior distribution that takes positive values for all the genres $g$ in which we would like to promote diversity in the recommendation. Again we also have a tuning parameter $\beta \in [0, 1]$ to control how "diversified" we would like our recommendations to be. As to how we decide which genres to promote diversity for? Well, that's a topic for another day, but to start with we can use a uniform distribution across all genres or the average over all the users' genre distribution.

After reading a lot of tutorials/papers that introduces new algorithms or new libraries/packages that boosts model performance in terms of ranking metric, coming across a paper on a different topic, calibrating recommendation, is a breath of fresh air. This sheds a new light on how I think about serving recommendations, so kudos to the original authors who brought this well-written paper to light. I encourage people to read the original paper if interested. [Paper: H. Steck - Calibrated Recommendation (2018)](https://dl.acm.org/citation.cfm?id=3240372)

**Side Note:**

- In the generating calibrated recommendation section, we introduced the maximum marginal relevance formula that has the knob for us to tune the balance between our original recommendation algorithm's score versus the calibration score. This "balancing" notion can actually be applied to many different areas. For example, In the original paper that introduced this idea, they were using this formula in the context of a search engine. To elaborate, when issuing a query on a search engine, the search engine often times assembles results that maximizes the relevance to the user query. By introducing the linear combination "marginal relevance" - a result is now said to have high marginal relevance if it is both relevant to the query and contains minimal similarity to previously selected results.
- The Greedy Algorithm we leveraged to actually generate our top-N items that maximizes the maximum marginal relevance formula is a discrete optimization method that's often times referred to as a submodular optimization. Submodular optimization is a generic optimization method that can also be applied to other areas such as influence maximization and can be a topic by itself and is discussed lightly deeper in another notebook. [Notebook: Submodular Optimization & Influence Maximization](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/networkx/max_influence/max_influence.ipynb)

# Reference

- [Paper: H. Steck - Calibrated Recommendation (2018)](https://dl.acm.org/citation.cfm?id=3240372)
- [Paper: J. Carbonell, J. Goldstein - The use of MMR, diversity-based reranking for
reordering documents and producing summaries (1998)](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.188.3982&rep=rep1&type=pdf)


# GSSR
Gene Splice Site Recognition by WAM, Bayesian Network and SVM approaches

## Abstract
xxx

## Introduction

### Gene splice site

https://en.wikipedia.org/wiki/RNA_splicing

RNA splicing, in molecular biology, is a form of RNA processing in which a newly made precursor messenger RNA (pre-mRNA) transcript is transformed into a mature messenger RNA (mRNA). During splicing, introns (Non-coding regions) are removed and exons (Coding Regions) are joined together. For nuclear-encoded genes, splicing takes place within the nucleus either during or immediately after transcription. For those eukaryotic genes that contain introns, splicing is usually required in order to create an mRNA molecule that can be translated into protein. For many eukaryotic introns, splicing is carried out in a series of reactions which are catalyzed by the spliceosome, a complex of small nuclear ribonucleo proteins (snRNPs). Self-splicing introns, or ribozymes capable of catalyzing their own excision from their parent RNA molecule, also exist.

### Models

https://academic.oup.com/bioinformatics/article-abstract/9/5/499/349215?redirectedFrom=PDF

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-477

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-S10-S7

and others

### Scikit-learn and Pgmpy

introductions of Scipy and Pgmpy

## Material and Method

### Environment

```python
import warnings
import import_ipynb
import numpy as np
import pandas as pd
import networkx as nx
from Utils import load_data
from Models import WAM, BN, SVM
from Evaluator import Evaluator
from matplotlib import pyplot as plt
from IPython.display import display, Markdown

warnings.filterwarnings("ignore")
training_set, testing_set = load_data()

# https://www.it-swarm.dev/zh/python/%E5%A6%82%E4%BD%95%E4%BB%8E%E7%BB%88%E7%AB%AF%E8%BF%90%E8%A1%8Cipynb-jupyter%E7%AC%94%E8%AE%B0%E6%9C%AC%EF%BC%9F/823861554/

n = 2500
training_set = pd.concat( (training_set[training_set['IsDonor']==0].sample(n=n),
                           training_set[training_set['IsDonor']==1].sample(n=n)), axis=0)
testing_set = pd.concat( (testing_set[testing_set['IsDonor']==0].sample(n=n),
                          testing_set[testing_set['IsDonor']==1].sample(n=n)), axis=0)
```

    importing Jupyter notebook from Utils.ipynb
    importing Jupyter notebook from Models.ipynb
    importing Jupyter notebook from Evaluator.ipynb

### Data curation

### Encoding for features

When encoding the sequence features to be numeric, a One-hot encoder were used. And to make the problem simpler, the ambiguous bases in sequences were disgarded. A simple example is that when deeling with sequence A, which can be represented as $acgtay^* aaccggttr^* y^* $, the ambiguous bases $y^*$ and $r^*$ are simply disgarded. So the sequence A becomes $acgtaaaccggtt$. And next, a One-hot encoder are used to encode sequence A to a numeric feature vector, which can be represented as $[1,0,0,0...0,0,0,1]$. In all of my experiments, I used 10 bases before and after the site to be tested, including its own sequence with a total length of 21.

### Model implements

The 3 models are mainly implemented using Python. Besides, the Bayesian network model and Support vector machine were implemented based on Pgmpy and Scikit-learn pacakge. The 3 models are designed to be *[put into practical use]* as much as possible. They contain several very easy-to-use program interfaces: `fit`, `predict_probas`, `predict_scores` and `predict_classes`. For all 3 models, the `fit` method are used for training, and the other 3 `predict` methods are used for prediction. The `predict_probas` are used for predicting the probabilities each sample belonging to each class, the `predict_scores` are used for calculating the $S(X)$, and the `predict_classes` are used for predicting the classes each sample belongs, this is simply done by comparing the $S(X)$ and given threshold $T$. Most importantly, the $S(X)$ are calculated by:
$$\tag{1}S(X) = ln(\frac{P^+ (X)}{P^- {X}})$$

#### Stratagies for WAM to perform recognition

When recoginizing Gene Splice Site, the 3 steps were mainly used to determine the potential of each sample being a splice site. Firstly, Laplace Pseudocounts are used to [...]. Secondly, the weights of positive and negative submodels, which are the frequencies of adjacent base pairs in WAM, are calculated using the following formula.
$$\tag{2} f(x) = \frac{C_{x,p}+C_p}{N},x \in [aa,ac,ag,...tt]$$ where $N$ is the total number of samples. $C_{x,p}$ is the total number of specific adjacent base pairs occuring at position $p-l+1...p$ in all samples. $C_p$ is the pseudocount, and $l$ is the length of array considered, which refers to the range of dependence between adjacent base pairs considered in experiments. And finally, the potential of new samples being a donor site can be easily calculated by
$$\tag{3} S(X) = ln(\frac{P^+(X)}{P^-(X)}) = \sum_{x=2}^{L}{ln(\frac{f^+(x)}{f^-(x)})}$$
where $L$ is the length of input sequence.
Most importantly, $l$ and $C_p$ are set to $2$ and $0.5$ respectively.

#### Stratagies for BN to learn structures and parameters from data

The min max hill climb (MMHC) algorithm *[REF]* were mainly used to learn BN structures from the training data. MMHC algorithm is a constrained-based learning algorithm, firstly be proposed by *[AUTHOR]*. *[ details of MMHC algorithm]*

#### Settings for SVM to classify splice sites

In the experiment, Gaussian Kernel function (see below) were used to map vectors to higher-dimensional feature space. The 惩罚系数 $C$ are used to .... And the key parameter $\gamma$ for Gaussian Kernel are set to [real gamma].
$$\tag{4}K(x_i, x_j) = exp(-\gamma||x_i - x_j||^2), \gamma > 0$$

```python
class Model:

    def __init__(self):
        self.wam = WAM()
        self.bn = BN()
        self.svm = SVM(kernel='rbf', gamma='scale', random_state=1, probability=True, verbose=True)

model = Model()
```

### Evaluation

To visualize and compare classification results of 3 models, the ROC Curves and Sn-Sp Curves are plotted using matplotlib package. To realize this, a sample-based benchmarking method is introduced.

$$
\begin{align}
TP_t = \sum_{i\in S_{TP,t}} 1, \quad S_{TP,t}& =\{i\mid S_i\geq t \land L_i = 1\} \tag{} \\
FP_t = \sum_{i\in S_{FP,t}} 1, \quad S_{FP,t}& =\{i\mid S_i\geq t \land L_i = 0\} \tag{} \\
TN_t = \sum_{i\in S_{TN,t}} 1, \quad S_{TN,t}& =\{i\mid S_i< t \land L_i = 0\} \tag{} \\
FN_t = \sum_{i\in S_{FN,t}} 1, \quad S_{FN,t}& =\{i\mid S_i< t \land L_i = 1\} \tag{} \\
\end{align}
$$
$$
\begin{align}
TPR_t & = \frac{TP_t}{TP_t+FN_t} \tag{} \\
FPR_t & = \frac{FP_t}{FP_t+TN_t} \tag{} \\
Sn_t & = \frac{TP_t}{TP_t+FN_t} \tag{} \\
Sp_t & = \frac{TN_t}{TN_t+FP_t} \tag{} \\
\end{align}
$$

where $ $ is ...... Finally, the ROC_AUC ( area under the ROC curve) were calculated using Composite Trapezoidal rule [ref]

$$
AUC = \sum_{i=1}^N \frac{(FPR_{i+1} - FPR_i)(TPR_i+TPR_{i+1})}{2} \tag{} \\
$$

https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/metrics/_ranking.py#L42

https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.trapz.html

https://github.com/numpy/numpy/blob/v1.16.2/numpy/lib/function_base.py#L3982-L4070

## Result and Discussion

### Weights for WAM

```python
#display(training_set)
model.wam.fit(training_set['Seq. features'], training_set['IsDonor'])
pd.set_option('display.max_columns', None)
p_rename = lambda x: 'P.{}'.format(x)
n_rename = lambda x: 'N.{}'.format(x)
```

```python
display(Markdown( '<h3 align="center">Weights for Weighted array model (Positive / Negative)</h3>' ))
model.wam.positive_weights.applymap(lambda x: '%.3f'%x) + '/' + \
model.wam.negative_weights.applymap(lambda x: '%.3f'%x)
```

Weights for Weighted array model (Positive / Negative)

      AA
      AC
      AG
      AT
      CA
      CC
      CG
      CT
      GA
      GC
      GG
      GT
      TA
      TC
      TG
      TT

      0
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan

      1
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan
      nan/nan

      2
      0.055/0.070
      0.049/0.050
      0.031/0.051
      0.063/0.054
      0.086/0.045
      0.117/0.069
      0.056/0.047
      0.059/0.045
      0.068/0.052
      0.069/0.053
      0.046/0.066
      0.036/0.061
      0.034/0.048
      0.067/0.056
      0.043/0.047
      0.076/0.078

      3
      0.095/0.047
      0.076/0.053
      0.037/0.058
      0.074/0.052
      0.067/0.061
      0.070/0.056
      0.072/0.060
      0.075/0.057
      0.058/0.052
      0.072/0.053
      0.027/0.055
      0.020/0.053
      0.057/0.063
      0.067/0.064
      0.044/0.062
      0.062/0.040

      4
      0.035/0.052
      0.137/0.057
      0.051/0.053
      0.078/0.065
      0.063/0.051
      0.087/0.062
      0.066/0.060
      0.082/0.055
      0.049/0.051
      0.071/0.062
      0.034/0.065
      0.020/0.053
      0.032/0.053
      0.066/0.052
      0.045/0.050
      0.056/0.055

      5
      0.043/0.048
      0.066/0.054
      0.033/0.053
      0.058/0.050
      0.072/0.061
      0.088/0.055
      0.078/0.061
      0.113/0.057
      0.059/0.055
      0.060/0.063
      0.040/0.058
      0.031/0.051
      0.029/0.054
      0.074/0.053
      0.060/0.054
      0.049/0.062

      6
      0.063/0.058
      0.056/0.062
      0.033/0.048
      0.079/0.055
      0.090/0.057
      0.061/0.057
      0.062/0.063
      0.068/0.059
      0.055/0.055
      0.078/0.051
      0.045/0.051
      0.021/0.059
      0.031/0.057
      0.066/0.060
      0.042/0.058
      0.142/0.047

      7
      0.041/0.058
      0.121/0.052
      0.044/0.054
      0.060/0.061
      0.079/0.053
      0.086/0.058
      0.050/0.055
      0.051/0.061
      0.049/0.058
      0.074/0.054
      0.029/0.047
      0.020/0.059
      0.038/0.054
      0.126/0.050
      0.055/0.056
      0.042/0.057

      8
      0.049/0.057
      0.089/0.060
      0.030/0.053
      0.078/0.055
      0.102/0.067
      0.064/0.052
      0.038/0.037
      0.135/0.048
      0.054/0.058
      0.084/0.051
      0.020/0.045
      0.041/0.056
      0.019/0.066
      0.068/0.058
      0.026/0.059
      0.062/0.055

      9
      0.119/0.051
      0.031/0.057
      0.024/0.074
      0.053/0.062
      0.168/0.058
      0.035/0.051
      0.048/0.045
      0.035/0.059
      0.073/0.051
      0.025/0.051
      0.009/0.063
      0.008/0.040
      0.135/0.048
      0.027/0.058
      0.078/0.054
      0.038/0.052

      10
      0.086/0.051
      0.008/0.053
      0.220/0.050
      0.041/0.055
      0.014/0.050
      0.005/0.054
      0.154/0.063
      0.016/0.059
      0.078/0.060
      0.002/0.056
      0.052/0.060
      0.005/0.069
      0.003/0.054
      0.005/0.054
      0.059/0.054
      0.006/0.055

      11
      0.024/0.054
      0.026/0.051
      0.076/0.055
      0.033/0.059
      0.000/0.057
      0.000/0.047
      0.050/0.058
      0.000/0.056
      0.012/0.061
      0.008/0.049
      0.423/0.052
      0.003/0.059
      0.000/0.056
      0.000/0.060
      0.042/0.063
      0.000/0.050

      12
      0.006/0.055
      0.018/0.063
      0.010/0.055
      0.019/0.053
      0.006/0.052
      0.010/0.055
      0.004/0.055
      0.011/0.050
      0.014/0.053
      0.035/0.063
      0.013/0.055
      0.473/0.057
      0.005/0.058
      0.003/0.054
      0.007/0.056
      0.013/0.057

      13
      0.007/0.057
      0.015/0.057
      0.006/0.052
      0.017/0.052
      0.009/0.056
      0.022/0.062
      0.016/0.064
      0.022/0.057
      0.011/0.053
      0.015/0.062
      0.008/0.055
      0.007/0.054
      0.403/0.057
      0.044/0.050
      0.219/0.056
      0.058/0.051

      14
      0.238/0.058
      0.060/0.050
      0.030/0.057
      0.083/0.056
      0.024/0.061
      0.021/0.053
      0.018/0.053
      0.023/0.061
      0.208/0.059
      0.044/0.055
      0.025/0.053
      0.012/0.051
      0.020/0.050
      0.027/0.048
      0.017/0.057
      0.014/0.061

      15
      0.012/0.065
      0.029/0.059
      0.265/0.053
      0.042/0.051
      0.026/0.053
      0.054/0.045
      0.092/0.065
      0.031/0.054
      0.012/0.060
      0.015/0.054
      0.045/0.057
      0.008/0.053
      0.009/0.059
      0.037/0.054
      0.044/0.061
      0.019/0.053

      16
      0.014/0.062
      0.027/0.056
      0.016/0.060
      0.032/0.058
      0.025/0.054
      0.029/0.055
      0.019/0.055
      0.038/0.052
      0.076/0.056
      0.129/0.052
      0.069/0.060
      0.220/0.057
      0.011/0.053
      0.026/0.055
      0.019/0.057
      0.034/0.049

      17
      0.035/0.050
      0.044/0.057
      0.029/0.059
      0.036/0.053
      0.055/0.056
      0.065/0.051
      0.057/0.049
      0.061/0.057
      0.035/0.050
      0.055/0.059
      0.017/0.063
      0.018/0.064
      0.106/0.049
      0.114/0.055
      0.104/0.050
      0.154/0.063

      18
      0.042/0.047
      0.086/0.052
      0.033/0.056
      0.097/0.056
      0.043/0.054
      0.087/0.051
      0.044/0.054
      0.086/0.058
      0.051/0.060
      0.082/0.061
      0.062/0.047
      0.036/0.057
      0.030/0.068
      0.077/0.061
      0.040/0.050
      0.069/0.056

      19
      0.036/0.059
      0.062/0.058
      0.032/0.059
      0.057/0.053
      0.071/0.054
      0.099/0.066
      0.074/0.057
      0.092/0.054
      0.036/0.056
      0.082/0.053
      0.037/0.048
      0.022/0.046
      0.039/0.053
      0.106/0.060
      0.055/0.056
      0.104/0.058

```python
scores_wam = model.wam.predict_scores(testing_set['Seq. features'].apply(str.upper))
```

### Graph for BN

```python
model.bn.fit(data=to_format_arr(training_set['Seq. features']),
             label=training_set['IsDonor'].astype(int),
             sign_level=0.01)
```

    Performing structure learning, using estimator: `<class 'pgmpy.estimators.MmhcEstimator.MmhcEstimator'>`
    Done !
    Performing parameter learning, using estimator: `<class 'pgmpy.estimators.BayesianEstimator.BayesianEstimator'>`

```python
plt.figure(figsize=(4,4), dpi=120)
G = model.bn.networks
bicolors = ['deepskyblue','salmon']
nodesize = np.array([float(G.degree(v) + 1) for v in G])
nodecolor = np.array([bicolors[int('label' in i)] for i in G])
nx.draw(G, with_labels=True, textsize=10000, pos=nx.spring_layout(G), node_size=nodesize*150,
        node_color=nodecolor)
```

```python
data = to_format_arr(testing_set['Seq. features'])
test_data = pd.DataFrame(data, columns=['F{}'.format(i) for i in range(data.shape[1])])
scores_bn = model.bn.predict_scores(test_data)
```

### Graph for SVM

```python
model.svm.fit(to_format_arr(training_set['Num. features']), training_set['IsDonor'])
scores_svm = model.svm.predict_scores(to_format_arr(testing_set['Num. features']))
```

    [LibSVM]

```python
arr = to_format_arr(training_set['Num. features'])
print(arr.shape)
arr
```

    (5000, 80)

    array([[0, 0, 0, ..., 1, 0, 0],
           [0, 1, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 1, 0],
           ...,
           [0, 0, 1, ..., 1, 0, 0],
           [0, 0, 1, ..., 0, 0, 1],
           [0, 0, 1, ..., 1, 0, 0]])

### Performance
- Table
- ROC Sn-Sp Curves

```python
evaltr = Evaluator(labels=testing_set['IsDonor'],
                  wam_scores=scores_wam,
                  bn_scores=scores_bn,
                  svm_scores=scores_svm)

display(evaltr.Cal_Sn_Sp(evaltr.Confusion_table(T_range=np.arange(0, 10, 0.1))['wam_scores']).round(2).T)
display(evaltr.Cal_Tpr_Fpr(evaltr.Confusion_table(T_range=np.arange(0, 10, 0.1))['wam_scores']).round(2).T)
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      65
      66
      67
      68
      69
      70
      71
      72
      73
      74
      75
      76
      77
      78
      79
      80
      81
      82
      83
      84
      85
      86
      87
      88
      89
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      Threshold
      0.00
      0.10
      0.20
      0.30
      0.40
      0.50
      0.60
      0.70
      0.80
      0.90
      1.00
      1.10
      1.20
      1.30
      1.40
      1.50
      1.60
      1.70
      1.80
      1.90
      2.00
      2.10
      2.20
      2.30
      2.40
      2.50
      2.60
      2.70
      2.80
      2.90
      3.00
      3.10
      3.20
      3.30
      3.40
      3.50
      3.60
      3.70
      3.80
      3.90
      4.00
      4.10
      4.20
      4.30
      4.40
      4.50
      4.60
      4.70
      4.80
      4.90
      5.00
      5.10
      5.20
      5.30
      5.40
      5.50
      5.60
      5.70
      5.8
      5.9
      6.0
      6.10
      6.20
      6.30
      6.40
      6.50
      6.60
      6.70
      6.80
      6.90
      7.00
      7.10
      7.20
      7.30
      7.40
      7.5
      7.60
      7.70
      7.80
      7.90
      8.00
      8.10
      8.20
      8.30
      8.40
      8.50
      8.6
      8.70
      8.80
      8.90
      9.00
      9.10
      9.20
      9.30
      9.40
      9.50
      9.6
      9.70
      9.80
      9.90

      Sensitivity
      0.85
      0.85
      0.85
      0.85
      0.84
      0.84
      0.84
      0.84
      0.84
      0.84
      0.84
      0.83
      0.83
      0.83
      0.83
      0.83
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.80
      0.80
      0.80
      0.80
      0.79
      0.79
      0.79
      0.79
      0.79
      0.78
      0.77
      0.77
      0.77
      0.77
      0.76
      0.76
      0.75
      0.75
      0.75
      0.74
      0.73
      0.73
      0.72
      0.72
      0.71
      0.71
      0.7
      0.7
      0.7
      0.69
      0.69
      0.68
      0.68
      0.67
      0.67
      0.66
      0.65
      0.65
      0.64
      0.64
      0.62
      0.62
      0.61
      0.6
      0.59
      0.59
      0.58
      0.57
      0.56
      0.55
      0.54
      0.53
      0.52
      0.51
      0.5
      0.49
      0.48
      0.47
      0.47
      0.46
      0.45
      0.44
      0.43
      0.41
      0.4
      0.39
      0.38
      0.36

      Specificity
      0.94
      0.94
      0.95
      0.95
      0.95
      0.95
      0.95
      0.95
      0.96
      0.96
      0.96
      0.96
      0.96
      0.96
      0.96
      0.97
      0.97
      0.97
      0.97
      0.97
      0.97
      0.97
      0.97
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.98
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      0.99
      1.00
      1.00
      1.00
      1.00
      1.00
      1.0
      1.0
      1.0
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.0
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.0
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00
      1.0
      1.00
      1.00
      1.00

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      65
      66
      67
      68
      69
      70
      71
      72
      73
      74
      75
      76
      77
      78
      79
      80
      81
      82
      83
      84
      85
      86
      87
      88
      89
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      Threshold
      0.00
      0.10
      0.20
      0.30
      0.40
      0.50
      0.60
      0.70
      0.80
      0.90
      1.00
      1.10
      1.20
      1.30
      1.40
      1.50
      1.60
      1.70
      1.80
      1.90
      2.00
      2.10
      2.20
      2.30
      2.40
      2.50
      2.60
      2.70
      2.80
      2.90
      3.00
      3.10
      3.20
      3.30
      3.40
      3.50
      3.60
      3.70
      3.80
      3.90
      4.00
      4.10
      4.20
      4.30
      4.40
      4.50
      4.60
      4.70
      4.80
      4.90
      5.00
      5.10
      5.20
      5.30
      5.40
      5.50
      5.60
      5.70
      5.8
      5.9
      6.0
      6.10
      6.20
      6.30
      6.40
      6.50
      6.60
      6.70
      6.80
      6.90
      7.00
      7.10
      7.20
      7.30
      7.40
      7.5
      7.60
      7.70
      7.80
      7.90
      8.00
      8.10
      8.20
      8.30
      8.40
      8.50
      8.6
      8.70
      8.80
      8.90
      9.00
      9.10
      9.20
      9.30
      9.40
      9.50
      9.6
      9.70
      9.80
      9.90

      Tpr
      0.85
      0.85
      0.85
      0.85
      0.84
      0.84
      0.84
      0.84
      0.84
      0.84
      0.84
      0.83
      0.83
      0.83
      0.83
      0.83
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.82
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.81
      0.80
      0.80
      0.80
      0.80
      0.79
      0.79
      0.79
      0.79
      0.79
      0.78
      0.77
      0.77
      0.77
      0.77
      0.76
      0.76
      0.75
      0.75
      0.75
      0.74
      0.73
      0.73
      0.72
      0.72
      0.71
      0.71
      0.7
      0.7
      0.7
      0.69
      0.69
      0.68
      0.68
      0.67
      0.67
      0.66
      0.65
      0.65
      0.64
      0.64
      0.62
      0.62
      0.61
      0.6
      0.59
      0.59
      0.58
      0.57
      0.56
      0.55
      0.54
      0.53
      0.52
      0.51
      0.5
      0.49
      0.48
      0.47
      0.47
      0.46
      0.45
      0.44
      0.43
      0.41
      0.4
      0.39
      0.38
      0.36

      Fpr
      0.06
      0.06
      0.05
      0.05
      0.05
      0.05
      0.05
      0.05
      0.04
      0.04
      0.04
      0.04
      0.04
      0.04
      0.04
      0.03
      0.03
      0.03
      0.03
      0.03
      0.03
      0.03
      0.03
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.02
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.00
      0.00
      0.00
      0.00
      0.00
      0.0
      0.0
      0.0
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.0
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.0
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.0
      0.00
      0.00
      0.00

```python
plt.figure(figsize=(10,5), dpi=120)
ax1 = plt.subplot(121)
evaltr.ROC_Curves(T_range=np.arange(-25, 25, 0.5))
ax1 = plt.subplot(122)
evaltr.Sn_Sp_Curves(T_range=np.arange(-25, 25, 0.5))
```

### Discussion

reintroduce the problem

describe results and its importance

future research

```python

```


```python
!pip install -r ../requirements.txt
```

    Requirement already satisfied: numpy in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 1)) (1.17.3)
    Requirement already satisfied: scipy in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 2)) (1.1.0)
    Requirement already satisfied: matplotlib in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 3)) (2.1.1)
    Requirement already satisfied: Pillow in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 4)) (4.3.0)
    Requirement already satisfied: ipywidgets>=0.6.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 5)) (7.5.1)
    Requirement already satisfied: requests in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from -r ../requirements.txt (line 6)) (2.14.2)
    Collecting SimPEG
      Using cached https://files.pythonhosted.org/packages/c4/05/6480fd00c7e6809f855134f9a47dd0cc61a2ed5538c367372290cdc6dcaf/SimPEG-0.12.0.tar.gz
    Collecting em_examples
      Using cached https://files.pythonhosted.org/packages/e8/39/1a285f68abfdbacc00deefbcd7d2053b6dd67d9ab384a4cbc70a523ae84c/em_examples-0.0.35-py3-none-any.whl
    Collecting empymod
      Using cached https://files.pythonhosted.org/packages/ad/01/b257911dbc987a440be8972384798a368b41be870526959eebc48142f49c/empymod-1.10.3-py3-none-any.whl
    Collecting deepdish
      Using cached https://files.pythonhosted.org/packages/6e/39/2a47c852651982bc5eb39212ac110284dd20126bdc7b49bde401a0139f5d/deepdish-0.3.6-py2.py3-none-any.whl
    Collecting pymatsolver>=0.1.2
      Using cached https://files.pythonhosted.org/packages/0f/5e/30e92dfe738cc3e28f52344484aa2c6e44872787b2edd789df0eaae04e3b/pymatsolver-0.1.2.tar.gz
    Requirement already satisfied: six>=1.10 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from matplotlib->-r ../requirements.txt (line 3)) (1.11.0)
    Requirement already satisfied: python-dateutil>=2.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from matplotlib->-r ../requirements.txt (line 3)) (2.8.1)
    Requirement already satisfied: pytz in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from matplotlib->-r ../requirements.txt (line 3)) (2016.6.1)
    Requirement already satisfied: cycler>=0.10 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from matplotlib->-r ../requirements.txt (line 3)) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from matplotlib->-r ../requirements.txt (line 3)) (2.1.4)
    Requirement already satisfied: olefile in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from Pillow->-r ../requirements.txt (line 4)) (0.44)
    Requirement already satisfied: nbformat>=4.2.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.2.0)
    Requirement already satisfied: ipykernel>=4.5.1 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.7.0)
    Requirement already satisfied: traitlets>=4.3.1 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.3.1)
    Requirement already satisfied: ipython>=4.0.0; python_version >= "3.3" in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (6.2.1)
    Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (3.5.1)
    Requirement already satisfied: cython in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from SimPEG->-r ../requirements.txt (line 7)) (0.24.1)
    Collecting properties>=0.5.2
      Using cached https://files.pythonhosted.org/packages/c0/9a/cd2def3421cd077d80d4973c802d29dcb3fba625190b3678480f228a85bf/properties-0.6.1.tar.gz
    Collecting vectormath>=0.2.0
      Using cached https://files.pythonhosted.org/packages/be/d3/5d49b6e961d87ba4845205509fbd0f2fa39875859ccb5e4778035518b89f/vectormath-0.2.2.tar.gz
    Collecting discretize>=0.4.0
      Using cached https://files.pythonhosted.org/packages/b1/41/4766aa8a1f6db9357f6941e6ffb1e6b572b9cfe9467d2d81d7dd431f20cd/discretize-0.4.10.tar.gz
    Collecting geoana>=0.0.4
      Using cached https://files.pythonhosted.org/packages/60/34/5715c1b3e35b4b7e0fdef463405c9ae5b073563f48118f2a54165be78cdc/geoana-0.0.5-py2.py3-none-any.whl
    Requirement already satisfied: jupyter in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from em_examples->-r ../requirements.txt (line 8)) (1.0.0)
    Requirement already satisfied: future in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from em_examples->-r ../requirements.txt (line 8)) (0.18.2)
    Requirement already satisfied: tables in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from deepdish->-r ../requirements.txt (line 10)) (3.4.2)
    Collecting pyMKL>=0.0.3
      Using cached https://files.pythonhosted.org/packages/43/9d/0657fdb64d31a83c5caab80a70e55dac7658ba3fac7d1f43cbcca726b286/pyMKL-0.0.3.tar.gz
    Requirement already satisfied: jupyter_client in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipykernel>=4.5.1->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (5.1.0)
    Requirement already satisfied: tornado>=4.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipykernel>=4.5.1->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.4.1)
    Requirement already satisfied: setuptools>=18.5 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (41.6.0)
    Requirement already satisfied: jedi>=0.10 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (0.11.0)
    Requirement already satisfied: decorator in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.4.1)
    Requirement already satisfied: pickleshare in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (0.7.4)
    Requirement already satisfied: simplegeneric>0.8 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (0.8.1)
    Requirement already satisfied: prompt_toolkit<2.0.0,>=1.0.4 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (1.0.15)
    Requirement already satisfied: pygments in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (2.1.3)
    Requirement already satisfied: pexpect in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.0.1)
    Requirement already satisfied: notebook>=4.4.1 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (5.0.0)
    Collecting utm
      Using cached https://files.pythonhosted.org/packages/b6/77/180f06153f2c1a8caf8409ff6365abc9423ec4ebc3991dfe4a3228bc09d4/utm-0.5.0.tar.gz
    Requirement already satisfied: numexpr>=2.5.2 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from tables->deepdish->-r ../requirements.txt (line 10)) (2.6.2)
    Requirement already satisfied: jupyter_core in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from jupyter_client->ipykernel>=4.5.1->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (4.4.0)
    Requirement already satisfied: pyzmq>=13 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from jupyter_client->ipykernel>=4.5.1->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (15.4.0)
    Requirement already satisfied: parso==0.1.* in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (0.1.1)
    Requirement already satisfied: wcwidth in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from prompt_toolkit<2.0.0,>=1.0.4->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=0.6.0->-r ../requirements.txt (line 5)) (0.1.7)
    Building wheels for collected packages: SimPEG, pymatsolver, properties, vectormath, discretize, pyMKL, utm
      Building wheel for SimPEG (setup.py) ... done
      Created wheel for SimPEG: filename=SimPEG-0.12.0-cp35-none-any.whl size=300157 sha256=adb0928480fe89c9518b13f9aaa2ac9a4dd09653d644472f622a0019459e2eaf
      Stored in directory: /home/nbuser/.cache/pip/wheels/2e/5a/e9/91fe65f3693bf49496c16b290be247c1ce2a83cbc0ecd6de89
      Building wheel for pymatsolver (setup.py) ... done
      Created wheel for pymatsolver: filename=pymatsolver-0.1.2-cp35-none-any.whl size=18199 sha256=e44bb10dc84d7c26fe2a0011608e919096c64542a503f66fc95cf3588fd2df5f
      Stored in directory: /home/nbuser/.cache/pip/wheels/09/66/a3/c540b8d29d1edab2bd4fabe368987aedc7fda18e742672be5b
      Building wheel for properties (setup.py) ... done
      Created wheel for properties: filename=properties-0.6.1-cp35-none-any.whl size=62368 sha256=e5ebe9c31f65485309599cf9e99097b024172e61f57597089373ac4e8d7cf007
      Stored in directory: /home/nbuser/.cache/pip/wheels/9d/d9/66/f1f984e6dbe9ca4fb0c51617e526a73945f90936d4ae040fc2
      Building wheel for vectormath (setup.py) ... done
      Created wheel for vectormath: filename=vectormath-0.2.2-cp35-none-any.whl size=9353 sha256=958cd9694614fe2b3a3b45b17b2646e7bd44c89aa9a43e68771333fafb5c662c
      Stored in directory: /home/nbuser/.cache/pip/wheels/82/45/db/21f8afb4447691ddc30be13170201154af9647c9653aa899b1
      Building wheel for discretize (setup.py) ... done
      Created wheel for discretize: filename=discretize-0.4.10-cp35-cp35m-linux_x86_64.whl size=2238852 sha256=091ee37106e078c5bdda54129b173201c657e89db866792f023a1d417459d8a1
      Stored in directory: /home/nbuser/.cache/pip/wheels/2f/64/bb/e65ceff1490fddea5cd123ba500360f0d400938fb0ce68872b
      Building wheel for pyMKL (setup.py) ... done
      Created wheel for pyMKL: filename=pyMKL-0.0.3-cp35-none-any.whl size=7069 sha256=0f45359485d13d5483dbde995cb1e52bbe5b8f96b9d4eec1c060aaa17f4edcac
      Stored in directory: /home/nbuser/.cache/pip/wheels/d6/40/40/df64f35d394a22fbecae00749cb8e7071798a3554fac9e5abe
      Building wheel for utm (setup.py) ... done
      Created wheel for utm: filename=utm-0.5.0-cp35-none-any.whl size=5944 sha256=d2cf95deae8cbdc20dc3ff3955d52a622159852735f2a23f25f22115f989474b
      Stored in directory: /home/nbuser/.cache/pip/wheels/22/85/c2/314ffed39b8e02ca623e9b7d2ad7fcf3f8544a0e77c096be76
    Successfully built SimPEG pymatsolver properties vectormath discretize pyMKL utm
    Installing collected packages: pyMKL, pymatsolver, properties, vectormath, discretize, utm, geoana, SimPEG, empymod, deepdish, em-examples
    Successfully installed SimPEG-0.12.0 deepdish-0.3.6 discretize-0.4.10 em-examples-0.0.35 empymod-1.10.3 geoana-0.0.5 properties-0.6.1 pyMKL-0.0.3 pymatsolver-0.1.2 utm-0.5.0 vectormath-0.2.2

```python
from em_examples import DCLayers
from IPython.display import display
%matplotlib inline
```

```python
from matplotlib import rcParams
rcParams['font.size'] = 14
```

# Purpose

## Investigating DC Resistivity Data

Using the widgets contained in this notebook we will explore the physical principals governing DC resistivity including the behavior of currents, electric field, electric potentials in a two layer earth.

The measured data in a DC experiment are potential differences, we will demonstrate how these  provide information about subsurface physical properties.

### Background: Computing Apparent Resistivity

In practice we cannot measure the potentials everywhere, we are limited to those locations where we place electrodes. For each source (current electrode pair) many potential differences are measured between M and N electrode pairs to characterize the overall distribution of potentials. The widget below allows you to visualize the potentials, electric fields, and current densities from a dipole source in a simple model with 2 layers. For different electrode configurations you can measure the potential differences and see the calculated apparent resistivities.

In a uniform halfspace the potential differences can be computed by summing up the potentials at each measurement point from the different current sources based on the following equations:

\begin{align}
    V_M = \frac{\rho I}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} \right] \\
    V_N = \frac{\rho I}{2 \pi} \left[ \frac{1}{AN} - \frac{1}{NB} \right]
\end{align}
where $AM$, $MB$, $AN$, and $NB$ are the distances between the corresponding electrodes.

The potential difference $\Delta V_{MN}$ in a dipole-dipole survey can therefore be expressed as follows,

\begin{equation}
    \Delta V_{MN} = V_M - V_N = \rho I \underbrace{\frac{1}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} - \frac{1}{AN} + \frac{1}{NB} \right]}_{G}
\end{equation}

and the resistivity of the halfspace $\rho$ is equal to,

$$
    \rho = \frac{\Delta V_{MN}}{IG}
$$

In this equation $G$ is often referred to as the geometric factor.

In the case where we are not in a uniform halfspace the above equation is used to compute the apparent resistivity ($\rho_a$) which is the resistivity of the uniform halfspace which best reproduces the measured potential difference.

In the top plot the location of the A electrode is marked by the red +, the B electrode is marked by the blue -, and the M/N potential electrodes are marked by the black dots. The $V_M$ and $V_N$ potentials are printed just above and to the right of the black dots. The calculted apparent resistivity is shown in the grey box to the right. The bottom plot can show the resistivity model, the electric fields (e), potentials, or current densities (j) depending on which toggle button is selected. Some patience may be required for the plots to update after parameters have been changed.

# LayeredEarth app

## Parameters:

 - **A**: (+) Current electrode  location
 - **B**: (-) Current electrode  location
 - **M**: (+) Potential electrode  location
 - **N**: (-) Potential electrode  location
 - **$\rho_1$**: Resistivity of the first layer
 - **$\rho_2$**: Resistivity of the second layer
 - **h**: Thickness of the first layer
 - **Plot**: Choice of 2D plot (Model, Potential, Electric field, Currents)

```python
out = DCLayers.plot_layer_potentials_app()
display(out)
```

    MyApp(children=(FloatSlider(value=-30.0, continuous_update=False, description='A', max=30.0, min=-30.0, step=1…

```python

```