This notebook is a work through for the implementation of a basic BM25 algorithm.

```python
import os
import nltk
nltk.download('popular');
from nltk.corpus import stopwords
# from nltk import word_tokenize
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
import string
import numpy as np
import re
import matplotlib.pyplot as plt
```

    [nltk_data] Downloading collection 'popular'
    [nltk_data]    |
    [nltk_data]    | Downloading package cmudict to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/cmudict.zip.
    [nltk_data]    | Downloading package gazetteers to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/gazetteers.zip.
    [nltk_data]    | Downloading package genesis to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/genesis.zip.
    [nltk_data]    | Downloading package gutenberg to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/gutenberg.zip.
    [nltk_data]    | Downloading package inaugural to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/inaugural.zip.
    [nltk_data]    | Downloading package movie_reviews to
    [nltk_data]    |     /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/movie_reviews.zip.
    [nltk_data]    | Downloading package names to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/names.zip.
    [nltk_data]    | Downloading package shakespeare to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/shakespeare.zip.
    [nltk_data]    | Downloading package stopwords to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/stopwords.zip.
    [nltk_data]    | Downloading package treebank to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/treebank.zip.
    [nltk_data]    | Downloading package twitter_samples to
    [nltk_data]    |     /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/twitter_samples.zip.
    [nltk_data]    | Downloading package omw to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/omw.zip.
    [nltk_data]    | Downloading package wordnet to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/wordnet.zip.
    [nltk_data]    | Downloading package wordnet31 to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/wordnet31.zip.
    [nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/wordnet_ic.zip.
    [nltk_data]    | Downloading package words to /root/nltk_data...
    [nltk_data]    |   Unzipping corpora/words.zip.
    [nltk_data]    | Downloading package maxent_ne_chunker to
    [nltk_data]    |     /root/nltk_data...
    [nltk_data]    |   Unzipping chunkers/maxent_ne_chunker.zip.
    [nltk_data]    | Downloading package punkt to /root/nltk_data...
    [nltk_data]    |   Unzipping tokenizers/punkt.zip.
    [nltk_data]    | Downloading package snowball_data to
    [nltk_data]    |     /root/nltk_data...
    [nltk_data]    | Downloading package averaged_perceptron_tagger to
    [nltk_data]    |     /root/nltk_data...
    [nltk_data]    |   Unzipping taggers/averaged_perceptron_tagger.zip.
    [nltk_data]    |
    [nltk_data]  Done downloading collection popular

Get the text document collection

```python
!wget https://storage.googleapis.com/pet-detect-239118/text_retrieval/documents.zip documents.zip
```

    --2021-10-30 00:52:50--  https://storage.googleapis.com/pet-detect-239118/text_retrieval/documents.zip
    Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.195.128, 74.125.199.128, 74.125.20.128, ...
    Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.195.128|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 226023 (221K) [application/x-zip-compressed]
    Saving to: ‘documents.zip’

    documents.zip       100%[===================>] 220.73K  --.-KB/s    in 0.002s

    2021-10-30 00:52:50 (99.1 MB/s) - ‘documents.zip’ saved [226023/226023]

    --2021-10-30 00:52:50--  http://documents.zip/
    Resolving documents.zip (documents.zip)... failed: Name or service not known.
    wget: unable to resolve host address ‘documents.zip’
    FINISHED --2021-10-30 00:52:51--
    Total wall clock time: 0.3s
    Downloaded: 1 files, 221K in 0.002s (99.1 MB/s)

```python
from zipfile import ZipFile
file_name = '/content/documents.zip'

with ZipFile(file_name, 'r',) as zip:
  zip.extractall()
  print('Done!!')
```

    Done!!

Parse the text document dataset into a dictionary

```python
def get_docDict(path):
  doc_dict = {}
  file_names = os.listdir(path)

  for file in file_names:
    full_path = path+'/'+file
    with open(full_path, 'r', errors='ignore') as f:
      data = f.readlines()
    text = "".join([i for i in data])
    # remove all the "\n" from the text
    text = re.sub("\n", " ", text)
    doc_dict[file] = text
  return doc_dict
```

```python
path = '/content/documents'
doc_dict = get_docDict(path)
```

Clean the text data

```python
def clean_text(doc_dict):
  """
  input - a dictionary of {filename : text}
  output - a dictionary of {filename : clean text}

  """
  clean_dict = {}
  stemmer = PorterStemmer()
  stopwords_english = stopwords.words('english')

  for name, doc in doc_dict.items():
    # remove extra white space
    text = re.sub(r"\s+", " ", doc)
    # remove extra ...
    text = re.sub(r"\.+"," ", doc)
    # remove hyphen
    text = re.sub(r"-","", text)
    text = text.lower()
    text_tokens = word_tokenize(text)
    text_clean = []
    for word in text_tokens:
      if (word not in stopwords_english and word not in string.punctuation):
        stem_word = stemmer.stem(word)
        text_clean.append(word)

    clean_dict[name] = text_clean

  return clean_dict
```

```python
clean_dict = clean_text(doc_dict)
clean_dict['A00-1000.pdf.txt'][:5]
```

    ['association', 'computational', 'linguistics', '6', 'th']

Build the dataset vocabulary

```python
def make_vocab(doc_dict):
  """
  input - a dictionary of {filename : clean text}
  output - a set of unique terms forms the dataset vocabulary
  """
  total_tokens = []
  for tokens in doc_dict.values():
    total_tokens += tokens
  vocab = list(set(total_tokens))
  return vocab
```

```python
vocab = make_vocab(clean_dict)
len(vocab)
```

    10811

Calculate TF-IDF

```python
# compute document-wise term frequency

def get_DocTF(doc_dict, vocab):
  """
  input - a dictionary of {filename : clean text}, the vocabulary of the whole dataset
  output - a dictionary of {filename : {term : count}}
  """
  tf_dict = {}
  # make the dict for filename=>{term:frequency}
  for doc_id in doc_dict.keys():
    tf_dict[doc_id] = {}

  for word in vocab:
    for doc_id, text in doc_dict.items():
      tf_dict[doc_id][word] = text.count(word)

  return tf_dict
```

```python
tf_dict = get_DocTF(clean_dict, vocab)
```

```python
# compute document frequency

def get_DocDF(clean_dict, vocab):
  """
  input - a dictionary of {filename : clean text}, the vocabulary of the whole dataset
  output - a dictionary of all terms in the vocabulary - {term : count}
  """
  df_dict = {}
  for word in vocab:
    freq = 0
    for text_tokens in clean_dict.values():
      if word in text_tokens:
        freq += 1
    df_dict[word] = freq

  return df_dict
```

```python
df_dict = get_DocDF(clean_dict, vocab)
```

```python
# compute IDF

def inverse_DF(df_dict, vocab, N):
  """
  input - a dictionary of DF {term : count}, the vocabulary of the whole dataset, total # of documents in the dataset
  output - a dictionary of IDF of all terms in the vocabulary - {term : inver_df}
  """
  idf_dict = {}

  for word in vocab:
    N_q = df_dict[word]
    # idf_dict[word] = np.log10(N / N_q)
    idf_dict[word] = np.log(((N - N_q + 0.5) / (N_q + 0.5)) + 1)

  return idf_dict
```

```python
N = len(tf_dict.keys())
idf_dict = inverse_DF(df_dict, vocab, N)
```

```python
idf_dict['language']
```

    0.02298951822469878

```python
# compute the TF-IDF dictionary

def get_tf_idf(tf_dict, idf_dict, doc_dict, vocab):
  tf_idf_dict = {}
  for doc_id in doc_dict.keys():
    tf_idf_dict[doc_id] = {}

  for word in vocab:
    for doc_id, text_tokens in doc_dict.items():
      tf_idf_dict[doc_id][word] = round((tf_dict[doc_id][word] * idf_dict[word]), 4)
  return tf_idf_dict
```

```python
tf_idf_dict = get_tf_idf(tf_dict, idf_dict, doc_dict, vocab)
tf_idf_dict['A00-1001.pdf.txt']['language']
```

    0.5288

To use the BM25 algorithm, we still need to compute TF and IDF.

The **TF** component includes the query frequency, and total number of terms as with TF-IDF, but we add two special parameters `k` and `b` which we can use to optimize our algorithm, but by default we stick with the values of `1.2` and `0.75` respectively. We also include two new values:
* Average document length (avgdl) - the average length of *all* documents
* *D* - the length of the current document.

Our new **IDF** component is similar, it maintains the parametes `N` and `N_q`, which are the number of documents and the number of documents *that contain* our query, respectively. And all we do is add a few values (0.5, 1) here and there.

```python
# compute average document length (use the cleaned text)

def get_avgdl(clean_dict):
  total_doc = len(clean_dict.keys())
  total_length = 0
  for text in clean_dict.values():
    total_length += len(text)

  avgdl = total_length / total_doc

  return round(avgdl, 4)
```

```python
avgdl = get_avgdl(clean_dict)
avgdl
```

    2764.3333

# Implement the BM25 algorithm
* compute a dictionary of a score of a given term from a given document
* the total score of a given document is presented as
\begin{align}
    BM25(d)=\sum_{t=q, f_t,_d >0} log(1 + \frac{N-df_t+0.5}{df_t+0.5}) \cdot \frac{f_t,_d}{f_t,_d + K \cdot (1-b+b\frac{l(d)}{avgdl}) }
\end{align}

*d* - document, *t* - term, *q* - word from the query, *N* - total number of documents

$f_t,_d$ - term frequenct (TF) given a word from query, $df_t$ - document frequency of a given term

$l(d)$ - total length of a given document d, $avgdl$ - average length of document in the dataset

```python
# compute BM25 score dictionary

def bm25(tf_dict, clean_dict, df_dict, vocab, k=1.2, b=0.75):
  bm25_dict = {}
  avgdl = get_avgdl(clean_dict)
  N = len(clean_dict.keys())

  # create the collection of dictionaries of all documents
  for doc_id in clean_dict.keys():
    bm25_dict[doc_id] = {}

  for word in vocab:
    for doc_id, text_tokens in clean_dict.items():
      freq = tf_dict[doc_id][word]
      # the TF in BM25
      tf = (freq*(k+1)) / (freq + k*(1-b+b*len(clean_dict[doc_id])/avgdl))
      # get DF
      N_q = df_dict[word]
      idf = np.log(((N - N_q + 0.5) / (N_q + 0.5)) + 1)
      score = round(tf*idf,4)
      bm25_dict[doc_id][word] = score

  return bm25_dict
```

```python
bm25_dict = bm25(tf_dict, clean_dict, df_dict, vocab)
```

```python
bm25_dict['A00-1020.pdf.txt']['language']
```

    0.0415

# Define the BM25 Model (Probabilistic Model)
* To find the most relevant documents related to the query
* Pass the query along with the document set (dictionary) and the BM25 scores of all words in the vocabulary
* Returns the top 5 documents ID and scores

```python
def BM25Model(query, doc_dict, bm25_dict):
  query_vocab = []
  query = query.lower()
  query = re.sub(r"\s+", " ", query)
  stopwords_english = stopwords.words('english')

  for word in query.split():
    if (word not in string.punctuation and word not in stopwords_english):
        query_vocab.append(word)

  query_wc = {}
  for word in query_vocab:
    query_wc[word] = query.split().count(word)

  relevance_scores = {}
  # use the raw doc_dict to get the filename only
  for doc_id in doc_dict.keys():
    score = 0
    for word in query_vocab:
      score += query_wc[word] * bm25_dict[doc_id][word]
    relevance_scores[doc_id] = round(score,4)

  # sort the relevance score and get the top-k ranking
  # sort the keys of the relevance score by value
  sort_keys = sorted(relevance_scores, key=relevance_scores.get , reverse = True)
  top_keys = sort_keys[:5]
  top_5 = {}
  for key in top_keys:
    top_5[key] = relevance_scores[key]

  return top_5
```

The vector space model (VSM)

```python
def get_tf_idf(tf_dict, idf_dict, doc_dict, vocab):
  tf_idf_dict = {}
  for doc_id in doc_dict.keys():
    tf_idf_dict[doc_id] = {}

  for word in vocab:
    for doc_id, text_tokens in doc_dict.items():
      tf_idf_dict[doc_id][word] = round((tf_dict[doc_id][word] * idf_dict[word]), 4)
  return tf_idf_dict

def vectorSpaceModel(query, doc_dict,tfidf_dict):
  query_vocab = []
  query = query.lower()
  query = re.sub(r"\s+", " ", query)
  stopwords_english = stopwords.words('english')

  for word in query.split():
    if (word not in string.punctuation and word not in stopwords_english):
        query_vocab.append(word)

  query_wc = {}
  for word in query_vocab:
    query_wc[word] = query.split().count(word)

  relevance_scores = {}
  for doc_id in doc_dict.keys():
    score = 0
    for word in query_vocab:
      score += query_wc[word] * tf_idf_dict[doc_id][word]
    relevance_scores[doc_id] = round(score,4)

  # sort the relevance score and get the top-k ranking
  # sort the keys of the relevance score by value
  sort_keys = sorted(relevance_scores, key=relevance_scores.get , reverse = True)
  top_keys = sort_keys[:5]
  top_5 = {}
  for key in top_keys:
    top_5[key] = relevance_scores[key]

  return top_5
```

```python
tfidf_dict = get_tf_idf(tf_dict, idf_dict, doc_dict, vocab)
```

```python
query1 = "Natural Language"
result1 = BM25Model(query1, doc_dict, bm25_dict)
print(f'ranked by BM25: {result1}')
result2 = vectorSpaceModel(query1, doc_dict, tfidf_dict)
print(f'ranked by VSM: {result2}')
```

    ranked by BM25: {'A00-1001.pdf.txt': 0.1977, 'A00-1007.pdf.txt': 0.1926, 'A00-1005.pdf.txt': 0.188, 'A00-1000.pdf.txt': 0.1856, 'A00-1009.pdf.txt': 0.1831}
    ranked by VSM: {'A00-1001.pdf.txt': 2.2942, 'A00-1007.pdf.txt': 1.8492, 'A00-1005.pdf.txt': 1.2415, 'A00-1016.pdf.txt': 0.9508, 'A00-1009.pdf.txt': 0.9115}

```python
query2 = "generative model"
result1 = BM25Model(query2, doc_dict, bm25_dict)
print(f'ranked by BM25: {result1}')
tfidf_dict = get_tf_idf(tf_dict, idf_dict, doc_dict, vocab)
result2 = vectorSpaceModel(query2, doc_dict, tfidf_dict)
print(f'ranked by VSM: {result2}')
```

    ranked by BM25: {'A00-1010.pdf.txt': 3.7823, 'A00-1014.pdf.txt': 2.5833, 'A00-1004.pdf.txt': 0.8908, 'A00-1019.pdf.txt': 0.8835, 'A00-1007.pdf.txt': 0.7827}
    ranked by VSM: {'A00-1004.pdf.txt': 20.8447, 'A00-1019.pdf.txt': 15.0082, 'A00-1010.pdf.txt': 6.0171, 'A00-1014.pdf.txt': 4.2593, 'A00-1007.pdf.txt': 3.3352}

```python
query3 = "text retrieval for nature language"
result1 = BM25Model(query3, doc_dict, bm25_dict)
print(f'ranked by BM25: {result1}')
tfidf_dict = get_tf_idf(tf_dict, idf_dict, doc_dict, vocab)
result2 = vectorSpaceModel(query3, doc_dict, tfidf_dict)
print(f'ranked by VSM: {result2}')
```

    ranked by BM25: {'A00-1019.pdf.txt': 4.0482, 'A00-1004.pdf.txt': 3.9572, 'A00-1018.pdf.txt': 3.1224, 'A00-1003.pdf.txt': 3.0071, 'A00-1000.pdf.txt': 2.6395}
    ranked by VSM: {'A00-1003.pdf.txt': 60.3235, 'A00-1004.pdf.txt': 12.596, 'A00-1018.pdf.txt': 12.0305, 'A00-1012.pdf.txt': 11.9561, 'A00-1019.pdf.txt': 8.0488}

Visualize score versus document length

```python
BM25_term_score = []
VSM_term_score = []
doc_length = []

for doc_id in doc_dict.keys():
  doc_length.append(len(clean_dict[doc_id]))
  vsm_score = np.sum([x for x in tfidf_dict[doc_id].values()])
  VSM_term_score.append(vsm_score)
  bm25_score = np.sum([x for x in bm25_dict[doc_id].values()])
  BM25_term_score.append(bm25_score)

```

```python
fig = plt.figure(figsize=(12, 8))
data = [BM25_term_score, VSM_term_score]
X = np.arange(21)

ax = fig.add_axes([0,0,1,1])
ax.bar(X + 0.00, data[0], color = 'r', width = 0.25)
ax.bar(X + 0.25, data[1], color = 'g', width = 0.25)
ax.legend(['BM25 score', 'VSM score'])
```


```python
# Importing packages
import matplotlib.pyplot as plt
import pandas as pd
import ipywidgets as widgets
import numpy as np
from ipywidgets import Layout
import seaborn as sns
import itertools
import math

import time
from scipy import linalg
import scipy.optimize as optimize
import sympy as sm

from data_gen import gen_df

# local module for linear algebra
%load_ext autoreload
%autoreload 2
```

```python
genre_list = {'Action', 'Adult', 'Adventure', 'Animation', 'Biography',
         'Comedy', 'Crime', 'Documentary', 'Drama', 'Family', 'Fantasy',
         'FilmNoir', 'GameShow', 'History', 'Horror', 'Music', 'Musical',
         'Mystery', 'News', 'RealityTV', 'Romance', 'SciFi', 'Short', 'Sport',
         'TalkShow', 'Thriller', 'War', 'Western'}

# Favorite Genre
def genre_func(genre_num, title, text, message):
    root = Tk.StringVar() # Create tk object
    root.title(title)

    # Add a grid
    mainframe = Frame(root)                           # Pass tk object to tkinter frame
    mainframe.grid(column=0,row=0, sticky=(N,W,E,S) ) # Add grid to frame
    mainframe.columnconfigure(0, weight = 1)
    mainframe.rowconfigure(0, weight = 1)
    mainframe.pack(pady = 100, padx = 100)

    # Create a Tkinter variable
    genre_gen = StringVar(root)

    # Call genre_list as options to choice from
    choices = genre_list
    genre_gen.set("") # Set the default option

    # Pop up window with drop down
    popupMenu = OptionMenu(mainframe, genre_gen, *choices)
    Label(mainframe, text=text).grid(row = 1, column = 1)
    popupMenu.grid(row = 2, column =1)

    # Function on change dropdown value
    def change_dropdown(*args):
        print(message + genre_gen.get() )

    # Link function to change dropdown
    genre_gen.trace('w', change_dropdown)

    root.mainloop()
    genre_num = genre_gen.get()

genre_func(genre_num="genre1", title="Favorite genre", text="What is your favorite genre?", message="Your favorite genre is ")
genre_func(genre_num="genre2", title="Second favorite genre", text="What is your second favorite genre?", message="Your second favorite genre is ")
genre_func(genre_num="genre3", title="Third favorite genre", text="What is your third favorite genre?", message="Your third favorite genre is ")
genre_func(genre_num="genre4", title="Fourth favorite genre", text="What is your fourth favorite genre?", message="Your fourth favorite genre is ")
genre_func(genre_num="genre5", title="Fifth favorite genre", text="What is your fifth favorite genre?", message="Your fifth favorite genre is ")

print(genre1)
print(genre2)
print(genre3)
print(genre4)
print(genre5)

```

    Your favorite genre is Animation
    Your second favorite genre is Romance
    Your third favorite genre is Comedy
    genre1
    genre2
    genre3

```python
decades = {"1920s", "1930s", "1940s", "1950s", "1960s", "1970s", "1980s", "1990s", "2000s", "2010s"}

# Favorite Decade
def decade_func(decade_choice, title, text, message):
    root = Tk() # Create tk object
    root.title(title)

    # Add a grid
    mainframe = Frame(root)                           # Pass tk object to tkinter frame
    mainframe.grid(column=0,row=0, sticky=(N,W,E,S) ) # Add grid to frame
    mainframe.columnconfigure(0, weight = 1)
    mainframe.rowconfigure(0, weight = 1)
    mainframe.pack(pady = 100, padx = 100)

    # Create a Tkinter variable
    decade_gen = StringVar(root)

    # Call genre_list as options to choice from
    choices = decades
    decade_gen.set("") # Set the default option

    # Pop up window with drop down
    popupMenu = OptionMenu(mainframe, decade_gen, *choices)
    Label(mainframe, text=text).grid(row = 1, column = 1)
    popupMenu.grid(row = 2, column =1)

    # Function on change dropdown value
    def change_dropdown(*args):
        print(message + decade_gen.get() )

    # Link function to change dropdown
    decade_gen.trace('w', change_dropdown)

    root.mainloop()
    decade_choice = decade_gen.get()

decade_func(decade_choice="decade", title="Favorite Decade", text="Which decade is your favorite movie decade?", message="Your favorite movie decade is ")

print(decade_func)

```

```python
genres = ['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
          'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
          'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western']

print(genres)

df_X["genre"] = ""

for i in genres:
    df_X.loc[df_X[i] == 1, "genre"] = i

df_X.head(10)

```


# Create SBML model from BioCRNpyler

```python
try:
    from biocrnpyler import *
    #Here are some parameters loaded into default_parameters.txt
    #Promoter Names: strong, medium, weak correspond to bicistronic RBSs: BCD2, BCD8 and BCD12
    #RBS Names: strong, medium, Weak correspond to Anderson Promoters: J23100, J23106, and J23103
    G = DNAassembly("X", promoter = "strong", rbs = "weak", transcript = None, protein = None)
    #Also notice that the names of transcript and protein can be changed, or set to Species.
    x0_dict = {"protein_RNAP_machinery":10., "protein_RNAase_machinery":5.0, "protein_Ribo_machinery":50.,
                   'dna_X':0.5, 'dna_cellular_processes':0.01}
    model_txtl = TxTlDilutionMixture("txtl", components = [G], parameter_file = "default_parameters.txt",
                                    initial_condition_dictionary = x0_dict)

    CRN = model_txtl.compile_crn()
    print(repr(CRN),"\n", CRN.pretty_print(show_attributes = True, show_material = True, show_rates = True),"\n\n")
    # except:
    #     print('BioCRNpyler not found. To run this notebook, make sure you run pip install biocrnpyler first.')
    CRN.write_sbml_file('biocrnpyler_gene_expression.xml')
except:
    print('BioCRNpyler not found. To run this notebook make sure you run pip install biocrpyler first.')
```

    Species = dna_X, protein_RNAP_machinery, rna_X, complex_dna_X_protein_RNAP_machinery_, protein_Ribo_machinery, protein_X, complex_protein_Ribo_machinery_rna_X_, protein_RNAase_machinery, dna_cellular_processes, rna_cellular_processes, complex_dna_cellular_processes_protein_RNAP_machinery_, protein_cellular_processes, complex_protein_Ribo_machinery_rna_cellular_processes_, complex_protein_RNAase_machinery_rna_X_, complex_complex_protein_Ribo_machinery_rna_X__protein_RNAase_machinery_, complex_protein_RNAase_machinery_rna_cellular_processes_, complex_complex_protein_Ribo_machinery_rna_cellular_processes__protein_RNAase_machinery_
    Reactions = [
    	dna[X]+protein[RNAP(machinery)] <--> complex[dna[X]:protein[RNAP]]
    	complex[dna[X]:protein[RNAP]] --> dna[X]+rna[X]+protein[RNAP(machinery)]
    	rna[X]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[X]]
    	complex[protein[Ribo]:rna[X]] --> rna[X]+protein[X]+protein[Ribo(machinery)]
    	dna[cellular_processes]+protein[RNAP(machinery)] <--> complex[dna[cellular_processes]:protein[RNAP]]
    	complex[dna[cellular_processes]:protein[RNAP]] --> dna[cellular_processes]+rna[cellular_processes]+protein[RNAP(machinery)]
    	rna[cellular_processes]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[cellular_processes]]
    	complex[protein[Ribo]:rna[cellular_processes]] --> rna[cellular_processes]+protein[cellular_processes]+protein[Ribo(machinery)]
    	rna[X]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[X]]
    	complex[protein[RNAase]:rna[X]] --> protein[RNAase(machinery)]
    	complex[protein[Ribo]:rna[X]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[X]]:protein[RNAase]]
    	complex[complex[protein[Ribo]:rna[X]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    	rna[cellular_processes]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[cellular_processes]]
    	complex[protein[RNAase]:rna[cellular_processes]] --> protein[RNAase(machinery)]
    	complex[protein[Ribo]:rna[cellular_processes]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
    	complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    	rna[X] -->
    	protein[X] -->
    	rna[cellular_processes] -->
    	protein[cellular_processes] -->
    ]
     Species (17) = {0. dna[X] init_conc = 0.5, 1. protein[RNAP(machinery)] init_conc = 10.0, 2. rna[X] init_conc = 0, 3. complex[dna[X]:protein[RNAP]] init_conc = 0, 4. protein[Ribo(machinery)] init_conc = 50.0, 5. protein[X] init_conc = 0, 6. complex[protein[Ribo]:rna[X]] init_conc = 0, 7. protein[RNAase(machinery)] init_conc = 5.0, 8. dna[cellular_processes] init_conc = 0.01, 9. rna[cellular_processes] init_conc = 0, 10. complex[dna[cellular_processes]:protein[RNAP]] init_conc = 0, 11. protein[cellular_processes] init_conc = 0, 12. complex[protein[Ribo]:rna[cellular_processes]] init_conc = 0, 13. complex[protein[RNAase]:rna[X]] init_conc = 0, 14. complex[complex[protein[Ribo]:rna[X]]:protein[RNAase]] init_conc = 0, 15. complex[protein[RNAase]:rna[cellular_processes]] init_conc = 0, 16. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] init_conc = 0}

    Reactions (20) = [
    0. dna[X]+protein[RNAP(machinery)] <--> complex[dna[X]:protein[RNAP]]
     Kf=k_forward * dna_X * protein_RNAP_machinery
     Kr=k_reverse * complex_dna_X_protein_RNAP_machinery_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=transcription_mm, partid=strong, name=kb).
      k_reverse=0.5
      found_key=(mech=None, partid=strong, name=ku).
      search_key=(mech=transcription_mm, partid=strong, name=ku).

    1. complex[dna[X]:protein[RNAP]] --> dna[X]+rna[X]+protein[RNAP(machinery)]
     Kf=k_forward * complex_dna_X_protein_RNAP_machinery_
      k_forward=3.926187672
      found_key=(mech=None, partid=strong, name=ktx).
      search_key=(mech=transcription_mm, partid=strong, name=ktx).

    2. rna[X]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[X]]
     Kf=k_forward * rna_X * protein_Ribo_machinery
     Kr=k_reverse * complex_protein_Ribo_machinery_rna_X_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=translation_mm, partid=weak, name=kb).
      k_reverse=5.0
      found_key=(mech=None, partid=weak, name=ku).
      search_key=(mech=translation_mm, partid=weak, name=ku).

    3. complex[protein[Ribo]:rna[X]] --> rna[X]+protein[X]+protein[Ribo(machinery)]
     Kf=k_forward * complex_protein_Ribo_machinery_rna_X_
      k_forward=0.05
      found_key=(mech=None, partid=None, name=ktl).
      search_key=(mech=translation_mm, partid=weak, name=ktl).

    4. dna[cellular_processes]+protein[RNAP(machinery)] <--> complex[dna[cellular_processes]:protein[RNAP]]
     Kf=k_forward * dna_cellular_processes * protein_RNAP_machinery
     Kr=k_reverse * complex_dna_cellular_processes_protein_RNAP_machinery_
      k_forward=500
      found_key=(mech=transcription, partid=None, name=kb).
      search_key=(mech=transcription_mm, partid=average_promoter, name=kb).
      k_reverse=50
      found_key=(mech=transcription, partid=None, name=ku).
      search_key=(mech=transcription_mm, partid=average_promoter, name=ku).

    5. complex[dna[cellular_processes]:protein[RNAP]] --> dna[cellular_processes]+rna[cellular_processes]+protein[RNAP(machinery)]
     Kf=k_forward * complex_dna_cellular_processes_protein_RNAP_machinery_
      k_forward=0.1
      found_key=(mech=transcription, partid=None, name=ktx).
      search_key=(mech=transcription_mm, partid=average_promoter, name=ktx).

    6. rna[cellular_processes]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[cellular_processes]]
     Kf=k_forward * rna_cellular_processes * protein_Ribo_machinery
     Kr=k_reverse * complex_protein_Ribo_machinery_rna_cellular_processes_
      k_forward=500
      found_key=(mech=translation, partid=None, name=kb).
      search_key=(mech=translation_mm, partid=average_rbs, name=kb).
      k_reverse=5
      found_key=(mech=translation, partid=None, name=ku).
      search_key=(mech=translation_mm, partid=average_rbs, name=ku).

    7. complex[protein[Ribo]:rna[cellular_processes]] --> rna[cellular_processes]+protein[cellular_processes]+protein[Ribo(machinery)]
     Kf=k_forward * complex_protein_Ribo_machinery_rna_cellular_processes_
      k_forward=0.1
      found_key=(mech=translation, partid=None, name=ktl).
      search_key=(mech=translation_mm, partid=average_rbs, name=ktl).

    8. rna[X]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[X]]
     Kf=k_forward * rna_X * protein_RNAase_machinery
     Kr=k_reverse * complex_protein_RNAase_machinery_rna_X_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=rna_degredation_mm, partid=rna_X, name=kb).
      k_reverse=10.0
      found_key=(mech=None, partid=None, name=ku).
      search_key=(mech=rna_degredation_mm, partid=rna_X, name=ku).

    9. complex[protein[RNAase]:rna[X]] --> protein[RNAase(machinery)]
     Kf=k_forward * complex_protein_RNAase_machinery_rna_X_
      k_forward=0.001
      found_key=(mech=rna_degredation_mm, partid=None, name=kdeg).
      search_key=(mech=rna_degredation_mm, partid=rna_X, name=kdeg).

    10. complex[protein[Ribo]:rna[X]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[X]]:protein[RNAase]]
     Kf=k_forward * complex_protein_Ribo_machinery_rna_X_ * protein_RNAase_machinery
     Kr=k_reverse * complex_complex_protein_Ribo_machinery_rna_X__protein_RNAase_machinery_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_X_, name=kb).
      k_reverse=10.0
      found_key=(mech=None, partid=None, name=ku).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_X_, name=ku).

    11. complex[complex[protein[Ribo]:rna[X]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
     Kf=k_forward * complex_complex_protein_Ribo_machinery_rna_X__protein_RNAase_machinery_
      k_forward=0.001
      found_key=(mech=rna_degredation_mm, partid=None, name=kdeg).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_X_, name=kdeg).

    12. rna[cellular_processes]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[cellular_processes]]
     Kf=k_forward * rna_cellular_processes * protein_RNAase_machinery
     Kr=k_reverse * complex_protein_RNAase_machinery_rna_cellular_processes_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=rna_degredation_mm, partid=rna_cellular_processes, name=kb).
      k_reverse=10.0
      found_key=(mech=None, partid=None, name=ku).
      search_key=(mech=rna_degredation_mm, partid=rna_cellular_processes, name=ku).

    13. complex[protein[RNAase]:rna[cellular_processes]] --> protein[RNAase(machinery)]
     Kf=k_forward * complex_protein_RNAase_machinery_rna_cellular_processes_
      k_forward=0.001
      found_key=(mech=rna_degredation_mm, partid=None, name=kdeg).
      search_key=(mech=rna_degredation_mm, partid=rna_cellular_processes, name=kdeg).

    14. complex[protein[Ribo]:rna[cellular_processes]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
     Kf=k_forward * complex_protein_Ribo_machinery_rna_cellular_processes_ * protein_RNAase_machinery
     Kr=k_reverse * complex_complex_protein_Ribo_machinery_rna_cellular_processes__protein_RNAase_machinery_
      k_forward=100.0
      found_key=(mech=None, partid=None, name=kb).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_cellular_processes_, name=kb).
      k_reverse=10.0
      found_key=(mech=None, partid=None, name=ku).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_cellular_processes_, name=ku).

    15. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
     Kf=k_forward * complex_complex_protein_Ribo_machinery_rna_cellular_processes__protein_RNAase_machinery_
      k_forward=0.001
      found_key=(mech=rna_degredation_mm, partid=None, name=kdeg).
      search_key=(mech=rna_degredation_mm, partid=complex_protein_Ribo_machinery_rna_cellular_processes_, name=kdeg).

    16. rna[X] -->
     Kf=k_forward * rna_X
      k_forward=0.001
      found_key=(mech=None, partid=None, name=kdil).
      search_key=(mech=global_degredation_via_dilution, partid=rna_X, name=kdil).

    17. protein[X] -->
     Kf=k_forward * protein_X
      k_forward=0.001
      found_key=(mech=None, partid=None, name=kdil).
      search_key=(mech=global_degredation_via_dilution, partid=protein_X, name=kdil).

    18. rna[cellular_processes] -->
     Kf=k_forward * rna_cellular_processes
      k_forward=0.001
      found_key=(mech=None, partid=None, name=kdil).
      search_key=(mech=global_degredation_via_dilution, partid=rna_cellular_processes, name=kdil).

    19. protein[cellular_processes] -->
     Kf=k_forward * protein_cellular_processes
      k_forward=0.001
      found_key=(mech=None, partid=None, name=kdil).
      search_key=(mech=global_degredation_via_dilution, partid=protein_cellular_processes, name=kdil).

    ]

    True

```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

from autoreduce import *
import numpy as np
from sympy import symbols
```

```python
from autoreduce.converters import load_sbml
x, f, P, params_values, x_init = load_sbml('biocrnpyler_gene_expression.xml')
# x[5] is desired output protein, set the output matrix C, in y = Cx, accordingly:
C = np.zeros( (1, len(x)), dtype = int)
C[0][5] = 1
# Set initial conditions:
sys = System(x, f, params = P, params_values = params_values, C = C, x_init = x_init)
```

```python
sys.x
```

    [dna_X,
     protein_RNAP_machinery,
     rna_X,
     complex_dna_X_protein_RNAP_machinery_,
     protein_Ribo_machinery,
     protein_X,
     complex_protein_Ribo_machinery_rna_X_,
     protein_RNAase_machinery,
     dna_cellular_processes,
     rna_cellular_processes,
     complex_dna_cellular_processes_protein_RNAP_machinery_,
     protein_cellular_processes,
     complex_protein_Ribo_machinery_rna_cellular_processes_,
     complex_protein_RNAase_machinery_rna_X_,
     complex_complex_protein_Ribo_machinery_rna_X__protein_RNAase_machinery_,
     complex_protein_RNAase_machinery_rna_cellular_processes_,
     complex_complex_protein_Ribo_machinery_rna_cellular_processes__protein_RNAase_machinery_]

# Create and solve the ODE for the model

```python
from autoreduce.utils import get_ODE
timepoints_ode = np.linspace(0, 200, 100)
sys_ode = get_ODE(sys, timepoints_ode)
sol = sys_ode.solve_system().T
try:
    import matplotlib.pyplot as plt
    plt.plot(timepoints_ode, np.transpose(np.array(C)@sol))
    plt.xlabel('Time')
    plt.ylabel('[Outputs]')
    plt.show()
except:
    print('Plotting libraries missing.')
```

# Local sensitivity analysis for BioCRNpyler model

```python
from autoreduce.utils import get_SSM
timepoints_ssm = np.linspace(0,20,10)
sys_ssm = get_SSM(sys, timepoints_ssm)
# Uncomment to run
# Ss = sys_ssm.compute_SSM() # len(timepoints) x len(params) x len(states)
```

    SSM Progress: |██████████████████████████████████████████████████| 100.0% Complete

```python
# out_Ss = []
# for i in range(len(P)):
#     out_Ss.append((np.array(C)@(Ss[:,i,:].T)))
# out_Ss = np.reshape(np.array(out_Ss), (len(timepoints_ssm), len(P), 1))
```

# Plot heatmap for sensitivity analysis

```python
# try:
    # import seaborn as sn
    # import matplotlib.pyplot as plt
    # for j in range(1):
    #     sn.heatmap(out_Ss[:,:,j].T)
    #     plt.xlabel('Time')
    #     plt.ylabel('Parameters')
    #     plt.title('Sensitivity of output[{0}] with respect to all parameters'.format(j))
    #     plt.show()
# except:
#     print('Plotting libraries missing.')
```

# Model reduction starts here:

```python
from autoreduce.utils import get_reducible
timepoints_ssm = np.linspace(0,20,10)
sys_reduce = get_reducible(sys, timepoints_ode, timepoints_ssm)
sys_reduce.nstates_tol_min = 3 # Minimum number of states tbat you want in the reduced model
```

# Conservation Laws (if any)

```python
# G, P, C1, T, R, C2, E, C3, X = sys.x
# conserved_quantities = [P + C1 - P_tot, R + C2 - R_tot, E + C3 - E_tot]
# states_to_eliminate = [1,4,6]
# f_cons = sys_reduce.set_conservation_laws(conserved_quantities, states_to_eliminate)
```

```python
# from autoreduce.utils import get_ODE
# timepoints_ode = np.linspace(0, 2, 100)
# params_values = [100, 10, 4, 10, 0.25, 2, 10, 0.5, 1, 10, 1000, 1000, 1000]
# sys_reduce.params_values = params_values
# sys_ode = get_ODE(sys_reduce, timepoints_ode)
# sol = sys_ode.solve_system().T
# try:
    # import matplotlib.pyplot as plt
    # plt.plot(timepoints_ode, np.transpose(np.array(sys_reduce.C)@sol))
    # plt.xlabel('Time')
    # plt.ylabel('[Outputs]')
    # plt.show()
# except:
#     print('Plotting libraries missing.')
```

# Derive model with no intermediate complexes

```python
# states_to_remove = [x[12], x[12], x[14], x[15], x[16]]
# retained_states = []
# for state in x:
#     index = sys.x.index(state)
#     if state not in states_to_remove:
#         retained_states.append(x[index])
reduced_sys, fast_ss = sys_reduce.solve_timescale_separation([x[0], x[1], x[2], x[3], x[4], x[5],
                                                             x[6], x[7], x[8], x[9], x[10], x[11]],
                                                             debug = False)
# reduced_sys.f[1]
# Se = sys_reduce.get_robustness_metric(reduced_sys, fast_ss, [0,2,5])
```

    count 1
    Successful time-scale separation solution obtained with states: [dna_X, protein_RNAP_machinery, rna_X, complex_dna_X_protein_RNAP_machinery_, protein_Ribo_machinery, protein_X, complex_protein_Ribo_machinery_rna_X_, protein_RNAase_machinery, dna_cellular_processes, rna_cellular_processes, complex_dna_cellular_processes_protein_RNAP_machinery_, protein_cellular_processes]!

```python
reduced_sys.f
```

    [complex_dna_X_protein_RNAP_machinery_*ktx_strong_ + complex_dna_X_protein_RNAP_machinery_*ku_strong_ - dna_X*kb__*protein_RNAP_machinery,
     complex_dna_X_protein_RNAP_machinery_*ktx_strong_ + complex_dna_X_protein_RNAP_machinery_*ku_strong_ + complex_dna_cellular_processes_protein_RNAP_machinery_*ktx__transcription + complex_dna_cellular_processes_protein_RNAP_machinery_*ku__transcription - dna_X*kb__*protein_RNAP_machinery - dna_cellular_processes*kb__transcription*protein_RNAP_machinery,
     complex_dna_X_protein_RNAP_machinery_*ktx_strong_ + complex_protein_Ribo_machinery_rna_X_*ktl__ + complex_protein_Ribo_machinery_rna_X_*ku_weak_ + kb__*ku__*protein_RNAase_machinery*rna_X/(kdeg__rna_degredation_mm + ku__) - kb__*protein_RNAase_machinery*rna_X - kb__*protein_Ribo_machinery*rna_X - kdil__*rna_X,
     -complex_dna_X_protein_RNAP_machinery_*ktx_strong_ - complex_dna_X_protein_RNAP_machinery_*ku_strong_ + dna_X*kb__*protein_RNAP_machinery,
     complex_protein_Ribo_machinery_rna_X_*kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery/(kdeg__rna_degredation_mm + ku__) + complex_protein_Ribo_machinery_rna_X_*ktl__ + complex_protein_Ribo_machinery_rna_X_*ku_weak_ + kb__*kb__translation*kdeg__rna_degredation_mm*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) - kb__*protein_Ribo_machinery*rna_X - kb__translation*protein_Ribo_machinery*rna_cellular_processes + ktl__translation*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation) + ku__translation*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation),
     complex_protein_Ribo_machinery_rna_X_*ktl__ - kdil__*protein_X,
     complex_protein_Ribo_machinery_rna_X_*kb__*ku__*protein_RNAase_machinery/(kdeg__rna_degredation_mm + ku__) - complex_protein_Ribo_machinery_rna_X_*kb__*protein_RNAase_machinery - complex_protein_Ribo_machinery_rna_X_*ktl__ - complex_protein_Ribo_machinery_rna_X_*ku_weak_ + kb__*protein_Ribo_machinery*rna_X,
     complex_protein_Ribo_machinery_rna_X_*kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery/(kdeg__rna_degredation_mm + ku__) + complex_protein_Ribo_machinery_rna_X_*kb__*ku__*protein_RNAase_machinery/(kdeg__rna_degredation_mm + ku__) - complex_protein_Ribo_machinery_rna_X_*kb__*protein_RNAase_machinery + kb__*kb__translation*kdeg__rna_degredation_mm*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery*rna_X/(kdeg__rna_degredation_mm + ku__) + kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery*rna_cellular_processes/(kdeg__rna_degredation_mm + ku__) + kb__*ku__*protein_RNAase_machinery*rna_X/(kdeg__rna_degredation_mm + ku__) + kb__*ku__*protein_RNAase_machinery*rna_cellular_processes/(kdeg__rna_degredation_mm + ku__) - kb__*protein_RNAase_machinery*rna_X - kb__*protein_RNAase_machinery*rna_cellular_processes - kb__*protein_RNAase_machinery*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation),
     complex_dna_cellular_processes_protein_RNAP_machinery_*ktx__transcription + complex_dna_cellular_processes_protein_RNAP_machinery_*ku__transcription - dna_cellular_processes*kb__transcription*protein_RNAP_machinery,
     complex_dna_cellular_processes_protein_RNAP_machinery_*ktx__transcription + kb__*ku__*protein_RNAase_machinery*rna_cellular_processes/(kdeg__rna_degredation_mm + ku__) - kb__*protein_RNAase_machinery*rna_cellular_processes - kb__translation*protein_Ribo_machinery*rna_cellular_processes - kdil__*rna_cellular_processes + ktl__translation*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation) + ku__translation*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation),
     -complex_dna_cellular_processes_protein_RNAP_machinery_*ktx__transcription - complex_dna_cellular_processes_protein_RNAP_machinery_*ku__transcription + dna_cellular_processes*kb__transcription*protein_RNAP_machinery,
     -kdil__*protein_cellular_processes + ktl__translation*(kb__*kb__translation*ku__*protein_RNAase_machinery*protein_Ribo_machinery*rna_cellular_processes/(kb__*kdeg__rna_degredation_mm*protein_RNAase_machinery + kdeg__rna_degredation_mm*ktl__translation + kdeg__rna_degredation_mm*ku__translation + ktl__translation*ku__ + ku__*ku__translation) + kb__translation*protein_Ribo_machinery*rna_cellular_processes)/(kb__*protein_RNAase_machinery + ktl__translation + ku__translation)]


# Data Analysis Search Results

```python
%load_ext autoreload
%autoreload 2
```

```python
import os
import re
import glob
import time
from collections import Counter
import json
from multiprocessing import Pool

from tqdm import tqdm
from lxml import html
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.colors import LinearSegmentedColormap
from matplotlib.patches import Patch
from matplotlib.lines import Line2D
import seaborn as sns
import numpy as np
from sklearn.utils import resample

from utils import value_counts, calculate_table
```

```python
# inputs
data_dir = '../data/output/datasets/'
fn_search = f'{data_dir}/searches_check.csv.xz'
fn_products = f'{data_dir}/products.csv.xz'
fn_queries = '../data/input/combined_queries_with_source.csv'
fn_amazon = f'{data_dir}/amazon_private_label.csv.xz'

# intermediary
fn_compare = f'{data_dir}/quality_and_sales_comparisons.csv'

# outputs
output_dir = '../data/output'
table1 = f'{output_dir}/tables/table1-not_always_labelled.csv'
fig2 = f'{output_dir}/figures/fig2-scatter.png'
fig2_csv = f'{output_dir}/figures/fig2-scatter.csv'
fig3a = f'{output_dir}/figures/fig3a-heatmap_amzn.png'
fig3b = f'{output_dir}/figures/fig3b-heatmap-unaffilated.png'
fig3c = f'{output_dir}/figures/fig3c-heatmap-sponsored.png'
fig3a_csv = f'{output_dir}/figures/fig3a-heatmap_amzn.csv'
fig3b_csv = f'{output_dir}/figures/fig3b-heatmap-unaffilated.csv'
fig3c_csv = f'{output_dir}/figures/fig3c-heatmap-sponsored.csv'
table3 = f'{output_dir}/tables/table3-appendix-generic.csv'
for subdir in ['figures', 'tables']:
    os.makedirs(f'{output_dir}/{subdir}', exist_ok=True)
```

```python
df_amazon = pd.read_csv(fn_amazon, compression='xz')
df_products = pd.read_csv(fn_products, compression='xz')
df_search = pd.read_csv(fn_search, compression='xz')
keywords = pd.read_csv(fn_queries)
```

```python
keywords.source.value_counts()
```

    autocomplete amazon     7696
    autocomplete walmart    3806
    seller                  2700
    generic                 1600
    Name: source, dtype: int64

```python
amazon_asin = set(df_amazon.asin.tolist())
len(amazon_asin)
```

    137428

## merge data

```python
_df_search = df_search[df_search.product_type.isin(['regular_placement'])]

searches_with_amazon_brands = _df_search[
    _df_search.asin.isin(amazon_asin)
].search_term.unique()
len(searches_with_amazon_brands)
```

    4567

```python
_df_search = _df_search[_df_search.search_term.isin(searches_with_amazon_brands)].merge(df_products, how='left')
_df_search['is_amazon'] = _df_search.asin.isin(amazon_asin)
```

```python
_df_search.iloc[0]
```

    asin                                                              B086VR2KY8
    product_name               All-New Toshiba 50LF621U21 50-inch Smart 4K UH...
    stars                                                                    4.6
    reviews                                                                 5282
    brand                                                                    NaN
    product_url                /gp/slredirect/picassoRedirect.html/ref=pa_sp_...
    is_prime                                                                True
    is_fresh                                                               False
    is_sponsored                                                           False
    is_featured_brand                                                       True
    is_amazons_choice                                                      False
    is_best_seller                                                         False
    has_sponsored_metadata                                                  True
    product_order                                                              1
    search_term                                                 50 inch smart tv
    product_type                                               regular_placement
    price                                                        $319.99 $379.99
    price_std                                                                NaN
    product_img                https://m.media-amazon.com/images/I/61XJCCT1mI...
    filename                   ../data/input/search-selenium/5/50/50 inch sma...
    fn                         ../data/input/selenium-products/6/6V/B086VR2KY...
    title                      All-New Toshiba 50LF621U21 50-inch Smart 4K UH...
    shipped_by                                                          Best Buy
    sold_by                                                             Best Buy
    has_third_party_sellers                                                False
    product_by_amazon                                                      False
    our_brands_carousel                                                    False
    ads                                                                       []
    no_buybox_winner                                                       False
    is_out_of_stock                                                        False
    is_page_gone                                                           False
    suggestions                ['B086VR2KY8', 'B086VR2KY8', 'B086VR2KY8', 'B0...
    n_ads                                                                      0
    is_amazon                                                               True
    is_sold_by_amazon                                                      False
    is_shipped_by_amazon                                                   False
    Name: 0, dtype: object

```python
_df_search["is_shipped_by_amazon_TRUE"] = _df_search.apply(lambda x: any([x['is_shipped_by_amazon'],
                                x['is_fresh'], x['is_prime']]), axis=1)
```

```python
kw_generic = keywords[keywords['source'] == 'generic'].search_term
kw_seller = keywords[keywords['source'] == 'seller'].search_term
kw_autocomplete = keywords[keywords['source'].isin(
    ['autocomplete amazon', 'autocomplete walmart']
)].search_term
```

```python
top_searches = set(kw_autocomplete.tolist() + kw_seller.tolist())
len(top_searches)
```

    11342

```python
df_top = _df_search[
    _df_search.search_term.isin(top_searches)
]
len(df_top), df_top.search_term.nunique()
```

    (187534, 3492)

```python
# how many searches had Amazon?
df_top.search_term.nunique() / len(top_searches)
```

    0.30788220772350555

```python
# how many products?
df_top.asin.nunique()
```

    125769

```python
# this is our main dataset
df = df_top
df.search_term.nunique()
```

    3492

```python
# this is the supplementary dataset
df_generic = _df_search[
   _df_search.search_term.isin(kw_generic)
]
len(df_generic), df_generic.search_term.nunique()
```

    (69287, 1217)

```python
df = df[~(df.is_sponsored.isnull()) &
   ~(df.is_sold_by_amazon.isnull()) &
   ~(df.is_amazon.isnull())]
```

```python
len(df)
```

    187534

```python
organic= df[df.is_sponsored != True]
```

```python
vals = {
    'all results': len(df),
    'sponsored' : len(df[df.is_sponsored == True]),
    'organic': len(organic),
    'sold by third party': len(organic[organic.is_sold_by_amazon != True]),
    'sold by amazon': len( organic[organic.is_sold_by_amazon == True] ),
    'amazon brand': len(organic[organic.asin.isin(amazon_asin)]),
    'competing brand' : len(organic[~organic.asin.isin(amazon_asin)])
}
```

## Produce plots

Our analysis centered around comparing these two metrics:

\begin{align}
{\# 1\ spot}_{Amazon} = \frac{N_{Amazon\ \#1\ slots}}{N_{\#1\ slots}}
\end{align}

\begin{align}
products_{Amazon} = \frac{N_{Amazon\ product\ slots}}{N_{product\ slots}}
\end{align}

```python
res = calculate_table(df, amazon_asin)
res
```

      Category
      Perc Products
      Perc #1 spot
      Perc first row

      0
      Unaffiliated
      5.808547
      4.209622
      13.917526

      1
      Amazon brands
      5.801081
      19.530355
      24.770905

      2
      Competing brands
      76.880992
      20.819015
      63.516609

      3
      Sponsored
      17.317926
      59.650630
      75.286369

      4
      Amazon sold
      27.093220
      22.451317
      47.079038

      5
      Third-party sold
      55.588853
      17.898053
      48.224513

      6
      Amazon shipped
      76.890591
      38.459336
      72.565865

      7
      Third-party shipped
      5.791483
      1.890034
      8.161512

```python
cat2color = {
    'Amazon brands' : "#ff9900",
    'Competing brands' : "#146eb4",
    'Sponsored': '#DB4437',
    'Organic': '#ee7e6c',
    'Amazon sold': '#146eb4',
    'Third-party sold': '#aabedf',
    'Unaffiliated': 'black',
}

mkr_dict = {
    'Unaffiliated': {
        'marker' : 'v',
        'color': ''
    },
    'Competing brands': {
        'marker' : 'o',
        'color' : ''
    },
    'Amazon sold' : {
        'marker': '+',
        'color' : ''
    },
    'Third-party sold': {
        'marker': '+',
        'color' : ''
    },
    'Amazon brands': {
        'marker': 'o',
        'color' : ''
    },
    'Sponsored' : {
        'marker' : '*',
        'color' : ''
    },
}
```

```python
cats = [
    'Sponsored',
    'Amazon brands',
    'Competing brands',
    'Unaffiliated'
]
plt.figure(figsize=(8, 6))
plt.title("What kind of products get the top search result on Amazon?")
ax = plt.gca()

to_plot = res[res.Category.isin(cats)]
to_plot.to_csv(fig2_csv)

# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')

ax.set_ylabel('Percentage of top spots')
ax.set_xlabel('Percentage of products')

ax.set_ylim(0, 75)
ax.set_xlim(0, 80)
ax.yaxis.get_major_ticks()[0].label1.set_visible(False)

for kind in cats:
    d = to_plot[to_plot.Category==kind]
    if kind in ['Unaffiliated']:
        ax.annotate(kind.title(), (d['Perc Products'] + 1.5,
                       d['Perc #1 spot'] - 1.5))
    elif kind == 'Competing brands':
        ax.annotate(kind.title(), (d['Perc Products'] - 17.5,
                       d['Perc #1 spot'] - 2))
    else:
        ax.annotate(kind.title(), (d['Perc Products'] + 1.6,
                           d['Perc #1 spot']))
    for j in range(1,6):
        plt.scatter(x=d['Perc Products'],
                    y=d['Perc #1 spot'],
                    c=cat2color[kind],
                    s=55,
                    alpha=.2 * j,
                    label=kind,
                    marker = mkr_dict[kind]['marker'])

ax.plot([0, 100], [0, 100],  '--', alpha=.4, color='grey')
ax.annotate("overrepresented",
            (52, 55), rotation=38)
ax.annotate("underrepresented",
            (54.5, 51), rotation=38)

plt.savefig(fig2);
```

## Heatmap

```python
def heatmap(df, control, cmap, title, vmin=0, vmax=100, fn_out=None):
    """Produces a heatmap of product slots"""
    amzn_label = (df
        .product_order.value_counts()
        .sort_index()
        .reindex(pd.RangeIndex(1, 61))[::-1])
    amzn_label = amzn_label / control * 100
    if fn_out:
        amzn_label.to_csv(fn_out)

    amzn_label_grid = amzn_label.iloc[::-1].values.reshape(15, 4)
    amzn_label_grid = pd.DataFrame(amzn_label_grid)
    amzn_label_grid.index = [_ + 1 for _ in amzn_label_grid.index]
    amzn_label_grid.columns = [_ + 1 for _ in amzn_label_grid.columns]

    fig = plt.figure(figsize=(6,12))
    sns.heatmap(amzn_label_grid,
                linewidths=4,
                fmt='g',
                cmap=cmap,
                vmin=vmax,
                vmax=vmin)
    plt.title(title)
    plt.ylabel("⟵ Rows of products going down the page")
    plt.xlabel("Columns of products across the page")
    plt.yticks(rotation=0)

    return fig
```

```python
n_samples = df.search_term.nunique()
```

```python
df_s = df[df.is_sponsored == True]
df_ns = df[df.is_sponsored == False]

df_non_amazon = df_ns[(~df_ns.asin.isin(amazon_asin))]
df_amazon_brand_ = df_ns[df_ns.asin.isin(amazon_asin)]

df_non_amazon_ = df_ns[
    (~df_ns.asin.isin(amazon_asin))
  & (df_ns.is_sold_by_amazon != True)
  & (df_ns.is_shipped_by_amazon != True)
  & (df_ns.is_prime != True)
  & (df_ns.is_fresh != True)
]
```

```python
control = (df.product_order.value_counts()
    .sort_index()
    .reindex(pd.RangeIndex(1, 61))[::-1])
```

```python
cmap = LinearSegmentedColormap.from_list(
    'mycmap', ['#f9f9f9', "#ff9900"]
)
fig = heatmap(
    df_amazon_brand_, control,
    cmap = LinearSegmentedColormap.from_list(None, ['#f9f9f9', "#ff9900"]),
    title=f"Where are Amazon Brands Placed\nDown and Across the Search Page? (N={n_samples:,})",
    vmax=20, vmin=2, fn_out=fig3a_csv
)
fig.savefig(fig3a);
```

```python
fig = heatmap(
    df_non_amazon, control,
    cmap = LinearSegmentedColormap.from_list(None, ['#f9f9f9', 'black']),
    title=f"Where are Competing Brands Placed\nDown and Across the Search Page? (N={n_samples:,})",
)
```

```python
fig = heatmap(
    df_non_amazon_, control,
    cmap = LinearSegmentedColormap.from_list(None, ['#f9f9f9', 'black']),
    title=f"Where are Entirely Unafilliated Products Placed\nDown and Across the Search Page? (N={n_samples:,})",
    vmin=2, vmax=20, fn_out=fig3b_csv
)
fig.savefig(fig3b)
```

```python
fig = heatmap(
    df_s, control,
    cmap=LinearSegmentedColormap.from_list(None, ['#f9f9f9', '#DB4437']),
    title=f"Where are Sponsored Products Placed\nDown and Across the Search Page? (N={n_samples:,})",
    fn_out=fig3c_csv
)
fig.savefig(fig3c)
```

## Not Clearly Labelled

```python
amazon_missed_all = '|'.join([
    'amazon basics',
    'amazonbasics',
    'amazon essentials',
    'whole foods market',
    'amazon commercial',
    'amazon collection',
    'amazon fresh',
    'amazon elements',
    '365 everyday value',
    'amazon exclusive',
    'amazon us exclusive',
    'amazon brand'
])

amazon_missed = '|'.join([
    'amazon basics',
    'amazon essentials',
    'whole foods market',
    'amazon commercial',
    'amazon collection',
    'amazon fresh',
    'amazon elements',
    'amazonbasics',
    '365 everyday value',
])

amazon_disclaimer = '|'.join([
    'amazon exclusive',
    'amazon us exclusive',
    'amazon brand'
])
```

```python
# How many products contain a disclaimer
disclaimer = (df[
  ((df.is_featured_brand) |
   (df.product_name.str.contains(amazon_disclaimer, case=False))) &
  (df.asin.isin(amazon_asin))
].asin.nunique() /
df[df.asin.isin(amazon_asin)].asin.nunique())

disclaimer
```

    0.4204641707606302

```python
# how many items contain a name with known brand
known_brand = (df[
  (df.product_name.str.contains(amazon_missed, case=False)) &
  (df.asin.isin(amazon_asin))
].asin.nunique() /
df[df.asin.isin(amazon_asin)].asin.nunique())

known_brand
```

    0.2879891580552262

```python
# How many do not contain a disclaimer or well known brand?
not_labelled = 1 - (df[
  ((df.product_name.str.contains(amazon_missed_all, case=False)) |
   (df.is_featured_brand)) &
  (df.asin.isin(amazon_asin))
].asin.nunique() /
df[df.asin.isin(amazon_asin)].asin.nunique())

not_labelled
```

    0.37438590547179396

```python
t1 = pd.DataFrame([
    {"" : "Labelled with a Disclaimer", "percentage": disclaimer * 100},
    {"" : "Well-Known Amazon private label", "percentage": known_brand * 100},
    {"" : "Total Labelled", "percentage": (1-not_labelled) * 100},
    {"" : "Total Unlabelled", "percentage": not_labelled * 100},
])
t1 = t1.round(1)
t1.to_csv(table1, index=False)
t1
```

      percentage

      0
      Labelled with a Disclaimer
      42.0

      1
      Well-Known Amazon private label
      28.8

      2
      Total Labelled
      62.6

      3
      Total Unlabelled
      37.4

```python
# perc of top amazon products with a "featured from our brands" tag
(df[
    (df.is_featured_brand) &
    (df.asin.isin(amazon_asin)) &
    (df.product_order == 1)
].search_term.nunique()) / df[(df.asin.isin(amazon_asin)) & (df.product_order == 1)].search_term.nunique()
```

    0.8387978142076503

```python
# perc of top "featured from our brands" products with "sponsored" metadata
(df[
    (df.is_featured_brand) &
    (df.asin.isin(amazon_asin)) &
    (df.product_order == 1) &
    (df.has_sponsored_metadata == True)
].search_term.nunique()) / \
(df[
    (df.is_featured_brand) &
    (df.asin.isin(amazon_asin)) &
    (df.product_order == 1)
].search_term.nunique())
```

    1.0

## More Stars and reviews?

```python
if not os.path.exists(fn_compare):
    data = []
    for st, _df in tqdm(df[df.is_sponsored == False].groupby("search_term")):
        _df.sort_values(by='product_order')
        amzn = _df[_df.asin.isin(amazon_asin)]
        non_amzn = _df[~_df.asin.isin(amazon_asin)]
        wnon_amzn =_df[
            (~_df.asin.isin(amazon_asin)) &
            (_df.is_sold_by_amazon == False) &
            (_df.is_fresh == False) &
            (_df.is_prime == False)
        ]

        if not amzn.empty:
            top_amazon =  amzn.product_order.iloc[0]
            amazon_star = amzn.stars.iloc[0]
            amazon_review = amzn.reviews.iloc[0]
        else:
            top_amazon = np.nan
            amazon_star = np.nan
            amazon_review = np.nan

        if not non_amzn.empty:
            top_non_amazon_brand = non_amzn.product_order.iloc[0]
            non_amazon_star = non_amzn.stars.iloc[0]
            non_amazon_review = non_amzn.reviews.iloc[0]
        else:
            top_non_amazon = np.nan
            non_amazon_star = np.nan
            non_amazon_review = np.nan

        if not wnon_amzn.empty:
            top_wnon_amazon = wnon_amzn.product_order.iloc[0]
            wnon_amazon_star = wnon_amzn.stars.iloc[0]
            wnon_amazon_review = wnon_amzn.reviews.iloc[0]

        else:
            top_wnon_amazon = np.nan
            wnon_amazon_star = np.nan
            wnon_amazon_review = np.nan
        record = {
            "search_term": st,
            "position_first_amazon": top_amazon,
            "position_first_non_amazon": top_non_amazon_brand,
            "position_first_wholly_non_amazon": top_wnon_amazon,

            "amazon_stars": amazon_star,
            "amazon_reviews": amazon_review,

            "non_amazon_stars": non_amazon_star,
            "non_amazon_reviews": non_amazon_review,

            "wnon_amazon_stars": wnon_amazon_star,
            "wnon_amazon_reviews": wnon_amazon_review,
        }
        data.append(record)
    df_compare = pd.DataFrame(data)
    df_compare.to_csv(fn_compare, index=False)

else:
    df_compare = pd.read_csv(fn_compare)
```

```python
# how often is Amazon placed before non-Amazon? Should be close to one-in-five
len(df_compare[df_compare.position_first_amazon < df_compare.position_first_non_amazon]) / len(df_compare)
```

    0.22880870561282932

```python
# how many Amazon products have more stars?
df_compare[
    df_compare.position_first_amazon < df_compare.position_first_non_amazon
].apply(
    lambda x:
    x['amazon_stars'] > x['non_amazon_stars'],
    axis=1
).value_counts(normalize=True)
```

    False    0.653317
    True     0.346683
    dtype: float64

```python
# how many Amazon products have more reivews?
df_compare[
    (df_compare.position_first_amazon < df_compare.position_first_non_amazon) &
    (~df_compare.non_amazon_reviews.isnull())
].apply(
    lambda x:
    x['amazon_reviews'] > x['non_amazon_reviews'],
    axis=1
).value_counts(normalize=True)
```

    False    0.517241
    True     0.482759
    dtype: float64

```python
# how many Amazon products have more reivews and stars?
df_compare[
    (df_compare.position_first_amazon < df_compare.position_first_non_amazon) &
    (~df_compare.non_amazon_reviews.isnull())
].apply(
    lambda x:
    (x['amazon_reviews'] < x['non_amazon_reviews']) and
    (x['amazon_stars'] < x['non_amazon_stars']),
    axis=1
).value_counts(normalize=True)
```

    False    0.720307
    True     0.279693
    dtype: float64

```python
# duplicates
asins_by_search = df[df.is_sponsored == False].groupby('search_term').asin.value_counts()
```

```python
# perc_searches_dupluicate_asin
asins_by_search[asins_by_search > 1].index.get_level_values(0).nunique() / df.search_term.nunique()
```

    0.09221076746849943

```python
# how many of these duplicates are Amazon products?
duplicate_asin = asins_by_search[asins_by_search > 1].index.get_level_values(1)
len([_ for _ in duplicate_asin if _ in amazon_asin]) / len(duplicate_asin)
```

    1.0

## Supplementary dataset

```python
df_generic.search_term.nunique()
```

    1217

```python
table_generic = calculate_table(df_generic, amazon_asin)
table_generic
```

      Category
      Perc Products
      Perc #1 spot
      Perc first row

      0
      Unaffiliated
      2.987573
      1.479047
      7.723911

      1
      Amazon brands
      8.225208
      24.897288
      35.332786

      2
      Competing brands
      73.530388
      10.764174
      54.478225

      3
      Sponsored
      18.244404
      64.338537
      84.223500

      4
      Amazon sold
      26.494148
      23.336072
      46.507806

      5
      Third-party sold
      55.261449
      12.325390
      46.918652

      6
      Amazon shipped
      78.762250
      34.921939
      70.172555

      7
      Third-party shipped
      2.993347
      0.739523
      3.779786

```python
table_top = calculate_table(df_top, amazon_asin)
table_top
```

      Category
      Perc Products
      Perc #1 spot
      Perc first row

      0
      Unaffiliated
      5.808547
      4.209622
      13.917526

      1
      Amazon brands
      5.801081
      19.530355
      24.770905

      2
      Competing brands
      76.880992
      20.819015
      63.516609

      3
      Sponsored
      17.317926
      59.650630
      75.286369

      4
      Amazon sold
      27.093220
      22.451317
      47.079038

      5
      Third-party sold
      55.588853
      17.898053
      48.224513

      6
      Amazon shipped
      76.890591
      38.459336
      72.565865

      7
      Third-party shipped
      5.791483
      1.890034
      8.161512

```python
df[df.is_amazon == True].asin.nunique() / df[(df.product_order == 1) & (df.is_amazon == True)].search_term.nunique()
```

    8.064207650273223

```python
df[df.is_amazon != True].asin.nunique() / df[(df.product_order == 1) & (df.is_amazon != True)].search_term.nunique()
```

    43.42971014492753

```python
df[(df.product_order == 1) & (df.is_amazon == True)].search_term.nunique()
```

    732

```python
table_diff = table_generic.set_index('Category').subtract(
    table_top.set_index('Category'))
```

```python
comparison = pd.concat([
    table_top[['Category', 'Perc Products',  'Perc #1 spot']].set_index('Category'),
    table_generic[['Category','Perc Products',  'Perc #1 spot']].set_index('Category'),
], axis=1, keys=["top", "generic"])
```

```python
comparison
```

      top
      generic

      Perc Products
      Perc #1 spot
      Perc Products
      Perc #1 spot

      Category

      Unaffiliated
      5.808547
      4.209622
      2.987573
      1.479047

      Amazon brands
      5.801081
      19.530355
      8.225208
      24.897288

      Competing brands
      76.880992
      20.819015
      73.530388
      10.764174

      Sponsored
      17.317926
      59.650630
      18.244404
      64.338537

      Amazon sold
      27.093220
      22.451317
      26.494148
      23.336072

      Third-party sold
      55.588853
      17.898053
      55.261449
      12.325390

      Amazon shipped
      76.890591
      38.459336
      78.762250
      34.921939

      Third-party shipped
      5.791483
      1.890034
      2.993347
      0.739523

```python
# note some manual changes are made to this CSV file before it went into our article.
comparison.round(1).to_csv(table3)
```

## Anecdotes

```python
# how many Amazon products have more reivews and stars
df_compare["better_stars_and_reviews"] = df_compare.apply(
    lambda x:
    (x['amazon_reviews'] < x['non_amazon_reviews']) and
    (x['amazon_stars'] < x['non_amazon_stars']),
    axis=1
)
```

```python
df_compare[
    (df_compare.position_first_amazon < df_compare.position_first_non_amazon) &
    (~df_compare.non_amazon_reviews.isnull()) &
    (df_compare["better_stars_and_reviews"] == True) &
    (df_compare['position_first_amazon'] == 1)
].search_term.to_csv('amazon_winner_searches.csv', index=False)
```

## How many more Sponsored products would we have if we counted carousels?

```python
df_carousels = df_search[~df_search.product_type.isin(['regular_placement'])]
df_carousels_ = df_carousels[df_carousels.search_term.isin(top_searches)]
```

```python
df_carousels_.product_type.value_counts()
```

    sponsored_banner             22923
    editorial_recs_carousel      22797
    featured_brands_carousel      8546
    regular_placement__missed     4907
    misc_carousel                 1859
    misc_sponsored_carousel       1284
    Name: product_type, dtype: int64

```python
sponsored_modules = [
    'sponsored_banner',
    'editorial_recs_carousel',
    'misc_carousel',
    'misc_sponsored_carousel'
]

numerator = len(df_top[df_top.is_sponsored == True]) + len(df_carousels_[df_carousels_.product_type.isin(sponsored_modules) ])
denominator = len(df_top) + len(df_carousels_[df_carousels_.product_type != 'regular_placement__missed'])
numerator / denominator
```

    0.31910884411095186


```python
from IPython.core.display import HTML
```

# Will's Lab-Books

## Will Boulton wboulton12@gmail.com

### 20 Sept. 2019

## Blurb

These are a collection of notebooks mainly for my own documentation, as well as resources and also teaching.
The aim will ultimately be to put them into a format helpful for others.

## Contents

1) [How to set up a notebook like this one](https://www.digitalocean.com/community/tutorials/how-to-install-run-connect-to-jupyter-notebook-on-remote-server)

2) More instructions on setting up Digital Ocean, and how to sync with Github and Google Drive

3) How to spin up more resource in Digital Ocean, and how to set up a database

4) A look at GCP, AirFlow, DataFlow, Apache Spark, Apache Beam, Kubernetes, Docker, Hadoop

5) Setting up Java environments with Java and Maven and a Jenkins build system

6) Running simulations and models in Biology, Physics and Chemistry etc

7) Running web services

8) Publishing these notes - https://nbviewer.jupyter.org/faq and https://jupyterhub.readthedocs.io/en/latest/ and
https://mybinder.org/

9) Notes about Java and Scala

10) Node.js and React, as well as other frontend (data vis with Plotly)

11) Machine learning in Python - web scraping, image processing, games, theory

12) Reverse Engineering and security

13) My Maths, coalgebra including $ (\Delta, \epsilon) \in C$ material

14) More maths, STEP preparation

15) Even more maths, blog posts

16) Computer science general

**Here is a markdown example of Maxwell's equations.**

\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

```python
%%html
<style>
 html {
    font-family: 'Lucida Sans', 'Lucida Sans Regular', 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva', Verdana, sans-serif;
}

body {
    background-image: url('wm-processed1.jpg');
    padding: 0px;
    margin: 0px;
    border: 0px;
    height: 100%;
}

/* --- Header --- */

.header {
    padding: 60px;
    text-align: center;
    background: #efefef;
    color: black;
    font-size: 30px;
    opacity: 1;
}

/* --- panel area --- */

.panel-background {
    display: flex;
    flex-wrap: wrap;
    height: 100%;
}

.panel-left {
  background-color: #ffffff;
  opacity: 0;
  flex: 10%; /* Set the width of the transparency */
  padding: 0px;
  margin: 0px;
  border: 0px;
}

.panel-main {
    flex: 80%;
    background-color: #efefef;
    padding: 0px;
    margin: 0px;
    border: 0px;
    height: 100%;
}

.panel-right {
    background-color: #ffffff;
    opacity: 0;
    flex: 10%; /* Set the width of the transparency */
    padding: 0px;
    margin: 0px;
    border: 0px;
}

/* --- Navbar --- */

/* Add a black background color to the top navigation */
.topnav {
  position: relative;
  background-color: #efefef;
  color: black;
  overflow: hidden;
}

/* Style the links inside the navigation bar */
.topnav a {
  float: left;
  text-align: center;
  padding: 16px;
  text-decoration: none;
  font-size: 17px;
  color: black;
}

/* Change the color of links on hover */
.topnav a:hover {
  text-decoration: underline;
  color: black;
}

/* Add a color to the active/current link */
.topnav a.active {
  text-decoration: underline;
  color: black;
}

/* Centered section inside the top navigation */
.topnav-centered div {
  float: none;
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
  color: black;
}

/* Right-aligned section inside the top navigation */
.topnav-right {
  float: right;
  color: black;
}

/* Responsive navigation menu - display links on top of each other instead of next to each other (for mobile devices) */
@media screen and (max-width: 600px) {
  .topnav a, .topnav-right {
    float: none;
    display: block;
    color: black;
  }

  .topnav-centered a {
    position: relative;
    top: 0;
    left: 0;
    transform: none;
    color: black;
  }
}

/*

.topnav {
    background-color: #efefef;
    overflow: hidden;
    font-size: 16;
    text-align: center;
}

/* Style the links inside the navigation bar *//*
.topnav a {
  float: left;
  color: black;
  text-align: center;
  padding: 14px 16px;
  text-decoration: none;
  font-size: 17px;
}

.topnav a:hover {
  text-decoration: underline;;
}

/* Add a underline to the currently active link *//*
.topnav a.active {
  text-underline-position: below;
}
*/

.content {
  border:10px;
  margin:10px;
  padding: 0px;
}
</style>
```

<style>
 html {
    font-family: 'Lucida Sans', 'Lucida Sans Regular', 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva', Verdana, sans-serif;
}

body {
    background-image: url('wm-processed1.jpg');
    padding: 0px;
    margin: 0px;
    border: 0px;
    height: 100%;
}

/* --- Header --- */

.header {
    padding: 60px;
    text-align: center;
    background: #efefef;
    color: black;
    font-size: 30px;
    opacity: 1;
}

/* --- panel area --- */

.panel-background {
    display: flex;
    flex-wrap: wrap;
    height: 100%;
}

.panel-left {
  background-color: #ffffff;
  opacity: 0;
  flex: 10%; /* Set the width of the transparency */
  padding: 0px;
  margin: 0px;
  border: 0px;
}

.panel-main {
    flex: 80%;
    background-color: #efefef;
    padding: 0px;
    margin: 0px;
    border: 0px;
    height: 100%;
}

.panel-right {
    background-color: #ffffff;
    opacity: 0;
    flex: 10%; /* Set the width of the transparency */
    padding: 0px;
    margin: 0px;
    border: 0px;
}

/* --- Navbar --- */

/* Add a black background color to the top navigation */
.topnav {
  position: relative;
  background-color: #efefef;
  color: black;
  overflow: hidden;
}

/* Style the links inside the navigation bar */
.topnav a {
  float: left;
  text-align: center;
  padding: 16px;
  text-decoration: none;
  font-size: 17px;
  color: black;
}

/* Change the color of links on hover */
.topnav a:hover {
  text-decoration: underline;
  color: black;
}

/* Add a color to the active/current link */
.topnav a.active {
  text-decoration: underline;
  color: black;
}

/* Centered section inside the top navigation */
.topnav-centered div {
  float: none;
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
  color: black;
}

/* Right-aligned section inside the top navigation */
.topnav-right {
  float: right;
  color: black;
}

/* Responsive navigation menu - display links on top of each other instead of next to each other (for mobile devices) */
@media screen and (max-width: 600px) {
  .topnav a, .topnav-right {
    float: none;
    display: block;
    color: black;
  }

  .topnav-centered a {
    position: relative;
    top: 0;
    left: 0;
    transform: none;
    color: black;
  }
}

/*

.topnav {
    background-color: #efefef;
    overflow: hidden;
    font-size: 16;
    text-align: center;
}

/* Style the links inside the navigation bar *//*
.topnav a {
  float: left;
  color: black;
  text-align: center;
  padding: 14px 16px;
  text-decoration: none;
  font-size: 17px;
}

.topnav a:hover {
  text-decoration: underline;;
}

/* Add a underline to the currently active link *//*
.topnav a.active {
  text-underline-position: below;
}
*/

.content {
  border:10px;
  margin:10px;
  padding: 0px;
}
</style>

# blah blah blah

```python

```