Importing a few useful libraries

```python
import bs4
from bs4 import BeautifulSoup
import requests
import os.path
import re
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import csv
import json
from datetime import datetime
from tqdm import tqdm
from collections import defaultdict
from collections import Counter
from IPython.core.display import HTML
import time
import nltk

# custom libraries
from data_collection import *
from tsv_management import *
from search_engine import *
from custom_search_engine import *

# nltk text preprocessing utilities
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
from nltk.stem import PorterStemmer
from nltk.corpus import wordnet
from nltk.stem import WordNetLemmatizer

# to be executed once
#nltk.download('wordnet')
#nltk.download('stopwords')
#nltk.download('punkt')
#nltk.download('averaged_perceptron_tagger')
#nltk.download('universal_tagset')
```

# 1. Data Collection

We start by retrieving the list of links associated to every anime.

```python
link_retrieval()
```

We retrieved a total of 19128 anime links.

We then we crawl every page in an html file.

```python
total_anime = data_crawling()
```

Because of the large number of requests, the site would stop the access every once in a while; to deal with this problem we set a waiting time between each request (0.1 s) and a 60 seconds pause every time the site returns a captcha page. This allowed us to download all the pages in about 12 hours.

Now we parse from every html file the data we are interested in into a dictionary. In doing so we also checked if some values contain a tab character, and we removed them. Finally we saved every dictionary into a tsv file.

```python
tsv_gen()
```

We wrote a set of useful functions to retrieve the informations about every anime from these tsv files. The details can be found in the library `tsv_management.py`. Here we give some simple examples on how to use them:

```python
titles = column_retrieval('animeTitle')  # returns the value of the input column for every anime

## NOTE: the tsv files are indexed starting from 1
anime_8 = tsv_retrieval(8)  # reads the data in anime_8.tsv and returns it in a dictionary

all_anime = load_all_tsv()   # returns the dictionaries associated to every anime in one list
```

```python
print(titles[7])
print(anime_8['animeTitle'])
print(all_anime[7]['animeTitle'])
```

    Ginga Eiyuu Densetsu
    Ginga Eiyuu Densetsu
    Ginga Eiyuu Densetsu

```python
animeType = column_retrieval('animeType')
Counter(animeType)
```

    Counter({'TV': 5515,
             'OVA': 3933,
             'Movie': 3476,
             'ONA': 2211,
             'Special': 2294,
             'Music': 1648,
             None: 51})

# 2. Search Engine

We start by defining some utilities to preprocess out data:

We will perform the following operations for text preprocessing:
1. Regex tokenization (to tokenize and remove punctuation all at once)
2. Convert every token to lowercase
3. Stopwords removal (according to NLTK stopwords library for the english language)
4. Stemming using Porter stemmer (to both normalize and stem the words)

We wrote a function for this exact purpose:

```python
preprocessing('Sample text\n These words will be tokenized and stemmed\n')
```

    ['sampl', 'text', 'word', 'token', 'stem']

We start by generating our vocabulary of all the words in the anime descriptions which associates every word to an integer:

```python
vocabulary_generation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [00:08<00:00, 2324.62it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [03:42<00:00, 85.89it/s]

The vocabulary is saved in a json file. We will read it from the disk as needed:

```python
vocabulary = vocabulary_retrieval()
```

We can also easly invert the dictionary to find the word corresponding to a certain integer:

```python
integer_to_word = vocabulary_inversion(vocabulary)
print(vocabulary[integer_to_word[7]], integer_to_word[7])
```

    7 reishin

## 2.1. Conjunctive query

Our first search engine will just sarch for all the anime that contain the query in the description.

We start by building our first inverted index using the anime descriptions. Because each word is associated to an integer, and because our inverted index will contain all the words in our vocabulary (from 0 to `len(vocabulary)`), we decided to store the inverted index as a list of lists, in which the $i^{th}$ entry corresponds to the inverted list associated to the word indexed by $i$.

```python
unranked_inverted_index_generation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [02:36<00:00, 122.28it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [03:29<00:00, 91.09it/s]
    Building the inverted index: 100%|██████████| 19128/19128 [00:02<00:00, 9160.42it/s]

Just as the vocabulary, our inverted index is saved in a json file, and we will read it as part of the search engine initialization. Before starting with the actual search, we want to comment some choices that we made during the implementation of the engine:

* We index the anime starting from $1$ (just as the tsv file names), so we decided to assign the index $0$ to a (virtual) document that contain every word in the vocabulary. Because of this choice every sublist of our inverted index list starts with this $0$ index. This choice has been dictated by a convenience in the computation of the intersection: having a universal document makes it so that the intersection between elements of the inverted index will never be empty, and allows us to terminate the algorithm with minimal effort in terms of length checking.

* The other choice we made regards the algorithm that computes the intersection between the elements of the inverted index. As said before, thanks to our choice of defining the universal $0$ document, the intersections will never be empty; this brings some convenience in terms of checking whether the algorithm has finished or not. Our other "unusual" choice regards the fact that we start intersecting the lists starting from their last element and proceeding backwards, so that the universal document will always be our last match.

We are now ready to initialize our search engine. What this does is loading the inverted index and the vocabulary in memory to have them ready for the search.

```python
## YOU NEED TO EXECUTE THIS ONLY ONCE
vocabulary, unranked_inverted_index = unranked_search_engine_initialization()
```

And we are now ready to search for some anime!

```python
unranked_search(vocabulary, unranked_inverted_index)
```

    Input a query: saiyan race

    4 search results found!

      animeTitle
      animeDescription
      Url

      Dragon Ball Z Special 1: Tatta Hitori no Saishuu Kessen
      Bardock, Son Goku's father, is a low-ranking Saiyan soldier who was given the power to see into the future by the last remaining alien on a planet he just destroyed. He witnesses the destruction of his race and must now do his best to stop Frieza's impending massacre.  (Source: ANN)
      https://myanimelist.net/anime/986/Dragon_Ball_Z_Special_1__Tatta_Hitori_no_Saishuu_Kessen

      Dragon Ball Kai
      Five years after the events of Dragon Ball, martial arts expert Gokuu is now a grown man married to his wife Chi-Chi, with a four-year old son named Gohan. While attending a reunion on Turtle Island with his old friends Master Roshi, Krillin, Bulma and others, the festivities are interrupted when a humanoid alien named Raditz not only reveals the truth behind Gokuu's past, but kidnaps Gohan as well.  With Raditz displaying power beyond anything Gokuu has seen before, he is forced to team up with his old nemesis, Piccolo, in order to rescue his son. But when Gokuu and Piccolo reveal the secret of the seven mystical wish-granting Dragon Balls to Raditz, he informs the duo that there is more of his race, the Saiyans, and they won’t pass up an opportunity to seize the power of the Dragon Balls for themselves.  These events begin the saga of Dragon Ball Kai, a story that finds Gokuu and his friends and family constantly defending the galaxy from increasingly more powerful threats. Bizarre, comical, heartwarming and threatening characters come together in a series of battles that push the powers and abilities of Gokuu and his friends beyond anything they have ever experienced.
      https://myanimelist.net/anime/6033/Dragon_Ball_Kai

      Dragon Ball Super: Broly
      Forty-one years ago on Planet Vegeta, home of the infamous Saiyan warrior race, King Vegeta noticed a baby named Broly whose latent power exceeded that of his own son. Believing that Broly's power would one day surpass that of his child, Vegeta, the king sends Broly to the desolate planet Vampa. Broly's father Paragus follows after him, intent on rescuing his son. However, his ship gets damaged, causing the two to spend years trapped on the barren world, unaware of the salvation that would one day come from an unlikely ally.  Years later on Earth, Gokuu Son and Prince Vegeta—believed to be the last survivors of the Saiyan race—are busy training on a remote island. But their sparring is interrupted when the appearance of their old enemy Frieza drives them to search for the last of the wish-granting Dragon Balls on a frozen continent. Once there, Frieza shows off his new allies: Paragus and the now extremely powerful Broly. A legendary battle that shakes the foundation of the world ensues as Gokuu and Vegeta face off against Broly, a warrior without equal whose rage is just waiting to be unleashed.  [Written by MAL Rewrite]
      https://myanimelist.net/anime/36946/Dragon_Ball_Super__Broly

      Dragon Ball Z
      Five years after winning the World Martial Arts tournament, Gokuu is now living a peaceful life with his wife and son. This changes, however, with the arrival of a mysterious enemy named Raditz who presents himself as Gokuu's long-lost brother. He reveals that Gokuu is a warrior from the once powerful but now virtually extinct Saiyan race, whose homeworld was completely annihilated. When he was sent to Earth as a baby, Gokuu's sole purpose was to conquer and destroy the planet; but after suffering amnesia from a head injury, his violent and savage nature changed, and instead was raised as a kind and well-mannered boy, now fighting to protect others.  With his failed attempt at forcibly recruiting Gokuu as an ally, Raditz warns Gokuu's friends of a new threat that's rapidly approaching Earth—one that could plunge Earth into an intergalactic conflict and cause the heavens themselves to shake. A war will be fought over the seven mystical dragon balls, and only the strongest will survive in Dragon Ball Z.  [Written by MAL Rewrite]
      https://myanimelist.net/anime/813/Dragon_Ball_Z

## 2.2. Conjunctive query & Ranking score

We now want to improve the engine by adding a ranking for the search results based on some similarity with the query. For this purpose we will consider every document as a vector in which the elements are the tfIdf scores of every word in the document.

We start by building the new inverted index, which will now consist of a dicionary fo the form:
```
{
term_id_1:[(document1, tfIdf_{term,document1}), (document2, tfIdf_{term,document2}), (document4, tfIdf_{term,document4}), ...],
term_id_2:[(document1, tfIdf_{term,document1}), (document3, tfIdf_{term,document3}), (document5, tfIdf_{term,document5}), (document6, tfIdf_{term,document6}), ...],
...}

```

```python
ranked_inverted_index_generation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [00:07<00:00, 2608.51it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [03:31<00:00, 90.39it/s]
    Ranked inverted index generation: 100%|██████████| 19128/19128 [00:02<00:00, 6879.75it/s]

Just as the vocabulary, our inverted index is saved in a json file, and we will read it as part of the search engine initialization. The intersection algorithm will be a specular version of the previous one, we just adapted it to the new data types. The similarity between the document and the query will be given by

\begin{equation}
\frac{\mathbf q \cdot \mathbf v }{\| \mathbf v \|}
\end{equation}

where $\mathbf q$ and $ \mathbf v$ are the tfidf vectors of the query and the document (respectively). For this purpose, in the previous function we also generated a dictionary of all the tfidf vectors and we saved it on the disk.

We are now ready to initialize our search engine. What this does is loading the inverted index, the vocabulary and the tfidf vector of every document in memory to have them ready for the search.

```python
## YOU NEED TO EXECUTE THIS ONLY ONCE
vocabulary, ranked_inverted_index, tfidf_vectors, total_documents = ranked_search_engine_initialization()
```

And we are now ready to search for some anime!

```python
ranked_search(vocabulary, ranked_inverted_index, tfidf_vectors, total_documents, k = 5)
```

    Input a query: saiyan race

      animeTitle
      animeDescription
      Url
      Similarity

      Dragon Ball Z Special 1: Tatta Hitori no Saishuu Kessen
      Bardock, Son Goku's father, is a low-ranking Saiyan soldier who was given the power to see into the future by the last remaining alien on a planet he just destroyed. He witnesses the destruction of his race and must now do his best to stop Frieza's impending massacre.  (Source: ANN)
      https://myanimelist.net/anime/986/Dragon_Ball_Z_Special_1__Tatta_Hitori_no_Saishuu_Kessen
      0.554850

      Dragon Ball Super: Broly
      Forty-one years ago on Planet Vegeta, home of the infamous Saiyan warrior race, King Vegeta noticed a baby named Broly whose latent power exceeded that of his own son. Believing that Broly's power would one day surpass that of his child, Vegeta, the king sends Broly to the desolate planet Vampa. Broly's father Paragus follows after him, intent on rescuing his son. However, his ship gets damaged, causing the two to spend years trapped on the barren world, unaware of the salvation that would one day come from an unlikely ally.  Years later on Earth, Gokuu Son and Prince Vegeta—believed to be the last survivors of the Saiyan race—are busy training on a remote island. But their sparring is interrupted when the appearance of their old enemy Frieza drives them to search for the last of the wish-granting Dragon Balls on a frozen continent. Once there, Frieza shows off his new allies: Paragus and the now extremely powerful Broly. A legendary battle that shakes the foundation of the world ensues as Gokuu and Vegeta face off against Broly, a warrior without equal whose rage is just waiting to be unleashed.  [Written by MAL Rewrite]
      https://myanimelist.net/anime/36946/Dragon_Ball_Super__Broly
      0.341980

      Dragon Ball Z
      Five years after winning the World Martial Arts tournament, Gokuu is now living a peaceful life with his wife and son. This changes, however, with the arrival of a mysterious enemy named Raditz who presents himself as Gokuu's long-lost brother. He reveals that Gokuu is a warrior from the once powerful but now virtually extinct Saiyan race, whose homeworld was completely annihilated. When he was sent to Earth as a baby, Gokuu's sole purpose was to conquer and destroy the planet; but after suffering amnesia from a head injury, his violent and savage nature changed, and instead was raised as a kind and well-mannered boy, now fighting to protect others.  With his failed attempt at forcibly recruiting Gokuu as an ally, Raditz warns Gokuu's friends of a new threat that's rapidly approaching Earth—one that could plunge Earth into an intergalactic conflict and cause the heavens themselves to shake. A war will be fought over the seven mystical dragon balls, and only the strongest will survive in Dragon Ball Z.  [Written by MAL Rewrite]
      https://myanimelist.net/anime/813/Dragon_Ball_Z
      0.233947

      Dragon Ball Kai
      Five years after the events of Dragon Ball, martial arts expert Gokuu is now a grown man married to his wife Chi-Chi, with a four-year old son named Gohan. While attending a reunion on Turtle Island with his old friends Master Roshi, Krillin, Bulma and others, the festivities are interrupted when a humanoid alien named Raditz not only reveals the truth behind Gokuu's past, but kidnaps Gohan as well.  With Raditz displaying power beyond anything Gokuu has seen before, he is forced to team up with his old nemesis, Piccolo, in order to rescue his son. But when Gokuu and Piccolo reveal the secret of the seven mystical wish-granting Dragon Balls to Raditz, he informs the duo that there is more of his race, the Saiyans, and they won’t pass up an opportunity to seize the power of the Dragon Balls for themselves.  These events begin the saga of Dragon Ball Kai, a story that finds Gokuu and his friends and family constantly defending the galaxy from increasingly more powerful threats. Bizarre, comical, heartwarming and threatening characters come together in a series of battles that push the powers and abilities of Gokuu and his friends beyond anything they have ever experienced.
      https://myanimelist.net/anime/6033/Dragon_Ball_Kai
      0.190423

As we can see, the results are now ranked by this measure of similarity between documents. The rank though, does not seem very satisfying to us, since the first place contains a special of the main anime, which is ranked third. This fact will be of guidance for the next exercise.

# 3. Define a new score!

At this point, since we have so many informations on each anime, we would like to be able to use more of them to achieve better search results. Before explaining our choices we wanto to make a remark: looking at the data we noticed that the informations were not all mutually independent, specifically we noticed that the anime are ranked in decreasing order of Score; we made the same hypotesis about the popularity and the number of members, but it seems only weakly verified (the ordering is not strictly preserved but is of by a small number of members, we think it is because of some delay between the update of the number of members and the update of the popularity in the site).

Having made these two remarks, we decided to only use the informations on the score, the popularity and the type of every anime. The informations about the aired dates seemed hard to use effectively without implementing an explicit filter, and the number of users did not seem useful to our purpose. We thought about using the anime related informations in a page rank fashion but we ended up discarding the idea to not add too much "randomness" to our formulas.

We also decided to treat the title, the characters names, the voices names and the staff names just like the description, so we ended up using 8 of the 15 (not independent) information we retrieved about every anime. Here is a summary of what we did:
1. We built a new vocabuary and inverted index based on the description, characters names, voices names and staff names
2. We assigned a custom popularity score to each anime
3. Given a query, we search for all the documents that have at least one word in common with the query
4. We rank these last results using a custom index that is based on the similarity and on the previously mentioned popularity score

We start with the first point: because most titles are not in english, we thought that the stemmer could return unexpected results and we changed the way we preprocess the documents. What we do is:
1. Regex tokenization (to tokenize and remove punctuation all at once)
2. Convert every token to lowercase
3. Stopwords removal (according to NLTK stopwords library for the english language)
4. Tag each word using the NLTK tagger
5. Lemmatize the couple (word, tag) using WordNet Lemmatizer

Because the lemmatizer is based on an actual database of words, we thought that the japanese titles would have a higher chance of survival than with the stemming. Here is an example:

```python
lemmatization_preprocessing('Sample text\n These words will be tokenized and lemmatized\n The title Ginga Eiyuu Densetsu is left untouched!')
```

    ['sample',
     'text',
     'word',
     'tokenized',
     'lemmatized',
     'title',
     'ginga',
     'eiyuu',
     'densetsu',
     'leave',
     'untouched']

We start, as usual, by building a new vocabulary of words based on all the textual informations that we have. We will save it to the disk and retrieve it as needed.

```python
titles_descriptions_vocabulary_generation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [00:08<00:00, 2311.17it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:06<00:00, 3039.95it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:06<00:00, 3020.09it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:06<00:00, 3099.35it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:06<00:00, 2898.71it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [04:29<00:00, 71.01it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:20<00:00, 947.15it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:30<00:00, 630.71it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:28<00:00, 670.66it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:19<00:00, 959.20it/s]

We wanted the user to be able to choose if he's searching for a specific anime or if he wants to do a broader search, so we had to keep the description separated from the other textual informations. Because of this, we created a tfidf vector for the descriptions, and a tfidf vector for the other informations (title, characters, voices and staff names; we will sometimes call this vector "partial" or just "titles"). Since we keep the tfidfs separated, we built an unranked inverted index (in the exact same way as the first search engine) and we stored it as a list of lists (we still adopt the universal 0 document convention that we established before). We saved these three objects in three distinct json files, to retrieve them when needed.

```python
titles_descriptions_inverted_index_generation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4032.32it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4466.58it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4560.52it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4531.45it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4231.51it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [03:47<00:00, 84.26it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:21<00:00, 904.74it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:31<00:00, 614.76it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:29<00:00, 649.66it/s]
    Preprocessing documents: 100%|██████████| 19128/19128 [00:20<00:00, 924.47it/s]
    Building the inverted index: 100%|██████████| 19128/19128 [00:02<00:00, 7698.43it/s]
    TfIdf vectors generation: 100%|██████████| 19128/19128 [00:02<00:00, 6712.85it/s]

We the proceeded to retrieve the other information that we wanted to use about the anime from the disk, specifically we retrieve for each anime:
* The Score
* The rank of Popularity
* The Type

Since Score (float in $[2,10]$) and rank of Popularity (integer in $[1,19128]$) are on totally different scales, we decided to normalize them in the interval $(0,1)$ in the following way:
* $newScore\ =\ f(Score,\ 10)$
* $newPopularity\ =\ f(19129-Popularity,\ 19128)$
where $f(x,m)$ is a function that squeezes (with respect to $x$) the interval $(0,m)$ in the interval $(0,1)$ in a nonlinear way near the borders (it gives the same weight to higher scores). We plot $f(x,1)$ below, see the implementation in the library `custom_search_engine.py`, in the function `anime_scoring_info_creation()` for the actual functional form of $f$.

```python
np.seterr(all="ignore")
x = np.linspace(0,1,1000)
plt.plot(x, bump_function(x, max = 1))
plt.show()
```

Once we obtain these new two scores, we save them to the disk along with the anime type:

```python
anime_scoring_info_creation()
```

    Retrieving tsv: 100%|██████████| 19128/19128 [00:06<00:00, 2859.34it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:04<00:00, 4199.38it/s]
    Retrieving tsv: 100%|██████████| 19128/19128 [00:03<00:00, 4787.86it/s]

Before starting the search, we give some more details about the actual implementation and the rankings. Once we retrieve all the documents that have at least one word in common with the query (see `union_pointers()` in `custom_search_engine.py` for the actual implementation of the union algorithm), we proceed to calculate the tfidf vector associated with the document in the following way:

$$tfidf_{document}(word) = \alpha * tfidf_{title}(word)+(1-\alpha) * tfidf_{description}(word)$$

where $tfidf_{title}(word)$ is the tfidf of a word seen as part of the title (and characters, etc..) document and $tfidf_{description}(word)$ is the tfidf of a word seen as part of the description document (where a tfidf is 0 if the word isn't in the document), and $\alpha$ is a parameter chosen by the user.

We then compute the cosine similarity (here we take into account the norm of the query too, because we want a result smaller than 1) between the query and this new tfidf vector. Having run some trials, we noticed that the similarity gave a lot of priority to documents with brief descriptions (as for these documents the tf comes out higher than for longer documents); the tfidf is better suited to work with large documents (for which the term frequency is more meaningful). To overcome this last problem we renormalize the final similarity with a number that depends on the length of the document; the functional form is an arch of ellipse, we choose the parameters with some trials and errors. You can look at the details in `description_titles_cosine_similarity()` inside `custom_search_engine.py`, here we limit ourselves to a simple plot:

```python
def ellipse (x):
    f = -20 * np.sqrt(1-0.000209*(x-70)**2)+21
    return(f)

x = np.linspace(1,69,1000)
plt.plot(x, ellipse(x))
plt.xlabel('number of words in the document')
plt.ylabel('renormalization factor')
plt.show()
```

After computing this new similarity score, we do two last modification to obtain the final score. Firstly we define for each anime the popularity ranking score as the average of the popularity and the rank score defined above; then we compute:

$$score = similarity * ( 1.5 - \alpha ) + PopularityRankingScore * \alpha$$
where $\alpha$ is another parameter chosen by the user. Finally, we let the user enter a preference for the anime type, and we increase the final score by a fixed percentage if the anime matches the desired type (see the global variable `type_modifier` inside `custom_search_engine.py` for the exact number).

We are now ready to initialize the search engine:

```python
custom_vocabulary, inverted_index, description_tf_idf, partial_tf_idf, anime_scoring_info, parameters_list, total_documents = custom_search_engine_initialization()
```

And here we can customize the engine parameters:

```python
parameters_list = search_engine_customization()
```

    Hi! Here you can tune some search engine parameters to fit your search needings.

    Do you want to have a detailed explanation of the role of every parameter? (Y/n) y

    To compute the final tfidf vector associated to the document we computed two separate tfidf vectors: one for the description and one for the titles, characters, voices and staff; and we combined the tfidf for each word according to the following formula:
     final_tfidf = alpha * title_tfidf + (1 - alpha) * description_tfidf
     where title_tfidf is the tfidf of the word in the titles, characters, voices and staff texts, and description_tfidf is the tfidf of the word in the description text. The parameter we ask you to choose is the alpha in the above formula, and it represents the weight of the titles texts in the final computation for the cosine similarity.
     alpha = 1 will give max priority to words in the query that match a word in titles, characters, voices and staff texts
     alpha = 0 will give max priority to words in the query that match a word in the description
     Thus by varying this parameter you will be able to decide if you search for a specific anime (alpha near 1), or if you want to do a more generic search (alpha near 0).
     NOTE: We will automatically renormalize your score in (0.1, 0.9) as choosing too extreme values can lead to unexpected behaviours.

    Are you searching for a specific anime or you want something generic that fits your query?
     Input a number in [0,1] where 0 is generic and 1 is specific:  0.9

    After computing the cosine similarity based on the parameter above, we convolute the similarity with a specific score that is based on the popularity of the anime according to the following:
     final_score = similarity * ( 1.5 - alpha ) + popularity_ranking_score * alpha
     where the popularity_ranking_score is computed starting from the animePopularity and animeScore in the data in the following way:
     rank_score = f(animeScore, 10)
     popularity_score = f(19129 - animePopularity, 19128)
     where f(x,m) is a certain smooth function that squeezes all the numbers in (0,m) in the interval (0,1) in a nonlinear way near the borders.
     See the implementation in the library custom_search_engine.py, in anime_scoring_info_creation() for more details about the function.
     After obtaining the rank_score and the popularity_score, we compute the final rank as a simple average:
     popularity_ranking_score = (popularity_score + rank_score)/2
     The parameter we ask you to choose is the alpha in the above formula, and it represents the weight of the popularity and the rank of each anime in the classification.
     alpha = 1 will give max priority to popularity and rank, sending up in the final classification the most popular anime
     alpha = 0 will give no priority to popularity and rank, letting the cosine similarity choose the classification
     NOTE: We will automatically renormalize your score in (0.1, 0.9) as choosing too extreme values can lead to unexpected behaviours.
     NOTE: Whatever you choose as the parameter, the final classification WILL be influenced by the rank and popularity of the anime. This is an intended choice and it
     reflects the assumption that anime which are not popular nor ranked high are probably of low quality in general and would not be searched much.
     Thus by varying this parameter you will be able to decide if you prefer to have popular anime (alpha near 1) or high ranked anime (alpha near 0) in your search results.

    Would you like to rank the anime based on their popularity or based on their score?
     Input a number in [0,1] where 0 is rank by popularity and 1 is rank by score:  0.9

    Here we ask you to specify if you prefer a specific type of anime among the following:
     Movie, Music, ONA, OVA, Special, TV
     This is an important choice since most of the anime that are not TV are special or side stories of a TV one.
     This will allow you to filter out some of the secondary story anime or to search specifically for them.
     The ranking boost works in this way: given a final similarity score, if the anime type coincides with the desired one
     the score will be increased by a fixed percentage (see the global variable type_modifier in the library custom_search_engine.py).
     NOTE: if you choose All or if you type an incorrect anime type, we will not take into account at all the type of the anime in the final score.

    Are you searching for a specific anime Type? Choose among the following:
    Movie, Music, ONA, OVA, Special, TV, All
     tv

    The parameter you just chose are passed to the search engine as a list:
     [isSpecific, Type, popularity_relevance]
     With this knowledge you can edit these parameters directly if you wish, and you can also bypass the restriction of the numerical parameters being in (0.1, 0.9).
     This last thing can bring unexpected results, experiment freely with this in mind.

```python
custom_search(custom_vocabulary, inverted_index, description_tf_idf, partial_tf_idf, total_documents, anime_scoring_info, parameters_list, k = 5)
```

    Input a query: saiyan race

      animeTitle
      animeDescription
      animeType
      animeScore
      animePopularity
      Url
      Custom score

      Dragon Ball Z Movie 14: Kami to Kami
      Following the defeat of a great adversary, Gokuu Son and his friends live peaceful lives on Earth. Meanwhile, in space, Beerus the God of Destruction awakens from his long slumber, having dreamed of an entity known as a Super Saiyan God. With the help of his assistant, Whis, Beerus looks for this powerful being, as he wishes to fight a worthy opponent. After discovering that the Saiyan home planet was destroyed, he tracks down the remaining Saiyans on Earth, looking for Gokuu specifically.  Having only heard of the Super Saiyan God in legends, Gokuu and his comrades summon Shen Long the Eternal Dragon, who they find out is afraid of Beerus. After learning the secret of the Super Saiyan God, an intense battle between Gokuu and Beerus commences, the immense power of which puts the Earth in terrible danger.  [Written by MAL Rewrite]
      Movie
      7.39
      987
      https://myanimelist.net/anime/14837/Dragon_Ball_Z_Movie_14__Kami_to_Kami
      0.737447

      Dragon Ball Z Special 1: Tatta Hitori no Saishuu Kessen
      Bardock, Son Goku's father, is a low-ranking Saiyan soldier who was given the power to see into the future by the last remaining alien on a planet he just destroyed. He witnesses the destruction of his race and must now do his best to stop Frieza's impending massacre.  (Source: ANN)
      Special
      7.55
      1785
      https://myanimelist.net/anime/986/Dragon_Ball_Z_Special_1__Tatta_Hitori_no_Saishuu_Kessen
      0.709900

      Dragon Ball Z Movie 08: Moetsukiro!! Nessen, Ressen, Chougekisen
      As Goku investigates the destruction of the Southern Galaxy, Vegeta is taken to be King of the New Planet Vegeta, and to destroy the Legendary Super Saiyan, Broly.  (Source: ANN)
      Movie
      7.32
      1271
      https://myanimelist.net/anime/901/Dragon_Ball_Z_Movie_08__Moetsukiro_Nessen_Ressen_Chougekisen
      0.685352

      Initial D Battle Stage
      There are many street races in the Initial D Series. This OVA has no plot other than to collect all the street races into one stage. The dialog during the original races is left in.
      OVA
      7.56
      3228
      https://myanimelist.net/anime/821/Initial_D_Battle_Stage
      0.672249

      Dragon Ball Z Movie 10: Kiken na Futari! Super Senshi wa Nemurenai
      After his loss to Goku, Broly crash lands and hibernates on earth. After some time, he is awakened by Trunks and Goten, who Broly believes is Kakarott, and goes on a rampage to kill both of them. At the same time, Gohan is on his way to challenge the Legendary Super Saiyan alone.
      Movie
      6.79
      1466
      https://myanimelist.net/anime/903/Dragon_Ball_Z_Movie_10__Kiken_na_Futari_Super_Senshi_wa_Nemurenai
      0.611405

## Search engine comparisons

Here we perform some comparisons between the our custom search engine and the ranked search engine we built before.

As a first example, by searching the top 3 for the query `naruto` we see that the ranked search gives only 2 anime about naruto, the first one and the third one, but neither is the TV series (which is the most popular). On the other hand the custom search, tuned by popularity, gives back the two main series in the first 3 places.

```python
ranked_search(vocabulary, ranked_inverted_index, tfidf_vectors, total_documents, k = 3)
```

    Input a query: naruto

      animeTitle
      animeDescription
      Url
      Similarity

      Naruto: Shippuuden - Shippuu! "Konoha Gakuen" Den
      Naruto school special.  Naruto is a new cool student and when he meets Sasuke they start fighting.
      https://myanimelist.net/anime/4134/Naruto__Shippuuden_-_Shippuu_Konoha_Gakuen_Den
      1.987909

      Juliet
      Based on the manga by U-Jin.  This is a story of two siblings, Anzu and Naruto Nozaki, heirs to the fabulous Nozaki fortune. Naruto appears to be an overbearing sis-con. In fact, he's madly in love with Anzu, because they're not blood-related. He confines her to the family mansion for "security," depriving her of companionship and a real life. At age 16, she starts to rebel. She smuggles in a kitten as a pet (Naruto hates cats) and then escapes in a delivery truck. Naruto catches up to her and realizes that she can't be confined like a bird in a cage. He agrees to let have a normal life, but just then, tragedy strikes   (Source: Collectr)
      https://myanimelist.net/anime/8993/Juliet
      1.241344

      Naruto: Shippuuden Movie 6 - Road to Ninja
      Returning home to Konohagakure, the young ninja celebrate defeating a group of supposed Akatsuki members. Naruto Uzumaki and Sakura Haruno, however, feel differently. Naruto is jealous of his comrades' congratulatory families, wishing for the presence of his own parents. Sakura, on the other hand, is angry at her embarrassing parents, and wishes for no parents at all. The two clash over their opposing ideals, but are faced with a more pressing matter when the masked Madara Uchiha suddenly appears and transports them to an alternate world.  In this world, Sakura's parents are considered heroes—for they gave their lives to protect Konohagakure from the Nine-Tailed Fox attack 10 years ago. Consequently, Naruto's parents, Minato Namikaze and Kushina Uzumaki, are alive and well. Unable to return home or find the masked Madara, Naruto and Sakura stay in this new world and enjoy the changes they have always longed for. All seems well for the two ninja, until an unexpected threat emerges that pushes Naruto and Sakura to not only fight for the Konohagakure of the alternate world, but also to find a way back to their own.  [Written by MAL Rewrite]
      https://myanimelist.net/anime/13667/Naruto__Shippuuden_Movie_6_-_Road_to_Ninja
      1.159498

```python
parameters_list = [0.1, 'all', 0.9]
custom_search(custom_vocabulary, inverted_index, description_tf_idf, partial_tf_idf, total_documents, anime_scoring_info, parameters_list, k = 3)
```

    Input a query: naruto

      animeTitle
      animeDescription
      animeType
      animeScore
      animePopularity
      Url
      Custom score

      Naruto: Shippuuden
      It has been two and a half years since Naruto Uzumaki left Konohagakure, the Hidden Leaf Village, for intense training following events which fueled his desire to be stronger. Now Akatsuki, the mysterious organization of elite rogue ninja, is closing in on their grand plan which may threaten the safety of the entire shinobi world.  Although Naruto is older and sinister events loom on the horizon, he has changed little in personality—still rambunctious and childish—though he is now far more confident and possesses an even greater determination to protect his friends and home. Come whatever may, Naruto will carry on with the fight for what is important to him, even at the expense of his own body, in the continuation of the saga about the boy who wishes to become Hokage.  [Written by MAL Rewrite]
      TV
      8.22
      16
      https://myanimelist.net/anime/1735/Naruto__Shippuuden
      1.157615

      Naruto: Shippuuden Movie 6 - Road to Ninja
      Returning home to Konohagakure, the young ninja celebrate defeating a group of supposed Akatsuki members. Naruto Uzumaki and Sakura Haruno, however, feel differently. Naruto is jealous of his comrades' congratulatory families, wishing for the presence of his own parents. Sakura, on the other hand, is angry at her embarrassing parents, and wishes for no parents at all. The two clash over their opposing ideals, but are faced with a more pressing matter when the masked Madara Uchiha suddenly appears and transports them to an alternate world.  In this world, Sakura's parents are considered heroes—for they gave their lives to protect Konohagakure from the Nine-Tailed Fox attack 10 years ago. Consequently, Naruto's parents, Minato Namikaze and Kushina Uzumaki, are alive and well. Unable to return home or find the masked Madara, Naruto and Sakura stay in this new world and enjoy the changes they have always longed for. All seems well for the two ninja, until an unexpected threat emerges that pushes Naruto and Sakura to not only fight for the Konohagakure of the alternate world, but also to find a way back to their own.  [Written by MAL Rewrite]
      Movie
      7.67
      640
      https://myanimelist.net/anime/13667/Naruto__Shippuuden_Movie_6_-_Road_to_Ninja
      1.154822

      Naruto
      Moments prior to Naruto Uzumaki's birth, a huge demon known as the Kyuubi, the Nine-Tailed Fox, attacked Konohagakure, the Hidden Leaf Village, and wreaked havoc. In order to put an end to the Kyuubi's rampage, the leader of the village, the Fourth Hokage, sacrificed his life and sealed the monstrous beast inside the newborn Naruto.  Now, Naruto is a hyperactive and knuckle-headed ninja still living in Konohagakure. Shunned because of the Kyuubi inside him, Naruto struggles to find his place in the village, while his burning desire to become the Hokage of Konohagakure leads him not only to some great new friends, but also some deadly foes.  [Written by MAL Rewrite]
      TV
      7.95
      8
      https://myanimelist.net/anime/20/Naruto
      1.153523

From now on will omit the actual search to not fill the notebook with wall of text.

We obtain similar result also by choosing as set of parameters: `[0.9, 'all', 0.1]`. In this last case the first ranked anime is Juliet, in which one of the characters is called Naruto as well, but the other two positions are occupied by the two main series.

Setting `[0.1, 'all', 0.9]`, if we search for 'one piece', we retrieve the main serie along with two movies with our search engine, while the ranked search only found the two associated movies.

One of the main advantages in using our search engine is the fact that we account for the title and the anime characters name, while the ranked search does not. This can bring to very different result, since one could wish to search by title.

# 4. BONUS: Understanding the anime's reviews
We start by crawling the reviews pages from the web:

```python
total_reviews = reviews_crawl()
```

And then we parse them and we save them in various txt files:

```python
txt_gen()
```

    100%|██████████| 19128/19128 [42:09<00:00,  7.56it/s]

and then we merge every review in one file for better file management (as it is hard to move around multiple smaller files)

```python
merge_txt()
```

    Retrieving txt: 100%|██████████| 19128/19128 [00:06<00:00, 3019.37it/s]

To perform the sentiment analysis we followed the guide found in https://towardsdatascience.com/sentiment-analysis-in-10-minutes-with-bert-and-hugging-face-294e8a04b671

We used the get_data function to get IMDB Reviews Dataset that contains 25,000 movie reviews for training and 25,000 for testing. All these 50,000 reviews are labeled. we downloaded the dataset from Stanford’s relevant directory `tf.keras.utils.get_file`. After getting the data we created 2 pandas dataset (train and test) from text_dataset_from_directory of tensorflow library. We had to convert the 2 pandas dataframe into objects that are suitable objects for the BERT model. We achieve this convertions using 2 functions `convert_data_to_examples()` and `convert_examples_to_tf_dataset()`. The first one converts every row of pandas dataframe into an InputExample object. The other one creates the required input format with the tokenized InputExample objects and then create an input dataset that we can feed to the model. We used a stochastic gradient descent method from tensorflow (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam), using CategoricalCrossentropy as our loss function, and SparseCategoricalAccuracy. After this we retrieved the reviews, we tokenized them, and than we gave to the pretrained model and we run a final softmax layer to get the predictions, using the argmax to get if a review is positive or negative. We wrote the result as a dict {anime_id:(number of positive,number of negative)} and we saved it in the `review_sentiment_dict.txt`.

## 5. Algorithmic question
You consult for a personal trainer who has a *back-to-back sequence* of requests for appointments. A sequence of requests is of the form
    > 30, 40, 25, 50, 30, 20
where each number is the time that the person who makes the appointment wants to spend.
You need to accept some requests, however you need a break between them, so you cannot accept two consecutive requests. For example, `[30, 50, 20]` is an acceptable solution (of duration *100*), but `[30, 40, 50, 20]` is not, because *30* and *40* are two consecutive appointments. Your goal is to provide to the personal trainer a schedule that maximizes the total length of the accepted appointments. For example, in the previous instance, the optimal solution is `[40, 50, 20]`, of total duration *110*.
1. Write an algorithm that computes the acceptable solution with the longest possible duration.
2. Implement a program that given in input an instance in the form given above, gives the optimal solution.

We start by making some remarks about the optimal solution. Let's suppose that, in a given day, the personal trainer receives $n$ requests $[r_1,r_2,\dots,r_n]$, and let's define $S = [s_1, \dots, s_k]$ as the (ordered) subset of appointments that will form our optimal solution for our problem in the given day.
* An important observation is that, by definition, if $r_i$ is in the optimal solution, then neither $r_{i-1}$ nor $r_{i+1}$ belong to this solution. This is expression of the fact that we do not want to accept two consecutive appointments.
* Another crucial observation that we want to make is that, given any couple of consecutive elements $r_i, r_j$ that are in the optimal solution, their distance is no more than $3$, i. e. $j-i\leq 3$. I other words, given any three consecutive appointments of durations $r_i, r_{i+1}, r_{i+2}$, at least one of them is in the optimal solution. This fact is, in some sense, pretty intuitive, and it comes from the fact that our optimal solution is the longest possible, i.e. the one for which the quantity $\sum s_j$ is maximized. The proof of this fact is trivial and we will just illustrate it with an example. Suppose that the followings are five consecutive appointments requested today: $[r_1,r_2,r_3,r_4,r_5]$ and assume that, among them, just $r_1,r_5$ are in the optimal solution; then, by adding $r_3$ to the solution we find another valid solution, which is longer than the optimal one, which is absurd.
* Finally, from the two observation above, given any three consecutive elements $r_i, r_{i+1}, r_{i+2}$, either one or two of them belong to the optimal solution (three of them is not acceptable by the first remark, none of them is not acceptable by the second remark).

One last note before starting: this problem may have more than one optimal solution, our algorithm returns just one of them but, with a bit fo work, can be generalized to find all the optimal solutions.

### Write an algorithm that computes the acceptable solution with the longest possible duration.
Given our observation, we now describe a possible algorithm to solve this problem, we illustrate an example, and then we give a description of the algorithm in pseudocode.

The core of the algorithm lies in the fact that, given any three consecutive elements $r_i, r_{i+1}, r_{i+2}, r_{i+3}$, if $r_{i+3}$ is in the optimal solution, then either $r_{i}$ or $r{i+1}$ is in the optimal solution as well. The algorithm will scan through each appointment, starting from the first, and will find the best possible solution (up to that point) that contains that appointment by confronting the best possible solution for the two preceeding appointment (not counting the adjacent one) and adding the appointment in question to that. In the end, we consider the two best solutions that contains the last two appointment, and we choose the best of the two as the optimal one.

We now describe the functioning of the algorithm with an example:

1. Consider the sequence of appointments given in the text `[30, 40, 25, 50, 30, 20]`.
2. The best solution that contains $30$ is `[30]`
3. The best solution that contains $40$ is `[40]`
4. The best solution that contains $25$ is `[30, 25]` with a length of $55$
5. There are two candidate solutions that contain $50$: the best solution that contains $30$, and the best solution that contains $40$. From the previous three points we know that these are, respectively, `[30]` and `[40]`; since we want the best possible solution, we choose the second, so the best possible solution that contains $50$ is `[40, 50]` with a length of $90$
6. There are two candidate solutions that contain $30$: the best solution that contains $40$, and the best solution that contains $25$. From the previous three points we know that these are, respectively, `[40]` and `[30, 25]` (which has a length of $55$); since we want the best possible solution, we choose the second, so the best possible solution that contains $30$ is `[30, 25, 30]` with a length of $85$
7. Finally, there are two candidate solutions that contain $20$: the best solution that contains $25$, and the best solution that contains $50$. From the previous three points we know that these are, respectively, `[30, 25]` with a length of $55$ and `[40, 50]` with a length of $90$; since we want the best possible solution, we choose the second, so the best possible solution that contains $20$ is `[40, 50, 20]` with a length of $110$
8. Now we consider just the last two appointments. We just have to compare the best possible solution that contains $30$, and the best possible solution that contains $20$; these are, respectively, `[30, 25, 30]` with a length of $85$ and `[40, 50, 20]` with a length of $110$. Since we want the best possible solution, we choose the second to be our optimal solution.

Finally, here is a description of the algorithm in pseudocode:
```
input: the list of requests
output: the (ordered) list of requests that belong to the optimal solution

function personal_trainer(req) {
    num_req = len(req)

    #manually taking care of the simple cases
    if (num_req <= 1) return(req)
    if (num_req == 2) return( [max(req)] )
    if (num_req == 3) {
        solution1 = req[0] + req[2]
        solution2 = req[1]
        if (solution1 > solution2) return( [req[0], req[2]] )
        else return( [req[2]] )
    }

    # generating all the three possible ways of choosing the first three element for a possible solution
    # we will store a possible solution in an array of length 2 which contains
    # in the first entry: the list of appointments in req that we want to have in the optimal solution
    # in the second entry: the total length of the appointments in the optimal solution, i.e. the quantity that we are trying to maximize

    solution1 = ([req[0]], req[0])
    solution2 = ([req[1]], req[1])
    solution3 = ([req[0], req[2]], req[0]+req[2])

    candidates = [solution1, solution2, solution3]

    for (i in 3 to num_req) {    #3 is included, num_req is excluded
        length_solution1 = candidates[-3][1]
        length_solution2 = candidates[-2][1]

        if (length_solution1 >= length_solution2) {
            appointments = candidates[-3][0] + [req[i]]   #here '+' means list concatenation
            length = candidates[-3][1] + req[i]

            candidates.append( (appointments, length) )
        } else {
            appointments = candidates[-2][0] + [req[i]]   #here '+' means list concatenation
            length = candidates[-2][1] + req[i]

            candidates.append( (appointments, length) )
        }
    }

    length_solution1 = candidates[-2][1]
    length_solution2 = candidates[-1][1]

    if (length_solution1 >= length_solution2) {
        return( candidates[-2][0] )
    } else {
        return( candidates[-1][0] )
    }
}
```

### Implement a program that given in input an instance in the form given above, gives the optimal solution.

```python
def personal_trainer(requests):
    '''
    input: the list of requests
    output: the (ordered) list of requests that belong to the optimal solution
    '''

    def best_solution(candidates):
        '''
        input: the list of the three last candidates solution
        output: the index of the best solution among the first two
        '''

        if (candidates[0][1] >= candidates[1][1]):
            return(0)
        else:
            return(1)

    number_of_requests = len(requests)

    ## manually taking care of the simple cases

    # if the list is empty or has just one element, we return the list itself
    if (number_of_requests <= 1):
        return(requests)

    # if the list has just two elements, we return the biggest of the two
    if (number_of_requests == 2):
        return( [max(requests)] )

    # if the list has just three elements, we return either the second appointment, or the first and the last
    if (number_of_requests == 3):

        solution1 = requests[0] + requests[2]
        solution2 = requests[1]

        if (solution1 > solution2):
            return( [requests[0], requests[2]] )
        else:
            return( [requests[2]] )

    # here we generate the best possible solution that contains a certain element for the first three elements
    # we will store a possible solution in a tuple of length 2 which contains:
    # in the first entry: the list of appointments that we want to have in the optimal solution
    # in the second entry: the total length of the appointments in the optimal solution, i.e. the quantity that we are trying to maximize

    solution1 = ([requests[0]], requests[0])
    solution2 = ([requests[1]], requests[1])
    solution3 = ([requests[0], requests[2]], requests[0]+requests[2])

    # candidates is a list that contains the best possible solution which contain a given three consecutive appointments
    candidates = [solution1, solution2, solution3]

    # here we loop through every appointment and compute the best possible solution that contains it
    for i in range(3, number_of_requests):
        best_solution_idx = best_solution(candidates)

        appointments_list = candidates[best_solution_idx][0] + [requests[i]]    #best possible solution that contains requests[i]
        length = candidates[best_solution_idx][1] + requests[i]    #total length of the best possible solution that contains requests[i]

        # here we add the solution relative to requests[i]
        # and remove the solution relative to requests[i-3]
        candidates.append((appointments_list, length))
        del(candidates[0])

    del(candidates[0])

    best_solution_idx = best_solution(candidates)

    return(candidates[best_solution_idx][0])
```

```python
requests = [30, 40, 25, 50, 30, 20]
optimal_solution = personal_trainer(requests)
```

```python
print('Given the sequence of requests', requests,\
      '\nTo maximize the total length of the accepted appointments without ever accepting two consecutive requests the personal trainer should follow this schedule:', optimal_solution)
```

    Given the sequence of requests [30, 40, 25, 50, 30, 20]
    To maximize the total length of the accepted appointments without ever accepting two consecutive requests the personal trainer should follow this schedule: [40, 50, 20]


```
import sys
sys.stdout.write('hello world\n')
sys.stdout.flush()
for i in range(3):
    sys.stdout.write('%s\n' % i)
    sys.stdout.flush()
sys.stderr.write('output to stderr\n')
sys.stderr.flush()
sys.stdout.write('some more stdout text\n')
sys.stdout.flush()
```

    hello world
    0
    1
    2

    output to stderr

    some more stdout text

# Markdown Cell

$ e^{ \pm i\theta } = \cos \theta \pm i\sin \theta + \beta $

*It* **really** is!

```
this is a syntax error
```

```
print('test')
```

```
from IPython.display import Latex
Latex('''The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\\begin{equation}
E=m
\\end{equation}''')
```

The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\begin{equation}
E=m
\end{equation}

```
from IPython.display import display
import ipywidgets as w
# s = w.IntSlider(0, 10)£
# s
a = w.IntSlider()
b = w.IntText()
w.jslink((a, 'value'), (b, 'value'))
display(a, b)
```

    IntSlider(value=0)

    IntText(value=0)

```
import plotly.express as px
df = px.data.tips()
fig = px.histogram(df, x="total_bill", y="tip", color="sex", marginal="rug",
                    hover_data=df.columns)
fig.show()
```

```
import numpy as np
size = 100
np.random.seed(0)
x_data = np.arange(size)
y_data = np.cumsum(np.random.randn(size) * 100.0)
from bqplot import pyplot as plt
plt.figure(title="My First Plot")
plt.plot(x_data, y_data)
plt.show()
```

    VBox(children=(Figure(axes=[Axis(scale=LinearScale()), Axis(orientation='vertical', scale=LinearScale())], fig…

```
from pyecharts.charts import Bar
from pyecharts import options as opts

bar = (
    Bar()
    .add_xaxis(["衬衫", "毛衣", "领带", "裤子", "风衣", "高跟鞋", "袜子"])
    .add_yaxis("商家A", [114, 55, 27, 101, 125, 27, 105])
    .add_yaxis("商家B", [57, 134, 137, 129, 145, 60, 49])
    .set_global_opts(title_opts=opts.TitleOpts(title="某商场销售情况"))
)
# for jupyterlab
# bar.load_javascript()
bar.render_notebook()
```

```
bar
```

    <pyecharts.charts.basic_charts.bar.Bar at 0x7fad1025f850>

```

```


```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir('../notebook_format')
from formats import load_style
load_style()
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Vollkorn');
@import url('http://fonts.googleapis.com/css?family=Arimo');
@import url('http://fonts.googleapis.com/css?family=Fira_sans');

    div.cell {
        width: 1000px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 600;
        font-size: 12pt;
        font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
    }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.text_cell_render{
        font-family: 'Fira sans', verdana,arial,sans-serif;
        line-height: 125%;
        font-size: 115%;
        text-align:justify;
        text-justify:inter-word;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
    }

    code{
      font-size: 70%;
    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       font-weight: bold;
       color:#447adb;
    }
    a:visited{
       font-weight: bold;
       color: #1d3b84;
    }
    a:hover{
       font-weight: bold;
       color: #1d3b84;
    }
    a:focus{
       font-weight: bold;
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2

from scipy.stats import beta
from collections import namedtuple

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,scipy
```

    Ethen 2017-01-14 23:02:25

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.3
    pandas 0.18.1
    matplotlib 1.5.1
    scipy 0.18.1

# Multi-Armed Bandits

Imagine this scenario: You’re in a casino. There are many different slot machines (known as "one-armed bandits", as they’re known for robbing you), each with a lever (an arm, if you will). You think that some slot machines payout more frequently than others do, and you’re goal is to walk out of the casino with the most money.

The question is, how do you learn which slot machine is the best and get the most money in the shortest amount of time? You could try all the slot machines out to get a sense of the expected return from playing each machine. But remember, each time you play a poor performing machine, you lower your take that you walk out of the casino with that night. In order to maximize how much money you walk out of the casino with, you will have to be efficient with how you collect your data.

Rewriting the scenario above into a business language. Each time a shopper looks comes to a webpage, we  show them one of the $K$ variations of the webpage. They either click on it or do not, and we log this information about the (binary) reward for each $K$ variations. Kext, we proceed to the next shopper or have to choose one of $K$ webpage variations again.

## Differences Between A/B Testing and Bandit Testing

In both scenarios above, we would normally determine our "winner" (the slot machine that pays the most, or the best webpage variations that gets the most clicks) using the well-known **A/B testing** approach. The **A/B testing** approach consists of a period of pure exploration, where you’re randomly assigning equal numbers of users to one of the $K$ variations and run the test until it's valid. After that, it jumps into pure exploitation, where you send 100% of your users to the more successful version of your site.

Two possible problems with the classical **A/B testing** approach is that:

- It jumps discretely from exploration to exploitation, when you might be able to transition more smoothly.
- During the exploratory phase (the test), it wastes resources exploring inferior options in order to gather as much data as possible.

Given the exploration - exploitation dilemma stated above, the **bandit testing** approach try to account for this. The following graph depicts the difference between the two types of testing method:

If we have three variations that we wish to test, the **A/B testing** approach we try out each of the three variations with equal proportions until we run our test at week 5, and then select the variation with the highest value.

As for **bandit testing**, it attempts to use what it knows about each variation from the very beginning, and it continuously updates the probabilities that it will select each variation throughout the optimization process. In the above chart we can see that with each new week, the **bandit testing** reduces how often it selects the lower performing options and increases how often if selects the highest performing option.

> You need to explore in order to figure out what works and what doesn’t. On the other hand, if you exploit you take advantage of what you have learned. The **bandit testing** approach highlights the fact that collecting data also has its cost.

To be specific, **bandit testing** algorithms will try to minimize what’s known as regret, which is the difference between your actual payoff and the payoff you would have collected had you played the optimal (best) options at every opportunity. There are tons of different bandit methods, in the next section we'll look at some of the more common ones.

## Bandit Algorithms

Before introducing the algorithms and trying them out through simulations, we'll denote some notations and terminologies to formally define the problem:

- **arms** is simply the variations that we're testing (webpages that we're testing) and there will be $K$ of them in total.
- In a simulation of t turns (how many samples in a simulation), we'll maintain **empirical means** of the reward for each arm (e.g. after trying out arm A for 10 turns, it got 3 clicks, the empirical means is simply 0.3) that are updated at every turn t.
- $u_i(t)$ is the empirical mean of arm $i$ after $t$ turns.
- $p_i(t)$ is the probability of picking arm $i$ at turn $t$.

Let's look at our simulated data beforing diving into each algorithms (hopefully the docstrings are self-explanatory).

```python
def generate_bernoulli_bandit_data( n_simulations, K ):
    """
    generate simluate data, that represents success / trial data

    Parameters
    ----------
    n_simulations : int
        the total number of turns in a simulation

    K : int
        the total number of arms

    Returns
    -------
    ctr : float 1d-array, shape(K, )
        the randomly generated empirical click through rate for each arm

    rewards : boolean 2d-array, shape(n_simulations, K)
        given the empirical ctr, simulate in each turn of the simulation,
        whether the arm gets pulled will obtain the
        reward or not (whether the webpage gets clicked)
    """

    ctr = np.random.rand(K)
    rewards = np.random.rand(n_simulations, K) < np.tile( ctr, (n_simulations, 1) )
    return ctr, rewards

K = 2
n_simulations = 5
ctr, rewards = generate_bernoulli_bandit_data(n_simulations, K)
print(ctr)
print(rewards)
```

    [ 0.75662543  0.5769374 ]
    [[ True False]
     [ True  True]
     [False False]
     [ True False]
     [ True False]]

### Algorithm 1 - Epsilon Greedy

At each round $t = 1, 2, ...$ the **Epsilon Greedy** algorithm will:

- Choose a random arm with the probability of $\epsilon$.
- Choose the arm with the current best empirical mean with probability of $1-\epsilon$.

In mathematical notations:

$$
p_i(t+1)=
\begin{cases}
    1 - \epsilon + \epsilon \big/ K & \quad \text{if i = } argmax_{j = 1, ..., K} \ u_j(t) \\
    \epsilon \big/ K  & \quad otherwise
\end{cases}
$$

Or more intuitively:

- When a new visitor comes to the site, the algorithm flips a coin that comes up tail with the probability of $\epsilon$. When it does in fact comes up tail, the algorithm is going to explore. The exploration phase is simply randomly choosing between any possible arm with equal (uniform) probability and showing it to the visitor.
- On the other hand, the algorithm will exploit the best known solution with the proability of $1- \epsilon$. To exploit, the algorithm simply looks up the current empirical means and shows the best one to the visitor.

The image below sums up the algorithm pretty well.

```python
def epsilon_greedy(counts, epsilon = 0.5, decrease_const = 1000):
    """
    adaptive epsilon greedy

    Parameters
    ----------
    counts : int 2d-array, shape(K, 2), where K = the total number of arms
        success and failures for each arm where column 0 represents
        success, 1 represents failure

    epsilon : float
        the initial probability of choosing a random arm;
        1 - epsilon is the probability of choosing the current best arm

    decrease_const : int
        parameter for the adaptive (annealing) epsilon

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    totals = counts.sum(axis = 1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    epsilon /= (1 + total_counts / decrease_const)
    if np.random.rand() > epsilon:
        return np.argmax(empirical_means)
    else:
        return np.random.randint(0, empirical_means.shape[0])

# counts : stores the counts of success and failures for each arm
# where column 0 represents success, 1 represents failure.
# each arm's count is initialiated as 1 to ensure that each arm is
# played at least once, to prevent "cold start" problem and
# 0 division in the beginning
counts = np.ones((K, 2))
print(counts)
epsilon_greedy(counts)
```

    [[ 1.  1.]
     [ 1.  1.]]

    0

The `decrease_const` parameter in the function above may look unfamiliar.

For the **Epsilon Greedy** algorithm, setting the $\epsilon$ can be a bit tricky. If it’s too small, exploration will be slow at the beginning, and you will be slow to react to changes. If we happen to sample, say, the second-best arm the first few times, it may take a long time to discover that another arm is actually better. If $\epsilon$ is too big, you’ll waste many trials pulling random arms without gaining much.

To accomodate for this situation, we will set the $\epsilon$ value at a higher value in the beginning and anneal (gradually lower) it over time. Intuitively, this simply means that after exploring around for a while, we become more certained about each arms' empirical means. After that, it's better to exploit.

In the function call above, the $\epsilon$ at turn $t$ will become:

$$\epsilon(t) = \epsilon(0) \Big/ (1 + t/T)$$

Where $T$ is a new parameter that represents a decreasing constant.

Note that there are different ways of annealing a parameter, but the spirit is the same.

```python
# show adaptive learning rate
epsilon = 0.5
decrease_const = 1000

# the epsilon value after 10 turns
total_counts = 10
print( epsilon / (1 + total_counts / decrease_const) )

# after 10000 turns
total_counts = 10000
print( epsilon / (1 + total_counts / decrease_const) )
```

    0.49504950495049505
    0.045454545454545456

### Algorithm 2 - Boltzmann Exploration (Softmax)

The **Softmax** algorithm picks each arm with a probability that is proportional to its average reward.

$$ p_i(t+1)= \frac{ e^{u_i(t) / \tau} }{ \sum_{j=1}^K  e^{u_j(t) / \tau} }$$

Where $\tau$ is a temperature parameter, controlling the randomness of the choice. When $\tau$ = 0, the algorithm acts like pure greedy. As $\tau$ grows to infinity, the algorithm will pick arms uniformly at random.

```python
def softmax(counts):
    """
    adaptive softmax

    Parameters
    ----------
    counts : int 2d-array, shape( K, 2 ), where K = the total number of arms
         success and failures for each arm where column 0 represents
         success, 1 represents failure

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    totals = counts.sum(axis = 1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    # annealing (adaptive learning rate)
    tau = 1 / np.log(total_counts + 0.000001)

    probs_n = np.exp(empirical_means / tau)
    probs_d = probs_n.sum()
    probs = probs_n / probs_d

    cum_prob = 0.
    z = np.random.rand()
    for idx, prob in enumerate(probs):
        cum_prob += prob
        if cum_prob > z:
            return(idx)

counts = np.ones((K, 2))
softmax(counts)
```

    1

### Algorithm 3 - Upper Confidence Bounds (UCB)

In the world of statistics, whenever you estimate some unknown parameter (such as the mean of a distribution) using random samples, there is a way to quantify the uncertainty inherent in your estimate.  For example, the true mean of a fair six-sided die is 3.5.  But if you only roll it once and get a 2, your best estimate of the mean is just 2.  Obviously that estimate is not very good, and we can quantify the confidence we have for our estimate.  There are confidence bounds which can be written, for example, as: "The mean of this die is 2, with a 95-th percentile lower bound of 1.4 and a 95-th percentile upper bound of 5.2."

The upper confidence bound (UCB) family of algorithms, as its name suggests, simply selects the arm with the largest upper confidence bound at each turn.  The intuition is this: the more times you roll the die, the tighter the confidence bounds, on the other hand, if your roll the die an infinite number of times then the width of the confidence bound is zero. In short, as the number of rolls increases, the uncertainty decreases, and so does the width of the confidence bound.

Thus, unlike the **Epsilon Greedy** and **Softmax** algorithm that only keeps track of the empirical means, the **UCB** algorithm also maitains the number of times that each arm has been played, denoted by $n_i(t)$. Initially, each arm is played once. Afterwards, at round t, the algorithm greedily picks the arm $j(t)$ as follows:

$$j(t) = argmax_{i = 1, ..., K} \left( u_i + \sqrt{\frac{2lnt}{n_i}} \right)$$

We can see that the **UCB** algorithm will try to learn about arms that we don't know enough about. The main advantages of these types of algorithms are:

- Take uncertainty of sample mean estimate into account in a smart way.
- No parameters (e.g. epsilon, annealing) to validate.

```python
def ucb(counts):
    """
    adaptive softmax

    Parameters
    ----------
    counts : int 2d-array, shape( K, 2 ), where K = the total number of arms
         success and failures for each arm where column 0 represents
         success, 1 represents failure

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    totals = counts.sum(axis = 1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    bonus = np.sqrt(2 * np.log(total_counts) / totals)
    return np.argmax(empirical_means + bonus)

counts = np.ones((K, 2))
softmax(counts)
```

    1

## Experimenting With Bandit Algorithms

In this section, we'll use our simulated data to experiment with our algorithms. To do this we'll also need a metric to calculate how well we are doing. Recall the absolute *best* we can do is to always pick the webpage (arm) with the largest click through rate (ctr). Denote this best arm's probability of $w_{opt}$. Our score should be relative to how well we would have done had we chosen the best arm from the beginning. This motivates the *total regret* of a strategy, defined as:

$$
\begin{align}
R_T & = \sum_{t=1}^{T} \left( w_{opt} - w_{I(t)} \right) \nonumber \\
& = Tw_{opt} - \sum_{t=1}^{T} \;  w_{I(t)} \nonumber
\end{align}
$$

Where $T$ is the total number of samples in the experiment, $w_{I(t)}$ is the probability of obtaining the reward (getting clicked) of the chosen arm in the $t_{th}$ turn. A total regret of 0 means the strategy is attaining the best possible score. This is likely not possible, as initially our algorithm will often make the wrong choice. Ideally, a strategy's total regret should flatten as it learns the best bandit. (Mathematically, we achieve $w_{I(t)} = w_{opt}$ often)

We'll run the experiment and plot the cumulative regret of the three algorithms below:

```python
def run_bandit_algo(rewards, ctr, algo, **kwargs):
    """
    Run different types of bandit algorithms

    Parameters
    ----------
    rewards, ctr :
        Return value of the `generate_bernoulli_bandit_data` function

    algo : bandit function
        [ epsilon_greedy, softmax, ucb ]

    **kwargs :
        additional parameters to pass in to the algo

    Returns
    -------
    cum_regret : 1d-array, shape( n_simulations, )
        The total regret accumulated over the experiment, where the regret
        is measured by the maximum ctr - the chosen arm's ctr

    opt_arm_percentage : float
        The percentage of plays in which the optimal arm is pulled
    """
    n_simulations, K = rewards.shape

    # counts : success and failures for each arm where column 0 represents
    # success, 1 represents failure. Each arm's count is initialiated as 1
    # to ensure that each arm is played at least once, to prevent "cold start"
    # problem and 0 division in the beginning
    counts = np.ones( (K, 2), dtype = int )
    regret = np.zeros(n_simulations)

    max_ctr_count = 0
    max_ctr = np.max(ctr)
    max_ctr_idx = np.argmax(ctr)

    for i in range(n_simulations):
        # 1. run the algorithm to obtain the arm that got pulled
        # 2. update the success / failure according to the generated rewards
        # 3. update the expected regret for each turn of the simulation
        # 4. if the arm that got pulled is the one with the opt ctr, increment this count
        arm = algo( counts, **kwargs )
        if rewards[i, arm] == 1:
            counts[arm, 0] += 1
        else:
            counts[arm, 1] += 1

        regret[i] = max_ctr - ctr[arm]
        if arm == max_ctr_idx:
            max_ctr_count += 1

    cum_regret = np.cumsum(regret)
    opt_arm_percentage = max_ctr_count / n_simulations
    return cum_regret, opt_arm_percentage
```

```python
K = 5
n_simulations = 10000
algorithms = [epsilon_greedy, softmax, ucb]

def run_experiment(K, n_simulations, algorithms):
    """
    Run the bandit algorithm's simulation by the
    specified number of samples for simulation, the number of arms
    and the different version of algorithm

    Parameters
    ----------
    n_simulations : int
        the total number of turns in a simulation

    K : int
        the total number of arms

    algorithms : list of functions
        the list of bandit algorithms to simulate

    Returns
    -------
    ctr : float 1d-array, shape( K, )
        the randomly generated empirical click through rate for each arm

    algo_opt_arm_percentage : float list
        the percentage of simulations that chose the best arm

    algo_cum_regret : float 2d-array, shape( n_simulations, length of the algorithm )
        each column stores the cumulative regret for one algorithm

    fig : matplotlib figure
        the cumulative regret for each bandit algorithm
    """
    algo_opt_arm_percentage = []
    algo_cum_regret = np.zeros( (n_simulations, len(algorithms)) )

    fig = plt.figure( figsize = (10, 7) )
    ctr, rewards = generate_bernoulli_bandit_data(n_simulations, K)

    for idx, algo in enumerate(algorithms):
        cum_regret, opt_arm_percentage = run_bandit_algo(rewards, ctr, algo = algo)
        algo_cum_regret[:, idx] = cum_regret
        algo_opt_arm_percentage.append(opt_arm_percentage)

        plt.semilogy(cum_regret, label = algo.__name__)

    plt.title( 'Simulated Bandit Performance for K = {}'.format(K) )
    plt.ylabel('Cumulative Expected Regret')
    plt.xlabel('Round Index')
    plt.legend(loc = 'lower right')

    return ctr, algo_opt_arm_percentage, algo_cum_regret, fig
```

```python
# change default figure size and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

ctr, algo_opt_arm_percentage, algo_cum_regret, fig = run_experiment(K, n_simulations, algorithms)
fig.show()
print(ctr)
print(algo_opt_arm_percentage)
```

**Section Conclusion:** The plot of the cumulative expected regret of the experiment above showed that all three different algorithms have converged (the cumulative expected regret gradually decreases to a steady level). And the **UCB** seems to be doing better than the other two algorithms in this limited horizon.

# Bayesian Bandits

All of that was great, so what's next? Well, it turns out that none of the algorithms we used in the last section are really suitable in real world applications. Why?

Recall that in the experiment above, we tested different kinds of bandit algorithms with the assumption that there is no delay between pulling an arm and observing the result. Or, more precisely, if there is a delay, it should happen before the next timing to pull an arm. This means the following timeline is impossible: 12:00 Visitor A sees Variation 1. 12:01 visitor B sees Variation 2. 12:02 Visitor A converts.

Also if you have limited computational resources, which means that you can only update your observed data in batch every 2 hours. For these kinds of delayed batch case, the algorithms described in the last section will pull the same arm every time for those 2 hours because it is deterministic in the absence of immediate updates. To sum up, the algorithms we just described needs the distributions to be updated every single round to work properly, which may not be applicable in a lot of practical cases ....

Having that caveat in mind, we'll introduce a Bayesian method that is more "immune" to this delayed feedback problem, namely **Thompson Sampling**.

Recall that the the problem we want to solve is the following. You have come up with $K$ different variations of the webpage (e.g. different layout) that now you wish to find the ones with the best click through rate (CTR), e.g. clicking to sign-up for the newsletter. Let's represent each CTR by $\theta_i$ - i.e., $\theta_i$ is the true probability that an individual user will click when they were shown with the $i_{th}$ webpage. It is important to note that we don't actually know what $\theta_i$ is - if we did, we could simply choose ii for which $\theta_i$ was largest and move on. We're simply pretending that we know in order to simulate the performance of the algorithm.

Using the Bayesian approach we will construct a prior probability distribution which represents our original belief about what the actual value of $\theta_i$, our ctr for the $i_{th}$ webpage is. The prior we'll use is the Beta distribution. Here's a quick recap of the distribution:

## Beta Distribution

The Beta distribution is very useful in Bayesian statistics. A random variable $X$ has a Beta distribution, with parameters $(\alpha, \beta)$, if its density function is:

$$f_X(x | \; \alpha, \beta ) = \frac{ x^{(\alpha - 1)}(1-x)^{ (\beta - 1) } }{B(\alpha, \beta) }$$

where $B$ is the [Beta function](http://en.wikipedia.org/wiki/Beta_function) (hence the name). The random variable $X$ is only allowed in [0,1], making the Beta distribution a popular distribution for decimal values, probabilities and proportions. The values of $\alpha$ and $\beta$, both positive values, provide great flexibility in the shape of the distribution. Below we plot some Beta distributions with different $\alpha$ and $\beta$ values:

```python
plt.figure( figsize = (12, 5) )

x = np.linspace(0.01, .99, 100)
params = [ (2, 5), (1, 1), (5, 5), (20, 4) ]

for a, b in params:
    y = beta.pdf(x, a, b)
    lines = plt.plot( x, y, label = "(%.1f,%.1f)" % (a, b), lw = 2 )
    plt.fill_between( x, 0, y, alpha = 0.2, color = lines[0].get_color() )
    plt.autoscale(tight = True)

plt.legend(loc = 'upper left', title = "(a,b)-parameters")
plt.show()
```

There are two important things to note about the Beta distribution:

The first is the presence of the flat distribution above, specified by parameters $(1,1)$. This is the Uniform distribution. Hence the Beta distribution is a generalization of the Uniform distribution.

The second is that there is an interesting connection between the Beta distribution and the Binomial distribution. Suppose we are interested in some unknown proportion or probability $p$. We assign a $\text{Beta}(\alpha, \beta)$ prior to $p$. We observe some data generated by a Binomial process, say $X \sim \text{Binomial}(N, p)$, with $p$ still unknown. Then our posterior *is again a Beta distribution*, i.e. $p | X \sim \text{Beta}( \alpha + X, \beta + N -X )$. Succinctly, one can relate the two by "a Beta prior with Binomial observations creates a Beta posterior".

In light of the above two paragraphs, if we start with a $\text{Beta}(1,1)$ prior on $p$ (which is a Uniform), observe data $X \sim \text{Binomial}(N, p)$, then our posterior is $\text{Beta}(1 + X, 1 + N - X)$.

## Thompson Sampling

So after assuming the priors on the probability of ctr for each webpage. To be explicit on the phrase "assuming the priors", we will assume that we're completely ignorant of these probabilities. So a very natural prior is the flat prior over 0 to 1, $\text{Beta}(\alpha=1,\beta=1)$. The algorithm then proceeds as follows:

For each turn:

1. Sample a random variable $X_i$ from the prior of arm $i$, for all $i$ ($K$ in total).
2. Select the arm with largest sample, i.e. select $i = \text{argmax}\; X_i$.
3. Observe the result of pulled arm $i$, and update your prior with that arm $i$.
4. Return to 1.

Like all the algorithms we've introduced before, **Thompson Sampling** suggests that we should not discard losers, but we should pick them at a decreasing rate as we gather confidence that there exist *better* webpages (arms). This follows because there is always a non-zero chance that a webpage with a lower ctr will get chosen, but the probability of this event decreases as we play more rounds.

```python
class BayesianBandit:
    """
    Thompson Sampling

    Parameters
    ----------
    K : int
        total number of arms

    prior_params : list of float length 2 tuple, default None, (optional)
        each element of the list is a tuple, where each tuple
        contains the alpha and beta parameter that represents the prior
        beta distribution for each arm. If not supplied
        it will assume that all arms's prior starts with an uniform distribution

    Attributes
    ----------
    trials, success : int 1d-array, shape( K, )
        stores the trials and success for each arm,
        e.g. trial = [ 1, 1 ] and success = [ 0, 1 ] means
        that both arm has been pulled once and arm 1 has generated
        the reward (clicked)
    """

    def __init__(self, K, prior_params = None):

        if prior_params:
            priors = namedtuple( "priors", ["alpha", "beta"] )
            prior  = [priors(*p) for p in prior_params]
            self.alphas = np.array([p.alpha for p in prior])
            self.betas  = np.array([p.beta  for p in prior])
        else:
            self.alphas = np.ones(K)
            self.betas  = np.ones(K)

        self.trials  = np.zeros(K, dtype = int)
        self.success = np.zeros(K, dtype = int)

    def get_recommendation(self):
        """
        for all arms, construct their beta distribution and
        draw a random sample from it, then return the arm
        with the maximum value random sample
        """
        theta = np.random.beta(self.alphas + self.success,
                               self.betas + self.trials - self.success)
        return np.argmax(theta)

    def update_result(self, arm, converted):
        """
        override the trials and success array, the success array
        will only be updated if it has generated a reward
        """
        self.trials[arm] += 1
        if converted:
            self.success[arm] += 1

        return self
```

```python
def experiment(T, K = None, ctr = None, prior_params = None):
    """
    run the experiment for Thompson Sampling,
    pass in ctr, the fixed ctr for each arm
    or K, the total number of arms to run the experiment,
    if K is supplied then it will be randomly generated

    Parameters
    ----------
    T : int
        number of simulation in an experiment

    K : int, , default = None, (optional)
        total number of arms

    ctr : float sequence, len = K, default = None, (optional)
        the empirical click through rate for each arm

    prior_params : list of float length 2 tuple, default None, (optional)
        each element of the list is a tuple, where each tuple
        contains the alpha and beta parameter that represents the prior
        beta distribution for each arm. If not supplied
        it will assume that all arms's prior starts with an uniform distribution

    Returns
    -------
    ctr : float sequence, len = K
        the supplied or the randomly generated ctr

    trials, success : 2d-array, shape( T, K )
        trials and success recorded for each turn of the experiment

    alphas, betas : float 1d-array, shape( K, )
        the alpha and beta parameters for each arm
    """
    if ctr:
        K = len(ctr)
    else:
        ctr = np.random.rand(K)

    trials  = np.zeros( (T, K), dtype = int )
    success = np.zeros( (T, K), dtype = int )

    bb = BayesianBandit(K, prior_params)
    for t in range(T):
        arm = bb.get_recommendation()
        converted = np.random.rand() < ctr[arm]
        bb.update_result(arm, converted)
        trials[t]  = bb.trials
        success[t] = bb.success

    return ctr, trials, success, bb.alphas, bb.betas
```

```python
def experiment_plot(ctr, trials, success):
    """
    Pass in the ctr, trials and success returned
    by the `experiment` function and plot
    the Cumulative Number of Turns For Each Arm and
    the CTR's Convergence Plot side by side
    """
    T, K = trials.shape
    n = np.arange(T) + 1
    fig = plt.figure( figsize = (14, 7) )

    plt.subplot(121)
    for i in range(K):
        plt.loglog( n, trials[:, i], label = "arm {}".format(i + 1) )

    plt.legend(loc = "upper left")
    plt.xlabel("Number of turns")
    plt.ylabel("Number of turns/arm")
    plt.title("Cumulative Number of Turns For Each Arm")

    plt.subplot(122)
    for i in range(K):
        plt.semilogx( n, np.zeros(T) + ctr[i], label = "arm {}'s CTR".format(i + 1) )

    plt.semilogx( n, ( success[:, 0] + success[:, 1] ) / n, label = "CTR at turn t" )

    plt.axis([0, T, 0, 1])
    plt.legend(loc = "upper left")
    plt.xlabel("Number of turns")
    plt.ylabel("CTR")
    plt.title("CTR's Convergence Plot")
    return fig
```

```python
# number of simulation in an experiment
T = 10000

# the empirical click through rate for each arm
ctr = 0.25, 0.35

ctr, trials, success, alphas, betas = experiment(T = T, ctr = ctr)
fig = experiment_plot(ctr, trials, success)
fig.show()
```

In our simulation, we gave the Bayesian bandit two webpages (arms) - one had a CTR of 0.25, the other had a CTR of 0.35. To start with, both webpages were displayed to the user with roughly equal probability. Over time, evidence accumulated that arm 2 was considerably better than arm 1. At this point the algorithm switched to displaying primarily webpage 1, and the overall CTR of the experiment converged to 0.35 (the optimal CTR).

We can also visualize our Beta distribution for each arms in different turns.

```python
def plot_beta_dist(ctr, trials, success, alphas, betas, turns):
    """
    Pass in the ctr, trials and success, alphas, betas returned
    by the `experiment` function and the number of turns
    and plot the beta distribution for all the arms in that turn
    """
    subplot_num = len(turns) / 2
    x = np.linspace(0.001, .999, 200)
    fig = plt.figure( figsize = (14, 7) )

    for idx, turn in enumerate(turns):
        plt.subplot(subplot_num, 2, idx + 1)

        for i in range( len(ctr) ):
            y = beta( alphas[i] + success[turn, i],
                      betas[i] + trials[turn, i] - success[ turn, i ] ).pdf(x)
            line = plt.plot( x, y, lw = 2, label = "arm {}".format( i + 1 ) )
            color = line[0].get_color()
            plt.fill_between(x, 0, y, alpha = 0.2, color = color)
            plt.axvline(x = ctr[i], color = color, linestyle = "--", lw = 2)
            plt.title("Posteriors After {} turns".format(turn) )
            plt.legend(loc = "upper right")

    return fig
```

```python
turns = [1, 100, 1000, 9999]
posterior_fig = plot_beta_dist(ctr, trials, success, alphas, betas, turns)
posterior_fig.show()
```

As you can see, we started out with some prior Beta distributions that represents the our initial beliefs about possible ctr values for each arm. We then update these beliefs based on evidence by showing different webpages to other randomly chosen users and observing the ctr. After doing this for many number of turns, we incrementally narrow the width of each arm's Beta distribution. Meaning that as we gather more data, we become more confident about each arm's actual ctr.

Note that we don't really care how accurate we become about the inference of the hidden probabilities — for this problem we are more interested in becoming more confident in choosing the best arm. This is basically why at the end of experiment, arm 1's distribution is wider. The algorithm is ignorant about what that hidden probability might be, but we are reasonably confident that it is not the best, so the algorithm chooses to ignore it.

From the above, we can see that starting after 100 pulls, the majority of arm 2's distribution already leads the pack, hence the algorithm will almost always choose this arm. This is good, as this arm is indeed better.

## Notes On Bandit Testings

In world settings, we need to account for situations such as delayed batch update or delay feedback. In such cases, algorithms such as **Epsilon Greedy**, **Softmax**, **UCB** needs the distributions to be updated every single round to work properly. On the other hand, **Thompson Sampling** is more realistic as it relies on random samples, which will be different every time even if the distributions are each arm aren't updated for a while.

So, after gaining some knowledge on **bandit testings**. The question that comes into our head is that: "In general, when is it preferred over the classical **A/B testing**?"

### Short-term testing

> "Whenever you have a small amount of time for both exploration and exploitation, use a bandit testing."

**Bandit testing** are conducive for short tests for clear reasons – if you were to run a classic **A/B testing** you’d not even be able to enjoy the period of pure exploitation. Instead, **bandit testing** allow you to adjust in real time and send more traffic, more quickly, to the better variation. Here are some possible use cases:

- **Headlines:** News has a short life cycle. Why would you run **A/B testing** on a headline if by the time you learn which variation is best, the time where the answer is applicable is over?
- **Holiday Campaigns:** e.g. If you’re running tests on an ecommerce site for Black Friday, an A/B testing isn’t that practical – you might only be confident in the result at the end of the day. On the other hand, a **bandit testing** will drive more traffic to the better-performing variation – and that in turn can increase revenue.

### Long-term testing

Because **Bandit testing** automatically shift traffic to higher performing (at the time) variations, thus it is effective in long term (or ongoing) testing where you can set it and forget about it. e.g. Serving specific ads and content to user sets (targeting ads).

## Reference

- [Notes: Algorithms for the multi-armed bandit problem](http://www.cs.mcgill.ca/~vkules/bandits.pdf)
- [Blog: Bandits for Recommendation Systems](http://engineering.richrelevance.com/bandits-recommendation-systems/)
- [Blog: When to Run Bandit Tests Instead of A/B Tests](http://conversionxl.com/bandit-tests/)
- [Blog: Bayesian Bandits - optimizing click throughs with statistics](https://www.chrisstucchio.com/blog/2013/bayesian_bandit.html)
- [Blog: Balancing Earning with Learning: Bandits and Adaptive Optimization](http://conductrics.com/balancing-earning-with-learning-bandits-and-adaptive-optimization/)
- [Notebook: Bayesian Methods for Hackers Chapter 6](http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter6_Priorities/Ch6_Priors_PyMC2.ipynb)