Processamento de Linguagem Natural

Gustavo C. A Corradi

robotreport.me

@gustavocorradi

# Lição  I - Representação de Texto

Nesta lição, veremos em alguns detalhes como podemos representar melhor o texto em nosso aplicativo. Vamos começar importando os módulos que usaremos

```python
import string
from collections import Counter
from pprint import pprint
import gzip
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

```python
import re
import nltk
from bs4 import BeautifulSoup

url = "http://www.rb.org.br/detalhe_artigo.asp?id=2295&idioma=Portugues"
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html)
for script in soup(["script", "style"]):
    script.extract()    # rip it out

# get text
text = soup.get_text()

# break into lines and remove leading and trailing space on each
#lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
#chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
# drop blank lines
#text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

```

    Radiologia Brasileira -
    Contrastes orais neutros para enterografia por tomografia computadorizada

    Radiologia Brasileira - Publicação Científica Oficial do Colégio Brasileiro de Radiologia
    AMB - Associação Médica Brasileira CNA - Comissão Nacional de Acreditação

          Idioma/Language:

    Home
    CBR
    RB History
    Editorial Board
    Publication Guidelines
    Advanced Search
    Current Issue
    Previous Issues
    Paper Submission
    Audience
    Supplements
    Ahead of Print
    Contact us

              Vol. 45 nº 3 - Maio / Jun.  of 2012

    EDITORIAL

     Print

    Page(s) V to VI

    					Contrastes orais neutros para enterografia por tomografia computadorizada

    Autho(rs): Daniella Braz Parente

     PDF Português
     PDF English

    Texto em Português
    English Text

    Durante muitos anos o intestino delgado foi um órgão difícil de ser estudado. Até bem pouco tempo, os únicos métodos de imagem disponíveis para o estudo deste segmento eram o trânsito delgado e a enteróclise convencional, que apresentam diversas limitações(1,2).

    Nos últimos anos, com o desenvolvimento da tecnologia, vários métodos têm sido desenvolvidos para o estudo do intestino delgado. No campo da endoscopia, dispõe-se hoje da cápsula endoscópica(3) e da enteroscopia com duplo-balão(4), métodos caros e não amplamente disponíveis em nosso meio. No campo da radiologia, surgiram a enterografia e a enteróclise por tomografia computadorizada (TC) ou por ressonância magnética (RM), que combinam as vantagens dos métodos radiológicos convencionais com as dos métodos seccionais, ou seja, a distensão do lúmen intestinal associada à visualização multiplanar de toda a cavidade abdominal e pélvica, sem sobreposição de imagens. Com estes métodos, toda a espessura da parede intestinal é visualizada, incluindo mucosa, submucosa, muscular e serosa, assim como gordura perientérica, vasos retos, linfonodos e padrão de realce pelo contraste, o que permite identificar espessamentos parietais, sinais inflamatórios, fístulas, coleções, além de focos de sangramento e tumores, entre outros(2,57).

    A distensão luminal é um passo fundamental na interpretação do estudo do intestino delgado, já que alças colabadas podem esconder ou simular segmentos doentes(2,57). Neste número da Radiologia Brasileira, D'Ippolito et al.(8) avaliam o desempenho de diferentes meios de contraste orais neutros, comparando a capacidade de distensão intestinal, a definição da parede intestinal, a aceitação e os efeitos colaterais. Os autores descrevem o estudo de 30 pacientes submetidos a enterografia por TC, randomizados entre três tipos de contraste oral neutro disponíveis em nosso meio: água, leite integral e polietilenoglicol (PEG). Neste estudo, a concentração de PEG empregada foi inferior à utilizada em estudos anteriores, com o objetivo de reduzir seus efeitos indesejados, mas mantendo sua capacidade de distensão da luz intestinal. O preparo com PEG resultou na melhor distensão intestinal (57,5%), não havendo diferença significativa entre o leite (35%) e a água (25%). Em todos os grupos foi possível a definição adequada da parede intestinal e os meios de contraste tiveram boa aceitação. Dos pacientes que ingeriram PEG, 80% apresentaram diarreia, efeito colateral não observado com o uso de leite ou água. O estudo mostra, de forma clara, que água e leite não são bons meios de contraste oral para uso na enterografia, já que são absorvidos pelo intestino e não distendem adequadamente as alças, o que é muito importante para a correta interpretação do exame. Mostra também que o PEG é um bom meio de contraste oral para o estudo do intestino delgado e que os pacientes toleram bem a ingestão de grande volume de PEG (1.500 ml). Além disso, demonstra que a diarreia é um efeito indesejado habitual, que ocorre na maioria dos pacientes, inerente à propriedade do PEG de não ser absorvido ao longo do trato gastrintestinal, mas geralmente autolimitada. Estudos futuros comparando o uso de PEG em baixa concentração com PEG na concentração habitual são necessários para melhor avaliação da distensibilidade luminal e incidência de efeitos colaterais, principalmente diarreia. Também seria interessante a realização de estudos comparando o PEG com o manitol, já que este último tem um custo muito baixo e é amplamente disponível na rede hospitalar pública.

    A avaliação de pacientes com suspeita ou diagnóstico de doença de Crohn é uma das principais indicações de estudo do intestino delgado. São pacientes jovens, que necessitam de reavaliações durante toda a vida. Desta forma, deve ser levada em consideração a dose de radiação a que estes pacientes são submetidos e tentar sempre reduzir ao máximo a dose de radiação utilizada(9). Este aspecto foi contemplado no estudo de D'Ippolito et al.(8), no qual foi realizada apenas a fase enterográfica, já que as fases pré-contraste, arterial e tardia não acrescentam significativamente para o diagnóstico e quadriplicam a dose de radiação a que os pacientes são expostos. Outra opção para esta população é a enterografia por RM, que tem acurácia diagnóstica semelhante e a vantagem de não utilizar radiação, podendo ser repetida múltiplas vezes sem malefícios ao paciente(6).

    A comparação entre diferentes tipos de contraste oral disponíveis em nosso meio para uso na enterografia por TC é um tema bastante importante, visto a grande prevalência de pacientes com doenças no intestino delgado que podem se beneficiar deste método de imagem, como pacientes com doença inflamatória intestinal, notadamente os com doença de Crohn, pacientes com síndrome do cólon irritável que precisam excluir outras causas para a dor abdominal, os com sangramento intestinal e os com tumores do intestino delgado. A TC está amplamente disponível em nosso país e é importante que os radiologistas estejam familiarizados com este método, para contribuir de forma significativa no diagnóstico, tratamento e acompanhamento dos pacientes com doenças intestinais.

    REFERÊNCIAS

    1. Saibeni S, Rondonotti E, Iozzelli A, et al. Imaging of the small bowel in Crohn's disease: a review of old and new techniques. World J Gastroenterol. 2007;13:327987.

    2. Lee SS, Kim AY, Yang SK, et al. Crohn disease of the small bowel: comparison of CT enterography, MR enterography, and small-bowel follow-through as diagnostic techniques. Radiology. 2009;251:75161.

    3. Freitas GP, Teixeira N, Feldman G. Capsule endoscopy in clinical practice: four years of experience from a single center. Arq Gastroenterol. 2011;48:2202.

    4. Albert JG. Interventional balloon-enteroscopy. J Interv Gastroenterol. 2012;2:4250.

    5. Costa-Silva L, Martins T, Passos MCF. Enterografia por tomografia computadorizada: experiência inicial na avaliação das doenças do intestino delgado. Radiol Bras. 2010;43:3038.

    6. Siddiki HA, Fidler JL, Fletcher JG, et al. Prospective comparison of state-of-the-art MR enterography and CT enterography in small-bowel Crohn's disease. AJR Am J Roentgenol. 2009;193:11321.

    7. Masselli G, Gualdi G. Evaluation of small bowel tumors: MR enteroclysis. Abdom Imaging. 2010;35:2330.

    8. D'Ippolito G, Braga FA, Resende MC, et al. Enterografia por tomografia computadorizada: uma avaliação de diferentes contrastes orais neutros. Radiol Bras. 2012;45:13943.

    9. Desmond AN, O'Regan K, Curran C, et al. Crohn's disease: factors associated with exposure to high levels of diagnostic radiation. Gut. 2008;57:15249.

    Doutoranda em Radiologia da Universidade Federal do Rio de Janeiro (UFRJ), Pesquisadora do Instituto D'Or de Pesquisa e Ensino, Radiologista da Rede Labs D'Or, Rio de Janeiro, RJ, Brasil. E-mail: daniella.parente@gmail.com
     Back  Top  Print

    GN1© Copyright 2020 - All rights reserved to Colégio Brasileiro de Radiologia e Diagnóstico por ImagemAv. Paulista, 37 - 7° andar - Conj. 71 - CEP 01311-902 - São Paulo - SP - Brazil - Phone: (11) 3372-4544 - Fax: (11) 3372-4554

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []
    print(temp)
    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        print(word[0])
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'had': 2,
             'a': 2,
             'was': 2,
             'everywhere': 2,
             'that': 2,
             'whose': 1,
             'fleece': 1,
             'white': 1,
             'as': 1,
             'snow': 1,
             'and': 1,
             'the': 1,
             'sure': 1,
             'to': 1,
             'go': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

And let's take a look at them:

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]

    Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('had', 2),
     ('a', 2),
     ('was', 2),
     ('everywhere', 2),
     ('that', 2),
     ('whose', 1),
     ('fleece', 1),
     ('white', 1),
     ('as', 1),
     ('snow', 1),
     ('and', 1),
     ('the', 1),
     ('sure', 1),
     ('to', 1),
     ('go', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the most common words in this large corpus:

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x:x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 2.0794415416798357,
              'lamb': 0.8630462173553426}),
     Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'whose': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'as': 1.3862943611198906,
              'snow': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 0.6931471805599453,
              'the': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'was': 0.6931471805599453,
              'sure': 1.3862943611198906,
              'to': 1.3862943611198906,
              'go': 1.3862943611198906})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False
    m = 0

    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followd by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.


# Run all cells

Look for instructions below

```python
!apt-get install clang-9
!git clone https://github.com/RumblingTurtle/SrdPy.git
!pip install git+https://github.com/rdeits/meshcat-python.git@master
!pip install ./SrdPy
```

    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following package was automatically installed and is no longer required:
      libnvidia-common-460
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      binfmt-support libclang-common-9-dev libclang-cpp9 libclang1-9 libffi-dev
      libllvm9 libomp-9-dev libomp5-9 libpfm4 llvm-9 llvm-9-dev llvm-9-runtime
      llvm-9-tools python-chardet python-pkg-resources python-pygments python-yaml
      python3-pkg-resources python3-pygments python3-yaml
    Suggested packages:
      clang-9-doc libomp-9-doc llvm-9-doc python-setuptools ttf-bitstream-vera
      python3-setuptools
    The following NEW packages will be installed:
      binfmt-support clang-9 libclang-common-9-dev libclang-cpp9 libclang1-9
      libffi-dev libllvm9 libomp-9-dev libomp5-9 libpfm4 llvm-9 llvm-9-dev
      llvm-9-runtime llvm-9-tools python-chardet python-pkg-resources
      python-pygments python-yaml python3-pkg-resources python3-pygments
      python3-yaml
    0 upgraded, 21 newly installed, 0 to remove and 34 not upgraded.
    Need to get 85.9 MB of archives.
    After this operation, 511 MB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-yaml amd64 3.12-1build2 [109 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 binfmt-support amd64 2.1.8-2 [51.6 kB]
    Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libllvm9 amd64 1:9-2~ubuntu18.04.2 [14.8 MB]
    Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libclang-cpp9 amd64 1:9-2~ubuntu18.04.2 [25.1 MB]
    Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libclang-common-9-dev amd64 1:9-2~ubuntu18.04.2 [3,861 kB]
    Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libclang1-9 amd64 1:9-2~ubuntu18.04.2 [6,701 kB]
    Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 clang-9 amd64 1:9-2~ubuntu18.04.2 [1,109 kB]
    Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libomp5-9 amd64 1:9-2~ubuntu18.04.2 [299 kB]
    Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libomp-9-dev amd64 1:9-2~ubuntu18.04.2 [58.6 kB]
    Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 llvm-9-runtime amd64 1:9-2~ubuntu18.04.2 [176 kB]
    Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpfm4 amd64 4.9.0-2 [225 kB]
    Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 llvm-9 amd64 1:9-2~ubuntu18.04.2 [4,874 kB]
    Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 libffi-dev amd64 3.2.1-8 [156 kB]
    Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-pygments all 2.2.0+dfsg-1ubuntu0.2 [576 kB]
    Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB]
    Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pygments all 2.2.0+dfsg-1ubuntu0.2 [574 kB]
    Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 llvm-9-tools amd64 1:9-2~ubuntu18.04.2 [250 kB]
    Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 llvm-9-dev amd64 1:9-2~ubuntu18.04.2 [26.6 MB]
    Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pkg-resources all 39.0.1-2 [128 kB]
    Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB]
    Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]
    Fetched 85.9 MB in 6s (14.7 MB/s)
    Selecting previously unselected package python3-yaml.
    (Reading database ... 160690 files and directories currently installed.)
    Preparing to unpack .../00-python3-yaml_3.12-1build2_amd64.deb ...
    Unpacking python3-yaml (3.12-1build2) ...
    Selecting previously unselected package binfmt-support.
    Preparing to unpack .../01-binfmt-support_2.1.8-2_amd64.deb ...
    Unpacking binfmt-support (2.1.8-2) ...
    Selecting previously unselected package libllvm9:amd64.
    Preparing to unpack .../02-libllvm9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libllvm9:amd64 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libclang-cpp9.
    Preparing to unpack .../03-libclang-cpp9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libclang-cpp9 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libclang-common-9-dev.
    Preparing to unpack .../04-libclang-common-9-dev_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libclang-common-9-dev (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libclang1-9.
    Preparing to unpack .../05-libclang1-9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libclang1-9 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package clang-9.
    Preparing to unpack .../06-clang-9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking clang-9 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libomp5-9:amd64.
    Preparing to unpack .../07-libomp5-9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libomp5-9:amd64 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libomp-9-dev.
    Preparing to unpack .../08-libomp-9-dev_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking libomp-9-dev (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package llvm-9-runtime.
    Preparing to unpack .../09-llvm-9-runtime_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking llvm-9-runtime (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libpfm4:amd64.
    Preparing to unpack .../10-libpfm4_4.9.0-2_amd64.deb ...
    Unpacking libpfm4:amd64 (4.9.0-2) ...
    Selecting previously unselected package llvm-9.
    Preparing to unpack .../11-llvm-9_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking llvm-9 (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package libffi-dev:amd64.
    Preparing to unpack .../12-libffi-dev_3.2.1-8_amd64.deb ...
    Unpacking libffi-dev:amd64 (3.2.1-8) ...
    Selecting previously unselected package python-pygments.
    Preparing to unpack .../13-python-pygments_2.2.0+dfsg-1ubuntu0.2_all.deb ...
    Unpacking python-pygments (2.2.0+dfsg-1ubuntu0.2) ...
    Selecting previously unselected package python-yaml.
    Preparing to unpack .../14-python-yaml_3.12-1build2_amd64.deb ...
    Unpacking python-yaml (3.12-1build2) ...
    Selecting previously unselected package python3-pygments.
    Preparing to unpack .../15-python3-pygments_2.2.0+dfsg-1ubuntu0.2_all.deb ...
    Unpacking python3-pygments (2.2.0+dfsg-1ubuntu0.2) ...
    Selecting previously unselected package llvm-9-tools.
    Preparing to unpack .../16-llvm-9-tools_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking llvm-9-tools (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package llvm-9-dev.
    Preparing to unpack .../17-llvm-9-dev_1%3a9-2~ubuntu18.04.2_amd64.deb ...
    Unpacking llvm-9-dev (1:9-2~ubuntu18.04.2) ...
    Selecting previously unselected package python-pkg-resources.
    Preparing to unpack .../18-python-pkg-resources_39.0.1-2_all.deb ...
    Unpacking python-pkg-resources (39.0.1-2) ...
    Selecting previously unselected package python-chardet.
    Preparing to unpack .../19-python-chardet_3.0.4-1_all.deb ...
    Unpacking python-chardet (3.0.4-1) ...
    Selecting previously unselected package python3-pkg-resources.
    Preparing to unpack .../20-python3-pkg-resources_39.0.1-2_all.deb ...
    Unpacking python3-pkg-resources (39.0.1-2) ...
    Setting up python3-yaml (3.12-1build2) ...
    Setting up binfmt-support (2.1.8-2) ...
    Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.service → /lib/systemd/system/binfmt-support.service.
    invoke-rc.d: could not determine current runlevel
    invoke-rc.d: policy-rc.d denied execution of start.
    Setting up python-yaml (3.12-1build2) ...
    Setting up libffi-dev:amd64 (3.2.1-8) ...
    Setting up python3-pkg-resources (39.0.1-2) ...
    Setting up python-pkg-resources (39.0.1-2) ...
    Setting up libomp5-9:amd64 (1:9-2~ubuntu18.04.2) ...
    Setting up libpfm4:amd64 (4.9.0-2) ...
    Setting up python-pygments (2.2.0+dfsg-1ubuntu0.2) ...
    Setting up libllvm9:amd64 (1:9-2~ubuntu18.04.2) ...
    Setting up libomp-9-dev (1:9-2~ubuntu18.04.2) ...
    Setting up python3-pygments (2.2.0+dfsg-1ubuntu0.2) ...
    Setting up libclang-common-9-dev (1:9-2~ubuntu18.04.2) ...
    Setting up python-chardet (3.0.4-1) ...
    Setting up libclang1-9 (1:9-2~ubuntu18.04.2) ...
    Setting up libclang-cpp9 (1:9-2~ubuntu18.04.2) ...
    Setting up llvm-9-runtime (1:9-2~ubuntu18.04.2) ...
    Setting up llvm-9-tools (1:9-2~ubuntu18.04.2) ...
    Setting up clang-9 (1:9-2~ubuntu18.04.2) ...
    Setting up llvm-9 (1:9-2~ubuntu18.04.2) ...
    Setting up llvm-9-dev (1:9-2~ubuntu18.04.2) ...
    Processing triggers for systemd (237-3ubuntu10.46) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

    Cloning into 'SrdPy'...
    remote: Enumerating objects: 1007, done.
    remote: Counting objects: 100% (1007/1007), done.
    remote: Compressing objects: 100% (483/483), done.
    remote: Total 1007 (delta 546), reused 957 (delta 501), pack-reused 0
    Receiving objects: 100% (1007/1007), 20.34 MiB | 21.01 MiB/s, done.
    Resolving deltas: 100% (546/546), done.
    Collecting git+https://github.com/rdeits/meshcat-python.git@master
      Cloning https://github.com/rdeits/meshcat-python.git (to revision master) to /tmp/pip-req-build-2fnmo930
      Running command git clone -q https://github.com/rdeits/meshcat-python.git /tmp/pip-req-build-2fnmo930
      Running command git submodule update --init --recursive -q
    Requirement already satisfied: ipython>=5 in /usr/local/lib/python3.7/dist-packages (from meshcat==0.1.1) (5.5.0)
    Collecting u-msgpack-python>=2.4.1
      Downloading https://files.pythonhosted.org/packages/a3/54/0400a3a22ff133633d343371821bf81010455fa3a981a93d7ff3e27a554e/u_msgpack_python-2.7.1-py2.py3-none-any.whl
    Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from meshcat==0.1.1) (1.19.5)
    Requirement already satisfied: tornado>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from meshcat==0.1.1) (5.1.1)
    Requirement already satisfied: pyzmq>=17.0.0 in /usr/local/lib/python3.7/dist-packages (from meshcat==0.1.1) (22.0.3)
    Collecting pyngrok>=4.1.6
      Downloading https://files.pythonhosted.org/packages/6b/4e/a2fe095bbe17cf26424c4abcd22a0490e22d01cc628f25af5e220ddbf6f0/pyngrok-5.0.5.tar.gz (745kB)
         |████████████████████████████████| 747kB 8.2MB/s
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (0.8.1)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (4.8.0)
    Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (5.0.5)
    Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (2.6.1)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (1.0.18)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (0.7.5)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (4.4.2)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython>=5->meshcat==0.1.1) (56.0.0)
    Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from pyngrok>=4.1.6->meshcat==0.1.1) (3.13)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect; sys_platform != "win32"->ipython>=5->meshcat==0.1.1) (0.7.0)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from traitlets>=4.2->ipython>=5->meshcat==0.1.1) (0.2.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=5->meshcat==0.1.1) (0.2.5)
    Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=5->meshcat==0.1.1) (1.15.0)
    Building wheels for collected packages: meshcat, pyngrok
      Building wheel for meshcat (setup.py) ... done
      Created wheel for meshcat: filename=meshcat-0.1.1-cp37-none-any.whl size=627107 sha256=17d90c56982e65d0b10d1d687116019e38437a019968289600aac544ba700a18
      Stored in directory: /tmp/pip-ephem-wheel-cache-99xicfwk/wheels/c2/57/90/f09f46ded98f6ea0f4dd05029529fe8d24d936a520786a60e0
      Building wheel for pyngrok (setup.py) ... done
      Created wheel for pyngrok: filename=pyngrok-5.0.5-cp37-none-any.whl size=19246 sha256=82f712a73666c161aa3058772765d483e6f03afea4597fbe493dcc2fea72cd92
      Stored in directory: /root/.cache/pip/wheels/0c/13/64/5ebbcc22eaf53fdf5766b397c1fb17c83f5775fdccf0ea1b88
    Successfully built meshcat pyngrok
    Installing collected packages: u-msgpack-python, pyngrok, meshcat
    Successfully installed meshcat-0.1.1 pyngrok-5.0.5 u-msgpack-python-2.7.1
    Processing ./SrdPy
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from SrdPy==0.1) (1.19.5)
    Collecting casadi
      Downloading https://files.pythonhosted.org/packages/d7/41/abb53746924c0669cc597d18d9fa77c51682f225222d130fec8217fbd256/casadi-3.5.5-cp37-none-manylinux1_x86_64.whl (34.2MB)
         |████████████████████████████████| 34.3MB 107kB/s
    Collecting control
      Downloading https://files.pythonhosted.org/packages/88/87/ee6cb7cdcf4efe5634231bd688b307773629a100ec4c83b0c3eb03edd39d/control-0.9.0.tar.gz (339kB)
         |████████████████████████████████| 348kB 42.7MB/s
    Collecting slycot
      Downloading https://files.pythonhosted.org/packages/85/21/4e7110462f3529b2fbcff8a519b61bf64e0604b8fcbe9a07649c9bed9d7a/slycot-0.4.0.0.tar.gz (1.5MB)
         |████████████████████████████████| 1.6MB 39.0MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
        Preparing wheel metadata ... done
    Collecting urdf-parser-py
      Downloading https://files.pythonhosted.org/packages/96/08/76bc3f7fc9a0345d45527c0054fbec8479d74649d6accb5c1ec174e5098c/urdf_parser_py-0.0.3.tar.gz
    Requirement already satisfied: pyngrok in /usr/local/lib/python3.7/dist-packages (from SrdPy==0.1) (5.0.5)
    Collecting qpsolvers
      Downloading https://files.pythonhosted.org/packages/d2/84/4775a9d3497fed47d77a1996baee409c8f67b670a00b75ac375036875168/qpsolvers-1.6.1-py3-none-any.whl
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from control->SrdPy==0.1) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from control->SrdPy==0.1) (3.2.2)
    Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from urdf-parser-py->SrdPy==0.1) (3.13)
    Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from urdf-parser-py->SrdPy==0.1) (4.2.6)
    Collecting quadprog>=0.1.8
      Downloading https://files.pythonhosted.org/packages/31/c7/833d7b5b53fff5b9042803aa8096207d8e038ca42052f074d4ad8cddbfc5/quadprog-0.1.8.tar.gz (269kB)
         |████████████████████████████████| 276kB 36.1MB/s
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control->SrdPy==0.1) (2.4.7)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control->SrdPy==0.1) (1.3.1)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control->SrdPy==0.1) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->control->SrdPy==0.1) (2.8.1)
    Requirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from quadprog>=0.1.8->qpsolvers->SrdPy==0.1) (0.29.22)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib->control->SrdPy==0.1) (1.15.0)
    Building wheels for collected packages: slycot
      Building wheel for slycot (PEP 517) ... done
      Created wheel for slycot: filename=slycot-0.4.0-cp37-cp37m-linux_x86_64.whl size=1417835 sha256=517fe74af2dba0f31ece207326a0db15eb504452187a4fca6c5238575b325da1
      Stored in directory: /root/.cache/pip/wheels/a2/46/56/f82cbb2fd06556f4f3952a2eb2396e8fd29264fffecbaad3cf
    Successfully built slycot
    Building wheels for collected packages: SrdPy, control, urdf-parser-py, quadprog
      Building wheel for SrdPy (setup.py) ... done
      Created wheel for SrdPy: filename=SrdPy-0.1-cp37-none-any.whl size=63107 sha256=e99a56e090922014480e70d922694bee7d8ce07bab507536b356091acaf3f1d9
      Stored in directory: /tmp/pip-ephem-wheel-cache-5zlrhlrr/wheels/f8/a7/cf/bd93c0f0858d9b8af946b5a7d08f2ee154ff8598b8672efd36
      Building wheel for control (setup.py) ... done
      Created wheel for control: filename=control-0.9.0-py2.py3-none-any.whl size=344920 sha256=4c9437a97c61c2bc307ad9ba2fa0704d65db2ba6159097f6c4cb18f9c32275df
      Stored in directory: /root/.cache/pip/wheels/35/be/ee/081b68ca3e4b2d253fba2f7f7e51960373903fb7b6d13c5532
      Building wheel for urdf-parser-py (setup.py) ... done
      Created wheel for urdf-parser-py: filename=urdf_parser_py-0.0.3-cp37-none-any.whl size=13122 sha256=99d3b7e4f8b1ce6f504d3c95b4983a28e956e3da1fd1327d845da7ea434a9b8f
      Stored in directory: /root/.cache/pip/wheels/61/30/cb/440cedefdba99ee2fa968bbf5bfcadc33e7433239b34b8fa88
      Building wheel for quadprog (setup.py) ... done
      Created wheel for quadprog: filename=quadprog-0.1.8-cp37-cp37m-linux_x86_64.whl size=336639 sha256=1c4cc436dcfc8d9789bf74f3b310584a1a130975add9807b408e0a4665d3512d
      Stored in directory: /root/.cache/pip/wheels/0c/39/5f/c712b0e462439dec9b30080cf34622f167929b39bd964d8baf
    Successfully built SrdPy control urdf-parser-py quadprog
    Installing collected packages: casadi, control, slycot, urdf-parser-py, quadprog, qpsolvers, SrdPy
    Successfully installed SrdPy-0.1 casadi-3.5.5 control-0.9.0 qpsolvers-1.6.1 quadprog-0.1.8 slycot-0.4.0 urdf-parser-py-0.0.3

```python
from SrdPy.URDFUtils import getLinkArrayFromURDF

from SrdPy.TableGenerators import generateConstraiedLinearModelTable
from SrdPy.TableGenerators import generateLinearModelTable
from SrdPy.LinksAndJoints import *
from SrdPy.Handlers import *
from SrdPy.InverseKinematics import *
from SrdPy.SymbolicUtils import *
from SrdPy.Loggers import *
from SrdPy.DynamicSolvers import *
from SrdPy.Controllers import *

from SrdPy.Visuals import Visualizer
from SrdPy import SymbolicEngine
from SrdPy import plotGeneric
from copy import deepcopy
from casadi import *

from SrdPy.TableGenerators import *
from SrdPy import Chain
from SrdPy import Profiler
import numpy as np
from scipy.integrate import solve_ivp
import os
```

```python
p = Profiler()
iiwaLinks = getLinkArrayFromURDF(os.path.abspath("./SrdPy/examples/iiwa/iiwa14.urdf"),True)
iiwaChain = Chain(iiwaLinks)

print(iiwaChain)
initialPosition = np.array([0.1701584,  -0.07586424,  0.42323069, -0.41380217,  0.01195943,  0.07095069,
  0.])

# CHANGE INITIAL POSITION HERE
# Data must produce a smooth condition number graph (with small maximum value), and a constant rank of 2.
initialPosition = np.array([0.5,  -0.2,  0.1, -0.4,  0.3,  0.7, 0.9])

blank_chain = deepcopy(iiwaChain)
blank_chain.update(initialPosition)

engine = SymbolicEngine(iiwaChain.linkArray)

deriveJacobiansForlinkArray(engine)
H = deriveJSIM(engine)

iN, dH = deriveGeneralizedInertialForces_dH(engine, H)
g = deriveGeneralizedGravitationalForces(engine)
d = deriveGeneralizedDissipativeForcesUniform(engine, 1)
T = deriveControlMap(engine)

description_gen_coord_model = generateDynamicsGeneralizedCoordinatesModel(engine,
                                                                          H=H,
                                                                          c=(iN + g + d),
                                                                          T=T,
                                                                          functionName_H="g_dynamics_H",
                                                                          functionName_c="g_dynamics_c",
                                                                          functionName_T="g_dynamics_T",
                                                                          casadi_cCodeFilename="g_dynamics_generalized_coordinates",
                                                                          path="./iiwa/Dynamics")

handlerGeneralizedCoordinatesModel = GeneralizedCoordinatesModelHandler(description_gen_coord_model)

description_linearization = generateDynamicsLinearization(engine,
                                                          H=H,
                                                          c=(iN + g + d),
                                                          T=T,
                                                          functionName_A="g_linearization_A",
                                                          functionName_B="g_linearization_B",
                                                          functionName_c="g_linearization_c",
                                                          casadi_cCodeFilename="g_dynamics_linearization",
                                                          path="./iiwa/Linearization")

handlerLinearizedModel = LinearizedModelHandler(description_linearization)

constraint6 = engine.links["iiwa_link_6"].absoluteFollower[0]

task = constraint6[:2]
print("task size is: ", task.size)

description_IK = generateSecondDerivativeJacobians(engine,
                                                task=task,
                                                functionName_Task="g_InverseKinematics_Task",
                                                functionName_TaskJacobian="g_InverseKinematics_TaskJacobian",
                                                functionName_TaskJacobianDerivative="g_InverseKinematics_TaskJacobian_derivative",
                                                casadi_cCodeFilename="g_InverseKinematics",
                                                path="./iiwa/InverseKinematics")

ikModelHandler = IKModelHandler(description_IK, engine.dof, task.shape[0])
```

    Unknown tag: material
    Unknown tag: self_collision_checking
    Unknown tag: material
    Unknown tag: material
    Unknown tag: material
    Unknown tag: material
    Unknown tag: material
    Unknown tag: material
    Unknown tag: material
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface
    Unknown tag: hardwareInterface

    Parsing URDF:/content/SrdPy/examples/iiwa/iiwa14.urdf
    Root node: base
    Chain
    Links:
    0. Ground
    1. iiwa_link_0
    2. iiwa_link_1
    3. iiwa_link_2
    4. iiwa_link_3
    5. iiwa_link_4
    6. iiwa_link_5
    7. iiwa_link_6
    8. iiwa_link_7

    Loaded existing .so at ./iiwa/Dynamics
    Loaded existing .so at ./iiwa/Linearization
    task size is:  2
    Starting writing function for the g_InverseKinematics_Task
    Starting writing function for the g_InverseKinematics_Task task jacobian
    Starting writing function for the derivative of g_InverseKinematics_Task
    Generated C code!

```python
IC_task = ikModelHandler.getTask(initialPosition)

task_1 = np.array([[0.1],  [0.3]])
# task_2 = np.array([[0.3],  [0.3]])
# task_3 = np.array([[0.3],  [0.1]])

# zeroOrderDerivativeNodes = np.hstack((IC_task, task_1, task_2, task_3))
zeroOrderDerivativeNodes = np.hstack((IC_task, task_1))

firstOrderDerivativeNodes = np.zeros(zeroOrderDerivativeNodes.shape)

secondOrderDerivativeNodes = np.zeros(zeroOrderDerivativeNodes.shape)

timeOfOneStage = 2
timeEnd = (len(zeroOrderDerivativeNodes[1]) - 1) * timeOfOneStage + 1
nodeTimes = np.arange(start=0, stop=timeEnd, step=timeOfOneStage)

handlerIK_taskSplines = IKtaskSplinesHandler(nodeTimes,
                                                zeroOrderDerivativeNodes,
                                                firstOrderDerivativeNodes,
                                                secondOrderDerivativeNodes)

timeTable = np.arange(handlerIK_taskSplines.timeStart, handlerIK_taskSplines.timeExpiration + 0.01, 0.01)

IKTable = generateIKTable(ikModelHandler, handlerIK_taskSplines, initialPosition, timeTable, method="lsqnonlin")
plotIKTable(ikModelHandler, timeTable, IKTable)

ikSolutionHandler = IKSolutionHandler(ikModelHandler, handlerIK_taskSplines, timeTable, IKTable, "linear")

tf = ikSolutionHandler.timeExpiration

n = handlerGeneralizedCoordinatesModel.dofConfigurationSpaceRobot

A_table, B_table, c_table, x_table, u_table, dx_table = generateLinearModelTable(handlerGeneralizedCoordinatesModel,handlerLinearizedModel,ikSolutionHandler,timeTable)

```

# Task

1. Implement control, but state observer (x can't be used by the controller, but $y = Cx$ can be)
2. Simulate the system with the observer and the controller
3. Simulate from different initial positions

### Deriving the dynamics of observer and controller
We have
\begin{equation}
\begin{cases}
\dot x = Ax+Bu+c\\
\dot{\hat x} = Ax+Bu + L(y - C \hat x) + c\\
y=Cx\\
u=-K (\hat x-x^*)+u^*
\end{cases}
\end{equation}
<br>
Collecting $\dot x$ and $\hat x$ we get:

<br>
\begin{equation}
\begin{bmatrix}
\dot x \\
\hat x
\end{bmatrix}
=
\begin{bmatrix}
A & -BK \\
LC & A-BK-LC
\end{bmatrix}
\begin{bmatrix}
x \\
\hat x
\end{bmatrix}
+
\begin{bmatrix}
BKx^*+Bu^*+c \\
BKx^*+Bu^*+c
\end{bmatrix}
\end{equation}
<br>

Let's introduce an abuse of notation
\begin{equation}
AA=
\begin{bmatrix}
A & -BK \\
LC & A-BK-LC
\end{bmatrix}
\end{equation}

<br>

\begin{equation}
cc=
\begin{bmatrix}
BKx^*+Bu^*+c \\
BKx^*+Bu^*+c
\end{bmatrix}
\end{equation}

Now we simulate the system with the observer and the controller

```python
# given the observer and the controller variables
# this function applies the formulas above and returns the AA and cc tables

def my_generateCloseLoopTable(A_table, B_table, C_table, K_Table, L_table, c_table, x_d_table, u_d_table):
    AA_table = []
    for i in range(count):
      AA_table.append(np.vstack((
                          np.hstack((A_table[i], -B_table[i].dot(K_table[i]))),
                          np.hstack((L_table[i].dot(C_table[i]), A_table[i]-B_table[i].dot(K_table[i])-L_table[i].dot(C_table[i])))
                      ))
                     )

    cc_table = []
    for i in range(count):
      cc_table.append(np.hstack((
                          B_table[i].dot(K_table[i]).dot(x_d_table[i]) + B_table[i].dot(u_d_table[i]) + c_table[i],
                          B_table[i].dot(K_table[i]).dot(x_d_table[i]) + B_table[i].dot(u_d_table[i]) + c_table[i]
                      ))
                     )
    return np.array(AA_table), np.array(cc_table)
```

```python
# this function turns every matrix in a table of matrices and returns the modified table
def trn_table(t):
  ret = []
  for i in t:
    ret.append(i.T)
  return np.array(ret)
```

```python
from control import lqr

# this function generate solves the LQR multiple times and returns a table of resutls
def my_generateLQRTable(A_table, B_table, Q_table, R_table):
    K_table = []

    for i in range(A_table.shape[0]):
        K, _, _ =  lqr(A_table[i], B_table[i], Q_table[i], R_table[i])
        K_table.append(K)

    return np.array(K_table)
```

```python
count = A_table.shape[0]

C = np.concatenate((np.eye(n), np.zeros((n, n))), axis=1) # y = C*x
C_table = np.array(count * [C])

Q = 0.1 * np.eye(2 * n)
R = 10 * np.eye(handlerGeneralizedCoordinatesModel.dofControl)

# Controller gains
K_table = my_generateLQRTable(A_table, B_table, np.tile(Q, [count, 1, 1]), np.tile(R, [count, 1, 1]))

# Observer gains
L_table = trn_table(my_generateLQRTable(trn_table(A_table), trn_table(C_table), np.tile(Q, [count, 1, 1]), np.tile(R, [count, 1, 1])))

# Generate closed-loop system
AA_table, cc_table = my_generateCloseLoopTable(A_table, B_table, C_table, K_table, L_table, c_table, x_table, u_table)

# Initial positions, initially, the observed values are the same as the actual values
x0 = np.hstack((initialPosition, np.zeros(n), initialPosition, np.zeros(n)))

# Now we have everything we needed for simulation
# from sympy import *
# display(Matrix(x0))
```

```python
# Simulation
ode_fnc_handle = ClosedLoopLinearSystemOdeFunctionHandler(AA_table, cc_table, timeTable)
sol = solve_ivp(ode_fnc_handle, [0, tf], x0, t_eval=timeTable,method="RK45")

# plot graphs
time_table_0 = sol.t
solution_tape_1 = sol.y[:n*2].T # Actual states
solution_tape_2 = sol.y[n*2:].T # Observed states

plotGeneric(time_table_0, solution_tape_1, figureTitle="", ylabel="ODE1")
plotGeneric(time_table_0, solution_tape_2, figureTitle="", ylabel="ODE2")

plotGeneric(timeTable, solution_tape_1[:, :n], figureTitle="position_actual", ylabel="q", plot=True)
plotGeneric(timeTable, solution_tape_2[:, :n], figureTitle="position_observed", ylabel="q", plot=True)

plotGeneric(timeTable, abs(solution_tape_1[:, :n] - solution_tape_2[:, :n]), figureTitle="position_error", ylabel="e", plot=True)

plotGeneric(timeTable, solution_tape_1[:, n:2*n], figureTitle="velocity_actual", ylabel="v", plot=True)
plotGeneric(timeTable, solution_tape_2[:, n:2*n], figureTitle="velocity_observed", ylabel="v", plot=True)

plotGeneric(timeTable, abs(solution_tape_1[:, n:2*n] - solution_tape_2[:, n:2*n]), figureTitle="velocity_error", ylabel="e", plot=True)

with open('anim_array.npy', 'wb') as f:
    np.save(f, solution_tape_2[:,:n])
```

```python
chainLinks = getLinkArrayFromURDF(os.path.abspath("./SrdPy/examples/iiwa/iiwa14.urdf"),True)
chain = Chain(chainLinks)

print(chain)
blank_chain = deepcopy(chain)
blank_chain.update(initialPosition)
with open('anim_array.npy', 'rb') as f:
    q = np.load(f)

blank_chain.update(q[0])
plotGeneric(np.arange(q.shape[0]),q,plot=True)
vis = Visualizer()
vis.animate(blank_chain,q,framerate=0.1,showMeshes=True)
```

Click the link above to see animation ^^^


# Abstract

4Chan is famous for it's anonymous and ephemeral boards. But is it really ephemeral? There are multiple internet archiving services that have been keeping copies of 4chan posts that go many years in the past. We looked into it to asnwer the following questions: Q1.) *Are 4chan archiving services reliable and offer a holistic view of the epehemeral 4chan threads?* q2.) *If (Q1) is true, does this provides any insights into the pseudo anonymity of the site?*

```python
%%capture
# needed to support mongo+srv connection strings
!pip install dnspython
!pip install pymongo
!pip install pandas
!pip install numpy
!pip install matplotlib
!pip install scipy
!pip install gensim
!pip install langdetect
!pip install tqdm
!pip install nltk
!pip install seaborn
!pip install pyLDAvis
!pip install wordcloud

## Only run once per kernel environment
## Doesn't need to be re-run when kernel is restarted
#import nltk
#nltk.download('punkt')
#nltk.download('averaged_perceptron_tagger')
#nltk.download('wordnet')
#nltk.download('stopwords')

```

```python
%%capture

from pymongo import MongoClient
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# Fixed period analysis - change these to extend/restrict the period
# Date as a yyyymmddhhMMss string
date_from = '202105010000'
date_to =   '202111010000'

# connectiong to mongodb cluster using a read only user

## REMOTE REPLICA SET
client = MongoClient("mongodb+srv://<usr>:<pass>@rs.prifti.us/?ssl=false&authSource=chan")

## LOCAL SINGLE INSTANCE
## 192.168.1.110 -- localhost
# client = MongoClient("mongodb://<usr>:<pass>@localhost:27777/?ssl=false&serverSelectionTimeoutMS=5000&connectTimeoutMS=10000&authSource=chan&authMechanism=SCRAM-SHA-256")

# connection to chan database
db = client.chan

# the threads collection contains API data collections from the 4chan api and 4pleb api
threads_collection = db['y-threads']

```

```python
#%%capture

# MongoDB aggregation query, joining data from thread and 'live-4chan' collection
# for each discovered 4chan thread - retreive a boolean representing if the API data was found

q1_pipeline = [
     {"$match": {"pleb": {"$exists": True } } }
    ,{ "$addFields": {
        "data": {
                "$ifNull": [ {"$objectToArray": "$pleb"}, [ ] ]
          }
     } }
    ,{ "$unwind": { "path": "$data", "preserveNullAndEmptyArrays": True } }
    ,{ "$project":{"posted": "$data.v.op.fourchan_date", "timestamp":"$data.v.op.timestamp", "thread_num": "$data.v.op.thread_num", "data": "$data.v", "url":"$value" } }
    ,{ "$addFields": {
        "data.flat_posts": {
                "$ifNull": [ {"$objectToArray": "$data.posts"}, [ ] ]
          }
     } }
    ,{"$project":{"posted": 1, "timestamp":1, "thread_num": 1, "data_op": "$data.op", "data_posts": "$data.flat_posts", "url": 1 } }
    ,{"$sort": {"thread_num": -1} }
    ,{"$limit": 10}
]

q1_pipeline

# run the aggregation
thread_data = pd.DataFrame(threads_collection.aggregate(q1_pipeline, allowDiskUse=True))

```

```python
thread_data
```

      _id
      posted
      timestamp
      thread_num
      url
      data_op
      data_posts

      0
      https://boards.4chan.org/pol/thread/346097093
      11/4/21(Thu)17:07
      1636060063
      346097093
      https://boards.4chan.org/pol/thread/346097093
      {'doc_id': '322233543', 'num': '346097093', 's...
      [{'k': '346097614', 'v': {'doc_id': '322234047...

      1
      https://boards.4chan.org/pol/thread/346097084
      11/4/21(Thu)17:07
      1636060059
      346097084
      https://boards.4chan.org/pol/thread/346097084
      {'doc_id': '322233544', 'num': '346097084', 's...
      [{'k': '346097546', 'v': {'doc_id': '322233977...

      2
      https://boards.4chan.org/pol/thread/346096992
      11/4/21(Thu)17:06
      1636060015
      346096992
      https://boards.4chan.org/pol/thread/346096992
      {'doc_id': '322233484', 'num': '346096992', 's...
      [{'k': '346097079', 'v': {'doc_id': '322233534...

      3
      https://boards.4chan.org/pol/thread/346096826
      11/4/21(Thu)17:05
      1636059949
      346096826
      https://boards.4chan.org/pol/thread/346096826
      {'doc_id': '322233273', 'num': '346096826', 's...
      [{'k': '346097342', 'v': {'doc_id': '322233756...

      4
      https://boards.4chan.org/pol/thread/346096822
      11/4/21(Thu)17:05
      1636059948
      346096822
      https://boards.4chan.org/pol/thread/346096822
      {'doc_id': '322233275', 'num': '346096822', 's...
      [{'k': '346096988', 'v': {'doc_id': '322233411...

      5
      https://boards.4chan.org/pol/thread/346096745
      11/4/21(Thu)17:05
      1636059920
      346096745
      https://boards.4chan.org/pol/thread/346096745
      {'doc_id': '322233274', 'num': '346096745', 's...
      [{'k': '346097218', 'v': {'doc_id': '322233647...

      6
      https://boards.4chan.org/pol/thread/346096723
      11/4/21(Thu)17:05
      1636059910
      346096723
      https://boards.4chan.org/pol/thread/346096723
      {'doc_id': '322233180', 'num': '346096723', 's...
      [{'k': '346096821', 'v': {'doc_id': '322233234...

      7
      https://boards.4chan.org/pol/thread/346096692
      11/4/21(Thu)17:04
      1636059899
      346096692
      https://boards.4chan.org/pol/thread/346096692
      {'doc_id': '322233177', 'num': '346096692', 's...
      [{'k': '346096954', 'v': {'doc_id': '322233423...

      8
      https://boards.4chan.org/pol/thread/346096690
      11/4/21(Thu)17:04
      1636059898
      346096690
      https://boards.4chan.org/pol/thread/346096690
      {'doc_id': '322233172', 'num': '346096690', 's...
      [{'k': '346097011', 'v': {'doc_id': '322233424...

      9
      https://boards.4chan.org/pol/thread/346096598
      11/4/21(Thu)17:04
      1636059858
      346096598
      https://boards.4chan.org/pol/thread/346096598
      {'doc_id': '322233113', 'num': '346096598', 's...
      [{'k': '346096938', 'v': {'doc_id': '322233387...

# Introduction

We have been collecting posts from the the /pol board of 4chan with high frequency (scaning the board multiple times per minute).

We have done so by simulating an end user that connects to the side with a conventional borwser (Firefox), using [OXPath](https://github.com/oxpath/oxpath)

We have been taking multiple snapshots of each thread until the thread was archived. When a thread is archived we then have been taking a snapshot of the thread from the 4chan api and one from 4pleb.org, an 4chan archiving service, believed to be the most complete archiving service on 4chan.

Two type of agents have been used to collect the data using OXpath.

- A discovery crawling agent that continuously crawls the /pol board looking for new threads and either a.) captures and queues the new thread URLs for scraping or b.) updates the *found* timestamp of a thread that had already been found by previous instances of the crawler.

- A extraction scraping agent that continuously scrapes threads queued by the crawling agent and stores the state of the thread together with the timestamp, unless the agent finds the thread to be *archived* or *deleted*. When the thread is found in the *archived* / *deleted* state, its attributes are updated and the thread is removed from the crawling queue.

```python
# %%capture

thread_op = pd.DataFrame(list(thread_data['data_op']))
thread_op = thread_op.drop(columns=['doc_id', 'subnum', 'thread_num', 'op', 'capcode', 'email', 'name', 'exif'])
thread_op = thread_op.drop(columns=['trip','sticky', 'locked', 'poster_hash_processed', 'poster_country_name_processed', 'troll_country_code', 'troll_country_name', 'since4pass', 'unique_ips', 'extra_data', 'media', 'board'])
thread_op = thread_op.drop(columns=['trip_processed', 'email_processed', 'name_processed', 'formatted'])

thread_op
#thread_posts.columns
```

      num
      timestamp
      timestamp_expired
      title
      comment
      poster_hash
      poster_country
      deleted
      nreplies
      nimages
      fourchan_date
      comment_sanitized
      comment_processed
      title_processed
      poster_country_name

      0
      346097093
      1636060063
      0
      Indiana Jones 5 camera operator found dead
      >be maxvaxxed to work in Hollywood\n>die in yo...
      Far+1kwU
      FR
      0
      None
      None
      11/4/21(Thu)17:07
      >be maxvaxxed to work in Hollywood\n>die in yo...
      <span class="greentext">&gt;be maxvaxxed to wo...
      Indiana Jones 5 camera operator found dead
      France

      1
      346097084
      1636060059
      0
      Interracial Romance in Hollywood
      Mark Dice's "Hollywood Propaganda" prepared me...
      m2MPJ1go
      US
      0
      None
      None
      11/4/21(Thu)17:07
      Mark Dice's "Hollywood Propaganda" prepared me...
      Mark Dice's &quot;Hollywood Propaganda&quot; p...
      Interracial Romance in Hollywood
      United States

      2
      346096992
      1636060015
      0
      Reminder Moving Out is Memed by the Elite
      Moving out is a meme of the elite so that cons...
      CoC3qC0h
      US
      0
      None
      None
      11/4/21(Thu)17:06
      Moving out is a meme of the elite so that cons...
      Moving out is a meme of the elite so that cons...
      Reminder Moving Out is Memed by the Elite
      United States

      3
      346096826
      1636059949
      0
      Polygamy will return.
      The covid injection bioweapon, based on past a...
      vuxbaAOu
      US
      0
      None
      None
      11/4/21(Thu)17:05
      The covid injection bioweapon, based on past a...
      The covid injection bioweapon, based on past a...
      Polygamy will return.
      United States

      4
      346096822
      1636059948
      0
      JOB FORCED JAB OR SLAB
      as a christian and religious leader how would ...
      Ab5voORM
      US
      0
      None
      None
      11/4/21(Thu)17:05
      as a christian and religious leader how would ...
      as a christian and religious leader how would ...
      JOB FORCED JAB OR SLAB
      United States

      5
      346096745
      1636059920
      0
      None
      Has this ever actually been successful before?...
      60jD9ne1
      US
      0
      None
      None
      11/4/21(Thu)17:05
      Has this ever actually been successful before?...
      Has this ever actually been successful before?...
      None
      United States

      6
      346096723
      1636059910
      0
      Can you buy Ivermectin in Spain?
      Asking for a friend
      nWeopydt
      ES
      0
      None
      None
      11/4/21(Thu)17:05
      Asking for a friend
      Asking for a friend
      Can you buy Ivermectin in Spain?
      Spain

      7
      346096692
      1636059899
      0
      Jews = Brahmins =White Christians
      All of these are assholes because they have hu...
      N+X/7+b9
      CA
      0
      None
      None
      11/4/21(Thu)17:04
      All of these are assholes because they have hu...
      All of these are assholes because they have hu...
      Jews = Brahmins =White Christians
      Canada

      8
      346096690
      1636059898
      0
      Why cant women turn anything off/lock up?
      >be me\n>get home from long day at work\n>fron...
      j1uGUd/p
      US
      0
      None
      None
      11/4/21(Thu)17:04
      >be me\n>get home from long day at work\n>fron...
      <span class="greentext">&gt;be me</span><br />...
      Why cant women turn anything off/lock up?
      United States

      9
      346096598
      1636059858
      0
      Do you think this guy owns a tie?
      Chauvin thing was way more entertaining.
      ugkY0E/9
      None
      0
      None
      None
      11/4/21(Thu)17:04
      Chauvin thing was way more entertaining.
      Chauvin thing was way more entertaining.
      Do you think this guy owns a tie?
      False

```python
# %%capture

deleted_threads['articles'] = deleted_threads['title'] + ' ' + deleted_threads['comment']
document_corp = list(deleted_threads['articles'])
document_corp = [x for x in document_corp if x is not None]
document_corp = pd.DataFrame(document_corp)
#document_corp.shape
document_corp.columns = ['articles']
document_corp = document_corp.dropna().reset_index(drop=True)
#temp_title = list(deleted_threads.query('title != None')['title'])
#temp_title = [x for x in temp_title if x is not None]

#temp_comments = list(deleted_threads.query('comment != None')['comment'])
#temp_comments = [x for x in temp_comments if x is not None]
#document_corp = temp_title + temp_comments
#len(document_corp)

```

```python
# %%capture

from langdetect import detect
from tqdm import notebook
notebook.tqdm().pandas()
data = document_corp.copy()

def try_detect(elem):
    try:
        return detect(elem)
    except:
        return 'n/a'

data['lang'] = data.articles.progress_map(try_detect)
data['lang'].head(5)

```

    HBox(children=(HTML(value=''), FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0…

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8537.0), HTML(value='')))

    0    en
    1    pl
    2    en
    3    en
    4    en
    Name: lang, dtype: object

```python
# %%capture

from nltk.tokenize import sent_tokenize

data.lang.value_counts()
data_en = data.loc[data.lang=='en']
data['sentences'] = data.articles.progress_map(sent_tokenize)
data['sentences'].head(1).tolist()[0][:3] # Print the first 3 sentences of the 1st article
```

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8537.0), HTML(value='')))

    ['DRY ASS PUSSY Imagine the smell.']

```python
# %%capture

from nltk.tokenize import word_tokenize
data['tokens_sentences'] = data['sentences'].progress_map(lambda sentences: [word_tokenize(sentence) for sentence in sentences])
print(data['tokens_sentences'].head(1).tolist()[0][:3])

```

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8537.0), HTML(value='')))

    [['DRY', 'ASS', 'PUSSY', 'Imagine', 'the', 'smell', '.']]

```python
# %%capture

from nltk import pos_tag
data['POS_tokens'] = data['tokens_sentences'].progress_map(lambda tokens_sentences: [pos_tag(tokens) for tokens in tokens_sentences])
print(data['POS_tokens'].head(1).tolist()[0][:3])
```

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8537.0), HTML(value='')))

    [[('DRY', 'NNP'), ('ASS', 'NNP'), ('PUSSY', 'NNP'), ('Imagine', 'NNP'), ('the', 'DT'), ('smell', 'NN'), ('.', '.')]]

```python
# %%capture

# Inspired from https://stackoverflow.com/a/15590384
from nltk.corpus import wordnet
from nltk.stem.wordnet import WordNetLemmatizer

def get_wordnet_pos(treebank_tag):

    if treebank_tag.startswith('J'):
        return wordnet.ADJ
    elif treebank_tag.startswith('V'):
        return wordnet.VERB
    elif treebank_tag.startswith('N'):
        return wordnet.NOUN
    elif treebank_tag.startswith('R'):
        return wordnet.ADV
    else:
        return ''

lemmatizer = WordNetLemmatizer()

# Lemmatizing each word with its POS tag, in each sentence
data['tokens_sentences_lemmatized'] = data['POS_tokens'].progress_map(
    lambda list_tokens_POS: [
        [
            lemmatizer.lemmatize(el[0], get_wordnet_pos(el[1]))
            if get_wordnet_pos(el[1]) != '' else el[0] for el in tokens_POS
        ]
        for tokens_POS in list_tokens_POS
    ]
)

data['tokens_sentences_lemmatized'].head(1).tolist()[0][:3]
```

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8537.0), HTML(value='')))

    [['DRY', 'ASS', 'PUSSY', 'Imagine', 'the', 'smell', '.']]

```python
# %%capture
from nltk.corpus import stopwords

stopwords_verbs = ['https', 'http', 'say', 'get', 'go', 'know', 'may', 'need', 'like', 'make', 'see', 'want', 'come', 'take', 'use', 'would', 'can']
stopwords_other = ['one', 'mr', 'image', 'getty', 'de', 'en', 'caption', 'also', 'copyright', 'something']
my_stopwords = stopwords.words('english') + stopwords_verbs + stopwords_other

```

```python
# %%capture

from itertools import chain # to flatten list of sentences of tokens into list of tokens
data['tokens'] = data['tokens_sentences_lemmatized'].map(lambda sentences: list(chain.from_iterable(sentences)))
data['tokens'] = data['tokens'].map(lambda tokens: [token.lower() for token in tokens if token.isalpha()
                                                    and token.lower() not in my_stopwords and len(token)>1])
data['tokens'].head(1).tolist()[0][:30]

```

    ['dry', 'ass', 'pussy', 'imagine', 'smell']

```python
# %%capture

from gensim.models import Phrases
from gensim import corpora

tokens = data['tokens'].tolist()
bigram_model = Phrases(tokens)
trigram_model = Phrases(bigram_model[tokens], min_count=1)
tokens = list(trigram_model[bigram_model[tokens]])

dictionary_LDA = corpora.Dictionary(tokens)
dictionary_LDA.filter_extremes(no_below=3)
corpus = [dictionary_LDA.doc2bow(tok) for tok in tokens]

```

```python
# %%capture

from gensim import models
import numpy as np

np.random.seed(123456)
num_topics = 20
%time lda_model = models.LdaModel(corpus, num_topics=num_topics, \
                                  id2word=dictionary_LDA, \
                                  passes=4, alpha=[0.01]*num_topics, \
                                  eta=[0.01]*len(dictionary_LDA.keys()))

```

    CPU times: user 30 s, sys: 14.1 ms, total: 30 s
    Wall time: 30 s

```python
# %%capture

import seaborn as sns;

topics = [lda_model[corpus[i]] for i in range(len(data))]

def topics_document_to_dataframe(topics_document, num_topics):
    res = pd.DataFrame(columns=range(num_topics))
    for topic_weight in topics_document:
        res.loc[0, topic_weight[0]] = topic_weight[1]
    return res

# Like TF-IDF, create a matrix of topic weighting, with documents as rows and topics as columns
document_topic = pd.concat([topics_document_to_dataframe(topics_document, num_topics=num_topics) for topics_document in topics]) \
  .reset_index(drop=True).fillna(0)

```

```python
# %%capture

document_topic.head()

%matplotlib inline
sns.set(rc={'figure.figsize':(10,20)})
sns.heatmap(document_topic.loc[document_topic.idxmax(axis=1).sort_values().index])
```

```python
# %%capture

sns.set(rc={'figure.figsize':(10,5)})
document_topic.idxmax(axis=1).value_counts().plot.bar(color='lightblue')
```

```python
# %%capture

# https://cran.r-project.org/web/packages/LDAvis/vignettes/details.pdf
# Here a short legend to explain the vis:
# size of bubble: proportional to the proportions of the topics across the N total tokens in the corpus
# red bars: estimated number of times a given term was generated by a given topic
# blue bars: overall frequency of each term in the corpus
# -- Relevance of words is computed with a parameter lambda
# -- Lambda optimal value ~0.6 (https://nlp.stanford.edu/events/illvi2014/papers/sievert-illvi2014.pdf)
%matplotlib inline
import pyLDAvis
import pyLDAvis.gensim
vis = pyLDAvis.gensim.prepare(topic_model=lda_model, corpus=corpus, dictionary=dictionary_LDA)
pyLDAvis.enable_notebook()
pyLDAvis.display(vis)

```

<link rel="stylesheet" type="text/css" href="https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css">

```python
# %%capture
from wordcloud import WordCloud,STOPWORDS

def showWordCloud(data):
    words = ' '.join(data)
    cleaned_word = " ".join([word for word in words.split()])
    wordcloud = WordCloud(stopwords = STOPWORDS,
                         background_color = 'black',
                         width = 2500,
                         height = 2500
                         ).generate(cleaned_word)
    plt.figure(1,figsize = (13,13))
    plt.imshow(wordcloud)
    plt.axis('off')
    plt.show()

flat_list = [item for sublist in tokens for item in sublist]

#flat_list
showWordCloud(flat_list)
```

# Comparison Questions

### Q1.) ***What is the archiving service 4plen coverage of 4chan and what is the quality of the coverage?***

Between the dates of **{{_from_date.strftime("%d/%m/%Y, %H:%M:%S") }}** and **{{_to_date.strftime("%d/%m/%Y, %H:%M:%S") }}** we have crawled **{{ "{:,}".format(len(match_result)) }}** threads on 4chan /pol board.

Of these, we have collected data from all three sources - 4chan scrap using OXPath, 4chan API and 4pleb API (i.e. scraped thread, posts content and metadata) on **{{full_match_len}} / {{"%.3f" % (full_match_len/len(match_result)*100)}} %** of the results.

We have found on the 4chan archiving service, 4pleb, **{{match_result.pleb.sum()}}  / {{"%.3f" % perc_match }}%** of the results.

Looking into the remainder **{{ len(match_result) - match_result.pleb.sum() }} / {{"%.3f" % perc_not_match }}%** that are not found in the archiving service we have noticed that:

- We have scraped **{{ len(not_pleb_yes_chan) }} / {{ "%.3f" % (len(not_pleb_yes_chan)/len(not_pleb_match)*100) }}%** threads from 4chan API but not found on 4pleb.
- We have scraped **{{ len(not_pleb_yes_chan_or_oxp) }} / {{ "%.3f" % (len(not_pleb_yes_chan_or_oxp)/len(not_pleb_match)*100) }}** using OXPath or 4chan API scraping methods. Of these **{{ len(not_pleb_match.query("oxp")) }}** is the number of items scraped using OXPath.

The analysis of the deleted threads could lead to insight on how threads are being monitored

- There are **{{ "{:,}".format(len(match_result.query("deleted==True"))) }}** threads that have been marked as deleted by the OXPath scraper. Of the deleted threads, **{{ "{:,}".format(len(match_result.query("pleb").query("deleted==True"))) }}** have been found been found on the 4pleb archiving API.
- There are **{{ len(id_chan_posts) }}** threads found on 4chan API that have been marked as deleted. **{{ num_del_chan }}** have also been found and marked as deleted by OXpath scraper
- There are **{{ len(pleb_posts.query("deleted=='1'")) }}** threads marked as deleted in 4pleb. **{{len(match_result[match_result['num'].isin(id_pleb_posts)].query('deleted==True'))}}** have also been found and marked as deleted by OXpath scraper

The deleted threads seems to behave according to the following pattern:

* Threads are posted by original poster (op)
* There is some activity on the thread (replies and/or reads)
* The thread becomes in-accessible and its URL return *404 - Page not found*
    * The OXPath scraper marks the thread as deleted
* Some threads are deleted permanently whilst others are returned on the main board and the activity continues
    * We can assume that **{{ "%.3f" % ( (len(pleb_posts.query("deleted=='1'")) / len(match_result.query("pleb").query("deleted==True"))) * 100)  }}%** of the cases that 4pleb also finds to be deleted, is the actual number of deleted threads

### Q1.1) - What topics of threads are being deleted and what is the difference between the ones permanently deleted and the ones put back on the board?

When analysing deleted threads we have to take into consideration the following question:
    ***How do we know when a thread has been deleted?***

The question is non trivial because ultimately ***all*** threads get deleted from 4chan. We are interested in a ***deleted*** status of the thread outside of it's expected life cycle.  The live expectancy of a thread $L_t$ (i.e. the elapsed amount of time during which a thread is reachable by it's uniquer URL) is composed of two components: $E_b$ - *Elapsed* time on the *board* and $E_a$ - *Elapsed* time *archived*.

\begin{equation} \label{eq1}
    L_t = E_b + E_a
\end{equation}

The second component $E_a$ is expected to be constant. All threads remain available for three days after they are archived. (add reference)

The first element, the elapsed time a thread spends on the board $E_b$, is an entity of time, and function of activity and janitor moderation. The activity on a thread is a combination of posts/images/authors. We can simplify this by only considering the number of posts (i.e. replies). Each new post bumps the thread up on the board, delaying it's archiving time (add reference) . The bumping effect is also a function that decays with time. Alternatively, since $E_b$ already represents time, it can be expressed based on the number of posts $p$. Similarly, the rate of decay $\beta$  grows as the number of posts on the thread grows (each posts bumps the thread up a bit, but every time less than the previous).

Furthermore, emerging from data, at times threads gets removed from the board and appear to be deleted. This is expected as 4chan has various levels of moderation by its *janitors*. Since threads can also be in a deleted state and then put back in the live board we are going to consider $P(d)$ as the probability of a threat being deleted with it's values $P(d) \in [0,1]  where  P(d)=0$ thread was not deleted, $P(d)=1$ thread was deleted, $P(d) \in (0,1)$ thread was deleted for a certain amount of time as percentage of its elapsed time and then put back in the live board.

\begin{equation}
    E_b = [log(np)(1-\frac{p}{max_p})^{log(p)}]*(1-P(d))
\end{equation}

Since we know that n = 200, and assuming $P(d)$ only affects threads in the live board

\begin{equation}
    L_t = [log(200p)(1-\frac{p}{max_p})^{log(p)}]*(1-P(d)) + E_a
\end{equation}

where

- $L_t$ Elapsed life (time) of a thread
- $n$ Number of threads on the board
- $p$ Number of posts on the thread
- $max_p$ The maximum number of posts any thread on the board has
- $\beta = p/max_p$ rate of decay of the thread through out the board
- $P(d)$ The probability that a thread has been deleted
- $E_a$ The elapsed time during which the thread is available in the archived board

Should a.) Load all threads, nr_replies, elapsed time between start date and last post, deleted flag b.) plot calculated time and observed time, c.) check difference and formula fit.

**(question outside of main quest - dropping it)**

```python
# %%capture
print('Plot showing the distribution of number of replies for missed threads)')

f2 = f1[f1.chan == True]
chan_op = pd.DataFrame(list(f2['chan_op']))
chan_op.drop(axis=1, inplace=True, columns=['md5', 'resto', 'imagelimit', 'troll_country'])#, 'm_img', 'tail_size', 'trip'])
chan_op.drop(axis=1, inplace=True, columns=['ext',	'w',	'h',	'tn_w',	'tn_h',
                      'tim',	'time',	'fsize',	'country',
                      'archived',	'bumplimit',	'semantic_url',
                      'country_name',	'images'])
chan_op = chan_op.sort_values('replies', ascending=False)
plt.hist(chan_op.replies, bins=20)

```

```python
# %%capture
import time
from datetime import datetime

print('Plot showing the time to being arcived of the threads not captured)')

time_op = pd.DataFrame([chan_op.no, chan_op.archived_on, chan_op.now])
time_op = time_op.transpose()
time_op['now'] = pd.to_datetime(time_op['now'] , format='%m/%d/%y(%a)%X')
#time_op['archived_on'] = time_op['archived_on'].apply(time.localtime)
time_op['archived_on'] = time_op['archived_on'].apply(datetime.fromtimestamp)
time_op['length'] =  (time_op.archived_on - time_op.now).astype('timedelta64[h]')
time_op = time_op.sort_values(['length'])
plt.hist(time_op.length, bins=20)

```

```python
# %%capture
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams.update({'figure.figsize':(7,5), 'figure.dpi':100})

plt.hist(count, bins=50)
plt.gca().set(title='Frequency Histogram', ylabel='Frequency');

```

```python
# %%capture
pipeline = [
            {"$match": {"api_scrap": True } }, { "$addFields": {
        "pleb_array": {
                "$ifNull": [ {"$objectToArray": "$pleb"}, [ ] ]
        }
    } },
    {"$project": {"_id": 0, "url": "$_id", "api_chan_posts": { "$size": { "$ifNull": [ "$chan.posts", [ ] ] } } ,
                "api_chan_op": {"$arrayElemAt": ["$chan.posts", 0 ] }, "pleb_array": {"$arrayElemAt": ["$pleb_array", 0 ] } } },
    {"$project": {"url": 1, "api_chan_posts": 1, "api_chan_op": 1
                ,"api_pleb_posts":  {"$ifNull": [ {"$objectToArray": "$pleb_array.v.posts"}, [ ] ] }
                ,"api_pleb_op": "$pleb_array.v.op"  } } ,
     {"$project": {"url": 1, "api_chan_posts": 1, "api_chan_op": 1, "api_pleb_op": 1, "api_pleb_posts": {"$size": "$api_pleb_posts"} } }
]

thread_data = pd.DataFrame(threads.aggregate(pipeline, allowDiskUse=True))

```

```python
# %%capture
pipeline = [
    {"$match": {"$and": [{ "data": {"$ne": None } },{"data": {"$type": "array" } } ] } }
   ,{"$project": { "url": 1, "chan_posts": {"$arrayElemAt": ["$data", -1 ] } } }
   ,{"$project": { "url": 1, "chan_posts": {"$arrayElemAt": ["$chan_posts.payload.thread", 0] } } }
   ,{"$project": { "_id":0, "url": 1, "chan_posts": {"$size": {"$ifNull": [ "$chan_posts.replies", [ ] ] } },
       "chan_op": "$chan_posts.op" } }
   ,{"$unwind": "$chan_op"}
]

original_thread_data = pd.DataFrame(livechan.aggregate(pipeline, allowDiskUse=True))
len(original_thread_data)
# sum(original_thread_data['chan_posts'])

```

```python
# %%capture
import json
#plt.plot(thread_data['api_chan_posts'] - thread_data['api_bleb_posts'])
merged_data = pd.merge(thread_data, original_thread_data, on=["url"])
merged_data['diff_posts'] = abs(merged_data['api_chan_posts'] - merged_data['api_pleb_posts'])
merged_data = merged_data.sort_values(by=['diff_posts'], ascending=False)
print(f'OXpath Threads {len(original_thread_data)}')
print(f'4Pleb and 4 Chan Threads {len(thread_data)}')
print(f'Merged match {len(merged_data)}')

# merged_data.api_chan_op[merged_data.api_chan_op == None]['api_chan_op'] = { }
# merged_data.api_chan_op1 = pd.DataFrame(list(merged_data.api_chan_op))

```

```python
# %%capture
import numpy
import scipy.stats as ss
no_api_chan = merged_data[~merged_data.api_chan_op.isnull()].sort_values(by=['diff_posts'], ascending=False)
no_api_pleb = no_api_chan[~no_api_chan.api_pleb_op.isnull()].sort_values(by=['diff_posts'], ascending=False)
#no_api_pleb = no_api_pleb[no_api_pleb.diff_posts > 0]
#print(no_api_pleb.diff_posts.mean())
#print(no_api_pleb.diff_posts.std())
#ss.describe(no_api_pleb.diff_posts)
#diff_pdf = ss.norm.pdf(no_api_pleb.diff_posts, 0, 1)
#plt.plot(no_api_pleb.diff_posts, diff_pdf, label='pdf')

mu = 0
sigma = 1
diff_cdf = ss.norm.cdf(no_api_pleb.diff_posts, mu, sigma)
diff_pdf = ss.norm.pdf(no_api_pleb.diff_posts, mu, sigma)

fig=plt.figure()
ax=fig.add_subplot(111)
ax.plot(no_api_pleb.diff_posts, diff_pdf, label='pdf')
ax.plot(no_api_pleb.diff_posts, diff_cdf, label='cdf')
ax.set_xlim(xmin=0.0, xmax=10)
ax.legend()
plt.show()

```

### Q2.) What is the persentage of posts that can be matched on all three crawling methods and what are the gaps

Whilst looking at the thread coverage from the archiving sites, we can see promising results with matching results well above $95.00$% another question is around the quality of the coverage. That is, do the threads discovered and retrieved from the archiving services (in this case 4pleb API) have the same information as what retrieved from 4chan API and/or OXPath scraping method?

The best way to look at quality would be to:
1. verify that the thread contains and matches some of the key metadata (such as date it was posted, date it was archived, author country)
2. verify that the thread contains and matches its main content (such as subject, comment, media) and lastly
3. verify that the linked posts match with what is reported from 4chan API and/or OXPath scraping method.

### Q3.) What can be said about the significat amount of threads that get flagged/marked as "deleted/404" and either permanently deleted or returned to the board. What topics get permanently deleted? What topics get flagged?