## Présentation

Twitter est un réseau social permettant aux utilisateurs de publier des informations et communiquer entre eux par le biais de messages, appelés tweets, pouvant contenir jusqu'à 280 caractères. Largement utilisé aujourd'hui, ce réseau peut être un outil pour des entreprises qui souhaitent évaluer l'avis de leurs clients.

Dans ce TP, on se met à la place d'une compagnie aérienne, qui souhaiterait détecter les tweets qui la mentionnent et analyser si ce sont des mentions positives ou négatives, en comparant leur résultat avec les autres compagnies.

Le *preprocessing* est une tâche cruciale en fouille de données. Elle permet de transformer les données brutes en un format adapté à l'application de méthodes de machine learning.

# INF8111 - Fouille de données

## TP1 Automne 2019 - Preprocessing de tweets pour de l'analyse de sentiments

##### Membres de l'équipe:

    - Amine Bellahsen
    - Abderrahim Khalifa
    - Hamza Sghir

# I/ Analyse de sentiments (13 Pts)

Usuellement dans la littérature, la tâche d'extraire le sentiment d'un texte est appelé *sentiment analysis*.
Ici pour se faire, nous allons utiliser un modèle *bag-of-words* (BoW).

## 1. Installation

Pour ce TP, vous aurez besoin des librairies `numpy`, `sklearn` et `scipy` (que vous avez sans doute déjà), ainsi que la librairie `nltk`, qui est une libraire utilisée pour faire du traitement du language (Natural Language Processing, NLP)

Installez les libraires en question et exécutez le code ci-dessous :

```python
# If you want, you can use anaconda and install after nltk library
# !pip install --user numpy
# !pip install --user sklearn
# !pip install --user scipy
# !pip install --user nltk

import sys
import nltk
nltk.download("punkt")
nltk.download("stopwords")
nltk.download("averaged_perceptron_tagger")
nltk.download("universal_tagset")
nltk.download('words')

```

    [nltk_data] Downloading package punkt to /home/aminebe/nltk_data...
    [nltk_data]   Package punkt is already up-to-date!
    [nltk_data] Downloading package stopwords to
    [nltk_data]     /home/aminebe/nltk_data...
    [nltk_data]   Package stopwords is already up-to-date!
    [nltk_data] Downloading package averaged_perceptron_tagger to
    [nltk_data]     /home/aminebe/nltk_data...
    [nltk_data]   Package averaged_perceptron_tagger is already up-to-
    [nltk_data]       date!
    [nltk_data] Downloading package universal_tagset to
    [nltk_data]     /home/aminebe/nltk_data...
    [nltk_data]   Package universal_tagset is already up-to-date!
    [nltk_data] Downloading package words to /home/aminebe/nltk_data...
    [nltk_data]   Package words is already up-to-date!

    True

## 2. Jeu de données

On utilise un jeu de donnée provenant de *Crowdflower's Data for Everyone library*: https://www.figure-eight.com/data-for-everyone/

Pour citer la source originale de la base :

    A sentiment analysis job about the problems of each major U.S. airline. Twitter data was scraped from February of 2015 and contributors were asked to first classify positive, negative, and neutral tweets, followed by categorizing negative reasons (such as "late flight" or "rude service").

Les compagnies incluses dans cette base de données sont Virgin America, United Airline, Southwest Airlines, jetBlue, USAirways, et American Airlines.

Dans le fichier zip du TP, vous trouverez le fichier *airline_tweets_database.csv*, qui est la base de données de tweets que nous allons manipuler.

Chaque ligne de ce fichier contient un tweet, avec plusieurs informations : l'identifiant du tweet, l'utilisateur, le contenu, le nombre de retweet... Ainsi que le label.

3 labels différents sont possibles dans ce dataset : *négatif*, *neutre* et *positif*, que l'on va représenter respectivement par 0, 1 et 2.

Pour ce TP, on ne va conserver ici que le texte et le label. On divise ensuite la base de données en 3 ensembles (entrainement/validation/test). Vous utiliserez l'ensemble d'entraînement et de validation pour cette partie, et l'ensemble de test à la partie suivante.

Le code ci-dessous permet de charger ces ensembles:

```python
import csv
from sklearn.model_selection import train_test_split

def load_dataset(path):

    x = []
    y = []

    with open(path, 'r', newline='', encoding="latin-1") as csvfile:

        reader = csv.reader(csvfile, delimiter=',')

        # Taking the header of the file + the index of useful columns:
        header = next(reader)
        ind_label = header.index('airline_sentiment')
        ind_text = header.index('text')

        for row in reader:
            x.append(row[ind_text])

            label = row[ind_label]

            if label == "negative":
                y.append(0)
            elif label == "neutral":
                y.append(1)
            elif label == "positive":
                y.append(2)

        assert len(x) == len(y)

        return x, y

# Path of the dataset
path = "data/airline_tweets_database.csv"

X, y = load_dataset(path)

train_valid_X, test_X, train_valid_Y, test_Y = train_test_split(X, y, test_size=0.15, random_state=12)

train_X, valid_X, train_Y, valid_Y = train_test_split(train_valid_X, train_valid_Y, test_size=0.18, random_state=12)

print("Length of training set : ", len(train_X))
print("Length of validation set : ", len(valid_X))
print("Length of test set : ", len(test_X))

```

    Length of training set :  10204
    Length of validation set :  2240
    Length of test set :  2196

## 3. Preprocessing

Nous allons ici implémenter la *tokenization* et le *stemming*, qui sont 2 étapes courantes de preprocessing en NLP. Ensuite, afin d'avoir un modèle qui s'adapte mieux au format de Twitter, nous ajouterons une étape spécifique supplémentaire.

### 3.1. Tokenization

Cette étape permet de séparer un texte en séquence de *tokens* (= jetons, ici des mots, symboles ou ponctuation).

Par exemple la phrase *"It's the student's notebook."* peut être séparé en liste de tokens de cette manière: ["it", " 's", "the", "student", " 's", "notebook", "."].

**De plus, tous les tokenizers ont également le rôle de mettre le texte en minuscule.**

##### Question 1. Implémentez les 2 tokenizers différents suivants: (0.5 Pts)

- Le **SpaceTokenizer**, qui est un tokenizer naïf qui sépare simplement en fonction des espaces.
- Le **NLTKTokenizer**, qui utilise la méthode du package *nltk* (https://www.nltk.org/api/nltk.html).

```python
from nltk.tokenize import TweetTokenizer

class SpaceTokenizer(object):
    """
    It tokenizes the tokens that are separated by whitespace (space, tab, newline).
    We consider that any tokenization was applied in the text when we use this tokenizer.

    For example: "hello\tworld of\nNLP" is split in ['hello', 'world', 'of', 'NLP']
    """

    def tokenize(self, text):
        # Have to return a list of tokens
        tokens = text.split()
        return tokens

class NLTKTokenizer(object):
    """
    This tokenizer uses the default function of nltk package (https://www.nltk.org/api/nltk.html) to tokenize the text.
    """

    def tokenize(self, text):
        # Have to return a list of tokens
        tokens = nltk.tokenize.word_tokenize(text)
        return tokens

```

#### Testez les deux tokenizers. Quelles différences pouvez-vous constater?

```python
text = "It's the student's notebook."

print("Testing with SpaceTokenizer")
spaceTokenizer = SpaceTokenizer()
print(spaceTokenizer.tokenize(text))

print("\n")

print("Testing with NLTKTokenizer")
nltkTokenizer = NLTKTokenizer()
print(nltkTokenizer.tokenize(text))
```

    Testing with SpaceTokenizer
    ["It's", 'the', "student's", 'notebook.']

    Testing with NLTKTokenizer
    ['It', "'s", 'the', 'student', "'s", 'notebook', '.']

### 3.2. Troncature (ou Stemming)

Dans les phrases "I should have bought a new shoes today" et "I spent too much money buying games", les mots "buying" et "bought" représentent la même idée. Considérer ces deux mots comme différents ne ferait qu'augmenter pour rien la complexité et la dimension du problème, ce qui peut avoir un impact négatif sur la performance globale. Ainsi, on peut donc plutôt une forme unique (comme la racine du mot) pour représenter ces deux mots de la même manière. Ce procédé de conversion de mots en racines permettant de réduire la dimension est appelé usuellement *stemming*, que l'on peut traduire par troncature.

#### Question 2. Récupérez les troncatures des tokens en utilisant l'attribut *stemmer* de la classe *Stemmer* (0.5 Pts)

```python
from nltk.stem.snowball import SnowballStemmer

class Stemmer(object):

    def __init__(self):
        self.stemmer = SnowballStemmer("english", ignore_stopwords=True)

    def stem(self, token):
        """
        token: a string that contain a token
        """
        # Have to return the stemmed token
        return list(map(self.stemmer.stem,token))

```

```python
stemmer = Stemmer()
print(stemmer.stem(["buying", "doing"]))
```

    ['buy', 'doing']

### 3.3. Twitter preprocessing

Parfois, appliquer uniquement ces deux étapes ne suffit pas, due aux particularités des données que nous manipulons, qui peuvent demander des étapes de preprocessing spécifiques afin d'obtenir un modèle plus adapté.

Couramment en NLP, un dictionnaire est utilisé pour stocker un ensemble de mots, et tous les mots n'appartenant pas au dictionnaire sont considérés comme inconnus. Ainsi, avec ce choix d'implémentation, la dimension de l'espace caractéristique du modèle est directement liée au nombre de mots du dictionnaire. Ainsi, pour des raisons de complexité mais aussi car les modèles à trop grande dimension peuvent souffrir du fléau de la dimensionnalité, il est préférable de réduire la taille de notre vocabulaire.

#### Question 3. Donnez, en expliquant brièvement, au moins deux exemples d'étapes de préprocessing qui permettent de réduire la taille du dictionnaire ici, puis implémentez-les.  (2.0 points)

Ces étapes de préprocessing doivent être en rapport aux charactéristiques spécifiques des données de Twitter. La suppression des mots vides ne compte pas comme une des deux étapes.

L'etape du preprocessing est une étape crutial pour le developement d'un modele assez puissant. On propose ici quelques traitements qu'on efectuera sur les tweet afin de reduire le vocabulaire et supprimer les tokens qui ne servent pas à notre problematique:

- **Remplacer les URL par un token:** les url sur twitter sont tous reduit vers un lien t.co qui ne donnent pas generalement aucune idée sur le contenu du tweet. On replace tous les url dans les tweet par le token `__URL__`

- **Remplacer les nombres par un token:** Un nombre ne permet pas dans un model BoW de savoir c'est quoi la nature du tweet sauf si on le met dans un context en specifiant l'unitée par exemple. On replace tous les nombre dans les tweet par le token `__NBR__`

- **Remplacer les TAGs par un token:** Lorsqu'on tag quelqu'un sur un tweet cela ne serve pas beaucoup à savoir l'emotion qu'on exprime. On replace tous les TAG ( les token qui commencent par @) dans les tweet par le token `__UNAME__`

- **Enlever les # des hashtags:** On va considerer les hashtags comme des mots.

- **Traiter les emojis**: Il s'agit ici de classifer les emojis, au moins les plus utilisés, selon deux categories: positives et negatives. Ceci permettra d'avoir une idée sur le type d'emotion exprimé dans le tweet, en remplacant soit par `EMO_POS` qui signifie emotion positive ou `EMO_NEG` qui signifie emotion negative. le code pour ce traitement on l'a depuis ce [lien](https://github.com/abdulfatir/twitter-sentiment-analysis/blob/master/code/preprocess.py?fbclid=IwAR2D44a0pwGWAhxsUVP0Gvr9EpRqd11qAgBXyfpp796mkCJPDXRraPurfu8).

```python
import string
import re
```

```python
class TwitterPreprocessing(object):

    def handle_urls(self, tweet):
        tweet = re.sub(r'(https:|http:)?\/\/t.co\/[a-zA-Z0-9]+', ' __URL__ ', tweet)
        return tweet

    def handle_numbers(self, tweet):
        tweet = re.sub(r'\b\d+\b', ' __NBR__ ', tweet)
        return tweet

    def handle_tags(self, tweet):
        tweet = re.sub(r'@\S+', ' __UNAME__ ', tweet)
        return tweet

    def handle_hashtags(self, tweet):
        tweet = re.sub(r'#(\S+)', r'\1', tweet)
        return tweet

    def handle_and(self, tweet):
        return tweet.replace("&", "&")

    def handle_emojis(self, tweet):
        # taken from https://github.com/abdulfatir/twitter-sentiment-analysis/blob/master/code/preprocess.py
        # Smile --
        tweet = re.sub(r'(:-\)|\(-:|:\'\))', ' __EMO_POS__ ', tweet)
        # Laugh --
        tweet = re.sub(r'(:-?D|x-?D|X-?D)', ' __EMO_POS__ ', tweet)
        # Love --
        tweet = re.sub(r'(<3|:\*)', ' __EMO_POS__ ', tweet)
        # Wink --
        tweet = re.sub(r'(;-?\)|;-?D|\(-?;)', ' __EMO_POS__ ', tweet)
        # Sad --
        tweet = re.sub(r'(:-\(|\)-:)', ' __EMO_NEG__ ', tweet)
        # Sad 2 --
        tweet = re.sub(r'(:\/|:-\/)', ' __EMO_NEG__ ', tweet)
        # Cry --
        tweet = re.sub(r'(:,\(|:\'\(|:"\()', ' __EMO_NEG__ ', tweet)
        return tweet

    def preprocess(self,data):

        data=self.handle_urls(data)
        data=self.handle_emojis(data)
        data=self.handle_tags(data)
        data=self.handle_hashtags(data)
        data=self.handle_and(data)
        # remove single letters and some repeated punctuation
        data = re.sub(r'\b[-\']\b', '', data)
        data = re.sub(r'\b\w\b', '', data)
        data = re.sub(r'[\?\'\.]{2,}', ' ', data)
        # remove encoding chars
        data = data.encode("cp1251","ignore").decode("utf8")
        return data
```

```python
# test
text = "Hello world \x8f this is my first & tweet 2019!poke @Hamza0, @Amine #LetsMakePolyGreatAgain :D ;) :-) :-D :-/ :/ meet us @ //t.co/kbb0B5FxMK https://t.co/gooNdg00Poly"
tweet_prep = TwitterPreprocessing()
tweet_prep.preprocess(text)
```

    'Hello world  this is my first & tweet 2019!poke  __UNAME__   __UNAME__  LetsMakePolyGreatAgain  __EMO_POS__   __EMO_POS__   __EMO_POS__   __EMO_POS__   __EMO_NEG__   __EMO_NEG__  meet us @  __URL__   __URL__ '

d'autres traitements peuvent être envisagés comme le traitement des ponctuations et les espaces.

### 3.4.  Pipeline

Une pipeline permet d'exécuter séquentiellement toutes les étapes de preprocessing, pour transformer les données brutes en une version utilisable pour notre modèle. La *PreprocessingPipeline* a été implémenter pour appliquer à la suite le tokenizer, les troncatures et le preprocessing spécifique à Twitter.

**N'hésitez pas à changer l'ordre des étapes de preprocessing si vous le souhaitez.**

```python
class PreprocessingPipeline:

    def __init__(self, tokenization, twitterPreprocessing, stemming):
        """
        tokenization: enable or disable tokenization.
        twitterPreprocessing: enable or disable twitter preprocessing.
        stemming: enable or disable stemming.
        """

        self.tokenizer = NLTKTokenizer() if tokenization else SpaceTokenizer()
        self.twitterPreprocesser = TwitterPreprocessing(
        ) if twitterPreprocessing else None
        self.stemmer = Stemmer() if stemming else None

    def preprocess(self, tweet):
        """
        Transform the raw data

        tokenization: boolean value.
        twitterPreprocessing: boolean value. Apply the
        stemming: boolean value.
        """
        if self.twitterPreprocesser:
            tweet_processed = self.twitterPreprocesser.preprocess(tweet)
        else:
            tweet_processed = tweet

        tokens = self.tokenizer.tokenize(tweet_processed)
        if self.stemmer:
            tokens = self.stemmer.stem(tokens)
        return tokens

```

Test de la pipeline :

```python
print("initial tweet:",train_X[:1])
pipeline = PreprocessingPipeline(tokenization=True, twitterPreprocessing=True, stemming=True)
print("results:")
print(list(map(pipeline.preprocess, train_X[:1])))
```

    initial tweet: ['@USAirways tells me to talk to @AmericanAir about my delayed flights. AA tells me to talk to US. #ihatemergers']
    results:
    [['__uname__', 'tell', 'me', 'to', 'talk', 'to', '__uname__', 'about', 'my', 'delay', 'flight', '.', 'aa', 'tell', 'me', 'to', 'talk', 'to', 'us', '.', 'ihatemerg']]

## 4. N-grams

Un n-gram est une séquence continue de *n* tokens dans un texte. Par exemple, les séquences *"nous a"* et *"la porte"* sont deux exemples de 2-grams de la phrase *"Il nous a dit au revoir en franchissant la porte."*. 1-gram, 2-gram et 3-gram sont respectivement appelés unigram, bigram et trigram.

Voici la liste de tous les unigrams, bigrams et trigrams possible pour la phrase *"Il nous a dit au revoir en franchissant la porte."* :
- Unigram: ['Il', 'nous', 'a', 'dit', 'au', 'revoir', 'en', 'franchissant', 'la', 'porte']
- Bigram: ['Il nous', 'nous a', 'a dit', 'dit au', 'au revoir', 'revoir en', 'en franchissant', 'franchissant la', 'la porte']
- Trigram: ['Il nous a', 'nous a dit', 'a dit au', 'dit au revoir', 'au revoir en', 'revoir en franchissant', 'en franchissant la', 'franchissant la porte']

##### Question 4. Implementez les fonctions `bigram` et `trigram`. (1 Pt)

Vous devez résoudre cette question sans utiliser de libraire exterieur comme scikit-learn par exemple.

```python
def bigram(tokens):
    """
    tokens: a list of strings
    """
    bigrams = []
    for words in zip(tokens[:-1],tokens[1:]):
        bigrams.append(" ".join(words))
    # This function returns the list of bigrams
    return bigrams

def trigram(tokens):
    """
    tokens: a list of strings
    """
    trigrams = []
    for words in zip(tokens[:-2],tokens[1:-1], tokens[2:]):
        trigrams.append(" ".join(words))
    # This function returns the list of trigrams
    return trigrams

```

```python
text = "He does not know what he is talking about"
nltktokenizer = NLTKTokenizer()
tokens = nltktokenizer.tokenize(text)
print('Liste des bigrams :'+"\n")
print('Bigrams =', bigram(tokens))
print('\n')
print('Liste des trigrams :'+"\n")
print('Trigrams = ', trigram(tokens))
```

    Liste des bigrams :

    Bigrams = ['He does', 'does not', 'not know', 'know what', 'what he', 'he is', 'is talking', 'talking about']

    Liste des trigrams :

    Trigrams =  ['He does not', 'does not know', 'not know what', 'know what he', 'what he is', 'he is talking', 'is talking about']

## 5. Bag-of-words

Régressions logistiques, SVM et d'autres modèles très courants demande des entrées qui soient toutes de la même taille, ce qui n'est forcément le cas pour des types de données comme les textes, qui peuvent avoir un nombre variable de mots.

Par exemple, considérons la phrase 1, ”Board games are much better than video games” et la phrase 2, ”Pandemic is an awesome game!”. La table ci-dessous montre un exemple d'un moyen de représentation de ces deux phrases en utilisant une représentation fixe :

|<i></i>     | an | are | ! | pandemic | awesome | better | games | than | video | much | board | is | game |
|------------|----|-----|---|----------|---------|--------|-------|------|-------|------|-------|----|------|
| Sentence 1 | 0  | 1   | 0 | 0        | 0       | 1      | 2     | 1    | 1     | 1    | 1     | 0  | 0    |
| Sentence 2 | 1  | 0   | 0 | 1        | 1       | 0      | 0     | 0    | 0     | 0    | 0     | 1  | 1    |

Chaque colonne représente un mot du vocabulaire (de longueur 13), tandis que chaque ligne contient l'occurence des mots dans une phrase. Ainsi, la valeur 2 à la position (1,7) est due au fait que le mot *"games"* apparait deux fois dans la phrase 1.

Ainsi, chaque ligne étant de longueur 13, on peut les utiliser comme vecteur pour représenter les phrases 1 et 2. Ainsi, c'est cette méthode que l'on appelle *Bag-of-Words* : c'est une représentation de documents par des vecteurs dont la dimension est égale à la taille du vocabulaire, et qui est construit en comptant le nombre d'occurence de chaque mot. Ainsi, chaque token est ici associé à une dimension.

##### Question 5. Implémentez le Bag-of-Words (2 Pts)

Pour cette question, vous ne pouvez pas utiliser de librairie Python externe comme scikit-learn, hormis si vous avez des problèmes de mémoire, vous pouvez utiliser la classe sparse.csr_matrix de scipy (https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html).

```python
# no need for the csr_matrix if we use an np.int8 type
# (we suppose the tweet are not too long and cannot contain the same token more than 256 times)
from scipy.sparse import csr_matrix
import numpy as np

class CountBoW(object):

    def __init__(self, pipeline, bigram=False, trigram=False):
        """
        pipelineObj: instance of PreprocesingPipeline
        bigram: enable or disable bigram
        trigram: enable or disable trigram
        words: list of words in the vocabulary
        """
        self.pipeline = pipeline
        self.bigram = bigram
        self.trigram = trigram
        self.words = None

    def computeBoW(self, tokens):
        """
        Calcule du BoW, à partir d'un dictionnaire de mots et d'une liste de tweets.
        On suppose que l'on a déjà collecté le dictionnaire sur l'ensemble d'entraînement.

        Entrée: tokens, une liste de vecteurs contenant les tweets (une liste de liste)

        Return: une csr_matrix
        """

        if self.words is None:
            raise Exception(
                "fit_transform() should be called first (no dictionnary available)"
            )

        word_to_idx = {word:idx for idx,word in enumerate(self.words)}

        bag_matrix = np.zeros((len(tokens), len(self.words)),dtype=np.int8)

        for tweet_idx,tweet_tokens in enumerate(tokens):
            all_tokens = tweet_tokens.copy()
            if self.bigram == True :
                all_tokens+=bigram(tweet_tokens)
            if self.trigram == True :
                all_tokens+=trigram(tweet_tokens)

            for token in all_tokens:
                word_idx = word_to_idx.get(token, -1)
                if word_idx>=0:
                    bag_matrix[tweet_idx,word_to_idx[token]] += 1

        return bag_matrix

    def fit_transform(self, X):
        """
        Cette méthode preprocess les données en utilisant la pipeline, ajoute les bigram et trigram
        si besoin, et transforme les textes en vecteurs d'entiers.

        Entrée: X, une liste de vecteurs contenant les tweets

        Return: une csr_matrix
        """

        toknized_tweets = list(map(self.pipeline.preprocess, X))
        words_dictionnary = set()
        for tweet in toknized_tweets:
            for token in tweet:
                words_dictionnary.add(token)
            if self.bigram == True :
                for token in bigram(tweet):
                    words_dictionnary.add(token)
            if self.trigram == True :
                for token in trigram(tweet):
                    words_dictionnary.add(token)

        self.words = list(words_dictionnary)

        return self.computeBoW(toknized_tweets)

    def transform(self, X):
        """
        Cette méthode preprocess les données en utilisant la pipeline, ajoute les bigram et trigram
        si besoin, et transforme les textes en vecteurs d'entiers.
        Différence avec fit_transform : on suppose qu'on dispose déjà du dictionnaire ici

        Entrée: X, une liste de vecteurs contenant les tweets
        Return: une csr_matrix
        """

        if self.words is None:
            raise Exception(
                "fit_transform() should be called first (no dictionnary available)"
            )

        toknized_tweets = list(map(self.pipeline.preprocess, X))
        return self.computeBoW(toknized_tweets)

```

```python
pipeline = PreprocessingPipeline(tokenization=True, twitterPreprocessing=True, stemming=True)
bow_test=CountBoW(pipeline, bigram=True, trigram=True)
bow_test.fit_transform(train_X[:2])
```

    array([[0, 4, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 2, 0, 1, 1, 1, 0, 0,
            0, 1, 0, 0, 0, 1, 1, 1, 0, 2, 1, 0, 1, 0, 2, 0, 0, 0, 0, 0, 2, 0,
            0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 2, 2, 0, 1, 0, 0,
            0, 0, 1, 1, 2, 1, 1, 0, 1, 1, 0, 1, 2, 1, 0, 1, 0, 0, 1, 0, 2, 0,
            0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 2, 0, 2, 0, 0, 0, 0, 0,
            0, 1, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0],
           [1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 2, 1, 1, 0, 1, 0, 0, 0, 1, 1,
            1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1,
            1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1,
            1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1,
            1, 2, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,
            1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1]], dtype=int8)

```python
list_list= bow_test.transform(train_X[3:5])
list_list
```

    array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int8)

## 6. TF-IDF

L'utilisation de la fréquence d'apparition brute des mots, comme c'est le cas avec le bag-of-words, peut être problématique. En effet, peu de tokens auront une fréquence très élevée dans un document, et de ce fait, le poids de ces mots sera beaucoup plus grand que les autres, ce qui aura tendance à biaiser l'ensemble des poids. De plus, les mots qui apparaissent dans la plupart des documents n'aident pas à les discriminer. Par exemple, le mot "*de*" apparaît dans beaucoup de tweets de la base de données, et pour autant, avoir ce mot en commun ne permet pas de conclure que des tweets sont similaires. Au contraire, le mot "*génial*" est plus rare, mais les documents qui contiennent ce mot sont plus susceptibles d'être positif. TF-IDF est donc une méthode qui permet de pallier à ce problème.

TF-IDF pondère le vecteur en utilisant une fréquence de document inverse (IDF) et une fréquence de termes (TF).

TF est l'information locale sur l'importance qu'a un mot dans un document donné, tandis que IDF mesure la capacité de discrimination des mots dans un jeu de données.

L'IDF d'un mot se calcule de la façon suivante:

\begin{equation}
	\text{idf}_i = \log\left( \frac{N}{\text{df}_i} \right),
\end{equation}

avec $N$ le nombre de documents dans la base de donnée, et $\text{df}_i$ le nombre de documents qui contiennent le mot $i$.

Le nouveau poids $w_{ij}$ d'un mot $i$ dans un document $j$ peut ensuite être calculé de la façon suivante:

\begin{equation}
	w_{ij} = \text{tf}_{ij} \times \text{idf}_i,
\end{equation}

avec $\text{tf}_{ij}$ la fréquence du mot $i$ dans le document $j$.

##### Question 6. Implémentez le bag-of-words avec la pondération de TF-IDF (3 Pts)

Pour cette question, vous ne pouvez pas utiliser de librairie Python externe comme scikit-learn, hormis si vous avez des problèmes de mémoire, vous pouvez utiliser la classe sparse.csr_matrix de scipy (https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html).

```python
from scipy.sparse import csr_matrix
import math

class TFIDFBoW(object):

    def __init__(self, pipeline, bigram=False, trigram=False):
        """
        pipelineObj: instance of PreprocesingPipeline
        bigram: enable or disable bigram
        trigram: enable or disable trigram
        words: list of words in the vocabulary
        idf: list of idfs for each document
        """
        self.pipeline = pipeline
        self.bigram = bigram
        self.trigram = trigram
        self.words = None
        self.idf = None

    def computeTFIDF(self, tokens):
        """
        Calcule du TF-IDF, à partir d'un dictionnaire de mots et d'une
        liste de tweets.
        On suppose que l'on a déjà collecté le dictionnaire ainsi que
        calculé le vecteur contenant l'idf pour chaque document.

        Entrée: tokens, une liste de vecteurs contenant les tweets (une liste de liste)

        Return: une csr_matrix
        """

        if self.words is None:
            raise Exception(
                "fit_transform() should be called first (no dictionnary available)"
            )

        word_to_idx = {word:idx for idx,word in enumerate(self.words)}

        tf = np.zeros((len(tokens), len(self.words)),dtype=np.int8)

        for tweet_idx,tweet_tokens in enumerate(tokens):
            all_tokens = tweet_tokens.copy()
            if self.bigram == True :
                all_tokens+=bigram(tweet_tokens)
            if self.bigram == True :
                all_tokens+=trigram(tweet_tokens)

            for token in all_tokens:
                word_idx = word_to_idx.get(token, -1)
                if word_idx>=0:
                    tf[tweet_idx,word_to_idx[token]] += 1

        # puisque ce n'est pas specifié on utilise le np.log
        if self.idf is None:
            self.idf = np.log(tf.shape[0] / (tf!=0).sum(axis=0))

        return np.multiply(tf, self.idf)

    def fit_transform(self, X):
        """
        Cette méthode preprocess les données en utilisant la pipeline, ajoute les bigram et trigram
        si besoin, et transforme les textes en vecteurs de flottants avec la pondération TF-IDF.

        Entrée : X, une liste de vecteurs contenant les tweets

        Return: une csr_matrix
        """
        toknized_tweets = list(map(self.pipeline.preprocess, X))
        words_dictionnary = set()
        for tweet in toknized_tweets:
            for token in tweet:
                words_dictionnary.add(token)
            if self.bigram == True :
                for token in bigram(tweet):
                    words_dictionnary.add(token)
            if self.trigram == True :
                for token in trigram(tweet):
                    words_dictionnary.add(token)

        self.words = list(words_dictionnary)
        return self.computeTFIDF(toknized_tweets)

    def transform(self, X):
        """
        Cette méthode preprocess les données en utilisant la pipeline, ajoute les bigram et trigram
        si besoin, et transforme les textes en vecteurs de flottants avec la pondération TF-IDF.
        Différence avec fit_transform : on suppose qu'on dispose déjà du dictionnaire et du calcul des idf ici.

        Entrée : X, une liste de vecteurs contenant les tweets

        Return: une csr_matrix
        """

        if self.words is None:
            raise Exception(
                "fit_transform() should be called first (no dictionnary available)"
            )

        toknized_tweets = list(map(self.pipeline.preprocess, X))
        return self.computeTFIDF(toknized_tweets)

```

```python
pipeline = PreprocessingPipeline(tokenization=True, twitterPreprocessing=True, stemming=True)
tfidf_test=TFIDFBoW(pipeline, bigram=False, trigram=False)
tfidf_test.fit_transform(train_X[:2])
```

    array([[0.        , 0.        , 0.        , 1.38629436, 2.77258872,
            0.69314718, 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.69314718, 0.        ,
            0.        , 0.69314718, 0.        , 0.        , 1.38629436,
            0.        , 0.        , 0.69314718, 0.        , 0.69314718,
            0.        , 1.38629436, 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.69314718, 0.        , 0.        ,
            0.        , 0.69314718],
           [0.69314718, 0.69314718, 0.69314718, 0.        , 0.        ,
            0.        , 0.69314718, 0.69314718, 0.        , 0.        ,
            0.69314718, 0.69314718, 0.69314718, 0.        , 0.69314718,
            0.69314718, 0.        , 0.69314718, 0.69314718, 0.        ,
            1.38629436, 0.69314718, 0.        , 0.69314718, 0.        ,
            0.69314718, 0.        , 0.69314718, 0.69314718, 1.38629436,
            0.69314718, 0.69314718, 0.        , 0.69314718, 0.69314718,
            0.69314718, 0.        ]])

```python
list_list= tfidf_test.transform(train_X[3:5])
list_list
```

    array([[0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.69314718, 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.69314718, 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.69314718, 0.69314718],
           [0.        , 0.        , 0.        , 0.        , 2.07944154,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            1.38629436, 0.        , 0.        , 0.69314718, 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.69314718, 0.        , 0.        , 0.        ,
            0.        , 0.        , 0.        , 0.        , 0.        ,
            0.        , 0.        ]])

## 7. Classification utilisant BoW

Pour la classification, nous allons effectuer une régression logisitique (vu en cours ou que vous allez voir bientôt).
Pour en savoir plus : https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

La méthode `train_evaluate` entraîne et évalue le modèle de régression logistique.

```python
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression

def train_evaluate(training_X, training_Y, validation_X, validation_Y, bowObj):
    """
    training_X: tweets from the training dataset
    training_Y: tweet labels from the training dataset
    validation_X: tweets from the validation dataset
    validation_Y: tweet labels from the validation dataset
    bowObj: Bag-of-word object

    :return: the classifier and its accuracy in the training and validation dataset.
    """

    classifier = LogisticRegression(n_jobs=-1)

    training_rep = bowObj.fit_transform(training_X)
    print("- Taille du dictionnaire:",len(bowObj.words))
    classifier.fit(training_rep, training_Y)

    trainAcc = accuracy_score(training_Y, classifier.predict(training_rep))
    validationAcc = accuracy_score(
        validation_Y, classifier.predict(bowObj.transform(validation_X)))

    print(f"- Training accuracy: {trainAcc:.5f}")
    print(f"- Validation accuracy: {validationAcc:.5f}")
    return classifier, trainAcc, validationAcc

```

##### Question 7. Entraînez et calculez la précision de la régression logistique sur les ensembles d'entraînement et de validation. (4 points)

Essayez les configurations suivantes :

    1. CountBoW + SpaceTokenizer(without tokenizer) + unigram
    2. CountBoW + NLTKTokenizer + unigram
    3. TFIDFBoW + NLTKTokenizer + Stemming + unigram
    4. TFIDFBoW + NLTKTokenizer + Twitter preprocessing + Stemming  + unigram
    5. TFIDFBoW + NLTKTokenizer + Twitter preprocessing + Stemming  + unigram + bigram
    6. TFIDFBoW + NLTKTokenizer + Twitter preprocessing + Stemming  + unigram + bigram + trigram

Outre la précision, reportez la taille du dictionnaire pour chacune des configurations. Enfin, décrivez vos résultats obtenus et répondez aux questions suivantes:
- Quelles étapes de preprocessing ont effectivement aidé le modèle ? Pourquoi ?
- La pondération avec TF-IDF a-t-elle aidé à obtenir une meilleure performance que le simple BoW ?
- Les bigrams et trigrams ont-ils amélioré la performance ? Expliquez pourquoi.

Indiquez quelle est la configuration que vous choisissez.

```python
configurations = [
    {"model":CountBoW, "tokenize":False},
    {"model":CountBoW, "tokenize":True},
    {"model":TFIDFBoW, "tokenize":True, "stemming": True},
    {"model":TFIDFBoW, "tokenize":True, "stemming": True, "preprocess":True},
    {"model":TFIDFBoW, "tokenize":True, "stemming": True, "preprocess":True, "bi":True},
    {"model":TFIDFBoW, "tokenize":True, "stemming": True, "preprocess":True, "bi":True,"tri":True}
]

results = []
for i,conf in enumerate(configurations):
    print(f"[+] Running configuration {i+1}...")
    stemming = conf.get("stemming",False)
    tw_prep = conf.get("preprocess",False)
    tokenize = conf.get("tokenize",False)

    pipeline = PreprocessingPipeline(tokenization = tokenize, twitterPreprocessing = tw_prep, stemming = stemming)
    obj = conf["model"](pipeline, bigram = conf.get("bi",False), trigram = conf.get("tri",False))
    results.append(train_evaluate(train_X, train_Y, train_valid_X, train_valid_Y, obj))

#- Taille du dictionnaire: 11064
#- Training accuracy: 0.98844
#- Validation accuracy: 0.94873
```

    [+] Running configuration 1...
    - Taille du dictionnaire: 23071

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.96139
    - Validation accuracy: 0.93161
    [+] Running configuration 2...
    - Taille du dictionnaire: 15540

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.94277
    - Validation accuracy: 0.91763
    [+] Running configuration 3...
    - Taille du dictionnaire: 11064

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.98844
    - Validation accuracy: 0.94873
    [+] Running configuration 4...
    - Taille du dictionnaire: 9282

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.98256
    - Validation accuracy: 0.94511
    [+] Running configuration 5...
    - Taille du dictionnaire: 79244

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.99647
    - Validation accuracy: 0.96255
    [+] Running configuration 6...
    - Taille du dictionnaire: 211105

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    - Training accuracy: 0.99667
    - Validation accuracy: 0.96183

```python
import matplotlib.pyplot as plt
%matplotlib inline

models, training_acc, validation_acc = zip(*results)

plt.figure(figsize=(10,5))
plt.plot(list(map(lambda i:f"config {i}",range(1,7))), training_acc, label="Training accuracy")
plt.plot(list(map(lambda i:f"config {i}",range(1,7))), validation_acc, label="Validation accuracy")
plt.title("Performance of different models per configuration")
plt.ylabel("Accuracy")
plt.legend();
```

- Quelles étapes de preprocessing ont effectivement aidé le modèle ? Pourquoi ?

Selon les résultats des différentes configuration de preprocessing ci-dessous, nous constatons que les étapes de preprocessing de la configuration 5 qu’ont effectivement aidé le modèle. En premier lieu, l'ajout du Twitter preprocessing augmente notre précision, ce qui est logique. Ensuite, l'ajout du bigram nous aide à augmenter encore plus cette précision, puisque le modèle considère deux mots en même temps (et donc donne un contexte) et non pas chaque mot indépendamment de l'autre.

- La pondération avec TF-IDF a-t-elle aidé à obtenir une meilleure performance que le simple BoW ?

OUI, TF-IDF a donné des précisions meilleures. En outre, l’avantage de TF-IDF provient du fait que certains mots peuvent être trouvés dans différents tweets; les gens ont tendance à s'exprimer avec des caractéristiques communes (mêmes mots souvent), ce qui n'est pas pris en compte dans CountBoW, mais apparaît dans le terme IDF de TF-IDF.

- Les bigrams et trigrams ont-ils amélioré la performance ? Expliquez pourquoi.

Les bigrams et trigrams ont bien amélioré les performances. Selon les résultats, la précision sur les données de validation a augmenté de 94,5% à 96,2%  grâce à l'utilisation de bigrams. Nous pouvons en justifier par le fait que le modèle peut stocker davantage de contexte. Si nous prenons l'exemple d'un mot bigram "Don’t like" ou "Quite pleasant", cela donne plus d'informations sur le sentiment que d'avoir les deux mots séparés. Concernant les trigrams, nous pouvons avoir un à priori sur leur précision comparée aux unigrams puisqu'ils prennent en compte le contexte quand même, mais peut être que considérer deux mots à la fois marche mieux. En tout cas, c'est le cas dans cette étude puisque la configuration n°6 ne donne pas un meilleur résultat que la configuration n°5.

# II/ Prototype (7 points)

Maintenant que nous avons un modèle de classification entraîné pour l'analyse de sentiments, nous pouvons l'appliquer à notre ensemble de tests et analyser le résultat.

## 1. Analyse de Sentiments

##### Question 9. Implémentez la fonction `detect_airline` qui détecte la compagnie aérienne d'un tweet. (1,5 points)

Expliquez votre approche, et les inconvénients possibles.

**Attention :** `detect_airline` doit être en mesure de gérer le cas où aucune compagnie n'est mentionnée (auquel cas `None` est retounée), mais aussi le cas où plusieurs compagnies sont mentionnées dans un tweet.

```python
from collections import Counter

knows_airlines = ['united', 'usairways', 'americanair', 'southwestair', 'jetblue', 'virginamerica',
                  'delta', 'spiritairlines', 'staralliance', 'virginatlantic', 'british_airways', 'expedia',
                  'southwest', 'gma', 'silverairways', 'aircanada']

print([k[0] for k in sorted(Counter(sum(list(map(lambda x:re.findall(r"@(\w+)",x.lower()),train_X)),[])).items(),
                            key=lambda x: -x[1]) if k[1]>2])
```

    ['united', 'usairways', 'americanair', 'southwestair', 'jetblue', 'virginamerica', 'delta', 'imaginedragons', 'phlairport', 'ladygaga', 'carrieunderwood', 'wsj', 'dfwairport', 'flytpa', 'spiritairlines', 'fortunemagazine', 'velourlive', 'staralliance', 'love_dragonss', 'bostonlogan', 'triflight', 'virginatlantic', 'dulles_airport', 'derekc21', 'ny_njairports', 'british_airways', 'cowboycerrone', 'perfectomobile', 'fly2ohare', 'jayvig', 'expedia', 'gg8929', 'southwest', 'directtv', 'jack_kairys', 'thenationaluae', 'zkatcher', 'gma', 'ellahenderson', 'kylejudah', 'annricord', 'aarp', 'dallaslovefield', 'lasairport', 'vincenzolandino', 'jedediahbila', 'roxydigital', 'cnn', 'abcnetwork', 'nytimes', 'silverairways', 'aircanada']

```python
import re
knows_airlines = ['united', 'usairways', 'americanair', 'southwestair', 'jetblue', 'virginamerica',
                  'delta', 'spiritairlines', 'staralliance', 'virginatlantic', 'british_airways', 'expedia',
                  'southwest', 'gma', 'silverairways', 'aircanada']

def detect_airline(tweet):
    """
    Detect and return the airline companies mentioned in the tweet

    tweet: represents the tweet message. You should define the data type

    Return: list of detected airline companies
    """

    found = set(knows_airlines).intersection(re.findall(r"@(\S+)",tweet.lower()))
    return list(found) if len(found)>0 else None

```

##### Question 10. Implémentez la fonction `extract_sentiment` qui, à partir de tweets et d'un classificateur, extrait leurs sentiments. (0.5 points)

```python

def extract_sentiment(classifier, tweets):
    """
    Extract the tweet sentiment

    classifier: classifier object
    tweets: represents the tweet message. processed tweets : matrix with numerical values (n_tweets x n_words)

    Return: list of detected airline companies
    """
    # We suppose that the classifier is trained
    return classifier.predict(tweets)

```

##### Question 11. En utilisant `extract_tweet_content`, `detect_airline` et `extract_sentiment`, générez un diagramme en bar contenant le nombre de tweets positives, négatifs et neutres pour chacune des compagnies. (2 points)

Décrivez brièvement le diagramme et analysez les résultats (par exemple, quelle est la compagnie avec le plus de tweets négatifs?). Expliquez comment un tel diagramme peut aider des compagnies aériennes.

```python
# init configuration
selected_conf = {"model":TFIDFBoW, "tokenize":True, "stemming": True, "preprocess":True, "bi":True}

stemming = selected_conf.get("stemming",False)
tw_prep = selected_conf.get("preprocess",False)
tokenize = selected_conf.get("tokenize",False)
bi = conf.get("bi",False)
tri = conf.get("tri",False)

# init preprocessing pipeline
pipeline = PreprocessingPipeline(tokenization = tokenize, twitterPreprocessing = tw_prep, stemming = stemming)
bowObj = conf["model"](pipeline, bigram = bi, trigram = tri)

training_rep = bowObj.fit_transform(train_X)
testing_rep = bowObj.transform(test_X)

# fit the classifier
classifier = LogisticRegression(n_jobs=-1)
classifier.fit(training_rep, train_Y)
```

    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
      "this warning.", FutureWarning)
    //anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 4.
      " = {}.".format(effective_n_jobs(self.n_jobs)))

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='warn', n_jobs=-1, penalty='l2',
                       random_state=None, solver='warn', tol=0.0001, verbose=0,
                       warm_start=False)

```python
airlines = list(map(detect_airline,test_X))
sentiments = extract_sentiment(classifier, testing_rep)
```

```python
import matplotlib.pyplot as plt
from collections import Counter, defaultdict

airlines_rates = defaultdict(lambda :defaultdict(int))
for airlines_tweet, sentim in zip(airlines, sentiments):
    if airlines_tweet is not None:
        for airln in airlines_tweet:
            airlines_rates[sentim][airln]+=1

```

```python
import pandas as pd

scores = pd.DataFrame(airlines_rates).fillna(0)
scores.columns = ["Negative", "Neutral", "Positive"]
scores.plot.bar(figsize=(20,5), rot=0)
plt.title("Histogram of sentiments expressed in comments for different airline companies")
plt.ylabel("Number of tweets")
plt.xlabel("Airline companies");
```

L’histogramme ci-dessus représente le nombre de tweets positives, négatifs et neutres pour chacune des compagnies. On remarque que le sentiment négatif est le plus dominant pour tous les compagnies. La compagnie United prend le dessus en terme de tweets négatifs vient après la compagnie American air, et puis UsairWays. En termes de tweets positives et neutres, on trouve qu’ils varient d’une compagnie à l’autre avec un grand écart par rapport aux tweets négatifs.
Ce diagramme aide énormément  les compagnies aériennes pour analyser la satisfaction de leurs clients par rapport à leurs concurrents.

## 2. Analyse de termes

Le POS-tagging (pour *part-of-speech tagging*, en français étiquetage grammatical) consiste à l'extraction de l'information grammaticale d'un token dans une phrase. Par exemple, la table ci-dessous donne un exemple du *POS-tagging* de la phrase *"The cat is white!"*

|   The   | cat  |  is  | white     |    !       |
|---------|------|------|-----------|------------|
| article | noun | verb | adjective | punctation |

Pour autant, le *POS-tagging* peut être plus complexe que les règles simples apprises à l'école. Il faut souvent des informations plus détaillées sur le rôle d'un terme dans une phrase. Pour notre problème, nous n'avons pas besoin d'utiliser un modèle linguistique plus complexe, nous allons utiliser ce qu'on appelle des *POS-tags* universelles.

En *POS-tagging*, chaque token est représenté par un tag. La liste des POS-tags utilisés sont disponibles ici :
https://universaldependencies.org/u/pos/ .

```python
# NLTK POS-tagger

import nltk

#before using pos_tag function, you have to tokenize the sentence.
s = ['The', 'cat', 'is',  'white', '!']
nltk.pos_tag(s,tagset='universal')

```

    [('The', 'DET'), ('cat', 'NOUN'), ('is', 'VERB'), ('white', 'ADJ'), ('!', '.')]

##### Question 12. Implémentez un code qui collecte les 10 termes les plus fréquents pour chaque compagnie aérienne. (2 Pts)

Ici, vous n'allez considérer que les termes apparaissant dans les tweets positifs et négatifs.

De plus, nous allons utiliser la définition suivante de "terme":

1. Un mot qui est soit un adjectif, soit un nom
2. Un N-gram composé d'adjectifs suivit par un nom (par exemple, "nice place"), ou un nom suivi par un autre nom (par exemple, "sports club").

Ensuite, **générez une table** contenant les 10 termes les plus fréquents, avec leurs fréquences (en pourcentage) pour chaque compagnie.

*N'oubliez pas de supprimer le nom de la compagnie parmi les termes !*

```python
from collections import Counter, defaultdict

selected_tweets = [tweet for tweet, sentiment in zip(train_X, train_Y) if sentiment!=1]
grams_companies = defaultdict(list)
prev_word = None
for tweet in selected_tweets:
    companies = detect_airline(tweet.lower())

    if companies is None: continue
    cleaned_tweet = tweet.lower()
    for c in companies:
        cleaned_tweet = cleaned_tweet.replace("@"+c,"")
    cleaned_tweet = cleaned_tweet.encode("cp1251","ignore").decode()

    for word in nltk.pos_tag(NLTKTokenizer().tokenize(cleaned_tweet),tagset='universal'):
        if word[1] in ["NOUN","ADJ"]:
            for company in companies:
                grams_companies[company].append(word[0])
                if prev_word is not None and word[1] == "NOUN" and prev_word[1] in ["NOUN","ADJ"]:
                    grams_companies[company].append(f"{prev_word[0]} {word[0]}")
        prev_word = word
```

```python
#from IPython.display import display, HTML

dataframes_dict = {}
for company_name in knows_airlines:
    terms_counts = Counter(grams_companies[company_name])
    most_freq_terms = terms_counts.most_common(10)
    total_counts = sum(terms_counts.values())
    most_freq_terms = list(map(lambda x:(x[0],x[1]/total_counts*100), most_freq_terms)) +[("","")]*(10 - len(most_freq_terms))

    index = [f"term N.{i}" for i in range(1,11)]
    dataframes_dict[company_name] = pd.DataFrame(most_freq_terms, index=index,columns=["term","frequency"])

    # second method
    #df = pd.DataFrame(most_freq_terms, index=index,columns=["term","frequency"])
    #df["frequency"] = df["frequency"] / total_counts*100
    #display(HTML(df.transpose().to_html()))

pd.Panel(dataframes_dict).to_frame().transpose()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead tr th {
        text-align: left;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr>
      <th>major</th>
      <th colspan="2" halign="left">term N.1</th>
      <th colspan="2" halign="left">term N.2</th>
      <th colspan="2" halign="left">term N.3</th>
      <th colspan="2" halign="left">term N.4</th>
      <th colspan="2" halign="left">term N.5</th>
      <th colspan="2" halign="left">term N.6</th>
      <th colspan="2" halign="left">term N.7</th>
      <th colspan="2" halign="left">term N.8</th>
      <th colspan="2" halign="left">term N.9</th>
      <th colspan="2" halign="left">term N.10</th>
    </tr>
    <tr>
      <th>minor</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
      <th>term</th>
      <th>frequency</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>united</th>
      <td>i</td>
      <td>4.40439</td>
      <td>flight</td>
      <td>3.53518</td>
      <td>service</td>
      <td>1.03839</td>
      <td>time</td>
      <td>0.816708</td>
      <td>thanks</td>
      <td>0.810874</td>
      <td>customer</td>
      <td>0.775872</td>
      <td>plane</td>
      <td>0.74087</td>
      <td>bag</td>
      <td>0.74087</td>
      <td>united</td>
      <td>0.711702</td>
      <td>gate</td>
      <td>0.618364</td>
    </tr>
    <tr>
      <th>usairways</th>
      <td>i</td>
      <td>4.85988</td>
      <td>flight</td>
      <td>3.54026</td>
      <td>service</td>
      <td>1.2132</td>
      <td>hold</td>
      <td>1.18482</td>
      <td>hours</td>
      <td>0.971976</td>
      <td>customer</td>
      <td>0.922313</td>
      <td>plane</td>
      <td>0.780419</td>
      <td>thanks</td>
      <td>0.759134</td>
      <td>customer service</td>
      <td>0.744945</td>
      <td>time</td>
      <td>0.695282</td>
    </tr>
    <tr>
      <th>americanair</th>
      <td>i</td>
      <td>5.18524</td>
      <td>flight</td>
      <td>3.68113</td>
      <td>service</td>
      <td>1.04497</td>
      <td>thanks</td>
      <td>1.00538</td>
      <td>@</td>
      <td>1.00538</td>
      <td>hours</td>
      <td>0.918303</td>
      <td>customer</td>
      <td>0.807473</td>
      <td>hold</td>
      <td>0.680811</td>
      <td>time</td>
      <td>0.641229</td>
      <td>call</td>
      <td>0.625396</td>
    </tr>
    <tr>
      <th>southwestair</th>
      <td>i</td>
      <td>5.25686</td>
      <td>flight</td>
      <td>3.45015</td>
      <td>thanks</td>
      <td>1.09926</td>
      <td>hold</td>
      <td>1.09926</td>
      <td>@</td>
      <td>1.03396</td>
      <td>service</td>
      <td>0.946887</td>
      <td>time</td>
      <td>0.827166</td>
      <td>customer</td>
      <td>0.816282</td>
      <td>http</td>
      <td>0.761863</td>
      <td>thank</td>
      <td>0.729212</td>
    </tr>
    <tr>
      <th>jetblue</th>
      <td>i</td>
      <td>4.95619</td>
      <td>flight</td>
      <td>3.8609</td>
      <td>thanks</td>
      <td>1.46495</td>
      <td>@</td>
      <td>1.20482</td>
      <td>jetblue</td>
      <td>1.04053</td>
      <td>time</td>
      <td>0.821468</td>
      <td>http</td>
      <td>0.780394</td>
      <td>service</td>
      <td>0.753012</td>
      <td>plane</td>
      <td>0.698248</td>
      <td>great</td>
      <td>0.684556</td>
    </tr>
    <tr>
      <th>virginamerica</th>
      <td>i</td>
      <td>5.66757</td>
      <td>flight</td>
      <td>3.54223</td>
      <td>http</td>
      <td>0.980926</td>
      <td>@</td>
      <td>0.817439</td>
      <td>thanks</td>
      <td>0.762943</td>
      <td>website</td>
      <td>0.762943</td>
      <td>help</td>
      <td>0.708447</td>
      <td>great</td>
      <td>0.653951</td>
      <td>flights</td>
      <td>0.653951</td>
      <td>seat</td>
      <td>0.599455</td>
    </tr>
    <tr>
      <th>delta</th>
      <td>i</td>
      <td>5.50725</td>
      <td>flight</td>
      <td>4.05797</td>
      <td>worst</td>
      <td>2.02899</td>
      <td>service</td>
      <td>1.73913</td>
      <td>customers</td>
      <td>1.44928</td>
      <td>airline</td>
      <td>1.15942</td>
      <td>time</td>
      <td>1.15942</td>
      <td>@</td>
      <td>0.869565</td>
      <td>good</td>
      <td>0.869565</td>
      <td>anything</td>
      <td>0.869565</td>
    </tr>
    <tr>
      <th>spiritairlines</th>
      <td>florida</td>
      <td>6.89655</td>
      <td>wrong</td>
      <td>3.44828</td>
      <td>3days</td>
      <td>3.44828</td>
      <td>florida 3days</td>
      <td>3.44828</td>
      <td>real</td>
      <td>3.44828</td>
      <td>airlines</td>
      <td>3.44828</td>
      <td>real airlines</td>
      <td>3.44828</td>
      <td>fly</td>
      <td>3.44828</td>
      <td>smalls</td>
      <td>3.44828</td>
      <td>glad</td>
      <td>3.44828</td>
    </tr>
    <tr>
      <th>staralliance</th>
      <td>tickets</td>
      <td>4.25532</td>
      <td>few</td>
      <td>2.12766</td>
      <td>weeks</td>
      <td>2.12766</td>
      <td>few weeks</td>
      <td>2.12766</td>
      <td>bright</td>
      <td>2.12766</td>
      <td>vibrant</td>
      <td>2.12766</td>
      <td>bright vibrant</td>
      <td>2.12766</td>
      <td>habitrails</td>
      <td>2.12766</td>
      <td>right</td>
      <td>2.12766</td>
      <td>thing</td>
      <td>2.12766</td>
    </tr>
    <tr>
      <th>virginatlantic</th>
      <td>i</td>
      <td>20</td>
      <td>representative</td>
      <td>5</td>
      <td>solution</td>
      <td>5</td>
      <td>loyal</td>
      <td>5</td>
      <td>customer</td>
      <td>5</td>
      <td>loyal customer</td>
      <td>5</td>
      <td>funny</td>
      <td>5</td>
      <td>lhr</td>
      <td>5</td>
      <td>mean</td>
      <td>5</td>
      <td>time</td>
      <td>5</td>
    </tr>
    <tr>
      <th>british_airways</th>
      <td>flight</td>
      <td>10</td>
      <td>agent</td>
      <td>5</td>
      <td>tonight</td>
      <td>5</td>
      <td>anyone</td>
      <td>5</td>
      <td>aa</td>
      <td>5</td>
      <td>weeks</td>
      <td>5</td>
      <td>today</td>
      <td>5</td>
      <td>miami</td>
      <td>5</td>
      <td>i</td>
      <td>5</td>
      <td>bag</td>
      <td>5</td>
    </tr>
    <tr>
      <th>expedia</th>
      <td>day</td>
      <td>4.7619</td>
      <td>costumer</td>
      <td>4.7619</td>
      <td>jetblue</td>
      <td>4.7619</td>
      <td>makingthingseasy</td>
      <td>4.7619</td>
      <td>feelbetter</td>
      <td>4.7619</td>
      <td>hour</td>
      <td>4.7619</td>
      <td>hold</td>
      <td>4.7619</td>
      <td>hour hold</td>
      <td>4.7619</td>
      <td>ticket</td>
      <td>4.7619</td>
      <td>easy</td>
      <td>4.7619</td>
    </tr>
    <tr>
      <th>southwest</th>
      <td>i</td>
      <td>5.26316</td>
      <td>change</td>
      <td>5.26316</td>
      <td>customer</td>
      <td>5.26316</td>
      <td>service</td>
      <td>5.26316</td>
      <td>customer service</td>
      <td>5.26316</td>
      <td>flight</td>
      <td>5.26316</td>
      <td>policy</td>
      <td>2.63158</td>
      <td>serious</td>
      <td>2.63158</td>
      <td>revision</td>
      <td>2.63158</td>
      <td>serious revision</td>
      <td>2.63158</td>
    </tr>
    <tr>
      <th>gma</th>
      <td>worse</td>
      <td>5.88235</td>
      <td>airline</td>
      <td>5.88235</td>
      <td>worse airline</td>
      <td>5.88235</td>
      <td>experience</td>
      <td>5.88235</td>
      <td>airline experience</td>
      <td>5.88235</td>
      <td>rude</td>
      <td>5.88235</td>
      <td>unhelpful</td>
      <td>5.88235</td>
      <td>help</td>
      <td>5.88235</td>
      <td>customer</td>
      <td>5.88235</td>
      <td>service</td>
      <td>5.88235</td>
    </tr>
    <tr>
      <th>silverairways</th>
      <td>preferred</td>
      <td>3.125</td>
      <td>security</td>
      <td>3.125</td>
      <td>preferred security</td>
      <td>3.125</td>
      <td>line</td>
      <td>3.125</td>
      <td>security line</td>
      <td>3.125</td>
      <td>tsa</td>
      <td>3.125</td>
      <td>pre-check</td>
      <td>3.125</td>
      <td>tsa pre-check</td>
      <td>3.125</td>
      <td>partner</td>
      <td>3.125</td>
      <td>flight</td>
      <td>3.125</td>
    </tr>
    <tr>
      <th>aircanada</th>
      <td>baggage</td>
      <td>16.6667</td>
      <td>person</td>
      <td>16.6667</td>
      <td>baggage person</td>
      <td>16.6667</td>
      <td>connecting</td>
      <td>16.6667</td>
      <td>flight</td>
      <td>16.6667</td>
      <td>connecting flight</td>
      <td>16.6667</td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
      <td></td>
    </tr>
  </tbody>
</table>
</div>

##### Question 13. Que conclure de la table généré à la question 12 pour les compagnies ? (1 Pt)

Il n’est pas facile de tirer les conclusions directes, car la fréquence des mots est trop faible. Les termes spécifiques discriminants utilisés contre / en faveur de la compagnie sont donc difficiles. Le seul problème avec l'extraction de mots bigrammes ou de mots est que les entreprises ne peuvent pas déterminer la raison du sentiment; ils devront lire l'intégralité du tweet. Sauf dans certains cas où on peut en deviner la raison: par exemple, "missed" signifie probablement que l'auteur a raté son vol

# III/ Bonus (2 points)

Les noms de personnes, les noms de sociétés et les emplacements sont appelés "entités nommées". La reconnaissance d'entité nommée (NER, pour *Named-entity recognition*) consiste à extraire les entités nommées en les classant à l'aide de catégories prédéfinies. Dans cette section bonus, vous utiliserez un outil de NER pour extraire automatiquement des entités nommées des tweets. Cette approche est suffisamment générique pour récupérer des informations sur d’autres sociétés ou même des noms de produits et de personnes.

**Pour le bonus, vous êtes libres d'utiliser n'importe quel NER implémenté en Python.**

##### Question Bonus 1.  Implémentez un code qui génère une table contenant le top 10 des NER de la base de données. (1 point)

Cette table doit contenir les fréquences des entités nommées. Ensuite, générez un diagramme en bar qui montre le nombre de tweets positifs, négatifs ou neutres pour chacunes des 10 NER. Décrivez le résultat obtenu.

```python
#!pip install spacy
#!python -m spacy download en

import spacy
import tqdm
spacy_nlp = spacy.load('en')

ner = []
for tweet in tqdm.tqdm_notebook(train_X):
    parsed_ner = spacy_nlp(re.sub("#|@","",tweet))

    for element in parsed_ner.ents:
        if element.label_ in ["GPE", "ORG", "PERSON"]:
            ner.append(str(element).lower())
```

    HBox(children=(IntProgress(value=0, max=10204), HTML(value='')))

```python
df = pd.DataFrame(Counter(ner).most_common(10),columns=["term","frequency"])
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>term</th>
      <th>frequency</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>americanair</td>
      <td>822</td>
    </tr>
    <tr>
      <th>1</th>
      <td>southwestair</td>
      <td>688</td>
    </tr>
    <tr>
      <th>2</th>
      <td>virginamerica</td>
      <td>276</td>
    </tr>
    <tr>
      <th>3</th>
      <td>jetblue</td>
      <td>224</td>
    </tr>
    <tr>
      <th>4</th>
      <td>cancelled flightled</td>
      <td>190</td>
    </tr>
    <tr>
      <th>5</th>
      <td>united</td>
      <td>135</td>
    </tr>
    <tr>
      <th>6</th>
      <td>jfk</td>
      <td>89</td>
    </tr>
    <tr>
      <th>7</th>
      <td>cancelled flighted</td>
      <td>69</td>
    </tr>
    <tr>
      <th>8</th>
      <td>lax</td>
      <td>64</td>
    </tr>
    <tr>
      <th>9</th>
      <td>us</td>
      <td>62</td>
    </tr>
  </tbody>
</table>
</div>

```python
from collections import defaultdict
freqs_ner = defaultdict(lambda :defaultdict(int))
for tweet, class_ in tqdm.tqdm_notebook(zip(train_X, train_Y)):
    lower_tw = tweet.lower()
    for ner_term in df.term.values:
        if ner_term in lower_tw:
            parsed_ner = spacy_nlp(re.sub("#|@","",tweet))
            for element in parsed_ner.ents:
                if str(element).lower() in df.term.values:
                    freqs_ner[str(element)][class_]+=1
            break
```

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

```python
df_req = pd.DataFrame(freqs_ner)
df_req.index = ["Negative","Neutral","Positive"]
df_req.plot.bar(figsize=(15,5),rot=0)
plt.title("le nombre de tweets positifs, négatifs ou neutres pour chacunes des 10 NER")
```

Tout comme le diagramme fait à la question 11, cela aide énormément les compagnies aériennes à analyser leurs points faibles et points forts afin d'atteindre une certaine satisfaction des clients (passagers). En effet, par le biais de ce diagramme, nous pouvons déduire de prime à bord que le nombre de commentaires négatifs et largement supérieur au nombre de commentaires positifs ou neutres. Cela est très remarqué pour les deux compagnies AmericanAir et SouthwestAir. En même temps, la copagnie SouthwestAir connait le meilleur nombre de commentaires positifs et neutres dans la liste. Ce qui serait intéressant et de mesurer le ratio entre les commentaire négatifs et positifs/neutres afin d'avoir une métrique de comparaison entre les différentes compagnies aériennes.

##### Question Bonus 2. Générez une table similaire à la question 12 pour le top 10 des NER pour chaque compagnie. (1 point)

Que peut-on conclure de ces résultats?

```python
# TODO
```


# Week 2
## Introduction to Solid State

```python
import numpy as np
import matplotlib.pyplot as plt
import os
import subprocess
import MSD as msd
import fig_params

from scipy import stats

def get_diffusion(file, atom):

    with open(file) as f:
        y = False
        for line in f:
            if str("atom        D ") in line:
                y = True
            if y == True and str(atom) in line:
                d = line.split()
                break
    return d
```

Now that you are familiar with molecular dynamics, you are now going to use it to tackle some real world problems. In the next three weeks you will investigate the transport properties of a simple fluorite material - Ca$F_2$. The transport properties of a material determine many properties that are utilised for modern technological applications. For example, solid oxide fuel cell (SOFCs - Alternative to batteries) materials are dependent on the movement of charge carriers through the solid electrolyte and  nuclear fuel materials oxidise and fall apart and this corrosive behaviour is dependent on the diffusion of oxygen into the lattice. Due to the importance of the transport properties of these materials, scientists and engineers spend large amounts of their time tring to optomise these properties using different stoichiometries, introducing defects and by using different syntheisis techniques. Over the next three weeks you will investigate how the transport properties of Ca$F_2$ are affected by temperature, structural defects (Schottky and Frenkel) and by chemcial dopants (e.g. different cations). A rough breakdown looks as follows
- Week 2
    - Introduction to DL_POLY
    - Tutorial on the calculation of diffusion coefficients
    - Tutorial on the Arhennius equation
    - Molecular dynamics simulations of stoichiomteric Ca$F_2$
- Week 3
    - Frenkel and Schottky defects
- Week 4
    - Dopants

## Introduction to DL_POLY

DL_POLY is a molecular dynamics program maintained by Daresbury laboratories. In contrast to pylj, DL_POLY is a three dimensional molecular dynamics code that is used worldwide by computational scientists for molecular simulation, but it should be noted that the theory is exactly the same and any understanding gained from pylj is completely applicable to DL_POLY. For the next three weeks you will use DL_POLY to run short molecular dynamics simulations on Ca$F_2$. You first need to understand the input files required for DL_POLY.
- CONTROL - This is the file that contains all of the simulation parameters, e.g. simulation temperature, pressure, number of steps e.t.c
- CONFIG - This is the file that contains the structure - i.e. the atomic coordinates of each atom.
- FIELD - This is the file that contains the force field or potential model e.g. Lennard Jones.

Contained within the folder "Input" you will find a file called input.txt. This is the main file that you will interact with over the next three weeks and is used to generate the FIELD, CONTROL and CONFIG. Essentially it is easier to meddle with input.txt than it is to meddle with the 3 DL_POLY files everytime you want to change something. To run metadise we will use the subprocess python module. You specify what program you want to run and the file that you want to run it in, you will need to ensure the file path is correct.

#### It is essential that the codes that were downloaded from [here](https://people.bath.ac.uk/chsscp/teach/adv.bho/progs.zip) are in the Codes/ folder in the parent directory, or this following cell will crash.

```python
subprocess.call('../Codes/metadise.exe', cwd='Input/')
os.rename('Input/control_o0001.dlp', 'Input/CONTROL')
os.rename('Input/config__o0001.dlp', 'Input/CONFIG')
os.rename('Input/field___o0001.dlp', 'Input/FIELD')
```

Now you should have a CONFIG, CONTROL and FIELD file within the input directory. In theory you could just call the DL_POLY program on this directory and your simulation would run. However we need to tweak the CONTROL file in order to set up our desired simulation. Make a new subdirectory in the week 2 directory named "Example" and copy CONFIG, CONTROL and FIELD to that subdirectory. Now edit the CONTROL file.

We want to change the following

`Temperature 300 ---> Temperature 1500`
`Steps 5001 ---> Steps 40000`
`ensemble nve ---> ensemble npt hoover 0.1 0.5`
`trajectory nstraj= 1    istraj=   250 keytrj=0 ---> trajectory nstraj= 0    istraj= 100  keytrj=0`

Now your simulation is ready. As a point of interest it is always good to check your structure before and after the simulation. You can view the CONFIG file in three dimensions using the VESTA program. It is available for free at http://www.jp-minerals.org/vesta/en/download.html . Download it and use it to view your CONFIG, a demonstrator can help if necessary. VESTA can generate nice pictures which will look very good in a lab report.

    Figure 1. Fluorite Ca$F_2$ unit cell visualised in VESTA.

To run DL_POLY from within a notebook use the below command. Keep in mind that this simulation will take 20 or so minutes so be patient.

If you are not comfortable with running things through this notebook then you can copy and paste the dlpoly_classic.exe executable into the Example/ sub directory and then double click the .exe file

```python
subprocess.call('../Codes/dlpoly_classic.exe', cwd='Example/')
```

    0

Once DL_POLY has completed you will find several files relating to your simulaton.
- HISTORY - This file contains the configuration of your system at each step during the simulation. You can view this as a movie using the VMD program - Ask a demonstrator for details
- REVCON - This is the configuration at the end of the simulation - Can be viewed in VESTA - why not check to see how it has changed.
- STATIS - Contains the stats at each step in the simulation.
- OUTPUT - Contains properties

It is now important to understand how we can actually use the details of the simulation to get some information on the properties of the material, e.g. Diffusion coefficients and activation energies.

## Mean Squared Displacements - Calculating diffusion coefficients

As we have seen molecules in liquds, gases and solids do not stay in the same place and move constantly. Think about a drop of dye in a glass of water, as time passes the dye distributes throughout the water. This process is called diffusion and is common throughout nature.

Using the dye as an example, the motion of a dye molecule is not simple. As it moves it is jostled by collisions with other molecules, preventing it from moving in a straight path. If the path is examined in close detail, it will be seen to be a good approximation to a random walk. In mathmatics a random walk is a series of steps, each taken in a random direction. This was analysed by Albert Einstein in a study of Brownian motion and he showed that the mean square of the distance travelled by a particle following a random walk is proportional to the time elapsed.
\begin{align}
\Big \langle r^2 \big \rangle & = 6 D_t + C
\end{align}

where $\Big \langle r^2 \big \rangle$ is the mean squared distance, t is time, D is the diffusion rate and C is a constant.

## What is the mean squared displacement

Going back to the example of the dye in water, lets assume for the sake of simplicity that we are in one dimension. Each step can either be forwards or backwards and we cannot predict which. From a given starting position, what distance is our dye molecule likely to travel after 1000 steps? This can be determined simply by adding together the steps, taking into account the fact that steps backwards subtract from the total, while steps forward add to the total. Since both forward and backward steps are equally probable, we come to the surprising conclusion that the probable distance travelled sums up to zero.

By adding the square of the distance we will always be adding positive numbers to our total which now increases linearly with time. Based upon equation 1 it should now be clear that a plot of $\Big \langle r^2 \big \rangle$ vs time with produce a line, the gradient of which is equal to 6D. Giving us direct access to the diffusion coefficient of the system.

Lets try explore this with an example. Run a short DL_POLY simulation on the input files provided.

You will a small MSD program called MSD.py to analyse your simulation results. First you need to read in the data, the HISTORY file contains a list of the atomic coordiantes held by the atoms during the simulation.

```python
# Read in the HISTORY file

## Provide the path to the simulation and the atom that you want data for.
data = msd.read_history("Example/HISTORY", "F")
```

data is a dictionary variable containing the atomic trajectories, lattice vectors, total number of atoms, and total number of timesteps.

data = {'trajectories':trajectories, 'lv':lv, 'timesteps':timesteps, 'natoms':natoms}

The next step is to calculate the MSD.

```python
# Run the MSD calculation
msd_data = msd.run_msd(data)
```

run_msd returns a dictionary containing the total MSD, the dimensional MSD values and the time.
msd_data = {'msd': msd, 'xmsd': xmsd, 'ymsd': ymsd, 'zmsd': zmsd, 'time': time}

This can then be plotted to give a nice linear relationship.

```python
plt.plot(msd_data['time'], msd_data['msd'], lw=2, color="red", label="MSD")
plt.plot(msd_data['time'], msd_data['xmsd'], lw=2, color="blue", label="X-MSD")
plt.plot(msd_data['time'], msd_data['ymsd'], lw=2, color="green", label="Y-MSD")
plt.plot(msd_data['time'], msd_data['zmsd'], lw=2, color="black", label="Z-MSD")

plt.ylabel("MSD (" r'$\AA$' ")", fontsize=15)
plt.xlabel("Time / ps", fontsize=15)
plt.ylim(0, np.amax(msd_data['msd']))
plt.xlim(0, np.amax(msd_data['time']))
plt.legend(loc=2, frameon=False)
plt.show()
```

To calculate the gradient we need to perform a linear regression on the data.

```python
slope, intercept, r_value, p_value, std_err = stats.linregress(msd_data['time'], msd_data['msd'])
```

The gradient is equal to 6D (6 = dimensionality). So our final diffusion coefficient for the simulation is given by

```python
diffusion_coefficient = (np.average(slope) / 6)
print("Diffusion Coefficient: ", diffusion_coefficient, " X 10 ^-9 (m^-2)")
```

    Diffusion Coefficient:  1.4550282026323467  X 10 ^-9 (m^-2)

## Simulation Length

It is important to consider the lenght of your simulation (Number of steps). Create a new folder called "Example_2", copy the CONFIG, FIELD and CONTROL files from your previous simulation but this time change the number of steps to 10000. Now rerun the simulation.

```python
subprocess.call('../Codes/dlpoly_classic.exe', cwd='Example_2/')
```

    0

```python
data = msd.read_history("Example_2/HISTORY", "F")
msd_data = msd.run_msd(data)

plt.plot(msd_data['time'], msd_data['msd'], lw=2, color="red", label="MSD")
plt.plot(msd_data['time'], msd_data['xmsd'], lw=2, color="blue", label="X-MSD")
plt.plot(msd_data['time'], msd_data['ymsd'], lw=2, color="green", label="Y-MSD")
plt.plot(msd_data['time'], msd_data['zmsd'], lw=2, color="black", label="Z-MSD")

plt.ylabel("MSD (" r'$\AA$' ")", fontsize=15)
plt.xlabel("Time / ps", fontsize=15)
plt.ylim(0, np.amax(msd_data['msd']))
plt.xlim(0, np.amax(msd_data['time']))
plt.legend(loc=2, frameon=False)
plt.show()
```

```python
slope, intercept, r_value, p_value, std_err = stats.linregress(msd_data['time'], msd_data['msd'])

diffusion_coefficient = (np.average(slope) / 6)
print("Diffusion Coefficient: ", diffusion_coefficient, " X 10 ^-9 (m^-2)")
```

    Diffusion Coefficient:  1.5361178863063245  X 10 ^-9 (m^-2)

You will hopefully see that your MSD plot has become considerably less linear. This shows that your simulation has not run long enough and your results will be unrealiable. You will hopefully also see a change to the value of your diffusion coefficient.  The length of your simulation is something that you should keep in mind for the next 3 weeks.

## Arrhenius

The next thing is to use the diffusion coefficients to calcaulte the activation energy for F diffusion. This rea=quires diffusion coefficients from a temperature range. Common sense and chemical intuition suggest that the higher the temperature, the faster a given chemical reaction will proceed. Quantitatively this relationship between the rate a reaction proceeds and its temperature is determined by the Arrhenius Equation. At higher temperatures, the probability that two molecules will collide is higher. This higher collision rate results in a higher kinetic energy, which has an effect on the activation energy of the reaction. The activation energy is the amount of energy required to ensure that a reaction happens.

\begin{align}
k = A * e^{(-Ea / RT)}
\end{align}

where k is the rate coefficient, A is a constant, Ea is the activation energy, R is the universal gas constant, and T is the temperature (in kelvin).

## Week 2 Exercise

Using what you have learned over the last 45 mins your task this week is to calculate the activation energy of F diffusion in Ca$F_2$. You will need to select a temperature range and carry out simulations at different temperatures within that range.
#### Questions to answer
- In what temperature range is Ca$F_2$ completely solid i.e. no diffusion?
- In what range is fluorine essentially liquid i.e. fluorine diffusion with no calcium diffusion?
- What is the melting temperature?
- Plot an Arrhenius plot and determine the activation energies in temperature range - You will need to rearange the equation.

You are encouraged to split the work up within your group and to learn how to view the simulation "movie" using VMD (Ask a demonstrator). VMD is a fantastic program that allows you to visualise your simulation, included below is a video showing a short snippet of an MD simulation of Ca$F_2$. A single F atom has been highlighted to show that diffusion is occuring.

```python
%%HTML
<div align="middle">
</div>
```

Furthermore, VMD can also be used to generate images showing the entire trajectory of the simulation, e.g.

    Figure 2. A figure showing all positions occupied by F during an MD simulation at 1500 K. F positions are shown in orange and Ca atoms are shown in green.

  To save you the time you can use the function declared at the start of this notebook to pull out a diffusion coefficient directly from the simulation output file. MSD.py is a small code to allow visualisation of the MSD plot but it is not neccesary every time you want the diffusion coefficient.

It is up to you how you organise/create your directories but it is reccomended that you start a new notebook. Use the commands/functions used in this notebook to generate your input files, run DL_POLY and extract the diffusion coefficients. The write your own code to generate an Arrhenius plot and calculate the activation energies.
If you finish early then feel free to start week 3.

```python

```


# Molecular Dynamics

##### Syma Khalid

```
from IPython.core.display import HTML
css_file = '../ipython_notebook_styles/ngcmstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 1000px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1200px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Open Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:900px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Arvo', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Arvo', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Arvo', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Arvo', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Arvo', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

## Introduction

Everything that will be done is classical; done in terms of Newtonian physics. Particles will be spherical balls, forces given by harmonic oscillator springs. This *cartoon* model is useful for quite complex situations.

The two lectures will be on

* Molecular mechanics force fields (functional forms and parameterisation).
* Energy minisation.
* Moleculr dynamics simulations
* Analysis and enhanced sampling methods.

### Recommended reading

* Leach
* Grant and ...

## Molecular mechanics

Molecular mechanics is a method by which molecular systems are modelled using classical mechanics.

We want to make models to predict the behaviour of molecules (changes in their structure, and how they interact with other molecules).

Can't do this in the lab thanks to cost, fiddling with underlying "physical" parameters (masses etc), etc.

## Force-fields

Underlying model is that the *potential energy* of a molecule can be described by the *force field* using the principles of classical mechanics.

For a typical force field, the potential energy of the molecule is determined in terms of the

* bond lengths
* bond angles
* dihedral angles
* electrostatics
* repulsion and dispersion
* as well as the Cartesian co-ordinates of the atoms.

Typically there's a massive *table* giving all this information

**Essential** : Cartesian coordinates, and prescription for the force field.

These interactions are not just within a single molecule; *inter*-molecular forces are also important, as well as the standard *intra*-molecular forces.

### Force-fields and potential energy

To formally relate potential energy to the positions of the atmos in our molecule of interest we use

$$
\begin{equation}
  \vec{F} = m \times \vec{a}
\end{equation}
$$

Using that the acceleration $\vec{a}$ is related to the position $\vec{r}$ as

$$
\begin{align}
  \vec{a} &= \frac{d \vec{v}}{d t}, \\
  \vec{v} &= \frac{d \vec{r}}{d t}
\end{align}
$$

and expressing the force as the (negative) gradient of the potential energy,

$$
\begin{equation}
  \vec{F} = - \frac{d V}{d \vec{r}},
\end{equation}
$$

we can combine these results to get

$$
\begin{equation}
  -\frac{d V}{d \vec{r}_i} = m_i \frac{d^2 \vec{r}_i}{d t^2}.
\end{equation}
$$

Here $i$ is thought of as an index on the atoms making up the molecule.

For simplicity divide the foce into two parts:

$$
\begin{equation}
  V_{\text{pot}} = \sum_{\text{bonded}} + \sum_{\text{non-bonded}}.
\end{equation}
$$

The bonded terms include bonds, angles, dihedrals (torsions).

The non-bonded terms include permanent electrostatics, disperion and repulsion.

### Bonded terms: bonds and angles

Usually use a harmonic potential (decent approximation):

$$
\begin{align}
  E &= \sum_{\text{bonds}} k_b (r - r_0)^2 \\ E &= \sum_{\text{angles}} k_{\theta} ( \theta - \theta_0 )^2.
\end{align}
$$

A Morse function (where the bond "breaks" after a certain length, leading to a constant potential above a certain length) would be more accurate for bond length.

The energetic penalty associated with deviating from the equilibrium has the same form for bond lengths *and* bond angles as shown above.

Changing the $k_{b, \theta}$ parameters will broaden or steepen the slope of the potential well

### Bonded terms: dihedrals

Cannot use a simple harmonic term for dihedral angles. This is because there's multiple configurations with the same minima, and two parameters to consider.

Simplest model is to use

$$
\begin{equation}
  E = \sum_{\text{dihedrals}} A \left[ 1 + \cos (n \phi - \phi_0) \right]
\end{equation}
$$

The $A$ parameter controls the amplitude of the potential, whilst the $n$ parameter controls its periodicity, and $\phi_0$ shifts the curve along the rotation angle axis.

### Non-bonded terms: dispersion and repulsion

Non-bonded energy represents the pair-wise sum of the energies of all possible interacting (non-bonded) atoms $i$ and $j$. Typically use the *Lennard-Jones* potential:

$$
\begin{equation}
  V(r) = 4 \epsilon \left[ \left( \frac{\sigma}{r_{ij}} \right)^{12} - \left( \frac{\sigma}{r_{ij}} \right)^{6} \right].
\end{equation}
$$

Dispersion is always attractive, occurs at short range and rapidly falls off with distance.

Repulsion occurs when the distance between interating atmos becomes slightly less than the sum of their contact radii.

The $\epsilon$ term defines the "stickiness" of the potential. What is the optimum interaction between the two particles? Do they "like" to be near each other?

The $\sigma$ term defines the size of the particles. Hard spheres cannot inter-penetrate, so the point at which the energy begins to rise steeply is the limit of the particle.

### Lennard-Jones: parameters

There are (large) tables of parameters giving the values of $\sigma$ and $\epsilon$ for given atoms. When the atmos mix you typically use the Lorenz-Berthelot mixing rules:

$$
\begin{align}
  \sigma_{AB} & = \tfrac{1}{2} \left( \sigma_{AA} + \sigma_{BB} \right), \\
  \epsilon_{AB} & = \sqrt{\epsilon_{AA} \epsilon_{AB}}.
\end{align}
$$

**Note**: you cannot take *one part* of a force field (eg the bond length) and mix with another part of a *different* force field (eg the non-bonded terms). They're self-consistent and come as a whole.

### Non-bonded terms: permanent electrostatics

Even when a molecule does not have a formal charge, because the charge distributions are not uniform the atoms have different *electronegativities*. This leads to atoms have *partial charges*. These charges are additive pairwise interactions again, *but* could be attractive *or* repulsive.

Two representations:

1. Partial charges
2. Multipole moments

The former is easier to code and explain, and more efficient to code. In this case we have

$$
\begin{equation}
  V = \sum_{\text{pairs } (i, j)} \frac{q_i q_j}{4 \pi \epsilon_0 r_{ij}}.
\end{equation}
$$

This is simply Coulomb's law.

This is often the most costly term in the system to solve due to the long range interactions; often the least interesting as well, but crucial.

Various ways of dealing with the long-range terms, including

* Cut-off (at large distance set to a fixed value, typically not zero).
* Reaction field (add an artificial field at large distance).
* Ewald (most comprehensive, most complex; split the region into two parts which separately converge).

## The actual simulations

### Overview

Need

* Suitable initial configuration of the system (usually at equilibrium).
* Force-field
* Simulation parameters (temperature, pressure, etc).

Then

* Integration of Newton's equations of motion

gives as output

* A collection of configurations showing the evolution of the system over the specified timescale.

This then needs to be analysed by the user.

### Newton's equations of motion

The DEs from Newton's second law are solved.

Trajectory that specifies hwo the positions and velocities of the atoms vary with time.

Equations solved by taking a small step in time, predicting new atom positions and velocities at the end of the step.

At the new positions, re-calculate the atomic forces and take another timestep.

### Finite difference methods

Coupled nature of the problem, so we solve them numerically.

Assume that the functions $\vec{r}, \vec{v}, \vec{a}$ can be approximated using a *Taylor series* expansion

$$
\begin{align}
  r(t + \delta t) & = r(t) + v(t) \, \delta t + \tfrac{1}{2} a(t) \, \delta t^2, \\
  v(t + \delta t) & = v(t) + a(t) \, \delta t + \tfrac{1}{2} b(t) \, \delta t^2, \\
  a(t + \delta t) & = a(t) + b(t) \, \delta t + \dots
\end{align}
$$

The **Verlet** and **velocity Verlet** algorithms are the most popular.

### Verlet Algorithm

Uses the positions and accelerations at time $t$ and positions from time $t - \delta t$ to calculate new positions at time $t + \delta t$.

This does not use explicit velocities.

Algorithm uses the Taylor series expansion for $r$ at $t \pm \delta t$ which, when summed, give

$$
\begin{equation}
  r(t + \delta t) + r(t - \delta t) = 2 r(t) + a(t) \, \delta t^2.
\end{equation}
$$

**Advantages**

1. Simple to code
2. Modest memory requirements

**Disadvantages**

1. Imprecise
2. Need to carry two sets of positions ($t$ and $t - \delta t$).
3. Velocities are not explicitly calculated, but estimated. Thus temperature is not reliably represented.

FEEG6016 Simulation and Modelling/2014/Molecular Dynamics Lecture 1.ipynb