Table of Contents

1  Leveraging Word2vec for Text Classification

1.1  Data Preparation

1.2  Gensim Implementation

1.3  Keras Implementation

1.4  Benchmarks

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

# change default style figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,sklearn,matplotlib,xgboost,keras
```

    Using TensorFlow backend.

    Ethen 2019-06-14 09:48:45

    CPython 3.6.4
    IPython 7.5.0

    numpy 1.16.1
    pandas 0.24.2
    sklearn 0.20.3
    matplotlib 3.0.3
    xgboost 0.81
    keras 2.2.2

# Leveraging Word2vec for Text Classification

Many machine learning algorithms requires the input features to be represented as a fixed-length feature
vector. When it comes to texts, one of the most common fixed-length features is one hot encoding methods such as bag of words or tf-idf. The advantage of these approach is that they have fast execution time, while the main drawback is they lose the ordering & semantics of the words.

The motivation behind converting text into semantic vectors (such as the ones provided by `Word2Vec`) is that not only do these type of methods have the capabilities to extract the semantic relationships (e.g. the word powerful should be closely related to strong as oppose to another word like bank), but they should be preserve most of the relevant information about a text while having relatively low dimensionality.

In this notebook, we'll take a look at how a `Word2Vec` model can also be used as a dimensionality reduction algorithm to feed into a text classifier. A good one should be able to extract the signal from the noise efficiently, hence improving the performance of the classifier.

## Data Preparation

We'll download the text classification data, read it into a `pandas` dataframe and split it into train and test set.

```python
import os
from subprocess import call

def download_data(base_dir='.'):
    """download Reuters' text categorization benchmarks from its url."""

    train_data = 'r8-train-no-stop.txt'
    test_data = 'r8-test-no-stop.txt'
    concat_data = 'r8-no-stop.txt'
    base_url = 'http://www.cs.umb.edu/~smimarog/textmining/datasets/'

    if not os.path.isdir(base_dir):
        os.makedirs(base_dir, exist_ok=True)

    dir_prefix_flag = ' --directory-prefix ' + base_dir

    # brew install wget
    # on a mac if you don't have it
    train_data_path = os.path.join(base_dir, train_data)
    if not os.path.isfile(train_data_path):
        call('wget ' + base_url + train_data + dir_prefix_flag, shell=True)

    test_data_path = os.path.join(base_dir, test_data)
    if not os.path.isfile(test_data_path):
        call('wget ' + base_url + test_data + dir_prefix_flag, shell=True)

    concat_data_path = os.path.join(base_dir, concat_data)
    if not os.path.isfile(concat_data_path):
        # concatenate train and test files, we'll make our own train-test splits
        # the > piping symbol directs the concatenated file to a new file, it
        # will replace the file if it already exists; on the other hand, the >> symbol
        # will append if it already exists
        train_test_path = os.path.join(base_dir, 'r8-*-no-stop.txt')
        call('cat {} > {}'.format(train_test_path, concat_data_path), shell=True)

    return concat_data_path
```

```python
base_dir = 'data'
data_path = download_data(base_dir)
data_path
```

    'data/r8-no-stop.txt'

```python
def load_data(data_path):
    texts, labels = [], []
    with open(data_path) as f:
        for line in f:
            label, text = line.split('\t')
            # texts are already tokenized, just split on space
            # in a real use-case we would put more effort in preprocessing
            texts.append(text.split())
            labels.append(label)

    return pd.DataFrame({'texts': texts, 'labels': labels})
```

```python
data = load_data(data_path)
data['labels'] = data['labels'].astype('category')
print('dimension: ', data.shape)
data.head()
```

    dimension:  (7674, 2)

      texts
      labels

      0
      [asian, exporters, fear, damage, japan, rift, ...
      trade

      1
      [china, daily, vermin, eat, pct, grain, stocks...
      grain

      2
      [australian, foreign, ship, ban, ends, nsw, po...
      ship

      3
      [sumitomo, bank, aims, quick, recovery, merger...
      acq

      4
      [amatil, proposes, two, for, bonus, share, iss...
      earn

```python
label_mapping = data['labels'].cat.categories
data['labels'] = data['labels'].cat.codes
X = data['texts']
y = data['labels']
```

```python
test_size = 0.1
random_state = 1234

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=test_size, random_state=random_state, stratify=y)

# val_size = 0.1
# X_train, X_val, y_train, y_val = train_test_split(
#     X_train, y_train, test_size=val_size, random_state=random_state, stratify=y_train)
```

## Gensim Implementation

After feeding the `Word2Vec` algorithm with our corpus, it will learn a vector representation for each word. This by itself, however, is still not enough to be used as features for text classification as each record in our data is a document not a word.

To extend these word vectors and generate document level vectors, we'll take the naive approach and use an average of all the words in the document (We could also leverage tf-idf to generate a weighted-average version, but that is not done here). The `Word2Vec` algorithm is wrapped inside a sklearn-compatible transformer which can be used almost the same way as `CountVectorizer` or `TfidfVectorizer` from `sklearn.feature_extraction.text`. Almost - because sklearn vectorizers can also do their own tokenization - a feature which we won't be using anyway because the corpus we will be using is already tokenized.

In the next few code chunks, we will build a pipeline that transforms the text into low dimensional vectors via average word vectors as use it to fit a boosted tree model, we then report the performance of the training/test set.

The `transformers` folder that contains the implementation is at the following [link](https://github.com/ethen8181/machine-learning/tree/master/keras/text_classification/transformers).

```python
from xgboost import XGBClassifier
from sklearn.pipeline import Pipeline
from transformers import GensimWord2VecVectorizer

gensim_word2vec_tr = GensimWord2VecVectorizer(size=50, min_count=3, sg=1, alpha=0.025, iter=10)
xgb = XGBClassifier(learning_rate=0.01, n_estimators=100, n_jobs=-1)
w2v_xgb = Pipeline([
    ('w2v', gensim_word2vec_tr),
    ('xgb', xgb)
])
w2v_xgb
```

    Pipeline(memory=None,
         steps=[('w2v', GensimWord2VecVectorizer(alpha=0.025, batch_words=10000, callbacks=(),
                 cbow_mean=1, compute_loss=False,
                 hashfxn=<built-in function hash>, hs=0, iter=10,
                 max_final_vocab=None, max_vocab_size=None, min_alpha=0.0001,
                 min_count=3, negati...tate=0, reg_alpha=0, reg_lambda=1, scale_pos_weight=1,
           seed=None, silent=True, subsample=1))])

```python
import time

start = time.time()
w2v_xgb.fit(X_train, y_train)
elapse = time.time() - start
print('elapsed: ', elapse)
w2v_xgb
```

    elapsed:  11.784907102584839

    Pipeline(memory=None,
         steps=[('w2v', GensimWord2VecVectorizer(alpha=0.025, batch_words=10000, callbacks=(),
                 cbow_mean=1, compute_loss=False,
                 hashfxn=<built-in function hash>, hs=0, iter=10,
                 max_final_vocab=None, max_vocab_size=None, min_alpha=0.0001,
                 min_count=3, negati...
           reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
           silent=True, subsample=1))])

```python
from sklearn.metrics import accuracy_score, confusion_matrix

y_train_pred = w2v_xgb.predict(X_train)
print('Training set accuracy %s' % accuracy_score(y_train, y_train_pred))
confusion_matrix(y_train, y_train_pred)
```

    Training set accuracy 0.9485954242687518

    array([[2026,    2,   29,    0,    0,    3,    0,    3],
           [  12,  313,    3,    0,    0,    0,    7,    1],
           [ 148,    0, 3381,    0,    0,    1,    0,    0],
           [   3,    1,    6,   23,    1,    0,    4,    8],
           [   5,    0,    3,    0,  205,   28,    1,    2],
           [   5,    1,    7,    0,    9,  238,    0,    4],
           [  16,    6,    5,    0,    2,    1,   96,    4],
           [   3,    1,    7,    0,    7,    6,    0,  269]])

```python
y_test_pred = w2v_xgb.predict(X_test)
print('Test set accuracy %s' % accuracy_score(y_test, y_test_pred))
confusion_matrix(y_test, y_test_pred)
```

    Test set accuracy 0.9244791666666666

    array([[224,   0,   3,   0,   0,   1,   0,   1],
           [  2,  34,   0,   0,   0,   0,   1,   1],
           [ 17,   0, 375,   0,   0,   0,   1,   0],
           [  0,   0,   0,   1,   0,   0,   1,   3],
           [  1,   1,   0,   0,  19,   5,   0,   1],
           [  2,   0,   1,   0,   3,  21,   0,   2],
           [  1,   1,   2,   0,   0,   0,   8,   2],
           [  0,   2,   0,   0,   0,   3,   0,  28]])

We can extract the Word2vec part of the pipeline and do some sanity check of whether the word vectors that were learned made any sense.

```python
vocab_size = len(w2v_xgb.named_steps['w2v'].model_.wv.index2word)
print('vocabulary size:', vocab_size)
w2v_xgb.named_steps['w2v'].model_.wv.most_similar(positive=['stock'])
```

    vocabulary size: 9846

    [('shares', 0.8288736939430237),
     ('common', 0.8102667927742004),
     ('dilutive', 0.7489669919013977),
     ('effected', 0.7259572744369507),
     ('warrants', 0.718142032623291),
     ('lazo', 0.7158320546150208),
     ('pubco', 0.7046886682510376),
     ('dealings', 0.7039074897766113),
     ('fractional', 0.7031500339508057),
     ('spartech', 0.7023240327835083)]

## Keras Implementation

We'll also show how we can use a generic deep learning framework to implement the `Wor2Vec` part of the pipeline. There are many variants of `Wor2Vec`, here, we'll only be implementing skip-gram and negative sampling.

The flow would look like the following:

An (integer) input of a target word and a real or negative context word. This is essentially the skipgram part where any word within the context of the target word is a real context word and we randomly draw from the rest of the vocabulary to serve as the negative context words.

An embedding layer lookup (i.e. looking up the integer index of the word in the embedding matrix to get the word vector).

A dot product operation. As the network trains, words which are similar should end up having similar embedding vectors. The most popular way of measuring similarity between two vectors $A$ and $B$ is the cosine similarity.

\begin{align}
similarity = cos(\theta) = \frac{\textbf{A}\cdot\textbf{B}}{\parallel\textbf{A}\parallel_2 \parallel \textbf{B} \parallel_2}
\end{align}

The denominator of this measure acts to normalize the result – the real similarity operation is on the numerator: the dot product between vectors $A$ and $B$.

Followed by a sigmoid output layer. Our network is a binary classifier since it's distinguishing words from the same context versus those that aren't.

```python
# the keras model/graph would look something like this:
from keras import layers, optimizers, Model

# adjustable parameter that control the dimension of the word vectors
embed_size = 100

input_center = layers.Input((1,))
input_context = layers.Input((1,))

embedding = layers.Embedding(vocab_size, embed_size, input_length=1, name='embed_in')
center = embedding(input_center)  # shape [seq_len, # features (1), embed_size]
context = embedding(input_context)

center = layers.Reshape((embed_size,))(center)
context = layers.Reshape((embed_size,))(context)

dot_product = layers.dot([center, context], axes=1)
output = layers.Dense(1, activation='sigmoid')(dot_product)
model = Model(inputs=[input_center, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop(lr=0.01))
model.summary()
```

    WARNING:tensorflow:From /Users/mingyuliu/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            (None, 1)            0
    __________________________________________________________________________________________________
    input_2 (InputLayer)            (None, 1)            0
    __________________________________________________________________________________________________
    embed_in (Embedding)            (None, 1, 100)       984600      input_1[0][0]
                                                                     input_2[0][0]
    __________________________________________________________________________________________________
    reshape_1 (Reshape)             (None, 100)          0           embed_in[0][0]
    __________________________________________________________________________________________________
    reshape_2 (Reshape)             (None, 100)          0           embed_in[1][0]
    __________________________________________________________________________________________________
    dot_1 (Dot)                     (None, 1)            0           reshape_1[0][0]
                                                                     reshape_2[0][0]
    __________________________________________________________________________________________________
    dense_1 (Dense)                 (None, 1)            2           dot_1[0][0]
    ==================================================================================================
    Total params: 984,602
    Trainable params: 984,602
    Non-trainable params: 0
    __________________________________________________________________________________________________

```python
# then we can feed in the skipgram and its label (whether the word pair is in or outside
# the context)
batch_center = [2354, 2354, 2354, 69, 69]
batch_context = [4288, 203, 69, 2535, 815]
batch_label = [0, 1, 1, 0, 1]
model.train_on_batch([batch_center, batch_context], batch_label)
```

    WARNING:tensorflow:From /Users/mingyuliu/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.

    0.6951684

The `transformers` folder that contains the implementation is at the following [link](https://github.com/ethen8181/machine-learning/tree/master/keras/text_classification/transformers).

```python
from transformers import KerasWord2VecVectorizer

keras_word2vec_tr = KerasWord2VecVectorizer(embed_size=50, min_count=3, epochs=5000,
                                            negative_samples=2)
keras_word2vec_tr
```

    KerasWord2VecVectorizer(batch_size=64, embed_size=50, epochs=5000,
                learning_rate=0.05, min_count=3, negative_samples=2,
                sort_vocab=True, use_sampling_table=True, window_size=5)

```python
keras_w2v_xgb = Pipeline([
    ('w2v', keras_word2vec_tr),
    ('xgb', xgb)
])

keras_w2v_xgb.fit(X_train, y_train)
```

    100%|██████████| 5000/5000 [02:49<00:00, 29.45it/s]

    Pipeline(memory=None,
         steps=[('w2v', KerasWord2VecVectorizer(batch_size=64, embed_size=50, epochs=5000,
                learning_rate=0.05, min_count=3, negative_samples=2,
                sort_vocab=True, use_sampling_table=True, window_size=5)), ('xgb', XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
         ...
           reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
           silent=True, subsample=1))])

```python
y_train_pred = keras_w2v_xgb.predict(X_train)
print('Training set accuracy %s' % accuracy_score(y_train, y_train_pred))
confusion_matrix(y_train, y_train_pred)
```

    Training set accuracy 0.9093541847668694

    array([[1966,    4,   69,    0,    6,    6,    2,   10],
           [  57,  255,   14,    0,    1,    1,    5,    3],
           [ 143,    4, 3377,    0,    2,    0,    2,    2],
           [  10,    2,    3,   16,    0,    2,    8,    5],
           [   5,    2,    5,    0,  209,   16,    0,    7],
           [  19,    6,   13,    0,   26,  177,    0,   23],
           [  43,    6,    7,    4,    0,    1,   65,    4],
           [  10,   14,   39,    0,    5,    7,    3,  215]])

```python
y_test_pred = keras_w2v_xgb.predict(X_test)
print('Test set accuracy %s' % accuracy_score(y_test, y_test_pred))
confusion_matrix(y_test, y_test_pred)
```

    Test set accuracy 0.8958333333333334

    array([[218,   1,   6,   1,   0,   0,   1,   2],
           [  7,  28,   1,   0,   0,   0,   1,   1],
           [ 20,   1, 371,   0,   1,   0,   0,   0],
           [  1,   1,   1,   0,   0,   0,   1,   1],
           [  0,   0,   4,   0,  18,   4,   0,   1],
           [  2,   0,   1,   0,   4,  20,   0,   2],
           [  2,   2,   1,   0,   0,   1,   7,   1],
           [  2,   0,   2,   0,   2,   1,   0,  26]])

```python
print('vocabulary size:', keras_w2v_xgb.named_steps['w2v'].vocab_size_)
keras_w2v_xgb.named_steps['w2v'].most_similar(positive=['stock'])
```

    vocabulary size: 9847

    [('common', 0.7450751),
     ('split', 0.72033733),
     ('annual', 0.7107709),
     ('acquistion', 0.7084387),
     ('remaining', 0.6948875),
     ('total', 0.69227046),
     ('shareholder', 0.68726707),
     ('shareholders', 0.6678084),
     ('associates', 0.64986753),
     ('board', 0.6477556)]

## Benchmarks

We'll compare the word2vec + xgboost approach with tfidf + logistic regression. The latter approach is known for its interpretability and fast training time, hence serves as a strong baseline.

Note that for sklearn's tfidf, we didn't use the default analyzer 'words', as this means it expects that input is a single string which it will try to split into individual words, but our texts are already tokenized, i.e. already lists of words.

```python
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english', analyzer=lambda x: x)
logistic = LogisticRegression(solver='liblinear', multi_class='auto')

tfidf_logistic = Pipeline([
    ('tfidf', tfidf),
    ('logistic', logistic)
])
```

```python
from scipy.stats import randint, uniform

w2v_params = {'w2v__size': [100, 150, 200]}
tfidf_params = {'tfidf__ngram_range': [(1, 1), (1, 2)]}
logistic_params = {'logistic__C': [0.5, 1.0, 1.5]}
xgb_params = {'xgb__max_depth': randint(low=3, high=12),
              'xgb__colsample_bytree': uniform(loc=0.8, scale=0.2),
              'xgb__subsample': uniform(loc=0.8, scale=0.2)}

tfidf_logistic_params = {**tfidf_params, **logistic_params}
w2v_xgb_params = {**w2v_params, **xgb_params}
```

```python
from sklearn.model_selection import RandomizedSearchCV

cv = 3
n_iter = 3
random_state = 1234
scoring = 'accuracy'

all_models = [
    ('w2v_xgb', w2v_xgb, w2v_xgb_params),
    ('tfidf_logistic', tfidf_logistic, tfidf_logistic_params)
]

all_models_info = []
for name, model, params in all_models:
    print('training:', name)
    model_tuned = RandomizedSearchCV(
        estimator=model,
        param_distributions=params,
        cv=cv,
        n_iter=n_iter,
        n_jobs=-1,
        verbose=1,
        scoring=scoring,
        random_state=random_state,
        return_train_score=False
    ).fit(X_train, y_train)

    y_test_pred = model_tuned.predict(X_test)
    test_score = accuracy_score(y_test, y_test_pred)
    info = name, model_tuned.best_score_, test_score, model_tuned
    all_models_info.append(info)

columns = ['model_name', 'train_score', 'test_score', 'estimator']
results = pd.DataFrame(all_models_info, columns=columns)
results = (results
           .sort_values('test_score', ascending=False)
           .reset_index(drop=True))
results
```

    training: w2v_xgb
    Fitting 3 folds for each of 3 candidates, totalling 9 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   4 out of   9 | elapsed:  2.0min remaining:  2.4min
    [Parallel(n_jobs=-1)]: Done   9 out of   9 | elapsed:  2.3min finished

    training: tfidf_logistic
    Fitting 3 folds for each of 3 candidates, totalling 9 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   4 out of   9 | elapsed:    1.4s remaining:    1.7s
    [Parallel(n_jobs=-1)]: Done   9 out of   9 | elapsed:    2.0s finished

      model_name
      train_score
      test_score
      estimator

      0
      tfidf_logistic
      0.949175
      0.962240
      RandomizedSearchCV(cv=3, error_score='raise-de...

      1
      w2v_xgb
      0.954532
      0.958333
      RandomizedSearchCV(cv=3, error_score='raise-de...

Note that different run may result in different performance being reported. And as our dataset changes, different approaches might that worked the best on one dataset might no longer be the best. Especially since the dataset we're working with here isn't very big, training an embedding from scratch will most likely not reach its full potential.

There are many other text classification techniques in the deep learning realm that we haven't yet explored, we'll leave that for another day.

# Reference

- [Blog: A Word2Vec Keras tutorial](https://adventuresinmachinelearning.com/word2vec-keras-tutorial/)
- [Blog: Text Classification With Word2Vec](http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/)


# Learning of constraints

## Objective
This notebook contains a very simple example showing how to train a feedforward neural network to learn how to explain its predictions.

## Outline
- [Objective](#Objective)
- [Problem description](#Problem-description)
    - [The course of the black box](#The-course-of-the-black-box)
    - [How to train like an adult](#How-to-train-like-an-adult)
    - [Learning of constraints](#Learning-of-constraints)
    - [Multi-label classification](#Multi-label-classification)
- [Libraries](#Libraries)
- [Dataset](#Dataset)
- [Learning concepts](#Learning-concepts)
- [**Explaining concepts**](#Explaining-concepts)
- [References](#References)

## Problem description

### The course of the black box
**Humans don't like black boxes**. We all (*humans*) suffer of incredulity. This cognitive bias prevents us from relying upon something we don't understand. In finding the trade-off between complexity and accuracy, we are often biased towards simple solutions. That's why we (*humans*) don't like deep learning. It's the Occam's razor revenge. Machine learning researchers are usually worried about the course of the dimensionality. But we actually need to realize that one of the main issues that is limiting deep learning applications is being a black box. The [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) has coursed neural networks. **It's the course of the black box**. <!--curse not course?-->

What is the actual issue here? Are neural networks intrinsically coursed? They are just optimizers with a scandalous number of parameters after all. How could we interpret a nonlinear decision function with millions of parameters? Well, in some sense we are doing it every single day, don't we? Human decisions and reasoning are a function of our brain and mind. Would this function have less parameters than an *artificial neural network*? I don't think so. So, what's the difference? Why do we trust our grannies but not our deep learning models? I think the main difference here is that our grannies can **explain** themselves. They can provide us a step-by-step explanation, *a logical reasoning*$^1$ that can help us understand what they are saying. Our nets don't do that... Nay, we dind't ask them! But how could we?

Let's step back a little. Let's define what we *really* want to ask them (our nets, not our grannies).

$^1$: it depends on your granny...

### How to train like an adult
So, what do we *really* want? I think the problem is that we are training our networks like a child. So now we get answers as from a child. Would you blame your child because his/her answers are not logically sound? That would be ridiculous if not inappropriate. We need a paradigm shift. **Instead of training our net like a child, why don't train it like an adult?** What's the difference? We need to teach them logic. So, let's start with defining what kind of logic we need them to learn about.

In most cases, researchers might be interested in scientific interpretability. One way of thinking about scientific interpretability is in terms of logic. **A (*scientific*) argument is interpretable from a human standpoint, if it can be described by a limited set of concepts linked by logical rules**. A simple example is the syllogism:

> All men are mortal.
>
> Socrates is a man.
>
> Therefore, Socrates is mortal.

So now our problem is: **how to make a neural network learn how to explain its predictions logically**?

### Learning of constraints
We can divide the problem into **3 steps**.

**First**, we need to teach our neural network basic concepts. How? As usual, just pick a problem and train your network (let's call it $N1$). We are now in the so called "concept space". For instance, the "perceptual space" of MNIST is an image of a digit, while the "concept space" is the "label" of that image, the digit, the abstract concept represented by the image. We, *humans*, tend to reason in this "concept space" rather than in our perceptual space. That's what we want from our net! So...

**Second**, we need to teach our network the logical relationships between concepts. How? One option is to use (by an ironic twist of fate) another neural network! Let me give you a simple example. Let's say we have trained our network $N1$ to recognize the following concepts: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, even, and odd. Let's say we are interested in understanding interpreting the prediction "even". In other words, we want to understand how our network is interpreting the concept "even" with respect to the other concepts. The idea is to train another neural network $N2$ taking as inputs the predictions of the network $N1$ and having just one output representing the concept "even". So now we have a neural network ($N2$) learning the concept "even" using the concepts: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and odd (we need to drop "even" as we might have ended up with a tautology!).

**Third**, we need to ask our network for a logical explanation. How? That's the fun part. We are going to generate the **truth table** of the network $N2$, representing its logical reasoning. Let's do it step by step:
1. generate an artificial dataset in the concept space representing **all possible combinations of truth values taken by each concept**. To make it easier, we will assume that each concept can just take 2 truth values: TRUE and FALSE.
2. feed the network $N2$ with this table.
3. The output of the network will represent the truth degree of each combination of concepts.

For instance, let's take the combination:

| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | odd |
|-|-|-|-|-|-|-|-|-|-|-|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |

If for this input the network predicts "even" with probability $\approx 0$, we might have just discovered something new! That is: "odd" $\implies$ not "even". That's a new concept!

Let's take another input:

| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | odd |
|-|-|-|-|-|-|-|-|-|-|-|
| 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

If for this input the network predicts "even" with probability $\approx 1$, we might have just discovered something new again! That is: "two" $\implies$ "even". That's another new concept!

So now we have a very simple way of interrogating our deep learning model! We can ask for explanations and we can extract general logical rules representing its decisions.

### Multi-label classification
In order to show a concrete example, in this notebook we consider a generic [**multi-label classification problem**](https://en.wikipedia.org/wiki/Multi-label_classification), that is a learning problem where each input example belongs to one or more classes.

We focus on **neural network-based** methods, that implicitly learn from supervisions.

More formally, we consider data belonging to the **perceptual space** $x \in \mathbb{R}^d$. Each sample $x$ is also associated with a boolean vector $y \in \mathbb{R}^n$. For any $x$, $y_i \in \{0, 1\}$ represents the membership degree of the example $x$ to the $i$-th class.

We consider a multi-output feedforward neural network classifier. Each output unit is associated to a **task function** $f_i: \mathbb{R}^d \rightarrow [0,1]$, for $i=1,...,n$, that predicts how strongly an input example belongs to the considered class. For any $x \in X$, $\hat{y}_i = f_i(x) \in [0, 1]$ represents the *predicted* membership degree of the example $x$ to the $i$-th class. We indicate with $f(x)$ the function that returns the $n$-dimensional vector $\hat{y} \in \mathbb{R}^n$ with the outputs of all the task functions. Such vector belongs to the so-called **concept space**.

This is the "standard network" learning basic concepts. Let's now consider another network $\psi_h: \mathbb{R}^{n-1} \rightarrow [0,1]$ whose input domain is the concept space (except for the target concept) and whose output represents the truth degree of the concept we want to explain.

Let's now dive into the code...

## Libraries
First we need to import some useful libraries:

```python
import torch
import numpy as np
from sklearn.datasets import load_digits
from sklearn.preprocessing import OneHotEncoder
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from itertools import product
import pandas as pd
```

## Dataset

In this notebook we will use a simplified version of the MNIST dataset called DIGITS. Each sample $x_i$ is an $8 \times 8$-pixels' handritten image representing a digit, while the supervision $y_i$ represents its numeric value.

We can first load the dataset and have a look at some of his properties:

```python
X, y = load_digits(return_X_y=True)

print(f'X shape: {X.shape}\nClasses: {np.unique(y)}')
```

    X shape: (1797, 64)
    Classes: [0 1 2 3 4 5 6 7 8 9]

As you can see, the images are flattened in a $64$-length vector, but we can easily visualize samples by reshaping the input:

```python
# show the first ten images
figs = X[:10].reshape((10, 8, 8))
plt.figure(figsize=[7, 7])
for i, fig in enumerate(figs):
    plt.subplot(5, 5, i+1)
    plt.title(f'Class: {y[i]}')
    sns.heatmap(fig, cbar=False)
    plt.axis('off')
plt.tight_layout()
plt.show()
```

As we are framing our problem as a multi-task classification problem, we need to encode our supervisions in a one-hot representation:

```python
enc = OneHotEncoder()
y1h = enc.fit_transform(y.reshape(-1, 1)).toarray()

print(f'Before: {y.shape}\nAfter: {y1h.shape}')
```

    Before: (1797,)
    After: (1797, 10)

To make it more fun, we will add two additional task functions: ODD (11-th column) and EVEN (12-th column).

```python
y2 = np.zeros((len(y), 2))
for i, yi in enumerate(y):
    if yi % 2:
        y2[i, 0] = 1
    else:
        y2[i, 1] = 1
y1h2 = np.hstack((y1h, y2))

print(f'Target vector shape: {y1h2.shape}')
for i in range(10):
    print(f'Example ({y[i]}): {y1h2[i]}')
```

    Target vector shape: (1797, 12)
    Example (0): [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
    Example (1): [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]
    Example (2): [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
    Example (3): [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1. 0.]
    Example (4): [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1.]
    Example (5): [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 1. 0.]
    Example (6): [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 1.]
    Example (7): [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0.]
    Example (8): [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1.]
    Example (9): [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0.]

We are now ready to split our dataset into a training and a test set:

```python
X_train_np, X_test_np, y_train_np, y_test_np = train_test_split(X, y1h2, test_size=0.33, random_state=42)
```

Finally, we need to transoform our data into torch tensors to be processed by the network:

```python
x_train = torch.FloatTensor(X_train_np)
y_train = torch.FloatTensor(y_train_np)
x_test = torch.FloatTensor(X_test_np)
y_test = torch.FloatTensor(y_test_np)
```

## Learning concepts
We will use a very simple (and quite standard) feedforward neural network with three layers, ReLU activations, and $12$ output units (one for each task function) with sigmoid activations. We are not using softmax as in our problem more than one predicate on the task function can be true at the same time (e.g. $isTwo(x)$ and $isEven(x)$ should be true at the same time).

```python
class FeedForwardNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(FeedForwardNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, H)
        self.linear3 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        h = self.linear1(x)
        h = torch.nn.functional.relu(h)
        h = self.linear2(h)
        h = torch.nn.functional.relu(h)
        h = self.linear3(h)
        y_pred = torch.sigmoid(h)
        return y_pred
```

We can now generate an instance of the network:

```python
din, dh, dout = x_train.shape[1], 20, y_train.shape[1]
model = FeedForwardNet(din, dh, dout)

print(model)
```

    FeedForwardNet(
      (linear1): Linear(in_features=64, out_features=20, bias=True)
      (linear2): Linear(in_features=20, out_features=20, bias=True)
      (linear3): Linear(in_features=20, out_features=12, bias=True)
    )

The train loop is quite standard:

```python
loss = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
model.train()
epoch = 2000
for epoch in range(epoch):
    optimizer.zero_grad()

    # Forward pass
    y_pred = model(x_train)
    y_pred_np = y_pred.detach().numpy()

    # Compute Loss
    tot_loss = loss(y_pred, y_train)

    # compute accuracy
    y_pred_d = (y_pred > 0.5).detach().numpy()
    accuracy = ((y_pred_d == y_train_np).sum(axis=1) == y_train_np.shape[1]).mean()

    if epoch % 100 == 0:
        print(f'Epoch {epoch + 1}: '
              f'total loss: {tot_loss.item():.4f} '
              f'| accuracy: {accuracy:.4f} ')

    # Backward pass
    tot_loss.backward()
    optimizer.step()
```

    Epoch 1: total loss: 0.8232 | accuracy: 0.0000
    Epoch 101: total loss: 0.0236 | accuracy: 0.9327
    Epoch 201: total loss: 0.0050 | accuracy: 0.9958
    Epoch 301: total loss: 0.0017 | accuracy: 1.0000
    Epoch 401: total loss: 0.0008 | accuracy: 1.0000
    Epoch 501: total loss: 0.0004 | accuracy: 1.0000
    Epoch 601: total loss: 0.0003 | accuracy: 1.0000
    Epoch 701: total loss: 0.0002 | accuracy: 1.0000
    Epoch 801: total loss: 0.0001 | accuracy: 1.0000
    Epoch 901: total loss: 0.0001 | accuracy: 1.0000
    Epoch 1001: total loss: 0.0001 | accuracy: 1.0000
    Epoch 1101: total loss: 0.0001 | accuracy: 1.0000
    Epoch 1201: total loss: 0.0001 | accuracy: 1.0000
    Epoch 1301: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1401: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1501: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1601: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1701: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1801: total loss: 0.0000 | accuracy: 1.0000
    Epoch 1901: total loss: 0.0000 | accuracy: 1.0000

Once the network is trained we can compute the test accuracy:

```python
y_pred = model(x_test)

# compute accuracy
y_pred_round = (y_pred > 0.5).to(torch.float).detach().numpy()
accuracy = ((y_pred_round == y_test_np).sum(axis=1) == y_test_np.shape[1]).mean()

print(f'accuracy: {accuracy:.4f}')
```

    accuracy: 0.8973

## Explaining concepts
Let's now focus on the concept "even". Our objective is to find an explanation for "even" predictions. What "even" means in terms of the other task functions?

Let's first define the $\psi$ function as a one-layer feedforward neural network with $n-1$ input (one for each task function except for $f_{even}$) and one output representing the truth degree of the concept "even".

```python
class ExplainEven(torch.nn.Module):
    def __init__(self, D_in):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(ExplainEven, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, 10)
        self.linear2 = torch.nn.Linear(10, 1)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        h = self.linear1(x)
        h = self.linear2(h)
        y_pred = torch.sigmoid(h)
        return y_pred
```

We can now generate the training set for this network. It is just the discretized output of the previous network:

```python
y_pred_train = model(x_train).detach().numpy().astype(float)
y_pred_test = model(x_test).detach().numpy().astype(float)

x_concepts_train_np, y_concepts_train_np = y_pred_train[:, :-1], y_pred_train[:, -1]
x_concepts_test_np, y_concepts_test_np = y_pred_test[:, :-1], y_pred_test[:, -1]

x_concepts_train, y_concepts_train = torch.FloatTensor(x_concepts_train_np), torch.FloatTensor(y_concepts_train_np)
x_concepts_test, y_concepts_test = torch.FloatTensor(x_concepts_test_np), torch.FloatTensor(y_concepts_test_np)
```

We are now ready to train our ExplainNet:

```python
D_in = y_pred_train.shape[1] - 1
even_net = ExplainEven(D_in)
print(even_net)

optimizer = torch.optim.Adam(even_net.parameters(), lr=0.01)
even_net.train()
epoch = 500
accuracy = 0
for epoch in range(epoch):
    optimizer.zero_grad()
    # Forward pass
    y_pred = even_net(x_concepts_train)
    y_pred_np = y_pred.detach().numpy()

    # Compute Loss
    tot_loss = loss(y_pred.squeeze(), y_concepts_train) + 0.08 * even_net.linear1.weight.norm(1)  * even_net.linear2.weight.norm(1)

    # compute accuracy
    y_pred_d = (y_pred > 0.5).detach().numpy().ravel()
    accuracy = (y_pred_d == (y_concepts_train_np>0.5)).mean()

    if epoch % 100 == 0:
        print(f'Epoch {epoch + 1}: '
              f'total loss: {tot_loss.item():.4f} '
              f'| accuracy: {accuracy:.4f} ')

    # Backward pass
    tot_loss.backward()
    optimizer.step()
```

    ExplainEven(
      (linear1): Linear(in_features=11, out_features=10, bias=True)
      (linear2): Linear(in_features=10, out_features=1, bias=True)
    )
    Epoch 1: total loss: 2.5795 | accuracy: 0.5079
    Epoch 101: total loss: 0.6931 | accuracy: 0.5079
    Epoch 201: total loss: 0.6922 | accuracy: 0.5079
    Epoch 301: total loss: 0.6710 | accuracy: 1.0000
    Epoch 401: total loss: 0.4969 | accuracy: 1.0000

## Rule extraction
Let's now find out how can we extract logical rules.

Lets just focus on a subset of weights (the ones representing strongest connections among concepts):

```python
weights, bias = [], []
for i, param in enumerate(even_net.parameters()):
    if i % 2 == 0:
        param_absneg = -torch.abs(param)
        idx = torch.topk(param_absneg, k=param_absneg.shape[1]-2)[1]
        for i in range(len(idx)):
            param[i, idx[i]] = 0
        param
        weights.append(param.detach().numpy())
    else:
        bias.append(param.detach().numpy())
```

```python
bias
```

    [array([ 0.3989522 ,  0.13632365, -0.40638036, -0.03830609, -0.35561085,
             0.05306427,  0.07716022,  0.04836779, -1.0059371 ,  0.41719967],
           dtype=float32),
     array([0.47904056], dtype=float32)]

```python
for i in even_net.parameters():
    print(i)
```

    Parameter containing:
    tensor([[ 3.3663e-03,  0.0000e+00,  0.0000e+00,  2.7577e-03,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00],
            [ 0.0000e+00,  0.0000e+00,  0.0000e+00, -5.2165e-03,  0.0000e+00,
              0.0000e+00,  0.0000e+00, -5.2380e-03,  0.0000e+00,  0.0000e+00,
              0.0000e+00],
            [-3.3996e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00, -3.0479e-03,
              0.0000e+00],
            [ 0.0000e+00,  3.5091e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  3.0491e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00],
            [-7.1541e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              4.0887e-03],
            [ 0.0000e+00,  5.4402e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              5.2893e-03,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00],
            [ 0.0000e+00,  0.0000e+00, -3.5255e-03,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  3.4183e-03,
              0.0000e+00],
            [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  5.4572e-03,
              4.0927e-03],
            [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
              0.0000e+00,  0.0000e+00,  0.0000e+00, -5.5383e-03,  0.0000e+00,
              2.8439e+00],
            [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00, -5.1365e-03,
              0.0000e+00,  0.0000e+00, -2.9250e-03,  0.0000e+00,  0.0000e+00,
              0.0000e+00]], grad_fn=<CopySlices>)
    Parameter containing:
    tensor([ 0.3990,  0.1363, -0.4064, -0.0383, -0.3556,  0.0531,  0.0772,  0.0484,
            -1.0059,  0.4172], requires_grad=True)
    Parameter containing:
    tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000, -0.0032,  0.0000,
             -1.1097,  0.0000]], grad_fn=<CopySlices>)
    Parameter containing:
    tensor([0.4790], requires_grad=True)

Now let's use the corresponding concepts to extract the rules:

```python
from intoCNF import booleanConstraint
f = booleanConstraint(weights,bias)
f
```

For very complex rules, we can use sympy to get more simple formulas:

```python
from sympy.logic import simplify_logic
sf = simplify_logic(f[0])
sf
```

Now recall that the 11th concept was represented by $isOdd$.

Nice! We have just learnt that our black box has learnt a very interesting concept: $\neg isOdd(x) \implies isEven(x)$!

That's all folks!

## References

Gori, M. (2017). Machine Learning: A constraint-based approach. Morgan Kaufmann.

Marra, G., Giannini, F., Diligenti, M., & Gori, M. (2019). Lyrics: a general interface layer to integrate ai and deep learning. arXiv preprint arXiv:1903.07534.

Ciravegna, G., Giannini, F., Gori, M., Maggini, M., & Melacci, S. Human-Driven FOL Explanations of Deep Learning. In 29th International Joint Conference on Artificial Intelligence (pp. 2234-2240).


# Assignment 5

Name 1:

Student id 1:

Email 1:

Name 2:

Student id 2:

Email 2:

**Instructions:** Read each question carefully.

Make sure you appropriately comment your code wherever required. Your final submission should contain the completed Notebook and the respective Python files for exercises 1 and 2. There is no need to submit the data files.

Upload the zipped folder in Teams. Make sure to click on "Turn-in" after your upload your submission, otherwise the assignment will not be considered as submitted. Only one from the group should make the submisssion.

---

## Exercise 1: Out Of Vocabulary Words (3.5 points)

As you saw in the lecture, the higher the number of unseen tokens in your language corpus, the higher the OOV rate. In this exercise, you will calculate the OOV rate for different languages for different vocabulary sizes. For each corpus, preprocess the data by lowercasing the text and applying tokenisation. Since there isn't any standard tokeniser that will work on all the languages, we recommend that you write your own function called `preprocess` in `exercise_1.py`.

### 1.1 Preprocess data (0.5 points)

Preprocess the data and partition it in a 70-30% train-test split. For this, write your own function `train_test_split_data` in `exercise_1.py`. You may modify the function signature and the code in the cell below appropriately.

```python
from importlib import reload
import os
import exercise_1
exercise_1 = reload(exercise_1)

# Walk through the data directory and read all the corpora
# For each corpus, read the text, preprocess it and create the train test split for each language

corpora = {} # To save the respective corpora

# TODO: Add a loop over each file
for filename in os.listdir('data/'):
    with open(os.path.join('data/', filename)) as f:
        text = f.read()
        pp = exercise_1.preprocess(text) #TODO: preprocess text
        train, test = exercise_1.train_test_split_data(pp, test_size=0.3) #TODO: split data
        #TODO: Add respective splits to the corpora dict
        lang = filename.split('.')[1]
        corpora[lang] = (train, test)
```

```python

```

### 1.2 Calculate OOV rates (1.5 points)
For every language, construct a vocabulary by taking the 15000 most frequent tokens in the training set. Compute the OOV rate for vocabulary sizes 1k, 2k, ..., 15k. Implement this in the function `get_oov_rates` in `exercise_1.py`.

```python
oov_rates = {}
for lang, (train, test) in corpora.items():
    oov_rates[lang] = exercise_1.get_oov_rates(train, test)
```

### 1.3 Plotting OOV rates (1 point)
* Using the loglog scale, plot the OOV rate against the vocabulary size for all the languages in a single plot. Make sure your legend identifies the languages appropriately and you label the axes.

* Describe your observations in 3-4 sentences.

```python
exercise_1.plot_oov_rates(oov_rates)
```

### 1.4 Handling OOV words (0.5 points)
* Before applying smoothing and backing-off models, we need to take care of the OOV words. Suggest 2 techniques to handle Out-Of-Vocabulary words your corpus.
* What are the advantages and disadvantages of each?

## Exercise 2: Smoothing (4 points)

### 2.1 Additive smoothing (1 point)

In the last assignments we largely ignored the issue of unseen words, i. s. words that are not in the train set/observed data but part of the test set. A very simple method to account for unseen words is [additive smoothing](https://en.wikipedia.org/wiki/Additive_smoothing). It assigns a small 'pseudo-count' to all unseen words AND to the words already in the language model, and then uses the updated counts to estimate the n-gram probabilities. The formula for unigram probabilities is:
\begin{equation}
p(w_i) = \frac{C(w_i) + \alpha}{N + \alpha |V|}
\end{equation}

Where
* $C(w_i)$ is the empirical count of the unigram $w_i$
* $N$ is the number of unigrams in the train set
* $|V|$ is the size of the vocabulary after smoothing
* $\alpha$ is the additive count.

If $\alpha = 1$ this is known as *Laplace* smoothing, if $0 < \alpha < 1$ *Lidstone* smoothing.

1. How would you estimate the bigram probability $p(w_i|w_{i-1})$ and the general case $p(w_i|w_{i-1}, ..., w_{n-i+1})$? Explain each part of the formula. (0.5 points)
2. Is it a good idea to set $\alpha$ to 1? What could be a more reasonable value, and why? (0.5 points)

### 2.2 Language model class (3 points)

Until now, you have implemented language models as a series of Python functions. We have provided to you a class skeleton in `lm.py` that should do all the tricks you need to estimate a language model. You will use the same corpora and train/test split as in Exercise 1.

1. Complete the implementation of the `LanguageModel` class. You may estimate the parameters of the language model as you like, but the method `perplexity` should perform the perplexity calculation (as in the below code block), and the method `lidstone_smoothing` should smooth the data. You may define new methods or change the signatures of existing ones, as long as you comment on your changes. Make sure that the relative frequencies and the conditional probabilities for each history sum up to 1. (1.5 points)

2. Choose $\alpha = 1$. Then, estimate $N = 1,2,3$ language models for the corpora from Exercise 1, and plot perplexity vs. $n$ for each of them. Do so by implementing the function `plot_pp` in `exercise_2.py` Do you observe any differences between the languages? Explain what you see in 3-4 sentences. (1 point)

```python

from importlib import reload
import lm
import exercise_2
lm = reload(lm)
exercise_2 = reload(exercise_2)

N = 3
PPs = {}
for lang, (train, test) in corpora.items():
    pp_list = []
    for i in range(1,N+1):
        LM = lm.LanguageModel(train, test, N=i, alpha=1)
        pp_list.append(LM.perplexity())
    PPs[lang] = pp_list

print(PPs)
exercise_2.plot_pp(PPs)
```

3. Now, find a good value for $\alpha$ for the *English* corpus. Do so by estimating $K=100$ trigram language models with $\alpha = 0.0, 0.01,...,0.99,1.0$, and plot trigram perplexity vs. increasing $\alpha$. You can write the code for the loop in the code cell below, the plotting code should be in `plot_pp_vs_alpha` in `exercise_2.py`. Does the $\alpha$ coincide with your estimate in 2.1.2? (0.5 points)

```python
import numpy as np

# lang = "corpus.en"
lang = "en"

N = 3
K = 100

PPs = []
alphas = np.arange(0.0,1.0,0.01)
train, test = corpora[lang]
# TODO: Loop
for alpha in alphas:
    LM = lm.LanguageModel(train, test, N=N, alpha=alpha)
    pp = LM.perplexity()
    PPs.append(pp)

exercise_2.plot_pp_vs_alpha(PPs, alphas)
```

## Exercise 3: Misc. (2.5 points)

## 3.1 Smoothed perplexity (1 point)

Assume you trained (MLE) an n-gram language model on datasets $D_\text{train}$ and $D_\text{test}$. You measure perplexities $p_{1,\text{train}}$ and $p_{1,\text{test}}$ respectively. You then smooth your n-gram language model and evaluate again on the two datasets, resulting in $p_{2,\text{train}}$ and $p_{2,\text{test}}$. Answer the following question with brief comments (e.g. _"X is always greater than Y because .."_). For a language model $p$, test perplexity can, for example, be computed as $2^{\frac{-1}{|D_\text{test}|} \sum_{w \in D_\text{test}}\log p(w|h)}$ and train perplexity as $2^{\frac{-1}{|D_\text{train}|} \sum_{w \in D_\text{train}}\log p(w|h)}$.

1. What is the relation of $p_{1,\text{train}}$ and $p_{1,\text{test}}$?
2. What is the relation of $p_{2,\text{train}}$ and $p_{2,\text{test}}$?
3. What is the relation of $p_{1,\text{train}}$ and $p_{2,\text{train}}$?
4. What is the relation of $p_{1,\text{test}}$ and $p_{2,\text{test}}$?
5. How does $n$ size affect the perplexities?

**Answers**

1. $p_{1,\text{train}}$ is less than $p_{1,\text{test}}$ because when training a language model (LM) on train data, the model has obviosly come across each word of vocabulary on the train set. Hence, computing perplexity on train data would always result in lower perplexity then test data as there might be some words in test set which the LM is seeing for the first time resulting in poorer perplexity.

2. $p_{2,\text{train}}$ and $p_{2,\text{test}}$

3. $p_{1,\text{train}}$ and $p_{2,\text{train}}$ would remain same as we do smoothing only on test set to cover up the occurances of unknown words. Hence, perplexity for the train set would remain same before and after smoothing.

4. $p_{1,\text{test}}$ will be greater than $p_{2,\text{test}}$ because after smoothing, some probability mass from the higher counts is assigned to the zero counts (i.e. the words which only occur in test set). This makes the distribution a little less discontinuous. Hence, smoothing causes improvement in perplexity and therefore becomes less than what it was before smoothing.

5. As $n$ (I assume we are talking about n-gram size, should have been specified explicitly) increases, perplexity decreases. A trigram LM would give better perplexity than a unigram LM. This is because computing probability of the	next word will turn out to be closely related to computing the probability of a	sequence of	words. So longer the sequence, more would be the conditional probablity $p(w|h)$

## 3.2 Infinite smoothing (0.5 points)

What distribution would you get if you applied additive or absolute discounting (choose one) smoothing infinitely? e.g. if $F_\text{smooth}$ is a function that smooths a language model (either additive or absolute discounting) and $\text{lm}^{(n+1)} = F_\text{smooth}(\text{lm}^{(n)})$. What will the language model $\lim_{n\rightarrow \infty} \text{lm}^{(n)}$ look similar to?

**Answers**

For $n \rightarrow \infty$, the LM distribution would become uniform.

## 3.3 Convex combination of LM models (1 point)

Consider the following quantity based on two independent language models $p_1$ and $p_2$.

$f_3(w|h) = \beta_1\cdot p_1(w|h) + \beta_2\cdot p_2(w|h)$ where $\beta_1 + \beta_2 = 1$ and $\beta_1 \ge 0, \beta_2 \ge 0$

- Is it still a language model (probability distribution given history $h$)? Show that all properties hold or find a counterexample for each: (1) non-negativity, (2) summation to 1 and (3) $\sigma$-additivity. See [Wikipedia - Probability Axioms](https://en.wikipedia.org/wiki/Probability_axioms). (0.5 points)
- What would be the possible gain of using the given function as a language model? (0.5 points)

**Answers**

1. *non-negativity*

Already given that $\beta_1 \ge 0, \beta_2 \ge 0$. Also, the probablity terms $p_1(w|h)$ and $p_2(w|h)$ would always be a non-negative quantity. So $f_3$ as a whole would be always non-negative.

2. *summation to 1*

Given two independent LMs $p_1$ and $p_2$.
$$\sum_{i} p_1(w_i|h_i) = 1$$
and
$$\sum_{i} p_2(w_i|h_i) = 1$$
Therefore,
$$\sum_{i} f_3(w_i|h_i) = \beta_1\cdot \sum_{i} p_1(w_i|h_i) + \beta_2\cdot \sum_{i} p_2(w_i|h_i)$$
$$\sum_{i} f_3(w_i|h_i) = \beta_1\cdot 1 + \beta_2\cdot 1$$
$$\sum_{i} f_3(w_i|h_i)) = 1$$

3. *$\sigma$-additivity*

One possible gain would be that the given quantity $f_3$ incorporates properties from both the languge models. There are slim chances that for a sequences both $p_1$ and $p_2$ give a poor prediction. Also, the distribution of each of these language models are being controlled by $\beta_1$ and $\beta_2$ which means that a favourable distribution would be given higher importance by controlling $\beta$ factor accordingly.

## Bonus (1 point)

Read about the following special language models techniques. The provided links shold only serve as a starting point.

#### 1. [Neural language models](https://en.wikipedia.org/wiki/Language_model#Neural_network) (0.5 points)

- Describe (~5 sentences or bullet points) the working, advantages and disadvantages of NLM.

#### 2. [Class-based language models](https://www.cs.cmu.edu/~roni/11761/PreviousYearsHandouts/classlm.pdf) (0.5 points)

- What is a class-based language model?
- What issues does it address?
- Can we utilize the output for something more than just language modelling (think about other NLP problems and classes you took)?