# Natural Language Processing Tutorial

Welcome to the second lab session of the [Mediterranean Machine Learning  summer school](https://www.m2lschool.org/home)!

[](https://colab.research.google.com/github/m2lschool/tutorials2021/blob/main/2_nlp/NLP_tutorial.ipynb)

In this tutorial, you will learn the fundamental components and main tasks in the Natural Language Processing (NLP) domain.

The tutorial is structured as follows:

* Imports and downloads (shared across sections)
* Part I: *Inner mechanisms of a vanilla Recurrent Neural Network (RNN)*
* Part II: *Character-level language modeling with Long-Short Term Memory (LSTM) networks*
* Part III: *Non-negative Matrix Factorization for topic modeling*
* Part IV: *From traditional (bag-of-words) to pretrained representations (BERT) for text classification*

All sections can be executed independently after running the initial import and download cells.

The sections marked as \[EXERCISE\] contain cells with missing code that you should complete.

Credits:
**[Federico Bianchi](https://federicobianchi.io)**, **[Debora Nozza](https://dnozza.github.io/)** and **[Francesco Visin](https://scholar.google.it/citations?user=kaAnZw0AAAAJ)**

### All Imports and Downloads

Here we are going to install and import everything we are going to need for this tutorial.

**Note**: *You can double-click the title of the collapsed cells (as the ones below) to expand them and read their content.*

```python
# @title Downloads and libraries installation
%%capture
!pip install git+https://github.com/deepmind/dm-haiku
!pip install optax
!pip install tensorflow-datasets
!pip install emoji
!pip install sentence-transformers
```

```python
# @title Imports
import emoji
from functools import partial
import re
import time
from tqdm import tqdm

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sentence_transformers import SentenceTransformer

import haiku as hk
import jax
import jax.numpy as jnp
from jax.nn.initializers import glorot_normal, normal, glorot_uniform, zeros
from jax import grad, jit, vmap, value_and_grad
from jax import random
from typing import Iterator, Mapping
from jax import lax
from jax.scipy.special import logsumexp
from jax import ops
import optax
import tensorflow.compat.v2 as tf
import tensorflow_datasets as tfds

from nltk.tokenize import RegexpTokenizer
```

# Part I. Vanilla RNN

**Goal**: Train a vanilla RNN to predict the next value of a temporal series given its previous values.

Specifically, we aim to predict

$$
sin (x +t \epsilon)
$$

from

$$
sin (x), sin (x + \epsilon), ..., sin (x + (t-1) \epsilon)
$$

In particular, we want the network to generate multiple predictions conditioned only on a few initial values. To do so, we will predict the next value of the function in a loop, conditioning on the value at the previous time-step (be it the initial, given, values or the ones predicted by the network at each previous step).

To learn the prediction model, we will use **teacher forcing**. This means that *during training* we will condition the model on values *coming from the ground truth* (i.e., the real sequence), rather than the output produced by the model at $t-1$. This makes training easier, because errors do not compound (a bad prediction at time $t$ does not influence the next prediction at time $t+1$).

At *inference time* (i.e., when we want to generate data from the model) we do not have access to the true sequence, so we must condition on the values *predicted by the model in previous time-steps*. This often leads to some compounding error, which generally makes it harder and harder to generate long sequences accurately.

To alleviate this problem we will also experiment with **warm starting**, which amounts to feeding the model with values coming from the ground truth (as done when teacher forcing) *only for a few initial steps*, and then feed it with its previous predictions instead. The rationale for this is to make training easier initially, to help the model learn the true function, and then feed it with its predictions to become robust to imperfect data.

## Training data

Let's store the sine wave over the interval $[0, 2\pi]$ as our training data. This will be our **ground truth**.

Note that differently from what is usually done, we don't have two separate training and test datasets here. Instead, we will train on random subsamples of the curve and then verify the ability of the network to generate the full curve when conditioned on a few initial values:

* Predict a few future values from a random starting point $x_s$;
* Generate the full trajectory, conditioned on a few initial points.

```python
t = np.arange(0, 2*np.pi, 0.1).reshape(-1, 1)
sin_t = np.sin(t)

plt.scatter(t, sin_t)
plt.xlabel('$t$')
plt.ylabel('$sin(t)$')
plt.show()
```

```python
# @title Hyperparameters
WARM_START = 10                   #@param {type:'integer'}
TEACHER_FORCING = False           #@param {type:'boolean'}
STEP_SIZE = 0.0001                #@param {type:'number'}
UNROLL_LENGTH = 30                #@param {type:'integer'}
HIDDEN_DIMENSION = 20             #@param {type:'integer'}
NUM_ITERATIONS = 10000            #@param {type:'integer'}
REPORTING_INTERVAL = 2000         #@param {type:'integer'}

```

As you have seen in the introductory lab, to use pseudo-random functions in JAX we need to instantiate a random number generator, and pass it explicitely to all the operations that work with random numbers (e.g. model initialization, dropout, etc, ...).

```python
rnd_key = random.PRNGKey(1)
```

## RNN cell \[EXERCISE\]

Implement a basic RNN cell using *jax.numpy (jnp)* functions

$$ h_t = f( Wx_t + Vh_{t-1}  + b) $$

Where,

* $x_t$ input at time $t$,
* $h_t$ hidden state at time $t$,
* $W$ input-to-hidden mapping (trainable),
* $V$ hidden-to-hidden mapping (trainable),
* $b$ bias (trainable),
* $f$ non-linearity chosen (we use *tanh*)

Start by implementing a function to return the initial parameters:

```python
def initialize_parameters(rnd_key):
    """
    Initialize and return the Vanilla RNN parameters.

    Args:
        rnd_key: random key

    Returns:
        A dict of weights with keys `V`, `W`, `bias`, `decoder`, `decoder_bias`.
    """

    input_dimension = 1   # the input is a scalar
    output_dimension = 1  # the output is a scalar

    ######################
    #   YOUR CODE HERE   #
    ######################
    # Initialize the Vanilla RNN parameters
    # using a glorot uniform distribution for W,
    # V, decoder and zero values for bias and
    # decoder_bias. Pay attention to the correct
    # shapes of all parameters.

    keys = random.split(rnd_key, 5)
    params = {

    }

    assert params is not None, 'Params should be initialized'

    return params
```

and then implement the RNN core itself:

```python
def RNN_cell(params, x_t, current_state):
    """
    This function will be called in a loop, when RNN core is connected to
    inputs and previous states.

    Args:
        params: neural network paramters
        x_t: jax DeviceArray containing the current input `x_{t}`
        prev_state: jax DeviceArray containing the previous state `h_{t-1}`

    Returns:
        A pair of RNN embedding and state.
    """

    ######################
    #   YOUR CODE HERE   #
    ######################
    # Compute the output of the RNN Cell by implementing the equation above.
    output = None

    assert output is not None, 'The RNN cell must compute the output'

    # The RNN cell returns pairs of (o_t, h_t), respectively the output and the
    # state at time $t$. For vanilla RNN these are the same.
    return output, output

def get_initial_rnn_state(rnd_key):
    return zeros(rnd_key, (HIDDEN_DIMENSION,))
```

Now let's check that the output of the RNN is as expected.

```python
rnd_key, k1, k2 = random.split(rnd_key, 3)
params = initialize_parameters(k1)
single_input = sin_t[0]
initial_state = get_initial_rnn_state(k2)

# RNN_cell returns two vectors, the output and the state. When the input is a
# single element (i.e., not a sequence) each should have shape HIDDEN_DIMENSION.
assert len(RNN_cell(params, single_input, initial_state)[0]) == HIDDEN_DIMENSION
assert len(RNN_cell(params, single_input, initial_state)[1]) == HIDDEN_DIMENSION
```

## Next element prediction [EXERCISE]

The `predict` method defined below supports both *teacher forcing* and *warmup*. For educational purposes, we unrolled the RNN loop manually so it is easier to understand and debug the code. You need to implement the missing bits, applying the `RNN_cell` and returning a sequence of scalar predictions.

The `sin(x)` input sequence is composed of scalars, which is what the `RNN_cell` expects so you don't need to preprocess them. The predictions of the `RNN_cell` have shape `(HIDDEN_DIMENSION,),` so you need to apply the $\textbf{decoder}$ to recover the sequence of scalars we want to predict.

When you are done debugging, uncomment the **`@jit` decorator** to ensure the code is optimized and runs faster.

```python
def encoder_decoder(params, encoded_input, current_state):
    rnn_embedding, new_state = RNN_cell(params, encoded_input, current_state)
    return jnp.dot(rnn_embedding, params['decoder']) + (
      params['decoder_bias'], new_state)

#@jit  # Uncomment for full speed!
def predict(params, initial_rnn_state, inputs, training=True):
    """Predict the next value of a sequence.

    Here we compute *one-step predictions*, i.e., we predict `sin_{t+1}` from
    `sin_{t}`. Note however that depending on the values of the TEACHER_FORCING
    and WARM_START parameters, sometimes we feed the model with the actual
    `sin_{t}` coming from the ground truth (i.e., `inputs`) and some other times
    with the last prediction of the model (which might or might not be
    accurate). This means that in some cases not all the values of `inputs` will
    be used for training.

    This code a simplified implementation of `hk.static_unroll` for educational
    purposes, do not use it in actual code. Statically unrolling the network has
    some caveats, e.g., it won't behave as expected if the length of the input
    sequence is variable. It is usually preferable to use `hk.dynamic_unroll`.

    Args:
        params: the parameters of the RNN
        initial_rnn_state: the initial state of the RNN
        inputs: the data passed in input to the RNN
        training: sets teacher forcing to True or False

    Returns:
        The next value predictions.
    """
    if training:
      teacher_forcing = TEACHER_FORCING
    else:
      teacher_forcing = False

    predictions = []
    current_state = initial_rnn_state

    # Unroll the rnn loop by hand.
    for t in range(len(inputs)):

        # When teacher forcing, the input is always the actual previous value
        # (from the ground truth). Otherwise, we will feed the model with the
        # actual previous values during the warm start period only, and then
        # input the previous predictions instead.
        if teacher_forcing or t <= WARM_START:
            input_ = inputs[t]
        else:
            input_ = prediction

        ######################
        #   YOUR CODE HERE   #
        ######################
        # Connect the RNN: we apply our cell to (input, state) pairs, and we
        # expect it to return (output, next_state) pairs; note how the current
        # state is updated with the next_state.
        rnn_embedding, next_state = None, None

        assert rnn_embedding is not None, 'The RNN must return an output'

        current_state = next_state

        ######################
        #   YOUR CODE HERE   #
        ######################
        # Create a linear mapping from RNN output to the target (scalar)
        # using the decoder and decoder bias parameters
        prediction = None

        assert prediction is not None, ('Connect the rnn embeddings and the '
                                        'decoder to get prediction')

        predictions.append(prediction)

    return jnp.stack(predictions)
```

### Shape check

Let's check that our RNN predictions returns the desired shape

```python
rnd_key, k1, k2 = random.split(rnd_key, 3)
params = initialize_parameters(k1)
initial_rnn_state = get_initial_rnn_state(k2)
predictions = predict(params, initial_rnn_state, sin_t[:UNROLL_LENGTH])
assert len(predictions) == UNROLL_LENGTH
```

## Loss and weights update [EXERCISE]

We update the parameters of the network using the **ADAM** optimizer. In this section you need to complete the `loss_fn` that must compute the loss as the mean squared error of the prediction and the targets. As before, once you are done debugging uncomment the `@jit` decorator to ensure the code runs at full speed.

The `optimize` function uses this `loss_fn` function to compute the loss and the gradient of the loss w.r.t. the parameters of the network, and takes one step of gradient descent to reduce such loss. This is effectively one step of training.

```python
# @jit
def loss_fn(params, rnn_state, inputs, targets):
    """
    Loss function is a simple mean squared error.

    Args:
        params: the parameters of the neural network
        rnn_state: the state of the rnn
        inputs: a sequence of inputs
        targets: a sequence of expected outputs

    Returns:
        The mean squared error.
    """

    ######################
    #   YOUR CODE HERE   #
    ######################
    # Obtain the prediction given the current state and input data
    # and compute the mean squared error with the targets.
    predictions = None
    mse = None

    assert predictions is not None, 'Get the RNN predictions'
    assert mse is not None, ('Compute the mean squared error between the '
                             'predictions and the target')
    assert predictions.shape == targets.shape, (
        f'The predictions should have the same shape as the input and target, '
        f'but are respectively {predictions.shape}, {inputs.shape}, '
        f'{targets.shape}')

    return mse

@jit
def optimize(params, opt_state, rnn_state, inputs, targets):
    """
    Updates the parameters and returns the loss values.

    Args:
        params: the parameters of the neural network
        opt_state: the state of the optimizer
        rnn_state: the state of the rnn
        inputs: a sequence of inputs
        targets: a sequence of expected outputs

    Returns:
        The loss, the new optimizer state and the updated network parameters.
    """
    loss_value, grads = jax.value_and_grad(loss_fn)(params, rnn_state, inputs,
                                                    targets)
    updates, opt_state = optimizer(grads, opt_state)
    params = optax.apply_updates(params, updates)
    return loss_value, opt_state, params
```

## Training [EXERCISE]

In this section we will finally use all the functions defined above to train the network.

The following code prints an example of input and output data for an unroll length of 4. Note how the target at time $t$ is the input at time $t+1$. This is because we train the network to predict the next value in the sequence.

```python
training_input = jnp.array(sin_t[:4])
training_target = jnp.array(sin_t[1:5])

print('Example of training sample')
print('input-> ', training_input.ravel())
print('target-> ', training_target.ravel())
```

At this point, all the functions and parameters have been defined. We can create the training loop, run it and visualize the output plot predictions

```python
######################
#   YOUR CODE HERE   #
######################
# Initialize parameters and get the initial state of the RNN
params = None
initial_rnn_state = None

assert params is not None, 'Initialize params of the RNN.'
assert initial_rnn_state is not None, 'Get the initial state of the RNN.'

opt_init, optimizer = optax.adam(STEP_SIZE)
opt_state = opt_init(params)

losses = []
for i in range(NUM_ITERATIONS+1):
    # Training.
    start = np.random.choice(range(len(sin_t) - UNROLL_LENGTH))
    training_input = jnp.array(sin_t[start: start+UNROLL_LENGTH])
    training_target = jnp.array(sin_t[start+1: start+UNROLL_LENGTH+1])
    # Note: `optimize` calls prediction, computes the loss and updates the
    # parameters of the network and the state of the optimizer.
    loss_value, opt_state, params = optimize(params, opt_state,
                                             initial_rnn_state, training_input,
                                             training_target)
    losses.append(loss_value.tolist())

    # Full sequence generation and plotting.
    if i % REPORTING_INTERVAL == 0:

        # Generate the full sequence (from the first to the last element).
        y_gen = predict(params, initial_rnn_state, sin_t[:-1], training=False)
        sampling_loss = loss_fn(params, initial_rnn_state, y_gen, sin_t[1:])
        plt.figure(figsize=(10, 5))

        plt.title(f'Training Loss {loss_value.tolist():.3f}, sampling loss '
                  f'{sampling_loss.tolist():.3f}, iteration {i}')

        plt.plot(t[1:].ravel(), sin_t[1:].ravel(), c='blue',
                 label='Ground truth',
                 linestyle=':', lw=6)

        if TEACHER_FORCING:
            plt.plot(t[start: start+UNROLL_LENGTH].ravel(),
                   training_input.ravel(),
                   alpha=0.7, lw=5, c='green', label='Training input')
        else:
            plt.plot(t[start: start+WARM_START].ravel(),
                   training_input[:WARM_START].ravel(),
                   alpha=0.7, lw=5, c='green', label='Training input')
        plt.plot(t[start+1: start+UNROLL_LENGTH+1].ravel(),
                 training_target.ravel(),
                 alpha=0.7, lw=2, c='black', label='Training target')

        plt.plot(t[start: start+UNROLL_LENGTH].ravel(),
                 predict(params, initial_rnn_state, training_input),
                 alpha=0.7, lw=3, c='gold', label='Network training prediction')

        plt.plot(t[1:].ravel(),
                 y_gen,
                 alpha=0.7, c='r', label='Network full generation')

        plt.legend()
        plt.show()

plt.figure()
plt.title('Training Loss')
_ = plt.plot(losses)
```

Now try changing the hyperparameters and see how they affect the training (e.g., how accurately does the network learn to model the curve, how fast does it converge, etc, ...)

## **What is worth trying/understanding here?**
- Difference between teacher forcing and learning on own samples:
 - What are the pros and cons of teacher forcing?
 - Why is the model struggling to learn in one of the setups?
 - What is it that we actually care about when we model sequences? How can this be negatively affected by teacher forcing?
- Effect of warm starting:
  - How does warm starting affect our training when teacher forcing is disabled? Why?
  - How does warm starting affect our sampling when teacher forcing is enabled? Why?
- What happens if the structure of interest is much longer than the unroll length?

# Part II: Character-level language modeling

**Goal**: Train a character level LSTM on text data - specifically Shakespeare's sonnets.

**What is an LSTM**: An LSTM is an advanced variant of the RNN. As opposed to the vanilla RNN the output and the state are separate, which allows for much more flexibility in deciding what to store. The LSTM is also characterized by a more advanced mechanism to determine which parts of the input to store into the memory and what to potentially forget. *For an in-depth analysis of the differences between various kinds of RNNs, we recommend you to read [this excellent guide](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Chris Olah.*

**Haiku**: In order to develop our network we will use [Haiku](https://github.com/deepmind/dm-haiku), which provides many common building blocks, including LSTMs!

**This tutorial**: Throughout this section you will build the network, train an LSTM on a dataset of Shakespeare's sonnets and finally you will be able to see how the quality of the generated text improves over time as the network trains.

```python
# @title Hyper-parameters
BATCH_SIZE = 32           #@param {type:'integer'}
SEQUENCE_LENGTH = 128     #@param {type:'integer'}
HIDDEN_SIZE = 256         #@param {type:'integer'}
SAMPLE_LENGTH = 128       #@param {type:'integer'}
LEARNING_RATE = 1e-3      #@param {type:'number'}
TRAINING_STEPS = 6500   #@param {type:'integer'}
SAMPLING_INTERVAL = 200   #@param {type:'integer'}
SEED = 43                 #@param {type:'integer'}
NUM_CHARS = 128           #@param {type:'integer'}
```

```python
#@title Dataset loader with encoding and decoding functions { form-width: "170px" }
Batch = Mapping[str, np.ndarray]

def load(split: tfds.Split, *, batch_size: int,sequence_length: int):
    '''Creates the Tiny Shakespeare dataset as a character modelling task.'''

    def preprocess_fn(x: Mapping[str, tf.Tensor]) -> Mapping[str, tf.Tensor]:
        x = x['text']
        x = tf.strings.unicode_split(x, 'UTF-8')
        x = tf.squeeze(tf.io.decode_raw(x, tf.uint8), axis=-1)
        x = tf.cast(x, tf.int32)
        return {'input': x[:-1], 'target': x[1:]}

    ds = tfds.load(name='tiny_shakespeare', split=split)
    ds = ds.map(preprocess_fn)
    ds = ds.unbatch()
    ds = ds.batch(sequence_length, drop_remainder=True)
    ds = ds.shuffle(100)
    ds = ds.repeat()
    ds = ds.batch(batch_size)
    ds = ds.map(lambda b: tf.nest.map_structure(tf.transpose, b))  # Time major.
    return tfds.as_numpy(ds)

def decode(x: np.ndarray) -> str:
    return ''.join([chr(x) for x in x])

def encode(x: str) -> np.ndarray:
    return np.array([ord(s) for s in x])
```

## Build the network with Haiku [EXERCISE]

In this section you will need to use Haiku to build the network. To do so, you will make use of the `hk.DeepRNN` ([doc](https://dm-haiku.readthedocs.io/en/latest/api.html#haiku.DeepRNN)) module to wrap together several layers into a single RNN core.

Implement the following architecture:

- Convert the input to a one-hot representation with `NUM_CHARS` elements (hint: use a lambda function `lambda x: jax.nn.one_hot(...)`) [(doc)](https://github.com/google/jax/blob/6c8fc1b031275c85b02cb819c6caa5afa002fa1d/jax/_src/nn/functions.py#L261)
- LSTM of `HIDDEN_SIZE` size ([doc](https://dm-haiku.readthedocs.io/en/latest/api.html?highlight=lstm#lstm))
- ReLU (non-linearity) ([doc](https://jax.readthedocs.io/en/latest/_autosummary/jax.nn.relu.html))
- LSTM of `HIDDEN_SIZE` size ([doc](https://dm-haiku.readthedocs.io/en/latest/api.html?highlight=lstm#lstm))
- An two-layer MLP composed of a layer of `HIDDEN_SIZE` size with ReLU activation, followed by a linear (no activation) layer mapping to the final problem size (number of characters: `NUM_CHARS`) ([docstring](https://github.com/deepmind/dm-haiku/blob/300e6a40be31e35940f0725ae7ed3457b737a5a3/haiku/_src/nets/mlp.py#L38))

```python
def make_network() -> hk.RNNCore:
    """Define the RNN core."""
    ######################
    #   YOUR CODE HERE   #
    ######################
    # Design the LSTM with Haiku
    # following the architecture
    # defined in the text above.

    rnn_core = hk.DeepRNN([
        # add code
    ])

    assert rnn_core is not None, 'LSTM model should be initialized'

    return rnn_core
```

## Loss and Update [EXERCISE]
As before, we update the parameters of the network using the **ADAM** optimizer.

In this section you need to complete the `loss_fn`, which should compute the loss as the categorical cross-entropy of the prediction and the targets. The `optimize` function (which you don't need to modify) uses this `loss_fn` function to compute the loss and the gradient of the loss w.r.t. the parameters of the network, and takes one step of gradient descent to reduce such loss. This is effectively one step of training.

As before, once you are done debugging uncomment the `@jit` decorator **to ensure the code runs at full speed**.

### Categorical Cross-entropy Recap
The categorial cross-entropy loss is computed as

$$\mathcal{L}_{\text{xent}} = - \mathbf{y} \cdot \log(\mathbf{\hat{y}})$$

where $\mathbf{y}$ is the true class probability and $\mathbf{\hat{y}}$ is the predicted class probability.

**Example**

Let's take as an example a classification problem with 3 classes. Assume our **target vector** for the current sample to be $[0, 1, 0]$ (i.e., the second class is the correct one, according to the ground truth). If our network's **predicted class probability** is $[0.2, 0.7, 0.1]$, the categorical cross-entropy loss for this prediction will be

$$
\begin{align}
  \mathcal{L}_{\text{xent}} =& - \bigg( 0 \cdot \log(0.2) + 1 \cdot \log(0.7) + 0 \cdot \log(0.1) \bigg)\\
  =& - \log(0.7) \\
  =& \; 0.35
\end{align}
$$

In this case, the network was fairly confident that the input belonged to the second class, which was correct, so the error is not very large.

Let's see what would have happened if the network was mistakenly confident that the true class was another one instead. Suppose the predicted vector was $[0.8, 0.05, 0.15]$, i.e., that the network was assigning a very low probability to the true class. The loss in this case would have been $-\log(0.05) = 3$, which is much higher than before.

```python
def loss_fn(batch):
    """
    Compute the categorical cross-entropy loss.

    Unroll the network over a sequence of batched inputs & targets, and
    compute the categorical cross-entropy loss.

    Args:
        logits: the logits from the network
        batch: a batch of input and target data

    Returns:
        The categorical cross-entropy loss.
    """
    sequence_length, batch_size = batch['input'].shape

    ######################
    #   YOUR CODE HERE   #
    ######################
    # create the network and unroll it (dynamic unrolling)
    # Obtain the prediction by computing the log_softmax on logits
    # Convert the log probabilities to one hot labels
    # and compute the categorical cross-entropy loss
    loss = None

    assert loss is not None, 'Loss should be computed'

    return loss

# @jit
def optimize(params, opt_state, batch):
    """
    Applies an update to the parameters.

    Args:
        params: the parameters of the neural network
        opt_state: the state of the optimizer
        batch:  batch of input and target data

    Returns:
        The new optimizer state and the updated network parameters.
    """
    ######################
    #   YOUR CODE HERE   #
    ######################
    # Similar to the RNN update function
    # - initialize the Adam optimizer
    # - apply grad function to loss
    # - compute params updates
    # - apply updates on params
    new_params = None
    new_opt_state = None

    # Note that since JAX is stateless (the state is passed explicitly to each
    # operation) here we create another optimizer function for simplicity,
    # instead of passing the one we create when we first initialize the state
    # of the optimizer.

    assert new_opt_state is not None, 'Compute the updated optimizer\'s state.'
    assert new_params is not None, 'Compute the updated parameters.'

    return new_params, new_opt_state
```

## Sampling function

This function draws samples from the model, given an initial context.

```python
def sample_fn(rng_key: jnp.ndarray, context: jnp.ndarray, sample_length: int):
    """
    Draws samples from the model, given an initial context.

    Note: this function is impure; we will hk.transform() it (and jit it) in the
    training loop.

    Args:
        rng_key: random key
        context: initial context given to the model
        sample_length: length of the sample

    Returns:
        the generated tokens
    """
    assert context.ndim == 1  # Sequence only, no batch

    def body_fn(t, values):
        tokens, state, rng_key = values
        token = tokens[t]
        next_logits, next_state = rnn_core(token, state)
        rng_key, k1 = jax.random.split(rng_key)
        next_token = jax.random.categorical(k1, next_logits, axis=-1)
        new_tokens = ops.index_update(tokens, ops.index[t + 1], next_token)
        return new_tokens, next_state, rng_key

    # Unroll over context (initial prompt).
    rnn_core =  make_network()
    initial_state = rnn_core.initial_state(None)  # no batch here!
    logits, state = hk.dynamic_unroll(rnn_core, context, initial_state)

    # Sample the first continuation token and initialize the output array.
    rng_key, k1 = jax.random.split(rng_key)
    first_token = jax.random.categorical(k1, logits[-1])
    tokens = np.zeros(sample_length, dtype=np.int32)
    tokens = ops.index_update(tokens, ops.index[0], first_token)

    # Sample the other tokens in a loop.
    initial_values = tokens, state, rng_key
    new_tokens, _, _ = lax.fori_loop(0, sample_length, body_fn, initial_values)

    return new_tokens
```

## Train and generate language
Finally, this training loop puts everything together. The network is trained for a fixed number of `TRAINING_STEPS` and is evaluated at fixed intervals. During the evaluation, given a *context* sentence provided as input, the network tries to generate some text to complete that prompt.

As you can see, for the first few iterations the model generates nonsensical sentences, but after some training you can observe that the model is actually learning to reproduce text that resembles Shakespeare's sonnets.

**Important:** make sure you uncommented the `@jit` decorators in the previous
code to enjoy accelerated training. Training should take less than 7 minutes.

```python
%%time
# Note that since we use Haiku, we use their PRNG instead of the one from JAX.
rng = hk.PRNGSequence(SEED)

train_data = load(tfds.Split.TRAIN, batch_size=BATCH_SIZE,
                  sequence_length=SEQUENCE_LENGTH)
train_data = iter(train_data)

# Out network doesn't have a nondifferentiable state (the inner states of the
# LSTMs is differentiable), so we don't need to use `hk.transform_with_state`
# and can use the typical `hk.transform` instead. Further, our network does not
# have stochastic operations, so we can wrap it in a `without_apply_rng` call.
# Note that here we transform `loss_fn` as a way to initialize the network
# parameters a few lines below.
transformed_loss = hk.without_apply_rng(hk.transform(loss_fn))
_, sample = hk.without_apply_rng(hk.transform(sample_fn))
sample = jax.jit(sample, static_argnums=[3])  # sample_length is static

# Setup optimizer.
params = transformed_loss.init(next(rng), next(train_data))
# Similar to what we do for the loss, here we create the optimizer just to get
# the initial optimizer state.
optimizer = optax.adam(LEARNING_RATE)
opt_state = optimizer.init(params)

BOLD = '\033[1m'
END =  '\033[0m'
SEP = '\n------------------------------\n'
for step in range(TRAINING_STEPS):
    train_batch = next(train_data)  # includes input and target

    # Run one step of training and update the parameters.
    params, opt_state = optimize(params, opt_state, train_batch)

    if step % SAMPLING_INTERVAL == 0:
        # Use the input text of the batch as context.
        context = train_batch['input'][:, 0]  # drop the batch
        rng_key = next(rng)
        # Sample generated text given context.
        samples = sample(params, rng_key, context, SAMPLE_LENGTH)
        # Decode context and samples to actual sentences.
        prompt = decode(context)
        continuation = decode(samples)

        print(f'{BOLD}>>> Prompt (from the dataset):{SEP}{END}\n{prompt}\n')
        print(f'{BOLD}Continuation (from the network):{END}\n{continuation}\n')
```

## **What is worth trying/understanding here?**
- After how many iterations the model is able to learn a sentence which makes sense?
- Why the first generated sentences do not make sense and what pattern can you see?
- How is the sample length parameters affecting the results?

Answers:

- same words repeated

# Part III: Topic modeling with Non-negative Matrix Factorization

**Goal**: Extract topics from a set of documents exploiting *matrix factorization*.

**What is topic modeling**: *Topic modeling* is a Natural Language Processing task whose aim is to discover the abstract *topics* that occur in a set of documents. Intuitively the idea is that, depending on the topic discussed in a document or a set of documents, specific words will appear more or less frequently. For instance, 'dog' and 'bone' will appear more often in documents about dogs, while 'cat' and 'meow' will appear in documents about cats. Common words, like 'the' and 'is', will tend to appear approximately uniformely in all kinds of documents so are not very informative to tell the documents apart. We can exploit this to cluster the documents in groups, according to their topic.

**Credits:** this code is an adaptation of [this fast.ai tutorial](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb).

## Load dataset

In this section, we will use the well-known benchmark dataset **20newsgroups** to model the main topics discussed in a group of conversations. Newsgroups are discussion groups on Usenet, which was popular in the 80s and 90s before the web really took off. This dataset includes 18,000 newsgroups posts on 20 different topics.

```python
categories = ['rec.motorcycles', 'sci.med', 'comp.graphics', 'sci.space']
remove = ('headers', 'footers', 'quotes')
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, remove=remove)
```

```python
# Print three random documents.
print('\n\n-----\n'.join(newsgroups_train.data[:3]))
```

```python
# Print the category of the previous documents.
np.array(newsgroups_train.target_names)[newsgroups_train.target[:3]]
```

## Document discrete representations

In order to represent the documents as a **term-document matrix**, we need to transform unstructured data (text) to structured data.

This can be performed by extracting count and/or frequencies of words in each documents.

- The **Bag of Words (BoW)** model is the simplest way to represent text in numerical form. As the name suggests (nomen omen!), given a vocabulary of words we can represent any sentence as a *bag of words vector* (i.e., a vector of numbers) using the *counts*, i.e. we count how many times each word in the dictionary appears in the documents.

- The **TF-IDF** (Term Frequency - Inverse Document Frequency) representation model is similar, but accounts for both the frequencies of words and their importance. For example, if a words appears in all the documents, it is not very important, as it does not allow us to differentiate documents.

We will use **TF-IDF** representations, which we will extract using the [sklearn](https://scikit-learn.org/) library.

```python
# Initialize the TF-IDF representation model.
tf_idf = TfidfVectorizer(stop_words='english', min_df=10)
# Apply TF-IDF representation to the documents.
X_train = tf_idf.fit_transform(newsgroups_train['data'])
X_train = X_train.todense()
X_train = jnp.array(X_train)
```

```python
X_train.shape
```

```python
# Each column of the representation matrix corresponds to a word.
vocab = np.array(tf_idf.get_feature_names())
print(vocab)
```

## Non-Negative Matrix Factorization [EXERCISE]

**Non-Negative Matrix Factorization** is a statistical method to reduce the dimension of the input corpora. It uses the [factor analysis](https://en.wikipedia.org/wiki/Factor_analysis) method to provide comparatively less weight to the words with least coherence.

To reason on a practical example, assume we have an input matrix $V$ of shape $m \times n$. This method factorizes $V$ into two matrices $W$ and $H$, such that the dimension of $W$ is $m \times k$ and that of $H$ is $k \times n$. In our case, $V$ represent the *term-document matrix* (the frequency of the terms that occur in the collection of documents), each row of the matrix $H$ is a word embedding and each column of the matrix $W$ represent the weight of each word in each of the sentences (i.e., the semantic relation of words with each sentence).

The assumption is that all the entries of $W$ and $H$ are positive, since all the entries of $V$ are positive.

```python
def initialize_parameters(rnd_key, X_train, k):
    """
    This function will initialize and returns the Vanilla RNN parameters.

    Args:
        rnd_key: random key
        X_train: term-document matrix
        k: hidden dimension

    Returns:
        A list of parameters [W, V, bias, decoder, decoder_bias].
    """
    ######################
    #   YOUR CODE HERE   #
    ######################
    # Initialize the NNMF parameters (W and H) using normal distribution
    # (mean zero and unit standard deviation) and correct shapes
    m, n = X_train.shape
    k1, k2 = random.split(rnd_key)
    params = None

    assert params is not None, 'Params should be initialized'

    return params

```

## Frobenius Norm

A way to perform NMF is by using the Frobenius norm, which is defined by the square root of the sum of the absolute squares of its elements.

$$ ||A||_F = \sqrt{\sum_{i=1}^m \sum_{j=1}^n | a_{i,j}|^2}$$

We want to minimize the Frobenious norm of

$$ V - WH $$

however, we also want to penalize the matrix $W$ and $H$ when they are negative. Thus, we will add a specific penalty function.

### Penalty [EXERCISE]

```python
def compute_penalty(matrix):
    """
    Return the average of the square of the negative values in the matrix.

    Args:
        matrix: the matrix on which we want to compute the penality

    Returns:
        The penalty

    """
    # Equivalent to `(matrix[matrix < 0]**2).mean()`
    return jnp.power(jnp.clip(matrix, a_max=0.), 2).mean()
```

Let's see this function in action: the "more negative" the matrix is, the higher the penalty will be

```python
A = jnp.array([[0, 1, 4], [5, 6, 7]]) # positive
B = jnp.array([[0, 1, -4], [5, 6, 7]]) # some negative elments
C = jnp.array([[0, 1, 4], [-5, -6, -7]]) # more negative elements

print("A", compute_penalty(A))
print("B", compute_penalty(B))
print("C", compute_penalty(C))
```

We can now use this into our general loss function

```python
def get_penalty():
    """
    We combine the penality for both matrices
    """
    return (compute_penalty(params['W']).mean() +
            compute_penalty(params['H']).mean())

def loss_fn(params, V):
    """
    Compute the Frobenius norm factorization loss

    The Frobenius norm of the matrix V and its matrix factorization (W*H),
    plus the penalty scaled by the regularization parameter.

    Args:
        params: params to optimize (W and H)
        V: matrix
    Returns:
        The Frobenius norm factorization loss.
    """
    ######################
    #   YOUR CODE HERE   #
    ######################
    # Compute the loss and sum the regularization.
    # Combine the frobenious norm with the penality (weight the penality by the
    # regularization parameter)
    loss = None
    regularization_param=1e6

    assert loss is not None, 'Loss should be computed'

    return loss
```

## Parameters update function
As in the previous sections, the `optimize` function (which you don't need to modify) uses the `loss_fn` function to compute the loss and the gradient of the loss w.r.t. the parameters of the network, and takes one step of gradient descent to reduce such loss. This is effectively one step of training.

```python
@jit
def optimize(params, opt_state, x):
    loss_value, grads = jax.value_and_grad(loss_fn)(params, x)
    updates, opt_state = opt_update(grads, opt_state)
    params = optax.apply_updates(params, updates)
    return loss_value, opt_state, params
```

## Train

```python
rnd_key = random.PRNGKey(1)
rnd_key, k1, k2 = random.split(rnd_key, 3)
params = initialize_parameters(k1, X_train, 5)
opt_init, opt_update = optax.adam(0.01)
opt_state = opt_init(params)
for _ in range(0, 200):
    loss_value, opt_state, params = optimize(params, opt_state, X_train)
```

## Visualize output topics

```python
# The number of most-common words that we want to visualize.
num_top_words = 8  #@param {type:"integer"}

def show_topics(H_matrix):
    """
    Shows the topic coming from the H matrix.
    To be used after we have trained our NMF model.

    Args:
        H_matrix: the matrix that contains the topics

    Returns:
        A list of lists with the topic words for each topic
    """
    top_words = lambda t: [vocab[i] for i in np.argsort(t)[:-num_top_words-1:-1]]
    topic_words = ([top_words(t) for t in H_matrix])
    return [' '.join(t) for t in topic_words]

show_topics(params['H'])
```

# Part IV: Text classification with neural networks

**Goal**: Perform text classification exploiting traditional and advanced representation models and assess their impact on classification performance.

**What is text classification**: *Text classification* is the most basic task of Natural Language Processing. The goal is to classify textual data as belonging to one (yes/no) or more categories.

In this section, we take the specific task of **Sentiment Analysis** applied to reviews. Sentiment analysis, also called *opinion mining*, is the field of study that analyzes people’s opinions, sentiments, evaluations, appraisals, attitudes, and emotions towards entities such as products, services, organizations, individuals, issues, events, topics, and their attributes. Commonly, Sentiment Analysis is performed as a classification problem where a text should be classified as being *positive* or *negative*.

To address this task, we will investigate various representation models with a simple neural network. The goal of this section is to show the impact of the different representations on the classification accuracy.

The code has been adapted from [[link](https://github.com/google/jax/blob/master/docs/notebooks/neural_network_with_tfds_data.ipynb)]

## Load dataset

Here we load the *IMDB reviews* dataset for binary sentiment classification. The training set is composed of 25,000 highly movie reviews.

```python
# Load and preprocess the data.
train_data, test_data = tfds.load(name='imdb_reviews', split=['train', 'test'],
                                  batch_size=-1, as_supervised=True)
train_examples, train_labels = tfds.as_numpy(train_data)
test_examples, test_labels = tfds.as_numpy(test_data)
train_examples = list(map(lambda x : x.decode('utf-8') , train_examples))
test_examples = list(map(lambda x : x.decode('utf-8') , test_examples))

def batch(X, y, batch_size=32):
    """
    This function will allow to select batches of training data.

    Args:
        X: train raw data
        y: train labels
        batch_size: size of the batch
    """
    total_len = len(X)
    for ndx in range(0, total_len, batch_size):
        yield (X[ndx:min(ndx + batch_size, total_len)],
               y[ndx:min(ndx + batch_size, total_len)])
```

## Initialize Variables

```python
PARAM_SCALE = 0.1 #@param {type:'number'}
STEP_SIZE = 0.01 #@param {type:'number'}
NUM_EPOCHS = 10 #@param {type:'integer'}
BATCH_SIZE = 128 #@param {type:'integer'}
N_TARGETS = 2 #@param {type:'integer'}
```

## Traditional Representation: TF-IDF [EXERCISE]

Similar to what we've previously seen, we use TF-IDF representation for the discrete representation of textual data.

```python
def get_discrete_data(train_examples,test_examples):

    ######################
    #   YOUR CODE HERE   #
    ######################
    # Transform test and train data with TF-IDF (using Sklearn TfidfVectorizer,
    # from sklearn) with min document frequency set to 80
    # transform arrays to dense arrays
    X_train = None
    X_test = None

    # we make these matrices dense
    X_train = X_train.todense()
    X_test = X_test.todense()

    assert X_train is not None, 'Data should be transformed with TF-IDF vectorizer'

    return X_train, X_test
```

## Initialize parameters [EXERCISE]

```python
def random_layer_params(m, n, key, scale=1e-2):
    w_key, b_key = random.split(key)
    return (scale * random.normal(w_key, (n, m)),
            scale * random.normal(b_key, (n,)))

def init_network_params(sizes, rnd_key):
    """
    Initialize neural network parameters.

    Args:
        sizes: layers' size
        key: random key

    Returns:
        The initialized parameters.
    """
    keys = random.split(rnd_key, len(sizes))
    return [random_layer_params(m, n, k)
            for m, n, k in zip(sizes[:-1], sizes[1:], keys)]
```

```python
def predict(params, sentence):
    """
    Predict class given a sentence.

    Args:
        params: the parameters of the neural network
        sentence: input data

    Returns:
        The predictions of the network
    """
    activations = sentence

    ######################
    #   YOUR CODE HERE   #
    ######################
    # For each layer in the neural network
    # compute the basic MLP (with weights and bias)
    # using relu as activation function.
    # Pay attention to the last layer

    output_last_layer = None

    assert output_last_layer is not None, 'Design MLP architecture'

    return jax.nn.log_softmax(output_last_layer)

# We use vmap to make the prediction batched.
batched_predict = vmap(predict, in_axes=(None, 0))
```

```python
 def accuracy(params, x, y):
    """
    Compute accuracy measure.

    Args:
        x: input data
        y: target data

    Returns:
        The classification accuracy
    """

    ######################
    #   YOUR CODE HERE   #
    ######################
    # Compute the accuracy.
    #
    # Reminder: the accuracy is
    # the average number of correctly
    # predicted classes.
    #
    # Hint: Use batched_predict to get the
    # classes ids predicted by the network,
    # and compare them with the target
    # classes from the ground truth.

    accuracy = None

    assert accuracy is not None, 'Accuracy should be computed'

    return accuracy
```

## Loss and update

```python
def loss(params, x, y):
    """
    Loss function.

    Args:
        params: the parameters of the neural network
        x: input data
        y: target data (class)

    Returns:
        The computed loss
    """
    preds = batched_predict(params, x)
    return -jnp.mean(preds * y)

@jit
def update(params, x, y):
    """
    The optimization function.

    Args:
        params: the parameters of the neural network
        x: input data
        y: target data (class)

    Returns:
        The updated parameters
    """
    loss_value, grads = jax.value_and_grad(loss)(params, x, y)
    return [(w - STEP_SIZE * dw, b - STEP_SIZE * db)
            for (w, b), (dw, db) in zip(params, grads)], loss_value
```

## Train

```python
def train_loop(train_examples, test_examples, train_labels, test_labels,
               layer_sizes=None):
    # Visualize learning progress with a bar.
    pbar = tqdm(total=NUM_EPOCHS, position=0, leave=True)

    X_train, X_test = get_discrete_data(train_examples, test_examples)
    if layer_sizes is None:
        layer_sizes = [X_train.shape[1], 256,  2]
    params = init_network_params(layer_sizes, random.PRNGKey(0))

    # Transform target data to one-hot representation.
    y = jax.nn.one_hot(train_labels, N_TARGETS)
    y_test = jax.nn.one_hot(test_labels, N_TARGETS)

    for epoch in range(NUM_EPOCHS):
        # Update parameters for each batch.
        for x_t, y_t in batch(X_train, y):
            params, loss_value = update(params, x_t, y_t)
        pbar.update(1)

        # Compute accuracy on training and test.
        train_acc = accuracy(params, X_train, y)
        test_acc = accuracy(params, X_test, y_test)

        pbar.set_description('Loss value is {0:.2f}, training accuracy is '
                            '{1:.5}, test accuracy is {2:.5}'.format(loss_value,
                                                                    train_acc,
                                                                    test_acc))
```

```python
train_loop(train_examples, test_examples, train_labels, test_labels)
```

## Preprocessing + TF-IDF

Text preprocessing is traditionally an important step for NLP tasks. It transforms text into a more digestible form so that machine learning algorithms can perform better. For example:
- it removes 'not meaningful words' such as a,an,the,for, etc. (so-called *stopwords*)
- lowercase everything
- remove symbols
- remove or process hashtags, mentions or emoticons (particularly important for social media text)

We expect that 'cleaning' the text will help us on obtaining better performance.

```python
# Preprocessing functions
def split_hashtag(tag):
    pattern = re.compile(r'[A-Z][a-z]+|\d+|[A-Z]+(?![a-z])')
    return pattern.findall(tag)

def quick_preprocessing(sentence, language, process_urls=False,
                        process_mentions=False, process_emoticon=False,
                        split_hashtags=False):
    """
    A quick preprocessing function that removes unwanted tokens from the text

    Args:
        sentence: the sentence to be preprocessed
        language: the language the sentence is in
        process_urls: whether to process URLs
        process_mentions: whether to process mentions
        process_emoticon: whether to process emoticons
        split_hashtags: whether to split hashtags

    Returns:
        The modified sentence given the preprocessing configuration

    """
    url_re = (r'http[s]?://(?:[a-z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-f]'
              r'[0-9a-f]))+')
    mention_re = r'(?:@[\w_]+)'
    emoticons_re = r'(?::|;|=)(?:-)?(?:\)|\(|D|P)'

    if process_urls:
        sentence = re.sub(url_re, 'TAGurl', sentence, flags=re.MULTILINE)
    if process_mentions:
        sentence = re.sub(mention_re, 'TAGmention', sentence,
                          flags=re.MULTILINE)
    if process_emoticon:
        sentence = re.sub(emoticons_re, 'TAGemoticon', sentence,
                          flags=re.MULTILINE)

    if split_hashtags:
        hashtag_list = list({tag.strip('#') for tag in sentence.split()
                             if tag.startswith('#')})
        hashtag_split_list = [' '.join(split_hashtag(h)) for h in hashtag_list]
        for i in range(len(hashtag_list)):
            sentence = sentence.replace(hashtag_list[i],
                                        ' ' + hashtag_split_list[i] + ' ')

    sentence = sentence.lower()
    sentence = ' '.join(
        re.sub('(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)',' ',
               sentence).split())
    tokenizer = RegexpTokenizer(r'\w+')
    words = tokenizer.tokenize(sentence)
    good_words = [w for w in words]
    return ' '.join(good_words)

def complete_preprocessing(sentence, language):
    return quick_preprocessing(sentence, language, True, True, True, True)
```

```python
# Apply preprocessing to train and test samples.
preprocessed_training = []
preprocessed_testing = []

pbar = tqdm(total=len(train_examples), position=0, leave=True)
for tr_example, ts_example in zip(train_examples, test_examples):
    preprocessed_training.append(complete_preprocessing(tr_example, 'english'))
    preprocessed_testing.append(complete_preprocessing(ts_example, 'english'))
    pbar.update(1)

pbar.close()
```

We use the same code as before for performing the training

```python
train_loop(preprocessed_training, preprocessed_testing, train_labels,
           test_labels)
```

## Entering BERT Embeddings [EXERCISE]

Recently, pretrained sentence embeddings, lead by Bidirectional Encoder Representations from Transformers ([BERT](https://www.aclweb.org/anthology/N19-1423/)), have become the standard in any NLP application. This means that sentences can be transformed into a dense and meaningful representation exploiting these large pretrained language models.

In this section, we use  the [Sentence-Transformers](https://github.com/UKPLab/sentence-transformers) library that permits to compute dense vector representations for sentences and paragraphs using transformer networks like BERT / RoBERTa / XLM-RoBERTa etc.

**NOTE**: this might generate an OOM errror because these embeddings are generated with pytroch (that is fighting with Jax for the GPU). One thing you can do is: restart the notebook, generate the embeddings, and then train this new network with jax. You can look at jax memory allocation settings [here](https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html).

%%capture
model_bert = SentenceTransformer('distilbert-base-nli-mean-tokens')

```python
%%capture
model_bert = SentenceTransformer('distilbert-base-nli-mean-tokens')
```

```python
# Transform the training and testing data into sentence embeddings.
# This might take a few minutes.
embeddings_train = jnp.array(model_bert.encode(train_examples, show_progress_bar=True))
embeddings_test = jnp.array(model_bert.encode(test_examples, show_progress_bar=True))
```

```python
def bert_train_loop(X_train, X_test, train_labels, test_labels,
               layer_sizes=None):
    # Visualize learning progress with a bar.
    pbar = tqdm(total=NUM_EPOCHS, position=0, leave=True)

    if layer_sizes is None:
        layer_sizes = [X_train.shape[1], 256,  2]
    params = init_network_params(layer_sizes, random.PRNGKey(0))

    # Transform target data to one-hot representation.
    y = jax.nn.one_hot(train_labels, N_TARGETS)
    y_test = jax.nn.one_hot(test_labels, N_TARGETS)

    for epoch in range(NUM_EPOCHS):
        # Update parameters for each batch.
        for x_t, y_t in batch(X_train, y):
            params, loss_value = update(params, x_t, y_t)
        pbar.update(1)

        # Compute accuracy on training and test.
        train_acc = accuracy(params, X_train, y)
        test_acc = accuracy(params, X_test, y_test)

        pbar.set_description('Loss value is {0:.2f}, training accuracy is '
                            '{1:.5}, test accuracy is {2:.5}'.format(loss_value,
                                                                    train_acc,
                                                                    test_acc))
    return params
```

```python
bert_network = bert_train_loop(embeddings_train, embeddings_test, train_labels, test_labels)
```

## Let's play with our new network :)

We are going to test our sentiment model on two positive documents and 1 negative one. Let's see if the model is able to predict the classes as expected.
You can add custom sentences and see the results!

```python
examples = jnp.array(model_bert.encode(["wow, they nice movie totally recommend it",
                                        "worst movie ever, I will never go to the cinema again",
                                        "I liked it a lot, best movie ever"]))
```

```python
jnp.argmax(jnp.exp(batched_predict(bert_network, examples)), axis=1)
```


```python
# %load /Users/facaiyan/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
plt.rcParams['axes.grid'] = False

import numpy as np

#from IPython.display import SVG
def show_image(filename, figsize=None, res_dir=True):
    if figsize:
        plt.figure(figsize=figsize)

    if res_dir:
        filename = './res/{}'.format(filename)

    plt.imshow(plt.imread(filename))
```

Chapter 6 Deep Feedforward Networks
===================================

feedforward: no feedback connections.

input layer -> hidden layers -> output layer

\begin{align}
    y &= f(x; \theta) \approx f^*(x) \\
      &= W^T \phi(x) + b
\end{align}

how to choose the mapping $\phi$?

+ use a very generic $\phi$, such as RBF kernel $\implies$ generation remains poor.
+ manually engineer $\phi$ $\implies$ classical feature engineer.
+ learn $\phi$ by deep learning. $y = f(x; \theta, w) = \phi(x; \theta)^T w$.
  - advantage: only needs to find the right general function family *VS* find the right function.

### 6.1 Example: Learning XOR

$h = g(W^T x + c)$, where an affine transformation followed by an activation function $g$.

```python
show_image("fig6_2.png", figsize=(5, 8))
```

default activation function is rectified linear unit (ReLU): $g(z) = max\{0, z\}$
+ advantage:
  1. piecewise linear function: very close to linear.
  2. easy to optimize with gradient-base methods.

```python
relu = lambda x: np.maximum(0, x)

x = np.linspace(-2, 2, 1000)
y =  relu(x)

plt.ylim([-1, 3])
plt.grid(True)
plt.plot(x, y)
```

### 6.2 Gradient-Based Learning

important:

+ initialize all weights to small random values.
+ initialize biases to zero or small positive values (push result to right area of ReLU).

#### 6.2.1 Cost Functions

In most cases,

+ our parametric model defines a distribution $p(y | x; \theta)$,
+ simply use the priciple of maximum likelihood.

$\implies$ [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) as the cost function.

##### 6.2.1.1 Learning Conditional Distributions with Maximum Likelihood

maximum likelihdood in neural networks => cost function is simply the negative log-likelihood == cross-entropy.

\begin{equation}
    J(\theta) = - \mathbb{E}_{x, y \sim \hat{p}_{data}} \log p_{model}(y | x)
\end{equation}

advantage:

+ Specifying a model p(y | x) automatically determines a cost function log p(y | x). => removes the burden of designing cost functions for each model.
+ Undo the exp of some output units => avoid staturate problem (flat area, very small gradient).

unusual property of the cross-entropy cose: does not have a minimum value (negative infinity). => regularization.

##### 6.2.1.2 Learning Conditional Statistics

cost L2: learn mean of y when x is given.

cost L1: learn median of y when x is given.

#### 6.2.2 Output Units

##### 6.2.2.1 Linear Units for Gaussian Output Distributions

Linear output layers are often used to produce the mean of a conditional Gaussian distribution.

这里意思是说，给定$x$，它对应的样本集$y$应是高斯分布。而用线性模型来学习，预测的正是样本集均值$f(x) = \bar{y}$。可见，这种情况常见于回归问题。

##### 6.2.2.2 Sigmoid Units for Bernoulli Output Distributions

binary classification

\begin{align}
    P(y) &= \delta((2y - 1) z) \quad \text{where } z = w^T h + b \\
    J(\theta) &= - \log P(y | x) \quad \text{undo exp} \\
         &= \zeta ((1 - 2y) z)
\end{align}

maximum likelihood is almost always the preferred approach to training sigmoid
output units.

##### 6.2.2.3 Softmax Units for Multinoulli Output Distributions

multiple classification

\begin{equation}
    \operatorname{softmax}(z)_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}
\end{equation}

\begin{align}
    \log \operatorname{softmax}(z)_i &= z_i - log \sum_j \exp(z_j) \\
        & \approx z_i - \max_j (z_j)
\end{align}

+ Overall, unregularized maximum likelihood will drive the softmax to predict the fraction of counts of counts of each outcome observed in the training set.

+ The argument $z$ can be produced in two different ways:
  1. one vs rest: N estimators
  2. choose one class as "pivot" class: N - 1 estimators

+ softmax provides a "softened" version of the argmax.

##### 6.2.2.4 Other Output Types

In general, think neural network as representing a function $f(x; \thetha) = w$, which provides the parameters for a distribution over $y$. Our loss function can then be inperpreted as $- \log p(y; w(x))$.

### 6.3 Hidden Units

+ Rectified linear units are an excellent default choice of hidden unit.
+ In practice, gradient descent still performs well enough for functions which are not actually differentiable.

#### 6.3.1 Rectified Linear Units and Their Generalizations

#### 6.3.2 Logistic Sigmoid and Hyperbolic Tangent

The widespread saturation of sigmoidal units => use as hidden units is now discouraged.

##### 6.3.3 Other Hidden Units

### 6.4 Architecture Design

+ layers: group of units
+ chain structure
  1. the depth of the network
  2. the width of each layer

#### 6.4.1 Universal Approximation Properties and Depth

universal approximation theorem: large MLP will be able to **represent** any function. However, we are not guaranteed that the training algorithm will be able to **learn** that function.

Empirically, greater depth does seem to result in better generalization.

#### 6.4.2 Other Architectural Considerations

### 6.5 Back-Propagation and Other Differentiation Algorithms

chain rule:

\begin{equation}
    \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx}
\end{equation}

### 6.6 Historical Notes

```python

```


# Defining Custom Display Logic for Your Own Objects

## Overview

In Python, objects can declare their textual representation using the `__repr__` method.  IPython expands on this idea and allows objects to declare other, richer representations including:

* HTML
* JSON
* PNG
* JPEG
* SVG
* LaTeX

This Notebook shows how you can add custom display logic to your own classes, so that they can be displayed using these rich representations. There are two ways of accomplishing this:

1. Implementing special display methods such as `_repr_html_`.
2. Registering a display function for a particular type.

In this Notebook we show how both approaches work.

Before we get started, we will import the various display functions for displaying the different formats we will create.

```
from IPython.display import display
from IPython.display import (
    display_html, display_jpeg, display_png,
    display_javascript, display_svg, display_latex
)
```

## Implementing special display methods

The main idea of the first approach is that you have to implement special display methods, one for each representation you want to use. Here is a list of the names of the special methods and the values they must return:

* `_repr_html_`: return raw HTML as a string
* `_repr_json_`: return raw JSON as a string
* `_repr_jpeg_`: return raw JPEG data
* `_repr_png_`: return raw PNG data
* `_repr_svg_`: return raw SVG data as a string
* `_repr_latex_`: return LaTeX commands in a string surrounded by "$".

### Model Citizen: pandas

A prominent example of a package that has IPython-aware rich representations of its objects is [pandas](http://pandas.pydata.org/).

A pandas DataFrame has a rich HTML table representation,
using `_repr_html_`.

```
import io
import pandas
```

```
%%writefile data.csv
Date,Open,High,Low,Close,Volume,Adj Close
2012-06-01,569.16,590.00,548.50,584.00,14077000,581.50
2012-05-01,584.90,596.76,522.18,577.73,18827900,575.26
2012-04-02,601.83,644.00,555.00,583.98,28759100,581.48
2012-03-01,548.17,621.45,516.22,599.55,26486000,596.99
2012-02-01,458.41,547.61,453.98,542.44,22001000,540.12
2012-01-03,409.40,458.24,409.00,456.48,12949100,454.53

```

    Writing data.csv

```
df = pandas.read_csv("data.csv")
pandas.set_option('display.notebook_repr_html', False)
df
```

             Date    Open    High     Low   Close    Volume  Adj Close
    0  2012-06-01  569.16  590.00  548.50  584.00  14077000     581.50
    1  2012-05-01  584.90  596.76  522.18  577.73  18827900     575.26
    2  2012-04-02  601.83  644.00  555.00  583.98  28759100     581.48
    3  2012-03-01  548.17  621.45  516.22  599.55  26486000     596.99
    4  2012-02-01  458.41  547.61  453.98  542.44  22001000     540.12
    5  2012-01-03  409.40  458.24  409.00  456.48  12949100     454.53

rich HTML can be activated via `pandas.set_option`.

```
pandas.set_option('display.notebook_repr_html', True)
df
```

      Date
      Open
      High
      Low
      Close
      Volume
      Adj Close

      0
       2012-06-01
       569.16
       590.00
       548.50
       584.00
       14077000
       581.50

      1
       2012-05-01
       584.90
       596.76
       522.18
       577.73
       18827900
       575.26

      2
       2012-04-02
       601.83
       644.00
       555.00
       583.98
       28759100
       581.48

      3
       2012-03-01
       548.17
       621.45
       516.22
       599.55
       26486000
       596.99

      4
       2012-02-01
       458.41
       547.61
       453.98
       542.44
       22001000
       540.12

      5
       2012-01-03
       409.40
       458.24
       409.00
       456.48
       12949100
       454.53

```
lines = df._repr_html_().splitlines()
print "\n".join(lines[:20])
```

          Date
          Open
          High
          Low
          Close
          Volume
          Adj Close

          0
           2012-06-01
           569.16
           590.00

### Exercise

Write a simple `Circle` Python class.  Don't even worry about properties such as radius, position, colors, etc. To help you out use the following representations (remember to wrap them in Python strings):

For HTML:

    ○

For SVG:

    <svg width="100px" height="100px">
        <circle cx="50" cy="50" r="20" stroke="black" stroke-width="1" fill="white"/>
    </svg>

For LaTeX (wrap with `$` and use a raw Python string):

    \bigcirc

After you write the class, create an instance and then use `display_html`, `display_svg` and `display_latex` to display those representations.

Tips : you can slightly tweek the representation to know from which `_repr_*_` method it came from.
For example in my solution the svg representation is blue, and the HTML one show "`HTML`" between brackets.

### Solution

Here is my simple `MyCircle` class:

```
%load soln/mycircle.py
```

Now create an instance and use the display methods:

```
c = MyCircle()
```

```
display_html(c)
```

○ (html)

```
display_svg(c)
```

```
display_latex(c)
```

$\bigcirc \LaTeX$

```
display_javascript(c)
```

## Adding IPython display support to existing objects

When you are directly writing your own classes, you can adapt them for display in IPython by following the above example.  But in practice, we often need to work with existing code we can't modify.  We now illustrate how to add these kinds of extended display capabilities to existing objects. To continue with our example above, we will add a PNG representation to our `Circle` class using Matplotlib.

### Model citizen: sympy

[SymPy](http://sympy.org) is another model citizen that defines rich representations of its object.
Unlike pandas above, sympy registers display formatters via IPython's display formatter API, rather than declaring `_repr_mime_` methods.

```
from sympy import Rational, pi, exp, I, symbols
x, y, z = symbols("x y z")
```

```
r = Rational(3,2)*pi + exp(I*x) / (x**2 + y)
r
```

    3*pi/2 + exp(I*x)/(x**2 + y)

SymPy provides an `init_printing` function that sets up advanced $\LaTeX$
representations of its objects.

```
from sympy.interactive.printing import init_printing
init_printing()
r
```

To add a display method to an existing class, we must use IPython's display formatter API.  Here we show all of the available formatters:

```
ip = get_ipython()
for mime, formatter in ip.display_formatter.formatters.items():
    print '%24s : %s' % (mime, formatter.__class__.__name__)

```

                   text/html : HTMLFormatter
                  image/jpeg : JPEGFormatter
               image/svg+xml : SVGFormatter
                   image/png : PNGFormatter
      application/javascript : JavascriptFormatter
                  text/latex : LatexFormatter
            application/json : JSONFormatter
                  text/plain : PlainTextFormatter

Let's grab the PNG formatter:

```
png_f = ip.display_formatter.formatters['image/png']
```

We will use the `for_type` method to register our display function.

```
png_f.for_type?
```

As the docstring describes, we need to define a function the takes the object as a parameter and returns the raw PNG data.

```
%matplotlib inline
import matplotlib.pyplot as plt
```

```
class AnotherCircle(object):
    def __init__(self, radius=1, center=(0,0), color='r'):
        self.radius = radius
        self.center = center
        self.color = color

    def __repr__(self):
        return "<%s Circle with r=%s at %s>" % (
                    self.color,
                    self.radius,
                    self.center,
                )

c = AnotherCircle()
c
```

    <r Circle with r=1 at (0, 0)>

```
from IPython.core.pylabtools import print_figure

def png_circle(circle):
    """Render AnotherCircle to png data using matplotlib"""
    fig, ax = plt.subplots()
    patch = plt.Circle(circle.center,
                       radius=circle.radius,
                       fc=circle.color,
                       )
    ax.add_patch(patch)
    plt.axis('scaled')
    data = print_figure(fig, 'png')
    # We MUST close the figure, otherwise IPython's display machinery
    # will pick it up and send it as output, resulting in a double display
    plt.close(fig)
    return data
```

```
c = AnotherCircle()
print repr(png_circle(c)[:10])
```

    '\x89PNG\r\n\x1a\n\x00\x00'

Now we register the display function for the type:

```
png_f.for_type(AnotherCircle, png_circle)
```

Now all `Circle` instances have PNG representations!

```
c2 = AnotherCircle(radius=2, center=(1,0), color='g')
c2
```

```
display_png(c2)
```

## return the object

```
# for demonstration purpose, I do the same with a circle that has no _repr_javascript method
class MyNoJSCircle(MyCircle):

    def _repr_javascript_(self):
        return

cNoJS = MyNoJSCircle()
```

Of course you can now still return the object, and this will use compute all the representations, store them in the notebook and show you the appropriate one.

```
cNoJS
```

Or just use `display(object)` if you are in a middle of a loop

```
for i in range(3):
    display(cNoJS)
```

Advantage of using `display()` versus `display_*()` is that all representation will be stored in the notebook document and notebook file, they are then availlable for other frontends or post-processing tool like `nbconvert`.

Let's compare `display()` vs `display_html()` for our circle in the Notebook Web-app and we'll see later the difference in nbconvert.

```
print "I should see a nice html circle in web-app, but"
print "nothing if the format I'm viewing the notebook in"
print "does not support html"
display_html(cNoJS)
```

```
print "Whatever the format I will see a representation"
print "of my circle"
display(cNoJS)
```

```
print "Same if I return the object"
cNoJS
```

```
print "But not if I print it"
print cNoJS
```

examples/IPython Kernel/Old Custom Display Logic.ipynb


# Fake News Detection: an application of classic NLP techniques
**Universidade de Brasília**

Faculdade de Tecnologia

Programa de Pós-graduação em Engenharia Elétrica (PPGEE)

## Author: Stefano M P C Souza (stefanomozart@ieee.org)
Advisor: Daniel G Silva
Advisor: Anderson C A Nascimento

## 1. Experiment design

We want to study the impact of various NLP preprocessing techniques in the task of text classification for fake news detection. We are going to use the pipeline from [[1](#bot)] for model traing, tuning (hyper-parameter search) and comparison. The following ML algorithms are used:
1. Naive Bayes:
2. Decision Trees:
2. K-Nearest Neighbour:
3. Logistic Regression:
3. Suport-Vector Machines:
4. Random Forest:
5. XGBoost:

All models are trained and tested on a binary (*fake*/real) classification task. The *pipeline*, written by the author, extends the `sklearn.pipeline.Pipeline` class, from scikit-learn, and consists of the following steps:
1. **Training and tuning**: uses a random search algorithm to select the best hyper-parameters for each ML model;
2. **Selection**: for each dataset, selects the models with best performance, on the selected metric, for the validation set. The selected model is trained one more time with the concatanation of the training and the valiudation set;
5. **Test**: the models selected on the previous step, and trained on training+validation sets are used to classify texts in the test set. The final score, on the selected metric, is record so we can compare .

## 2. Natural Language Processing

### 2.1. Selected techniques

1. **Tokenization**: the text, a sequence of caracters, is transformed in a ordered collection of tokens (words, punctiation marks, emojis, etc);
2. **Stopword removal (SwR)**: removing words that do not add information, in the statistical learning sense, to any specific class in the sample. Most algorithms rely on experts dictionaries or on statistical measures such as *Mutual Information*;
3. **Stemming**: Stemming is the reduction of variant forms of a word, eliminating inflectional morphemes such as verbal tense or plural suffixes, in order to provide a common representation, the root or stem. The intuition is to perform a dimensionality reduction on the dataset, removing rare morphological word variants, and reduce the risk of bias on word statistics measured on the documents;
4. **Lemmatization:** Lemmatization consists on the reduction of each token to a linguistically valid root or lemma. The goal, from the statistical perspective, is exactly the same as in stemming: reduce variance in term frequency. It is sometimes compared to the normalization of the word sample, and aims to provide more accurate transformations than stemming, from the linguistic perspective;
5. **Bag-of-Words (BoW)**: The BoW algorithm used in most NLP libraries is based on the *Vector Space Model* (VSM) and associates the tokens with with the corresponding term frequency: the number of occurrences of that token in that document. This algorithm produces an unordered set that does not retain any information on word order or proximity in the document ;
6. **Term Frequency/Inverse Document Frequency (TF-IDF)**: Similar to the Vector Space Model Bag-of-Words, the TF-IDF (sometimes expressed as TF*IDF) document representation will associate each token in a document with a normalized or smoothed term frequency, weighted by the inverse of the frequency at which the term occurs in $D$, the corpus, or in the list of documents under processing. That is, $f_{t_i, d_j}$, the number of occurrences of token $t_i$ in document $d_j$, is replaced by $\mathrm{tf\cdot{idf}}$, where:

\begin{equation}
\begin{split}
  \mathrm{tf}(t_i,d_j) &=1 + \log \frac{f_{t_i,d_j}}{\sum_{t\in d_j}{f_{t,d_j}}} \\
  \mathrm{idf}(t_i, D) &=  1 + \log \frac{|D|+1}{|\{d \in D : t_i \in d\}|+1}
\end{split}
\end{equation}

### 2.2. Datasets

We selected 2 datasets in English and 2 in Portuguese. Each pair has a dataset with full-length news
articles and a dataset comprised of short statements, or sentences. The purpose of experimenting
with different languages and text sizes was to observe how these variables may impact preprocessing
and training cost, and, ultimately, model performance.

The selected datasets are:
  - **Liar Dataset (liar):** curated by the UC Santa Barbara NLP Group, contains 12791 claims
  by North-American politicians and celebrities, classified as `true`, `mostly-true`, `half-true`,
  `barely-true`, `false` and `pants-on-fire` [[2](#liar)];

  - **Source Based Fake News Classification (sbnc):** 2020 full-length news manually labeled
  as `Real` or `Fake` [[3](#sbnc)];

  - **FactCk.br:** 1313 claims by Brazilian politicians, manually annotated by fact checking agencies\footnote{\url{https://piaui.folha.uol.com.br/lupa}, \url{https://www.aosfatos.org} and \url{https://apublica.org}} as `true`, `false`, `imprecise` and `others` [[4](#factckbr)];

  - **Fake.br:** 7200 full-length news articles, with text and metadata, manually flagged as `real` or `fake` news [[5](#fakebr)].

The classification experiments were preceded by a dataset preparation so that each dataset would have the same structure:
1. **label**: (boolean) indicating if that text was labeled as *fake news*;
2. **text**: (string) a concatenation of title (when available) and news body.

## 3. Processing
#### Daset preparation

```python
# importando bibliotecas de propósito geral, utilizada na manipulação dos datasets
import pandas as pd
import numpy as np
import joblib

import os, sys, inspect, time
sys.path.insert(0, os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))))
```

```python
# Os datasets serão armezenados em um dicionário a fim de facilitar
# a iteração de cada experimento sobre todos os datasets
datasets = [
    # Dataset 1: Liar
    {'name':  'liar', 'lang': 'en', 'df': pd.read_csv('datasets/liar/liar.csv')},

    # Dataset 2: Source Based FK Detection
    {'name': 'sbnc', 'lang': 'en', 'df': pd.read_csv('datasets/sbnc/sbnc.csv')},

    # Dataset 3: Fake.br
    {'name': 'fake.br', 'lang': 'pt', 'df': pd.read_csv('datasets/fake.br/fake.br.csv')},

    # Dataset 4: FactCk.br
    {'name': 'factck.br', 'lang': 'pt', 'df': pd.read_csv("datasets/factck.br/factck.br.csv")}
]

experiments = {
   "E01": {'preprocessing_time': {}, 'name': 'bow'},
   "E02": {'preprocessing_time': {}, 'name': 'bow.swr'},
   "E03": {'preprocessing_time': {}, 'name': 'bow.stem'},
   "E04": {'preprocessing_time': {}, 'name': 'bow.lemm'},
   "E05": {'preprocessing_time': {}, 'name': 'bow.lemm.swr'},
   "E06": {'preprocessing_time': {}, 'name': 'tfidf'},
   "E07": {'preprocessing_time': {}, 'name': 'tfidf.swr'},
   "E08": {'preprocessing_time': {}, 'name': 'tfidf.stem'},
   "E09": {'preprocessing_time': {}, 'name': 'tfidf.lemm'},
   "E10": {'preprocessing_time': {}, 'name': 'tfidf.lemm.swr'},
}
```

#### Data split

```python
from sklearn.model_selection import train_test_split

for d in datasets:
    train_valid, test = train_test_split(d['df'], stratify=d['df'].label, test_size=0.2, random_state=42)
    train, valid = train_test_split(train_valid, stratify=train_valid.label, test_size=0.2, random_state=42)

    train_valid.to_csv(f"datasets/{d['name']}/train.valid.csv", index=False)
    train.to_csv(f"datasets/{d['name']}/train.csv", index=False)
    valid.to_csv(f"datasets/{d['name']}/valid.csv", index=False)
    test.to_csv(f"datasets/{d['name']}/test.csv", index=False)

    d['train.valid'] = train_valid
    d['train'] = train
    d['valid'] = valid
    d['test'] = test
```

### 3.1. Bag of Words (BoW)

```python
from sklearn.feature_extraction.text import CountVectorizer
from scipy.sparse import save_npz

for d in datasets:
    t = time.process_time()

    cv = CountVectorizer()
    train = cv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.bow.npz", train)
    valid = cv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.bow.npz", valid)

    cv = CountVectorizer()
    train = cv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.bow.npz", train)
    test = cv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.bow.npz", test)

    experiments["E01"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 3.2. BoW and Stopword Removal  (BoW + SwR):

```python
import nltk

swr = {
    'en': nltk.corpus.stopwords.words("english"),
    'pt': nltk.corpus.stopwords.words("portuguese")
}

for d in datasets:
    t = time.process_time()

    cv = CountVectorizer(stop_words=swr[d['lang']])
    train = cv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.bow.swr.npz", train)
    valid = cv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.bow.swr.npz", valid)

    cv = CountVectorizer(stop_words=swr[d['lang']])
    train = cv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.bow.swr.npz", train)
    test = cv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.bow.swr.npz", test)

    experiments["E02"]['preprocessing_time'][d['name']] = time.process_time() - t
```

### 3.3. BoW and Stemming

```python
cv_analyzer = CountVectorizer().build_analyzer()

snowball = {
    'en': nltk.stem.SnowballStemmer('english'),
    'pt': nltk.stem.SnowballStemmer('portuguese')
}

def en_stemmer(doc):
    return (snowball['en'].stem(w) for w in cv_analyzer(doc))

def pt_stemmer(doc):
    return (snowball['pt'].stem(w) for w in cv_analyzer(doc))

cv_stemmer = {
    'en': en_stemmer,
    'pt': pt_stemmer
}
```

```python
for d in datasets:
    t = time.process_time()

    cv = CountVectorizer(analyzer=cv_stemmer[d['lang']])
    train = cv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.bow.stem.npz", train)
    valid = cv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.bow.stem.npz", valid)

    train = cv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.bow.stem.npz", train)
    test = cv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.bow.stem.npz", test)

    experiments["E03"]['preprocessing_time'][d['name']] = time.process_time() - t
```

### 3.4. BoW and Lemmatization

```python
import stanza
stanza_pt = stanza.Pipeline(lang='pt', processors='tokenize,mwt,pos,lemma')

wordnet = nltk.stem.WordNetLemmatizer()

def en_lemma(doc):
    return [wordnet.lemmatize(token) for token in nltk.word_tokenize(doc)]

def pt_lemma(doc):
    d = stanza_pt(doc).sentences
    return [w.lemma for s in d for w in s.words]

lemmatizer = {
    'en': en_lemma,
    'pt': pt_lemma
}

```

    2021-06-08 13:57:05 INFO: Loading these models for language: pt (Portuguese):
    =======================
    | Processor | Package |
    -----------------------
    | tokenize  | bosque  |
    | mwt       | bosque  |
    | pos       | bosque  |
    | lemma     | bosque  |
    =======================

    2021-06-08 13:57:05 INFO: Use device: gpu
    2021-06-08 13:57:05 INFO: Loading: tokenize
    2021-06-08 13:57:09 INFO: Loading: mwt
    2021-06-08 13:57:09 INFO: Loading: pos
    2021-06-08 13:57:11 INFO: Loading: lemma
    2021-06-08 13:57:11 INFO: Done loading processors!

```python
for d in datasets:
    t = time.process_time()

    cv = CountVectorizer(tokenizer=lemmatizer[d['lang']])
    train = cv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.bow.lemm.npz", train)
    valid = cv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.bow.lemm.npz", valid)

    cv = CountVectorizer(tokenizer=lemmatizer[d['lang']])
    train = cv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.bow.lemm.npz", train)
    test = cv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.bow.lemm.npz", test)

    experiments["E04"]['preprocessing_time'][d['name']] = time.process_time() - t
```

### 3.5. BoW, Lemmatization and SwR

```python
for d in datasets:
    t = time.process_time()

    cv = CountVectorizer(tokenizer=lemmatizer[d['lang']], stop_words=swr[d['lang']])
    train = cv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.bow.lemm.swr.npz", train)
    valid = cv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.bow.lemm.swr.npz", valid)

    cv = CountVectorizer(tokenizer=lemmatizer[d['lang']], stop_words=swr[d['lang']])
    train = cv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.bow.lemm.swr.npz", train)
    test = cv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.bow.lemm.swr.npz", test)

    experiments["E05"]['preprocessing_time'][d['name']] = time.process_time() - t
```

    /home/dev/.local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py:388: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ["'d", "'ll", "'re", "'s", "'ve", 'could', 'doe', 'ha', 'might', 'must', "n't", 'need', 'sha', 'wa', 'wo', 'would'] not in stop_words.
      warnings.warn('Your stop_words may be inconsistent with '
    /home/dev/.local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py:388: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['estar', 'estivar', 'fôr', 'haver', 'ir', 'm', 'ser', 'ter', 'vós'] not in stop_words.
      warnings.warn('Your stop_words may be inconsistent with '

### 3.6. Term-Frequency/Inverse Document Frequency (TF-IDF)

```python
from sklearn.feature_extraction.text import TfidfVectorizer

for d in datasets:
    t = time.process_time()

    tv = TfidfVectorizer()
    train = tv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.tfidf.npz", train)
    valid = tv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.tfidf.npz", valid)

    tv = TfidfVectorizer()
    train = tv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.tfidf.npz", train)
    test = tv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.tfidf.npz", test)

    experiments["E06"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 3.7. TF-IDF and SwR

```python
from sklearn.feature_extraction.text import TfidfVectorizer

for d in datasets:
    t = time.process_time()

    tv = TfidfVectorizer(stop_words=swr[d['lang']])
    train = tv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.tfidf.swr.npz", train)
    valid = tv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.tfidf.swr.npz", valid)

    tv = TfidfVectorizer(stop_words=swr[d['lang']])
    train = tv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.tfidf.swr.npz", train)
    test = tv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.tfidf.swr.npz", test)

    experiments["E07"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 3.8. TF-IDF and Stemming

```python
#norm_count_vec = TfidfVectorizer(use_idf=False, norm='l2')
tf_analyzer = TfidfVectorizer().build_analyzer()

snowball = {
    'en': nltk.stem.SnowballStemmer('english'),
    'pt': nltk.stem.SnowballStemmer('portuguese')
}

def en_stemmer(doc):
    return (snowball['en'].stem(w) for w in tf_analyzer(doc))

def pt_stemmer(doc):
    return (snowball['pt'].stem(w) for w in tf_analyzer(doc))

tf_stemmer = {
    'en': en_stemmer,
    'pt': pt_stemmer
}
```

```python
from sklearn.feature_extraction.text import TfidfVectorizer

for d in datasets:
    t = time.process_time()

    tv = TfidfVectorizer(tokenizer=tf_stemmer[d['lang']])
    train = tv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.tfidf.stem.npz", train)
    valid = tv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.tfidf.stem.npz", valid)

    tv = TfidfVectorizer(tokenizer=tf_stemmer[d['lang']])
    train = tv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.tfidf.stem.npz", train)
    test = tv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.tfidf.stem.npz", test)

    experiments["E08"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 3.9. TF-IDF and Lemmatization

```python
from sklearn.feature_extraction.text import TfidfVectorizer

for d in datasets:
    t = time.process_time()

    tv = TfidfVectorizer(tokenizer=lemmatizer[d['lang']])
    train = tv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.tfidf.lemm.npz", train)
    valid = tv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.tfidf.lemm.npz", valid)

    tv = TfidfVectorizer(tokenizer=lemmatizer[d['lang']])
    train = tv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.tfidf.lemm.npz", train)
    test = tv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.tfidf.lemm.npz", test)

    experiments["E09"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 3.10. TF-IDF, Lemmatization and SwR

```python
from sklearn.feature_extraction.text import TfidfVectorizer

for d in datasets:
    t = time.process_time()

    tv = TfidfVectorizer(tokenizer=lemmatizer[d['lang']], stop_words=swr[d['lang']])
    train = tv.fit_transform(d['train'].text)
    save_npz(f"datasets/{d['name']}/train.tfidf.lemm.swr.npz", train)
    valid = tv.transform(d['valid'].text)
    save_npz(f"datasets/{d['name']}/valid.tfidf.lemm.swr.npz", valid)

    tv = TfidfVectorizer(tokenizer=lemmatizer[d['lang']], stop_words=swr[d['lang']])
    train = tv.fit_transform(d['train.valid'].text)
    save_npz(f"datasets/{d['name']}/train.valid.tfidf.lemm.swr.npz", train)
    test = tv.transform(d['test'].text)
    save_npz(f"datasets/{d['name']}/test.tfidf.lemm.swr.npz", test)

    experiments["E10"]['preprocessing_time'][d['name']] = time.process_time() - t

```

### 4. Saving the pre-processed datasets

```python
import joblib

joblib.dump(datasets, 'datasets.pyd')
joblib.dump(experiments, 'experiments.pyd')
```

    ['experiments.pyd']

## References

[1]: Souza, S.M.P. et al. *Tuning machine learning models to detect bots on Twitter*. 2020 Workshop on Communication Networks and Power Systems (WCNPS). Brasilia, 2020.

[2] Wlliam Yang Wang, "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection, to appear in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), short paper, Vancouver, BC, Canada, July 30-August 4, ACL.

[3]. A.  Bharadwaj,  B.  Ashar,  P.  Barbhaya,  R.  Bhatia,  Z.  Shaikh,  Source based fake news classification using machine learning (Aug 2020).URL https://kaggle.com/ruchi798/source-based-news-classification

[4]. J. a. Moreno, G. Bressan, Factck.br:  A new dataset to study fake news,in:  Proceedings of the 25th Brazillian Symposium on Multimedia andthe  Web,  WebMedia  ’19,  Association  for  Computing  Machinery,  NewYork, NY, USA, 2019, p. 525–527.  doi:10.1145/3323503.3361698.

[5]. Monteiro R.A., Santos R.L.S., Pardo T.A.S., de Almeida T.A., Ruiz E.E.S., Vale O.A. (2018) Contributions to the Study of Fake News in Portuguese: New Corpus and Automatic Detection Results. In: Villavicencio A. et al. (eds) Computational Processing of the Portuguese Language. PROPOR 2018. Lecture Notes in Computer Science, vol 11122. Springer, Cham.