# [COM4513-6513] Assignment 2: Text Classification with a Feedforward Network

### Instructor: Nikos Aletras

The goal of this assignment is to develop a Feedforward network for text classification.

For that purpose, you will implement:

- Text processing methods for transforming raw text data into input vectors for your network  (**1 mark**)
- A Feedforward network consisting of:
    - **One-hot** input layer mapping words into an **Embedding weight matrix** (**1 mark**)
    - **One hidden layer** computing the mean embedding vector of all words in input followed by a **ReLU activation function** (**1 mark**)
    - **Output layer** with a **softmax** activation. (**1 mark**)
- The Stochastic Gradient Descent (SGD) algorithm with **back-propagation** to learn the weights of your Neural network. Your algorithm should:
    - Use (and minimise) the **Categorical Cross-entropy loss** function (**1 mark**)
    - Perform a **Forward pass** to compute intermediate outputs (**4 marks**)
    - Perform a **Backward pass** to compute gradients and update all sets of weights (**4 marks**)
    - Implement and use **Dropout** after each hidden layer for regularisation (**2 marks**)
- Discuss how did you choose hyperparameters? You can tune the learning rate (hint: choose small values), embedding size {e.g. 50, 300, 500}, the dropout rate {e.g. 0.2, 0.5} and the learning rate. Please use tables or graphs to show training and validation performance for each hyperparam combination  (**2 marks**).
- After training the model, plot the learning process (i.e. training and validation loss in each epoch) using a line plot and report accuracy.
- Re-train your network by using pre-trained embeddings ([GloVe](https://nlp.stanford.edu/projects/glove/)) trained on large corpora. Instead of randomly initialising the embedding weights matrix, you should initialise it with the pre-trained weights. During training, you should not update them (i.e. weight freezing) and backprop should stop before computing gradients for updating embedding weights. Report results by performing hyperparameter tuning and plotting the learning process. Do you get better performance? (**3 marks**).

- **BONUS:** Extend you Feedforward network by adding more hidden layers (e.g. one more). How does it affect the performance? Note: You need to repeat hyperparameter tuning, but the number of combinations grows exponentially. Therefore, you need to choose a subset of all possible combinations (**+2 extra marks**)

### Data

The data you will use for Task 2 is a subset of the [AG News Corpus](http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html) and you can find it in the `./data_topic` folder in CSV format:

- `data_topic/train.csv`: contains 2,400 news articles, 800 for each class to be used for training.
- `data_topic/dev.csv`: contains 150 news articles, 50 for each class to be used for hyperparameter selection and monitoring the training process.
- `data_topic/test.csv`: contains 900 news articles, 300 for each class to be used for testing.

### Pre-trained Embeddings

You can download pre-trained GloVe embeddings trained on Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download) from [here](http://nlp.stanford.edu/data/glove.840B.300d.zip). No need to unzip, the file is large.

### Save Memory

To save RAM, when you finish each experiment you can delete the weights of your network using `del W` followed by Python's garbage collector `gc.collect()`

### Submission Instructions

You should submit a Jupyter Notebook file (assignment2.ipynb) and an exported PDF version (you can do it from Jupyter: `File->Download as->PDF via Latex`).

You are advised to follow the code structure given in this notebook by completing all given funtions. You can also write any auxilliary/helper functions (and arguments for the functions) that you might need but note that you can provide a full solution without any such functions. Similarly, you can just use only the packages imported below but you are free to use any functionality from the [Python Standard Library](https://docs.python.org/2/library/index.html), NumPy, SciPy and Pandas. You are not allowed to use any third-party library such as Scikit-learn (apart from metric functions already provided), NLTK, Spacy, Keras etc.. You are allowed to re-use your code from Assignment 1.

Please make sure to comment your code. You should also mention if you've used Windows to write and test your code. There is no single correct answer on what your accuracy should be, but correct implementations usually achieve F1 of ~75-80% and ~85% without and with using pre-trained embeddings respectively.

This assignment will be marked out of 20. It is worth 20\% of your final grade in the module. If you implement the bonus question you can get up to 2 extra points but your final grade will be capped at 20.

The deadline for this assignment is **23:59 on Mon, 18 May 2020** and it needs to be submitted via Blackboard (MOLE). Standard departmental penalties for lateness will be applied. We use a range of strategies to detect [unfair means](https://www.sheffield.ac.uk/ssid/unfair-means/index), including Turnitin which helps detect plagiarism, so make sure you do not plagiarise.

```python
import pandas as pd
import numpy as np
from collections import Counter
from functools import partial
import re
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import random
from time import localtime, strftime
from scipy.stats import spearmanr,pearsonr
import zipfile
import gc

# fixing random seed for reproducibility
random.seed(123)
np.random.seed(123)
```

## Transform Raw texts into training and development data

First, you need to load the training, development and test sets from their corresponding CSV files (tip: you can use Pandas dataframes).

```python
def load_data(folder='topic'):
    """
        Load data from the folder and stor it in the dictionary

        Parameters
        ----------

        folder: {data_topic}
            name of the folder to extract the data from. (default is to get topic data)

    """
    names = ['text', 'label']
    if folder=='topic':
        names = names[::-1]
    all_data = {}
    for file in ['train', 'dev', 'test']:
        all_data[file] = pd.read_csv(f'data_{folder}/{file}.csv', names=names)
    return all_data

def get_raw_data(data_files):
    """
        Converts pandas data to lists

        Parameters
        ----------
        data_files: dict
            Dictionary containing pandas dataframes read from the data files
    """
    raw_data = {}
    for key, value in data_files.items():
        raw_data[f'X_{key}'] = value.text.values.tolist()
        raw_data[f'y_{key}'] = value.label.values
    return raw_data
```

```python
raw_data = get_raw_data(load_data())
```

# Create input representations

To train your Feedforward network, you first need to obtain input representations given a vocabulary. One-hot encoding requires large memory capacity. Therefore, we will instead represent documents as lists of vocabulary indices (each word corresponds to a vocabulary index).

## Text Pre-Processing Pipeline

To obtain a vocabulary of words. You should:
- tokenise all texts into a list of unigrams (tip: you can re-use the functions from Assignment 1)
- remove stop words (using the one provided or one of your preference)
- remove unigrams appearing in less than K documents
- use the remaining to create a vocabulary of the top-N most frequent unigrams in the entire corpus.

```python
stop_words = ['a','in','on','at','and','or',
              'to', 'the', 'of', 'an', 'by',
              'as', 'is', 'was', 'were', 'been', 'be',
              'are','for', 'this', 'that', 'these', 'those', 'you', 'i', 'if',
             'it', 'he', 'she', 'we', 'they', 'will', 'have', 'has',
              'do', 'did', 'can', 'could', 'who', 'which', 'what',
              'but', 'not', 'there', 'no', 'does', 'not', 'so', 've', 'their',
             'his', 'her', 'they', 'them', 'from', 'with', 'its']
```

### Unigram extraction from a document

You first need to implement the `extract_ngrams` function. It takes as input:
- `x_raw`: a string corresponding to the raw text of a document
- `ngram_range`: a tuple of two integers denoting the type of ngrams you want to extract, e.g. (1,2) denotes extracting unigrams and bigrams.
- `token_pattern`: a string to be used within a regular expression to extract all tokens. Note that data is already tokenised so you could opt for a simple white space tokenisation.
- `stop_words`: a list of stop words
- `vocab`: a given vocabulary. It should be used to extract specific features.

and returns:

- a list of all extracted features.

```python
def extract_ngrams(x_raw, ngram_range=(1,3), token_pattern=r'\b[A-Za-z][A-Za-z]+\b', stop_words=set(), vocab=set()):
    """
        Extracts ngram representation of the text data from a string

        Parameters
        ----------
        x_raw: str
            String representign a single text file

        ngram_range: tuple, optional
            range for ngrams to return (default is (1, 3) means unigrams, bigrams and trigrams will be returned)

        token_pattern: str, optional
            a partern for regex to split the string into tokens

        stop_words: set, optional
            tokens to get rid of (default is empty meaning all tokens will be kept)

        vocab: set, optional
            a set of ngrams that should be kept (default is empty returning all ngrams)
    """

    # Tokenise the text
    tokens = re.findall(token_pattern, x_raw.lower()) # lowercase to reducde number of unique tokens

    # Remove the stop words
    no_stop = [word for word in tokens if word not in stop_words]

    # Create ngrams
    if ngram_range[0] == 1:
        ngram_range = (ngram_range[0]+1, ngram_range[1])
        ngrams = no_stop.copy()
    else:
        ngrams = []

    for n in range(ngram_range[0], ngram_range[1]+1):
        ngrams += zip(*[no_stop[i:] for i in range(n)])

    # Keep only ngrams that are in vocab:
    if vocab:
        ngrams = [ngram for ngram in ngrams if ngram in vocab]

    return ngrams
```

### Create a vocabulary of n-grams

Then the `get_vocab` function will be used to (1) create a vocabulary of ngrams; (2) count the document frequencies of ngrams; (3) their raw frequency. It takes as input:
- `X_raw`: a list of strings each corresponding to the raw text of a document
- `ngram_range`: a tuple of two integers denoting the type of ngrams you want to extract, e.g. (1,2) denotes extracting unigrams and bigrams.
- `token_pattern`: a string to be used within a regular expression to extract all tokens. Note that data is already tokenised so you could opt for a simple white space tokenisation.
- `stop_words`: a list of stop words
- `min_df`: keep ngrams with a minimum document frequency.
- `keep_topN`: keep top-N more frequent ngrams.

and returns:

- `vocab`: a set of the n-grams that will be used as features.
- `df`: a Counter (or dict) that contains ngrams as keys and their corresponding document frequency as values.
- `ngram_counts`: counts of each ngram in vocab

```python
def get_vocab(X_raw, ngram_range=(1,3), token_pattern=r'\b[A-Za-z][A-Za-z]+\b',
              min_df=0, keep_topN=0, stop_words=set()):

    """
        Extracts the full vocabluary of ngrams from a list of strings (texts)

        Parameters
        ----------
        X_raw: list[dtype=str]
            String to extrac the vocabluary from

        ngram_range: tuple, optional
            range for ngrams to return (default is (1, 3) means unigrams, bigrams and trigrams will be returned)

        token_pattern: str, optional
            a partern for regex to split the string into tokens

        stop_words: set, optional
            tokens to get rid of (default is empty meaning all tokens will be kept)

        min_df: int, optional
            minimum document frequency to keep the ngram in the vocabluary (default is 0 to keep all ngrams)

        keep_topN: int, optinal
            keep only top number of occuring ngrams (default is 0 to keep all ngrams)
    """

    df = Counter()
    ngram_counts = Counter()

    # Extract the full vocabluary:
    for x in X_raw:
        ngrams = extract_ngrams(x, ngram_range=ngram_range, token_pattern=token_pattern, stop_words=stop_words)
        ngram_counts.update(ngrams)
        df.update(set(ngrams))

    # Keep only ngrams that occur in certain number of documents
    if min_df:
        df = Counter({k: v for k, v in df.items() if v >= min_df})
        ngram_counts = Counter({k: v for k, v in ngram_counts.items() if k in df.keys()})

    # Keep only ngrams that are common overall
    if keep_topN and keep_topN < len(df.keys()):
        ngram_counts = Counter(dict(ngram_counts.most_common()[:keep_topN]))
        df = Counter({k: v for k, v in df.items() if k in ngram_counts.keys()})

    return list(df.keys()), df, ngram_counts
```

Now you should use `get_vocab` to create your vocabulary and get document and raw frequencies of unigrams:

```python
vocab, df, ngram_counts = get_vocab(raw_data['X_train'], ngram_range=(1,1), stop_words=stop_words, keep_topN = 5000)
print(len(vocab))
print()
print(list(vocab)[:100])
print()
print(df.most_common()[:10])
```

    5000

    ['new', 'remove', 'hugo', 'wing', 'out', 'referendum', 'him', 'historic', 'vote', 'chavez', 'two', 'give', 'numbers', 'next', 'left', 'office', 'either', 'venezuelans', 'president', 'large', 'sunday', 'mandate', 'govern', 'reuters', 'years', 'turned', 'early', 'controversial', 'send', 'seoul', 'more', 'korean', 'water', 'police', 'troops', 'used', 'protesters', 'south', 'government', 'central', 'decision', 'urging', 'disperse', 'cannon', 'reverse', 'least', 'iraq', 'thousands', 'jails', 'said', 'care', 'minister', 'better', 'began', 'starved', 'death', 'conditions', 'israel', 'hunger', 'didn', 'israeli', 'security', 'palestinian', 'strike', 'prisoners', 'afp', 'overnight', 'violation', 'fragile', 'servicemen', 'disputed', 'georgian', 'wounding', 'gunfire', 'took', 'ossetia', 'ceasefire', 'seven', 'place', 'sporadic', 'region', 'shelling', 'armed', 'black', 'against', 'area', 'african', 'deployed', 'soldiers', 'flew', 'sudan', 'foreign', 'into', 'ap', 'troubled', 'farmers', 'killing', 'since', 'arab', 'militiamen']

    [('reuters', 631), ('said', 432), ('tuesday', 413), ('wednesday', 344), ('new', 325), ('after', 295), ('ap', 275), ('athens', 245), ('monday', 221), ('first', 210)]

Then, you need to create vocabulary id -> word and id -> word dictionaries for reference:

```python
id_word = dict(zip(range(len(vocab)), vocab))
word_id = dict(zip(vocab, range(len(vocab))))
```

### Convert the list of unigrams  into a list of vocabulary indices

Storing actual one-hot vectors into memory for all words in the entire data set is prohibitive. Instead, we will store word indices in the vocabulary and look-up the weight matrix. This is equivalent of doing a dot product between an one-hot vector and the weight matrix.

First, represent documents in train, dev and test sets as lists of words in the vocabulary:

```python
ngram_partial = partial(extract_ngrams, stop_words=stop_words, vocab=set(vocab))

ngram_data = {}
for key, value in raw_data.items():
    if key.startswith('X'):
        ngram_data[key] = list(map(ngram_partial, value))
    else:
        ngram_data[key] = value
```

```python
ngram_data['X_train'][0]
```

    ['reuters',
     'venezuelans',
     'turned',
     'out',
     'early',
     'large',
     'numbers',
     'sunday',
     'vote',
     'historic',
     'referendum',
     'either',
     'remove',
     'left',
     'wing',
     'president',
     'hugo',
     'chavez',
     'office',
     'give',
     'him',
     'new',
     'mandate',
     'govern',
     'next',
     'two',
     'years']

Then convert them into lists of indices in the vocabulary:

```python
word_to_id = lambda l : [word_id[w] for w in l]
for key, value in ngram_data.items():
    if key.startswith('X'):
        ngram_data[key] = list(map(word_to_id, ngram_data[key]))
```

```python
ngram_data['X_train'][1]
```

    [23,
     37,
     31,
     33,
     35,
     32,
     43,
     39,
     29,
     20,
     42,
     45,
     36,
     41,
     38,
     44,
     27,
     40,
     28,
     30,
     34,
     46]

Put the labels `Y` for train, dev and test sets into arrays:

#### Puting data extraction procedure together:

```python
def get_data(folder, ngram_range=(1,3), token_pattern=r'\b[A-Za-z][A-Za-z]+\b',
             min_df=0, keep_topN=0, stop_words=set()):
    """
        Combines data extraction procedure into a single method

        Parameters
        ----------
        folder: str
            name of the folder to extract the data from

        tfidf: bool, optional
            determines tfidf should be used or not (defaults to False)

        ngram_range: tuple, optional
            range for ngrams to return (default is (1, 3) means unigrams, bigrams and trigrams will be returned)

        token_pattern: str, optional
            a partern for regex to split the string into tokens

        stop_words: set, optional
            tokens to get rid of (default is empty meaning all tokens will be kept)

        min_df: int, optional
            minimum document frequency to keep the ngram in the vocabluary (default is 0 to keep all ngrams)

        keep_topN: int, optinal
            keep only top number of occuring ngrams (default is 0 to keep all ngrams)

        Returns
        -------
        data_dict: dict
            Dictionary with all data splits, vocabluary and word_id, id_word mappings.
    """
    data_dict = {}

    data_files = load_data(folder=folder)
    raw_data = get_raw_data(data_files)

    vocab, df, ngram_counts = get_vocab(raw_data['X_train'], ngram_range=ngram_range, stop_words=stop_words,
                                       token_pattern=token_pattern, min_df=min_df, keep_topN=keep_topN)

    id_word = dict(zip(range(len(vocab)), vocab))
    word_id = dict(zip(vocab, range(len(vocab))))

    ngram_partial = partial(extract_ngrams, stop_words=stop_words, vocab=set(vocab))
    word_to_id = lambda l : [word_id[w] for w in l]

    for key, value in raw_data.items():
        if key.startswith('X'):
            word_vec = list(map(ngram_partial, value))
            id_vec = list(map(word_to_id, word_vec))
            # Keep only non-empty documents:
            keep_idx, id_vec = map(list, zip(*[(i, x) for i, x in enumerate(id_vec) if x]))
            data_dict[key] = id_vec
        else:
            data_dict[key] = value[keep_idx]

    data_dict['id_word'] = id_word
    data_dict['word_id'] = word_id
    data_dict['vocab'] = np.array(vocab)

    return data_dict
```

# Network Architecture

Your network should pass each word index into its corresponding embedding by looking-up on the embedding matrix and then compute the first hidden layer $\mathbf{h}_1$:

$$\mathbf{h}_1 = \frac{1}{|x|}\sum_i W^e_i, i \in x$$

where $|x|$ is the number of words in the document and $W^e$ is an embedding matrix $|V|\times d$, $|V|$ is the size of the vocabulary and $d$ the embedding size.

Then $\mathbf{h}_1$ should be passed through a ReLU activation function:

$$\mathbf{a}_1 = relu(\mathbf{h}_1)$$

Finally the hidden layer is passed to the output layer:

$$\mathbf{y} = \text{softmax}(\mathbf{a}_1W^T) $$
where $W$ is a matrix $d \times |{\cal Y}|$, $|{\cal Y}|$ is the number of classes.

During training, $\mathbf{a}_1$ should be multiplied with a dropout mask vector (elementwise) for regularisation before it is passed to the output layer.

You can extend to a deeper architecture by passing a hidden layer to another one:

$$\mathbf{h_i} = \mathbf{a}_{i-1}W_i^T $$

$$\mathbf{a_i} = relu(\mathbf{h_i}) $$

# Network Training

First we need to define the parameters of our network by initiliasing the weight matrices. For that purpose, you should implement the `network_weights` function that takes as input:

- `vocab_size`: the size of the vocabulary
- `embedding_dim`: the size of the word embeddings
- `hidden_dim`: a list of the sizes of any subsequent hidden layers (for the Bonus). Empty if there are no hidden layers between the average embedding and the output layer
- `num_clusses`: the number of the classes for the output layer

and returns:

- `W`: a dictionary mapping from layer index (e.g. 0 for the embedding matrix) to the corresponding weight matrix initialised with small random numbers (hint: use numpy.random.uniform with from -0.1 to 0.1)

See the examples below for expected outputs. Make sure that the dimensionality of each weight matrix is compatible with the previous and next weight matrix, otherwise you won't be able to perform forward and backward passes. Consider also using np.float32 precision to save memory.

```python
def network_weights(vocab_size=1000, embedding_dim=300, hidden_dim=[], num_classes=3, init_val=0.1):
    W = {}

    dist = partial(np.random.uniform, -init_val, init_val)
    sizes = [vocab_size, embedding_dim] + hidden_dim + [num_classes]

    for i in range(len(sizes) - 1):
        W[i] = dist(size=(sizes[i], sizes[i+1])).astype(np.float32)

    return W
```

```python
W = network_weights(vocab_size=5,embedding_dim=10,hidden_dim=[], num_classes=2)

print('W_emb:', W[0].shape)
print('W_out:', W[1].shape)
```

    W_emb: (5, 10)
    W_out: (10, 2)

```python
W = network_weights(vocab_size=3,embedding_dim=4,hidden_dim=[2], num_classes=2)
```

```python
print('W_emb:', W[0].shape)
print('W_h1:', W[1].shape)
print('W_out:', W[2].shape)
```

    W_emb: (3, 4)
    W_h1: (4, 2)
    W_out: (2, 2)

```python
W[0]
```

    array([[-0.08085749,  0.07706536,  0.02544979,  0.04468327],
           [-0.09677416,  0.01888638,  0.01135704, -0.06820807],
           [-0.06938589,  0.03910591, -0.03624671,  0.03839406]],
          dtype=float32)

Then you need to develop a `softmax` function (same as in Assignment 1) to be used in the output layer. It takes as input:

- `z`: array of real numbers

and returns:

- `sig`: the softmax of `z`

```python
def softmax(z, axis=None):
    exp = np.exp(z)
    den = np.sum(exp, axis=axis)
    den = den.reshape(-1, 1) if axis else den
    return exp/den
```

Now you need to implement the categorical cross entropy loss by slightly modifying the function from Assignment 1 to depend only on the true label `y` and the class probabilities vector `y_preds`:

```python
def to_categorical(arr, num_classes):
    return np.eye(num_classes)[arr-1]
```

```python
def categorical_loss(y, y_pred):
    y_cat = to_categorical(y, y_pred.shape[-1])
    if np.isscalar(y_cat[0]):
        y_cat = np.array([y_cat])
    l = -y_cat * np.log(y_pred)
    return np.mean(np.sum(l, axis=1))
```

```python
def predict_class(prob):
    return np.argmax(prob, axis=1) + 1
```

```python
# example for 5 classes
y = 2 #true label
y_preds = softmax(np.array([[-2.1,1.,0.9,-1.3,1.5]]))

print('y_preds: ',y_preds)
print('loss:', categorical_loss(y, y_preds))
```

    y_preds:  [[0.01217919 0.27035308 0.24462558 0.02710529 0.44573687]]
    loss: 1.3080264848567502

Then, implement the `relu` function to introduce non-linearity after each hidden layer of your network (during the forward pass):

$$relu(z_i)= max(z_i,0)$$

and the `relu_derivative` function to compute its derivative (used in the backward pass):

\begin{equation}
  \text{relu_derivative}(z_i)=\begin{cases}
    0, & \text{if $z_i<=0$}.\\
    1, & \text{otherwise}.
  \end{cases}
\end{equation}

Note that both functions take as input a vector $z$

Hint use .copy() to avoid in place changes in array z

```python
def relu(z):
    return np.maximum(z, 0, z.copy())

def relu_derivative(z):
    return np.heaviside(z, 0)
```

During training you should also apply a dropout mask element-wise after the activation function (i.e. vector of ones with a random percentage set to zero). The `dropout_mask` function takes as input:

- `size`: the size of the vector that we want to apply dropout
- `dropout_rate`: the percentage of elements that will be randomly set to zeros

and returns:

- `dropout_vec`: a vector with binary values (0 or 1)

```python
def dropout_mask(size, dropout_rate):
    return np.random.binomial(1, 1-dropout_rate, size)
```

```python
print(dropout_mask(10, 0.2))
print(dropout_mask(10, 0.2))
```

    [1 1 1 1 1 1 1 1 1 1]
    [1 1 0 1 0 1 1 1 1 0]

Now you need to implement the `forward_pass` function that passes the input x through the network up to the output layer for computing the probability for each class using the weight matrices in `W`. The ReLU activation function should be applied on each hidden layer.

- `x`: a list of vocabulary indices each corresponding to a word in the document (input)
- `W`: a list of weight matrices connecting each part of the network, e.g. for a network with a hidden and an output layer: W[0] is the weight matrix that connects the input to the first hidden layer, W[1] is the weight matrix that connects the hidden layer to the output layer.
- `dropout_rate`: the dropout rate that is used to generate a random dropout mask vector applied after each hidden layer for regularisation.

and returns:

- `out_vals`: a dictionary of output values from each layer: h (the vector before the activation function), a (the resulting vector after passing h from the activation function), its dropout mask vector; and the prediction vector (probability for each class) from the output layer.

```python
def forward_pass(x, W, dropout_rate=0.2):

    out_vals = {}
    out_vals['h'] = []
    out_vals['a'] = []
    out_vals['dropout_vec'] = []

    # Ensure x is list of lists:
    if np.isscalar(x[0]):
        x = [x]
        axis = None  # Axis to perform softmax over
    else:
        axis = 1

    for i, w in W.items():
        if not i:  # Calculate embeding:
            x = np.array([np.sum(w[x_i], axis=0)/len(x_i) for x_i in x])
        else:
            x = np.dot(x, w)

        if i != (len(W) - 1):  # unless last layer
            out_vals['h'].append(x)
            x = relu(x)
            out_vals['a'].append(x)
            if dropout_rate:
                drop = dropout_mask(len(x[0]), dropout_rate)
                out_vals['dropout_vec'].append(drop)
                x *= drop
        else:  # Calculate output
            y = softmax(x, axis=axis)
            out_vals['y'] = y

    return out_vals
```

```python
W = network_weights(vocab_size=3, embedding_dim=4, hidden_dim=[5], num_classes=2)

for i in range(len(W)):
    print('Shape W'+str(i), W[i].shape)

print()
print(forward_pass([2,1], W, dropout_rate=0.1))
```

    Shape W0 (3, 4)
    Shape W1 (4, 5)
    Shape W2 (5, 2)

    {'h': [array([[-0.07048225,  0.05083556,  0.0807253 ,  0.01912668]],
          dtype=float32), array([[-0.00537986, -0.00211071,  0.00318677,  0.00552425, -0.00987258]],
          dtype=float32)], 'a': [array([[0.        , 0.05083556, 0.0807253 , 0.01912668]], dtype=float32), array([[0.        , 0.        , 0.00318677, 0.00552425, 0.        ]],
          dtype=float32)], 'dropout_vec': [array([1, 1, 1, 1]), array([1, 1, 1, 1, 1])], 'y': array([[0.4998965, 0.5001035]], dtype=float32)}

The `backward_pass` function computes the gradients and update the weights for each matrix in the network from the output to the input. It takes as input

- `x`: a list of vocabulary indices each corresponding to a word in the document (input)
- `y`: the true label
- `W`: a list of weight matrices connecting each part of the network, e.g. for a network with a hidden and an output layer: W[0] is the weight matrix that connects the input to the first hidden layer, W[1] is the weight matrix that connects the hidden layer to the output layer.
- `out_vals`: a dictionary of output values from a forward pass.
- `learning_rate`: the learning rate for updating the weights.
- `freeze_emb`: boolean value indicating whether the embedding weights will be updated.

and returns:

- `W`: the updated weights of the network.

Hint: the gradients on the output layer are similar to the multiclass logistic regression.

```python
def backward_pass(x, y, W, out_vals, lr=0.001, freeze_emb=False):

    dW = []  # Store weight updates

    y = to_categorical(y, out_vals['y'].shape[-1])

    # Compute outer layer gradient:
    grad = out_vals['y'] - y
    dW_out = np.dot(out_vals['a'][-1].T, grad)
    dW.append(dW_out)

    # Propagate back to the next layer
    grad = np.dot(grad, W[len(W)-1].T)

    for l in list(W.keys())[::-1][1:]:

        # Multiply by dropout applied durning training:
        if out_vals['dropout_vec']:
            grad = grad * out_vals['dropout_vec'][l]

        # Derivative with respect to activation:
        grad *= relu_derivative(out_vals['h'][l])

        if l: # Unless first layer
            # Compute hidden layer update
            dW_hid = np.dot(out_vals['a'][l-1].T, grad)
            dW.append(dW_hid)
            # Propagate back to the next layer
            grad = np.dot(grad, W[l].T)

        else:  # Update the embeding weights
            if not freeze_emb:
                count = Counter(x)
                # Only calculate gradient for activated inputs:
                dW_in = (np.array(list(count.values())).reshape(-1, 1) * grad) / len(x)
                # Can update straight away as W[0] won't be reused
                W[0][list(count.keys())] -= lr * dW_in

    # Update the weights:
    for i in range(len(dW)):
        W[i+1] -= lr * dW.pop()

    return W
```

Finally you need to modify SGD to support back-propagation by using the `forward_pass` and `backward_pass` functions.

The `SGD` function takes as input:

- `X_tr`: array of training data (vectors)
- `Y_tr`: labels of `X_tr`
- `W`: the weights of the network (dictionary)
- `X_dev`: array of development (i.e. validation) data (vectors)
- `Y_dev`: labels of `X_dev`
- `lr`: learning rate
- `dropout`: regularisation strength
- `epochs`: number of full passes over the training data
- `tolerance`: stop training if the difference between the current and previous validation loss is smaller than a threshold
- `freeze_emb`: boolean value indicating whether the embedding weights will be updated (to be used by the backward pass function).
- `print_progress`: flag for printing the training progress (train/validation loss)

and returns:

- `weights`: the weights learned
- `training_loss_history`: an array with the average losses of the whole training set after each epoch
- `validation_loss_history`: an array with the average losses of the whole development set after each epoch

```python
def SGD(X_tr, Y_tr, W, X_dev=[], Y_dev=[], lr=0.001,
        dropout=0.2, epochs=5, tolerance=0.001, freeze_emb=False, print_progress=True):
    """
        Performs Stochastic Gradient Descent optimization for classification task with a Neural Net

        Parameters
        ----------
        X_tr: np.array
            Training input data

        Y_tr: np.array
            Training labels (first dimension should be equivalent to X_tr)

        W: dict
            Initialised weights for each of the layers in Neural Network

        X_dev: np.array, optional
            Development input data for calculation of validation loss

        Y_dev: np.array, optional
            Development labels (only used if X_dev is suplied)

        dropout: float, (0, 1)
            Percentage of neurons to cut out between the hidden layers (regularization)

        lr: float, optional
            Learning rate to use durning training (default to optimal parameter determined durning hyperparameter optimization)

        epochs: int, optional
            Number of times to go over full dataset (defaults to optimal)

        tolerance: float, optional
            Difference between concurent development losses at which to stop the training (default to 0.0001)

        freeze_emb: bool, optional
            Determines if to freeze the first layer of the network durning training

        print_progress: bool, optional
            Determines if to print the updates for each epoch (default to True)
    """

    training_loss_history = []
    validation_loss_history = []

    n_points = len(X_tr)

    for epoch in range(epochs):

        rand_idx = np.random.permutation(n_points)

        # Peform the update
        for i in rand_idx:
            out_vals = forward_pass(X_tr[i], W, dropout_rate=dropout)
            W = backward_pass(X_tr[i], Y_tr[i], W, out_vals, lr=lr, freeze_emb=freeze_emb)

        # Calculate training loss for epoch
        tr_pred = forward_pass(X_tr, W, dropout_rate=0)['y']
        tr_loss = categorical_loss(Y_tr, tr_pred)
        training_loss_history.append(tr_loss)

        # Calcualte validation loss for epoch
        if len(X_dev):
            val_pred = forward_pass(X_dev, W, dropout_rate=0)['y']
            val_loss = categorical_loss(Y_dev, val_pred)
            validation_loss_history.append(val_loss)

        if print_progress:
            if len(X_dev):
                print(f'Epoch: {epoch}| Training Loss: {tr_loss:.3f}| Validation loss: {val_loss:.3f}')
            else:
                print(f'Epoch: {epoch}| Training Loss: {tr_loss:.3f}|')

        if epoch and len(X_dev) and tolerance:
            if (validation_loss_history[-2] - val_loss) < tolerance:
                break

    return W, training_loss_history, validation_loss_history, epoch+1
```

Now you are ready to train and evaluate you neural net. First, you need to define your network using the `network_weights` function followed by SGD with backprop:

```python
data = get_data('topic', ngram_range=(1,1), keep_topN=5000, stop_words=stop_words)
W = network_weights(vocab_size=len(data['vocab']), embedding_dim=300, hidden_dim=[], num_classes=3)

for i in range(len(W)):
    print('Shape W'+str(i), W[i].shape)

W, loss_tr, dev_loss, _ = SGD(data['X_train'], data['y_train'], W,
                           X_dev=data['X_dev'], Y_dev=data['y_dev'],
                           lr=0.005, epochs=150, dropout=0.4, tolerance=0.0001,
                           freeze_emb=False)
```

    Shape W0 (5000, 300)
    Shape W1 (300, 3)
    Epoch: 0| Training Loss: 1.098| Validation loss: 1.098
    Epoch: 1| Training Loss: 1.097| Validation loss: 1.098
    Epoch: 2| Training Loss: 1.096| Validation loss: 1.097
    Epoch: 3| Training Loss: 1.096| Validation loss: 1.097
    Epoch: 4| Training Loss: 1.095| Validation loss: 1.096
    Epoch: 5| Training Loss: 1.094| Validation loss: 1.096
    Epoch: 6| Training Loss: 1.093| Validation loss: 1.095
    Epoch: 7| Training Loss: 1.092| Validation loss: 1.095
    Epoch: 8| Training Loss: 1.091| Validation loss: 1.094
    Epoch: 9| Training Loss: 1.089| Validation loss: 1.093
    Epoch: 10| Training Loss: 1.088| Validation loss: 1.092
    Epoch: 11| Training Loss: 1.087| Validation loss: 1.092
    Epoch: 12| Training Loss: 1.085| Validation loss: 1.091
    Epoch: 13| Training Loss: 1.084| Validation loss: 1.090
    Epoch: 14| Training Loss: 1.082| Validation loss: 1.089
    Epoch: 15| Training Loss: 1.080| Validation loss: 1.088
    Epoch: 16| Training Loss: 1.078| Validation loss: 1.087
    Epoch: 17| Training Loss: 1.076| Validation loss: 1.085
    Epoch: 18| Training Loss: 1.074| Validation loss: 1.084
    Epoch: 19| Training Loss: 1.071| Validation loss: 1.082
    Epoch: 20| Training Loss: 1.068| Validation loss: 1.081
    Epoch: 21| Training Loss: 1.065| Validation loss: 1.079
    Epoch: 22| Training Loss: 1.061| Validation loss: 1.077
    Epoch: 23| Training Loss: 1.057| Validation loss: 1.074
    Epoch: 24| Training Loss: 1.053| Validation loss: 1.072
    Epoch: 25| Training Loss: 1.048| Validation loss: 1.069
    Epoch: 26| Training Loss: 1.042| Validation loss: 1.066
    Epoch: 27| Training Loss: 1.036| Validation loss: 1.062
    Epoch: 28| Training Loss: 1.030| Validation loss: 1.058
    Epoch: 29| Training Loss: 1.022| Validation loss: 1.054
    Epoch: 30| Training Loss: 1.014| Validation loss: 1.049
    Epoch: 31| Training Loss: 1.005| Validation loss: 1.044
    Epoch: 32| Training Loss: 0.996| Validation loss: 1.038
    Epoch: 33| Training Loss: 0.985| Validation loss: 1.031
    Epoch: 34| Training Loss: 0.973| Validation loss: 1.024
    Epoch: 35| Training Loss: 0.961| Validation loss: 1.016
    Epoch: 36| Training Loss: 0.947| Validation loss: 1.007
    Epoch: 37| Training Loss: 0.933| Validation loss: 0.998
    Epoch: 38| Training Loss: 0.917| Validation loss: 0.988
    Epoch: 39| Training Loss: 0.901| Validation loss: 0.976
    Epoch: 40| Training Loss: 0.884| Validation loss: 0.965
    Epoch: 41| Training Loss: 0.866| Validation loss: 0.952
    Epoch: 42| Training Loss: 0.848| Validation loss: 0.938
    Epoch: 43| Training Loss: 0.829| Validation loss: 0.924
    Epoch: 44| Training Loss: 0.810| Validation loss: 0.909
    Epoch: 45| Training Loss: 0.791| Validation loss: 0.893
    Epoch: 46| Training Loss: 0.772| Validation loss: 0.877
    Epoch: 47| Training Loss: 0.753| Validation loss: 0.861
    Epoch: 48| Training Loss: 0.734| Validation loss: 0.844
    Epoch: 49| Training Loss: 0.716| Validation loss: 0.827
    Epoch: 50| Training Loss: 0.698| Validation loss: 0.809
    Epoch: 51| Training Loss: 0.680| Validation loss: 0.792
    Epoch: 52| Training Loss: 0.663| Validation loss: 0.774
    Epoch: 53| Training Loss: 0.646| Validation loss: 0.757
    Epoch: 54| Training Loss: 0.630| Validation loss: 0.739
    Epoch: 55| Training Loss: 0.613| Validation loss: 0.721
    Epoch: 56| Training Loss: 0.597| Validation loss: 0.703
    Epoch: 57| Training Loss: 0.581| Validation loss: 0.685
    Epoch: 58| Training Loss: 0.566| Validation loss: 0.668
    Epoch: 59| Training Loss: 0.551| Validation loss: 0.650
    Epoch: 60| Training Loss: 0.535| Validation loss: 0.633
    Epoch: 61| Training Loss: 0.521| Validation loss: 0.616
    Epoch: 62| Training Loss: 0.506| Validation loss: 0.599
    Epoch: 63| Training Loss: 0.491| Validation loss: 0.582
    Epoch: 64| Training Loss: 0.477| Validation loss: 0.566
    Epoch: 65| Training Loss: 0.463| Validation loss: 0.550
    Epoch: 66| Training Loss: 0.450| Validation loss: 0.535
    Epoch: 67| Training Loss: 0.437| Validation loss: 0.520
    Epoch: 68| Training Loss: 0.424| Validation loss: 0.506
    Epoch: 69| Training Loss: 0.412| Validation loss: 0.492
    Epoch: 70| Training Loss: 0.400| Validation loss: 0.479
    Epoch: 71| Training Loss: 0.389| Validation loss: 0.467
    Epoch: 72| Training Loss: 0.378| Validation loss: 0.455
    Epoch: 73| Training Loss: 0.368| Validation loss: 0.444
    Epoch: 74| Training Loss: 0.358| Validation loss: 0.433
    Epoch: 75| Training Loss: 0.349| Validation loss: 0.423
    Epoch: 76| Training Loss: 0.340| Validation loss: 0.414
    Epoch: 77| Training Loss: 0.332| Validation loss: 0.405
    Epoch: 78| Training Loss: 0.324| Validation loss: 0.397
    Epoch: 79| Training Loss: 0.316| Validation loss: 0.389
    Epoch: 80| Training Loss: 0.309| Validation loss: 0.381
    Epoch: 81| Training Loss: 0.302| Validation loss: 0.374
    Epoch: 82| Training Loss: 0.295| Validation loss: 0.368
    Epoch: 83| Training Loss: 0.289| Validation loss: 0.362
    Epoch: 84| Training Loss: 0.283| Validation loss: 0.356
    Epoch: 85| Training Loss: 0.277| Validation loss: 0.350
    Epoch: 86| Training Loss: 0.271| Validation loss: 0.345
    Epoch: 87| Training Loss: 0.266| Validation loss: 0.340
    Epoch: 88| Training Loss: 0.261| Validation loss: 0.334
    Epoch: 89| Training Loss: 0.256| Validation loss: 0.330
    Epoch: 90| Training Loss: 0.251| Validation loss: 0.326
    Epoch: 91| Training Loss: 0.246| Validation loss: 0.322
    Epoch: 92| Training Loss: 0.242| Validation loss: 0.318
    Epoch: 93| Training Loss: 0.237| Validation loss: 0.314
    Epoch: 94| Training Loss: 0.233| Validation loss: 0.310
    Epoch: 95| Training Loss: 0.229| Validation loss: 0.307
    Epoch: 96| Training Loss: 0.225| Validation loss: 0.303
    Epoch: 97| Training Loss: 0.221| Validation loss: 0.300
    Epoch: 98| Training Loss: 0.217| Validation loss: 0.297
    Epoch: 99| Training Loss: 0.213| Validation loss: 0.294
    Epoch: 100| Training Loss: 0.209| Validation loss: 0.292
    Epoch: 101| Training Loss: 0.206| Validation loss: 0.289
    Epoch: 102| Training Loss: 0.202| Validation loss: 0.287
    Epoch: 103| Training Loss: 0.199| Validation loss: 0.284
    Epoch: 104| Training Loss: 0.195| Validation loss: 0.282
    Epoch: 105| Training Loss: 0.192| Validation loss: 0.280
    Epoch: 106| Training Loss: 0.188| Validation loss: 0.278
    Epoch: 107| Training Loss: 0.185| Validation loss: 0.276
    Epoch: 108| Training Loss: 0.182| Validation loss: 0.275
    Epoch: 109| Training Loss: 0.179| Validation loss: 0.273
    Epoch: 110| Training Loss: 0.176| Validation loss: 0.271
    Epoch: 111| Training Loss: 0.173| Validation loss: 0.270
    Epoch: 112| Training Loss: 0.170| Validation loss: 0.268
    Epoch: 113| Training Loss: 0.167| Validation loss: 0.266
    Epoch: 114| Training Loss: 0.164| Validation loss: 0.265
    Epoch: 115| Training Loss: 0.161| Validation loss: 0.264
    Epoch: 116| Training Loss: 0.159| Validation loss: 0.263
    Epoch: 117| Training Loss: 0.156| Validation loss: 0.261
    Epoch: 118| Training Loss: 0.154| Validation loss: 0.260
    Epoch: 119| Training Loss: 0.151| Validation loss: 0.259
    Epoch: 120| Training Loss: 0.149| Validation loss: 0.258
    Epoch: 121| Training Loss: 0.146| Validation loss: 0.257
    Epoch: 122| Training Loss: 0.144| Validation loss: 0.256
    Epoch: 123| Training Loss: 0.141| Validation loss: 0.256
    Epoch: 124| Training Loss: 0.139| Validation loss: 0.255
    Epoch: 125| Training Loss: 0.136| Validation loss: 0.254
    Epoch: 126| Training Loss: 0.134| Validation loss: 0.253
    Epoch: 127| Training Loss: 0.132| Validation loss: 0.252
    Epoch: 128| Training Loss: 0.130| Validation loss: 0.252
    Epoch: 129| Training Loss: 0.128| Validation loss: 0.251
    Epoch: 130| Training Loss: 0.125| Validation loss: 0.250
    Epoch: 131| Training Loss: 0.123| Validation loss: 0.250
    Epoch: 132| Training Loss: 0.122| Validation loss: 0.249
    Epoch: 133| Training Loss: 0.119| Validation loss: 0.249

Plot the learning process:

```python
def plot_results(losses, labels):
    for i, loss in enumerate(losses):
        plt.plot(loss, label=labels[i])
    plt.xlabel('Epoch')
    plt.ylabel('Binary Loss')
    plt.title('Loss trough Training')
    plt.legend()
    plt.draw()
```

```python
plot_results([loss_tr, dev_loss], ['count train', 'count dev'])
```

```python
print(f'Difference between final development and training loss is {dev_loss[-1] - loss_tr[-1]}')
```

    Difference between final development and training loss is 0.1296248079242911

Compute accuracy, precision, recall and F1-Score:

```python
def evaluation_metrics(actual, predicted, average='binary'):
    acc = accuracy_score(actual, predicted)
    prec = precision_score(actual, predicted, average=average)
    rec = recall_score(actual, predicted, average=average)
    f1 = f1_score(actual, predicted, average=average)

    return [acc, prec, rec, f1]
```

```python
pred = predict_class(forward_pass(data['X_test'], W, dropout_rate=0)['y'])
metrics = evaluation_metrics(data['y_test'], pred, average ='macro')

results = pd.DataFrame(data=metrics, index=['Accuracy', 'Precision', 'Recall', 'F1-Score'], columns=['Avr Embeding'])
results.round(3)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Avr Embeding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Accuracy</th>
      <td>0.864</td>
    </tr>
    <tr>
      <th>Precision</th>
      <td>0.867</td>
    </tr>
    <tr>
      <th>Recall</th>
      <td>0.864</td>
    </tr>
    <tr>
      <th>F1-Score</th>
      <td>0.864</td>
    </tr>
  </tbody>
</table>
</div>

### Discuss how did you choose model hyperparameters ?

```python
def optimize(data, lr_range, drop_range, layer_sizes, n_itter=100, hidden_range=[0], w_glove=None):

    columns = ['lr', 'epoch', 'dropout', 'embeding']
    for i in range(max(hidden_range)):
        columns.append(f'hidden_{i+1}')
    columns += ['dev_loss', 'F1']

    results = []

    # Print top row:
    str_len = []
    top_str = ''
    for c in columns:
        g = int((12 - len(c))/2)
        entry = '|' + g*' ' + str(c) + g*' '
        top_str += entry
        str_len.append(len(entry))
    print(top_str + '|')

    min_dev_loss = 1

    for i in range(n_itter):

        hidden = []

        lr = random.choice(lr_range)
        dropout = random.choice(drop_range)
        n_hidden = random.choice(hidden_range)

        if w_glove is None:
            embeding = random.choice(layer_sizes)
        else:
            embeding = 300

        for h in range(n_hidden):
            hidden.append(random.choice(layer_sizes))

        res = opt_run(data, lr, dropout, embeding, hidden, max(hidden_range), w_glove=w_glove)
        results.append(res)

        # Print out results for current itteration:
        if res[-2] < min_dev_loss:
            min_dev_loss = res[-2]
            res_str = '\x1b[95m'
            esc = '\x1b[0m'
        else:
            res_str = ''
            esc = ''
        for i, r in enumerate(res):
            v_s = f'{res[i]}'
            v_s = v_s if len(v_s) < 5 else v_s[:6]
            g = int(np.floor((12 - len(v_s))/2))
            res_str += '|' + g*' ' + v_s + (str_len[i]-g-len(v_s)-1)*' '
        print(res_str + '|' + esc)

    return pd.DataFrame(data=np.vstack(results), columns=columns)

def opt_run(data, lr, dropout, embeding, hidden, n_hidden, w_glove=None):

    W = network_weights(vocab_size=len(data['vocab']), embedding_dim=embeding, hidden_dim=hidden, num_classes=3)

    if w_glove is not None:
        W[0] = w_glove
        freeze_emb = True
    else:
        freeze_emb = False

    W, loss_tr, dev_loss, e = SGD(data['X_train'], data['y_train'], W,
                               X_dev=data['X_dev'], Y_dev=data['y_dev'],
                               lr=lr, epochs=200, dropout=dropout, tolerance=0.0001,
                               freeze_emb=freeze_emb, print_progress=False)

    pred = predict_class(forward_pass(data['X_dev'], W, dropout_rate=0)['y'])
    metrics = evaluation_metrics(data['y_dev'], pred, average ='macro')

    res = [lr, e, dropout, embeding]
    for h in range(n_hidden):
        try:
            res.append(hidden[h])
        except:
            res.append('-')
    res += [dev_loss[-1], metrics[-1]]

    return res
```

```python
lr_range = np.linspace(0.001, 0.01, 10)
drop_range = np.linspace(0.1, 0.5, 5)
layer_sizes = np.linspace(200, 400, 5).astype(int)

opt_results_ = optimize(data, lr_range, drop_range, layer_sizes, n_itter=100)
```

    |     lr     |   epoch   |  dropout  |  embeding  |  dev_loss  |     F1     |
    |   0.001    |    200    |   0.3000  |    350     |   0.8499   |   0.7771   |
    |   0.005    |    150    |    0.1    |    350     |   0.2627   |   0.9008   |
    |   0.0090   |     83    |    0.5    |    300     |   0.2515   |   0.9207   |
    |   0.001    |    200    |    0.2    |    300     |   0.7819   |   0.8403   |
    |   0.0090   |     78    |   0.3000  |    250     |   0.2437   |   0.9207   |
    |   0.001    |    200    |    0.4    |    400     |   0.9368   |   0.8013   |
    |   0.007    |    109    |    0.1    |    300     |   0.2520   |   0.9143   |
    |   0.008    |     89    |    0.1    |    200     |   0.2527   |   0.9008   |
    |   0.003    |    200    |    0.2    |    300     |   0.2660   |   0.9143   |
    |   0.007    |    106    |    0.5    |    300     |   0.2477   |   0.9207   |
    |   0.008    |     89    |    0.1    |    300     |   0.2498   |   0.9143   |
    |   0.0090   |     81    |    0.4    |    400     |   0.2584   |   0.9075   |
    |   0.0090   |     88    |    0.5    |    200     |   0.2383   |   0.9143   |
    |   0.007    |    108    |    0.5    |    400     |   0.2489   |   0.9075   |
    |    0.01    |     76    |    0.4    |    300     |   0.2401   |   0.9142   |
    |   0.0090   |     84    |    0.1    |    200     |   0.2492   |   0.9143   |
    |   0.008    |     93    |   0.3000  |    300     |   0.2518   |   0.9143   |
    |   0.006    |    137    |    0.5    |    200     |   0.2355   |   0.9079   |
    |   0.008    |     96    |    0.4    |    200     |   0.2458   |   0.9075   |
    |   0.004    |    194    |    0.5    |    300     |   0.2481   |   0.9139   |
    |   0.001    |    200    |   0.3000  |    350     |   0.8779   |   0.8201   |
    |   0.0090   |     86    |    0.5    |    300     |   0.2496   |   0.9207   |
    |   0.008    |     96    |    0.2    |    250     |   0.2494   |   0.9075   |
    |   0.003    |    200    |    0.5    |    350     |   0.2752   |   0.8951   |
    |   0.008    |    101    |    0.1    |    250     |   0.2477   |   0.9143   |
    |   0.008    |     83    |    0.2    |    300     |   0.2592   |   0.9079   |
    |   0.006    |    117    |   0.3000  |    350     |   0.2515   |   0.9143   |
    |   0.001    |    200    |    0.1    |    400     |   0.6657   |   0.9131   |
    |   0.006    |    123    |    0.1    |    350     |   0.2485   |   0.8944   |
    |   0.001    |     2     |    0.5    |    200     |   1.0996   |   0.2480   |
    |   0.0090   |     82    |    0.4    |    300     |   0.2436   |   0.9014   |
    |   0.005    |    143    |    0.4    |    400     |   0.2537   |   0.8946   |
    |   0.006    |    125    |    0.4    |    200     |   0.2406   |   0.9207   |
    |   0.003    |    200    |    0.4    |    250     |   0.2638   |   0.8951   |
    |   0.006    |    113    |    0.2    |    200     |   0.2460   |   0.9207   |
    |   0.0090   |     84    |    0.4    |    250     |   0.2448   |   0.9143   |
    |   0.005    |    140    |    0.4    |    250     |   0.2546   |   0.9015   |
    |   0.0090   |     76    |   0.3000  |    250     |   0.2471   |   0.9015   |
    |    0.01    |     74    |    0.4    |    200     |   0.2379   |   0.9143   |
    |   0.003    |    200    |    0.4    |    300     |   0.2488   |   0.9078   |
    |   0.008    |     96    |    0.4    |    350     |   0.2468   |   0.9207   |
    |   0.001    |    200    |    0.2    |    400     |   0.7506   |   0.8590   |
    |   0.0090   |     83    |    0.5    |    300     |   0.2550   |   0.9079   |
    |   0.0090   |     84    |    0.1    |    250     |   0.2657   |   0.9004   |
    |    0.01    |     73    |   0.3000  |    300     |   0.2514   |   0.9079   |
    |   0.0090   |     85    |    0.2    |    300     |   0.2524   |   0.9143   |
    |   0.005    |    146    |    0.4    |    400     |   0.2483   |   0.9078   |
    |   0.002    |    200    |    0.2    |    200     |   0.3279   |   0.9143   |
    |   0.004    |    164    |   0.3000  |    200     |   0.2501   |   0.9075   |
    |    0.01    |     75    |   0.3000  |    300     |   0.2471   |   0.9075   |
    |   0.0090   |     85    |    0.5    |    400     |   0.2466   |   0.9079   |
    |   0.003    |    200    |    0.4    |    350     |   0.2641   |   0.9143   |
    |   0.001    |    200    |   0.3000  |    300     |   0.8805   |   0.8118   |
    |   0.007    |     94    |    0.2    |    400     |   0.2600   |   0.9079   |
    |   0.001    |     4     |    0.5    |    300     |   1.0978   |   0.2658   |
    |    0.01    |     69    |    0.1    |    300     |   0.2565   |   0.9012   |
    |   0.008    |     93    |   0.3000  |    300     |   0.2559   |   0.9143   |
    |   0.006    |    115    |    0.2    |    300     |   0.2523   |   0.9079   |
    |   0.004    |    176    |    0.2    |    200     |   0.2437   |   0.9079   |
    |   0.007    |    103    |    0.4    |    250     |   0.2439   |   0.9207   |
    |   0.0090   |     84    |    0.4    |    200     |   0.2390   |   0.9143   |
    |   0.005    |    154    |    0.1    |    400     |   0.2542   |   0.9011   |
    |    0.01    |     76    |   0.3000  |    350     |   0.2496   |   0.9079   |
    |   0.001    |     2     |    0.5    |    300     |   1.0992   |   0.2142   |
    |   0.002    |    200    |    0.4    |    250     |   0.3886   |   0.9075   |
    |   0.008    |     93    |   0.3000  |    400     |   0.2417   |   0.9143   |
    |    0.01    |     70    |   0.3000  |    350     |   0.2472   |   0.9075   |
    |   0.0090   |     87    |    0.5    |    300     |   0.2494   |   0.9143   |
    |   0.001    |     2     |    0.2    |    200     |   1.0986   |   0.3049   |
    |   0.007    |    101    |    0.4    |    350     |   0.2474   |   0.9011   |
    |   0.0090   |     91    |    0.1    |    400     |   0.2544   |   0.9143   |
    |   0.003    |    200    |   0.3000  |    250     |   0.2470   |   0.9143   |
    |    0.01    |     84    |    0.1    |    400     |   0.2503   |   0.9140   |
    |    0.01    |     69    |   0.3000  |    300     |   0.2525   |   0.9140   |

```python
opt_results_.sort_values('dev_loss').head(3)
```

```python
plot_columns = ['lr', 'epoch', 'dropout', 'embeding']
for c in columns:
    opt_results_.plot.scatter(x=c, y='dev_loss', c='F1', colormap='Blues', sharex=False)
```

# Use Pre-trained Embeddings

Now re-train the network using GloVe pre-trained embeddings. You need to modify the `backward_pass` function above to stop computing gradients and updating weights of the embedding matrix.

Use the function below to obtain the embedding martix for your vocabulary.

```python
def get_glove_embeddings(f_zip, f_txt, word2id, emb_size=300):

    w_emb = np.zeros((len(word2id), emb_size))

    with zipfile.ZipFile(f_zip) as z:
        with z.open(f_txt) as f:
            for line in f:
                line = line.decode('utf-8')
                word = line.split()[0]

                if word in vocab:
                    emb = np.array(line.strip('\n').split()[1:]).astype(np.float32)
                    w_emb[word2id[word]] +=emb
    return w_emb
```

```python
w_glove = get_glove_embeddings("glove.840B.300d.zip","glove.840B.300d.txt", word_id)
```

First, initialise the weights of your network using the `network_weights` function. Second, replace the weigths of the embedding matrix with `w_glove`. Finally, train the network by freezing the embedding weights:

```python
W = network_weights(vocab_size=len(data['vocab']), embedding_dim=300, hidden_dim=[], num_classes=3)
W[0] = w_glove

W, loss_tr, dev_loss, _ = SGD(data['X_train'], data['y_train'], W,
                           X_dev=data['X_dev'], Y_dev=data['y_dev'],
                           lr=0.004, epochs=200, dropout=0.3, tolerance=0.0001,
                           freeze_emb=True)
```

```python
pred = predict_class(forward_pass(data['X_test'], W, dropout_rate=0)['y'])
metrics = evaluation_metrics(data['y_test'], pred, average ='macro')

results['Pre-Trained Embeding'] = metrics
results.round(3)
```

### Discuss how did you choose model hyperparameters ?

```python
lr_range = np.linspace(0.001, 0.005, 5)
drop_range = np.linspace(0.1, 0.5, 5)
layer_sizes = []

opt_results_ = optimize(data, lr_range, drop_range, layer_sizes, n_itter=100, w_glove=w_glove)
```

```python
opt_results_emb.sort_values('dev_loss').head(5)
```

```python
plot_columns = ['lr', 'epoch', 'dropout']
for c in columns:
    opt_results_emb.plot.scatter(x=c, y='dev_loss', c='F1', colormap='Blues', sharex=False)
```

# Extend to support deeper architectures (Bonus)

Extend the network to support back-propagation for more hidden layers. You need to modify the `backward_pass` function above to compute gradients and update the weights between intermediate hidden layers. Finally, train and evaluate a network with a deeper architecture.

```python
data = get_data('topic', ngram_range=(1,1), keep_topN=5000, stop_words=stop_words)
W = network_weights(vocab_size=len(data['vocab']), embedding_dim=300, hidden_dim=[300], num_classes=3)
W[0] = w_glove

W, loss_tr, dev_loss, _ = SGD(data['X_train'], data['y_train'], W,
                           X_dev=data['X_dev'], Y_dev=data['y_dev'],
                           lr=0.004, epochs=200, dropout=0.3, tolerance=0.0001,
                           freeze_emb=True)
```

```python
pred = predict_class(forward_pass(data['X_test'], W, dropout_rate=0)['y'])
metrics = evaluation_metrics(data['y_test'], pred, average ='macro')

results['Pre-trained Embeding + 1 Hidden'] = metrics
results.round(3)
```

```python
lr_range = np.linspace(0.0001, 0.0005, 5)
drop_range = np.linspace(0.1, 0.5, 5)
layer_sizes = np.linspace(200, 400, 5).astype(int)
hidden_range = [1, 2]

opt_results_emb = optimize(data, lr_range, drop_range, layer_sizes, n_itter=100, hidden_range=hidden_range, w_glove=w_glove)
```

## Full Results

Add your final results here:

| Model | Precision  | Recall  | F1-Score  | Accuracy
|:-:|:-:|:-:|:-:|:-:|
| Average Embedding  |   |   |   |   |
| Average Embedding (Pre-trained)  |   |   |   |   |
| Average Embedding (Pre-trained) + X hidden layers (BONUS)   |   |   |   |   |


# Physics 256
## Fast Fourier Transform

```python
import style
style._set_css_style('../include/bootstrap.css')
```

## Last Time

### [Notebook Link: 21 Spectral Analysis](./21_SpectralAnalysis.ipynb)

- Spectral analysis
- Discrete Fourier transform
- Nyquist frequency

## Today

- Fast Fourier transform

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
```

## Discrete Fourier Transform

For a discrete time series $\vec{y}$ measured at times $t_j = j \Delta t$, the discrete Fourier transform is given by:

\begin{equation}
\hat{y}_k  = \sum_{j=0}^{N-1} y_j \mathrm{e}^{-i 2\pi k j /N} .
\end{equation}

for $k=0,\ldots,N-1$.  The minimum non-zero frequency is $\omega_{\rm min} = 2\pi/(N\Delta t)$ while the maximum frequency is given by: $\omega_{\rm Ny} = \pi/\Delta t$.

Last time we considered a *brute force* implementation of $\hat{y}_k$ that is formally of order $\mathrm{O}(N^2)$.  We have to perform $N$ summations, each containint $N$ terms.  Historically, when computational abilities were more limited, this scaling prevented a **big** problem and better algorithms were needed.

```python
from scipy.constants import pi as π
def fourier_transform(t,y):
    '''Return the discrete Fourier transform of y.'''
    N = y.size
    Δt = t[1]-t[0]
    ŷ = np.zeros([N],dtype=complex)

    for k in range(N):
        ŷ[k] = np.sum(y*np.exp(-complex(0,1)*2*π*np.arange(N)*k/N))

    ω = 2*π*np.arange(N)/(N*Δt)

    return ω,ŷ
```

## Fast Fourier Transform

In 1965, Cooley and Tukey introduced an algorithm, now known as the *fast fourier transform* (FFT) which is of order: $\mathrm{O}(N\log_2N)$ *provided* that $N=2^n;\; n\in\mathbb{Z}$.  Deriving the algorithm is quite complicated, but all languages provide **black box** routines to return the FFT.  It can be evaluated in Python via `np.fft.fft`: https://docs.scipy.org/doc/numpy/reference/routines.fft.html

A nice video which includes a step-by-step derivation is below

```python
from IPython.display import YouTubeVideo
YouTubeVideo('htCj9exbGo0')
```

```python
def fast_fourier_transform(t,y):
    '''Return the fast Fourier transform of y.'''
    ŷ = np.fft.fft(y)
    ω = 2*π*np.fft.fftfreq(t.size,t[1]-t[0])
    return ω,ŷ
```

```python
# the time series
Δt = 0.1
t = np.arange(0.0,50.0,Δt)
ωₒ = 2.0*π*(0.2)
ϕ = 0.5*π
y = np.sin(ωₒ*t + ϕ) + np.sin(2*ωₒ*t)

# Plot the time series
fig,ax = plt.subplots(1,2, figsize=(8,4))
fig.subplots_adjust(wspace=0.4)

ax[0].plot(t,y)
ax[0].set_xlabel('Time [s]')
ax[0].set_ylabel('Amplitude [arb.]')

# # Get the discrete FT
ω,ŷ = fast_fourier_transform(t,y)

# Plot the real and imaginary parts of the DFT
ax[1].plot(ω,ŷ.real,label='real.', color=colors[0])
ax[1].plot(ω,ŷ.imag,linestyle='--',label='imag.', color=colors[-2])

ax[1].set_xlabel('Frequency [rad/s]')
ax[1].set_ylabel('Transform [arb.]')
#ax[1].set_xlim(0,np.max(ω))
ax[1].legend(loc='lower right')
```

np.fft.fft wraps frequencies above the Nyquist frequency to $\omega < 0$.

### Let's confirm the speedup factor

```python
import timeit
N = np.array([2**n for n in range(6,15)])
dft,fft = [],[]
for cN in N:
    t = np.linspace(0,20,cN)
    y = np.sin(t)

    dft.append(timeit.timeit('fourier_transform(t,y)', number=1, globals=globals()))
    fft.append(timeit.timeit('fast_fourier_transform(t,y)', number=1, globals=globals()))

dft = np.array(dft)
fft = np.array(fft)
```

### Try to fit the cuves

```python
from scipy.optimize import curve_fit

def N2(x,*a):
    '''N^2'''
    return a[0] + a[1]*x**2

def NlogN(x,*a):
    '''power function.'''
    return a[0] + a[1]*x*np.log2(x)

# perform the fits
a1,a1_cov = curve_fit(N2,N,dft,p0=(1,1))
a2,a2_cov = curve_fit(NlogN,N,fft,p0=(1,1))
```

### Plot the results

```python
# plot the timing results
plt.loglog(N,dft,'o', markersize=6, mec='None', mfc=colors[0], label='DFT')
plt.loglog(N,fft,'o', markersize=6, mec='None', mfc=colors[-2], label='FFT')

# Plot the fits
plt.loglog(N,N2(N,*a1),'-', color=colors[0], zorder=0, linewidth=1.5, label=r'$N^2$')
plt.loglog(N,NlogN(N,*a2),'-', color=colors[-2], zorder=0, linewidth=1.5, label=r'$N\log_2N$')

plt.xlim(50,20000)
plt.legend(loc='upper left')
plt.xlabel('N')
plt.ylabel('Execution Time [s]')
```

 Group Programming challenge

Revisit the driven dissipative non-linear pendulum and plot the power spectrum for $F_D = 0.5,1.2$.  What does this tell you about the onset/origin of chaos?

```python
from scipy.constants import g
from scipy.constants import pi as π
def euler(t,FD,ℓ,θₒ,ωₒ,γ,ΩD):
    ''' Semi-implicit Euler Method for the non-linear, dissipative, driven pendulum.'''

    Δt = t[1]-t[0]
    ω,θ = np.zeros_like(t),np.zeros_like(t)
    θ[0],ω[0] = θₒ,ωₒ

    # perform the numerical integration
    for n in range(t.size-1):
        ω[n+1] = ω[n] + (-(g/ℓ)*np.sin(θ[n]) - γ*ω[n] + FD*np.sin(ΩD*t[n]))*Δt
        θ[n+1] = θ[n] + ω[n+1]*Δt

        # keep theta in [-pi,pi)
        if θ[n+1] < -π: θ[n+1] += 2.0*π
        if θ[n+1] >= π: θ[n+1] -= 2.0*π

    return θ
```

### Setup initial conditions

```python
params = ℓ,θ0,ω0,γ,ΩD = g, 0.2, 0.0, 0.5, 2.0/3.0
FD = [0.5,1.2]
Δt = 0.01

# We determine the maximum time such that N = 2*n
pass

# setup the time array
t = np.arange(0.0,N*Δt,Δt)
```

### Perform a spectral analysis of θ(t) for the different values of driving force defined above

```python
pass
```

```python

```


# Tasas de Interés

## Librerías

```python
from finrisk import QC_Financial_3 as Qcf
from IPython.display import Image
from IPython.core.display import HTML
```

## Convenciones de Tasas

Consideremos la siguiente situación:

- El día 10-09-2020, nos conectamos al sitio web de un banco de la plaza para contratar un depósito a plazo en CLP, a 30 días, por 1 MM CLP.
- La página del banco nos ofrece una tasa de 0,11% para ese plazo.
- ¿Tenemos claro cuál es el negocio que nos están ofreciendo?

```python
Image(url="img/20200910_dap_online_0.png", width=500, height=400)
```

Al apretar el botón para **Continuar** se despliega el siguiente resultado:

Veamos como se llega a este resultado. Nuestra solicitud fue:

```python
monto_inicial = 1000000
plazo = 30
```

La pantalla inicial indica que la tasa para este plazo es:

```python
valor_tasa = .0011
```

¿Porqué el plazo de la operación simulada es de 33 días?

```python
fecha_inicial = Qcf.QCDate(10, 9, 2020)

fecha_final = fecha_inicial.add_days(plazo)

print(fecha_final.description(True))

print(fecha_final.week_day())
```

    10-10-2020
    SAT

Sumando 30 días al 10-09-2020 llegamos al 10-10-2020 que es un sábado (`SAT`). Para corregir esto, consideremos dentro del cálculo los días feriados.

```python
calendario = Qcf.BusinessCalendar(fecha_inicial, 1)

fecha_final = calendario.next_busy_day(fecha_final)

print(fecha_final.description(True))

print(fecha_final.week_day())
```

    12-10-2020
    MON

Vemos que, al pasar a la fecha hábil siguiente, llegamos al 12-10-2020. Sin embargo, la fecha final del depósito simulado es el 13-10-2020. El objeto calendario considera por default como feriados los días sábado y domingo, pero el lunes 12-10-2020 también es feriado y hay que agregarlo manualmente al calendario.

```python
calendario.add_holiday(fecha_final) # Se agrega el 12-10-2020 como feriado al calendario.

fecha_final = calendario.next_busy_day(fecha_final) # Se recalcula la fecha final.

print(f'Fecha final: {fecha_final.description(True)}.')

print(f'Día final: {fecha_final.week_day()}.')

plazo_op = fecha_inicial.day_diff(fecha_final)

print(f'Plazo: {plazo_op} días.')
```

    Fecha final: 13-10-2020.
    Día final: TUE.
    Plazo: 33 días.

¿Cómo se calcula el interés? La convención de tasa en Chile para depósitos a plazo en CLP es Lineal Act/30.

```python
yf = Qcf.QCAct30() # Act/30

wf = Qcf.QCLinearWf() # Lineal

tasa = Qcf.QCInterestRate(valor_tasa, yf, wf) # Tasa con ese valor y convención
```

Así, el monto final se calcula como:

```python
monto_final = monto_inicial * tasa.wf(fecha_inicial, fecha_final) # wf: factor de capitalización

print(f'Monto final: {monto_final:,.0f}')
```

    Monto final: 1,001,210

Veamos explícitamente este cálculo:

```python
print(f'Monto final: {monto_inicial * (1 + valor_tasa * plazo_op / 30): ,.0f}')
```

    Monto final:  1,001,210

El monto final, entonces, se calcula utilizando la siguiente fórmula:

$$
\begin{equation}
M_{final}=M_{inicial}\cdot\left(1+r\cdot\frac{fecha_{final}-fecha_{inicial}}{30}\right)
\end{equation}
$$

donde:

- $M_{inicial}$ es el monto inicial del depósito.
- $M_{final}$ es el monto final del depósito.
- $r$ es el valor de la tasa ofrecida por el banco.
- $fecha_{final}-fecha_{inicial}$ representa la diferencia real en días entre las fechas inicial y final del depósito.

¿Porqué se usa la fórmula anterior?

Por convención:

- en Chile las tasas de los depósitos a plazo en CLP se cotizan de esta forma.
- La fórmula tiene distintas componentes, las cuales, en su conjunto, definen el tipo de tasa de interés asociado a los depósitos en CLP.

Abstrayendo la estructura de la fórmula (1) podemos escribir:

$$
\begin{equation}
M_{final}=M_{inicial}\cdot wf(valorTasa, yf(fecha_{inicial},fecha_{final}))
\end{equation}
$$

Donde:

- $yf$ es la función *fracción de año* que determina el período de tiempo que transcurre entre ambas fechas y
- $wf$ es la función *factor de capitalización* que establece el factor a aplicar al monto inicial de la inversión para obtener el monto final de la inversión.

Para el cálculo de la fracción de año, existen distintas maneras de contar los días:

- Actual (días reales o la diferencia de fechas que acabamos de ver)
- En base 30 (todos los meses tienen 30 días)

y distintas maneras de representar un *período* de tasa:

- Base 30 (meses)
- Año de 360 días
- Año de 365 días

También hay distintas maneras de calcular el factor de capitalización, entre ellas:

- Lineal:

$$
\begin{equation}
1+r_{lineal}\cdot yf(fecha_{inicial},fecha_{final})
\end{equation}
$$

- Compuesto

$$
\begin{equation}
\left(1+r_{compuesto}\right)^{yf(fecha_{inicial},fecha_{final})}
\end{equation}
$$

- Exponencial
$$
\begin{equation}
\exp\left(r_{exponencial}\cdot yf\left(fecha_{inicial},fecha_{final}\right)\right)
\end{equation}
$$

Plazo de la Tasa:

- En Chile se suele hablar de la tasa a 30 días, 90 días, 180 días etc. Como vimos en el ejemplo anterior, no todos los días podemos tener una tasa a 30 días (por el vencimiento en día inhábil).
- Esto hace difícil la comparación en el tiempo de las tasas a un mismo plazo (de hecho no todos los días podemos tener tasa a un cierto plazo).
- En los mercados desarrollados, esto se resuelve a través del concepto de plazo estándar o Tenor.

Tenor

- Por ejemplo, en vez de tasa a 30 días se habla de la tasa a 1 mes (1M) que significa, por ejemplo (es una convención de mercado), que la tasa es al mismo día de hoy, pero del mes siguiente.
- Si ese día es inhábil se adopta una convención que puede ser ir al próximo día hábil. Así tenemos también 3M, 6M, 1Y (un año) etc.

```python
un_mes = Qcf.Tenor('1M')
print(f'Un mes sin corrección: {fecha_inicial.add_months(un_mes.get_months()).description(True)}')

temp = calendario.next_busy_day(fecha_inicial.add_months(un_mes.get_months()))
print(f'Un mes con corrección al día hábil siguiente: {temp.description(True)}')
```

    Un mes sin corrección: 10-10-2020
    Un mes con corrección al día hábil siguiente: 13-10-2020

Consideremos la siguiente inversión.

- Fecha inicial: 10-09-2020
- Plazo de la inversión: 1Y (corrección al día hábil siguiente).
- En la fecha final recibo 10 CLP por cada 100 CLP invertidos.
- Dado el plazo de la inversión, ésta queda definida, por su factor de capitalización $\frac{110}{100}=1.1$.

## Ejercicio

Utilizando la librería `QC_Financial_3` replicar los valores de la siguiente tabla:

```python
Image(url="img/tabla_ejercicio.png", width=550, height=440)
```


# Probabilistic Programming 3: Hidden Markov modeling
## Monte Carlo sampling

In this notebook, we will be looking at forecasting a time-series. To be precise, we will implement a Hidden Markov Model using Turing.

### Preliminaries

- Goal
  - Learn to apply Turing to a time-series data set.
- Materials
  - Mandatory
    - These lecture notes.
  - Optional
    - [Tutorial on Bayesian HMM using Turing.jl](https://turing.ml/dev/tutorials/4-bayeshmm/)
    - [Probabilistic Programming notebook](https://github.com/bertdv/BMLIP/tree/master/lessons/notebooks/probprog/Probabilistic-Programming.ipynb)
    - [Cheatsheets: how does Julia differ from Matlab / Python](https://docs.julialang.org/en/v1/manual/noteworthy-differences/index.html).

```julia
# Package managing
using Pkg
Pkg.activate("workspace")
Pkg.instantiate()

include("../scripts/pp-3.jl")

using Logging; disable_logging(LogLevel(0))
using Distributions
using StatsPlots
using MCMCChains
using Turing
Turing.setadbackend(:forward_diff);
```

## Data generation

In this problem setting, data is generated in the form of states over time. For example, proteins that fold into one of several functional configurations or machine translation, where one observes writing in one language and aims to infer writing in another language (both being finite sets of symbols). We don't have perfect measurement tools and have to assume measurement noise.

We will generate a synthetic data set as follows: we pick an initial state and evolve the state using a fixed state transition matrix. Then we observe the state using a fixed observation probability, referred to as an _emission_ matrix.

```julia
# There are 3 possible states and each variable is in one of those (one-hot encoding)
K = 3

# Length of time-series
T = 50

# Transition matrix of latent variables
transition = [0.3 0.6 0.1;
              0.5 0.2 0.3;
              0.2 0.7 0.1]

# Emission matrix for observed variables
emission = [0.9 0.1 0.0;
            0.1 0.8 0.1;
            0.0 0.2 0.8]

# Preallocate data arrays
states = Array{Int64,1}(undef, T)
observations = Array{Int64,1}(undef, T)

# Initial state
state_0 = 1

# Keep previous state in memory
state_tmin = state_0

# Generate data for entire time-series
for t = 1:T

    # Transition from previous state
    states[t] = rand(Categorical(transition[state_tmin, :]), 1)[1]

    # Emission of current state
    observations[t] = rand(Categorical(emission[states[t], :]), 1)[1]

    # Update previous state
    state_tmin = states[t]

end

# Visualization.
plot(1:T, states, color="red", label="states", ylim=(0, 5), grid=false)
scatter!(1:T, observations, color="blue", label="observations", xlabel="time (t)")
```

## Model specification

The Hidden Markov model is a state-space model of the form:

$$\begin{align}
x_t \sim&\ \text{Cat}(A x_{t-1}) \\
y_t \sim&\ \text{Cat}(B x_t)
\end{align}$$

where $A$ is the state transition matrix and $B$ is the emission matrix. The states $x$ are indexed by time $t$, ranging from $0$ to $T$. The observations $y$ are indexed from $1$ to $T$.

Again, we start simple and slowly improve the model.

### Model 1: estimate states

We will first assume we know the transition and emission matrices. Based on these, we purely want to estimate states from observations.

```julia
# Turing model definition.
@model hidden_markov_model(y, K, A, B) = begin

    # Time series length
    T = length(y)

    # State sequence.
    x = Vector{Int}(undef, T)

    # Define initial state
    x_0 ~ Categorical(ones(K,)./K)

    # Initialize "previous state" variable
    x_tmin = x_0

    # Loop over entire sequence
    for t = 1:T

        # State transition
        x[t] ~ Categorical(A[x_tmin, :])

        # Observation emission
        y[t] ~ Categorical(B[x[t], :])

        # Update previous state
        x_tmin = x[t]
    end
end;

# Call instance of the model
model1 = hidden_markov_model(observations, K, transition, emission);
```

The Categorical distribution is a known parametric distribution, which we can marginalize. It is therefore suited to Gibbs sampling.

```julia
# Length of chain
len_chain = 500

# Define sampler
sampler1 = Gibbs(PG(50, :x))

# Call sampler
chain1 = sample(model1, sampler1, len_chain);
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:03:51

Let's inspect the chain again. Note that $x$ is an array of $T$ latent variables, so this would extend for a quite bit.Instead, we're going to inspect the last state.

```julia
plot(chain1["x["*string(T)*"]"])
```

Since the state space is discrete, we cannot take the sample average of the chain to obtain estimates. The sample average is continuous, after all. To obtain discrete estimates, we take the median of the chain.

```julia
# Extract mean of chain
x_estimated = median_chain(chain1, :x)

# Visualization
plot(1:T, states, color="red", label="states", ylim=(0, 5), grid=false)
plot!(1:T, x_estimated, color="green", label="inferred")
scatter!(1:T, observations, color="blue", label="observations", xlabel="time (t)")
```

### Model 2: transition matrix estimation

At the moment, we assume we know how the states evolve over time. This is not an unreasonable assumption; in protein folding we know exactly _how_ the protein changes state, i.e. through the signaling of another molecule, and we know the concentration of signal molecules. However, in other applications we do not necessarily know the transition matrix.

We can estimate the transition matrix by letting the coefficients be latent variables and posing a prior.

```julia
# Turing model definition.
@model hidden_markov_model(y, K, B) = begin

    # Time series length.
    T = length(y)

    # State sequence.
    x = Vector{Int}(undef, T)

    # Define initial state
    x_0 ~ Categorical(ones(K,)./K)

    # Initialize "previous state" variable
    x_tmin = x_0

    # Preallocate transition matrix
    A = Vector{Vector}(undef, K)

    # Loop over rows of the transition matrix
    for k = 1:K

        # Loose prior for state transitions
        A[k] ~ Dirichlet(ones(K))
    end

    # Loop over entire sequence
    for t = 1:T

        # State transition
        x[t] ~ Categorical(A[x_tmin])

        # Observation emission
        y[t] ~ Categorical(B[x[t], :])

        # Update previous state
        x_tmin = x[t]
    end
end;

# Call instance of the model
model2 = hidden_markov_model(observations, K, emission);
```

We will extend the current sampler with a Hamiltonian MC for the transition matrix.

```julia
# Length of chain
len_chain = 500

# Define sampler
sampler2 = Gibbs(PG(50, :x), HMC(0.1, 5, :A))

# Call sampler
chain2 = sample(model2, sampler2, len_chain);
```

    ┌ Warning: The current proposal will be rejected due to numerical error(s).
    │   isfinite.((θ, r, ℓπ, ℓκ)) = (true, true, false, true)
    └ @ AdvancedHMC /home/wmkouw/.julia/packages/AdvancedHMC/haUrH/src/hamiltonian.jl:44
    ┌ Warning: The current proposal will be rejected due to numerical error(s).
    │   isfinite.((θ, r, ℓπ, ℓκ)) = (true, true, false, true)
    └ @ AdvancedHMC /home/wmkouw/.julia/packages/AdvancedHMC/haUrH/src/hamiltonian.jl:44
    ┌ Warning: The current proposal will be rejected due to numerical error(s).
    │   isfinite.((θ, r, ℓπ, ℓκ)) = (true, true, false, true)
    └ @ AdvancedHMC /home/wmkouw/.julia/packages/AdvancedHMC/haUrH/src/hamiltonian.jl:44
    ┌ Warning: The current proposal will be rejected due to numerical error(s).
    │   isfinite.((θ, r, ℓπ, ℓκ)) = (true, true, false, false)
    └ @ AdvancedHMC /home/wmkouw/.julia/packages/AdvancedHMC/haUrH/src/hamiltonian.jl:44
    Progress: 100%|█████████████████████████████████████████| Time: 0:04:08

```julia
plot(chain2[:A])
```

```julia
# Extract mean of chain
x_estimated = median_chain(chain2, :x)

# Visualization
plot(1:T, states, color="red", label="states", ylim=(0, 5), grid=false)
plot!(1:T, x_estimated, color="green", label="inferred")
scatter!(1:T, observations, color="blue", label="observations", xlabel="time (t)")
```

Let's visualize the estimated transition matrix.

```julia
# Extract state estimates
transition_estimated = mean(chain2[:A].value.data[:,:,1], dims=1)

# Visualization
l = @layout [a b]
h1 = bar(0:9, reshape(transition, (9,)), label="true", ylims=[0., 1.])
h2 = bar(reshape(transition_estimated, (9,)), label="estimated", ylims=[0., 1.])
plot(h1, h2, layout=l, size=(1000,500))
```

### Model 3: states, transition and emission

We can extend the model further to also estimate the emission matrix.

```julia
@model hidden_markov_model(y, K) = begin

    # Time series length.
    T = length(y)

    # State sequence.
    x = Vector{Int}(undef, T)

    # Define initial state
    x_0 ~ Categorical(ones(K,)./K)

    # Initialize "previous state" variable
    x_tmin = x_0

    # Preallocate transition and emission matrices
    A = Vector{Vector}(undef, K)
    B = Vector{Vector}(undef, K)

    # Loop over rows of the transition and emission matrices
    for k = 1:K

        # Loose prior for state transitions
        A[k] ~ Dirichlet(ones(K))

        # Sharper prior for emissions
        B[k] ~ Dirichlet(10*one_hot(k,K) .+ 1)
    end

    # Loop over entire sequence
    for t = 1:T

        # State transition
        x[t] ~ Categorical(A[x_tmin])

        # Observation emission
        y[t] ~ Categorical(B[x[t]])

        # Update previous state
        x_tmin = x[t]

    end
end;

# Call instance of the model
model3 = hidden_markov_model(observations, K);
```

```julia
# Length of chain
len_chain = 500

# Define sampler
sampler3 = Gibbs(PG(50, :x), HMC(0.001, 2, :A, :B))

# Call sampler
chain3 = sample(model3, sampler3, len_chain);
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:03:50

```julia
plot(chain3[:B])
```

```julia
# Extract mean of chain
x_estimated = median_chain(chain3, :x)

# Visualization
plot(1:T, states, color="red", label="states", ylim=(0, 5), grid=false)
plot!(1:T, x_estimated, color="green", label="inferred")
scatter!(1:T, observations, color="blue", label="observations", xlabel="time (t)")
```

```julia
# Extract state estimates
transition_estimated = mean(chain3[:A].value.data[:,:,1], dims=1)

# Visualization
l = @layout [a b]
h1 = bar(0:9, reshape(transition, (9,)), label="true", ylims=[0., 1.])
h2 = bar(reshape(transition_estimated, (9,)), label="estimated", ylims=[0., 1.])
plot(h1, h2, layout=l, size=(1000,500))
```

```julia
# Extract emission estimates
emission_estimated = mean(chain3[:B].value.data[:,:,1], dims=1)

# Visualization
l = @layout [a b]
h1 = bar(0:9, reshape(emission, (9,)), label="true", ylims=[0., 1.])
h2 = bar(reshape(emission_estimated, (9,)), label="estimated", ylims=[0., 1.])
plot(h1, h2, layout=l, size=(1000,500))
```

### Assignment: priors

Note that it becomes harder to identify whether an observation was noisily generated from one state or precisely generated from another state. Here, the prior can help: if we have some idea of how noisy the sensor is, but not exact enough to _fix_ the emission matrix to particular values, then we can still favor certain solutions over others.

Play around with the prior parameters of the state transition and emission matrices. How does the state estimation depend on the priors? Is this model more sensitive to the choice of priors than the mixture model or regression models?

```julia
### Your code here
```

### Optional assignment: continuous observations

We can extend the problem setting to one where the states are discrete, but the observations are continuous. This type of problem occurs often. Think for example of speech recognition: each utterance belongs to a discrete finite set of syllables. The generative model is formulated as follows:

$$\begin{align}
x_t \sim&\ \text{Cat}(A x_{t-1}) \\
y_t \sim&\ \mathcal{N}(x_t, \sigma)
\end{align}$$

with the customary $x_0 \sim \text{Cat}(1/K)$. We've provided the data generation setup below:

```julia
# There are 3 possible states and each variable is in one of those (one-hot encoding)
K = 3

# Length of time-series
T = 50

# Transition matrix of latent variables
transition = [0.3 0.6 0.1;
              0.5 0.2 0.3;
              0.2 0.7 0.1]

# Measurement noise for observations y
measurement_noise = 1/2

# Preallocate data arrays
states = Array{Int64,1}(undef, T)
observations = Array{Float64,1}(undef, T)

# Initial state
state_0 = 1

# Keep previous state in memory
state_tmin = state_0

# Generate data for entire time-series
for t = 1:T

    # Transition from previous state
    states[t] = rand(Categorical(transition[state_tmin, :]), 1)[1]

    # Emission of current state
    observations[t] = rand(Normal(states[t], measurement_noise), 1)[1]

    # Update previous state
    state_tmin = states[t]

end

# Visualization.
plot(1:T, states, color="red", label="states", ylim=(0, 5), grid=false)
scatter!(1:T, observations, color="blue", label="observations", xlabel="time (t)")
```

Can you change the model such that it infers the discrete states from continuous observations?

```julia
### Your code here
```