```python
%run ../../common/import_all.py

from common.setup_notebook import set_css_style, config_ipython
config_ipython()
set_css_style()
```

# Probabilistic Latent Semantic Analysis

## What is and how it works

PLSA, also called PLSI (I for "indexing"), is a probabilistic version of LSA which models the co-occurrence of words and documents as a mixture decomposition; it came out in [[1]](#1). Note that in the following, we will use the words *concept* and *topic* interchangeably.

Given word $w$ and document $d$, the probability of co-occurrence (as, the probability that $w$ occurs in $d$) is given by a mixture of conditionally independent multinomial distributions. Given latent categories (topics) $c$, we write indeed:

$$
\begin{align}
    P(w, d) &= \sum_c P(c) P(d | c) P(w | c) \\
            &= P(d) \sum_c P(c | d) P(w | c)
\end{align}
$$

In the fist formulation, $w$ and $d$ are thought as generated from the latent class $c$ in similar ways by using the conditional probabilities $P(d|c)$ and $P(w|c)$. In the second formulation instead, $\forall d$, a latent class is chosen conditionally to it according to the probability $P(c|d)$ and a word is generated from that class following $P(w|c)$. The number of parameters is $c(w+d)$, so it grows linearly with the number of documents. Said parameters are estimated using the [EM algorithm](../../prob-stats/methods/em.ipynb).

## Concepts, documents, words and queries

PLSA is an improved variation of LSA in the sense that

* documents might not contain similar terms but still refer to the same concept
* queries can contain words not present in a document and still be very relevant to the document

This is why is uses the probability of a word, or full query $q$ given the context ($r \in \{0, 1\}$ is the relevance of the document):

$$
\begin{align}
    P(r=1 | q) &= \frac{P(q | r=1) P(r=1)}{P(q)} \\
               &\propto P(q|r=1) P(r=1)
\end{align}
$$

where in the last writing we have $P(q|r=1)$, telling us given a document, how probable is the query, and $P(r=1)$, which can be uniform or dependent on the popularity of the document.

$P(q | r=1)$ is calculated as:

* $\forall d$, compute the probability of each word $w$ to be relevant for it $P(w | r=1)$
* compute the conditional probability of words in $q$

The model relies on the idea that each concept has a distribution over words, each document is a mixture of concepts and each word is drawn from the topics. We will have, according to the equation above,

$$
\begin{align}
    P(w, d) &= P(d) \sum_c P(w | c) P(c | d) \\
            &= \sum_c P(d|c) P(c) P(w|c)
\end{align}
$$

and this is a factorisation of the original matrix into three matrices (hence the relation to LSA):

* a matrix $U$ which maps documents to concepts
* a matrix $\Sigma$ which contains the concepts
* a matrix $V$ which maps concepts to words

But, differently from an SVD factorisation used in LSA, here there is no orthonormality condition for $U$ and $V$ and their elements are non-negative because they are probabilities.

Now, we need to find all parameters such that the probability of observing the corpus is maximised. Using a [MLE](../../prob-stats/methods/mle.ipynb) approach, we need to maximise the likelihood

$$
\mathcal{L} = \Pi_{i=1}^n \Pi_{j=1}^m P(w_j, d_i)^{n(w_j, d_i)} \ ,
$$

where the exponent gives the multiplicity of $w_j$ in $d_i$, as in, their count. So, computing the log,

$$
\begin{align}
    \log \mathcal{L} &= \sum_{i=1}^n \sum_{j=1}^m n(w_j, d_i) \log P(w_j, d_i) \\
                     &= \sum_{i=1}^n \sum_{j=1}^m n(w_j, d_i) \log \left[ \sum_{k=1}^K P(d_i) P(c_k|d_i) P(w_j|c_k) \right] \\
                     &= \sum_{i=1}^n n(d_i) \sum_{j=1}^m \frac{n(w_j, d_i)}{n(d_i)} \log \left[\sum_{k=1}^K P(d_i) P(c_k|d_i) P(w_j|c_k) \right] \\
                     &= \sum_{i=1}^n n(d_i) \sum_{j=1}^m \frac{n(w_j, d_i)}{n(d_i)} \left( \log P(d_i) + \log \left[ \sum_k P(c_k| d_i) P(w_j|c_k) \right]  \right) \\
                     &= \sum_{i=1}^n n(d_i) \left[ \sum_{j=1}^m \frac{n(w_j, d_i)}{n(d_i)} \log P(d_i) + \sum_{j=1}^m \frac{n(w_j, d_i)}{n(d_i)} \log \sum_k P(c_k | d_i) P(w_j | c_k) \right] \\
                     &= \sum_{i=1}^n n(d_i) \left( \log P(d_i) + \sum_{j=1}^m \frac{n(w_j, d_i)}{n(d_i)} \log \left[ \sum_k P(c_k | d_i) P(w_j | c_k) \right] \right)
\end{align} \ ,
$$

this because $n(w_j, d_i)$ is the count of $w_j$ in $d_i$ and $n(d_i)$ is the number of words in $d_i$, hence the semplification at the first addend, as $\sum_j \frac{n(w_j, d_i)}{n(d_i)} = \sum_j P(w_j) = 1$.

Because of the coupling elements given by the $c_k$, this is a hard optimisation problem and the solution can be found via the EM algorithm:

* *E step*: calculate the posterior using the current estimates of the parameters:

$$
P(c_k|d_i, w_j) = \frac{P(w_j, c_k | d_i)}{P(w_j | d_i)} = \frac{P(w_j|c_k, d_i) P(z_k|d_i)}{\sum_k P(w_j|c_i, d_i) P(c_i, d_i)}
$$

* *M step*: maximise the logarithm of the likelihood from the posteriors

## References

1.  T Hofmann [**Probabilistic latent semantic indexing**](http://www.csie.ntu.edu.tw/~b97020/DSP/p50-hofmann.pdf) *Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, ACM* 1999
2.  [Wikipedia on PLSA](https://en.wikipedia.org/wiki/Probabilistic_latent_semantic_analysis)

```python

```


# Word Embeddings: Ungraded Practice Notebook

In this ungraded notebook, you'll try out all the individual techniques that you learned about in the lecture. Practicing on small examples will prepare you for the graded assignment, where you will combine the techniques in more advanced ways to create word embeddings from a real-life corpus.

This notebook is made of two main parts: data preparation, and the continuous bag-of-words (CBOW) model.

To get started, import and initialize all the libraries you will need.

```python
import sys
!{sys.executable} -m pip install emoji
```

    Collecting emoji
      Downloading emoji-0.5.4.tar.gz (43 kB)
         |████████████████████████████████| 43 kB 4.4 MB/s  eta 0:00:01
    Building wheels for collected packages: emoji
      Building wheel for emoji (setup.py) ... done
      Created wheel for emoji: filename=emoji-0.5.4-py3-none-any.whl size=42174 sha256=87187bff9f854b89b65261248401ec79a0e0cc98c5556d1d3b1d2fea3eb19635
      Stored in directory: /home/jovyan/.cache/pip/wheels/f6/65/82/d742fe456cd8aa21ffe6c4c1eaeedf3c2d430689811bf328e1
    Successfully built emoji
    Installing collected packages: emoji
    Successfully installed emoji-0.5.4
    WARNING: You are using pip version 20.1; however, version 20.1.1 is available.
    You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.

```python
import re
import nltk
from nltk.tokenize import word_tokenize
import emoji
import numpy as np

from utils2 import get_dict

nltk.download('punkt')  # download pre-trained Punkt tokenizer for English
```

    [nltk_data] Downloading package punkt to /home/jovyan/nltk_data...
    [nltk_data]   Unzipping tokenizers/punkt.zip.

    True

# Data preparation

In the data preparation phase, starting with a corpus of text, you will:

- Clean and tokenize the corpus.

- Extract the pairs of context words and center word that will make up the training data set for the CBOW model. The context words are the features that will be fed into the model, and the center words are the target values that the model will learn to predict.

- Create simple vector representations of the context words (features) and center words (targets) that can be used by the neural network of the CBOW model.

## Cleaning and tokenization

To demonstrate the cleaning and tokenization process, consider a corpus that contains emojis and various punctuation signs.

```python
corpus = 'Who ❤ "word embeddings" in 2020? I do!!!'
```

First, replace all interrupting punctuation signs — such as commas and exclamation marks — with periods.

```python
print(f'Corpus:  {corpus}')
data = re.sub(r'[,!?;-]+', '.', corpus)
print(f'After cleaning punctuation:  {data}')
```

    Corpus:  Who ❤ "word embeddings" in 2020? I do!!!
    After cleaning punctuation:  Who ❤ "word embeddings" in 2020. I do.

Next, use NLTK's tokenization engine to split the corpus into individual tokens.

```python
print(f'Initial string:  {data}')
data = nltk.word_tokenize(data)
print(f'After tokenization:  {data}')
```

    Initial string:  Who ❤ "word embeddings" in 2020. I do.
    After tokenization:  ['Who', '❤', '``', 'word', 'embeddings', "''", 'in', '2020', '.', 'I', 'do', '.']

Finally, as you saw in the lecture, get rid of numbers and punctuation other than periods, and convert all the remaining tokens to lowercase.

```python
print(f'Initial list of tokens:  {data}')
data = [ ch.lower() for ch in data
         if ch.isalpha()
         or ch == '.'
         or emoji.get_emoji_regexp().search(ch)
       ]
print(f'After cleaning:  {data}')
```

    Initial list of tokens:  ['Who', '❤', '``', 'word', 'embeddings', "''", 'in', '2020', '.', 'I', 'do', '.']
    After cleaning:  ['who', '❤', 'word', 'embeddings', 'in', '.', 'i', 'do', '.']

Note that the heart emoji is considered as a token just like any normal word.

Now let's streamline the cleaning and tokenization process by wrapping the previous steps in a function.

```python
def tokenize(corpus):
    data = re.sub(r'[,!?;-]+', '.', corpus)
    data = nltk.word_tokenize(data)  # tokenize string to words
    data = [ ch.lower() for ch in data
             if ch.isalpha()
             or ch == '.'
             or emoji.get_emoji_regexp().search(ch)
           ]
    return data
```

Apply this function to the corpus that you'll be working on in the rest of this notebook: "I am happy because I am learning"

```python
corpus = 'I am happy because I am learning'
print(f'Corpus:  {corpus}')
words = tokenize(corpus)
print(f'Words (tokens):  {words}')
```

    Corpus:  I am happy because I am learning
    Words (tokens):  ['i', 'am', 'happy', 'because', 'i', 'am', 'learning']

**Now try it out yourself with your own sentence.**

```python
tokenize("Now it's your turn: try with your own sentence!")
```

    ['now', 'it', 'your', 'turn', 'try', 'with', 'your', 'own', 'sentence', '.']

## Sliding window of words

Now that you have transformed the corpus into a list of clean tokens, you can slide a window of words across this list. For each window you can extract a center word and the context words.

The `get_windows` function in the next cell was introduced in the lecture.

```python
def get_windows(words, C):
    i = C
    while i < len(words) - C:
        center_word = words[i]
        context_words = words[(i - C):i] + words[(i+1):(i+C+1)]
        yield context_words, center_word
        i += 1
```

The first argument of this function is a list of words (or tokens). The second argument, `C`, is the context half-size. Recall that for a given center word, the context words are made of `C` words to the left and `C` words to the right of the center word.

Here is how you can use this function to extract context words and center words from a list of tokens. These context and center words will make up the training set that you will use to train the CBOW model.

```python
for x, y in get_windows(
            ['i', 'am', 'happy', 'because', 'i', 'am', 'learning'],
            2
        ):
    print(f'{x}\t{y}')
```

    ['i', 'am', 'because', 'i']	happy
    ['am', 'happy', 'i', 'am']	because
    ['happy', 'because', 'am', 'learning']	i

The first example of the training set is made of:

- the context words "i", "am", "because", "i",

- and the center word to be predicted: "happy".

**Now try it out yourself. In the next cell, you can change both the sentence and the context half-size.**

```python
for x, y in get_windows(tokenize("Now it's your turn: try with your own sentence!"), 1):
    print(f'{x}\t{y}')
```

    ['now', 'your']	it
    ['it', 'turn']	your
    ['your', 'try']	turn
    ['turn', 'with']	try
    ['try', 'your']	with
    ['with', 'own']	your
    ['your', 'sentence']	own
    ['own', '.']	sentence

## Transforming words into vectors for the training set

To finish preparing the training set, you need to transform the context words and center words into vectors.

### Mapping words to indices and indices to words

The center words will be represented as one-hot vectors, and the vectors that represent context words are also based on one-hot vectors.

To create one-hot word vectors, you can start by mapping each unique word to a unique integer (or index). We have provided a helper function, `get_dict`, that creates a Python dictionary that maps words to integers and back.

```python
word2Ind, Ind2word = get_dict(words)
```

Here's the dictionary that maps words to numeric indices.

```python
word2Ind
```

    {'am': 0, 'because': 1, 'happy': 2, 'i': 3, 'learning': 4}

You can use this dictionary to get the index of a word.

```python
print("Index of the word 'i':  ",word2Ind['i'])
```

    Index of the word 'i':   3

And conversely, here's the dictionary that maps indices to words.

```python
Ind2word
```

    {0: 'am', 1: 'because', 2: 'happy', 3: 'i', 4: 'learning'}

```python
print("Word which has index 2:  ",Ind2word[2] )
```

    Word which has index 2:   happy

Finally, get the length of either of these dictionaries to get the size of the vocabulary of your corpus, in other words the number of different words making up the corpus.

```python
V = len(word2Ind)
print("Size of vocabulary: ", V)
```

    Size of vocabulary:  5

### Getting one-hot word vectors

Recall from the lecture that you can easily convert an integer, $n$, into a one-hot vector.

Consider the word "happy". First, retrieve its numeric index.

```python
n = word2Ind['happy']
n
```

    2

Now create a vector with the size of the vocabulary, and fill it with zeros.

```python
center_word_vector = np.zeros(V)
center_word_vector
```

    array([0., 0., 0., 0., 0.])

You can confirm that the vector has the right size.

```python
len(center_word_vector) == V
```

    True

Next, replace the 0 of the $n$-th element with a 1.

```python
center_word_vector[n] = 1
```

And you have your one-hot word vector.

```python
center_word_vector
```

    array([0., 0., 1., 0., 0.])

**You can now group all of these steps in a convenient function, which takes as parameters: a word to be encoded, a dictionary that maps words to indices, and the size of the vocabulary.**

```python
def word_to_one_hot_vector(word, word2Ind, V):
    # BEGIN your code here
    one_hot_vector = np.zeros(V)
    one_hot_vector[word2Ind[word]] = 1
    # END your code here
    return one_hot_vector
```

Check that it works as intended.

```python
word_to_one_hot_vector('happy', word2Ind, V)
```

    array([0., 0., 1., 0., 0.])

**What is the word vector for "learning"?**

```python
# BEGIN your code here
word_to_one_hot_vector('learning', word2Ind, V)
# END your code here
```

    array([0., 0., 0., 0., 1.])

Expected output:

    array([0., 0., 0., 0., 1.])

### Getting context word vectors

To create the vectors that represent context words, you will calculate the average of the one-hot vectors representing the individual words.

Let's start with a list of context words.

```python
context_words = ['i', 'am', 'because', 'i']
```

Using Python's list comprehension construct and the `word_to_one_hot_vector` function that you created in the previous section, you can create a list of one-hot vectors representing each of the context words.

```python
context_words_vectors = [word_to_one_hot_vector(w, word2Ind, V) for w in context_words]
context_words_vectors
```

    [array([0., 0., 0., 1., 0.]),
     array([1., 0., 0., 0., 0.]),
     array([0., 1., 0., 0., 0.]),
     array([0., 0., 0., 1., 0.])]

And you can now simply get the average of these vectors using numpy's `mean` function, to get the vector representation of the context words.

```python
np.mean(context_words_vectors, axis=0)
```

    array([0.25, 0.25, 0.  , 0.5 , 0.  ])

Note the `axis=0` parameter that tells `mean` to calculate the average of the rows (if you had wanted the average of the columns, you would have used `axis=1`).

**Now create the `context_words_to_vector` function that takes in a list of context words, a word-to-index dictionary, and a vocabulary size, and outputs the vector representation of the context words.**

```python
def context_words_to_vector(context_words, word2Ind, V):
    # BEGIN your code here
    context_words_vectors = [word_to_one_hot_vector(w, word2Ind, V) for w in context_words]
    context_words_vectors = np.mean(context_words_vectors, axis=0)
    # END your code here
    return context_words_vectors
```

And check that you obtain the same output as the manual approach above.

```python
context_words_to_vector(['i', 'am', 'because', 'i'], word2Ind, V)
```

    array([0.25, 0.25, 0.  , 0.5 , 0.  ])

**What is the vector representation of the context words "am happy i am"?**

```python
# BEGIN your code here
context_words_to_vector(['am', 'happy', 'i', 'am'], word2Ind, V)
# END your code here
```

    array([0.5 , 0.  , 0.25, 0.25, 0.  ])

Expected output:

    array([0.5 , 0.  , 0.25, 0.25, 0.  ])

## Building the training set

You can now combine the functions that you created in the previous sections, to build a training set for the CBOW model, starting from the following tokenized corpus.

```python
words
```

    ['i', 'am', 'happy', 'because', 'i', 'am', 'learning']

To do this you need to use the sliding window function (`get_windows`) to extract the context words and center words, and you then convert these sets of words into a basic vector representation using `word_to_one_hot_vector` and `context_words_to_vector`.

```python
for context_words, center_word in get_windows(words, 2):  # reminder: 2 is the context half-size
    print(f'Context words:  {context_words} -> {context_words_to_vector(context_words, word2Ind, V)}')
    print(f'Center word:  {center_word} -> {word_to_one_hot_vector(center_word, word2Ind, V)}')
    print()
```

    Context words:  ['i', 'am', 'because', 'i'] -> [0.25 0.25 0.   0.5  0.  ]
    Center word:  happy -> [0. 0. 1. 0. 0.]

    Context words:  ['am', 'happy', 'i', 'am'] -> [0.5  0.   0.25 0.25 0.  ]
    Center word:  because -> [0. 1. 0. 0. 0.]

    Context words:  ['happy', 'because', 'am', 'learning'] -> [0.25 0.25 0.25 0.   0.25]
    Center word:  i -> [0. 0. 0. 1. 0.]

In this practice notebook you'll be performing a single iteration of training using a single example, but in this week's assignment you'll train the CBOW model using several iterations and batches of example.
Here is how you would use a Python generator function (remember the `yield` keyword from the lecture?) to make it easier to iterate over a set of examples.

```python
def get_training_example(words, C, word2Ind, V):
    for context_words, center_word in get_windows(words, C):
        yield context_words_to_vector(context_words, word2Ind, V), word_to_one_hot_vector(center_word, word2Ind, V)
```

The output of this function can be iterated on to get successive context word vectors and center word vectors, as demonstrated in the next cell.

```python
for context_words_vector, center_word_vector in get_training_example(words, 2, word2Ind, V):
    print(f'Context words vector:  {context_words_vector}')
    print(f'Center word vector:  {center_word_vector}')
    print()
```

    Context words vector:  [0.25 0.25 0.   0.5  0.  ]
    Center word vector:  [0. 0. 1. 0. 0.]

    Context words vector:  [0.5  0.   0.25 0.25 0.  ]
    Center word vector:  [0. 1. 0. 0. 0.]

    Context words vector:  [0.25 0.25 0.25 0.   0.25]
    Center word vector:  [0. 0. 0. 1. 0.]

Your training set is ready, you can now move on to the CBOW model itself.

# The continuous bag-of-words model

The CBOW model is based on a neural network, the architecture of which looks like the figure below, as you'll recall from the lecture.

<div style="width:image width px; font-size:100%; text-align:center;"> Figure 1 </div>

This part of the notebook will walk you through:

- The two activation functions used in the neural network.

- Forward propagation.

- Cross-entropy loss.

- Backpropagation.

- Gradient descent.

- Extracting the word embedding vectors from the weight matrices once the neural network has been trained.

## Activation functions

Let's start by implementing the activation functions, ReLU and softmax.

### ReLU

ReLU is used to calculate the values of the hidden layer, in the following formulas:

\begin{align}
 \mathbf{z_1} &= \mathbf{W_1}\mathbf{x} + \mathbf{b_1}  \tag{1} \\
 \mathbf{h} &= \mathrm{ReLU}(\mathbf{z_1})  \tag{2} \\
\end{align}

Let's fix a value for $\mathbf{z_1}$ as a working example.

```python
np.random.seed(10)
z_1 = 10*np.random.rand(5, 1)-5
z_1
```

    array([[ 2.71320643],
           [-4.79248051],
           [ 1.33648235],
           [ 2.48803883],
           [-0.01492988]])

To get the ReLU of this vector, you want all the negative values to become zeros.

First create a copy of this vector.

```python
h = z_1.copy()
```

Now determine which of its values are negative.

```python
h < 0
```

    array([[False],
           [ True],
           [False],
           [False],
           [ True]])

You can now simply set all of the values which are negative to 0.

```python
h[h < 0] = 0
```

And that's it: you have the ReLU of $\mathbf{z_1}$!

```python
h
```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

**Now implement ReLU as a function.**

```python
def relu(z):
    # BEGIN your code here
    result = z.copy()
    result[result < 0] = 0
    # END your code here

    return result
```

**And check that it's working.**

```python
z = np.array([[-1.25459881], [ 4.50714306], [ 2.31993942], [ 0.98658484], [-3.4398136 ]])
relu(z)
```

    array([[0.        ],
           [4.50714306],
           [2.31993942],
           [0.98658484],
           [0.        ]])

Expected output:

    array([[0.        ],
           [4.50714306],
           [2.31993942],
           [0.98658484],
           [0.        ]])

### Softmax

The second activation function that you need is softmax. This function is used to calculate the values of the output layer of the neural network, using the following formulas:

\begin{align}
 \mathbf{z_2} &= \mathbf{W_2}\mathbf{h} + \mathbf{b_2}   \tag{3} \\
 \mathbf{\hat y} &= \mathrm{softmax}(\mathbf{z_2})   \tag{4} \\
\end{align}

To calculate softmax of a vector $\mathbf{z}$, the $i$-th component of the resulting vector is given by:

$$ \textrm{softmax}(\textbf{z})_i = \frac{e^{z_i} }{\sum\limits_{j=1}^{V} e^{z_j} }  \tag{5} $$

Let's work through an example.

```python
z = np.array([9, 8, 11, 10, 8.5])
z
```

    array([ 9. ,  8. , 11. , 10. ,  8.5])

You'll need to calculate the exponentials of each element, both for the numerator and for the denominator.

```python
e_z = np.exp(z)
e_z
```

    array([ 8103.08392758,  2980.95798704, 59874.1417152 , 22026.46579481,
            4914.7688403 ])

The denominator is equal to the sum of these exponentials.

```python
sum_e_z = np.sum(e_z)
sum_e_z
```

    97899.41826492078

And the value of the first element of $\textrm{softmax}(\textbf{z})$ is given by:

```python
e_z[0]/sum_e_z
```

    0.08276947985173956

This is for one element. You can use numpy's vectorized operations to calculate the values of all the elements of the $\textrm{softmax}(\textbf{z})$ vector in one go.

**Implement the softmax function.**

```python
def softmax(z):
    # BEGIN your code here
    e_z = np.exp(z)
    sum_e_z = np.sum(e_z)
    return e_z / sum_e_z
    # END your code here
```

**Now check that it works.**

```python
softmax([9, 8, 11, 10, 8.5])
```

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

Expected output:

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

## Dimensions: 1-D arrays vs 2-D column vectors

Before moving on to implement forward propagation, backpropagation, and gradient descent, let's have a look at the dimensions of the vectors you've been handling until now.

Create a vector of length $V$ filled with zeros.

```python
x_array = np.zeros(V)
x_array
```

    array([0., 0., 0., 0., 0.])

This is a 1-dimensional array, as revealed by the `.shape` property of the array.

```python
x_array.shape
```

    (5,)

To perform matrix multiplication in the next steps, you actually need your column vectors to be represented as a matrix with one column. In numpy, this matrix is represented as a 2-dimensional array.

The easiest way to convert a 1D vector to a 2D column matrix is to set its `.shape` property to the number of rows and one column, as shown in the next cell.

```python
x_column_vector = x_array.copy()
x_column_vector.shape = (V, 1)  # alternatively ... = (x_array.shape[0], 1)
x_column_vector
```

    array([[0.],
           [0.],
           [0.],
           [0.],
           [0.]])

The shape of the resulting "vector" is:

```python
x_column_vector.shape
```

    (5, 1)

So you now have a 5x1 matrix that you can use to perform standard matrix multiplication.

## Forward propagation

Let's dive into the neural network itself, which is shown below with all the dimensions and formulas you'll need.

<div style="width:image width px; font-size:100%; text-align:center;"> Figure 2 </div>

Set $N$ equal to 3. Remember that $N$ is a hyperparameter of the CBOW model that represents the size of the word embedding vectors, as well as the size of the hidden layer.

```python
N = 3
```

### Initialization of the weights and biases

Before you start training the neural network, you need to initialize the weight matrices and bias vectors with random values.

In the assignment you will implement a function to do this yourself using `numpy.random.rand`. In this notebook, we've pre-populated these matrices and vectors for you.

```python
W1 = np.array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
               [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
               [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])

W2 = np.array([[-0.22182064, -0.43008631,  0.13310965],
               [ 0.08476603,  0.08123194,  0.1772054 ],
               [ 0.1871551 , -0.06107263, -0.1790735 ],
               [ 0.07055222, -0.02015138,  0.36107434],
               [ 0.33480474, -0.39423389, -0.43959196]])

b1 = np.array([[ 0.09688219],
               [ 0.29239497],
               [-0.27364426]])

b2 = np.array([[ 0.0352008 ],
               [-0.36393384],
               [-0.12775555],
               [-0.34802326],
               [-0.07017815]])
```

**Check that the dimensions of these matrices match those shown in the figure above.**

```python
# BEGIN your code here
print(f'V (vocabulary size): {V}')
print(f'N (embedding size / size of the hidden layer): {N}')
print(f'size of W1: {W1.shape} (NxV)')
print(f'size of b1: {b1.shape} (Nx1)')
print(f'size of W2: {W1.shape} (VxN)')
print(f'size of b2: {b2.shape} (Vx1)')
# END your code here
```

    V (vocabulary size): 5
    N (embedding size / size of the hidden layer): 3
    size of W1: (3, 5) (NxV)
    size of b1: (3, 1) (Nx1)
    size of W2: (3, 5) (VxN)
    size of b2: (5, 1) (Vx1)

### Training example

Run the next cells to get the first training example, made of the vector representing the context words "i am because i", and the target which is the one-hot vector representing the center word "happy".

> You don't need to worry about the Python syntax, but there are some explanations below if you want to know what's happening behind the scenes.

```python
training_examples = get_training_example(words, 2, word2Ind, V)
```

> `get_training_examples`, which uses the `yield` keyword, is known as a generator. When run, it builds an iterator, which is a special type of object that... you can iterate on (using a `for` loop for instance), to retrieve the successive values that the function generates.
>
> In this case `get_training_examples` `yield`s training examples, and iterating on `training_examples` will return the successive training examples.

```python
x_array, y_array = next(training_examples)
```

> `next` is another special keyword, which gets the next available value from an iterator. Here, you'll get the very first value, which is the first training example. If you run this cell again, you'll get the next value, and so on until the iterator runs out of values to return.
>
> In this notebook `next` is used because you will only be performing one iteration of training. In this week's assignment with the full training over several iterations you'll use regular `for` loops with the iterator that supplies the training examples.

The vector representing the context words, which will be fed into the neural network, is:

```python
x_array
```

    array([0.25, 0.25, 0.  , 0.5 , 0.  ])

The one-hot vector representing the center word to be predicted is:

```python
y_array
```

    array([0., 0., 1., 0., 0.])

Now convert these vectors into matrices (or 2D arrays) to be able to perform matrix multiplication on the right types of objects, as explained above.

```python
x = x_array.copy()
x.shape = (V, 1)
print('x')
print(x)
print()

y = y_array.copy()
y.shape = (V, 1)
print('y')
print(y)
```

    x
    [[0.25]
     [0.25]
     [0.  ]
     [0.5 ]
     [0.  ]]

    y
    [[0.]
     [0.]
     [1.]
     [0.]
     [0.]]

### Values of the hidden layer

Now that you have initialized all the variables that you need for forward propagation, you can calculate the values of the hidden layer using the following formulas:

\begin{align}
 \mathbf{z_1} = \mathbf{W_1}\mathbf{x} + \mathbf{b_1}  \tag{1} \\
 \mathbf{h} = \mathrm{ReLU}(\mathbf{z_1})  \tag{2} \\
\end{align}

First, you can calculate the value of $\mathbf{z_1}$.

```python
z1 = np.dot(W1, x) + b1
```

> `np.dot` is numpy's function for matrix multiplication.

As expected you get an $N$ by 1 matrix, or column vector with $N$ elements, where $N$ is equal to the embedding size, which is 3 in this example.

```python
z1
```

    array([[ 0.36483875],
           [ 0.63710329],
           [-0.3236647 ]])

You can now take the ReLU of $\mathbf{z_1}$ to get $\mathbf{h}$, the vector with the values of the hidden layer.

```python
h = relu(z1)
h
```

    array([[0.36483875],
           [0.63710329],
           [0.        ]])

Applying ReLU means that the negative element of $\mathbf{z_1}$ has been replaced with a zero.

### Values of the output layer

Here are the formulas you need to calculate the values of the output layer, represented by the vector $\mathbf{\hat y}$:

\begin{align}
 \mathbf{z_2} &= \mathbf{W_2}\mathbf{h} + \mathbf{b_2}   \tag{3} \\
 \mathbf{\hat y} &= \mathrm{softmax}(\mathbf{z_2})   \tag{4} \\
\end{align}

**First, calculate $\mathbf{z_2}$.**

```python
# BEGIN your code here
z2 = np.dot(W2, h) + b2
# END your code here

z2
```

    array([[-0.31973737],
           [-0.28125477],
           [-0.09838369],
           [-0.33512159],
           [-0.19919612]])

Expected output:

    array([[-0.31973737],
           [-0.28125477],
           [-0.09838369],
           [-0.33512159],
           [-0.19919612]])

This is a $V$ by 1 matrix, where $V$ is the size of the vocabulary, which is 5 in this example.

**Now calculate the value of $\mathbf{\hat y}$.**

```python
# BEGIN your code here
y_hat = softmax(z2)
# END your code here

y_hat
```

    array([[0.18519074],
           [0.19245626],
           [0.23107446],
           [0.18236353],
           [0.20891502]])

Expected output:

    array([[0.18519074],
           [0.19245626],
           [0.23107446],
           [0.18236353],
           [0.20891502]])

As you've performed the calculations with random matrices and vectors (apart from the input vector), the output of the neural network is essentially random at this point. The learning process will adjust the weights and biases to match the actual targets better.

**That being said, what word did the neural network predict?**

<details>
<summary>
    <font size="3" color="darkgreen"><b>Solution</b></font>
</summary>
<p>The neural network predicted the word "happy": the largest element of $\mathbf{\hat y}$ is the third one, and the third word of the vocabulary is "happy".</p>
<p>Here's how you could implement this in Python:</p>
<p><code>print(Ind2word[np.argmax(y_hat)])</code></p>

Well done, you've completed the forward propagation phase!

## Cross-entropy loss

Now that you have the network's prediction, you can calculate the cross-entropy loss to determine how accurate the prediction was compared to the actual target.

> Remember that you are working on a single training example, not on a batch of examples, which is why you are using *loss* and not *cost*, which is the generalized form of loss.

First let's recall what the prediction was.

```python
y_hat
```

    array([[0.18519074],
           [0.19245626],
           [0.23107446],
           [0.18236353],
           [0.20891502]])

And the actual target value is:

```python
y
```

    array([[0.],
           [0.],
           [1.],
           [0.],
           [0.]])

The formula for cross-entropy loss is:

$$ J=-\sum\limits_{k=1}^{V}y_k\log{\hat{y}_k} \tag{6}$$

**Implement the cross-entropy loss function.**

Here are a some hints if you're stuck.

<details>
<summary>
    <font size="3" color="darkgreen"><b>Hint 1</b></font>
</summary>
    <p>To multiply two numpy matrices (such as <code>y</code> and <code>y_hat</code>) element-wise, you can simply use the <code>*</code> operator.</p>

<details>
<summary>
    <font size="3" color="darkgreen"><b>Hint 2</b></font>
</summary>
<p>Once you have a vector equal to the element-wise multiplication of <code>y</code> and <code>y_hat</code>, you can use <code>np.sum</code> to calculate the sum of the elements of this vector.</p>

```python
def cross_entropy_loss(y_predicted, y_actual):
    # BEGIN your code here
    loss = np.sum(-np.log(y_hat)*y)
    # END your code here
    return loss
```

**Now use this function to calculate the loss with the actual values of $\mathbf{y}$ and $\mathbf{\hat y}$.**

```python
cross_entropy_loss(y_hat, y)
```

    1.4650152923611106

Expected output:

    1.4650152923611106

This value is neither good nor bad, which is expected as the neural network hasn't learned anything yet.

The actual learning will start during the next phase: backpropagation.

## Backpropagation

The formulas that you will implement for backpropagation are the following.

\begin{align}
 \frac{\partial J}{\partial \mathbf{W_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right )\mathbf{x}^\top \tag{7}\\
 \frac{\partial J}{\partial \mathbf{W_2}} &= (\mathbf{\hat{y}} - \mathbf{y})\mathbf{h^\top} \tag{8}\\
 \frac{\partial J}{\partial \mathbf{b_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right ) \tag{9}\\
 \frac{\partial J}{\partial \mathbf{b_2}} &= \mathbf{\hat{y}} - \mathbf{y} \tag{10}
\end{align}

> Note: these formulas are slightly simplified compared to the ones in the lecture as you're working on a single training example, whereas the lecture provided the formulas for a batch of examples. In the assignment you'll be implementing the latter.

Let's start with an easy one.

**Calculate the partial derivative of the loss function with respect to $\mathbf{b_2}$, and store the result in `grad_b2`.**

$$\frac{\partial J}{\partial \mathbf{b_2}} = \mathbf{\hat{y}} - \mathbf{y} \tag{10}$$

```python
# BEGIN your code here
grad_b2 = y_hat - y
# END your code here

grad_b2
```

    array([[ 0.18519074],
           [ 0.19245626],
           [-0.76892554],
           [ 0.18236353],
           [ 0.20891502]])

Expected output:

    array([[ 0.18519074],
           [ 0.19245626],
           [-0.76892554],
           [ 0.18236353],
           [ 0.20891502]])

**Next, calculate the partial derivative of the loss function with respect to $\mathbf{W_2}$, and store the result in `grad_W2`.**

$$\frac{\partial J}{\partial \mathbf{W_2}} = (\mathbf{\hat{y}} - \mathbf{y})\mathbf{h^\top} \tag{8}$$

> Hint: use `.T` to get a transposed matrix, e.g. `h.T` returns $\mathbf{h^\top}$.

```python
# BEGIN your code here
grad_W2 = np.dot(y_hat - y, h.T)
# END your code here

grad_W2
```

    array([[ 0.06756476,  0.11798563,  0.        ],
           [ 0.0702155 ,  0.12261452,  0.        ],
           [-0.28053384, -0.48988499,  0.        ],
           [ 0.06653328,  0.1161844 ,  0.        ],
           [ 0.07622029,  0.13310045,  0.        ]])

Expected output:

    array([[ 0.06756476,  0.11798563,  0.        ],
           [ 0.0702155 ,  0.12261452,  0.        ],
           [-0.28053384, -0.48988499,  0.        ],
           [ 0.06653328,  0.1161844 ,  0.        ],
           [ 0.07622029,  0.13310045,  0.        ]])

**Now calculate the partial derivative with respect to $\mathbf{b_1}$ and store the result in `grad_b1`.**

$$\frac{\partial J}{\partial \mathbf{b_1}} = \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right ) \tag{9}$$

```python
# BEGIN your code here
grad_b1 = relu(np.dot(W2.T, y_hat - y))
# END your code here

grad_b1
```

    array([[0.        ],
           [0.        ],
           [0.17045858]])

Expected output:

    array([[0.        ],
           [0.        ],
           [0.17045858]])

**Finally, calculate the partial derivative of the loss with respect to $\mathbf{W_1}$, and store it in `grad_W1`.**

$$\frac{\partial J}{\partial \mathbf{W_1}} = \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right )\mathbf{x}^\top \tag{7}$$

```python
# BEGIN your code here
grad_W1 = np.dot(relu(np.dot(W2.T, y_hat - y)), x.T)
# END your code here

grad_W1
```

    array([[0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.04261464, 0.04261464, 0.        , 0.08522929, 0.        ]])

Expected output:

    array([[0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.        , 0.        , 0.        , 0.        , 0.        ],
           [0.04261464, 0.04261464, 0.        , 0.08522929, 0.        ]])

Before moving on to gradient descent, double-check that all the matrices have the expected dimensions.

```python
# BEGIN your code here
print(f'V (vocabulary size): {V}')
print(f'N (embedding size / size of the hidden layer): {N}')
print(f'size of grad_W1: {grad_W1.shape} (NxV)')
print(f'size of grad_b1: {grad_b1.shape} (Nx1)')
print(f'size of grad_W2: {grad_W1.shape} (VxN)')
print(f'size of grad_b2: {grad_b2.shape} (Vx1)')
# END your code here
```

    V (vocabulary size): 5
    N (embedding size / size of the hidden layer): 3
    size of grad_W1: (3, 5) (NxV)
    size of grad_b1: (3, 1) (Nx1)
    size of grad_W2: (3, 5) (VxN)
    size of grad_b2: (5, 1) (Vx1)

## Gradient descent

During the gradient descent phase, you will update the weights and biases by subtracting $\alpha$ times the gradient from the original matrices and vectors, using the following formulas.

\begin{align}
 \mathbf{W_1} &:= \mathbf{W_1} - \alpha \frac{\partial J}{\partial \mathbf{W_1}} \tag{11}\\
 \mathbf{W_2} &:= \mathbf{W_2} - \alpha \frac{\partial J}{\partial \mathbf{W_2}} \tag{12}\\
 \mathbf{b_1} &:= \mathbf{b_1} - \alpha \frac{\partial J}{\partial \mathbf{b_1}} \tag{13}\\
 \mathbf{b_2} &:= \mathbf{b_2} - \alpha \frac{\partial J}{\partial \mathbf{b_2}} \tag{14}\\
\end{align}

First, let set a value for $\alpha$.

```python
alpha = 0.03
```

The updated weight matrix $\mathbf{W_1}$ will be:

```python
W1_new = W1 - alpha * grad_W1
```

Let's compare the previous and new values of $\mathbf{W_1}$:

```python
print('old value of W1:')
print(W1)
print()
print('new value of W1:')
print(W1_new)
```

    old value of W1:
    [[ 0.41687358  0.08854191 -0.23495225  0.28320538  0.41800106]
     [ 0.32735501  0.22795148 -0.23951958  0.4117634  -0.23924344]
     [ 0.26637602 -0.23846886 -0.37770863 -0.11399446  0.34008124]]

    new value of W1:
    [[ 0.41687358  0.08854191 -0.23495225  0.28320538  0.41800106]
     [ 0.32735501  0.22795148 -0.23951958  0.4117634  -0.23924344]
     [ 0.26509758 -0.2397473  -0.37770863 -0.11655134  0.34008124]]

The difference is very subtle (hint: take a closer look at the last row), which is why it takes a fair amount of iterations to train the neural network until it reaches optimal weights and biases starting from random values.

**Now calculate the new values of $\mathbf{W_2}$ (to be stored in `W2_new`), $\mathbf{b_1}$ (in `b1_new`), and $\mathbf{b_2}$ (in `b2_new`).**

\begin{align}
 \mathbf{W_2} &:= \mathbf{W_2} - \alpha \frac{\partial J}{\partial \mathbf{W_2}} \tag{12}\\
 \mathbf{b_1} &:= \mathbf{b_1} - \alpha \frac{\partial J}{\partial \mathbf{b_1}} \tag{13}\\
 \mathbf{b_2} &:= \mathbf{b_2} - \alpha \frac{\partial J}{\partial \mathbf{b_2}} \tag{14}\\
\end{align}

```python
# BEGIN your code here
W2_new = W2 - alpha * grad_W2
b1_new = b1 - alpha * grad_b1
b2_new = b2 - alpha * grad_b2
# END your code here

print('W2_new')
print(W2_new)
print()
print('b1_new')
print(b1_new)
print()
print('b2_new')
print(b2_new)
```

    W2_new
    [[-0.22384758 -0.43362588  0.13310965]
     [ 0.08265956  0.0775535   0.1772054 ]
     [ 0.19557112 -0.04637608 -0.1790735 ]
     [ 0.06855622 -0.02363691  0.36107434]
     [ 0.33251813 -0.3982269  -0.43959196]]

    b1_new
    [[ 0.09688219]
     [ 0.29239497]
     [-0.27875802]]

    b2_new
    [[ 0.02964508]
     [-0.36970753]
     [-0.10468778]
     [-0.35349417]
     [-0.0764456 ]]

Expected output:

    W2_new
    [[-0.22384758 -0.43362588  0.13310965]
     [ 0.08265956  0.0775535   0.1772054 ]
     [ 0.19557112 -0.04637608 -0.1790735 ]
     [ 0.06855622 -0.02363691  0.36107434]
     [ 0.33251813 -0.3982269  -0.43959196]]

    b1_new
    [[ 0.09688219]
     [ 0.29239497]
     [-0.27875802]]

    b2_new
    [[ 0.02964508]
     [-0.36970753]
     [-0.10468778]
     [-0.35349417]
     [-0.0764456 ]]

Congratulations, you have completed one iteration of training using one training example!

You'll need many more iterations to fully train the neural network, and you can optimize the learning process by training on batches of examples, as described in the lecture. You will get to do this during this week's assignment.

## Extracting word embedding vectors

Once you have finished training the neural network, you have three options to get word embedding vectors for the words of your vocabulary, based on the weight matrices $\mathbf{W_1}$ and/or $\mathbf{W_2}$.

### Option 1: extract embedding vectors from $\mathbf{W_1}$

The first option is to take the columns of $\mathbf{W_1}$ as the embedding vectors of the words of the vocabulary, using the same order of the words as for the input and output vectors.

> Note: in this practice notebook the values of the word embedding vectors are meaningless after a single iteration with just one training example, but here's how you would proceed after the training process is complete.

For example $\mathbf{W_1}$ is this matrix:

```python
W1
```

    array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],
           [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],
           [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])

The first column, which is a 3-element vector, is the embedding vector of the first word of your vocabulary. The second column is the word embedding vector for the second word, and so on.

The first, second, etc. words are ordered as follows.

```python
for i in range(V):
    print(Ind2word[i])
```

    am
    because
    happy
    i
    learning

So the word embedding vectors corresponding to each word are:

```python
# loop through each word of the vocabulary
for word in word2Ind:
    # extract the column corresponding to the index of the word in the vocabulary
    word_embedding_vector = W1[:, word2Ind[word]]

    print(f'{word}: {word_embedding_vector}')
```

    am: [0.41687358 0.32735501 0.26637602]
    because: [ 0.08854191  0.22795148 -0.23846886]
    happy: [-0.23495225 -0.23951958 -0.37770863]
    i: [ 0.28320538  0.4117634  -0.11399446]
    learning: [ 0.41800106 -0.23924344  0.34008124]

### Option 2: extract embedding vectors from $\mathbf{W_2}$

The second option is to take $\mathbf{W_2}$ transposed, and take its columns as the word embedding vectors just like you did for $\mathbf{W_1}$.

```python
W2.T
```

    array([[-0.22182064,  0.08476603,  0.1871551 ,  0.07055222,  0.33480474],
           [-0.43008631,  0.08123194, -0.06107263, -0.02015138, -0.39423389],
           [ 0.13310965,  0.1772054 , -0.1790735 ,  0.36107434, -0.43959196]])

```python
# loop through each word of the vocabulary
for word in word2Ind:
    # extract the column corresponding to the index of the word in the vocabulary
    word_embedding_vector = W2.T[:, word2Ind[word]]

    print(f'{word}: {word_embedding_vector}')
```

    am: [-0.22182064 -0.43008631  0.13310965]
    because: [0.08476603 0.08123194 0.1772054 ]
    happy: [ 0.1871551  -0.06107263 -0.1790735 ]
    i: [ 0.07055222 -0.02015138  0.36107434]
    learning: [ 0.33480474 -0.39423389 -0.43959196]

### Option 3: extract embedding vectors from $\mathbf{W_1}$ and $\mathbf{W_2}$

The third option, which is the one you will use in this week's assignment, uses the average of $\mathbf{W_1}$ and $\mathbf{W_2^\top}$.

**Calculate the average of $\mathbf{W_1}$ and $\mathbf{W_2^\top}$, and store the result in `W3`.**

```python
# BEGIN your code here
W3 = (W1+W2.T)/2
# END your code here

W3
```

    array([[ 0.09752647,  0.08665397, -0.02389858,  0.1768788 ,  0.3764029 ],
           [-0.05136565,  0.15459171, -0.15029611,  0.19580601, -0.31673866],
           [ 0.19974284, -0.03063173, -0.27839106,  0.12353994, -0.04975536]])

Expected output:

    array([[ 0.09752647,  0.08665397, -0.02389858,  0.1768788 ,  0.3764029 ],
           [-0.05136565,  0.15459171, -0.15029611,  0.19580601, -0.31673866],
           [ 0.19974284, -0.03063173, -0.27839106,  0.12353994, -0.04975536]])

Extracting the word embedding vectors works just like the two previous options, by taking the columns of the matrix you've just created.

```python
# loop through each word of the vocabulary
for word in word2Ind:
    # extract the column corresponding to the index of the word in the vocabulary
    word_embedding_vector = W3[:, word2Ind[word]]

    print(f'{word}: {word_embedding_vector}')
```

    am: [ 0.09752647 -0.05136565  0.19974284]
    because: [ 0.08665397  0.15459171 -0.03063173]
    happy: [-0.02389858 -0.15029611 -0.27839106]
    i: [0.1768788  0.19580601 0.12353994]
    learning: [ 0.3764029  -0.31673866 -0.04975536]

You're now ready to take on this week's assignment!

### How this practice relates to and differs from the upcoming graded assignment

- In the assignment, for each iteration of training you will use batches of examples instead of a single example. The formulas for forward propagation and backpropagation will be modified accordingly, and you will use cross-entropy cost instead of cross-entropy loss.
- You will also complete several iterations of training, until you reach an acceptably low cross-entropy cost, at which point you can extract good word embeddings from the weight matrices.
- After extracting the word embedding vectors, you will use principal component analysis (PCA) to visualize the vectors, which will enable you to perform an intrinsic evaluation of the quality of the vectors, as explained in the lecture.

Natural Language Processing/Course 2 - Natural Language Processing with Probabilistic Models/Assignments/Week 4/Word Embeddings.ipynb


# PyTorch tutorial の内容について(5)

- date: 2020-0722
- author: 浅川伸一

[https://github.com/pytorch/tutorials/tree/master/beginner_source/nlp](https://github.com/pytorch/tutorials/tree/master/beginner_source/nlp) を見ると
PyTorch で 自然言語処理を行う場合のチュートリアルは以下とおりである

# Deep Learning for NLP with Pytorch

1. [pytorch_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/pytorch_tutorial.py):
	[PyTorch 入門 Introduction to PyTorch](https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html)

2. [deep_learning_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/deep_learning_tutorial.py):
	[PyTorch による深層学習 Deep Learning with PyTorch](https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html)

3. [word_embeddings_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/word_embeddings_tutorial.py):
	[単語埋め込み:語彙的意味の符号化 Word Embeddings: Encoding Lexical Semantics](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html)

4. [sequence_models_tutorial.py]((https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/sequence_models_tutorial.py):
	[系列モデルと LSTM Sequence Models and Long-Short Term Memory Networks](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html)

5. [advanced_tutorial.py]((https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/advanced_tutorial.py):
	[動的意思決定と双方向 LSTM 条件付き確率場 Advanced: Making Dynamic Decisions and the Bi-LSTM CRF](https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html)

以下では，このうちの 5 について解説している。

```python
%matplotlib inline
```

# Advanced: Making Dynamic Decisions and the Bi-LSTM CRF

## Dynamic versus Static Deep Learning Toolkits

Pytorch is a *dynamic* neural network kit.
Another example of a dynamic kit is `Dynet <https://github.com/clab/dynet>`__ (I mention this because working with Pytorch and Dynet is similar.
If you see an example in Dynet, it will probably help you implement it in Pytorch).
The opposite is the *static* tool kit, which includes Theano, Keras, TensorFlow, etc.
The core difference is the following:

* In a static toolkit, you define  a computation graph once, compile it, and then stream instances to it.
* In a dynamic toolkit, you define a computation graph *for each   instance*. It is never compiled and is executed on-the-fly

Without a lot of experience, it is difficult to appreciate the difference. One example is to suppose we want to build a deep constituent parser.
Suppose our model involves roughly the following
steps:

* We build the tree bottom up
* Tag the root nodes (the words of the sentence)
* From there, use a neural network and the embeddings
  of the words to find combinations that form constituents. Whenever you
  form a new constituent, use some sort of technique to get an embedding
  of the constituent. In this case, our network architecture will depend
  completely on the input sentence. In the sentence "The green cat
  scratched the wall", at some point in the model, we will want to combine
  the span $(i,j,r) = (1, 3, \text{NP})$ (that is, an NP constituent
  spans word 1 to word 3, in this case "The green cat").

However, another sentence might be "Somewhere, the big fat cat scratched the wall".
In this sentence, we will want to form the constituent $(2, 4, NP)$ at some point.
The constituents we will want to form will depend on the instance.
If we just compile the computation graph once, as in a static toolkit, it will be exceptionally difficult or
impossible to program this logic.
In a dynamic toolkit though, there isn't just 1 pre-defined computation graph. There can be a new computation graph for each instance, so this problem goes away.

Dynamic toolkits also have the advantage of being easier to debug and the code more closely resembling the host language (by that I mean that Pytorch and Dynet look more like actual Python code than Keras or Theano).

## Bi-LSTM Conditional Random Field Discussion

For this section, we will see a full, complicated example of a Bi-LSTM Conditional Random Field for named-entity recognition.
The LSTM tagger above is typically sufficient for part-of-speech tagging, but a sequence model like the CRF is really essential for strong performance on NER.
Familiarity with CRF's is assumed. Although this name sounds scary, all the model is is a CRF but where an LSTM provides the features.
This is an advanced model though, far more complicated than any earlier model in this tutorial.
If you want to skip it, that is fine. To see if you're ready, see if you can:

-  Write the recurrence for the viterbi variable at step i for tag k.
-  Modify the above recurrence to compute the forward variables instead.
-  Modify again the above recurrence to compute the forward variables in log-space (hint: log-sum-exp)

If you can do those three things, you should be able to understand the code below.
Recall that the CRF computes a conditional probability.
Let $y$ be a tag sequence and $x$ an input sequence of words.
Then we compute

\begin{align}P(y|x) = \frac{\exp{(\text{Score}(x, y)})}{\sum_{y'} \exp{(\text{Score}(x, y')})}\end{align}

Where the score is determined by defining some log potentials $\log \psi_i(x,y)$ such that

\begin{align}\text{Score}(x,y) = \sum_i \log \psi_i(x,y)\end{align}

To make the partition function tractable, the potentials must look only
at local features.

In the Bi-LSTM CRF, we define two kinds of potentials: emission and transition.
The emission potential for the word at index $i$ comes from the hidden state of the Bi-LSTM at timestep $i$.
The transition scores are stored in a $|T|x|T|$ matrix $\textbf{P}$, where $T$ is the tag set.
In my implementation, $\textbf{P}_{j,k}$ is the score of transitioning to tag $j$ from tag $k$. So:

\begin{align}\text{Score}(x,y) = \sum_i \log \psi_\text{EMIT}(y_i \rightarrow x_i) + \log \psi_\text{TRANS}(y_{i-1} \rightarrow y_i)\end{align}

\begin{align}= \sum_i h_i[y_i] + \textbf{P}_{y_i, y_{i-1}}\end{align}

where in this second expression, we think of the tags as being assigned unique non-negative indices.

If the above discussion was too brief, you can check out [`this`](http://www.cs.columbia.edu/%7Emcollins/crf.pdf) write up from Michael Collins on CRFs.

## Implementation Notes

The example below implements the forward algorithm in log space to compute the partition function, and the viterbi algorithm to decode.
Backpropagation will compute the gradients automatically for us.
We don't have to do anything by hand.

The implementation is not optimized. If you understand what is going on, you'll probably quickly see that iterating over the next tag in the forward algorithm could probably be done in one big operation.
I wanted to code to be more readable.
If you want to make the relevant change, you could probably use this tagger for real tasks.

```python
# Author: Robert Guthrie

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.optim as optim

torch.manual_seed(1)
```

Helper functions to make the code more readable.

```python
def argmax(vec):
    # return the argmax as a python int
    _, idx = torch.max(vec, 1)
    return idx.item()

def prepare_sequence(seq, to_ix):
    idxs = [to_ix[w] for w in seq]
    return torch.tensor(idxs, dtype=torch.long)

# Compute log sum exp in a numerically stable way for the forward algorithm
def log_sum_exp(vec):
    max_score = vec[0, argmax(vec)]
    max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])
    return max_score + \
        torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))
```

Create model

```python
class BiLSTM_CRF(nn.Module):

    def __init__(self, vocab_size, tag_to_ix, embedding_dim, hidden_dim):
        super(BiLSTM_CRF, self).__init__()
        self.embedding_dim = embedding_dim
        self.hidden_dim = hidden_dim
        self.vocab_size = vocab_size
        self.tag_to_ix = tag_to_ix
        self.tagset_size = len(tag_to_ix)

        self.word_embeds = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim // 2,
                            num_layers=1, bidirectional=True)

        # Maps the output of the LSTM into tag space.
        self.hidden2tag = nn.Linear(hidden_dim, self.tagset_size)

        # Matrix of transition parameters.  Entry i,j is the score of
        # transitioning *to* i *from* j.
        self.transitions = nn.Parameter(
            torch.randn(self.tagset_size, self.tagset_size))

        # These two statements enforce the constraint that we never transfer
        # to the start tag and we never transfer from the stop tag
        self.transitions.data[tag_to_ix[START_TAG], :] = -10000
        self.transitions.data[:, tag_to_ix[STOP_TAG]] = -10000

        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (torch.randn(2, 1, self.hidden_dim // 2),
                torch.randn(2, 1, self.hidden_dim // 2))

    def _forward_alg(self, feats):
        # Do the forward algorithm to compute the partition function
        init_alphas = torch.full((1, self.tagset_size), -10000.)
        # START_TAG has all of the score.
        init_alphas[0][self.tag_to_ix[START_TAG]] = 0.

        # Wrap in a variable so that we will get automatic backprop
        forward_var = init_alphas

        # Iterate through the sentence
        for feat in feats:
            alphas_t = []  # The forward tensors at this timestep
            for next_tag in range(self.tagset_size):
                # broadcast the emission score: it is the same regardless of
                # the previous tag
                emit_score = feat[next_tag].view(
                    1, -1).expand(1, self.tagset_size)
                # the ith entry of trans_score is the score of transitioning to
                # next_tag from i
                trans_score = self.transitions[next_tag].view(1, -1)
                # The ith entry of next_tag_var is the value for the
                # edge (i -> next_tag) before we do log-sum-exp
                next_tag_var = forward_var + trans_score + emit_score
                # The forward variable for this tag is log-sum-exp of all the
                # scores.
                alphas_t.append(log_sum_exp(next_tag_var).view(1))
            forward_var = torch.cat(alphas_t).view(1, -1)
        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
        alpha = log_sum_exp(terminal_var)
        return alpha

    def _get_lstm_features(self, sentence):
        self.hidden = self.init_hidden()
        embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)
        lstm_out, self.hidden = self.lstm(embeds, self.hidden)
        lstm_out = lstm_out.view(len(sentence), self.hidden_dim)
        lstm_feats = self.hidden2tag(lstm_out)
        return lstm_feats

    def _score_sentence(self, feats, tags):
        # Gives the score of a provided tag sequence
        score = torch.zeros(1)
        tags = torch.cat([torch.tensor([self.tag_to_ix[START_TAG]], dtype=torch.long), tags])
        for i, feat in enumerate(feats):
            score = score + \
                self.transitions[tags[i + 1], tags[i]] + feat[tags[i + 1]]
        score = score + self.transitions[self.tag_to_ix[STOP_TAG], tags[-1]]
        return score

    def _viterbi_decode(self, feats):
        backpointers = []

        # Initialize the viterbi variables in log space
        init_vvars = torch.full((1, self.tagset_size), -10000.)
        init_vvars[0][self.tag_to_ix[START_TAG]] = 0

        # forward_var at step i holds the viterbi variables for step i-1
        forward_var = init_vvars
        for feat in feats:
            bptrs_t = []  # holds the backpointers for this step
            viterbivars_t = []  # holds the viterbi variables for this step

            for next_tag in range(self.tagset_size):
                # next_tag_var[i] holds the viterbi variable for tag i at the
                # previous step, plus the score of transitioning
                # from tag i to next_tag.
                # We don't include the emission scores here because the max
                # does not depend on them (we add them in below)
                next_tag_var = forward_var + self.transitions[next_tag]
                best_tag_id = argmax(next_tag_var)
                bptrs_t.append(best_tag_id)
                viterbivars_t.append(next_tag_var[0][best_tag_id].view(1))
            # Now add in the emission scores, and assign forward_var to the set
            # of viterbi variables we just computed
            forward_var = (torch.cat(viterbivars_t) + feat).view(1, -1)
            backpointers.append(bptrs_t)

        # Transition to STOP_TAG
        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
        best_tag_id = argmax(terminal_var)
        path_score = terminal_var[0][best_tag_id]

        # Follow the back pointers to decode the best path.
        best_path = [best_tag_id]
        for bptrs_t in reversed(backpointers):
            best_tag_id = bptrs_t[best_tag_id]
            best_path.append(best_tag_id)
        # Pop off the start tag (we dont want to return that to the caller)
        start = best_path.pop()
        assert start == self.tag_to_ix[START_TAG]  # Sanity check
        best_path.reverse()
        return path_score, best_path

    def neg_log_likelihood(self, sentence, tags):
        feats = self._get_lstm_features(sentence)
        forward_score = self._forward_alg(feats)
        gold_score = self._score_sentence(feats, tags)
        return forward_score - gold_score

    def forward(self, sentence):  # dont confuse this with _forward_alg above.
        # Get the emission scores from the BiLSTM
        lstm_feats = self._get_lstm_features(sentence)

        # Find the best path, given the features.
        score, tag_seq = self._viterbi_decode(lstm_feats)
        return score, tag_seq
```

Run training

```python
START_TAG = "<START>"
STOP_TAG = "<STOP>"
EMBEDDING_DIM = 5
HIDDEN_DIM = 4

# Make up some training data
training_data = [(
    "the wall street journal reported today that apple corporation made money".split(),
    "B I I I O O O B I O O".split()
), (
    "georgia tech is a university in georgia".split(),
    "B I O O O O B".split()
)]

word_to_ix = {}
for sentence, tags in training_data:
    for word in sentence:
        if word not in word_to_ix:
            word_to_ix[word] = len(word_to_ix)

tag_to_ix = {"B": 0, "I": 1, "O": 2, START_TAG: 3, STOP_TAG: 4}

model = BiLSTM_CRF(len(word_to_ix), tag_to_ix, EMBEDDING_DIM, HIDDEN_DIM)
optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)

# Check predictions before training
with torch.no_grad():
    precheck_sent = prepare_sequence(training_data[0][0], word_to_ix)
    precheck_tags = torch.tensor([tag_to_ix[t] for t in training_data[0][1]], dtype=torch.long)
    print(model(precheck_sent))

# Make sure prepare_sequence from earlier in the LSTM section is loaded
for epoch in range(
        300):  # again, normally you would NOT do 300 epochs, it is toy data
    for sentence, tags in training_data:
        # Step 1. Remember that Pytorch accumulates gradients.
        # We need to clear them out before each instance
        model.zero_grad()

        # Step 2. Get our inputs ready for the network, that is,
        # turn them into Tensors of word indices.
        sentence_in = prepare_sequence(sentence, word_to_ix)
        targets = torch.tensor([tag_to_ix[t] for t in tags], dtype=torch.long)

        # Step 3. Run our forward pass.
        loss = model.neg_log_likelihood(sentence_in, targets)

        # Step 4. Compute the loss, gradients, and update the parameters by
        # calling optimizer.step()
        loss.backward()
        optimizer.step()

# Check predictions after training
with torch.no_grad():
    precheck_sent = prepare_sequence(training_data[0][0], word_to_ix)
    print(model(precheck_sent))
# We got it!
```

## Exercise: A new loss function for discriminative tagging

It wasn't really necessary for us to create a computation graph when doing decoding, since we do not backpropagate from the viterbi path score.
Since we have it anyway, try training the tagger where the loss function is the difference between the Viterbi path score and the score of the gold-standard path.
It should be clear that this function is non-negative and 0 when the predicted tag sequence is the correct tag sequence. This is essentially *structured perceptron*.

This modification should be short, since Viterbi and score\_sentence are already implemented.
This is an example of the shape of the computation graph *depending on the training instance*.
Although I haven't tried implementing this in a static toolkit, I imagine that it is possible but much less straightforward.

Pick up some real data and do a comparison!

```python

```