# Assignment 4:  Question duplicates

Welcome to the fourth assignment of course 3. In this assignment you will explore Siamese networks applied to natural language processing. You will further explore the fundamentals of Trax and you will be able to implement a more complicated structure using it. By completing this assignment, you will learn how to implement models with different architectures.

## Outline

- [Overview](#0)
- [Part 1: Importing the Data](#1)
    - [1.1 Loading in the data](#1.1)
    - [1.2 Converting a question to a tensor](#1.2)
    - [1.3 Understanding the iterator](#1.3)
        - [Exercise 01](#ex01)
- [Part 2: Defining the Siamese model](#2)
    - [2.1 Understanding Siamese Network](#2.1)
        - [Exercise 02](#ex02)
    - [2.2 Hard  Negative Mining](#2.2)
        - [Exercise 03](#ex03)
- [Part 3: Training](#3)
    - [3.1 Training the model](#3.1)
        - [Exercise 04](#ex04)
- [Part 4: Evaluation](#4)
    - [4.1 Evaluating your siamese network](#4.1)
    - [4.2 Classify](#4.2)
        - [Exercise 05](#ex05)
- [Part 5: Testing with your own questions](#5)
    - [Exercise 06](#ex06)
- [On Siamese networks](#6)

### Overview
In this assignment, concretely you will:

- Learn about Siamese networks
- Understand how the triplet loss works
- Understand how to evaluate accuracy
- Use cosine similarity between the model's outputted vectors
- Use the data generator to get batches of questions
- Predict using your own model

By now, you are familiar with trax and know how to make use of classes to define your model. We will start this homework by asking you to preprocess the data the same way you did in the previous assignments. After processing the data you will build a classifier that will allow you to identify whether to questions are the same or not.

You will process the data first and then pad in a similar way you have done in the previous assignment. Your model will take in the two question embeddings, run them through an LSTM, and then compare the outputs of the two sub networks using cosine similarity. Before taking a deep dive into the model, start by importing the data set.

# Part 1: Importing the Data

### 1.1 Loading in the data

You will be using the Quora question answer dataset to build a model that could identify similar questions. This is a useful task because you don't want to have several versions of the same question posted. Several times when teaching I end up responding to similar questions on piazza, or on other community forums. This data set has been labeled for you. Run the cell below to import some of the packages you will be using.

```python
import os
import nltk
import trax
from trax import layers as tl
from trax.supervised import training
from trax.fastmath import numpy as fastnp
import numpy as np
import pandas as pd
import random as rnd

# set random seeds
trax.supervised.trainer_lib.init_random_number_generators(34)
rnd.seed(34)
```

    INFO:tensorflow:tokens_length=568 inputs_length=512 targets_length=114 noise_density=0.15 mean_noise_span_length=3.0

**Notice that for this assignment Trax's numpy is referred to as `fastnp`, while regular numpy is referred to as `np`.**

You will now load in the data set. We have done some preprocessing for you. If you have taken the deeplearning specialization, this is a slightly different training method than the one you have seen there. If you have not, then don't worry about it, we will explain everything.

```python
data = pd.read_csv("questions.csv")
N=len(data)
print('Number of question pairs: ', N)
data.head()
```

    Number of question pairs:  404351

      id
      qid1
      qid2
      question1
      question2
      is_duplicate

      0
      0
      1
      2
      What is the step by step guide to invest in sh...
      What is the step by step guide to invest in sh...
      0

      1
      1
      3
      4
      What is the story of Kohinoor (Koh-i-Noor) Dia...
      What would happen if the Indian government sto...
      0

      2
      2
      5
      6
      How can I increase the speed of my internet co...
      How can Internet speed be increased by hacking...
      0

      3
      3
      7
      8
      Why am I mentally very lonely? How can I solve...
      Find the remainder when [math]23^{24}[/math] i...
      0

      4
      4
      9
      10
      Which one dissolve in water quikly sugar, salt...
      Which fish would survive in salt water?
      0

We first split the data into a train and test set. The test set will be used later to evaluate our model.

```python
N_train = 300000
N_test  = 10*1024
data_train = data[:N_train]
data_test  = data[N_train:N_train+N_test]
print("Train set:", len(data_train), "Test set:", len(data_test))
del(data) # remove to free memory
```

    Train set: 300000 Test set: 10240

As explained in the lectures, we select only the question pairs that are duplicate to train the model.

We build two batches as input for the Siamese network and we assume that question $q1_i$ (question $i$ in the first batch) is a duplicate of $q2_i$ (question $i$ in the second batch), but all other questions in the second batch are not duplicates of $q1_i$.
The test set uses the original pairs of questions and the status describing if the questions are duplicates.

```python
td_index = (data_train['is_duplicate'] == 1).to_numpy()
td_index = [i for i, x in enumerate(td_index) if x]
print('number of duplicate questions: ', len(td_index))
print('indexes of first ten duplicate questions:', td_index[:10])
```

    number of duplicate questions:  111486
    indexes of first ten duplicate questions: [5, 7, 11, 12, 13, 15, 16, 18, 20, 29]

```python
print(data_train['question1'][5])  #  Example of question duplicates (first one in data)
print(data_train['question2'][5])
print('is_duplicate: ', data_train['is_duplicate'][5])
```

    Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?
    I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?
    is_duplicate:  1

```python
Q1_train_words = np.array(data_train['question1'][td_index])
Q2_train_words = np.array(data_train['question2'][td_index])

Q1_test_words = np.array(data_test['question1'])
Q2_test_words = np.array(data_test['question2'])
y_test  = np.array(data_test['is_duplicate'])
```

Above, you have seen that you only took the duplicated questions for training our model.
You did so on purpose, because the data generator will produce batches $([q1_1, q1_2, q1_3, ...]$, $[q2_1, q2_2,q2_3, ...])$  where $q1_i$ and $q2_k$ are duplicate if and only if $i = k$.

Let's print to see what your data looks like.

```python
print('TRAINING QUESTIONS:\n')
print('Question 1: ', Q1_train_words[0])
print('Question 2: ', Q2_train_words[0], '\n')
print('Question 1: ', Q1_train_words[5])
print('Question 2: ', Q2_train_words[5], '\n')

print('TESTING QUESTIONS:\n')
print('Question 1: ', Q1_test_words[0])
print('Question 2: ', Q2_test_words[0], '\n')
print('is_duplicate =', y_test[0], '\n')
```

    TRAINING QUESTIONS:

    Question 1:  Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?
    Question 2:  I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?

    Question 1:  What would a Trump presidency mean for current international master’s students on an F1 visa?
    Question 2:  How will a Trump presidency affect the students presently in US or planning to study in US?

    TESTING QUESTIONS:

    Question 1:  How do I prepare for interviews for cse?
    Question 2:  What is the best way to prepare for cse?

    is_duplicate = 0

You will now encode each word of the selected duplicate pairs with an index.
 Given a question, you can then just encode it as a list of numbers.

First you tokenize the questions using `nltk.word_tokenize`.

You need a python default dictionary which later, during inference, assigns the values $0$ to all Out Of Vocabulary (OOV) words.

Then you encode each word of the selected duplicate pairs with an index. Given a question, you can then just encode it as a list of numbers.

```python
#create arrays
Q1_train = np.empty_like(Q1_train_words)
Q2_train = np.empty_like(Q2_train_words)

Q1_test = np.empty_like(Q1_test_words)
Q2_test = np.empty_like(Q2_test_words)
```

```python
# Building the vocabulary with the train set         (this might take a minute)
from collections import defaultdict

vocab = defaultdict(lambda: 0)
vocab['<PAD>'] = 1

for idx in range(len(Q1_train_words)):
    Q1_train[idx] = nltk.word_tokenize(Q1_train_words[idx])
    Q2_train[idx] = nltk.word_tokenize(Q2_train_words[idx])
    q = Q1_train[idx] + Q2_train[idx]
    for word in q:
        if word not in vocab:
            vocab[word] = len(vocab) + 1
print('The length of the vocabulary is: ', len(vocab))
```

    The length of the vocabulary is:  36268

```python
print(vocab['<PAD>'])
print(vocab['Astrology'])
print(vocab['Astronomy'])  #not in vocabulary, returns 0
```

    1
    2
    0

```python
for idx in range(len(Q1_test_words)):
    Q1_test[idx] = nltk.word_tokenize(Q1_test_words[idx])
    Q2_test[idx] = nltk.word_tokenize(Q2_test_words[idx])
```

```python
print('Train set has reduced to: ', len(Q1_train) )
print('Test set length: ', len(Q1_test) )
```

    Train set has reduced to:  111486
    Test set length:  10240

### 1.2 Converting a question to a tensor

You will now convert every question to a tensor, or an array of numbers, using your vocabulary built above.

```python
# Converting questions to array of integers
for i in range(len(Q1_train)):
    Q1_train[i] = [vocab[word] for word in Q1_train[i]]
    Q2_train[i] = [vocab[word] for word in Q2_train[i]]

for i in range(len(Q1_test)):
    Q1_test[i] = [vocab[word] for word in Q1_test[i]]
    Q2_test[i] = [vocab[word] for word in Q2_test[i]]
```

```python
print('first question in the train set:\n')
print(Q1_train_words[0], '\n')
print('encoded version:')
print(Q1_train[0],'\n')

print('first question in the test set:\n')
print(Q1_test_words[0], '\n')
print('encoded version:')
print(Q1_test[0])
```

    first question in the train set:

    Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?

    encoded version:
    [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]

    first question in the test set:

    How do I prepare for interviews for cse?

    encoded version:
    [32, 38, 4, 107, 65, 1015, 65, 11509, 21]

You will now split your train set into a training/validation set so that you can use it to train and evaluate your Siamese model.

```python
# Splitting the data
cut_off = int(len(Q1_train)*.8)
train_Q1, train_Q2 = Q1_train[:cut_off], Q2_train[:cut_off]
val_Q1, val_Q2 = Q1_train[cut_off: ], Q2_train[cut_off:]
print('Number of duplicate questions: ', len(Q1_train))
print("The length of the training set is:  ", len(train_Q1))
print("The length of the validation set is: ", len(val_Q1))
```

    Number of duplicate questions:  111486
    The length of the training set is:   89188
    The length of the validation set is:  22298

### 1.3 Understanding the iterator

Most of the time in Natural Language Processing, and AI in general we use batches when training our data sets. If you were to use stochastic gradient descent with one example at a time, it will take you forever to build a model. In this example, we show you how you can build a data generator that takes in $Q1$ and $Q2$ and returns a batch of size `batch_size`  in the following format $([q1_1, q1_2, q1_3, ...]$, $[q2_1, q2_2,q2_3, ...])$. The tuple consists of two arrays and each array has `batch_size` questions. Again, $q1_i$ and $q2_i$ are duplicates, but they are not duplicates with any other elements in the batch.

The command ```next(data_generator)```returns the next batch. This iterator returns the data in a format that you could directly use in your model when computing the feed-forward of your algorithm. This iterator returns a pair of arrays of questions.

### Exercise 01

**Instructions:**
Implement the data generator below. Here are some things you will need.

- While true loop.
- if `index >= len_Q1`, set the `idx` to $0$.
- The generator should return shuffled batches of data. To achieve this without modifying the actual question lists, a list containing the indexes of the questions is created. This list can be shuffled and used to get random batches everytime the index is reset.
- Append elements of $Q1$ and $Q2$ to `input1` and `input2` respectively.
- if `len(input1) == batch_size`, determine `max_len` as the longest question in `input1` and `input2`. Ceil `max_len` to a power of $2$ (for computation purposes) using the following command:  `max_len = 2**int(np.ceil(np.log2(max_len)))`.
- Pad every question by `vocab['<PAD>']` until you get the length `max_len`.
- Use yield to return `input1, input2`.
- Don't forget to reset `input1, input2`  to empty arrays at the end (data generator resumes from where it last left).

```python
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: data_generator
def data_generator(Q1, Q2, batch_size, pad=1, shuffle=True):
    """Generator function that yields batches of data

    Args:
        Q1 (list): List of transformed (to tensor) questions.
        Q2 (list): List of transformed (to tensor) questions.
        batch_size (int): Number of elements per batch.
        pad (int, optional): Pad character from the vocab. Defaults to 1.
        shuffle (bool, optional): If the batches should be randomnized or not. Defaults to True.
    Yields:
        tuple: Of the form (input1, input2) with types (numpy.ndarray, numpy.ndarray)
        NOTE: input1: inputs to your model [q1a, q2a, q3a, ...] i.e. (q1a,q1b) are duplicates
              input2: targets to your model [q1b, q2b,q3b, ...] i.e. (q1a,q2i) i!=a are not duplicates
    """

    input1 = []
    input2 = []
    idx = 0
    len_q = len(Q1)
    question_indexes = [*range(len_q)]

    if shuffle:
        rnd.shuffle(question_indexes)

    ### START CODE HERE (Replace instances of 'None' with your code) ###
    while True:
        if idx >= len_q:
            # if idx is greater than or equal to len_q, set idx accordingly
            # (Hint: look at the instructions above)
            idx = len_q
            # shuffle to get random batches if shuffle is set to True
            if shuffle:
                rnd.shuffle(question_indexes)

        # get questions at the `question_indexes[idx]` position in Q1 and Q2
        q1 = Q1[question_indexes[idx]]
        q2 = Q2[question_indexes[idx]]

        # increment idx by 1
        idx += 1
        # append q1
        input1.append(q1)
        # append q2
        input2.append(q2)
        if len(input1) == batch_size:
            # determine max_len as the longest question in input1 & input 2
            # Hint: use the `max` function.
            # take max of input1 & input2 and then max out of the two of them.
            max_len = max(max([len(q) for q in input1]),max([len(q) for q in input2]))
            # pad to power-of-2 (Hint: look at the instructions above)
            max_len = 2**int(np.ceil(np.log2(max_len)))
            b1 = []
            b2 = []
            for q1, q2 in zip(input1, input2):
                # add [pad] to q1 until it reaches max_len
                q1 = q1 + [pad] * (max_len - len(q1))
                # add [pad] to q2 until it reaches max_len
                q2 = q2 + [pad] * (max_len - len(q2))
                # append q1
                b1.append(q1)
                # append q2
                b2.append(q2)
            # use b1 and b2
            yield np.array(b1), np.array(b2)
    ### END CODE HERE ###
            # reset the batches
            input1, input2 = [], []  # reset the batches
```

```python
batch_size = 2
res1, res2 = next(data_generator(train_Q1, train_Q2, batch_size))
print("First questions  : ",'\n', res1, '\n')
print("Second questions : ",'\n', res2)
```

    First questions  :
     [[  30   87   78  134 2132 1981   28   78  594   21    1    1    1    1
         1    1]
     [  30   55   78 3541 1460   28   56  253   21    1    1    1    1    1
         1    1]]

    Second questions :
     [[  30  156   78  134 2132 9508   21    1    1    1    1    1    1    1
         1    1]
     [  30  156   78 3541 1460  131   56  253   21    1    1    1    1    1
         1    1]]

**Note**: The following expected output is valid only if you run the above test cell **_once_** (first time). The output will change on each execution.

If you think your implementation is correct and it is not matching the output, make sure to restart the kernel and run all the cells from the top again.

**Expected Output:**
```CPP
First questions  :
 [[  30   87   78  134 2132 1981   28   78  594   21    1    1    1    1
     1    1]
 [  30   55   78 3541 1460   28   56  253   21    1    1    1    1    1
     1    1]]

Second questions :
 [[  30  156   78  134 2132 9508   21    1    1    1    1    1    1    1
     1    1]
 [  30  156   78 3541 1460  131   56  253   21    1    1    1    1    1
     1    1]]
```
Now that you have your generator, you can just call it and it will return tensors which correspond to your questions in the Quora data set.
Now you can go ahead and start building your neural network.

# Part 2: Defining the Siamese model

### 2.1 Understanding Siamese Network
A Siamese network is a neural network which uses the same weights while working in tandem on two different input vectors to compute comparable output vectors.The Siamese network you are about to implement looks like this:

You get the question embedding, run it through an LSTM layer, normalize $v_1$ and $v_2$, and finally use a triplet loss (explained below) to get the corresponding cosine similarity for each pair of questions. As usual, you will start by importing the data set. The triplet loss makes use of a baseline (anchor) input that is compared to a positive (truthy) input and a negative (falsy) input. The distance from the baseline (anchor) input to the positive (truthy) input is minimized, and the distance from the baseline (anchor) input to the negative (falsy) input is maximized. In math equations, you are trying to maximize the following.

$$\mathcal{L}(A, P, N)=\max \left(\|\mathrm{f}(A)-\mathrm{f}(P)\|^{2}-\|\mathrm{f}(A)-\mathrm{f}(N)\|^{2}+\alpha, 0\right)$$

$A$ is the anchor input, for example $q1_1$, $P$ the duplicate input, for example, $q2_1$, and $N$ the negative input (the non duplicate question), for example $q2_2$.

$\alpha$ is a margin; you can think about it as a safety net, or by how much you want to push the duplicates from the non duplicates.

### Exercise 02

**Instructions:** Implement the `Siamese` function below. You should be using all the objects explained below.

To implement this model, you will be using `trax`. Concretely, you will be using the following functions.

- `tl.Serial`: Combinator that applies layers serially (by function composition) allows you set up the overall structure of the feedforward. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.combinators.Serial) / [source code](https://github.com/google/trax/blob/1372b903bb66b0daccee19fd0b1fdf44f659330b/trax/layers/combinators.py#L26)
    - You can pass in the layers as arguments to `Serial`, separated by commas.
    - For example: `tl.Serial(tl.Embeddings(...), tl.Mean(...), tl.Dense(...), tl.LogSoftmax(...))`

-  `tl.Embedding`: Maps discrete tokens to vectors. It will have shape (vocabulary length X dimension of output vectors). The dimension of output vectors (also called d_feature) is the number of elements in the word embedding. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.core.Embedding) / [source code](https://github.com/google/trax/blob/1372b903bb66b0daccee19fd0b1fdf44f659330b/trax/layers/core.py#L113)
    - `tl.Embedding(vocab_size, d_feature)`.
    - `vocab_size` is the number of unique words in the given vocabulary.
    - `d_feature` is the number of elements in the word embedding (some choices for a word embedding size range from 150 to 300, for example).

-  `tl.LSTM` The LSTM layer. It leverages another Trax layer called [`LSTMCell`](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.rnn.LSTMCell). The number of units should be specified and should match the number of elements in the word embedding. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.rnn.LSTM) / [source code](https://github.com/google/trax/blob/1372b903bb66b0daccee19fd0b1fdf44f659330b/trax/layers/rnn.py#L87)
    - `tl.LSTM(n_units)` Builds an LSTM layer of n_units.

- `tl.Mean`: Computes the mean across a desired axis. Mean uses one tensor axis to form groups of values and replaces each group with the mean value of that group. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.core.Mean) / [source code](https://github.com/google/trax/blob/1372b903bb66b0daccee19fd0b1fdf44f659330b/trax/layers/core.py#L276)
    - `tl.Mean(axis=1)` mean over columns.

- `tl.Fn` Layer with no weights that applies the function f, which should be specified using a lambda syntax. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.base.Fn) / [source doce](https://github.com/google/trax/blob/70f5364dcaf6ec11aabbd918e5f5e4b0f5bfb995/trax/layers/base.py#L576)
    - $x$ -> This is used for cosine similarity.
    - `tl.Fn('Normalize', lambda x: normalize(x))` Returns a layer with no weights that applies the function `f`

- `tl.parallel`: It is a combinator layer (like `Serial`) that applies a list of layers in parallel to its inputs. [docs](https://trax-ml.readthedocs.io/en/latest/trax.layers.html#trax.layers.combinators.Parallel) / [source code](https://github.com/google/trax/blob/37aba571a89a8ad86be76a569d0ec4a46bdd8642/trax/layers/combinators.py#L152)

```python
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: Siamese
def Siamese(vocab_size=len(vocab), d_model=128, mode='train'):
    """Returns a Siamese model.

    Args:
        vocab_size (int, optional): Length of the vocabulary. Defaults to len(vocab).
        d_model (int, optional): Depth of the model. Defaults to 128.
        mode (str, optional): 'train', 'eval' or 'predict', predict mode is for fast inference. Defaults to 'train'.

    Returns:
        trax.layers.combinators.Parallel: A Siamese model.
    """

    def normalize(x):  # normalizes the vectors to have L2 norm 1
        return x / fastnp.sqrt(fastnp.sum(x * x, axis=-1, keepdims=True))

    ### START CODE HERE (Replace instances of 'None' with your code) ###
    q_processor = tl.Serial(  # Processor will run on Q1 and Q2.
        tl.Embedding(vocab_size, d_model), # Embedding layer
        tl.LSTM(d_model), # LSTM layer
        tl.Mean(axis=1), # Mean over columns
        tl.Fn('Normalize', lambda x: normalize(x))  # Apply normalize function
    )  # Returns one vector of shape [batch_size, d_model].

    ### END CODE HERE ###

    # Run on Q1 and Q2 in parallel.
    model = tl.Parallel(q_processor, q_processor)
    return model

```

Setup the Siamese network model

```python
# check your model
model = Siamese()
print(model)
```

    Parallel_in2_out2[
      Serial[
        Embedding_41699_128
        LSTM_128
        Mean
        Normalize
      ]
      Serial[
        Embedding_41699_128
        LSTM_128
        Mean
        Normalize
      ]
    ]

**Expected output:**

```CPP
Parallel_in2_out2[
  Serial[
    Embedding_41699_128
    LSTM_128
    Mean
    Normalize
  ]
  Serial[
    Embedding_41699_128
    LSTM_128
    Mean
    Normalize
  ]
]
```

### 2.2 Hard  Negative Mining

You will now implement the `TripletLoss`.

As explained in the lecture, loss is composed of two terms. One term utilizes the mean of all the non duplicates, the second utilizes the *closest negative*. Our loss expression is then:

\begin{align}
 \mathcal{Loss_1(A,P,N)} &=\max \left( -cos(A,P)  + mean_{neg} +\alpha, 0\right) \\
 \mathcal{Loss_2(A,P,N)} &=\max \left( -cos(A,P)  + closest_{neg} +\alpha, 0\right) \\
\mathcal{Loss(A,P,N)} &= mean(Loss_1 + Loss_2) \\
\end{align}

Further, two sets of instructions are provided. The first set provides a brief description of the task. If that set proves insufficient, a more detailed set can be displayed.

### Exercise 03

**Instructions (Brief):** Here is a list of things you should do:

- As this will be run inside trax, use `fastnp.xyz` when using any `xyz` numpy function
- Use `fastnp.dot` to calculate the similarity matrix $v_1v_2^T$ of dimension `batch_size` x `batch_size`
- Take the score of the duplicates on the diagonal `fastnp.diagonal`
- Use the `trax` functions `fastnp.eye` and `fastnp.maximum` for the identity matrix and the maximum.

    More Detailed Instructions

We'll describe the algorithm using a detailed example. Below, V1, V2 are the output of the normalization blocks in our model. Here we will use a batch_size of 4 and a d_model of 3. As explained in lecture, the inputs, Q1, Q2 are arranged so that corresponding inputs are duplicates while non-corresponding entries are not. The outputs will have the same pattern.

This testcase arranges the outputs, v1,v2, to highlight different scenarios. Here, the first two outputs V1[0], V2[0] match exactly - so the model is generating the same vector for Q1[0] and Q2[0] inputs. The second outputs differ, circled in orange, we set, V2[1] is set to match V2[**2**], simulating a model which is generating very poor results. V1[3] and V2[3] match exactly again while V1[4] and V2[4] are set to be exactly wrong - 180 degrees from each other, circled in blue.

The first step is to compute the cosine similarity matrix or `score` in the code. As explained in lecture, this is $$V_1 V_2^T$$ This is generated with `fastnp.dot`.

The clever arrangement of inputs creates the data needed for positive *and* negative examples without having to run all pair-wise combinations. Because Q1[n] is a duplicate of only Q2[n], other combinations are explicitly created negative examples or *Hard Negative* examples. The matrix multiplication efficiently produces the cosine similarity of all positive/negative combinations as shown above on the left side of the diagram. 'Positive' are the results of duplicate examples and 'negative' are the results of explicitly created negative examples. The results for our test case are as expected, V1[0]V2[0] match producing '1' while our other 'positive' cases (in green) don't match well, as was arranged. The V2[2] was set to match V1[3] producing a poor match at `score[2,2]` and an undesired 'negative' case of a '1' shown in grey.

With the similarity matrix (`score`) we can begin to implement the loss equations. First, we can extract $$cos(A,P)$$ by utilizing `fastnp.diagonal`. The goal is to grab all the green entries in the diagram above. This is `positive` in the code.

Next, we will create the *closest_negative*. This is the nonduplicate entry in V2 that is closest (has largest cosine similarity) to an entry in V1. Each row, n, of `score` represents all comparisons of the results of Q1[n] vs Q2[x] within a batch. A specific example in our testcase is row `score[2,:]`. It has the cosine similarity of V1[2] and V2[x]. The *closest_negative*, as was arranged, is V2[2] which has a score of 1. This is the maximum value of the 'negative' entries (blue entries in the diagram).

To implement this, we need to pick the maximum entry on a row of `score`, ignoring the 'positive'/green entries. To avoid selecting the 'positive'/green entries, we can make them larger negative numbers. Multiply `fastnp.eye(batch_size)` with 2.0 and subtract it out of `scores`. The result is `negative_without_positive`. Now we can use `fastnp.max`, row by row (axis=1), to select the maximum which is `closest_negative`.

Next, we'll create *mean_negative*. As the name suggests, this is the mean of all the 'negative'/blue values in `score` on a row by row basis. We can use `fastnp.eye(batch_size)` and a constant, this time to create a mask with zeros on the diagonal. Element-wise multiply this with `score` to get just the 'negative values. This is `negative_zero_on_duplicate` in the code. Compute the mean by using `fastnp.sum` on `negative_zero_on_duplicate` for `axis=1` and divide it by `(batch_size - 1)` . This is `mean_negative`.

Now, we can compute loss using the two equations above and `fastnp.maximum`. This will form `triplet_loss1` and `triplet_loss2`.

`triple_loss` is the `fastnp.mean` of the sum of the two individual losses.

Once you have this code matching the expected results, you can clip out the section between ### START CODE HERE and ### END CODE HERE it out and insert it into TripletLoss below.

<\details>

```python
# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: TripletLossFn
def TripletLossFn(v1, v2, margin=0.25):
    """Custom Loss function.

    Args:
        v1 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to Q1.
        v2 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to Q2.
        margin (float, optional): Desired margin. Defaults to 0.25.

    Returns:
        jax.interpreters.xla.DeviceArray: Triplet Loss.
    """
    ### START CODE HERE (Replace instances of 'None' with your code) ###

    # use fastnp to take the dot product of the two batches (don't forget to transpose the second argument)
    scores = fastnp.dot(v1, v2.T)  # pairwise cosine sim
    # calculate new batch size
    batch_size = len(scores)
    # use fastnp to grab all postive `diagonal` entries in `scores`
    positive = fastnp.diagonal(scores)  # the positive ones (duplicates)
    # multiply `fastnp.eye(batch_size)` with 2.0 and subtract it out of `scores`
    negative_without_positive = scores - 2.0 * fastnp.eye(batch_size)
    # take the row by row `max` of `negative_without_positive`.
    # Hint: negative_without_positive.max(axis = [?])
    closest_negative = negative_without_positive.max(axis=1)
    # subtract `fastnp.eye(batch_size)` out of 1.0 and do element-wise multiplication with `scores`
    negative_zero_on_duplicate = scores * (1.0 - fastnp.eye(batch_size))
    # use `fastnp.sum` on `negative_zero_on_duplicate` for `axis=1` and divide it by `(batch_size - 1)`
    mean_negative = np.sum(negative_zero_on_duplicate, axis=1) / (batch_size-1)
    # compute `fastnp.maximum` among 0.0 and `A`
    # A = subtract `positive` from `margin` and add `closest_negative`
    triplet_loss1 = fastnp.maximum(0.0, margin - positive + closest_negative)
    # compute `fastnp.maximum` among 0.0 and `B`
    # B = subtract `positive` from `margin` and add `mean_negative`
    triplet_loss2 = fastnp.maximum(0.0, margin - positive + mean_negative)
    # add the two losses together and take the `fastnp.mean` of it
    triplet_loss = fastnp.mean(triplet_loss1 + triplet_loss2)

    ### END CODE HERE ###

    return triplet_loss
```

```python
v1 = np.array([[0.26726124, 0.53452248, 0.80178373],[0.5178918 , 0.57543534, 0.63297887]])
v2 = np.array([[ 0.26726124,  0.53452248,  0.80178373],[-0.5178918 , -0.57543534, -0.63297887]])
TripletLossFn(v2,v1)
print("Triplet Loss:", TripletLossFn(v2,v1))
```

    Triplet Loss: 0.5

**Expected Output:**
```CPP
Triplet Loss: 0.5
```

To make a layer out of a function with no trainable variables, use `tl.Fn`.

```python
from functools import partial
def TripletLoss(margin=0.25):
    triplet_loss_fn = partial(TripletLossFn, margin=margin)
    return tl.Fn('TripletLoss', triplet_loss_fn)
```

# Part 3: Training

Now you are going to train your model. As usual, you have to define the cost function and the optimizer. You also have to feed in the built model. Before, going into the training, we will use a special data set up. We will define the inputs using the data generator we built above. The lambda function acts as a seed to remember the last batch that was given. Run the cell below to get the question pairs inputs.

```python
batch_size = 256
train_generator = data_generator(train_Q1, train_Q2, batch_size, vocab['<PAD>'])
val_generator = data_generator(val_Q1, val_Q2, batch_size, vocab['<PAD>'])
print('train_Q1.shape ', train_Q1.shape)
print('val_Q1.shape   ', val_Q1.shape)
```

    train_Q1.shape  (89188,)
    val_Q1.shape    (22298,)

### 3.1 Training the model

You will now write a function that takes in your model and trains it. To train your model you have to decide how many times you want to iterate over the entire data set; each iteration is defined as an `epoch`. For each epoch, you have to go over all the data, using your training iterator.

### Exercise 04

**Instructions:** Implement the `train_model` below to train the neural network above. Here is a list of things you should do, as already shown in lecture 7:

- Create `TrainTask` and `EvalTask`
- Create the training loop `trax.supervised.training.Loop`
- Pass in the following depending on the context (train_task or eval_task):
    - `labeled_data=generator`
    - `metrics=[TripletLoss()]`,
    - `loss_layer=TripletLoss()`
    - `optimizer=trax.optimizers.Adam` with learning rate of 0.01
    - `lr_schedule=lr_schedule`,
    - `output_dir=output_dir`

You will be using your triplet loss function with Adam optimizer. Please read the [trax](https://trax-ml.readthedocs.io/en/latest/trax.optimizers.html?highlight=adam#trax.optimizers.adam.Adam) documentation to get a full understanding.

This function should return a `training.Loop` object. To read more about this check the [docs](https://trax-ml.readthedocs.io/en/latest/trax.supervised.html?highlight=loop#trax.supervised.training.Loop).

```python
lr_schedule = trax.lr.warmup_and_rsqrt_decay(400, 0.01)

# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: train_model
def train_model(Siamese, TripletLoss, lr_schedule, train_generator=train_generator, val_generator=val_generator, output_dir='model/'):
    """Training the Siamese Model

    Args:
        Siamese (function): Function that returns the Siamese model.
        TripletLoss (function): Function that defines the TripletLoss loss function.
        lr_schedule (function): Trax multifactor schedule function.
        train_generator (generator, optional): Training generator. Defaults to train_generator.
        val_generator (generator, optional): Validation generator. Defaults to val_generator.
        output_dir (str, optional): Path to save model to. Defaults to 'model/'.

    Returns:
        trax.supervised.training.Loop: Training loop for the model.
    """
    output_dir = os.path.expanduser(output_dir)

    ### START CODE HERE (Replace instances of 'None' with your code) ###

    train_task = training.TrainTask(
        labeled_data=train_generator,            # Use generator (train)
        loss_layer=TripletLoss(),                # Use triplet loss. Don't forget to instantiate this object
        optimizer=trax.optimizers.Adam(0.01),    # Don't forget to add the learning rate parameter
        lr_schedule=lr_schedule,                  # Use Trax multifactor schedule function
    )

    eval_task = training.EvalTask(
        labeled_data=val_generator,       # Use generator (val)
        metrics=[TripletLoss()],          # Use triplet loss. Don't forget to instantiate this object
    )

    ### END CODE HERE ###

    training_loop = training.Loop(Siamese(),
                                  train_task,
                                  eval_task=eval_task,
                                  output_dir=output_dir)

    return training_loop
```

```python
train_steps = 5
training_loop = train_model(Siamese, TripletLoss, lr_schedule)
training_loop.run(train_steps)
```

    Step      1: train TripletLoss |  0.49954823
    Step      1: eval  TripletLoss |  0.49950948

The model was only trained for 5 steps due to the constraints of this environment. For the rest of the assignment you will be using a pretrained model but now you should understand how the training can be done using Trax.

# Part 4:  Evaluation

### 4.1 Evaluating your siamese network

In this section you will learn how to evaluate a Siamese network. You will first start by loading a pretrained model and then you will use it to predict.

```python
# Loading in the saved model
model = Siamese()
model.init_from_file('model.pkl.gz')
```

### 4.2 Classify
To determine the accuracy of the model, we will utilize the test set that was configured earlier. While in training we used only positive examples, the test data, Q1_test, Q2_test and y_test, is setup as pairs of questions, some of which are duplicates some are not.
This routine will run all the test question pairs through the model, compute the cosine simlarity of each pair, threshold it and compare the result to  y_test - the correct response from the data set. The results are accumulated to produce an accuracy.

### Exercise 05

**Instructions**
 - Loop through the incoming data in batch_size chunks
 - Use the data generator to load q1, q2 a batch at a time. **Don't forget to set shuffle=False!**
 - copy a batch_size chunk of y into y_test
 - compute v1, v2 using the model
 - for each element of the batch
        - compute the cos similarity of each pair of entries, v1[j],v2[j]
        - determine if d > threshold
        - increment accuracy if that result matches the expected results (y_test[j])
 - compute the final accuracy and return

Due to some limitations of this environment, running classify multiple times may result in the kernel failing. If that happens *Restart Kernal & clear output* and then run from the top. During development, consider using a smaller set of data to reduce the number of calls to model().

```python
# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: classify
def classify(test_Q1, test_Q2, y, threshold, model, vocab, data_generator=data_generator, batch_size=64):
    """Function to test the accuracy of the model.

    Args:
        test_Q1 (numpy.ndarray): Array of Q1 questions.
        test_Q2 (numpy.ndarray): Array of Q2 questions.
        y (numpy.ndarray): Array of actual target.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        vocab (collections.defaultdict): The vocabulary used.
        data_generator (function): Data generator function. Defaults to data_generator.
        batch_size (int, optional): Size of the batches. Defaults to 64.

    Returns:
        float: Accuracy of the model.
    """
    accuracy = 0
    ### START CODE HERE (Replace instances of 'None' with your code) ###
    for i in range(0, len(test_Q1), batch_size):
        # Call the data generator (built in Ex 01) with shuffle=False using next()
        # use batch size chuncks of questions as Q1 & Q2 arguments of the data generator. e.g x[i:i + batch_size]
        # Hint: use `vocab['<PAD>']` for the `pad` argument of the data generator
        q1, q2 = next(data_generator(test_Q1[i:i + batch_size], test_Q2[i:i + batch_size], batch_size, vocab['<PAD>'], shuffle=False))
        # use batch size chuncks of actual output targets (same syntax as example above)
        y_test = y[i:i + batch_size]
        # Call the model
        v1, v2 = model((q1, q2))

        for j in range(batch_size):
            # take dot product to compute cos similarity of each pair of entries, v1[j], v2[j]
            # don't forget to transpose the second argument
            d = np.dot(v1[j], v2[j].T)
            # is d greater than the threshold?
            res = d > threshold
            # increment accurancy if y_test is equal `res`
            accuracy += (y_test[j] == res)
    # compute accuracy using accuracy and total length of test questions
    accuracy = accuracy / len(test_Q1)
    ### END CODE HERE ###

    return accuracy
```

```python
# this takes around 1 minute
accuracy = classify(Q1_test,Q2_test, y_test, 0.7, model, vocab, batch_size = 512)
print("Accuracy", accuracy)
```

    Accuracy 0.69091796875

**Expected Result**
Accuracy ~0.69

# Part 5: Testing with your own questions

In this section you will test the model with your own questions. You will write a function `predict` which takes two questions as input and returns $1$ or $0$ depending on whether the question pair is a duplicate or not.

But first, we build a reverse vocabulary that allows to map encoded questions back to words:

Write a function `predict`that takes in two questions, the model, and the vocabulary and returns whether the questions are duplicates ($1$) or not duplicates ($0$) given a similarity threshold.

### Exercise 06

**Instructions:**
- Tokenize your question using `nltk.word_tokenize`
- Create Q1,Q2 by encoding your questions as a list of numbers using vocab
- pad Q1,Q2 with next(data_generator([Q1], [Q2],1,vocab['<PAD>']))
- use model() to create v1, v2
- compute the cosine similarity (dot product) of v1, v2
- compute res by comparing d to the threshold

```python
# UNQ_C6 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: predict
def predict(question1, question2, threshold, model, vocab, data_generator=data_generator, verbose=False):
    """Function for predicting if two questions are duplicates.

    Args:
        question1 (str): First question.
        question2 (str): Second question.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        vocab (collections.defaultdict): The vocabulary used.
        data_generator (function): Data generator function. Defaults to data_generator.
        verbose (bool, optional): If the results should be printed out. Defaults to False.

    Returns:
        bool: True if the questions are duplicates, False otherwise.
    """
    ### START CODE HERE (Replace instances of 'None' with your code) ###
    # use `nltk` word tokenize function to tokenize
    q1 = nltk.word_tokenize(question1)  # tokenize
    q2 = nltk.word_tokenize(question2)  # tokenize
    Q1, Q2 = [], []
    for word in q1:  # encode q1
        # increment by checking the 'word' index in `vocab`
        Q1 += [vocab[word]]
    for word in q2:  # encode q2
        # increment by checking the 'word' index in `vocab`
        Q2 += [vocab[word]]

    # Call the data generator (built in Ex 01) using next()
    # pass [Q1] & [Q2] as Q1 & Q2 arguments of the data generator. Set batch size as 1
    # Hint: use `vocab['<PAD>']` for the `pad` argument of the data generator
    Q1, Q2 = next(data_generator([Q1], [Q2], 1, vocab['<PAD>']))
    # Call the model
    v1, v2 = model((Q1, Q2))
    # take dot product to compute cos similarity of each pair of entries, v1, v2
    # don't forget to transpose the second argument
    d = np.dot(v1[0], v2[0].T)
    # is d greater than the threshold?
    res = d > threshold

    ### END CODE HERE ###

    if(verbose):
        print("Q1  = ", Q1, "\nQ2  = ", Q2)
        print("d   = ", d)
        print("res = ", res)

    return res
```

```python
# Feel free to try with your own questions
question1 = "When will I see you?"
question2 = "When can I see you again?"
# 1 means it is duplicated, 0 otherwise
predict(question1 , question2, 0.7, model, vocab, verbose = True)
```

    Q1  =  [[585  76   4  46  53  21   1   1]]
    Q2  =  [[ 585   33    4   46   53 7280   21    1]]
    d   =  0.88113236
    res =  True

    True

##### Expected Output
If input is:
```CPP
question1 = "When will I see you?"
question2 = "When can I see you again?"
```

Output is (d may vary a bit):
```CPP
Q1  =  [[585  76   4  46  53  21   1   1]]
Q2  =  [[ 585   33    4   46   53 7280   21    1]]
d   =  0.88113236
res =  True
True
```

```python
# Feel free to try with your own questions
question1 = "Do they enjoy eating the dessert?"
question2 = "Do they like hiking in the desert?"
# 1 means it is duplicated, 0 otherwise
predict(question1 , question2, 0.7, model, vocab, verbose=True)
```

    Q1  =  [[  443  1145  3159  1169    78 29017    21     1]]
    Q2  =  [[  443  1145    60 15302    28    78  7431    21]]
    d   =  0.477536
    res =  False

    False

##### Expected output

If input is:
```CPP
question1 = "Do they enjoy eating the dessert?"
question2 = "Do they like hiking in the desert?"
```

Output  (d may vary a bit):

```CPP
Q1  =  [[  443  1145  3159  1169    78 29017    21     1]]
Q2  =  [[  443  1145    60 15302    28    78  7431    21]]
d   =  0.477536
res =  False
False
```

You can see that the Siamese network is capable of catching complicated structures. Concretely it can identify question duplicates although the questions do not have many words in common.

###   On Siamese networks

Siamese networks are important and useful. Many times there are several questions that are already asked in quora, or other platforms and you can use Siamese networks to avoid question duplicates.

Congratulations, you have now built a powerful system that can recognize question duplicates. In the next course we will use transformers for machine translation, summarization, question answering, and chatbots.

3 - Natural Language Processing with Sequence Models/Week 4/C3W4_A1_Question duplicates.ipynb


# Tutorial 2: Introduction to GANs and Density Ratio Estimation Perspective of GANs

**Week 2, Day 5: Generative Models**

**By Neuromatch Academy**

__Content creators:__ Kai Xu, Seungwook Han, Akash Srivastava

__Content reviewers:__ Polina Turishcheva, Melvin Selim Atay, Hadi Vafaei, Deepak Raya, Charles J. Edelson

__Content editors:__ Charles J. Edelson, Spiros Chavlis

__Production editors:__ Arush Tagade, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---

## Tutorial Objectives

The goal of this tutorial is two-fold; first you will be introduced to GANs training, and you will be able to understand how GANs are connected to other generative models that we have been before.

By the end of the first part of this tutorial you will be able to:
- Understand, at a high level, how GANs are implemented.
- Understand the training dynamics of GANs.
- Know about a few failure modes of GAN training.
- Understand density ratio estimation using a binary classifier
- Understand the connection between GANs and other generative models.
- Implement a GAN.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/dftym/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# Imports
import torch
import numpy as np
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt

from time import time
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

ld_true = [-7.0066e-01, -2.6368e-01, -2.4250e+00, -2.0247e+00, -1.1795e+00,
        -4.5558e-01, -7.1316e-01, -1.0932e-01, -7.8608e-01, -4.5838e-01,
        -1.0530e+00, -9.1201e-01, -3.8020e+00, -1.7787e+00, -1.2246e+00,
        -6.5677e-01, -3.6001e-01, -2.2313e-01, -1.8262e+00, -1.2649e+00,
        -3.8330e-01, -8.8619e-02, -9.2357e-01, -1.3450e-01, -8.6891e-01,
        -5.9257e-01, -4.8415e-02, -3.3197e+00, -1.6862e+00, -9.8506e-01,
        -1.1871e+00, -7.0422e-02, -1.7378e+00, -1.3099e+00, -1.8926e+00,
        -3.4508e+00, -1.5696e+00, -7.2787e-02, -3.2420e-01, -2.9795e-01,
        -6.4189e-01, -1.4120e+00, -5.3684e-01, -3.4066e+00, -1.9753e+00,
        -1.4178e+00, -2.0399e-01, -2.3173e-01, -1.2792e+00, -7.2990e-01,
        -1.9872e-01, -2.9378e-03, -3.5890e-01, -5.6643e-01, -1.8003e-01,
        -1.5818e+00, -5.2227e-01, -2.1862e+00, -1.8743e+00, -1.4200e+00,
        -3.1988e-01, -3.5513e-01, -1.5905e+00, -4.2916e-01, -2.5556e-01,
        -8.2807e-01, -6.5568e-01, -4.8475e-01, -2.1049e-01, -2.0104e-02,
        -2.1655e+00, -1.1496e+00, -3.6168e-01, -8.9624e-02, -6.7098e-02,
        -6.0623e-02, -5.1165e-01, -2.7302e+00, -6.0514e-01, -1.6756e+00,
        -3.3807e+00, -5.7368e-02, -1.2763e-01, -6.6959e+00, -5.2157e-01,
        -8.7762e-01, -8.7295e-01, -1.3052e+00, -3.6777e-01, -1.5904e+00,
        -3.8083e-01, -2.8388e-01, -1.5323e-01, -3.7549e-01, -5.2722e+00,
        -1.7393e+00, -2.8814e-01, -5.0310e-01, -2.2077e+00, -1.5507e+00,
        -6.8569e-01, -1.4620e+00, -9.2639e-02, -1.4160e-01, -3.6734e-01,
        -1.0053e+00, -6.7353e-01, -2.2676e+00, -6.0812e-01, -1.0005e+00,
        -4.2908e-01, -5.1369e-01, -2.2579e-02, -1.8496e-01, -3.4798e-01,
        -7.3089e-01, -1.1962e+00, -1.6095e+00, -1.7558e-01, -3.3166e-01,
        -1.1445e+00, -2.4674e+00, -5.0600e-01, -2.0727e+00, -5.4371e-01,
        -8.0499e-01, -3.0521e+00, -3.6835e-02, -2.0485e-01, -4.6747e-01,
        -3.6399e-01, -2.6883e+00, -1.9348e-01, -3.1448e-01, -1.6332e-01,
        -3.2233e-02, -2.3336e-01, -2.6564e+00, -1.2841e+00, -1.3561e+00,
        -7.4717e-01, -2.7926e-01, -8.7849e-01, -3.3715e-02, -1.4933e-01,
        -2.7738e-01, -1.6899e+00, -1.5758e+00, -3.2608e-01, -6.5770e-01,
        -1.7136e+00, -5.8316e+00, -1.1988e+00, -8.3828e-01, -1.8033e+00,
        -2.3017e-01, -8.9936e-01, -1.1917e-01, -1.6659e-01, -2.7669e-01,
        -1.2955e+00, -1.2076e+00, -2.2793e-01, -1.0528e+00, -1.4894e+00,
        -5.7428e-01, -7.3208e-01, -9.5673e-01, -1.6617e+00, -3.9169e+00,
        -1.2182e-01, -3.8092e-01, -1.1924e+00, -2.4566e+00, -2.7350e+00,
        -2.8332e+00, -9.1506e-01, -6.7432e-02, -7.8965e-01, -2.0727e-01,
        -3.4615e-02, -2.8868e+00, -2.1218e+00, -1.2368e-03, -9.0038e-01,
        -5.3746e-01, -5.4080e-01, -3.1625e-01, -1.1786e+00, -2.2797e-01,
        -1.1498e+00, -1.3978e+00, -1.9515e+00, -1.1614e+00, -5.1456e-03,
        -1.9316e-01, -1.3849e+00, -9.2799e-01, -1.1649e-01, -2.3837e-01]

def plotting_ld(ld, true=ld_true):
  fig, ax = plt.subplots(figsize=(7, 7))
  ax.plot([-6, 1], [-6, 1], label="Ground Truth")
  ax.scatter(true, ld, marker="x",
             label="Your implementation")
  ax.set_xlabel("Loss from oracle implementation")
  ax.set_ylabel("Loss from your implementation")
  ax.legend()
  ax.set_title("Discriminator Loss")

lg_true = [-7.0066e-01, -2.6368e-01, -2.4250e+00, -2.0247e+00, -1.1795e+00,
        -4.5558e-01, -7.1316e-01, -1.0932e-01, -7.8608e-01, -4.5838e-01,
        -1.0530e+00, -9.1201e-01, -3.8020e+00, -1.7787e+00, -1.2246e+00,
        -6.5677e-01, -3.6001e-01, -2.2313e-01, -1.8262e+00, -1.2649e+00,
        -3.8330e-01, -8.8619e-02, -9.2357e-01, -1.3450e-01, -8.6891e-01,
        -5.9257e-01, -4.8415e-02, -3.3197e+00, -1.6862e+00, -9.8506e-01,
        -1.1871e+00, -7.0422e-02, -1.7378e+00, -1.3099e+00, -1.8926e+00,
        -3.4508e+00, -1.5696e+00, -7.2787e-02, -3.2420e-01, -2.9795e-01,
        -6.4189e-01, -1.4120e+00, -5.3684e-01, -3.4066e+00, -1.9753e+00,
        -1.4178e+00, -2.0399e-01, -2.3173e-01, -1.2792e+00, -7.2990e-01,
        -1.9872e-01, -2.9378e-03, -3.5890e-01, -5.6643e-01, -1.8003e-01,
        -1.5818e+00, -5.2227e-01, -2.1862e+00, -1.8743e+00, -1.4200e+00,
        -3.1988e-01, -3.5513e-01, -1.5905e+00, -4.2916e-01, -2.5556e-01,
        -8.2807e-01, -6.5568e-01, -4.8475e-01, -2.1049e-01, -2.0104e-02,
        -2.1655e+00, -1.1496e+00, -3.6168e-01, -8.9624e-02, -6.7098e-02,
        -6.0623e-02, -5.1165e-01, -2.7302e+00, -6.0514e-01, -1.6756e+00,
        -3.3807e+00, -5.7368e-02, -1.2763e-01, -6.6959e+00, -5.2157e-01,
        -8.7762e-01, -8.7295e-01, -1.3052e+00, -3.6777e-01, -1.5904e+00,
        -3.8083e-01, -2.8388e-01, -1.5323e-01, -3.7549e-01, -5.2722e+00,
        -1.7393e+00, -2.8814e-01, -5.0310e-01, -2.2077e+00, -1.5507e+00]

def plotting_lg(lg, true=lg_true):
  fig, ax = plt.subplots(figsize=(7, 7))
  ax.plot([-6, 1], [-6, 1], label="Ground Truth")
  ax.scatter(true, lg, marker="x",
             label="Your implementation")
  ax.set_xlabel("Loss from oracle implementation")
  ax.set_ylabel("Loss from your implementation")
  ax.legend()
  ax.set_title("Generator loss")

def plotting_ratio_impl(ax, x_real, x_fake, ratio, yscale="linear"):
  dist_p = torch.distributions.normal.Normal(loc=0, scale=1)
  dist_q = torch.distributions.normal.Normal(loc=-2, scale=1)
  x = torch.linspace(-3, 5, 100)
  prob_p = torch.exp(dist_p.log_prob(x))
  prob_q = torch.exp(dist_q.log_prob(x))
  trueRatio = prob_p / prob_q
  ax.plot(x, trueRatio, label="True tatio")

  x = torch.cat([x_real, x_fake])
  ax.scatter(x[:,0][::10], ratio[:,0][::10], marker="x",
             label="Ratio from discriminator")
  ax.hist(x_real[:,0], density=True, bins=50, histtype="step", label="Real")
  ax.hist(x_fake[:,0], density=True, bins=50, histtype="step", label="Fake")
  ax.set_yscale(yscale)
  title = "Densities and the ratio from discriminator"
  if yscale == "log":
    title += " in log scale"
  ax.set_title(title)
  ax.legend()

def plotting_ratio(x_real, x_fake, ratio):
  fig, axes = plt.subplots(1, 2, figsize=(2 * 7, 7))
  plotting_ratio_impl(axes[0], x_real, x_fake, ratio, yscale="linear")
  plotting_ratio_impl(axes[1], x_real, x_fake, ratio, yscale="log")

class Interactive:
  def display_widgets(self):
    for widget in self.widgets:
      display(widget)

  def __init__(self, widgets, handler):
    def handler_with_extra_steps(b):
      clear_output(wait=True)
      handler(*map(lambda w: w.value, widgets))
      self.display_widgets()
    self.widgets = widgets
    for widget in self.widgets:
      widget.observe(handler_with_extra_steps,
                     names=['value'])
    handler(*map(lambda w: w.value, widgets))
    self.display_widgets()

# Using Interactive
# All widgets: https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html

def make_plot(xmax, title, ftype):
  fig, ax = plt.subplots()
  x = np.linspace(-2, xmax, 100)
  if ftype == "sin":
    y = np.sin(x)
  if ftype == "cos":
    y = np.cos(x)
  if ftype == "tanh":
    y = np.tanh(x)
  ax.scatter(x, y)
  ax.set_xlim(-2.1, 2.1)
  ax.set_ylim(-2, 2)
  if title:
    ax.set_title(f"Range from -1 to {xmax}")
  return fig
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

---
# Section 1: How to train GANs.

```python
# @title Video 1: Generative Adversarial Networks
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV19V411H72M", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"fCGnJPW2RKM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

GANs consist two networks: A critic or discriminator (`disc`) and a generator (`gen`) that are trained by alternating between the following two steps:
- In step 1, we update the parameters (`disc.params`) of the discriminator by backpropagating through the discriminator loss (BCE loss) `disc.loss`.
- In step 2, we update the parameters (`gen.params`) of the generator by backpropagating through the generator loss, `gen.loss` (-1 * BCE loss).

We will now implement a simple GAN training loop!

## Coding Exercise 1: The GAN training loop

To get you started we have implemented a simple GAN in pseudocode. All you have to do is to implement the training loop.

__Your goal__ is to arrange the functions given below in the correct order in the `train_gan_iter` function
- `disc.loss(x_real, x_fake)`: Discriminator loss
- `disc.classify(x)`: Classify `x` as real or fake
- `gen.loss(x_fake, disc_fn)`: Generator loss
- `disc_fn(x)` is a function to check `x` is real or fake.
- `gen.sample(num_samples)`: Generate samples from the generator
- `backprop(loss, model)`: Compute gradient of `loss` wrt `model`
- `model` is either `disc` or `gen`

We have already taken care of most of these functions. So you only have to figure out the placement of `disc.loss` and `gen.loss` functions.

__We highly recommend studying `train_gan_iter` function to understand how the GAN training loop is structured.__

```python
# @markdown *Execute this cell to enable helper functions*

def get_data():
  return "get_data"

class Disc:

  def loss(self, x_real, x_fake):
    assert x_real == "get_data" and x_fake == "gen.sample","Inputs to disc.loss is wrong"

  def classify(self, x):
    return "disc.classify"

class Gen:

  def loss(self, x_fake, disc_fn):
    assert x_fake == "gen.sample" and disc_fn(None) == "disc.classify", "Inputs to gen.loss is wrong"

  def sample(self, num_samples):
    return "gen.sample"

def backprop(loss, model):
  pass

def update(model, grad):
  pass
```

```python
def train_gan_iter(data, disc, gen):
  """Update the discriminator (`disc`) and the generator (`gen`) using `data`

  Args:
    data (ndarray): An array of shape (N,) that contains the data
    disc (Disc): The discriminator
    gen (Gen): The generator

  Returns:
  """
  #################################################
  # Intructions for students:                        #
  # Fill out ... in the function and remove below #
  #################################################

  # Number of samples in the data batch
  num_samples = 200

  # The data is the real samples
  x_real = data

  ## Discriminator training

  # Ask the generator to generate some fake samples
  x_fake = gen.sample(num_samples)

  #################################################
  ## TODO for students: details of what they should do ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: Write code to compute disc_loss")
  #################################################
  # Compute the discriminator loss
  disc_loss = ...

  # Compute the gradient for discriminator
  disc_grad = backprop(disc_loss, disc)

  # Update the discriminator
  update(disc, disc_grad)

  ## Generator training

  # Ask the generator to generate some fake samples
  x_fake = gen.sample(num_samples)

  #################################################
  ## TODO for students: details of what they should do ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: Write code to compute gen_loss")
  #################################################
  # Compute the generator loss
  gen_loss = ...

  # Compute the gradient for generator
  gen_grad = backprop(gen_loss, gen)

  # Update the generator
  update(gen, gen_grad)

  print("Your implementation passes the check!")

data = get_data()
disc = Disc()
gen = Gen()
## Uncomment below to check your function
# train_gan_iter(data, disc, gen)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D5_GenerativeModels/solutions/W2D5_Tutorial2_Solution_dc5b38a9.py)

---
# Section 2: The difficulty of GAN training.

In this section we will develop an intuition for the training dynamics of GANs.

## Interactive Demo 2: Failure modes of GAN training

GAN training is notoriously difficult because
it is very sensitive to hyper-parameters such as learning rate and model architecture. To help you develop a sense of this here is a very simple GAN training demo that we have borrowed from [Andrej Karpathy's website](https://cs.stanford.edu/people/karpathy/gan/).

The generator $G$, pictured in red, takes inputs sampled from a uniform distribution, $z$. It attempts to transform these to match a data distribution, shown below in blue. Meanwhile, the discriminator $D$ attempts to determine whether a sample is from the data distribution or the generating distribution. In the demo, the green curve represents the output of the discriminator. Its value is high where the discriminator is more confident that a sample with that value is drawn from the data distribution.

Even though the GAN in this demo is very simple and operates in either 1D or 2D spaces, it is however very sensitive to the learning rate. Try it for yourself!

```python
# @title GAN training demo
# @markdown Make sure you execute this cell to enable the widget!

from IPython.display import IFrame
IFrame(src='https://xukai92.github.io/gan_demo/index.html', width=900, height=600)
```

## Exercise 2: What makes GANs hard to train?

You have played with the demo and it's time to think about a few questions

1. Which target is more stable to train, 1D or 2D?
2. If you keep increasing the learning rate, what happens? Does it happen in both the cases, i.e., 1D/2D targets?
3. Can you think of some drawbacks of using small learning rates?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D5_GenerativeModels/solutions/W2D5_Tutorial2_Solution_8febc070.py)

---
# Section 3: GAN Training Objective

The training objective of GANs consists of the losses for generators and discriminators respectively. In this section we will be implementing these objectives.

```python
# @title Video 2: Principles of GANs
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Fq4y1s7pf", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"bTJfZro6A9Q", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

## Section 3.1:  Discriminator Loss

The critic or the discriminator in a vanilla GAN is trained as a binary classifier using the BCE criteria. In this section, we will implement the training objective for the discriminator.

\begin{equation}
\text{BCE}_\omega = \mathbb{E}_{x \sim p}[\log(\sigma(D_\omega(x)))] + \mathbb{E}_{x \sim q}[\log(1 - \sigma(D_\omega(x)))]
\end{equation}

Here, $p$ is the data distribution and $q$ is the generator distribution. $D_\omega$ is the logit, which represents $\log \frac{p}{q}$. $\sigma$ is the sigmoid function and therfore, $\sigma(D_\omega)$ represents $\frac{p}{p+q}$.

### Coding Exercise 3.1: Implement Discriminator Loss

To get you started we have implemented a simple GAN in pseudocode and partially implemented the discriminator training objective.

**Your goal** is to complete the missing part in the training objective of the discriminator in the function `loss_disc`.

`loss_disc` also allows you evaluate the loss function on some random samples.
If your implementation is correct, you will see a plot where the loss values from your implementation will match the ground truth loss values.

In practice, given $N$ samples, we estimate BCE as

\begin{equation}
\text{BCE}_\omega = -\frac{1}{N} \sum_{i=1}^N y_i \log(\sigma(D_\omega(x_i)) + (1-y_i) \log(1-\sigma(D_\omega(x_i))).
\end{equation}

Here, $y$ is the label. $y=1$ when $x \sim p$ (real data) and $y=0$ when $x \sim q$ (i.e., fake data).

Please note, `disc.classify` = $\sigma(D_\omega)$ in `loss_disc`.

```python
# @markdown *Execute this cell to enable helper functions*

def get_data(num_samples=100, seed=0):
  set_seed(seed)
  return torch.randn([num_samples, 1])

class DummyGen:
  def sample(self, num_samples=100, seed=1):
    set_seed(seed)
    return torch.randn([num_samples, 1]) + 2

class DummyDisc:
  def classify(self, x, seed=0):
    set_seed(seed)
    return torch.rand([x.shape[0], ])
```

```python
def loss_disc(disc, x_real, x_fake):
  """Compute the discriminator loss for `x_real` and `x_fake` given `disc`

  Args:
    disc: The discriminator
    x_real (ndarray): An array of shape (N,) that contains the real samples
    x_fake (ndarray): An array of shape (N,) that contains the fake samples

  Returns:
    ndarray: The discriminator loss
  """

  label_real = 1
  #################################################
  # TODO for students: Loss for real data
  raise NotImplementedError("Student exercise: Implement loss for real samples")
  #################################################
  loss_real = label_real * ...

  label_fake = 0
  #################################################
  # TODO for students: Loss for fake data
  raise NotImplementedError("Student exercise: Implement loss for fake samples")
  #################################################
  loss_fake = ... * torch.log(1 - disc.classify(x_fake))

  return torch.cat([loss_real, loss_fake])

disc = DummyDisc()
gen = DummyGen()

x_real = get_data()
x_fake = gen.sample()

## Uncomment to check your function
# ld = loss_disc(disc, x_real, x_fake)
# plotting_ld(ld)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D5_GenerativeModels/solutions/W2D5_Tutorial2_Solution_393f89f3.py)

*Example output:*

**A note on numerical stability**

It is common that functions like $\log$ throw a numerical error.
For $\log$, it happens when $x$ in $\log(x)$ is very close to 0.
The most common practice is to always add some very small value $\epsilon$ to $x$, i.e. use $\log(x + \epsilon)$ instead.
Most build-in functions in modern DL frameworks like TensorFlow or PyTorch handle such things in their build-in loss already, e.g., `torch.nn.BCE`, which is equivalent to the loss you implemented above.

## Section 3.2:  Density ratio estimation and the optimal discriminator

As explained in the lecture, the critic in GAN that trains as a binary classifier, upon training becomes a *density ratio estimator* of $\frac{p}{q}$.

The estimated density ratio allows for quantifying how far the real and generator distributions are from each other using $f$-divergence. The generator minimizes this estimated $f$-divergence during training.

We will now train a discriminator to see how it estimates the density ratio between two distributions.

### Coding Exercise 3.2: Estimating the density ratio by the discriminator

We have provided an implementation of a binary critic in the class `OptimalDisc`.

__Your goal__ is to complete the implementation of the function `ratio_disc` below using the function `classify` from `OptimalCritic` class.

Upon correct implementation, you should see that the plot of the ratios from the optimal discriminator align to the true density ratio.

__Remember, $\frac{p}{p + q} = \sigma(D_\omega(x))$__

```python
# @markdown *Execute this cell to enable the optimal discriminator: `OptimalDisc`*

class OptimalDisc:
  def classify(self, x):
    dist_p = torch.distributions.normal.Normal(loc=0, scale=1)
    dist_q = torch.distributions.normal.Normal(loc=-2, scale=1)
    prob_p = torch.exp(dist_p.log_prob(x))
    prob_q = torch.exp(dist_q.log_prob(x))

    return prob_p / (prob_p + prob_q)
```

```python
def ratio_disc(disc, x_real, x_fake):
  """Compute the density ratio between real distribution and fake distribution for `x`

  Args:
    disc: The discriminator
    x (ndarray): An array of shape (N,) that contains the samples to evaluate

  Returns:
    ndarray: The density ratios
  """

  # Put samples together
  x = torch.cat([x_real, x_fake])

  #################################################
  # TODO for students: Compute p / (p + q) i.e. p(D=1|x)
  raise NotImplementedError("Student exercise: Implement p_over_pplusq")
  #################################################
  p_over_pplusq = ...

  #################################################
  # TODO for students: Compute q / (p + q) i.e. 1 - p(D=1|x)
  raise NotImplementedError("Student exercise: Implement q_over_pplusq")
  #################################################
  q_over_pplusq = ...

  # Compute p / q
  p_over_q = p_over_pplusq / q_over_pplusq

  return p_over_q

disc = OptimalDisc()
gen = DummyGen()

x_real = get_data(1_000)
x_fake = gen.sample(1_000)

## Uncomment below to check your function
# ratio = ratio_disc(disc, x_real, x_fake)
# plotting_ratio(x_real, x_fake, ratio)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D5_GenerativeModels/solutions/W2D5_Tutorial2_Solution_007bde00.py)

*Example output:*

## Section 3.3:  The generator loss

Now that we have a trained critic, lets see how to train the generator using it.

### Coding Exercise 3.3: The generator loss

We will now implement the generator loss function and evaluate it on some fixed points.

**Your goal** is to complete the implementation of the function `loss_gen` using the optimal critic from above.

Upon correct implementation, you shall see a plot where the loss values from generator samples align with the "Correct" values.

**HINT:** You simply need to change the labels.

```python
def loss_gen(disc, x_fake):
  """Compute the generator loss for `x_fake` given `disc`

  Args:
    disc: The generator
    x_fake (ndarray): An array of shape (N,) that contains the fake samples

  Returns:
    ndarray: The generator loss
  """

  #################################################
  # TODO for students: Loss for fake data
  raise NotImplementedError("Student exercise: Implement loss for fake data")
  #################################################
  label_fake = ...
  loss_fake = label_fake * ...

  return loss_fake

disc = DummyDisc()
gen = DummyGen()

x_fake = gen.sample()
## Uncomment below to check your function
# lg = loss_gen(disc, x_fake)
# plotting_lg(lg)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D5_GenerativeModels/solutions/W2D5_Tutorial2_Solution_04a08f8f.py)

*Example output:*

**Did you notice?**

The loss you implemented for generator is essentially the part for real data in `loss_disc`, i.e., it is saying, "*the data I am feeding to you is real and not fake*".

---
# Section 4: GAN training in action!

In this section we will be playing with a complete implementation of GAN.

## Interactive Demo 4: GAN training in action

```python
# @markdown *Execute this cell to enable the implemented GAN*
# @markdown
# @markdown You are encouraged to take a look at the implementation as well.
from IPython.display import clear_output

class Generator(nn.Module):
  def __init__(self, latent_dim, layers, output_activation=None):

    super(Generator, self).__init__()
    self.latent_dim = latent_dim
    self.output_activation = output_activation
    self._init_layers(layers)

  def _init_layers(self, layers):
    """Initialize the layers and store as self.module_list."""
    self.module_list = nn.ModuleList()
    last_layer = self.latent_dim
    for index, width in enumerate(layers):
      self.module_list.append(nn.Linear(last_layer, width))
      last_layer = width
      if index + 1 != len(layers):
        self.module_list.append(nn.LeakyReLU())
    else:
      if self.output_activation is not None:
        self.module_list.append(self.output_activation())

  def forward(self, input_tensor):
    """Forward pass; map latent vectors to samples."""
    intermediate = input_tensor
    for layer in self.module_list:
      intermediate = layer(intermediate)
    return intermediate

class Discriminator(nn.Module):
  def __init__(self, input_dim, layers):

    super(Discriminator, self).__init__()
    self.input_dim = input_dim
    self._init_layers(layers)

  def _init_layers(self, layers):
    """Initialize the layers and store as self.module_list."""
    self.module_list = nn.ModuleList()
    last_layer = self.input_dim
    for index, width in enumerate(layers):
      self.module_list.append(nn.Linear(last_layer, width))
      last_layer = width
      if index + 1 != len(layers):
        self.module_list.append(nn.LeakyReLU())
    else:
      self.module_list.append(nn.Sigmoid())

  def forward(self, input_tensor):
    """Forward pass; map samples to confidence they are real [0, 1]."""
    intermediate = input_tensor
    for layer in self.module_list:
      intermediate = layer(intermediate)
    return intermediate

class VanillaGAN():
  def __init__(self, generator, discriminator, noise_fn, data_fn,
                batch_size=100, device='cpu', lr_d=1e-3, lr_g=2e-4):

    self.generator = generator
    self.generator = self.generator.to(device)
    self.discriminator = discriminator
    self.discriminator = self.discriminator.to(device)
    self.noise_fn = noise_fn
    self.data_fn = data_fn
    self.batch_size = batch_size
    self.device = device
    self.criterion = nn.BCELoss()
    self.optim_d = optim.Adam(discriminator.parameters(),
                              lr=lr_d, betas=(0.5, 0.999))
    self.optim_g = optim.Adam(generator.parameters(),
                              lr=lr_g, betas=(0.5, 0.999))
    self.target_ones = torch.ones((batch_size, 1)).to(device)
    self.target_zeros = torch.zeros((batch_size, 1)).to(device)

  def generate_samples(self, latent_vec=None, num=None):
    """Sample from the generator.
    """
    num = self.batch_size if num is None else num
    latent_vec = self.noise_fn(num) if latent_vec is None else latent_vec
    with torch.no_grad():
      samples = self.generator(latent_vec)
    return samples

  def real_data(self, num=None):
    """Real Data
    """
    num = self.batch_size if num is None else num
    with torch.no_grad():
      samples = self.data_fn(num)
    return samples

  def train_step_generator(self):
    """Train the generator one step and return the loss."""
    self.generator.zero_grad()

    latent_vec = self.noise_fn(self.batch_size)
    generated = self.generator(latent_vec)
    classifications = self.discriminator(generated)
    loss = self.criterion(classifications,
                          self.target_ones)
    loss.backward()
    self.optim_g.step()
    return loss.item()

  def train_step_discriminator(self):
    """Train the discriminator one step and return the losses."""
    self.discriminator.zero_grad()

    # real samples
    real_samples = self.data_fn(self.batch_size)
    pred_real = self.discriminator(real_samples)
    loss_real = self.criterion(pred_real,
                               self.target_ones)

    # generated samples
    latent_vec = self.noise_fn(self.batch_size)
    with torch.no_grad():
        fake_samples = self.generator(latent_vec)
    pred_fake = self.discriminator(fake_samples)
    loss_fake = self.criterion(pred_fake,
                               self.target_zeros)

    # combine
    loss = (loss_real + loss_fake) / 2
    loss.backward()
    self.optim_d.step()
    return loss_real.item(), loss_fake.item()

  def train_step(self):
    """Train both networks and return the losses."""
    loss_d = self.train_step_discriminator()
    loss_g = self.train_step_generator()
    return loss_g, loss_d

def train(mean=0., sigma=.2, device='cpu'):
  epochs = 30
  batches = 100
  generator = Generator(2, [64, 32, 2])
  discriminator = Discriminator(2, [64, 32, 1])
  noise_fn = lambda x: torch.rand((x, 2), device=device)
  data_fn = lambda x: mean + sigma*torch.randn((x, 2), device=device)
  gan = VanillaGAN(generator,
                   discriminator,
                   noise_fn,
                   data_fn,
                   lr_d=1e-3,
                   lr_g=2e-4,
                   device=device)
  loss_g, loss_d_real, loss_d_fake = [], [], []
  start = time()

  fig, ax = plt.subplots(1, 1,figsize=(5,5))
  x, y = np.random.random((2, 200))
  data_scat = ax.scatter(x, y,
                         label='Data',
                         alpha=0.9,
                         s=10.,
                         c='r')
  gan_scat = ax.scatter(x,y,
                        label='GAN',
                        alpha=0.9,
                        s=10.,
                        c='b')
  ax.legend()
  ax.set_xlim(-2, 2)
  ax.set_ylim(-2, 2)
  for epoch in range(epochs):
    loss_g_running = 0
    loss_d_real_running = 0
    loss_d_fake_running = 0
    for batch in range(batches):
      lg_, (ldr_, ldf_) = gan.train_step()
      loss_g_running += lg_
      loss_d_real_running += ldr_
      loss_d_fake_running += ldf_
    loss_g.append(loss_g_running / batches)
    loss_d_real.append(loss_d_real_running / batches)
    loss_d_fake.append(loss_d_fake_running / batches)
    print(f"Epoch {epoch+1}/{epochs} ({int(time() - start)}s):"
          f" G={loss_g[-1]:.3f},"
          f" Dr={loss_d_real[-1]:.3f},"
          f" Df={loss_d_fake[-1]:.3f}")

    gan_scat.set_offsets(gan.generate_samples())
    data_scat.set_offsets(gan.real_data())

    clear_output(wait=True)
    display(fig)
  plt.close(1)
```

Notice in the implementaiton above, we use `torch.nn.BCELoss` to implement the discriminator and generator loss with proper "labels".
They are actually equivalent to your implementation - do you want to varify this fact?

```python
# @title GAN demo
# @markdown Make sure you execute this cell to enable the widget!
import ipywidgets

# https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html
slider_mean = ipywidgets.FloatSlider(value=0., min=-1, max=1,
                                     step=.1, readout_format='.5f',
                                     description="Set Target Mean",
                                     style={'description_width': 'initial'})
slider_sigma = ipywidgets.FloatSlider(value=.2, min=.2, max=1.,
                                      step=.1, readout_format='.5f',
                                      description="Set Target Sigma",
                                      style={'description_width': 'initial'})
button = ipywidgets.Button(description="Start Training!")
output = ipywidgets.Output()

def on_button_clicked(b):
  # Display the message within the output widget.
  with output:
    train(slider_mean.value, slider_sigma.value)

button.on_click(on_button_clicked)
display(slider_mean, slider_sigma, button, output)
```

---
# Summary

Through this tutorial, we have learned

- How to implement the training loop of GANs.
- Developed an intuition about the training dynamics of GANs.
- How to implement the training objectives for the generator and discriminator of GANs.
- How are GANs connected to density ratio estimation.

Next tutorial will cover conditional GANs and ethical issues of DL.


```python
import pandas as pd
import numpy as np
```

```python
data = pd.read_excel('data.xlsx')
```

```python
data.head()
```

      Q1_性别
      Q2_身高（厘米）
      Q3_体重 （公斤）
      Q4_头发长度（厘米）

      0
      男
      190
      70
      7

      1
      女
      160
      45
      20

      2
      男
      179
      61
      5

      3
      女
      173
      60
      50

      4
      男
      175
      70
      15

```python
data = data.rename(columns={'Q1_性别': 'label',
                            'Q2_身高（厘米）': 'height',
                            'Q3_体重 （公斤）': 'weight',
                            'Q4_头发长度（厘米）': 'hair'})
```

```python
data['label'] = data['label'].apply(lambda x : {'男': 0, '女': 1}[x])
```

```python
data.head()
```

      label
      height
      weight
      hair

      0
      0
      190
      70
      7

      1
      1
      160
      45
      20

      2
      0
      179
      61
      5

      3
      1
      173
      60
      50

      4
      0
      175
      70
      15

```python
features = data[['height', 'weight', 'hair']].to_numpy()
```

```python
mean = np.mean(features, axis=0)
std = np.std(features, axis=0)
```

```python
features = (features - mean)/std  #归一化，正态分布
```

```python
label = data['label'].to_numpy()
```

```python
features
```

    array([[ 9.86969285e-01, -2.25614423e-03, -4.63916995e-01],
           [-7.03854590e-01, -8.10707828e-01, -1.44545566e-01],
           [ 3.67000531e-01, -2.93298750e-01, -5.13051061e-01],
           [ 2.88357560e-02, -3.25636818e-01,  5.92465423e-01],
           [ 1.41557348e-01, -2.25614423e-03, -2.67380731e-01],
           [-5.91132998e-01, -5.52003289e-01,  1.01124764e-01],
           [-1.40246631e-01,  3.21124529e-01, -5.13051061e-01],
           [-2.52968223e-01, -4.87327154e-01, -5.13051061e-01],
           [ 1.41557348e-01, -1.63946481e-01, -5.37618094e-01],
           [ 1.41557348e-01, -2.25614423e-03, -5.37618094e-01],
           [-8.16576182e-01, -7.46031693e-01,  5.92465423e-01],
           [ 4.23361327e-01, -6.69322789e-02, -5.62185127e-01],
           [ 4.23361327e-01,  1.59434192e-01, -5.13051061e-01],
           [-7.03854590e-01, -6.81355558e-01, -1.44545566e-01],
           [-3.09329019e-01, -3.25636818e-01, -3.41081830e-01],
           [-7.03854590e-01, -6.81355558e-01,  3.46795093e-01],
           [ 1.41557348e-01, -1.31608414e-01, -2.67380731e-01],
           [ 8.74247694e-01, -2.25614423e-03, -2.67380731e-01],
           [-8.38858357e-02, -2.28622616e-01, -3.90215896e-01],
           [ 1.41557348e-01,  1.59434192e-01, -3.90215896e-01],
           [-2.75250398e-02, -1.63946481e-01, -5.62185127e-01],
           [-7.03854590e-01, -5.84341356e-01,  5.92465423e-01],
           [ 4.79722123e-01,  1.91772260e-01, -3.41081830e-01],
           [ 1.41557348e-01, -3.25636818e-01,  1.82081707e+00],
           [-2.75250398e-02, -3.57974885e-01, -5.13051061e-01],
           [ 2.88357560e-02, -2.93298750e-01, -5.62185127e-01],
           [-1.40246631e-01, -3.25636818e-01, -4.63916995e-01],
           [ 4.23361327e-01, -3.90312952e-01, -5.13051061e-01],
           [-1.96607427e-01, -4.87327154e-01,  1.01124764e-01],
           [ 4.23361327e-01, -1.96284548e-01,  5.92465423e-01],
           [-7.03854590e-01, -6.49017491e-01,  3.46795093e-01],
           [-2.75250398e-02, -2.60960683e-01, -5.62185127e-01],
           [ 3.10639735e-01, -2.25614423e-03, -4.88484028e-01],
           [ 3.10639735e-01, -2.25614423e-03, -3.90215896e-01],
           [ 8.51965518e-02,  1.61464722e+00, -3.90215896e-01],
           [ 4.31225624e+00,  4.16935454e+00,  3.73704564e+00],
           [ 7.05165306e-01, -3.25636818e-01, -2.67380731e-01],
           [-4.08550234e+00,  4.16935454e+00,  4.22838630e+00],
           [-5.34772202e-01, -4.54989087e-01,  1.01124764e-01],
           [-8.38858357e-02,  1.59434192e-01, -5.62185127e-01],
           [ 2.88357560e-02, -1.63946481e-01, -5.13051061e-01],
           [-1.96607427e-01, -5.19665222e-01, -9.54115001e-02],
           [-2.75250398e-02,  6.24199904e-02, -5.62185127e-01]])

```python
label
```

    array([0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0])

# Picking a Link Function
Generalized linear models usually tranform a linear model of the predictors by using a [link function](https://en.wikipedia.org/wiki/Generalized_linear_model#Link_function). In logistic regression, the link function is the [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function). We can implement this really easily.

```python
def sigmoid(scores):
    return 1 / (1 + np.exp(-scores))
```

# Maximizing the Likelihood

To maximize the likelihood, I need a way to compute the likelihood and the gradient of the likelihood. Fortunately, the likelihood (for binary classification) can be reduced to a fairly intuitive form by switching to the log-likelihood. We're able to do this without affecting the weights parameter estimation because log transformation are [monotonic](https://en.wikipedia.org/wiki/Monotonic_function).

For anyone interested in the derivations of the functions I'm using, check out Section 4.4.1 of Hastie, Tibsharani, and Friedman's [Elements of Statistical Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/). For those less mathematically inclined, Carlos Guestrin (Univesity of Washington) details one possible derivation of the log-likelihood in a series of short lectures on [Coursera](https://www.coursera.org/learn/ml-classification/lecture/1ZeTC/very-optional-expressing-the-log-likelihood) using indicator functions.

## Calculating the Log-Likelihood

The log-likelihood can be viewed as as sum over all the training data. Mathematically,
损失函数
$$\begin{equation}
ll = \sum_{i=1}^{N}y_{i}\beta ^{T}x_{i} - log(1+e^{\beta^{T}x_{i}})
\end{equation}$$ 越小越好

where $y$ is the target class, $x_{i}$ represents an individual data point, and $\beta$ is the weights vector.

I can easily turn that into a function and take advantage of matrix algebra.

```python
#分类问题的损失函数
def log_likelihood(features, target, weights):
    scores = np.dot(features, weights)
    ll = np.sum( target*scores - np.log(1 + np.exp(scores)) )
    return ll
```

## Calculating the Gradient

Now I need an equation for the gradient of the log-likelihood. By taking the derivative of the equation above and reformulating in matrix form, the gradient becomes:

$$\begin{equation}
\bigtriangledown ll = X^{T}(Y - Predictions)
\end{equation}$$

Again, this is really easy to implement. It's so simple I don't even need to wrap it into a function. The gradient here looks very similar to the output layer gradient in a neural network (see my [post](https://beckernick.github.io/neural-network-scratch/) on neural networks if you're curious).

This shouldn't be too surprising, since a neural network is basically just a series of non-linear link functions applied after linear manipulations of the input data.

# Building the Logistic Regression Function

Finally, I'm ready to build the model function. I'll add in the option to calculate the model with an intercept, since it's a good option to have.

权重的迭代公式：

```python
#梯度下降最优化：取误差函数的极小值
def logistic_regression(features, target, num_steps, learning_rate, add_intercept = False):
    if add_intercept: #intercept：有截距b
        intercept = np.ones((features.shape[0], 1)) #features.shape返回(60,3)[0]返回60。生成60行，1列的1
        features = np.hstack((intercept, features)) #在feature前加上一列1  np.vstack():在竖直方向上堆叠；np.hstack():在水平方向上平铺

    weights = np.zeros(features.shape[1])  #features.shape[1]返回3，3个权重

    for step in range(num_steps):  #迭代多少步
        scores = np.dot(features, weights)
        predictions = sigmoid(scores)

        # Update weights with log likelihood gradient
        output_error_signal = target - predictions

        gradient = np.dot(features.T, output_error_signal)
        weights += learning_rate * gradient

        # Print log-likelihood every so often
        if step % 10000 == 0:
            print(log_likelihood(features, target, weights))

    return weights
```

Time to do the regression.

```python
weights = logistic_regression(features, label,
                     num_steps = 50000, learning_rate = 5e-5, add_intercept=True)  #收敛了
```

    -29.794300659677482
    -12.17665666438134
    -10.368615637493027
    -9.699007253564737
    -9.355548233950786

```python
print(weights)
```

    [-1.62788588 -3.1418227  -2.31358577  2.1596935 ]

```python
def predict(features, weights):
    global mean
    global std
    features = (features - mean)/std
    intercept = np.ones((features.shape[0], 1))
    features = np.hstack((intercept, features))
    scores = np.dot(features, weights)
    predictions = sigmoid(scores)

    return predictions
```

```python
student1 = np.array([[188, 85, 2]])
print(predict(student1, weights))
```

    [0.00115921]

```python
student2 = np.array([[165, 50, 25]])
print(predict(student2, weights))
```

    [0.76002054]