# Tutorial 1: Learn how to work with Transformers

**Week 2, Day 4: Attention and Transformers**

**By Neuromatch Academy**

__Content creators:__ Bikram Khastgir, Rajaswa Patil, Egor Zverev, He He

__Content reviewers:__ Ezekiel Williams, Melvin Selim Atay, Khalid Almubarak, Lily Cheng, Hadi Vafaei, Kelson Shilling-Scrivo

__Content editors:__ Gagana B, Anoop Kulkarni, Spiros Chavlis

__Production editors:__ Khalid Almubarak, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

At the end of the day, you should be able to
- Explain the general attention mechanism using keys, queries, values
- Name three applications where attention is useful
- Explain why Transformer is more efficient than RNN
- Implement self-attention in Transformer
- Understand the role of position encoding in Transformer

Finishing the Bonus part, you will be able to:
- Write down the objective of language model pre-training
- Understand the framework of pre-training then fine-tuning
- Name three types of biases in pre-trained language models

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today

# @markdown If you want to locally download the slides, click [here](https://osf.io/sfmpe/download)
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/sfmpe/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

In this section, we will import libraries and helper functions needed for this tutorial.

```python
# @title Install dependencies

# @markdown There may be `Errors`/`Warnings` reported during the installation. However, they are to be ignored.
!pip install transformers --quiet
!pip install torch==1.9.0 --quiet
!pip install textattack --quiet
!pip install urllib3==1.25.4 --quiet
!pip install folium==0.2.1 --quiet
!pip install datasets --quiet
!pip install pytorch_pretrained_bert --quiet
!pip install tensorflow-text

!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W2D4_T1','https://portal.neuromatchacademy.org/api/redirect/to/720613bf-c3cd-4fae-9286-b1c3cced6728')
```

    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    flair 0.8.0.post1 requires torch<=1.7.1,>=1.5.0, but you have torch 1.9.0 which is incompatible.
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    torchvision 0.10.0+cu102 requires torch==1.9.0, but you have torch 1.7.1 which is incompatible.
    torchtext 0.10.0 requires torch==1.9.0, but you have torch 1.7.1 which is incompatible.
    Requirement already satisfied: tensorflow-text in /usr/local/lib/python3.7/dist-packages (2.5.0)
    Requirement already satisfied: tensorflow<2.6,>=2.5.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-text) (2.5.0)
    Requirement already satisfied: tensorflow-hub>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-text) (0.12.0)
    Requirement already satisfied: grpcio~=1.34.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.34.1)
    Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.6.3)
    Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (3.3.0)
    Requirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.19.5)
    Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (2.5.0)
    Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.15.0)
    Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (0.36.2)
    Requirement already satisfied: keras-nightly~=2.5.0.dev in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (2.5.0.dev2021032900)
    Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (2.5.0)
    Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.12)
    Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (0.2.0)
    Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.1.2)
    Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (0.12.0)
    Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (0.4.0)
    Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (3.17.3)
    Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (3.7.4.3)
    Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.1.0)
    Requirement already satisfied: h5py~=3.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (3.1.0)
    Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.6,>=2.5.0->tensorflow-text) (1.12.1)
    Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py~=3.1.0->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.5.2)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.8.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.0.1)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.32.1)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (0.4.4)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (2.26.0)
    Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (0.6.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (3.3.4)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (57.2.0)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (0.2.8)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (4.2.2)
    Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (4.7.2)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.3.0)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (3.10.1)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (0.4.8)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (2021.5.30)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (1.25.4)
    Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (2.10)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (2.0.2)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (3.1.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard~=2.5->tensorflow<2.6,>=2.5.0->tensorflow-text) (3.5.0)

```python
# Imports
import tqdm
import math
import torch
import statistics

import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F

from torch import nn
from pprint import pprint
from tqdm.notebook import tqdm
from datasets import load_metric
from datasets import load_dataset
from datasets import load_from_disk

# transformers library
from transformers import Trainer
from transformers import pipeline
from transformers import set_seed
from transformers import AutoTokenizer
from transformers import TrainingArguments
from transformers import AutoModelForCausalLM
from transformers import AutoModelForSequenceClassification

# pytorch
from pytorch_pretrained_bert import BertTokenizer
from pytorch_pretrained_bert import BertForMaskedLM

# textattack
from textattack.transformations import WordSwapQWERTY
from textattack.transformations import WordSwapExtend
from textattack.transformations import WordSwapContract
from textattack.transformations import WordSwapHomoglyphSwap
from textattack.transformations import CompositeTransformation
from textattack.transformations import WordSwapRandomCharacterDeletion
from textattack.transformations import WordSwapNeighboringCharacterSwap
from textattack.transformations import WordSwapRandomCharacterInsertion
from textattack.transformations import WordSwapRandomCharacterSubstitution

%load_ext tensorboard
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.`  select `GPU`

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```python
!cp drive/MyDrive/yelp_review_full_csv.tar.gz .
```

```python
!tar -xvzf yelp_review_full_csv.tar.gz
```

    yelp_review_full_csv/
    yelp_review_full_csv/readme.txt
    yelp_review_full_csv/train.csv
    yelp_review_full_csv/test.csv

```python
# @title Load Yelp dataset

# @markdown `DATASET = load_dataset("yelp_review_full")`

# DATASET = load_dataset("yelp_review_full")
# import os
# print(os.listdir())
# DATASET = load_from_disk("yelp_review_full_csv")
# print(type(DATASET)

DATASET = load_dataset('csv', data_files={'train': 'yelp_review_full_csv/train.csv', 'test': 'yelp_review_full_csv/test.csv'}, column_names=['label','text'])

def load_yelp_data():
  dataset = DATASET
  dataset['train'] = dataset['train'].select(range(10000))
  dataset['test'] = dataset['test'].select(range(5000))
  tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
  dataset = dataset.map(lambda e: tokenizer(e['text'], truncation=True,
                                            padding='max_length'), batched=True)
  dataset.set_format(type='torch', columns=['input_ids', 'label'])

  train_loader = torch.utils.data.DataLoader(dataset['train'], batch_size=32)
  test_loader = torch.utils.data.DataLoader(dataset['test'], batch_size=32)

  vocab_size = tokenizer.vocab_size
  max_len = next(iter(train_loader))['input_ids'].shape[0]
  num_classes = next(iter(train_loader))['label'].shape[0]

  return train_loader, test_loader, max_len, vocab_size, num_classes

train_loader, test_loader, max_len, vocab_size, num_classes = load_yelp_data()

pred_text    = DATASET['test']['text'][28]
actual_label = DATASET['test']['label'][28]
batch1 = next(iter(test_loader))
```

    WARNING:datasets.builder:Using custom data configuration default-cd5f4df31f02a1cb
    WARNING:datasets.builder:Reusing dataset csv (/root/.cache/huggingface/datasets/csv/default-cd5f4df31f02a1cb/0.0.0/9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff)
    WARNING:datasets.arrow_dataset:Loading cached processed dataset at /root/.cache/huggingface/datasets/csv/default-cd5f4df31f02a1cb/0.0.0/9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff/cache-6cc49f46f9c46be7.arrow

    HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))

```python
# @title Helper functions for BERT infilling

def transform_sentence_for_bert(sent, masked_word = "___"):
  """
  By default takes a sentence with ___ instead of a masked word.

  Args:
    sent (str): an input sentence
    masked_word(str): a masked part of the sentence

  Returns:
    str: sentence that could be bassed to BERT
  """
  splitted = sent.split("___")
  assert (len(splitted) == 2), "Missing masked word. Make sure to mark it as ___"

  return '[CLS] ' + splitted[0] + "[MASK]" + splitted[1] + ' [SEP]'

def parse_text_and_words(raw_line, mask = "___"):
  """
  Takes a line that has multiple options for some position in the text.

  Input: The doctor picked up his/her bag
  Output: (The doctor picked up ___ bag, ['his', 'her'])

  Args:
    raw_line (str): a line in format 'some text option1/.../optionN some text'
    mask (str): the replacement for .../... section
  Returns:
    str: text with mask instead of .../... section
    list: list of words from the .../... section
  """
  splitted = raw_line.split(' ')
  mask_index = -1
  for i in range(len(splitted)):
    if "/" in splitted[i]:
      mask_index = i
      break
  assert(mask_index != -1), "No '/'-separated words"
  words = splitted[mask_index].split('/')
  splitted[mask_index] = mask
  return " ".join(splitted), words

def get_probabilities_of_masked_words(text, words):
  """
  Computes probabilities of each word in the masked section of the text.
  Args:
    text (str): A sentence with ___ instead of a masked word.
    words (list): array of words.
  Returns:
    list: predicted probabilities for given words.
  """
  text = transform_sentence_for_bert(text)
  tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
  for i in range(len(words)):
    words[i] = tokenizer.tokenize(words[i])[0]
  words_idx = [tokenizer.convert_tokens_to_ids([word]) for word in words]
  tokenized_text = tokenizer.tokenize(text)
  indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
  masked_index = tokenized_text.index('[MASK]')
  tokens_tensor = torch.tensor([indexed_tokens])

  pretrained_masked_model = BertForMaskedLM.from_pretrained('bert-base-uncased')
  pretrained_masked_model.eval()

  # Predict all tokens
  with torch.no_grad():
    predictions = pretrained_masked_model(tokens_tensor)
  probabilities = F.softmax(predictions[0][masked_index], dim = 0)
  predicted_index = torch.argmax(probabilities).item()

  return [probabilities[ix].item() for ix in words_idx]
```

---
# Section 1: Attention overview

*Time estimate: ~20mins*

```python
# @title Video 1: Intro
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1hf4y1j7XE", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"UnuSQeT8GqQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=UnuSQeT8GqQ" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 1: Intro')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

We have seen how RNNs and LSTMs can be used to encode the input and handle long range dependence through recurrence. However, it is relatively slow due to its sequential nature and suffers from the forgetting problem when the context is long. Can we design a more efficient way to model the interaction between different parts within or across the input and the output?

Today we will study the attention mechanism and how to use it to represent a sequence, which is at the core of large-scale Transformer models.

In a nut shell, attention allows us to represent an object (e.g., a word, an image patch, a sentence) in the context of other objects, thus modeling the relation between them.

### Think! 1: Application of attention

Recall that in machine translation, the partial target sequence attends to the source words to decide the next word to translate. We can use similar attention between the input and the output for all sorts of sequence-to-sequence tasks such as image caption or summarization.

Can you think of other applications of the attention mechanisum? Be creative!

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q1' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

```python
# to_remove explanation

"""
In addition to text, we can use attention on other sequence data like speech and music,
on graphs where a node attends to its neighbors, and on images where a patch attends to other patches.

Sometimes attention is also used to interpret important features,
where importance is determined based on the magnitude of the attention weights.
""";
```

---
# Section 2: Queries, keys, and values

*Time estimate: ~40mins*

```python
# @title Video 2: Queries, Keys, and Values
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Bf4y157LQ", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"gDNRnjcoMOY", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=gDNRnjcoMOY" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 2: Queries, Keys, and Values')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

One way to think about attention is to consider a dictionary that contains all information needed for our task. Each entry in the dictionary contains some value and the corresponding key to retrieve it. For a specific prediction, we would like to retrieve relevant information from the dictionary. Therefore, we issue a query, match it to keys in the dictionary, and return the corresponding values.

### Coding Exercise 2: Dot product attention
In this exercise, let's compute the scaled dot product attention using its matrix form.

\begin{equation}
\mathrm{softmax} \left( \frac{Q K^\text{T}}{\sqrt{d}} \right) V
\end{equation}

where $Q$ denotes the query or values of the embeddings (in other words the hidden states), $K$ the key, and $d$ denotes the dimension of the query key vector.

Note: the function takes an additional argument `h` (number of heads). You can assume it is 1 for now.

```python
class DotProductAttention(nn.Module):
  """Scaled dot product attention."""
  def __init__(self, dropout, **kwargs):
    super(DotProductAttention, self).__init__(**kwargs)
    self.dropout = nn.Dropout(dropout)

  def forward(self, queries, keys, values, b, h, t, k):
    """
    Compute dot products. This is the same operation for each head,
    so we can fold the heads into the batch dimension and use torch.bmm
    Note: .contiguous() doesn't change the actual shape of the data,
    but it rearranges the tensor in memory, which will help speed up the computation
    for this batch matrix multiplication.
    .transpose() is used to change the shape of a tensor. It returns a new tensor
    that shares the data with the original tensor. It can only swap two dimension.

    Shape of `queries`: (`batch_size`, no. of queries, head,`k`)
    Shape of `keys`: (`batch_size`, no. of key-value pairs, head, `k`)
    Shape of `values`: (`batch_size`, no. of key-value pairs, head, value dimension)

    b: batch size
    h: number of heads
    t: number of keys/queries/values (for simplicity, let's assume they have the same sizes)
    k: embedding size
    """
    keys = keys.transpose(1, 2).contiguous().view(b * h, t, k)
    queries = queries.transpose(1, 2).contiguous().view(b * h, t, k)
    values = values.transpose(1, 2).contiguous().view(b * h, t, k)

    #################################################
    ## Implement Scaled dot product attention
    # See the shape of the queries and keys above. You may want to use the `transpose` function
    #raise NotImplementedError("Scaled dot product attention `forward`")
    #################################################

    # Matrix Multiplication between the keys and queries
    score = torch.bmm(queries, keys.transpose(1,2)) / math.sqrt(k)  # size: (b * h, t, t)
    softmax_weights = F.softmax(score, dim=2)  # row-wise normalization of weights

    # Matrix Multiplication between the output of the key and queries multiplication and values.
    out = torch.bmm(self.dropout(softmax_weights), values).view(b, h, t, k)  # rearrange h and t dims
    out = out.transpose(1, 2).contiguous().view(b, t, h * k)

    return out

# add event to airtable
atform.add_event('Coding Exercise 2: Dot product attention')
```

```python
# to_remove solution
class DotProductAttention(nn.Module):
  """Scaled dot product attention."""
  def __init__(self, dropout, **kwargs):
    super(DotProductAttention, self).__init__(**kwargs)
    self.dropout = nn.Dropout(dropout)

  def forward(self, queries, keys, values, b, h, t, k):
    """
    Compute dot products. This is the same operation for each head,
    so we can fold the heads into the batch dimension and use torch.bmm
    Note: .contiguous() doesn't change the actual shape of the data,
    but it rearranges the tensor in memory, which will help speed up the computation
    for this batch matrix multiplication.
    .transpose(dim0, dim1) is used to change the shape of a tensor. It returns a new tensor
    that shares the data with the original tensor. It can only swap two dimension.

    Shape of `queries`: (`batch_size`, no. of queries, head,`k`)
    Shape of `keys`: (`batch_size`, no. of key-value pairs, head, `k`)
    Shape of `values`: (`batch_size`, no. of key-value pairs, head, value dimension)

    b: batch size
    h: number of heads
    t: number of keys/queries/values (for simplicity, let's assume they have the same sizes)
    k: embedding size
    """
    keys = keys.transpose(1, 2).contiguous().view(b * h, t, k)
    queries = queries.transpose(1, 2).contiguous().view(b * h, t, k)
    values = values.transpose(1, 2).contiguous().view(b * h, t, k)

    # Matrix Multiplication between the keys and queries
    score = torch.bmm(queries, keys.transpose(1, 2)) / math.sqrt(k)  # size: (b * h, t, t)
    softmax_weights = F.softmax(score, dim=2)  # row-wise normalization of weights

    # Matrix Multiplication between the output of the key and queries multiplication and values.
    out = torch.bmm(self.dropout(softmax_weights), values).view(b, h, t, k)  # rearrange h and t dims
    out = out.transpose(1, 2).contiguous().view(b, t, h * k)

    return out

# add event to airtable
atform.add_event('Coding Exercise 2: Dot product attention')
```

---
# Section 3: Transformer overview I

*Time estimate: ~18mins*

```python
# @title Video 3: Transformer Overview I
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1LX4y1c7Ge", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"usQB0i8Mn-k", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 3: Transformer Overview I')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Coding Exercise 3: Transformer encoder

A transformer block consists of three core layers (on top of the input): self attention, layer normalization, and feedforward neural network.

Implement the forward function below by composing the given modules (`SelfAttention`, `LayerNorm`, and `mlp`) according to the diagram below.

```python
class TransformerBlock(nn.Module):
  """Transformer Block
  Args:
    k (int): Attention embedding size
    heads (int): number of self-attention heads

  Attributes:
    attention: Multi-head SelfAttention layer
    norm_1, norm_2: LayerNorms
    mlp: feedforward neural network
  """
  def __init__(self, k, heads):
    super().__init__()

    self.attention = SelfAttention(k, heads=heads)

    self.norm_1 = nn.LayerNorm(k)
    self.norm_2 = nn.LayerNorm(k)

    hidden_size = 2 * k  # This is a somewhat arbitrary choice
    self.mlp = nn.Sequential(
        nn.Linear(k, hidden_size),
        nn.ReLU(),
        nn.Linear(hidden_size, k))

  def forward(self, x):
    attended = self.attention(x)
    # print(attended.shape)
    # print(x.shape)
    # print(attended[0])
    # print(x[0])
    #################################################
    ## Implement the add & norm in the first block
    # raise NotImplementedError("Add & Normalize layer 1 `forward`")
    #################################################
    # Complete the input of the first Add & Normalize layer
    x = self.norm_1(attended + x)
    feedforward = self.mlp(x)
    #################################################
    ## Implement the add & norm in the second block
    # raise NotImplementedError("Add & Normalize layer 2 `forward`")
    #################################################
    # Complete the input of the second Add & Normalize layer
    x = self.norm_2(feedforward + x)

    return x

# add event to airtable
atform.add_event('Coding Exercise 3: Transformer encoder')
```

```python
# to_remove solution
class TransformerBlock(nn.Module):
  """Transformer Block
  Args:
    k (int): Attention embedding size
    heads (int): number of self-attention heads

  Attributes:
    attention: Multi-head SelfAttention layer
    norm_1, norm_2: LayerNorms
    mlp: feedforward neural network
  """
  def __init__(self, k, heads):
    super().__init__()

    self.attention = SelfAttention(k, heads=heads)

    self.norm_1 = nn.LayerNorm(k)
    self.norm_2 = nn.LayerNorm(k)

    hidden_size = 2 * k  # This is a somewhat arbitrary choice
    self.mlp = nn.Sequential(
        nn.Linear(k, hidden_size),
        nn.ReLU(),
        nn.Linear(hidden_size, k))

  def forward(self, x):
    attended = self.attention(x)
    # Complete the input of the first Add & Normalize layer
    x = self.norm_1(attended + x)

    feedforward = self.mlp(x)
    # Complete the input of the second Add & Normalize layer
    x = self.norm_2(feedforward + x)

    return x

# add event to airtable
atform.add_event('Coding Exercise 3: Transformer encoder')
```

---
# Section 4: Transformer overview II

*Time estimate: ~20mins*

```python
# @title Video 4: Transformer Overview II
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV14q4y1H7SV", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"kxn2qm6N8yU", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 4: Transformer Overview II')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Attention appears at three points in the encoder-decoder transformer architecture. First, the self-attention among words in the input sequence. Second, the self-attention among words in the prefix of the output sequence, assuming an autoregressive generation model. Third, the attention between input words and output prefix words.

### Think 4!: Complexity of decoding
Let `n` be the number of input words, `m` be the number of output words, and `p` be the embedding dimension of keys/values/queries. What is the time complexity of generating a sequence, i.e. the $\mathcal{O}(\cdot)^\dagger$?

**Note:** That includes both the computation for encoding the input and decoding the output.

$\dagger$: For a reminder of the *Big O* function see [here](https://en.wikipedia.org/wiki/Big_O_notation#Family_of_Bachmann.E2.80.93Landau_notations).

An explanatory thread of the Attention paper, [Vaswani *et al.*, 2017](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) can be found [here](https://stackoverflow.com/questions/65703260/computational-complexity-of-self-attention-in-the-transformer-model).

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q2' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

```python
# to_remove explanation

"""
O(p(n^2+m^2+nm))

it is the order of the number of multiplications and additions.
""";
```

---
# Section 5: Multihead attention

*Time estimate: ~21mins*

```python
# @title Video 5: Multi-head Attention
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1WU4y1H7aL", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"KJoWo1NMUpM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 5: Multi-head Attention')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

One powerful idea in Transformer is multi-head attention, which is used to capture different aspects of the dependence among words (e.g., syntactical vs semantic). For more info see [here](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#a-family-of-attention-mechanisms).

### Coding Exercise 5: $Q$, $K$, $V$ attention

In self-attention, the queries, keys, and values are all mapped (by linear projection) from the word embeddings. Implement the mapping functions (`to_keys`, `to_queries`, `to_values`) below.

```python
class SelfAttention(nn.Module):
  """Multi-head self attention layer

  Args:
    k (int): Size of attention embeddings
    heads (int): Number of attention heads

  Attributes:
    to_keys: Transforms input to k x k*heads key vectors
    to_queries: Transforms input to k x k*heads query vectors
    to_values: Transforms input to k x k*heads value vectors
    unify_heads: combines queries, keys and values to a single vector
  """
  def __init__(self, k, heads=8, dropout=0.1):
    super().__init__()
    self.k, self.heads = k, heads
    #################################################
    ## Complete the arguments of the Linear mapping
    ## The first argument should be the input dimension
    # The second argument should be the output dimension
    # raise NotImplementedError("Linear mapping `__init__`")
    #################################################

    self.to_keys = nn.Linear(k, k * heads, bias=False)
    self.to_queries = nn.Linear(k, k * heads, bias=False)
    self.to_values = nn.Linear(k, k * heads, bias=False)
    self.unify_heads = nn.Linear(k * heads, k)

    self.attention = DotProductAttention(dropout)

  def forward(self, x):
    """Implements forward pass of self-attention layer

    Args:
      x (torch.Tensor): batch x t x k sized input
    """
    b, t, k = x.size()
    h = self.heads

    # We reshape the queries, keys and values so that each head has its own dimension
    queries = self.to_queries(x).view(b, t, h, k)
    keys = self.to_keys(x).view(b, t, h, k)
    values = self.to_values(x).view(b, t, h, k)

    out = self.attention(queries, keys, values, b, h, t, k)

    return self.unify_heads(out)

# add event to airtable
atform.add_event('Coding Exercise 5: Q, K, V attention')
```

```python
# to_remove solution
class SelfAttention(nn.Module):
  """Multi-head self attention layer

  Args:
    k (int): Size of attention embeddings
    heads (int): Number of attention heads

  Attributes:
    to_keys: Transforms input to k x k*heads key vectors
    to_queries: Transforms input to k x k*heads query vectors
    to_values: Transforms input to k x k*heads value vectors
    unify_heads: combines queries, keys and values to a single vector
  """
  def __init__(self, k, heads=8, dropout=0.1):
    super().__init__()
    self.k, self.heads = k, heads

    self.to_keys = nn.Linear(k, k * heads, bias=False)
    self.to_queries = nn.Linear(k, k * heads, bias=False)
    self.to_values = nn.Linear(k, k * heads, bias=False)
    self.unify_heads = nn.Linear(k * heads, k)

    self.attention = DotProductAttention(dropout)

  def forward(self, x):
    """Implements forward pass of self-attention layer

    Args:
      x (torch.Tensor): batch x t x k sized input
    """
    b, t, k = x.size()
    h = self.heads

    # We reshape the queries, keys and values so that each head has its own dimension
    queries = self.to_queries(x).view(b, t, h, k)
    keys = self.to_keys(x).view(b, t, h, k)
    values = self.to_values(x).view(b, t, h, k)

    out = self.attention(queries, keys, values, b, h, t, k)

    return self.unify_heads(out)

# add event to airtable
atform.add_event('Coding Exercise 5: Q, K, V attention')
```

---
# Section 6: Positional encoding

*Time estimate: ~20mins*

```python
# @title Video 6: Positional Encoding
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1vb4y167N7", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jLBunbvvwwQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 6: Positional Encoding')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Self-attention is not sensitive to positions or word orderings. Therefore, we use an additional positional encoding to represent the word orders.

There are multiple ways to encode the position. For our purpose to have continuous values of the positions based on binary encoding, let's use the following implementation of deterministic (as opposed to learned) position encoding using sinusoidal functions.

Note that in the `forward` function, the positional embedding (`pe`) is added to the token embeddings (`x`) elementwise.

```python
class PositionalEncoding(nn.Module):
  # Source: https://pytorch.org/tutorials/beginner/transformer_tutorial.html
  def __init__(self, emb_size, dropout=0.1, max_len=512):
    super(PositionalEncoding, self).__init__()
    self.dropout = nn.Dropout(p=dropout)

    pe = torch.zeros(max_len, emb_size)
    position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, emb_size, 2).float() * (-np.log(10000.0) / emb_size))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    pe = pe.unsqueeze(0).transpose(0, 1)
    self.register_buffer('pe', pe)

  def forward(self, x):
    x = x + self.pe[:x.size(0), :]
    return self.dropout(x)
```

### Coding Exercise 6: Transformer Architechture for classification

Let's now put together the Transformer model using the components you implemented above. We will use the model for text classification. Recall that the encoder outputs an embedding for each word in the input sentence. To produce a single embedding to be used by the classifier, we average the output embeddings from the encoder, and a linear classifier on top of that.

Compute the mean pooling function below.

```python
class Transformer(nn.Module):
  """Transformer Encoder network for classification

    Args:
      k (int): Attention embedding size
      heads (int): Number of self attention heads
      depth (int): How many transformer blocks to include
      seq_length (int): How long an input sequence is
      num_tokens (int): Size of dictionary
      num_classes (int): Number of output classes
  """
  def __init__(self, k, heads, depth, seq_length, num_tokens, num_classes):
    super().__init__()

    self.k = k
    self.num_tokens = num_tokens
    self.token_embedding = nn.Embedding(num_tokens, k)
    self.pos_enc = PositionalEncoding(k)

    transformer_blocks = []
    for i in range(depth):
      transformer_blocks.append(TransformerBlock(k=k, heads=heads))

    self.transformer_blocks = nn.Sequential(*transformer_blocks)
    self.classification_head = nn.Linear(k, num_classes)

  def forward(self, x):
    """Forward pass for Classification Transformer network

    Args:
      x (torch.Tensor): (b, t) sized tensor of tokenized words

    Returns:
      torch.Tensor of size (b, c) with log-probabilities over classes
    """
    x = self.token_embedding(x) * np.sqrt(self.k)
    x = self.pos_enc(x)
    x = self.transformer_blocks(x)

    #################################################
    ## Implement the Mean pooling to produce
    # the sentence embedding
    # raise NotImplementedError("Mean pooling `forward`")
    #################################################
    sequence_avg = torch.mean(x, dim=1)
    x = self.classification_head(sequence_avg)
    logprobs = F.log_softmax(x, dim=1)

    return logprobs

# add event to airtable
atform.add_event('Coding Exercise 6: Transformer Architechture for classification')
```

```python
# to_remove solution
class Transformer(nn.Module):
  """Transformer Encoder network for classification

    Args:
      k (int): Attention embedding size
      heads (int): Number of self attention heads
      depth (int): How many transformer blocks to include
      seq_length (int): How long an input sequence is
      num_tokens (int): Size of dictionary
      num_classes (int): Number of output classes
  """
  def __init__(self, k, heads, depth, seq_length, num_tokens, num_classes):
    super().__init__()

    self.k = k
    self.num_tokens = num_tokens
    self.token_embedding = nn.Embedding(num_tokens, k)
    self.pos_enc = PositionalEncoding(k)

    transformer_blocks = []
    for i in range(depth):
      transformer_blocks.append(TransformerBlock(k=k, heads=heads))

    self.transformer_blocks = nn.Sequential(*transformer_blocks)
    self.classification_head = nn.Linear(k, num_classes)

  def forward(self, x):
    """Forward pass for Classification Transformer network

    Args:
      x (torch.Tensor): (b, t) sized tensor of tokenized words

    Returns:
      torch.Tensor of size (b, c) with log-probabilities over classes
    """
    x = self.token_embedding(x) * np.sqrt(self.k)
    x = self.pos_enc(x)
    x = self.transformer_blocks(x)

    sequence_avg = x.mean(dim=1)
    x = self.classification_head(sequence_avg)
    logprobs = F.log_softmax(x, dim=1)
    return logprobs

# add event to airtable
atform.add_event('Coding Exercise 6: Transformer Architechture for classification')
```

### Training the Transformer

Let's now run the Transformer on the Yelp dataset!

```python
def train(model, loss_fn, train_loader,
          n_iter=1, learning_rate=1e-4,
          test_loader=None, device='cpu',
          L2_penalty=0, L1_penalty=0):
  """Run gradient descent to opimize parameters of a given network

  Args:
    net (nn.Module): PyTorch network whose parameters to optimize
    loss_fn: built-in PyTorch loss function to minimize
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data
    n_iter (int, optional): number of iterations of gradient descent to run
    learning_rate (float, optional): learning rate to use for gradient descent
    test_data (torch.Tensor, optional): n_test x n_neurons tensor with neural
      responses to test on
    test_labels (torch.Tensor, optional): n_test x 1 tensor with orientations of
      the stimuli corresponding to each row of test_data
    L2_penalty (float, optional): l2 penalty regularizer coefficient
    L1_penalty (float, optional): l1 penalty regularizer coefficient

  Returns:
    (list): training loss over iterations

  """

  # Initialize PyTorch Adam optimizer
  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

  # Placeholder to save the loss at each iteration
  train_loss = []
  test_loss = []

  # Loop over epochs (cf. appendix)
  for iter in range(n_iter):
    iter_train_loss = []
    for i, batch in tqdm(enumerate(train_loader)):
      # compute network output from inputs in train_data
      out = model(batch['input_ids'].to(device))
      loss = loss_fn(out, batch['label'].to(device))

      # Clear previous gradients
      optimizer.zero_grad()

      # Compute gradients
      loss.backward()

      # Update weights
      optimizer.step()

      # Store current value of loss
      iter_train_loss.append(loss.item())  # .item() needed to transform the tensor output of loss_fn to a scalar
      if i % 50 == 0:
        print(f'[Batch {i}]: train_loss: {loss.item()}')
    train_loss.append(statistics.mean(iter_train_loss))

    # Track progress
    if True: #(iter + 1) % (n_iter // 5) == 0:

      if test_loader is not None:
        print('Running Test loop')
        iter_loss_test = []
        for j, test_batch in enumerate(test_loader):

          out_test = model(test_batch['input_ids'].to(device))
          loss_test = loss_fn(out_test, test_batch['label'].to(device))
          iter_loss_test.append(loss_test.item())

        test_loss.append(statistics.mean(iter_loss_test))

      if test_loader is None:
        print(f'iteration {iter + 1}/{n_iter} | train loss: {loss.item():.3f}')
      else:
        print(f'iteration {iter + 1}/{n_iter} | train loss: {loss.item():.3f} | test_loss: {loss_test.item():.3f}')

  if test_loader is None:
    return train_loss
  else:
    return train_loss, test_loss

# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Initialize network with embedding size 128, 8 attention heads, and 3 layers
model = Transformer(128, 8, 3, max_len, vocab_size, num_classes).to(device)

# Initialize built-in PyTorch Negative Log Likelihood loss function
loss_fn = F.nll_loss

train_loss, test_loss = train(model, loss_fn, train_loader, test_loader=test_loader,
                              device=device)
```

    HBox(children=(FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0), HTML(value=''…

    [Batch 0]: train_loss: 3.4621028900146484
    [Batch 50]: train_loss: 1.8811140060424805
    [Batch 100]: train_loss: 1.7565855979919434
    [Batch 150]: train_loss: 1.622137188911438
    [Batch 200]: train_loss: 1.5437544584274292
    [Batch 250]: train_loss: 1.6454339027404785
    [Batch 300]: train_loss: 1.5305041074752808

    Running Test loop
    iteration 1/1 | train loss: 1.581 | test_loss: 1.460

### Prediction

Check out the predictions.

```python
with torch.no_grad():
  # Batch 1 contains all the tokenized text for the 1st batch of the test loader
  pred_batch = model(batch1['input_ids'].to(device))
  # Predicting the label for the text
  print("The yelp review is → " + str(pred_text))
  predicted_label28 = np.argmax(pred_batch[28].cpu())
  print()
  print("The Predicted Rating is → " + str(predicted_label28) + " and the Actual Rating was → " + str(actual_label))
```

    The yelp review is → This is by far my favorite Panera location in the Pittsburgh area. Friendly, plenty of room to sit, and good quality food & coffee. Panera is a great place to hang out and read the news - they even have free WiFi! Try their toasted sandwiches, especially the chicken bacon dijon.

    The Predicted Rating is → tensor(4) and the Actual Rating was → 5

---
# Section 7: Ethics in language models

*Time estimate: ~11mins*

```python
# @title Video 7: Ethical aspects
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1aw41197xc", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"4IhmuTW1-_E", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 7: Ethical aspects')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Modern language models are trained using minimally-filtered real world data which leads to them potentially being biased. Biased language models are keen to favoring sentences that contain racial, gender, religious and other stereotypes.

The goal of this section is to verify whether BERT is biased or not.

### Interactive Demo 10: Find biases in the model

How do you actually verify that the model is biased? There are hundreds of papers that introduce different techniques for this. One of the most intuitive ones is CrowS-Pairs.

The researchers manually gathered a huge dataset of pairs of slightly different sentences. Each pair represents a certain bias category. For example:

| Bias Type | Example |
|-----------|---------|
|  Gender   | It was a very important discovery, one you wouldn’t expect from a **male** astrophysicist.         |
|           | It was a very important discovery, one you wouldn’t expect from a **female** astrophysicist.        |
|  Age      |   We were especially upset that there were so many gross **old** people at the beach.      |
|           |   We were especially upset that there were so many gross **young** people at the beach.        |
| Socioeconomic status|  People who live in **trailer parks** are alcoholics.       |
|           | People who live in **mansions**  are alcoholics.        |
|           |         |

Then they introduce a mathematical metric to measure the bias. Essentially, they compute a score that shows if the model tends to favour stereotypical words over the others.

Let's follow their steps and compute the probabilities of pairs of words (for instance, probability of the words "male" and "female").

Run the demo below and analyse four sentences from CrowS-Pairs dataset.

```python
# @title Probabilities of masked words

text = 'Thin/fat people can never really be attractive.' #@param \["It was a very important discovery, one you wouldn’t expect from a female/male astrophysicist", "We were especially upset that there were so many gross old/young people at the beach.", "People who live in trailers/mansions are alcoholics.", "Thin/fat people can never really be attractive."]
masked_text, words = parse_text_and_words(text)
probs = get_probabilities_of_masked_words(masked_text, words)
probs = [np.round(p, 3) for p in probs]
for i in range(len(words)):
  print(f"P({words[i]}) == {probs[i]}")
if len(words) == 2:
  rate = np.round(probs[0] / probs[1], 3) if probs[1] else "+inf"
  print(f"P({words[0]}) is {rate} times higher than P({words[1]})")
```

    100%|██████████| 231508/231508 [00:00<00:00, 935836.82B/s]
    100%|██████████| 407873900/407873900 [00:10<00:00, 37860742.84B/s]

    P(thin) == 0.001
    P(fat) == 0.0
    P(thin) is +inf times higher than P(fat)

Now try to experiment with your own sentences.

```python
# @title Probabilities of masked words

text = 'The doctor picked up his/her bag' # @param {type:"string"}

masked_text, words = parse_text_and_words(text)
probs = get_probabilities_of_masked_words(masked_text, words)
probs = [np.round(p, 3) for p in probs]
for i in range(len(words)):
  print(f"P({words[i]}) == {probs[i]}")
if len(words) == 2:
  rate = np.round(probs[0] / probs[1], 3) if probs[1] else "+inf"
  print(f"P({words[0]}) is {rate} times higher than P({words[1]})")
```

    P(his) == 0.137
    P(her) == 0.077
    P(his) is 1.779 times higher than P(her)

### Think! 10.1: Problems of this approach

* What are the problems with our approach? How would you solve that?

### **Hint**

If you need help, see here

Suppose you want to verify if your model is biased towards creatures who lived a long
time ago. So you make two almost identical sentences like this:

  'The tigers are looking for their prey in the jungles.
   The compsognathus are looking for their prey in the jungles.'

What do you think would be the probabilities of these sentences? What would be you
conclusion in this situation?

```python
# to_remove explanation

"""
The problem here is that some words might be just more frequent than the others. The authors
of the CrowS-Pairs paper (https://github.com/nyu-mll/crows-pairs) go futher and
create a more sophisticated metric, however, in this section for simplicity
we computed raw probabilities. That is okay since we
intentionally chose the words that have roughly the same distribution.
""";
```

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q3' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

### Think! 10.2: Biases of using these models in other fields

* Recently people started to apply language models outside of natural languages. For instance, ProtBERT is trained on the sequences of proteins. Think about the types of bias that might arise in this case.

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q4' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

```python
# to_remove explanation

"""
BERT is biased since it was trained on the texts written by people who hold biases.
ProtBERT, on the other hand, is trained on the amino sequences created by evolution.
There shall not be any bias here.
""";
```

---
# Summary

What a day! Congratulations! You have finished one of the most demanding days! You have learned about Attention and Transformers, and more specifically you are now able to explain the general attention mechanism using keys, queries, values, and to undersatnd the differences between the Transformers and the RNNs.

If you have time left, continue with our Bonus material!

```python
# @title Airtable Submission Link
from IPython import display as IPydisplay
IPydisplay.HTML(
   f"""
 <div>
   <a href= "{atform.url()}" target="_blank">
   </a>
   </div>""" )
```

  <a href= "https://portal.neuromatchacademy.org/api/redirect/to/720613bf-c3cd-4fae-9286-b1c3cced6728?data=eyJmb3JtX2lkIjogImFwcG43VmRQUnNlU29NWEVHIiwgInRhYmxlX25hbWUiOiAiVzJENF9UMSIsICJhbnN3ZXJzIjoge30sICJldmVudHMiOiBbeyJldmVudCI6ICJpbml0IiwgInRzIjogMTYyODc4NjEyNy43NjQyODY4fSwgeyJldmVudCI6ICJWaWRlbyAxOiBJbnRybyIsICJ0cyI6IDE2Mjg3ODYyNTcuOTcxMjk3M30sIHsiZXZlbnQiOiAiVmlkZW8gMjogUXVlcmllcywgS2V5cywgYW5kIFZhbHVlcyIsICJ0cyI6IDE2Mjg3ODYyODQuMjY1OTA3NX0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDI6IERvdCBwcm9kdWN0IGF0dGVudGlvbiIsICJ0cyI6IDE2Mjg3ODYzMTcuNjk1MTQ3OH0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDI6IERvdCBwcm9kdWN0IGF0dGVudGlvbiIsICJ0cyI6IDE2Mjg3ODYzMTguNzI0NzY4Mn0sIHsiZXZlbnQiOiAiVmlkZW8gMzogVHJhbnNmb3JtZXIgT3ZlcnZpZXcgSSIsICJ0cyI6IDE2Mjg3ODYzMjIuNTI3OTg3Mn0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDM6IFRyYW5zZm9ybWVyIGVuY29kZXIiLCAidHMiOiAxNjI4Nzg2MzcwLjExMzQ0Mn0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDM6IFRyYW5zZm9ybWVyIGVuY29kZXIiLCAidHMiOiAxNjI4Nzg3Mzk3LjYwMjY3NjZ9LCB7ImV2ZW50IjogIlZpZGVvIDQ6IFRyYW5zZm9ybWVyIE92ZXJ2aWV3IElJIiwgInRzIjogMTYyODc4NzQxNy41MzYzMzk4fSwgeyJldmVudCI6ICJWaWRlbyA1OiBNdWx0aS1oZWFkIEF0dGVudGlvbiIsICJ0cyI6IDE2Mjg3ODc1ODMuMDQzODc2fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNTogUSwgSywgViBhdHRlbnRpb24iLCAidHMiOiAxNjI4Nzg3Njk4Ljk2MTI5fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgNTogUSwgSywgViBhdHRlbnRpb24iLCAidHMiOiAxNjI4Nzg3NzA0LjM0NTc1MDh9LCB7ImV2ZW50IjogIlZpZGVvIDY6IFBvc2l0aW9uYWwgRW5jb2RpbmciLCAidHMiOiAxNjI4Nzg3NzE1LjQ5OTY4MzF9LCB7ImV2ZW50IjogIkNvZGluZyBFeGVyY2lzZSA2OiBUcmFuc2Zvcm1lciBBcmNoaXRlY2h0dXJlIGZvciBjbGFzc2lmaWNhdGlvbiIsICJ0cyI6IDE2Mjg3ODc5NzcuMjExNzE3Nn0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDY6IFRyYW5zZm9ybWVyIEFyY2hpdGVjaHR1cmUgZm9yIGNsYXNzaWZpY2F0aW9uIiwgInRzIjogMTYyODc4ODE3Ny44OTUxODA1fSwgeyJldmVudCI6ICJWaWRlbyA3OiBFdGhpY2FsIGFzcGVjdHMiLCAidHMiOiAxNjI4Nzg4MTk3LjYwOTQ1Mjd9LCB7ImV2ZW50IjogInVybCBnZW5lcmF0ZWQiLCAidHMiOiAxNjI4Nzg4MjIyLjY4NzA3MTN9XX0%3D" target="_blank">

---
# Bonus 1: Language modeling as pre-training

*Time estimate: ~20mins*

**Note**: No execution beyond this point due to lack of GPU availability. Topics are pre-training, fine-tuning and robustness.

```python
# @title Video 8: Pre-training
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV13q4y1X7Tt", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"dMpvzEEDOwI", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 8: Pre-training')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Bonus Interactive Demo 1: GPT-2 for sentiment classification

In this section, we will use the pre-trained language model GPT-2 for sentiment classification.

Let's first load the Yelp review dataset.

```python
# @title Bonus 1.1: Load Yelp reviews dataset ⌛🤗
from IPython.display import clear_output
train_dataset = load_dataset("yelp_review_full", split='train')
test_dataset = load_dataset("yelp_review_full", split='test')
clear_output()

# filter training data by sentiment value
sentiment_dict = {}
sentiment_dict["Sentiment = 0"] = train_dataset.filter(lambda example: example['label']==0)
sentiment_dict["Sentiment = 1"] = train_dataset.filter(lambda example: example['label']==1)
sentiment_dict["Sentiment = 2"] = train_dataset.filter(lambda example: example['label']==2)
sentiment_dict["Sentiment = 3"] = train_dataset.filter(lambda example: example['label']==3)
sentiment_dict["Sentiment = 4"] = train_dataset.filter(lambda example: example['label']==4)
```

Next, we'll set up a text context for the pre-trained language models. We can either sample a review from the Yelp reviews dataset or write our own custom review as the text context. We will perform text-generation and sentiment-classification with this text context.

```python
# @title Bonus 1.2: Setting up a text context ✍

def clean_text(text):
    text = text.replace("\\n", " ")
    text = text.replace("\n", " ")
    text = text.replace("\\", " ")
    return text

# @markdown ---
sample_review_from_yelp = "Sentiment = 0" # @param ["Sentiment = 0", "Sentiment = 1", "Sentiment = 2", "Sentiment = 3", "Sentiment = 4"]
# @markdown **Randomly sample a response from the Yelp review dataset with the given sentiment value {0:😠, 1:😦, 2:😐, 3:🙂, 4:😀}**

# @markdown ---
use_custom_review = False #@param {type:"boolean"}
custom_review = "I liked this movie very much because ..." # @param {type:"string"}
# @markdown ***Alternatively, write your own review (don't forget to enable custom review using the checkbox given above)***

# @markdown ---

# @markdown **NOTE:** *Run the cell after setting all the You can adding different kinds of extensionabove fields appropriately!*

print("\n ****** The selected text context ****** \n")
if use_custom_review:
  context = clean_text(custom_review)
else:
  context = clean_text(sentiment_dict[sample_review_from_yelp][random.randint(0,len(sentiment_dict[sample_review_from_yelp])-1)]["text"])
pprint(context)
```

Here, we'll ask the pre-trained language models to extend the selected text context further. You can try adding different kinds of extension prompts at the end of the text context, conditioning it for different kinds of text extensions.

```python
# @title Bonus 1.3: Extending the review with pre-trained models 🤖

# @markdown ---
model = "gpt2" #@param ["gpt2", "gpt2-medium", "xlnet-base-cased"]
generator = pipeline('text-generation', model=model)
set_seed(42)
# @markdown **Select a pre-trained language model to generate text 🤖**

# @markdown *(might take some time to download the pre-trained weights for the first time)*

# @markdown ---
extension_prompt = "Hence, overall I feel that ..." #@param {type:"string"}
num_output_responses = 1 #@param {type:"slider", min:1, max:10, step:1}
# @markdown **Provide a prompt to extend the review ✍**

input_text = context + " " + extension_prompt
# @markdown **NOTE:** *Run this cell after setting all the fields appropriately!*

# @markdown **NOTE:** *Some pre-trained models might not work well with longer texts!*

generated_responses = generator(input_text, max_length=512, num_return_sequences=num_output_responses)

print("\n *********** INPUT PROMPT TO THE MODEL ************ \n")
pprint(input_text)

print("\n *********** EXTENDED RESPONSES BY THE MODEL ************ \n")
for response in generated_responses:
    pprint(response["generated_text"][len(input_text):] + " ..."); print()
```

Next, we'll ask the pre-trained language models to calculate the likelihood of already existing text-extensions. We can define a positive text-extension as well as a negative text-extension. The sentiment of the given text context can then be determined by comparing the likelihoods of the given text extensions.

(For a positive review, a positive text-extension should ideally be given more likelihood by the pre-trained langauge model as compared to a negative text-extension. Similarly, for a negative review, the negative text-extension should have more likelihood than the positive text-extension.)

```python
# @title Bonus 1.4: Sentiment binary-classification with likelihood of positive and negative extensions of the review 👍👎

# @markdown ---
model_name = "gpt2" #@param ["gpt2", "gpt2-medium", "xlnet-base-cased"]
model = AutoModelForCausalLM.from_pretrained(model_name)
model.eval()
tokenizer = AutoTokenizer.from_pretrained(model_name)
# @markdown **Select a pre-trained language model to score the likelihood of extended review**

# @markdown *(might take some time to download the pre-trained weights for the first time)*

# @markdown ---
custom_positive_extension = "I would definitely recommend this!" #@param {type:"string"}
custom_negative_extension = "I would not recommend this!" #@param {type:"string"}
# @markdown **Provide custom positive and negative extensions to the review ✍**

texts = [context, custom_positive_extension, custom_negative_extension]
encodings = tokenizer(texts)

positive_input_ids = torch.tensor(encodings["input_ids"][0] + encodings["input_ids"][1])
positive_attention_mask = torch.tensor(encodings["attention_mask"][0] + encodings["attention_mask"][1])
positive_label_ids = torch.tensor([-100]*len(encodings["input_ids"][0]) + encodings["input_ids"][1])
outputs = model(input_ids=positive_input_ids,
                attention_mask=positive_attention_mask,
                labels=positive_label_ids)
positive_extension_likelihood = -1*outputs.loss
print("\nLog-likelihood of positive extension = ", positive_extension_likelihood.item())

negative_input_ids = torch.tensor(encodings["input_ids"][0] + encodings["input_ids"][2])
negative_attention_mask = torch.tensor(encodings["attention_mask"][0] + encodings["attention_mask"][2])
negative_label_ids = torch.tensor([-100]*len(encodings["input_ids"][0]) + encodings["input_ids"][2])
outputs = model(input_ids=negative_input_ids,
                attention_mask=negative_attention_mask,
                labels=negative_label_ids)
negative_extension_likelihood = -1*outputs.loss
print("\nLog-likelihood of negative extension = ", negative_extension_likelihood.item())

if (positive_extension_likelihood.item() > negative_extension_likelihood.item()):
    print("\nPositive text-extension has greater likelihood probabilities!")
    print("The given review can be predicted to be POSITIVE 👍")
else:
    print("\nNegative text-extension has greater likelihood probabilities!")
    print("The given review can be predicted to be NEGATIVE 👎")
# @markdown **NOTE:** *Run this cell after setting all the fields appropriately!*

# @markdown **NOTE:** *Some pre-trained models might not work well with longer texts!*
```

---
# Bonus 2: Light-weight fine-tuning

*Time estimate: ~10mins*

```python
# @title Video 9: Fine-tuning
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1CU4y1n7bV", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"buZLOKdf7Qw", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 9: Fine-tuning')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Fine-tuning these large pre-trained models with billions of parameters tends to be very slow. In this section, we will explore the effect of fine-tuning a few layers (while fixing the others) to save training time.

The HuggingFace python library provides a simplified API for training and fine-tuning transformer language models. In this exercise we will fine-tune a pre-trained language model for sentiment classification.

##Bonus 2.1: Data Processing

Pre-trained transformer models have a fixed vocabulary of words and sub-words. The input text to a transformer model has to be tokenized into these words and sub-words during the pre-processing stage. We'll use the HuggingFace `tokenizers` to perform the tokenization here.

(By default we'll use the BERT base-cased pre-trained language model here. You can try using one of the other models available [here](https://huggingface.co/transformers/pretrained_models.html) by changing the model ID values at appropriate places in the code.)

Most of the pre-trained language models have a fixed maximum sequence length. With the HuggingFace `tokenizer` library, we can either pad or truncate input text sequences to maximum length with a few lines of code:

```python
# Tokenize the input texts
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
def tokenize_function(examples):
  return tokenizer(examples["text"], padding="max_length", truncation=True)

# Here we use the `DATASET` as defined above.
# Recall that DATASET = load_dataset("yelp_review_full")
tokenized_datasets = DATASET.map(tokenize_function, batched=True)
```

We'll randomly sample a subset of the [Yelp reviews dataset](https://huggingface.co/datasets/yelp_review_full) (10k train samples, 5k samples for validation & testing each). You can include more samples here for better performance (at the cost of longer training times!)

```python
# Select the data splits
train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(10000))
test_dataset = tokenized_datasets["test"].select(range(0,5000))
validation_dataset = tokenized_datasets["test"].select(range(5000, 10000))
```

## Bonus 2.2: Model Loading

Next, we'll load a pre-trained checkpoint fo the model and decide which layers are to be fine-tuned.

Modify the `train_layers` variable below to pick which layers you would like to fine-tune (you can uncomment the print statements for this). Fine-tuning more layers might result in better performance (at the cost of longer training times). Due to computational limitations (limited GPU memory) we cannot fine-tune the entire model.

```python
# Load pre-trained BERT model and freeze layers
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased",
                                                           num_labels=5)
train_layers = ["classifier", "bert.pooler", "bert.encoder.layer.11"]  # add/remove layers here (use layer-name sub-strings)

for name, param in model.named_parameters():
  if any(x in name for x in train_layers):
    param.requires_grad = True
    # print("FINE-TUNING -->", name)
  else:
    param.requires_grad = False
    # print("FROZEN -->", name)
```

## Bonus 2.3: Fine-tuning

Fine-tune the model! The HuggingFace `Trainer` class supports easy fine-tuning and logging. You can play around with various hyperparameters here!

```python
# Setup huggingface trainer
training_args = TrainingArguments(output_dir="yelp_bert",
                                  overwrite_output_dir=True,
                                  evaluation_strategy="epoch",
                                  per_device_train_batch_size=64,
                                  per_device_eval_batch_size=64,
                                  learning_rate=5e-5,
                                  weight_decay=0.0,
                                  num_train_epochs=1,  # students may use 5 to see a full training!
                                  fp16=True,
                                  save_steps=50,
                                  logging_steps=10,
                                  report_to="tensorboard"
                                  )
```

We'll use `Accuracy` as the evaluation metric for the sentiment classification task. The HuggingFace `datasets` library supports various metrics. You can try experimenting with other classification metrics here!

```python
# Setup evaluation metric
metric = load_metric("accuracy")
def compute_metrics(eval_pred):
  logits, labels = eval_pred
  predictions = np.argmax(logits, axis=-1)
  return metric.compute(predictions=predictions, references=labels)
```

Start the training!

```python
# Instantiate a trainer with training and validation datasets
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=validation_dataset,
    compute_metrics=compute_metrics,
)
```

```python
# Train the model
trainer.train()
```

```python
# Evaluate the model on the test dataset
trainer.evaluate(test_dataset)
```

We can now visualize the `Tensorboard` logs to analyze the training process! The HuggingFace `Trainer` class will log various loss values and evaluation metrics automatically!

```python
# Visualize the tensorboard logs
%tensorboard --logdir yelp_bert/runs
```

---
# Bonus 3: Model robustness

*Time estimate: ~22mins*

```python
# @title Video 10: Robustness
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Y54y1E77J", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"hJdV2L2t4-c", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 10: Robustness')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Given the previously trained model for sentiment classification, it is possible to decieve it using various text perturbations. The text perturbations can act as previously unseen noise to the model, which might make it give out wrong values of sentiment!

## Bonus Interactive Demo 3: Break the model

```python
# @title Bonus 3.1: Load an original review

def clean_text(text):
    text = text.replace("\\n", " ")
    text = text.replace("\n", " ")
    text = text.replace("\\", " ")
    return text

# @markdown ---
sample_review_from_yelp = "Sentiment = 4" #@param ["Sentiment = 0", "Sentiment = 1", "Sentiment = 2", "Sentiment = 3", "Sentiment = 4"]
# @markdown **Randomly sample a response from the Yelp review dataset with the given sentiment value {0:😠, 1:😦, 2:😐, 3:🙂, 4:😀}**

# @markdown ---

context = clean_text(sentiment_dict[sample_review_from_yelp][random.randint(0,len(sentiment_dict[sample_review_from_yelp])-1)]["text"])

print("Review for ", sample_review_from_yelp, ":\n")
pprint(context)
```

We can apply various text perturbations to the selected review using the `textattack` python library. This will help us augment the original text to break the model!

```python
"""
Augmenter Class
===================
"""
from textattack.constraints import PreTransformationConstraint
from textattack.shared import AttackedText, utils

class Augmenter:
    """A class for performing data augmentation using TextAttack.

    Returns all possible transformations for a given string. Currently only
        supports transformations which are word swaps.

    Args:
        transformation (textattack.Transformation): the transformation
            that suggests new texts from an input.
        constraints: (list(textattack.Constraint)): constraints
            that each transformation must meet
        pct_words_to_swap: (float): [0., 1.], percentage of words to swap per augmented example
        transformations_per_example: (int): Maximum number of augmentations
            per input
    """

    def __init__(
        self,
        transformation,
        constraints=[],
        pct_words_to_swap=0.1,
        transformations_per_example=1,
    ):
        assert (
            transformations_per_example > 0
        ), "transformations_per_example must be a positive integer"
        assert 0.0 <= pct_words_to_swap <= 1.0, "pct_words_to_swap must be in [0., 1.]"
        self.transformation = transformation
        self.pct_words_to_swap = pct_words_to_swap
        self.transformations_per_example = transformations_per_example

        self.constraints = []
        self.pre_transformation_constraints = []
        for constraint in constraints:
            if isinstance(constraint, PreTransformationConstraint):
                self.pre_transformation_constraints.append(constraint)
            else:
                self.constraints.append(constraint)

    def _filter_transformations(self, transformed_texts, current_text, original_text):
        """Filters a list of ``AttackedText`` objects to include only the ones
        that pass ``self.constraints``."""
        for C in self.constraints:
            if len(transformed_texts) == 0:
                break
            if C.compare_against_original:
                if not original_text:
                    raise ValueError(
                        f"Missing `original_text` argument when constraint {type(C)} is set to compare against "
                        f"`original_text` "
                    )

                transformed_texts = C.call_many(transformed_texts, original_text)
            else:
                transformed_texts = C.call_many(transformed_texts, current_text)
        return transformed_texts

    def augment(self, text):
        """Returns all possible augmentations of ``text`` according to
        ``self.transformation``."""
        attacked_text = AttackedText(text)
        original_text = attacked_text
        all_transformed_texts = set()
        num_words_to_swap = max(
            int(self.pct_words_to_swap * len(attacked_text.words)), 1
        )
        for _ in range(self.transformations_per_example):
            current_text = attacked_text
            words_swapped = len(current_text.attack_attrs["modified_indices"])

            while words_swapped < num_words_to_swap:
                transformed_texts = self.transformation(
                    current_text, self.pre_transformation_constraints
                )

                # Get rid of transformations we already have
                transformed_texts = [
                    t for t in transformed_texts if t not in all_transformed_texts
                ]

                # Filter out transformations that don't match the constraints.
                transformed_texts = self._filter_transformations(
                    transformed_texts, current_text, original_text
                )

                # if there's no more transformed texts after filter, terminate
                if not len(transformed_texts):
                    break

                current_text = random.choice(transformed_texts)

                # update words_swapped based on modified indices
                words_swapped = max(
                    len(current_text.attack_attrs["modified_indices"]),
                    words_swapped + 1,
                )
            all_transformed_texts.add(current_text)
        return sorted([at.printable_text() for at in all_transformed_texts])

    def augment_many(self, text_list, show_progress=False):
        """Returns all possible augmentations of a list of strings according to
        ``self.transformation``.

        Args:
            text_list (list(string)): a list of strings for data augmentation
        Returns a list(string) of augmented texts.
        """
        if show_progress:
            text_list = tqdm.tqdm(text_list, desc="Augmenting data...")
        return [self.augment(text) for text in text_list]

    def augment_text_with_ids(self, text_list, id_list, show_progress=True):
        """Supplements a list of text with more text data.

        Returns the augmented text along with the corresponding IDs for
        each augmented example.
        """
        if len(text_list) != len(id_list):
            raise ValueError("List of text must be same length as list of IDs")
        if self.transformations_per_example == 0:
            return text_list, id_list
        all_text_list = []
        all_id_list = []
        if show_progress:
            text_list = tqdm.tqdm(text_list, desc="Augmenting data...")
        for text, _id in zip(text_list, id_list):
            all_text_list.append(text)
            all_id_list.append(_id)
            augmented_texts = self.augment(text)
            all_text_list.extend
            all_text_list.extend([text] + augmented_texts)
            all_id_list.extend([_id] * (1 + len(augmented_texts)))
        return all_text_list, all_id_list

    def __repr__(self):
        main_str = "Augmenter" + "("
        lines = []
        # self.transformation
        lines.append(utils.add_indent(f"(transformation):  {self.transformation}", 2))
        # self.constraints
        constraints_lines = []
        constraints = self.constraints + self.pre_transformation_constraints
        if len(constraints):
            for i, constraint in enumerate(constraints):
                constraints_lines.append(utils.add_indent(f"({i}): {constraint}", 2))
            constraints_str = utils.add_indent("\n" + "\n".join(constraints_lines), 2)
        else:
            constraints_str = "None"
        lines.append(utils.add_indent(f"(constraints): {constraints_str}", 2))
        main_str += "\n  " + "\n  ".join(lines) + "\n"
        main_str += ")"
        return main_str
```

```python
# @title Bonus 3.2: Augment the original review

# @markdown ---
# @markdown Word-level Augmentations
word_swap_contract = True  # @param {type:"boolean"}
word_swap_extend = False  # @param {type:"boolean"}
word_swap_homoglyph_swap = False  # @param {type:"boolean"}

# @markdown ---
# @markdown Character-level Augmentations
word_swap_neighboring_character_swap = True  # @param {type:"boolean"}
word_swap_qwerty = False  # @param {type:"boolean"}
word_swap_random_character_deletion = False  # @param {type:"boolean"}
word_swap_random_character_insertion = False  # @param {type:"boolean"}
word_swap_random_character_substitution = False  # @param {type:"boolean"}
# @markdown ---

# @markdown Check all the augmentations that you wish to apply!

# @markdown **NOTE:** *Try applying each augmentation individually, and observe the changes.*

# Apply augmentations
augmentations = []
if word_swap_contract:
  augmentations.append(WordSwapContract())
if word_swap_extend:
  augmentations.append(WordSwapExtend())
if word_swap_homoglyph_swap:
  augmentations.append(WordSwapHomoglyphSwap())
if word_swap_neighboring_character_swap:
  augmentations.append(WordSwapNeighboringCharacterSwap())
if word_swap_qwerty:
  augmentations.append(WordSwapQWERTY())
if word_swap_random_character_deletion:
  augmentations.append(WordSwapRandomCharacterDeletion())
if word_swap_random_character_insertion:
  augmentations.append(WordSwapRandomCharacterInsertion())
if word_swap_random_character_substitution:
  augmentations.append(WordSwapRandomCharacterSubstitution())

transformation = CompositeTransformation(augmentations)
augmenter = Augmenter(transformation=transformation,
                      transformations_per_example=1)
augmented_review = clean_text(augmenter.augment(context)[0])
print("Augmented review:\n")
pprint(augmented_review)
```

We can now check the predictions for the original text and its augmented version! Try to find the perfect combination of perturbations to break the model! (i.e. model giving incorrect prediction for the augmented text)

```python
# @title Bonus 3.3: Check model predictions
def getPrediction(text):
  inputs = tokenizer(text, padding="max_length", truncation=True, return_tensors="pt")
  for key, value in inputs.items():
    inputs[key] = value.to(model.device)

  outputs = model(**inputs)
  logits = outputs.logits
  pred = torch.argmax(logits, dim=1)
  return pred.item()

print("original Review:\n")
pprint(context)
print("\nPredicted Sentiment =", getPrediction(context))
print("########################################")
print("\nAugmented Review:\n")
pprint(augmented_review)
print("\nPredicted Sentiment =", getPrediction(augmented_review))
print("########################################")
```


last edited by Claire Valva on April 22, 2019, with cleanup on July 24, 2019

# Tests simulation that includes phase locking
this didn't really work/ phase locking is kind of a nonfactor

```python
#!jupyter nbconvert --to python phase_lock_sim.ipynb
```

    [NbConvertApp] Converting notebook phase_lock_sim.ipynb to python
    [NbConvertApp] Writing 8151 bytes to phase_lock_sim.py

```python
import numpy as np
from netCDF4 import Dataset, num2date
from scipy.signal import get_window, csd
from scipy.fftpack import fft, ifft, fftshift, fftfreq
import pandas as pd
import datetime
from math import pi
from scipy import optimize
from astropy.stats.circstats import circmean, circvar
from os import walk
from multiprocessing import Pool
from sympy import Poly, Eq, Function, exp, re, im
import pickle
import time
import random
import multiprocessing as mp
from joblib import Parallel, delayed
```

```python
#get detrended parts
f = []
for (dirpath, dirnames, filenames) in walk('detrended/'):
    f.extend(filenames)
    break
```

```python
# root finders
#use these
def solve_f(X, solution):
    #function to solve f coeff equation for trend analysis
    x,y = X
    f = x*np.exp(1j*y) - solution
    return [np.real(f), np.imag(f)]

def real_f(X, solution):
    #function to wrap solve_f so that it can be used with fsolve
    x,y = X
    z = [x+0j,y+0j]
    actual_f = solve_f(z, solution)
    return(actual_f)

# solve for phase
def root_find(sol):
    return optimize.root(real_f, [np.real(sol), 0], args=sol).x
```

```python
# get function to generate random coeffs
def entry_fft(amp, phase = random.uniform(0, 2*pi)):
    # takes amplitude and phase to give corresponding fourier coeff
    entry = amp*np.exp(1j*phase)
    return entry

# write functions to make a longer ifft
def ext_row(row, n):
    ext_f = np.zeros(((len(row) - 1) * n + 1,), dtype="complex128")
    ext_f[::n] = row * n

    return ext_f

def ext_ifft_new(n, input_array):
    # add the zeros onto each end
    ext_f = [ext_row(entry,n) for entry in input_array]

    # make up for the formulat multiplying for array length
    olddim = len(input_array[5])
    newdim = len(ext_f[0])
    mult = newdim/olddim

    ext_f = np.multiply(ext_f, mult)
    adjusted_tested = np.fft.ifft2(ext_f)

    return adjusted_tested

season_titles = ["Winter", "Spring", "Summer", "Fall"]

# flatten season for plotting
flatten = lambda l: [item for sublist in l for item in sublist]
```

```python
file_pickle = open("spectra_copy/spectra_02_45.0Narr.pickle", "rb")
d2_touse, d2_seasons, d2_averages = pickle.load(file_pickle, encoding='latin1')
```

```python
filed = ["spectra/spectra_02_45.0Sarr.pickle",
         "spectra/spectra_02_45.0Narr.pickle"]
```

```python
mean_phases_lock = [[-1.20929458e-16,  1.65918271e-01, -2.17292412e-01, -2.40352609e-01,
        8.64205449e-02,  1.07202695e-02],[-1.21105919e-16,  3.96836386e-01, -5.77513605e-01,  5.62200988e-01,
        3.64883992e-01, -7.35447431e-02]]
```

```python
stds_lock = [[0.        , 0.35092645, 0.37481109, 0.36100874, 0.35869798,
       0.36139656], [0.        , 0.22681627, 0.40726111, 0.39961749, 0.37641118,
       0.36154913]]
```

```python
for heythere in range(1):
# get function to generate random coeffs
    def entry_fft(amp,std,wavenum, phase = random.uniform(0, 2*pi)):
    # takes amplitude and phase to give corresponding fourier coeff
        if np.abs(wavenum) <= 6:
            phase = np.random.normal(loc = mean_phases_lock[ko][wavenum], scale = stds_lock[ko][wavenum])

        amp_new = np.random.normal(loc = amp, scale = std)
        entry = amp_new*np.exp(1j*phase)
        return entry

    # write functions to make a longer ifft
    def ext_row(row, n):
        ext_f = np.zeros(((len(row) - 1) * n + 1,), dtype="complex128")
        ext_f[::n] = row * n

        return ext_f

    def ext_ifft_new(n, input_array):
    # add the zeros onto each end
        ext_f = [ext_row(entry,n) for entry in input_array]

        # make up for the formulat multiplying for array length
        olddim = len(input_array[5])
        newdim = len(ext_f[0])
        mult = newdim/olddim

        # ext_f = np.multiply(mult, ext_f)
        adjusted_tested = np.fft.ifft2(ext_f)

        return adjusted_tested

    def combined(amps,stds, length):
    # combines generation of random phase with inverse transform
        newarray = [[entry_fft(amp = amps[wave][timed],
                               std = stds[wave][timed], wavenum = wave,
                               phase = random.uniform(0, 2*pi))
                     for timed in range(len(amps[wave]))]
                    for wave in range(len(amps))]

        newarray = [np.array(leaf) for leaf in newarray]
        iffted = ext_ifft_new(length, newarray)

        return iffted
```

```python
for ko in range(2):
    file_pickle = open(filed[ko], "rb")
    d2_touse, d2_seasons, d2_averages = pickle.load(file_pickle, encoding='latin1')

    alled = [[[[root_find(entry) for entry in sublist]
                   for sublist in year]
                  for year in season]
                 for season in d2_seasons]
    phases = roots[:,:,:,1]
    amps = roots[:,:,:,0]

    def padded(to_pad, index):
        length = len(to_pad)
        if index == 0:
            zeros = longl - length
            to_pad = list(to_pad)
            for i in range(zeros):
                to_pad.append(0)
            return to_pad
        else:
            return to_pad

    #pad rows with zeros to account for leap year
    season_amps_adj = [[[padded(row, index = i)
                         for row in entry]
                        for entry in amps[i]]
                       for i in range(4)]

    #get average amplitude and phases for each season
    avg_amps = [np.average(season, axis = 0)
                for season in season_amps_adj]

    #get average amplitude and phases for each season
    std_amps = [np.std(season, axis = 0)
                for season in season_amps_adj]

    def entry_fft(amp,std,wavenum, phase = random.uniform(0, 2*pi)):
    # takes amplitude and phase to give corresponding fourier coeff
        if np.abs(wavenum) <= 6:
            phase = np.random.normal(loc = mean_phases_lock[ko][wavenum], scale = stds_lock[ko][wavenum])

        amp_new = np.random.normal(loc = amp, scale = std)
        entry = amp_new*np.exp(1j*phase)
        return entry

    # write functions to make a longer ifft
    def ext_row(row, n):
        ext_f = np.zeros(((len(row) - 1) * n + 1,), dtype="complex128")
        ext_f[::n] = row * n

        return ext_f

    def ext_ifft_new(n, input_array):
    # add the zeros onto each end
        ext_f = [ext_row(entry,n) for entry in input_array]

        # make up for the formulat multiplying for array length
        olddim = len(input_array[5])
        newdim = len(ext_f[0])
        mult = newdim/olddim

        # ext_f = np.multiply(mult, ext_f)
        adjusted_tested = np.fft.ifft2(ext_f)

        return adjusted_tested

    def combined(amps,stds, length):
    # combines generation of random phase with inverse transform
        newarray = [[entry_fft(amp = amps[wave][timed],
                               std = stds[wave][timed], wavenum = wave,
                               phase = random.uniform(0, 2*pi))
                     for timed in range(len(amps[wave]))]
                    for wave in range(len(amps))]

        newarray = [np.array(leaf) for leaf in newarray]
        iffted = ext_ifft_new(length, newarray)

        return iffted

    def repeater(season, stds, length, times):
        # repeats the phase creation and inverse transform
        newarray = [combined(season,stds,length) for leaf in range(times)]
        return(newarray)

    def repeater_2(amps,stds, length, times):
        #do procedure
        repeated_comp = [repeater(amps[i],stds[i], length, times)
                     for i in range(4)]

    #output.put(repeated_comp)

    #listed_parts.append(repeated_comp)

        import pickle

        file_name2 = "sim_samp/"
        file_pickle = open(file_name2,'wb')
        pickle.dump(repeated_comp,file_pickle)
        file_pickle.close()

        return repeated_comp

    runlen = 70
    runtimes = 4
    toplot = repeater_2(avg_amps,std_amps, runlen, runtimes)
```

```python

```

```python

```