# First Concepts
Reinforcement Learning is all about training agents in an environement with or without prior knowledge about the problem to solve.

# States and Observations
- It's a complete overview and vision of the environement.
- An Observation is a partial view of the state.
- Both of them are represented by vectors or matrices.
- If Observation = State we say that the environement is "Fully Observed" else it's "Partially Observed" (no complete information)
-  The action is conditioned on the observation because the agent does not have access to the state.

# Action Spaces
- Set of actions avaible for the agent for the actual environement depending on the state.
- Some games have **discrete** action spaces (finite number of moves) like **GO** and **Chess** and other ones have **Continuous** one like : controlling a **robot in real environement** or **Starcraft II** ( real valued vectors ).

# Policy
Represent the rule for the agent on how to choose the correct action (his brain).
It can be deterministic denoted by $\mu$
\begin{align}
a_t & = \mu(S_t)
\end{align}

Or stochastic denoted by $ \pi $ :
\begin{align}
a_t & = \pi_\theta(.|S_t)
\end{align}

In deep RL, we deal with parameterized policies: policies whose outputs are computable functions that depend on a set of parameters (eg the weights and biases of a neural network) which we can adjust to change the behavior via some optimization algorithm.

## Stochastic Policies
The two most common kinds of stochastic policies in deep RL are categorical policies and diagonal Gaussian policies.

Categorical policies can be used in discrete action spaces, while diagonal Gaussian policies are used in continuous action spaces.

Two key computations are centrally important for using and training stochastic policies:

- sampling actions from the policy,
- and computing log likelihoods of particular actions $\log \pi_{\theta}(a|s)$ .

### Categorical Policies
A categorical policy is like a classifier over discrete actions. You build the neural network for a categorical policy the same way you would for a classifier: the input is the observation, followed by some number of layers (possibly convolutional or densely-connected, depending on the kind of input), and then you have one final linear layer that gives you logits for each action, followed by a softmax to convert the logits into probabilities.

**Sampling** Given the probabilities for each action, frameworks like Tensorflow have built-in tools for sampling. For example, see the tf.distributions.Categorical documentation, or tf.multinomial.

**Log-Likelihood** Denote the last layer of probabilities as P_{\theta}(s). It is a vector with however many entries as there are actions, so we can treat the actions as indices for the vector. The log likelihood for an action a can then be obtained by indexing into the vector:
$ \log \pi_{\theta}(a|s) = \log \left[P_{\theta}(s)\right]_a $ .

## Trajectory
A trajectory $\tau$ is a sequence of states and actions in the world,

$\tau = (s_0, a_0, s_1, a_1, ...)$.

The very first state of the world, $s_0$, is randomly sampled from the start-state distribution, sometimes denoted by $\rho_0$:

$s_0 \sim \rho_0(\cdot)$.

State transitions (what happens to the world between the state at time t, $s_t$, and the state at t+1, $s_{t+1})$, are governed by the natural laws of the environment, and depend on only the most recent action, $a_t$. They can be either deterministic,

$s_{t+1} = f(s_t, a_t)$

or stochastic,

$s_{t+1} \sim P(\cdot|s_t, a_t)$.

Actions come from an agent according to its policy.

# Reward and Return
The reward function **R** is the most important part of Reinforcement Learning, it depends on the current state, the action taken and the resulting state:
\begin{align}
r_t = R(s_t, a_t, s_{t+1})
\end{align}

although frequently this is simplified to just a dependence on the current state, $r_t = R(s_t)$, or state-action pair $r_t = R(s_t,a_t)$.
The goal of the agent is to maximize some notion of cumulative reward over a trajectory.

One kind of return is the finite-horizon undiscounted return, which is just the sum of rewards obtained in a fixed window of steps:

$R(\tau) = \sum_{t=0}^T r_t$.

Another kind of return is the infinite-horizon discounted return, which is the sum of all rewards ever obtained by the agent, but discounted by how far off in the future they’re obtained. This formulation of reward includes a discount factor \gamma \in (0,1):

$R(\tau) = \sum_{t=0}^{\infty} \gamma^t r_t$.

Why would we ever want a discount factor, though? Don’t we just want to get all rewards? We do, but the discount factor is both intuitively appealing and mathematically convenient. On an intuitive level: cash now is better than cash later. Mathematically: an infinite-horizon sum of rewards may not converge to a finite value, and is hard to deal with in equations. But with a discount factor and under reasonable conditions, the infinite sum converges.

## Reinforcement Learning Problem
Whatever Return measure is chosen, the goal of Reinforcement Learning agent is to develop a policy to maximize it
To talk about expected return, we first have to talk about probability distributions over trajectories.

Let’s suppose that both the environment transitions and the policy are stochastic. In this case, the probability of a T -step trajectory is:

$P(\tau|\pi) = \rho_0 (s_0) \prod_{t=0}^{T-1} P(s_{t+1} | s_t, a_t) \pi(a_t | s_t)$

The expected return (for whichever measure), denoted by J(\pi), is then:

$J(\pi) = \int_{\tau} P(\tau|\pi) R(\tau) = \underE{\tau\sim \pi}{R(\tau)}$

The central optimization problem in RL can then be expressed by

$\pi^* = \arg \max_{\pi} J(\pi)$

with $\pi^*$ being the optimal policy.

## Value Function
It’s often useful to know the value of a state, or state-action pair. By value, we mean the expected return if you start in that state or state-action pair, and then act according to a particular policy forever after. Value functions are used, one way or another, in almost every RL algorithm.

There are four main functions of note here.
The On-Policy Value Function, $V^{\pi}(s)$, which gives the expected return if you start in state s and always act according to policy \pi:

$V^{\pi}(s) = \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s\right.}$

The On-Policy Action-Value Function, $Q^{\pi}(s,a)$, which gives the expected return if you start in state s, take an arbitrary action a (which may not have come from the policy), and then forever after act according to policy \pi:

$Q^{\pi}(s,a) = \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s, a_0 = a\right.}$

The Optimal Value Function, V^*(s), which gives the expected return if you start in state s and always act according to the optimal policy in the environment:

$V^*(s) = \max_{\pi} \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s\right.}$

The Optimal Action-Value Function, $Q^*(s,a)$, which gives the expected return if you start in state s, take an arbitrary action a, and then forever after act according to the optimal policy in the environment:

$Q^*(s,a) = \max_{\pi} \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s, a_0 = a\right.}$

You Should Know

When we talk about value functions, if we do not make reference to time-dependence, we only mean expected infinite-horizon discounted return. Value functions for finite-horizon undiscounted return would need to accept time as an argument. Can you think about why? Hint: what happens when time’s up?

You Should Know

There are two key connections between the value function and the action-value function that come up pretty often:

$V^{\pi}(s) = \underE{a\sim \pi}{Q^{\pi}(s,a)}$

and

$V^*(s) = \max_a Q^* (s,a)$

These relations follow pretty directly from the definitions just given: can you prove them?

## The Optimal Q-Function and the Optimal Action
There is an important connection between the optimal action-value function $Q^*(s,a)$ and the action selected by the optimal policy. By definition, $Q^*(s,a)$ gives the expected return for starting in state s, taking (arbitrary) action a, and then acting according to the optimal policy forever after.

The optimal policy in s will select whichever action maximizes the expected return from starting in s. As a result, if we have $Q^*$, we can directly obtain the optimal action, $a^*(s)$, via

$a^*(s) = \arg \max_a Q^* (s,a)$

Note: there may be multiple actions which maximize $Q^*(s,a)$, in which case, all of them are optimal, and the optimal policy may randomly select any of them. But there is always an optimal policy which deterministically selects an action.

 ## Advantage Function
 Because sometimes we don't need to measure the best action, but to know how much an action is better than the other ones, that's the goal of the advantage function
 The advantage function $A^{\pi}(s,a)$ corresponding to a policy $\pi$ describes how much better it is to take a specific action a in state s, over randomly selecting an action according to $\pi(\cdot|s)$, assuming you act according to $\pi$ forever after. Mathematically, the advantage function is defined by

$A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s)$

## Formalism

So far, we’ve discussed the agent’s environment in an informal way, but if you try to go digging through the literature, you’re likely to run into the standard mathematical formalism for this setting: Markov Decision Processes (MDPs). An MDP is a 5-tuple, \langle S, A, R, P, \rho_0 \rangle, where

- S is the set of all valid states,
- A is the set of all valid actions,
- R : $S \times A \times S \to \mathbb{R}$ is the reward function, with $r_t = R(s_t, a_t, s_{t+1})$,
- P : $S \times A \to \mathcal{P}(S)$ is the transition probability function, with P(s'|s,a) being the probability of transitioning into state s' if you start in state s and take action a,
    and $\rho_0$ is the starting state distribution.

The name Markov Decision Process refers to the fact that the system obeys the Markov property: transitions only depend on the most recent state and action, and no prior history.

## Bellman Equation
All four of the value functions obey special self-consistency equations called Bellman equations. The basic idea behind the Bellman equations is this:

    The value of your starting point is the reward you expect to get from being there, plus the value of wherever you land next.

The Bellman equations for the on-policy value functions are

\begin{align*} V^{\pi}(s) &= \undersetE{a \sim \pi \\ s'\sim P}{r(s,a) + \gamma V^{\pi}(s')}, \\ Q^{\pi}(s,a) &= \undersetE{s'\sim P}{r(s,a) + \gamma \undersetE{a'\sim \pi}{Q^{\pi}(s',a')}}, \end{align*}

where s' \sim P is shorthand for s' \sim P(\cdot |s,a), indicating that the next state s' is sampled from the environment’s transition rules; a \sim \pi is shorthand for a \sim \pi(\cdot|s); and a' \sim \pi is shorthand for a' $\sim \pi(\cdot|s')$.

The Bellman equations for the optimal value functions are

\begin{align*} V^*(s) &= \max_a \undersetE{s'\sim P}{r(s,a) + \gamma V^*(s')}, \\ Q^*(s,a) &= \undersetE{s'\sim P}{r(s,a) + \gamma \max_{a'} Q^*(s',a')}. \end{align*}

The crucial difference between the Bellman equations for the on-policy value functions and the optimal value functions, is the absence or presence of the $\max$ over actions. Its inclusion reflects the fact that whenever the agent gets to choose its action, in order to act optimally, it has to pick whichever action leads to the highest value.

```python

```

Spinning Up in Deep RL/Introduction to RL.ipynb


## Домашнее задание №2, теоретическое. Логические методы

Решите предложенные задачи. Каждая задача должна быть подробно обоснована, задачи без
обоснования не засчитываются. Решения пишутся в свободной форме, однако так, чтобы проверяющий смог разобраться. Если проверяющий не смог разобраться в решении какой-нибудь задачи, то
она автоматически не засчитывается. Если используются какие-либо внешние источники, их нужно обязательно указывать.

 ### Задача 1 (1 балл). Кроссвалидация, LOO, k-fold.

 Объясните, стоит ли использовать оценку `leave-one-out-CV` или `k-fold-CV` с небольшим $k$ в случае, когда:

 * обучающая выборка содержит очень малое количество объектов;
 * обучающая выборка содержит очень большое количество объектов.

### Задача 2 (1 балл). Логистическая регрессия, вывод функции потерь.

Рассмотрим выборку объектов $X = \{x_1,...,x_\ell\}$ и их целевых меток $Y = \{y_1,...,y_\ell\}$, где $y_i ∈ \{0, 1\}$. Предположим, что мы хотим обучить линейный классификатор:

$$Q(w, X^\ell) = \sum\limits_{i=1}^\ell\mathcal{L}(y_i\left<w,x_i\right>) \rightarrow \min\limits_w$$

где $w$ – веса линейной модели, $\mathcal{L}(y, z)$ – некоторая гладкая функция потерь.

Так как решается задача двухклассовой классификации, то будем обучать классификатор предсказывать вероятности принадлежности объекта классу $1$, то есть решать задачу логистической регрессии. Для измерения качества такого классификатора обычно используют правдоподобие $P(Y|X)$ целевых меток $Y$ при заданных объектах $X$ в соответствии с предсказанными распределениями $p$ — чем выше правдоподобие, тем точнее классификатор. Для удобства с вычислительной точки зрения обычно используется отрицательный логарифм правдоподобия, также называемый LogLoss (Logarithmic Loss). Будем считать, что пары объект-ответ $(x_i,y_i)$ независимы между собой для разных $i$.

#### 1. (0.5 балла)
Покажите что:

$$\text{LogLoss} =  -\text{LogLikelihood} = - \log(P(Y|X)) = - \sum\limits_{i=1}^\ell (y_i \log \tilde y_i + (1-y_i) \log (1-\tilde y_i))$$

#### 2. (0.5 балла)
Для того, чтобы классификатор возвращал числа из отрезка $[0,1]$, положите
$$
p(y_i=1|x_i) = \sigma \left ( \left< w, x_i \right> \right) =
\frac{1}{1 + \exp{\left(- \left< w, x_i \right> \right)}};
$$

сигмоидная функция монотонно возрастает, поэтому чем больше скалярное призведение, тем большая вероятность положительного класса будет предсказана объекту.

Подставьте трансформированный ответ линейной модели в логарифм правдоподобия. К какой функции потерь мы пришли? (Обратите внимание, что функция обычно записывается для классов $\{-1, 1\}$).

### Задача 3 (1.5 балла). Логистическая регрессия, решение оптимизационной задачи.

#### 1. (0.8 балла)

Докажите, что в случае линейно разделимой выборки не существует вектора параметров (весов), который бы максимизировал правдоподобие вероятностной модели логистической регрессии в задаче двухклассовой классификации.

#### 2. (0.4 балла)

Предложите, как можно модифицировать вероятностную модель, чтобы оптимум достигался.

#### 3. (0.3 балла)
Выпишите формулы пересчета значений параметров при оптимизации методом градиентного спуска для обычной модели логистической регрессии и предложенной модификации.

### Задача 4 (1 балл). Мультиномиальная регрессия

В случае многоклассовой классификации логистическую регрессию можно обобщить: пусть для каждого класса $k$ есть свой вектор весов $w_k$. Тогда вероятность принадлежности классу $k$ запишем следующим образом:

$$
P(y=k | x, W) = \frac{e^{\langle w_k, x\rangle}}{\sum\limits_{j=1}^K e^{\langle w_j, x\rangle}}
$$

Тогда оптимизируемая функция примет вид:

$$
\mathcal{L}_{sm}(W) = -\sum_{i=1}^N \sum_{k=1}^K [y_i=k]\ln P(y_i=k | x_i, W),~\text{где}~[y_i=k]=\begin{cases}
1, y_i=k,\\
0, \text{иначе}
\end{cases}
$$

Пусть количество классов $K=2$. Для простоты положим, что выборка линейно неразделима.

#### 1. (0.5 балла)
 Единственно ли решение задачи? Почему?

#### 2. (0.5 балла)
 Покажите, что предсказанные распределения вероятностей на классах в случае логистической и мультиномиальной регрессий будут совпадать.

### Задача 5 (1.5 балла) Решающие деревья, константное предсказание, функции потерь.

Допустим, при построении решающего дерева в некоторый лист попало $N$ объектов $x_1, ... , x_N$ с метками $y_1, ... , y_N$.
Предсказание в каждом листе дерева - константа. Найдите, какое значение $\tilde y$
должен предсказывать этот лист для минимизации следующих функций потерь:

#### 1. (0.5 балла)
Mean Squared Error (средний квадрат ошибки) для задачи регрессии:
        \begin{equation}Q=\frac{1}{N}\sum_{i=1}^N (y_i - \tilde y)^2;\end{equation}

#### 2. (0.5 балла)
 Mean Absolute Error (средний модуль отклонения) для задачи регрессии:
        \begin{equation}Q=\frac{1}{N}\sum_{i=1}^N |y_i - \tilde y|.\end{equation}

#### 3. (0.5 балла)
 $\text{LogLoss}$ (логарифмические потери) для задачи классификации:
        $$Q=-\frac{1}{N}\sum_{i=1}^N \left(y_i\log\tilde y+(1-y_i)\log(1-\tilde y)\right),
            \quad \tilde y\in[0,1], \quad y_i \in \{0,1\}.$$

 ### Задача 6 (1 балл). Решающие деревья, функции потерь, impurity functions.
$$
 \Phi(U) - \frac{|U_1|}{|U|}\Phi(U_1) - \frac{|U_2|}{|U|}\Phi(U_2) \to \max
$$
таким выражением в лекции задается критерий, по которому происходит ветвление вершины  решающего дерева. Давайте разберемся подробнее.

Impurity function $\Phi(U)$ («функция неопределенности» или «функция нечистоты») используется для того, чтобы измерить степень неоднородности целевых меток $y_1,\dots, y_\ell$ для множества объектов $U$ размера $\ell$. Например, при обучении решающего дерева в текущем листе  выбирается такое разбиение множества объектов $U$ на два непересекающихся множества $U_1$ и $U_2$, чтобы impurity function $\Phi(U)$ исходного множества $U$ как можно сильнее превосходила нормированную impurity function в новых листьях $\frac{|U_1|}{|U|}\Phi(U_1) + \frac{|U_2|}{|U|}\Phi(U_2)$. Отсюда и получается, что нужно выбрать разбиение, решающее  задачу
$$
 \Phi(U) - \frac{|U_1|}{|U|}\Phi(U_1) - \frac{|U_2|}{|U|}\Phi(U_2) \to \max.
$$
Полученную разность называют Gain (выигрыш), и она показывает, на сколько удалось уменьшить «неопределенность» от разбиения листа два новых.

В соответствии с одним из возможных определений, impurity function — это значение функционала ошибки $Q = \frac{1}{\ell}\sum\limits_{i=1}^\ell \mathcal{L}(y_i, \tilde{y})$ в листе с множеством объектов $U$ при константном предсказании $\tilde{y}$, оптимальном для $Q$ (см. задачу 7):
    $$
        \Phi(U) = \frac{1}{\ell}\sum\limits_{i=1}^\ell \mathcal L (y_i, \tilde y).
    $$
Понятно, что каждому критерию разбиения соответствует своя impurity function $\Phi(U)$, а в основе каждой $\Phi(U)$ лежит некоторая функция потерь. Давайте разберемся, откуда берутся различные критерии разбиения.

#### 1. (0.5 балла)
Покажите, что для квадратичных потерь $\mathcal L (y_i, \tilde y) = (y_i - \tilde y)^2$ в задаче регрессии $y_i \in \mathbb{R}$ impurity function $\Phi(U)$ равна выборочной дисперсии целевых меток объектов, попавших в лист дерева.

#### 2. (0.5 балла)
 Покажите, что для функции потерь $\text{Logloss}$  $\mathcal L (y_i, \tilde y) =-y_i\log(\tilde y) - (1-y_i)\log(1 - \tilde y)$ в задаче классификации $y_i \in \{0,1\}$ impurity function $\Phi(U)$ соответствует энтропийному критерию разбиения.

### Задача 7 (1 балл). Решающие деревья, индекс Джини

Пусть имеется построенное решающее дерево для задачи многоклассовой классификации. Рассмотрим лист дерева с номером $m$ и объекты $R_m$, попавшие в него. Обозначим за $p_{mk}$ долю объектов $k$-го класса в листе $m$. *Индексом Джини* этого листа называется величина
$$\sum_{k = 1}^{K} p_{mk} (1 - p_{mk}),$$
где $K$ — общее количество классов. Индекс Джини обычно служит мерой того, насколько хорошо в данном листе выделен какой-то один класс (см. impurity function в предыдущей задаче).

#### 1. (0.5 балла)
Поставим в соответствие листу $m$ алгоритм классификации $a(x)$, который предсказывает класс случайно, причем класс $k$ выбирается с вероятностью $p_{mk}$. Покажите, что матожидание частоты ошибок этого алгоритма на объектах из $R_m$ равно индексу Джини.

#### 2. (0.5 балла)
*Дисперсией класса $k$* назовем дисперсию выборки $\{ [y_i = k]:\ x_i \in R_m$\},
    где $y_i$ - класс объекта $x_i$, $[f]$ — индикатор истинности выражения $f$, равный 1 если $f$ верно, и нулю в противном случае, а $R_m$ — множество объектов в листе.
    Покажите, что сумма дисперсий всех классов в заданном листе равна его индексу Джини.

### Задача 8 (2 балла). Бинарные решающие деревья, MSE

Предложите алгоритм построения *оптимального* бинарного решающего дерева для задачи регрессии на $\ell$ объектах в $n$-мерном пространстве с асимптотической сложностью $O(n \ell \log \ell)$. В качестве предикатов нужно рассматривать пороговые правила (наиболее распространенный случай на практике). Для простоты можно считать, что  получающееся дерево близко к сбалансированному (т.е. его глубина имеет порядок $O(\log \ell)$) и в качестве функции ошибки  используется Mean Squared Error (MSE):
	$$Q=\frac{1}{\ell}\sum_{i=1}^\ell (y_i - \tilde y_i)^2$$
Под оптимальностью в данной задаче подразумевается, что в каждом узле дерева делается оптимальное с точки зрения MSE разбиение на два поддерева.


# Assignment 1
The first assignment has two parts. The first part concerns PyTorch and the second part is about feature engineering for a basic NLP task.
## Instructions

1.   Make a copy of this notebook
  - Click on "File -> Save a copy in Drive" and open it in Colab afterwards
  - Alternatively, download the notebook and work on it on your local machine. However, keep in mind that you will have to make sure it still runs on Colab afterwards and does not depend on any packages that you installed locally
2.   Rename your notebook to **surname_forename_studentnumber.ipynb**
  - Make sure to exactly follow this naming scheme (don't replace `_` with `-` or something like that)
  - **Failure to comply with this scheme results in -10 points!**
3.   For math exercises, use $\LaTeX$  to typset your answer
4.   For coding exercises, insert your code at `# TODO` statements
5.   For multiple-choice questions, choose an answer from the drop-down list
6.   Before submitting your notebook, **make sure that it runs without errors when executed from start to end on Colab**
  - To check this, reload your notebook and the Python kernel, and run the notebook from the first to the last cell
  - **If your notebook throws any errors, you will be penalized by -25 points in addition to any penalities from incorrect answers**
  - We are not going to fix any errors (no matter how small) to make your code work
7.  Download your notebook and submit it on Moodle
  - Click on "File -> Download .ipynb"

## Notebook Setup [don't change!]

```
%%shell
pip install torch
```

    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.7.0+cu101)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch) (0.16.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch) (3.7.4.3)
    Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch) (0.8)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.19.5)

```
import torch
from torch import nn
torch.__version__
```

    '1.7.0+cu101'

# Part I: PyTorch [50 points]

## Linear Algebra [30 points]

### PyTorch Tensors [5 points]

#### Construct Scaled Identity Matrix [1 point]
Given $n \in \mathbb{N}$ and $c \in \mathbb{R}$, construct a matrix $\mathbf{X} \in \mathbb{R}^{n\ \times\ n}$ where $\mathbf{X}$ has $c$ on its diagonal and zeros everywhere else.

```
def construct_scaled_identity(n, c):
  a = torch.zeros((n, n))
  # np.fill_diagonal(a, c)
  torch.diagonal(a).fill_(c)
  return a

construct_scaled_identity(4, 3.2)
```

    tensor([[3.2000, 0.0000, 0.0000, 0.0000],
            [0.0000, 3.2000, 0.0000, 0.0000],
            [0.0000, 0.0000, 3.2000, 0.0000],
            [0.0000, 0.0000, 0.0000, 3.2000]])

#### Mean Diagonal [1 point]
Given a square matrix $\mathbf{X}\in\mathbb{R}^{n\ \times\ n}$, return the mean of its diagonal.

```
def mean_diagonal(x):
  a = torch.mean(torch.diagonal(x))
  return a

x = torch.arange(0, 16, dtype=torch.float).view(4, 4)
mean_diagonal(x)
```

    tensor(7.5000)

#### Indexing [1 point]
Given a matrix $\mathbf{X}\in\mathbb{R}^{n\ \times\ m}$ and $i,j \in \mathbb{N}$, return the submatrix $\mathbf{Y}\in\mathbb{R}^{i\ \times\ j}$ of the last i rows and last j columns of $\mathbf{X}$ (i.e. the bottom right submatrix of the given size). You can assume that $i \leq n$ and $j \leq m$.

```
def bottom_right_matrix(x, i, j):
  a = x[-i:, -j:]
  return a

x = torch.arange(0, 12).view(3, 4)
bottom_right_matrix(x, 2, 2)
```

    tensor([[ 6,  7],
            [10, 11]])

#### Transpose Sum [2 points]
Given a tensor $\mathcal{X}\in\mathbb{R}^{i\ \times\ j\ \times\ k}$, return a transposed tensor $\mathcal{y}\in\mathbb{R}^{j\ \times\ i}$ whose values in the third dimension are summed up.

```
def transpose_sum(x):
  a = torch.sum(x, dim=2)
  a = torch.transpose(a, 0, 1)
  return a

x = torch.arange(0, 12).view(2, 3, 2)
transpose_sum(x)
```

    tensor([[ 1, 13],
            [ 5, 17],
            [ 9, 21]])

### Matrix-vector Multiplication [10 points]

Implement five unique ways for multiplying a matrix A with a vector b. **Each PyTorch function is allowed to be used in only one of the five implementations**. For instance, if you use `unsqueeze` in one of the methods, you are not allowed to use it for the other five implementations. Furthermore, functions in `torch` and in `torch.Tensor` are treated as the same function (i.e. using `torch.add(x, y)`, `x.add(y)` and `x + y` are all treated as the same function and hence are not allowed to be used in more than one implementation). Your code needs to be applicable to any matrix $A \in \mathbb{R}^{n\ \times\ m }$ and vector $b\in\mathbb{R}^m$.

```
def matrixvector1(A, b):
  return torch.matmul(A, b)

def matrixvector2(A, b):
  return A @ b

def matrixvector3(A, b):
  return A.mm(b.unsqueeze(1)).squeeze(-1)

def matrixvector4(A, b):
  return torch.einsum('ij,j->i', A, b)

def matrixvector5(A, b):
  return torch.mv(A, b)
```

### Backprop [15 points]

#### Forward [2 points]
Implement $\mathbf{y}\odot\text{tanh}\left(\mathbf{W}\mathbf{x}+\mathbf{b}\right)$ in PyTorch without using a linear layer implementation (i.e. do the matrix-vector mulitplication and addition of a bias term yourself). Note that we are not looking for a batched implementation, so assume $\mathbf{y},\mathbf{b} \in \mathbb{R}^n, \mathbf{x}\in\mathbb{R}^m$ and $\mathbf{W}\in\mathbb{R}^{n\ \times\ m}$

```
def fw(y, W, x, b):
  result = y * torch.tanh(torch.matmul(W, x) + b)
  return result
```

#### Gradient [10 points]
Derive $\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}}\left[\mathbf{y}\odot\text{tanh}\left(\mathbf{W}\mathbf{x}+\mathbf{b}\right)\right]$ analytically. Here $\mathbf{z}$ is an _upstream (error) gradient_ and we are interested in calculating the _downstream gradient_ for $\mathbf{x}$. Make sure to write down all intermediate steps and not just the final result.

Let's define:
\begin{equation}
h = \text{tanh}(z)
\\
z = Wx + b
\end{equation}
We can therefore write:
\begin{align}
\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}}\left[\mathbf{y}\odot\text{tanh}\left(\mathbf{W}\mathbf{x}+\mathbf{b}\right)\right]
&=\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}} [\mathbf{y}\odot{h}]
\end{align}
Also, by using the chain rule:
\begin{equation}
\frac{\partial}{\partial \mathbf{x}} = \frac{\partial}{\partial h}\frac{\partial h}{\partial z}\frac{\partial z}{\partial \mathbf{x}}
\end{equation}
And, deriving the product:
\begin{equation}
\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}} [\mathbf{y}\odot{h}] = \mathbf{z}^\top \mathbf{y} \frac{\partial} {\partial \mathbf{x}} \odot{h} + {h} \odot \frac{\partial} {\partial \mathbf{x}} \mathbf{y}
\end{equation}
We can remove the second portion since:
\begin{equation}
\frac{\partial} {\partial \mathbf{x}} \mathbf{y} = 0
\end{equation}
So finally:
\begin{equation}
\\
\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}} [\mathbf{y}\odot{h}] = \mathbf{z}^\top\frac{\partial}{\partial h}\frac{\partial h}{\partial z}\frac{\partial z}{\partial \mathbf{x}} [\mathbf{y}\odot{h}]
\\
\mathbf{z}^\top\frac{\partial}{\partial h}\frac{\partial h}{\partial z}\frac{\partial z}{\partial \mathbf{x}} [\mathbf{y}\odot{h}] = \mathbf{z}^\top diag(\mathbf{y}\odot\frac{\partial h}{\partial z}\frac{\partial z}{\partial \mathbf{x}}[h])
\\
\mathbf{z}^\top diag(\mathbf{y}\odot\frac{\partial h}{\partial z}\frac{\partial z}{\partial \mathbf{x}}[\text{tanh}(z)] = \mathbf{z}^\top diag(\mathbf{y}\odot(1 - \text{tanh}(z)^2))\frac{\partial z}{\partial \mathbf{x}}[z]
\\
\mathbf{z}^\top diag(\mathbf{y}\odot(1 - \text{tanh}(z)^2))\frac{\partial z}{\partial \mathbf{x}}[z] = \mathbf{z}^\top diag(\mathbf{y}\odot(1 - \text{tanh}(z)^2))W
\end{equation}
Where:
\begin{equation}
\frac{\partial}{\partial h} = diag(\mathbf{y})
\\
\frac{\partial h}{\partial z} = 1 - tanh(z)^2
\\
\frac{\partial z}{\partial \mathbf{x}} = W
\end{equation}
The first derivative is defined as such by using a property of the Hadamard product where:
\begin{equation}
\mathbf{A}\odot\mathbf{X} = diag(\mathbf{A})\mathbf{X}
\end{equation}

#### Backward [3 points]
Implement the calculation for $\mathbf{z}^\top\frac{\partial}{\partial \mathbf{x}}\left[\mathbf{y}\odot\text{tanh}\left(\mathbf{W}\mathbf{x}+\mathbf{b}\right)\right]$  in PyTorch (i.e. without using PyTorch Autograd's `.backward`) using your derivation above.

```
def bw(y, W, x, b, grad_output):
  result = torch.diag(y * (1 - torch.tanh(torch.matmul(W, x) + b)**2))
  result = grad_output.T.matmul(result.matmul(W))
  return result
```

## SortBy PyTorch Autograd Function [10 points]

Implement a PyTorch Autograd function `SortBy` which takes two inputs:
- `x` is a matrix of size `m x n`
- `s` is an accompanying vector of size `m`

`SortBy` should sort the position of the row vectors in `x` using the accompanying scores in `s` in ascending order. For example, given
$$
\begin{align}
\mathbf{X} &= \left[\begin{matrix}
0.2 & -0.4 & 0.3\\
1.2 & 2.3 & -2.1\\
0.1 & -0.1 & 2
\end{matrix}\right]
&\mathbf{s} &=\left[\begin{matrix}
0.2\\
-0.1\\
3
\end{matrix}\right]
\end{align}
$$ the forward pass of `SortBy` should return
$$
\mathbf{Y} = \left[\begin{matrix}
1.2 & 2.3 & -2.1\\
0.2 & -0.4 & 0.3\\
0.1 & -0.1 & 2
\end{matrix}\right]
$$.

Furthermore, given an upstream gradient `grad_output`  (i.e. a matrix of the same size as X), the backward pass of `SortBy` should calculate the gradient of `x`, effectively rerouting the gradient to the original position of the vectors before sorting. For example, if the first row vector of the upstream gradient  in our example above is a vector $\mathbf{z}$, the gradient of `x` would have $\mathbf{z}$ as its second row vector.

Note that, `SortBy` will only be differentiable w.r.t. to x, and is not be differentiable w.r.t. the sorting procedure to provie a gradient for `s`. **You are not allowed to use any Python loops in your implementation. If you use Python loops for your solution, we will only give you half of the points!**

Hints:
- You are allowed to use `torch.sort` in your implementation of the forward pass.
- Similarly to the example we had in the lecture, you can use the context `ctx` to save tensors on the forward pass that you might need to reuse on the backward pass.

```
from torch.autograd import Function

class SortBy(Function):
  @staticmethod
  def forward(ctx, x, s):
    result = x[s.sort().indices]
    ctx.save_for_backward(result, s)
    return result

  @staticmethod
  def backward(ctx, grad_output):
    result, s = ctx.saved_tensors
    return grad_output[s.sort().indices], None
```

## Multiple Choice Quiz [10 points]

Answer the following questions by selecting the correct most specific answer (or `None` in case all answers are wrong).

1. Which of the following operations cannot be calculated using `@`?
2. What is gradient checking for?
3. Why don't we use the finite differences method of gradient checking to calculate gradients instead of using backpropagation?
4. Which of the following operations cannot be expressed as a single einsum string?
5. When should you prefer using `view` instead of `reshape`?
6. Which of the following statements is true if you construct a PyTorch tensor from a NumPy array using `torch.from_numpy`?
7. Which one is a sufficient condition for being able to broadcast an operation between two tensors?
8. What is the difference between a torch.Tensor and a torch.nn.Parameter?
9. Given a convex loss function and a sufficiently small learning rate, stochastic gradient descent is guaranteed to?
10. Given a non-convex loss function and a very large learning rate, stochastic gradient descent is guaranteed to?

```
#@title Answers { run: "auto" }
Q1 = "None of the above" #@param ["Matrix-matrix multiplication", "Matrix-vector multiplication", "Vector-vector multiplication", "Tensor-matrix multiplication", "Tensor-vector multiplication", "None of the above"]
Q2 = "It tests whether the forward pass of a function is consistent with the backward pass" #@param ["It tests whether the forward pass of a function is consistent with the backward pass", "It is used at runtime to check for numerical instabilities in the backward pass", "It tests wether the function and its gradient have been implemented correctly", "It tests whether the norm of the gradients of a function are bounded", "None of the above"]
Q3 = "It would be too slow" #@param ["It cannot be used to approximate the gradient accurately enough", "It can only be used to calculate the gradient of single functions and not for chained functions which are commonly used in deep learning models", "It would be too slow", "None of the above"]
Q4 = "None of the above" #@param ["The transpose of an order-three tensor", "The sum of the diagonal of a square matrix", "The outer product of two matrices", "None of the above"]
Q5 = "When the tensor is contiguous" #@param ["When the tensor is non-contiguous", "When the tensor is contiguous", "None of the above"]
Q6 = "They point to the same memory and altering one will change the other" #@param ["Gradients can be calculated using both, the PyTorch tensor and the NumPy array", "They point to the same memory and altering one will change the other", "The PyTorch tensor cannot be mapped back to a NumPy array", "None of the above"]
Q7 = "One of the two tensors is a scalar" #@param ["One of the two tensors is a scalar", "One of the tensors has a singleton dimension", "The two tensors have the same number of dimensions", "None of the above"]
Q8 = "Parameters get associated with a model when assigned to a member of the model's modules" #@param ["Parameters are mutable and tensors are not", "Parameters get associated with a model when assigned to a member of the model's modules", "Parameters need to be flattened into vectors whereas tensors can be high-dimensional", "None of the above"]
Q9 = "All of the above" #@param ["Find a local optimum", "Find the global optimum", "All of the above", "None of the above"]
Q10 = "None of the above" #@param ["Find a local optimum", "Find the global optimum", "Converge to a saddle point", "All of the above", "None of the above"]
```

# Part II: Feature Engineering [50 points]

In this section you will develop a logistic regression model for sentiment prediction.

## Setup
First we download the [sentence polarity dataset v1.0](http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz) from this [website](http://www.cs.cornell.edu/people/pabo/movie-review-data/) using a few shell commands.

```
%%shell
wget http://www.cs.cornell.edu/People/pabo/movie-review-data/rt-polaritydata.tar.gz
tar -xzf rt-polaritydata.tar.gz
mv rt-polaritydata.README.1.0.txt rt-polaritydata
cd rt-polaritydata
iconv -f cp1252 -t utf-8 < rt-polarity.neg > rt-polarity.neg.utf8
iconv -f cp1252 -t utf-8 < rt-polarity.pos > rt-polarity.pos.utf8
perl -ne 'print "neg\t" . $_' <  rt-polarity.neg.utf8 > rt-polarity.neg.utf8.tsv
perl -ne 'print "pos\t" . $_' <  rt-polarity.pos.utf8 > rt-polarity.pos.utf8.tsv
cat rt-polarity.neg.utf8.tsv rt-polarity.pos.utf8.tsv > rt-polarity.utf8.tsv
```

    --2021-01-31 21:25:20--  http://www.cs.cornell.edu/People/pabo/movie-review-data/rt-polaritydata.tar.gz
    Resolving www.cs.cornell.edu (www.cs.cornell.edu)... 132.236.207.36
    Connecting to www.cs.cornell.edu (www.cs.cornell.edu)|132.236.207.36|:80... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz [following]
    --2021-01-31 21:25:20--  http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz
    Reusing existing connection to www.cs.cornell.edu:80.
    HTTP request sent, awaiting response... 200 OK
    Length: 487770 (476K) [application/x-gzip]
    Saving to: ‘rt-polaritydata.tar.gz.5’

    rt-polaritydata.tar 100%[===================>] 476.34K  2.47MB/s    in 0.2s

    2021-01-31 21:25:20 (2.47 MB/s) - ‘rt-polaritydata.tar.gz.5’ saved [487770/487770]

Now we install [AllenNLP](https://allennlp.org/).

```
%%shell
pip install allennlp==0.9
```

    Requirement already satisfied: allennlp==0.9 in /usr/local/lib/python3.6/dist-packages (0.9.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.4.1)
    Requirement already satisfied: tqdm>=4.19 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (4.41.1)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.22.2.post1)
    Requirement already satisfied: conllu==1.3.1 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.3.1)
    Requirement already satisfied: requests>=2.18 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (2.23.0)
    Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.16.63)
    Requirement already satisfied: gevent>=1.3.6 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (21.1.2)
    Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.6.4)
    Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.2.5)
    Requirement already satisfied: editdistance in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.5.3)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (2018.9)
    Requirement already satisfied: parsimonious>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.8.1)
    Requirement already satisfied: pytorch-pretrained-bert>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.6.2)
    Requirement already satisfied: tensorboardX>=1.2 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (2.1)
    Requirement already satisfied: flaky in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.7.0)
    Requirement already satisfied: torch>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.7.0+cu101)
    Requirement already satisfied: flask>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.1.2)
    Requirement already satisfied: unidecode in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.1.2)
    Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.2.2)
    Requirement already satisfied: numpydoc>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.1.0)
    Requirement already satisfied: word2number>=1.1 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.1)
    Requirement already satisfied: overrides in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.1.0)
    Requirement already satisfied: ftfy in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (5.8)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.19.5)
    Requirement already satisfied: responses>=0.7 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.12.1)
    Requirement already satisfied: flask-cors>=3.0.7 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (3.0.10)
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (2.10.0)
    Requirement already satisfied: sqlparse>=0.2.4 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.4.1)
    Requirement already satisfied: jsonpickle in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.5.1)
    Requirement already satisfied: jsonnet>=0.10.0; sys_platform != "win32" in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (0.17.0)
    Requirement already satisfied: spacy<2.2,>=2.1.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (2.1.9)
    Requirement already satisfied: pytorch-transformers==1.1.0 in /usr/local/lib/python3.6/dist-packages (from allennlp==0.9) (1.1.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->allennlp==0.9) (1.0.0)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==0.9) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==0.9) (2020.12.5)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==0.9) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp==0.9) (1.25.11)
    Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->allennlp==0.9) (0.3.4)
    Requirement already satisfied: botocore<1.20.0,>=1.19.63 in /usr/local/lib/python3.6/dist-packages (from boto3->allennlp==0.9) (1.19.63)
    Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->allennlp==0.9) (0.10.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from gevent>=1.3.6->allennlp==0.9) (51.3.3)
    Requirement already satisfied: zope.interface in /usr/local/lib/python3.6/dist-packages (from gevent>=1.3.6->allennlp==0.9) (5.2.0)
    Requirement already satisfied: greenlet<2.0,>=0.4.17; platform_python_implementation == "CPython" in /usr/local/lib/python3.6/dist-packages (from gevent>=1.3.6->allennlp==0.9) (1.0.0)
    Requirement already satisfied: zope.event in /usr/local/lib/python3.6/dist-packages (from gevent>=1.3.6->allennlp==0.9) (4.5.0)
    Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (0.7.1)
    Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (1.10.0)
    Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (20.3.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (1.15.0)
    Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (1.4.0)
    Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp==0.9) (8.6.0)
    Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert>=0.6.0->allennlp==0.9) (2019.12.20)
    Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorboardX>=1.2->allennlp==0.9) (3.12.4)
    Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch>=1.2.0->allennlp==0.9) (0.8)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.2.0->allennlp==0.9) (0.16.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch>=1.2.0->allennlp==0.9) (3.7.4.3)
    Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp==0.9) (7.1.2)
    Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp==0.9) (1.1.0)
    Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp==0.9) (1.0.1)
    Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp==0.9) (2.11.2)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp==0.9) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp==0.9) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp==0.9) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp==0.9) (2.4.7)
    Requirement already satisfied: sphinx>=1.6.5 in /usr/local/lib/python3.6/dist-packages (from numpydoc>=0.8.0->allennlp==0.9) (1.8.5)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy->allennlp==0.9) (0.2.5)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from jsonpickle->allennlp==0.9) (3.4.0)
    Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (0.8.1)
    Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (2.0.5)
    Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (1.0.5)
    Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (7.0.8)
    Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (0.2.4)
    Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (2.0.1)
    Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (1.0.5)
    Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.1.0->allennlp==0.9) (0.9.6)
    Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from pytorch-transformers==1.1.0->allennlp==0.9) (0.1.95)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.10.1->flask>=1.0.2->allennlp==0.9) (1.1.1)
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (20.8)
    Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (2.6.1)
    Requirement already satisfied: sphinxcontrib-websupport in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (1.2.4)
    Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (2.9.0)
    Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (0.7.12)
    Requirement already satisfied: docutils>=0.11 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (0.16)
    Requirement already satisfied: imagesize in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (1.2.0)
    Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (2.1.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->jsonpickle->allennlp==0.9) (3.4.0)
    Requirement already satisfied: sphinxcontrib-serializinghtml in /usr/local/lib/python3.6/dist-packages (from sphinxcontrib-websupport->sphinx>=1.6.5->numpydoc>=0.8.0->allennlp==0.9) (1.1.4)

Next we implement a AllenNLP data loader for this data.

```
from typing import Iterator, List, Dict, Optional
import torch
import torch.optim as optim
import numpy as np
from allennlp.data import Instance
from allennlp.data.fields import TextField, SequenceLabelField, LabelField
from allennlp.data.dataset_readers import DatasetReader
from allennlp.common.file_utils import cached_path
from allennlp.data.token_indexers import TokenIndexer, SingleIdTokenIndexer
from allennlp.data.tokenizers import Token
from allennlp.data.vocabulary import Vocabulary
from allennlp.models import Model
from allennlp.modules.text_field_embedders import TextFieldEmbedder, BasicTextFieldEmbedder
from allennlp.modules.token_embedders import Embedding
from allennlp.modules.seq2seq_encoders import Seq2SeqEncoder, PytorchSeq2SeqWrapper
from allennlp.nn.util import get_text_field_mask, sequence_cross_entropy_with_logits
from allennlp.training.metrics import CategoricalAccuracy
from allennlp.data.iterators import BucketIterator
from allennlp.training.trainer import Trainer
from allennlp.predictors import SentenceTaggerPredictor

class PolarityDatasetReader(DatasetReader):
    """
    DatasetReader for polarity data like

        neg\tI had better gone to Imperial
    """
    def __init__(self, token_indexers: Dict[str, TokenIndexer] = None,
                 tokenize_and_preprocess = lambda text: text.split(" ")) -> None:
        super().__init__(lazy=False)
        self.token_indexers = token_indexers or {"tokens": SingleIdTokenIndexer()}
        self.tokenize_and_preprocess = tokenize_and_preprocess

    def text_to_instance(self, tokens: List[Token], label: Optional[str] = None) -> Instance:
        sentence_field = TextField(tokens, self.token_indexers)
        fields = {"sentence": sentence_field}

        if label:
            label_field = LabelField(label=label)
            fields["label"] = label_field

        return Instance(fields)

    def _tokenize_and_preprocess(text):
        return text.split(" ")

    def _read(self, file_path: str) -> Iterator[Instance]:
        with open(file_path) as f:
            for line in f:
                label, text = line.split("\t")
                tokens = [Token(word) for word in self.tokenize_and_preprocess(text)]
                yield self.text_to_instance(tokens, label)

```

# Preprocessing [7pts]

In order to fit our model, we will need to tokenize and preprocess the data. Write a dataset loader that preprocesses the data.

Tokenization is an important field of NLP, and can make a large difference to downstream performance. Luckily fo us, the dataset has already been tokenized, so we just need to split the input text by whitespace to get the tokens. The tokenization is not perfect though. Your preprocessing function should should fix all instances where "Mr." have been tokenized as two tokens to instances where "Mr." is a single token.

Implement the above by changing and possibly extending the code below.

```
def collapse_mr_dot(text):
    """
    Args:
      tokens: a list of tokens
    """
    result = []
    text = text.lower()
    text = text.replace("mr .", "mr.")
    text = text.replace("mr  .", "mr.")
    result = text.split(" ")
    return result

torch.manual_seed(1)
reader = PolarityDatasetReader(tokenize_and_preprocess=collapse_mr_dot)
data_pre = reader.read(cached_path("rt-polaritydata/rt-polarity.utf8.tsv"))
rt_polarity_pre = data_pre
for instance in data_pre:
  if "mr." in [t.text for t in instance['sentence']]:
    print(instance['sentence'][:])
    break
```

    10662it [00:00, 10675.45it/s]

    [as, is, often, the, case, with, ambitious, ,, eager, first-time, filmmakers, ,, mr., murray, ,, a, prolific, director, of, music, videos, ,, stuffs, his, debut, with, more, plot, than, it, can, comfortably, hold, .,
    ]

## Logistic Regression

Below we provide a simple implementation of a model, that combined with the corresponding loss, amounts to logistic regression.

```
import torch
import torch.nn as nn

# Model
class LogisticRegression(nn.Module):
    """
    Simple Logistic Regression implementation based on torchtext input format.
    """
    def __init__(self, num_features):
        super(LogisticRegression, self).__init__()
        self.weights = nn.Parameter(torch.normal(torch.zeros(num_features)),
                                requires_grad=True)

    def forward(self, sentence):
        """
        Args:
          sentence: a dictionary of ...
        """
        tokens = sentence['tokens']
        active_tokens_mask = get_text_field_mask(sentence)
        # retrieve weights and set those to zero that come from padding cells
        filtered = active_tokens_mask * self.weights[tokens]
        # sum pooling along the token position dimension
        logits = filtered.sum(dim=1)
        return logits

# model = LogisticRegression(vocab.get_vocab_size("tokens"))
# model.forward(sentence)
```

## Formulation [5pts]

In the class we have presented the model as encoder $f(\mathbf{x})$ follwed by a linear decoder
$$s(\mathbf{x}) = \boldsymbol{\theta}^T  f(\mathbf{x}) = \boldsymbol{\theta}^T \sum_{w\in \mathbf{x}} f(w) $$ where $f(\mathbf{x})$ is the representation of the input text.

The implementation here achieves the same output, but the calculation is performed slightly differently due to technical reasons when working with pytorch. Can you give a mathematical description of this implementation here that captures the order in which computation happens? Below $f(w)$ is a one-hot representation of a word, as per lecture 2.

The candidate answers are:

1. $s(\mathbf{x}) = \left[\sum_{w\in \mathbf{x}}  f(w)  \right]^T \boldsymbol{\theta} $
2. $s(\mathbf{x}) = \sum_{w\in \mathbf{x}}  \boldsymbol{\theta}^T f(w)$
3. $s(\mathbf{x}) = \frac{1}{|\mathbf{x}|}\boldsymbol{\theta}^T \sum_{\mathbf{x}\in x}  f(w)$
4. $s(\mathbf{x}) = \left[\sum_{w\in \mathbf{x}}  f(w)  \right]^T \boldsymbol{\theta} \frac{1}{|\mathbf{x}|} $

```
#@title Answers { run: "auto" }
QFormulation = "Eq 1" #@param ["Eq 1", "Eq 2", "Eq 3", "Eq 4", "None of the above"]
```

## Mean Pooling [8pts]

Create a new version of the logistic regression module, using mean pooling.

```
# Model
class LogisticRegressionMeanPooling(nn.Module):
    """
    Simple Logistic Regression implementation based on torchtext input format.
    """
    def __init__(self, num_features):
        super(LogisticRegressionMeanPooling, self).__init__()
        self.weights = nn.Parameter(torch.normal(torch.zeros(num_features)),
                                requires_grad=True)

    def forward(self, sentence):
        """
        Args:
          sentence: a dictionary of ...
        """
        tokens = sentence['tokens']
        active_tokens_mask = get_text_field_mask(sentence)
        # retrieve weights and set those to zero that come from padding cells
        filtered = active_tokens_mask * self.weights[tokens]
        # sum pooling along the token position dimension
        logits = filtered.sum(dim=1)
        # Get the number of words for each sentence
        n = active_tokens_mask.sum(dim=1)
        # Divide by the number of words
        mean_logits = logits/n

        return mean_logits

```

## Add Features [20pts]

Add the features below to the preprocessing pipeline shown below.

### Bias Feature [6 pts]

It is common practice to add a *bias* term of linear classifiers:

$$s(\mathbf{x}) = \boldsymbol{\theta}^T  f(\mathbf{x}) + b $$

One way to achieve this in general is to augment $f(\mathbf{x}) $ with an extra component that is always set to $1$. In our implementation, this can be achieved by augmenting the sentence field appropriately when loading the data, and setting the `add_features` argument in the dataset loader. Implement this below.

### Bigram Feature [7 pts]

Use the `add_features` pipeline to implement a feature that captures whether word *pairs* $w_1, w_2$ appear consecutively in the sentence.  This feature should be *combined* with the standard unigram and bias features.

### Max Pooling [7 pts]

Use the `add_features` pipeline to implement max pooling such that any feature appearing more than once in the sentence is only counted once, as per the lecture slides of week 1.

```
def add_features(text):
  """
  Args:
    features: a list of tokens
  """
  # TODO implement this function based on instructions above.
  toks = collapse_mr_dot(text)

  # Bias Feature
  toks.append('--ciao--')

  # Biagram
  # I am doing this after the bias as the question is asking for it.
  sentence_len = len(toks)
  for i in range(sentence_len - 1):
    toks.append(toks[i] + ' ' + toks[i + 1])

  # Max Pooling
  unique_toks = list(dict.fromkeys(toks))
  result = list(unique_toks)

  return result

reader = PolarityDatasetReader(tokenize_and_preprocess=add_features)
data_pre_2 = reader.read(cached_path("rt-polaritydata/rt-polarity.utf8.tsv"))
data_pre_2[10]['sentence'][:]
```

    10662it [00:01, 7681.21it/s]

    [a,
     sentimental,
     mess,
     that,
     never,
     rings,
     true,
     .,

    ,
     --ciao--,
     a sentimental,
     sentimental mess,
     mess that,
     that never,
     never rings,
     rings true,
     true .,
     .
    ,

     --ciao--]

## Hyperparameters Search and Analysis [10 pts]

### Early Stopping [5pts]

Finding the right number of iterations is important (why can running to convergence be bad?). One way to to do this is to iterate for a max number K, and then choose the iteration with the largest dev set performance. But this can be slow and unnecessary if we assume that dev-set performance doesn't go up again once it starts to go down (dev set performance concave). Implement a variant of the training loop that implements this idea.  Specifically, the loop should terminate if there has been no increase in development set accuracy when comparing the current accuracy to that from 10 epochs ago.

### Grid Search [5pts]
Using all the features you developed in the above "Add Features" section (or the base model in case you could not address the question), find the best combination of

* Learning Rate in {1.0, 0.1}
* Number of Training epochs (via early stopping, use 1000 as maximum)
* L2 regularisation weight in {0.001, 0.0001, 0}

After grid search, the value of the variables `best_acc`, `best_l2`, `best_lr` and `best_epochs` should be appropriately.

```
def accuracy(dataset, model, iterator):
  # Testing the model and returning the accuracy on the given dataset
  total = 0
  correct = 0
  for batch in iterator(dataset, num_epochs=1):
      sentence = batch['sentence']
      label = batch['label']
      output = model(sentence)
      total += len(label)
      prediction = (output > 0).long()
      correct += (prediction == label).sum()

  return float(correct) / total

def training_loop(model, iterator, train_set, dev_set, num_epochs=100,
                  lr=0.1, weight_decay=0.0, early_stop_decision = 'simple'):
  """
  Should return the best dev_set accuracy and the number of epochs used.
  """
  criterion = torch.nn.BCEWithLogitsLoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=lr,
                              weight_decay=weight_decay)
  # Training the Model
  epoch_accuracies = []
  best_epoch = 0
  best_accuracy = 0.0
  for epoch in range(num_epochs):
      for i, batch in enumerate(iterator(train_set,num_epochs=1)):
          sentence = batch['sentence']
          label = batch['label'].float()

          # Forward + Backward + Optimize
          optimizer.zero_grad()
          logits = model(sentence)
          loss = criterion(logits, label)
          loss.backward()
          optimizer.step()

          if (i+1) % 100 == 0:
              print ('Epoch: [%d/%d], Step: [%d/%d], Loss: %.4f, Dev: %.4f'
                     % (epoch+1, num_epochs, i+1, len(train_set)//iterator._batch_size,
                        loss.data, accuracy(dev_set, model, iterator)))

      epoch_accuracies.append(accuracy(dev_set, model, iterator))
      if epoch_accuracies[-1] > best_accuracy:
          best_accuracy = epoch_accuracies[-1]
          best_epoch = epoch

      # TODO: implement early stopping here
      # The exercise require the early stopping to happen if the comparison with the epoch accuracy has not gone up in th elast 10 iterations.
      # We can compare this with the 10th last result or we can use a moving average to ensure the smoothness of the function.
      # I added an argument "early_stop_decision" in training_loop with a default to simple to check for this
      # I use best set accuracy as specified here: https://moodle.ucl.ac.uk/mod/forum/discuss.php?d=539287
      if early_stop_decision == 'simple':
        if len(epoch_accuracies) > 10:
          if best_accuracy <= epoch_accuracies[-11]:
            print("Early Stop")
            break
      elif early_stop_decision == 'complex':
        if len(epoch_accuracies) > 10:
          if best_accuracy <= sum(epoch_accuracies[-11:-2])/10:
            print("Early Stop")
            break

  return best_accuracy, best_epoch

reader = PolarityDatasetReader(tokenize_and_preprocess=add_features)
data = reader.read(cached_path("rt-polaritydata/rt-polarity.utf8.tsv"))
training_data = data[0:-1000]
dev_data = data[-1000:]
vocab = Vocabulary.from_instances(training_data)
iterator = BucketIterator(batch_size=32, sorting_keys=[("sentence", "num_tokens")])
iterator.index_with(vocab)
print(len(training_data))
print(len(dev_data))
print(len(data))
```

    10662it [00:01, 8418.81it/s]
    100%|██████████| 9662/9662 [00:00<00:00, 25092.59it/s]

    9662
    1000
    10662

```
best_acc = 0.0 # best accuracy achieved
best_lr = 0.0 # best learning rate at best accuracy
best_l2 = 0.0 # best l2 regularizing weight
best_epochs = 0 # best number of epochs
# TODO: implement grid search to make sure the above 4 variable have
for lr in [1.0, 0.1]:
  for l2 in [0.001, 0.0001, 0]:
    for early_stop_decision in ["simple"]:
      model = LogisticRegression(num_features=vocab.get_vocab_size("tokens"))
      acc, epochs = training_loop(model, iterator, training_data, dev_data, lr=lr, weight_decay = l2, early_stop_decision = early_stop_decision)

    if acc > best_acc:
      print(epochs)
      best_acc = acc
      best_lr = lr
      best_l2 = l2
      best_epochs = epochs
      best_early_stop_decision = early_stop_decision
```

```
print("Best Accuracy working with LogisticRegression and simple early stop:")
best_acc, best_epochs, best_lr, best_l2, best_early_stop_decision
```

    Best Accuracy working with LogisticRegression and simple early stop:

    (0.991, 15, 1.0, 0, 'simple')

```
best_acc = 0.0 # best accuracy achieved
best_lr = 0.0 # best learning rate at best accuracy
best_l2 = 0.0 # best l2 regularizing weight
best_epochs = 0 # best number of epochs
# TODO: implement grid search to make sure the above 4 variable have
for lr in [1.0, 0.1]:
  for l2 in [0.001, 0.0001, 0]:
    for early_stop_decision in ["complex"]:
      model = LogisticRegression(num_features=vocab.get_vocab_size("tokens"))
      acc, epochs = training_loop(model, iterator, training_data, dev_data, lr=lr, weight_decay = l2, early_stop_decision = early_stop_decision)

    if acc > best_acc:
      print(epochs)
      best_acc = acc
      best_lr = lr
      best_l2 = l2
      best_epochs = epochs
      best_early_stop_decision = early_stop_decision
```

```
print("Best Accuracy working with LogisticRegression and complex early stop:")
best_acc, best_epochs, best_lr, best_l2, best_early_stop_decision
```

    Best Accuracy working with LogisticRegression and complex early stop:

    (0.951, 44, 0.1, 0.001, 'complex')

```
best_acc = 0.0 # best accuracy achieved
best_lr = 0.0 # best learning rate at best accuracy
best_l2 = 0.0 # best l2 regularizing weight
best_epochs = 0 # best number of epochs
# TODO: implement grid search to make sure the above 4 variable have
for lr in [1.0, 0.1]:
  for l2 in [0.001, 0.0001, 0]:
    for early_stop_decision in ["simple"]:
      model = LogisticRegressionMeanPooling(num_features=vocab.get_vocab_size("tokens"))
      acc, epochs = training_loop(model, iterator, training_data, dev_data, lr=lr, weight_decay = l2, early_stop_decision = early_stop_decision)

    if acc > best_acc:
      print(epochs)
      best_acc = acc
      best_lr = lr
      best_l2 = l2
      best_epochs = epochs
      best_early_stop_decision = early_stop_decision
```

```
print("Best Accuracy working with LogisticRegressionMeanPooling and simple early stop:")
best_acc, best_epochs, best_lr, best_l2, best_early_stop_decision
```

    Best Accuracy working with LogisticRegressionMeanPooling and simple early stop:

    (0.618, 0, 1.0, 0, 'simple')

```
best_acc = 0.0 # best accuracy achieved
best_lr = 0.0 # best learning rate at best accuracy
best_l2 = 0.0 # best l2 regularizing weight
best_epochs = 0 # best number of epochs
# TODO: implement grid search to make sure the above 4 variable have
for lr in [1.0, 0.1]:
  for l2 in [0.001, 0.0001, 0]:
    for early_stop_decision in ["complex"]:
      model = LogisticRegressionMeanPooling(num_features=vocab.get_vocab_size("tokens"))
      acc, epochs = training_loop(model, iterator, training_data, dev_data, lr=lr, weight_decay = l2, early_stop_decision = early_stop_decision)

    if acc > best_acc:
      print(epochs)
      best_acc = acc
      best_lr = lr
      best_l2 = l2
      best_epochs = epochs
      best_early_stop_decision = early_stop_decision
```

```
print("Best Accuracy working with LogisticRegressionMeanPooling and complex early stop:")
best_acc, best_epochs, best_lr, best_l2, best_early_stop_decision
```

    Best Accuracy working with LogisticRegressionMeanPooling and complex early stop:

    (0.985, 0, 0.1, 0.001, 'complex')

# Evaluation

## Validity Check
This is a way for you to check whether you accidentially renamed answer variables or functions that we will use for automatic evaluation. Note that this is not a comprehensive list and we do not check here whether you accidentially changed the function signatures, so failing this validity check is only a sufficient condition for telling you something went wrong.

```
for answer in [Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q10, QFormulation]:
  assert isinstance(answer, str)

for fun in [
    construct_scaled_identity,
    mean_diagonal,
    bottom_right_matrix,
    transpose_sum,
    matrixvector1,
    matrixvector2,
    matrixvector3,
    matrixvector4,
    matrixvector5,
    fw,
    bw,
    SortBy,
    collapse_mr_dot,
    LogisticRegressionMeanPooling,
    add_features,
    accuracy,
    training_loop
    ]:
  assert callable(fun)
```