# Focal Loss for Dense Object Detection

Follow [Andrew Ng's](https://www.youtube.com/watch?v=733m6qBH-jI&list=PLoROMvodv4rOABXSygHTsbvUz4G_YQhOb&index=9&t=0s) suggested strategy of multiple passes over a paper. Namely:

1. Title, Abstract and Figures
2. Intro/Conclusion and Skim the rest (Skipping related work)
3. Read All Remaining Sections (however Skip the maths)
4. Read Everything (but skip the parts that don't make sense)

After some background reading (medium posts, blog posts, other papers, youtube videos from [cs231 2016](https://www.youtube.com/watch?v=GxZrEKZfW2o&t=3s) [cs231 2017](https://www.youtube.com/watch?v=nDPWywWRIRo&t=3909s)), I have made notes below following each pass.

## Title, Abstract and Figures

### Title
Focal Loss for Dense Object Detection
Comments:

What is `Focal Loss`? A new type of loss function called focal loss is introduced in this paper.

What is `Dense Object Detection`? Dense object detection is looking for objects in the same image over many densely covered spatial positions, scales and aspect ratios - it is one stage detection approach.

### Abstract

**Comments**

Object detection is a task of finding bounding boxes for zero of more instances of predefined classes on an image. The bounding box is a rectangle that locates and bounds the instance of a class. A solution which does this is called an object detector.

Here is an example:

There are two classes of object detection algorithm that are state of the art on competing metrics - namely:

1. Dense object detection performs at the state of the art level of performance, as measured by an accuracy metric: average precision (area under precision recall curve - for different classification thresholds).

2. Sparse object detection performs at the state of the art level of performance, as measured by speed of prediction on test cases, for a respectable level but not excellent of performance of accuracy on the average precision metric.

```
Aside: The dense object detection approach is for one stage learning models - those that do proposal, classification, and localization in one shot - such as OverFeat, SSD or YOLO - many (dense) blind possibly overlapping candidate boxes are made and fitted over a grid.

The sparse object detection approach is a two step approach - where first regions that may contain an object are proposed and then these boxes are put through a classifier - some examples being R-CNN, Fast R-CNN, Faster R-CNN and Mask R-CNN.
```

The first approach produces excellent results. However, the second approach provides a

> In contrast, one-stage detectors that are applied
over a regular, dense sampling of possible object locations
have the potential to be faster and simpler, but have trailed
the accuracy of two-stage detectors thus far.

`Question: Simpler in what sense? Still very difficult to interpret or understand a sparse object detection model.`

The authors suggest that the paper identifies the reason for poor performance on sparse object detection. They claim that they are able to pinpoint the blame on *extremely unbalanced data* (no objects >> objects) occurring at training time. That is to say, dense detectors suffer from many more examples of no objects in boxes over an image than examples of boxes with images.

They have a way to address this imbalance. They notice that no object *boxes* are assigned much lower cost $-log(Pr($no object | box$))$ than $-log(Pr($class j | box$))$ - the probability of class (object) $j$ the classes of objects when they occur in boxes.

Effectively, they are noticing an artifact that the probabilities of no object are more peaked than they should be and the probabilities of the object classes are flatter than they should be.

`Idea: Could what the authors do be translated into applying a regularization or prior to make the optimizer prefer models with flatter no-object probability and more peaky object probabilities. Could we explicitly set the Bayesian prior to achieve the same result`

The authors view their algorithm as having a *focal loss* which focuses (by having a higher cost) on harder examples (examples with objects present - those with lower probabilities) and having less focus on easy negatives (examples with no object and higher probability - the no object class). They believe this alleviates the problem of the detector being overwhelmed by the imbalanced data, so that learning can still take place.

To see how well their loss does for dense objects detectors, the authors present a network they dub RetinaNet. Their results show that with the *focal loss*  RetinaNet is as fast as the popular one stage detectors, and yet have accuracy (as measures by average precision) as good as the state of the art two stage detectors.

### Figures

#### Figure 1

A graph showing how the proposed novel loss function - focal loss - has the desired properties of increasing cost for low probability values and reducing cost for high probability values. A hyperparameter $\gamma$ is shown over a range of values.

Note that the proposed loss function is pinned at the extremes of 0 loss for predicting with probability 1 the ground truth and shoots off to infinity asymptote as the probability of predicting the ground truth goes to 0.

#### Figure 2

This is a plot of Speed vs accuracy (Average Precision) for RetinaNet with base feature extractor network as ResNet100 or ResNet50 compared to other popular competitive algorithms. It shows that RetinaNet is forms an upper enveloper over the other methods and so is superior in speed and accuracy vs the popular algorithms - with one exception. YOLOv2 is faster, but at a much lower accuracy than RetinaNet.

#### Figure 3
Outline of RetinaNet. The backbone of the network is a Feature Pyramid Network (FPN) - which allows the network to work at various scales. FPN receives inputs from a ConvNet Feature Extractor, and it is connected to two heads: a classifier for the anchor boxes, a regressor to fine tune the anchor boxes to get estimates for the bounding box - the ground truth boxes and IoU metric is used for this. The authors claim that this design is simple enough to allow the focal loss to show it's power.

```
Question: If the network as well as the loss function are being changed - then doesn't this make it hard to ensure the changes are orthogonal and not interfering and impacting the results?
```

#### Figure 4

This graph shows the q-q plot or CDF of normalized loss against the CDF of sample, for each of foreground(object) and background (no object) examples from the ResNet-101 predictions, at different values of gamma for Focal Loss (including Cross Entropy Loss).

This graph is used to identify that the loss on

```
normalized loss: I guess this is the normalized cost function
CDF to CDF - p-p plot.
```

Actually, reading the section for this figure - I don't get this section well - there is the dataset, the model, some normalization of loss, splits into unbalanced foreground and background labels, quite a lot that needs to be teased out. I will come back to this on the third or fourth pass.

#### Figure 5

This graph and the following two are used to show that a similar function to the Focal Loss (FL), $FL^*$ (which depends on a $\beta$ as well as $\gamma$ parameter) is close to FL.

```
There is no proper rigorous analysis, but empirical results and plots - is this good enough?
```

#### Figure 6

Figure 6 show the derivatives are similar in nature and magnitude for FL and $FL^*$.

#### Figure 7

This shows the effective variations of the parameters $\beta$ and $\gamma$ such that the loss function has desired behavior. It also shows less desirable loss functions in light black lines vs the better loss functions in heavier blue lines.

## Introduction and Conclusion

### 1. Introduction

The benchmark data set for researchers working on the object detection task is MS COCO (or COCO).

The reference metric is AP (Average Precision).

```
Need to find references and blog posts on AP
```

State Of The Art (SOTA) object detectors on the AP (average precision) metric are all from a family of sparse object detectors that rely on two stage (proposal then classification+localization) technologies.

Although their performance is to be applauded (still not good enough for many production applications) these two stage detectors are quite slow - as measured in Frames Per Second, FPS.

A promising direction has been the development of one stage detectors, they run one order of magnitude faster (7 FPS for `Faster R-CNN` two stage detector vs 21 FPS for `YOLO Fast` using VGG-16 as a CNN feature extractor). However the performance of one stage detectors has lagged significantly (73.2 - `Faster R-CNN` vs 66.4 `YOLO Fast` AP).

This paper introduces a few technologies that demonstrate a one stage detector with comparable speed to SOTA one stage detectors for speed and comparable AP as SOTA two stage detectors - what they call an upper envelope of performance. The authors assert that the main driver of this improved performance is their new loss function - which combats extremely imbalanced foreground/background data in one stage detectors. (The authors believe  extremely imbalanced foreground/background data  this to be the impediment to better one stage detector performance).

Although two stage detectors also see class imbalance, it is less severe and can be remedied by:

- sampling heuristics (fixing the imbalance ratio by sampling)
- online hard example mining (OHEM is a technique applied inside a minibatch to only select *high loss function* examples from the minibatch forward pass and use them in the backward pass)

Imbalance is less of an issue for two stage detectors because the first stage proposals are chosen as very likely to have foreground objects - most two stage detectors only present 1-2k proposal regions.

One shot detectors need to deal with orders of magnitude more locations that can be overlapping densely covered with many aspect ratios. As a consequence imbalances of 1000 to 1 are normal in background/foreground object classes. At training time this imbalance dominates the loss function, enough that effective learning for the foreground object classes doesn't happen.

Strategies like sampling and OHEM can help but are not as efficient because the training algorithm still concentrates weight changing efforts on the easily classified background examples - there are so many more background examples due to the imbalance.

Typically treatment for this imbalance is to use:

- bootstrapping the dataset to create a distribution of parameters and averaging the resultant probabilities.

- hard example mining (produce many data augmented examples possibly via crops/reflections of mis-classifications from prior training -foregrounds predicted as backgrounds - so that algorithm becomes better at learning these mappings.)

The authors believe their novel loss function alleviates the imbalance problem - by down-weighting easy examples and up-weighting hard examples. The exact form of the loss function is not important - the authors show that similar functions work.

To show how good the focal loss function is, the authors produced a one stage learning algorithm with many state of the art innovations (FPN backbone, ResNet) - called RetinaNet. It's performance of 5 FPS and very good AP of 39.1 - by comparison YOLOv2 achieves 21.2 AP, with undocumented speed.

### 6. Conclusion

The authors reiterate that the problem with one stage detector AP performance has been extreme background/foreground class imbalance.

They present a modulating term applied to the cross entropy loss to create a new focal loss function. They believe this is a high quality solution as it results a better detector. The authors are pleased with a new one stage detector network trained using this focal loss, one that is better on speed and accuracy metrics.

## All Remaining Section (except the Mathematics)

### 2. Related Work

**Classic Object Detectors**

There have some old school object detectors, starting around 2001, including:

**Viola-Jones** was the first set of algorithms to produce useful results for object detection - applied to face recognition. Essentially taking rectangles and matching them to eyes, cheeks, nose and mouth, features are extracted. Then a boosting algorithm of the adaBoost variety is applied to these features to detect faces. **LeCun's** seminal paper proposed sliding windows to detect characters. **Histogram of Oriented Gradients** of integral channels produced good results for pedestrian detection - essentially finding edges in images, again powered by sliding windows. With the onset of Deep Learning, these fell behind in SOTA performance.

**Two Stage Detectors** have been suggested by some of the authors of this paper. They have pioneered innovations and improvements in this space. The main idea is to propose a smaller number of regions (relative to sliding windows) where the propasal step is done so each region has a high probability of having a foreground object class. Then a feature extractor CNN is used to get features and then a classifier and located at most one object class in each region proposal.

The first iteration of the methodology R-CNN using a fixed method - they used selective search algorithm was used for the region proposal, CNN feature extractor run on this and then SVMs used classify objects, and regression head to get the bounding boxes. However, this was slow.

Later this improved by using the faster R-CNN algorithm. This would move things around a little and change the SVM. An image is processed by a CNN feature extractor, then the region proposal takes place, then this is fed into a CNN which outputs two heads - a classifier using a fully connected layer and a bounding box regression, both of which take the CNN as input.

This was still too slow for many applications. So, Faster R-CNN algorithm was proposed. With this algorithm, the region proposal step is also trained using a CNN, using a IoU criteria to train region finding - taking input from a CNN feature extractor network that is pre-trained. Then the rest follows as before for the fast R-CNN algorithm.

There have been several extensions since then to the same basic idea - first classify/set regions likely to have foreground objects, then classify and regress bounding boxes on them.

**One Stage Detectors**

One stage detectors are all from the dense object detector family. They propose several orders of magnitude more regions - using anchors has been a key insight. Then they finesse the bounding boxes around these anchors and classify the object contained.  Among those that have been suggested are:

- OverFeat
- SSD
- YOLO

They all are highly performant on speed, but lag accuracy materially compared to two stage detectors.

RetinaNet is a one stage detector taken from this family of dense detectors using FPN for the scaling to boost performance. It achieves higher accuracy and maintains speed by having the novel loss function - focal loss.

**Class Imbalance**

The one stage detectors suffer from extreme imbalance, $~10^5$ regions. This poses two problems:

- overwhelming the cost function at training time with easy negative (background) classifications
- inefficient estimation as most regions produce no signal

Hard negative mining has been the traditional solution. Other more complex re-sampling strategies can be used.

The central theme of this paper, focal loss is shown to be good at dealing with such imbalance.

**Robust Estimation**

Robust Loss functions have been used to down-weight or reduce the influence of outliers - outliers being determined by their frequency in the data. This paper instead focuses on down-weighting or reducing the impact of easy examples - even if they are the majority of the imbalanced data.

### 3. Focal Loss

Traditional cross entropy loss is given in equation 1. It is the average number of bits needed to learn the outcome of a random draw from the true distribution. One can also think of it as a measure how different the true distribution q is from the distribution being used p.

One hot data vectors are used as instances to compute the cross entropy for data against a model distribution. Then this averaged cross entropy function is the cost function which is usually run through an optimization algorithm (often from the gradient descent family of optimizers) to find the best parameters that minimize the cost.

By minimizing the cross entropy function, we can get a best fit. This also reduces the NLL of MLEs for classifiers.

#### 3.1 Balanced Cross Entropy

Although the above is a good strategy, the reality is that sometimes we come across imbalanced data in classification tasks. For example our current case of dense object detection. What this imbalance does is make it very difficult for gradient descent algorithms to find a good local optima. The reason is that the class(es) with the highest frequency contribute so much more to the loss function, that they dominate the loss function and the other examples are not concentrated on enough by the loss function optimizer.

Usually re-sampling strategies are run to artificially add more examples of the lower/least frequency classes. This can be done through:

- over-sampling the lower frequency classes
- under-sampling the higher frequency classes
- setting a weight hyper-parameter to increase the contribution of the lower frequency classes and/or reduce the contribution of the higher frequency classes.

Equation (3) lists the expression for the last of these strategies - in a compacted form.

#### 3.2 Focal Loss Definition

The tunable focal loss equation is introduced in equations 4 (one parameter) and 5 (two parameter). Somewhat strange, but at training time an approximation to this is used.

The authors outline how the focal loss has the desired property of reducing the loss of misclassifying high probability classes (such as background) and increasing the loss of misclassifying more complicated examples.

#### 3.3 Class Imbalance and Model Initialization

With extreme class imbalance of around $10^3-10^6$ to 1, the usual parameter initialization strategy for binary classification leads to unstable training early on for focal loss. The authors suggest instead giving initial conditions that corresponds to 0.01 probability for the foreground (lower frequency) class. They even call this a prior - which it doesn't feel like in a Bayesian sense - but given that neural networks can have many local optima - perhaps it is like a Bayesian prior, in that it impacts the parameter choices and final probability estimates.

#### 3.4 Class Imbalance and Two Stage Detectors

Class imbalance is less of an issue for two stage detectors because of the proposal layer, that filters for likely regions. There are still 1000-2000 proposal locations usually. Of these, again many will not have a foreground object. However, the regions are proposed to be much more likely to have a foreground object. This helps reduce the class imbalance. Secondly at training time, the minibatches are constructed to have reasonable ratios of classes - say 1:3, effectively re-sampling within the minibatches.

The authors note this is like re-weighting the terms of the loss function with a $\alpha$ weight hyperparameter.

By contrast the focal loss function works at a different level of abstraction.

```
It would have been interesting to see how focal loss does with two stage detectors.
```

### 4. RetinaNet Detector

**Feature Pyramid Network**
Given a single image input resolution, it is normal that some objects are at very different scales to others. In that case, performance can be improved by having classification go through a range of scales. A Feature Pyramid Network is a collection of layers that achieves this. It works as follows:

- lateral connections are made from the convolutional layers to each scale layer of the pyramid.

- top down connections feed into the scale layers through the convolutions.

- the scale layers connect to their own classification and box sub networks.

These then feed into the final set of classification and bounding box predictions.

Scale layer $P_l$ reduces input resolution by by a factor $2^l$.

**Anchors**

Default boxes, or anchors are spaced out across the image data. They then assign to a K hot vector of classifications and probabilities and the bounding box regression. These anchors are presented at the various scales corresponding to the FPNs.

**Classification Subnets**

This head predicts the classification probability for each of the classes for each anchor box.

**Box Regression Subnet**

This finesses the anchors box to get the bounding box for each object in the anchor box.

#### 4.1 Inference and Training

**Inference**

This is a FCN with two heads per bounding box, a classification head and a bounding box regression head. The architecture includes the backbone FPN and a CNN extractor.

To improve speed of training, after some threshold only the top 1000 detected objects are used to continue training on. Non-max suppression is used when at a threshold of 0.5 to keep only the most promising predictions.

**Focal Loss**

The authors found in their hyperparameter tuning that focal loss with $\gamma = 2$, $\alpha = 0.25$, worked best. They suggest reducing $\alpha$ as $\gamma$ is increased.

**Initialization**

The hidden layers were initialized with the usual bias being 0 and small variance ($\sigma=0.01$) random gaussian noise.

For the final layer of the classification head, the bias parameter is set as the inverse logit of 0.01.

**Optimization**

minibatch (16 images) stochastic gradient descent using GPUs is used to train RetinaNet for 90000 iterations and  decaying learning parameter. Focal loss is used for the classification part and L1 loss for the bounding box regression. They trained for around 35 hours.

### 5. Experiments

Training is done on 40k images with some randomized selection from the dataset. The test-dev set is found for data where no public labels are given and the test-dev features are tested and predicted on using an evaluation server.

#### 5.1 Training Dense Detection

**Network Initialization**
If the usual initialization techniques are used, the network diverges. However, the earlier outlined strategy of using the prior probs of 0.01 and inverse logit to get bias parameters of the final layer of the network, make it converge to good AP values. This is all for the cross entropy loss.

**Balanced Cross Entropy**

Weighted or balanced cross entropy loss with $\alpha=0.9$ gives the best results.

**Focal Loss**
The hyperparameter search worked as explained above. The table shows a range of values used.

**Analysis of the Focal Loss**

This section is important as it shows that the FL reduces the loss for all but the most extreme negative background examples contribute to the FL over the dataset. That is, the majority of background easy classifications do not contribute very much to the loss function.

I feel some other kind of chart could be used to indicate this - maybe plotting the cumulative loss function cumulatively against the probability estimate of the foreground and background classes.

**Online Hard Example Mining**

OHEM discards the easy examples and has the loss function optimized on the hard examples that were misclassified - this happens inside the gradient descent algorithm where examples are included or discarded. Note that focal loss still keeps the easy examples, but reduces their effect on the loss function.

The authors note that the AP was higher (better) for focal loss by 3.6 points than OHEM.

**Hinge Loss**

Lin et al. tried to train with hinge loss - setting loss to zero after a certain threshold, but it didn't work producing unstable results that didn't converge.

#### 5.2 Model Architecture Design

**Anchor Density**
How high a density to have for for the anchors and possibly having the same anchor locations for different tall and wide boxes is not an exact science, but impacts performance a great deal for one stage detectors. The authors find that after a certain density, increasing the number and spread of anchors doesn't help performance.

**Speed Vs Accuracy**

Faster networks need smaller backbone networks. The speed accuracy tradeoff is show in the figures. it shows an upper envelope of better accuracy and faster speed compared to most models that are at the SOTA.

### 5.3 Comparison to State of the Art

The authors reiterate their faster and more accurate model over a range of backbone networks. They show exact numbers.

## Everything Else

### Mathematics

#### Equation 1

Under this binary regime either $y=1$ or $y=-1$ are the two classes.

$$
\begin{equation}
  CE(p,y) = \left \{
  \begin{aligned}
    &-\log(p), && \text{if}\ y=1 \\
    &-\log(1-p), && \text{if}\ y=-1
  \end{aligned} \right.
\end{equation}
$$

This is the equation for cross entropy loss for a binary classifier - it is the average number of bits needed under assumed probability distribution $p$ to identify an event drawn using the true distribution of $y$.

This is the natural loss function for binary classification and nicely links to MLE estimation/NLL cost function.

#### Equation 2a

$$
\begin{equation}
  p_t  \left \{
  \begin{aligned}
    & p, && \text{if}\ y=1 \\
    & 1-p, && \text{if}\ y=-1
  \end{aligned} \right.
\end{equation}
$$

This equation is a little odd because, there is no index or variable $t$ being referenced by the subscript. It might have been better to replace that with something $\tilde{p} = \tilde{p}(p,y)$, to keep a relation with p and indicate that it is transform of $p$ and $y$.

One way that it makes sense to keep the subscript t notation is if we were looking at multiclass classification. In that case,

$$
\begin{equation}
  p_t  \left \{
  \begin{aligned}
    & p(y_t), && \text{if}\ y=y_t \\
    & 1-p(y_t), && \text{if}\ y \neq y_t
  \end{aligned} \right.
\end{equation}
$$

However, even then this is messy notation.

#### Equation 2a

From the equation above we can rewrite $CE(p, y) = CE(p_t) = − log(p_t)$, keeping the same odd subscript $t$ notation.

#### Equation 3

$CE(p_t) = -\alpha_t log(p_t)$

This is an equation made to put weight ($\alpha_t \in [0,1]$) to increase or decrease the cost of different outcomes. It's making adjustments (unnatural from some perspectives) to the cross entropy loss function.

This is a well known practice for dealing with imbalanced data.

In fact, note that this can be expressed as:

$CE(p_t) = - log(p_t^{\alpha_t})$

Now if we have $y_0 = 1$ and $y_1 = -1$ (so $t=0$ or $t=1$) for the binary case, then in order to be consistent probability distributions in this representation we need:

$(1-p_0)^{\alpha_1} + p_0^{\alpha_0} = 1$

This follows because $p_1 = 1 - p_0$.

which we can write as saying that $\alpha_0$ is free to be any value in [0,1], but that the following must hold - for probability consistency:

${\alpha_1}  = \frac{1- p_0^{\alpha_0}}{log(1-p_0)}$

With a similar constraint existing if we extend to multiclass classification.

The effect is to increase the probability $p_0$ and decrease the probability $p_1$ with a functional form that does not depend on the weights or features of the network. If we assigned foreground images to the label $y_0=1$, then it makes sense to upweight the class with fewer examples.

If this modification was acting on the weights, then it would be called a regularizer, or a flat Bayesian prior. But since it is at the level of the loss function we don't call it that but the effect is similar.

Of course one downside is that we need to use hyperparameter tuning to find the best value of the loss function weight parameter.

#### Equation 4

The central topic of this paper is the tunable focal loss, given by the equation:

$FL(p_t) = −(1 − p_t)^{\gamma} log(p_t)$

for $\gamma \geq 0$.

The authors find that in practice the exact choice of gamma does not affect much the accuracy. They suggest $\gamma \in [2,5]$ for good tuning results based on their experiments.

Let's do our analysis of this function via Taylor series. Using $log(x) \approx (x-1) - (1/2)(x-1)^2$ for small enough x, well then:

$FL(p_t) \approx −(1 − p_t)^{\gamma} ((p_t-1) - ^2)$

$ = −(1 − p_t)^{\gamma} ((p_t-1) - (1/2)(p_t^2- 2*p_t + 1))$

$ = −(1 − p_t)^{\gamma} (1/2)(p_t-1)(3 - p_t)$

$ = (1/2)(1 − p_t)^{\gamma + 1 }(3 - p_t)$

There is no real insight to be had from this expansion, but it might have been useful.

This $FL$ function is still pinned at 0 and infinity for perfect and worst fit to the one hot vector of actual observation y to distribution estimated via $p_t$. What it does however, is to lower the loss for good estimations ($p_t$ close to y) and increase the loss for bad estimation ($p_t$ far from $y$).

In effect, the focal loss is applying a modulating factor to the cross entropy loss. This might be seen as a kind of flat Bayesian prior, or regularizer to encourage the kind of fit that does well for imbalanced data. Only this flattening is not at the level of the weights, but higher up in the model, acting at the loss function level.

```
There is something not satisfying that the true probabilities cannot be estimated well due to imbalanced data, so much so that we need to resort to modifying the cross entropy loss function using heuristics.

The hope would be to have methods which deal with imbalance at a lower level, at the parameter or architecture level. The reason I believe this to be important is that the cross entropy function is well understood with connections to maximum likelihood estimation. To introduce extra parameters - some of which might be better pushed down the network parameter estimation level feels wrong.

However, since focal loss produces state of the art results we should applaud it's contribution to the field. It also is specifically used for data imbalance - where other strategies have failed to get solutions efficiently. Also, it is not post processing the estimated probabilities - but directly optimizing them, albeit in a novel way.

Perhaps it is time to consider others families of loss functions.
```
#### Equation 5

In practice the authors suggest that a weight factor applied to the modulated loss function produces good results. So one could express focal loss as:

$FL(p_t) = -\alpha_t (1 − p_t)^{\gamma} log(p_t)$

Now with two tunable parameters.

```
Let's see what happens if we try to express the probability model implied by this loss - for the binary case:
```
Say, we set:

$$ \phi_t = -\alpha_t (1 − p_t)^{\gamma} log(p_t)$$

Then:

$$ exp(- \frac{\phi_t}{\alpha_t} (1 − p_t)^{-\gamma}) = p_t$$

replacing the ratio $\frac{\phi_t}{\alpha_t} = \Gamma_t$

$$ exp( - \Gamma_t (1 − p_t)^{-\gamma}) = p_t $$

```
So as we are optimizing the loss function, we are restricting the solution to be on a level set with the form above. I wonder if constrained optimization could be used, maybe not relevant
```

#### Equation 6

This and the following two equations are in the appendix and are supposed to show that there is a easier to optimize function that is very similar to the focal loss described earlier.

$x_t = yx$

It really isn't clear what x is supposed to represent. It may just be that for any scalar $x$, we denote by $x_t$ the product of $y$ and $x$. We are told that $y \in {-1,1}$ is the ground truth label. I initially thought $x$ was a reference to the features $x$, but that doesn't seem to be the case. We are also told that

$p_t = \sigma(x_t)$

is compatible with equation 2. There isn't an explicit definition of $\sigma(\cdot)$. I had thought that it might be the sigmoid function. However matching up the earlier definition from equation 2

$$
\begin{equation}
  p_t  \left \{
  \begin{aligned}
    & p, && \text{if}\ y=1 \\
    & 1-p, && \text{if}\ y=-1
  \end{aligned} \right.
\end{equation}
$$

it's hard to tell. So, let's investigate:

$p_t = \sigma(x_t) = \sigma(yx)$ is to be $p$ if $y=1$ and $1-p$ if $y=-1$.

Again, it's difficult to be sure. So what we'll do it look further ahead at equation 9. From this, we can see that the differential equation can be evaluated only if $\sigma(\cdot)$ is the sigmoid function. It would have been nice if the authors had made this explicit.

#### Equation 7

$p_t^{∗} = \sigma(\gamma x_t + \beta)$

#### Equation 8

$FL∗ = − \frac{log(p_t^{∗})}{\gamma}$

The authors assert that $FL^{*}$ for the settings of $\gamma=2$ and $\beta=1$ is very close to FL and computationally easier. The show graphs where the function is similar within a good range.

```
Question: As neural networks are excellent function approximators, I'm inclined to wonder why they couldn't just approximate these specific transformations. Perhaps the extra help is needed because of the overwhelming imbalance.
```

#### Equation 9

$\frac{dCE}{dx} = y(p_t − 1)$

This equation only works if $\sigma(\cdot)$ is the sigmoid function. We'll do the arithmetic here:

$CE = -log(p_t)  = -log(\sigma(x_t)) = -log(\sigma(yx))$

$\frac{dCE}{dx}  = -[\sigma(yx)]^{-1} * \frac{d[\sigma(yx)]}{dx}$

$ = -[\sigma(yx)]^{-1} * \sigma(yx) * [1- \sigma(yx)] y$

$= y(p_t − 1)$

The reason for giving this and the following two equations seems to be to allow parameter updates via gradient descent when training. In addition, they serve to demonstrate the closeness of $FL$ to $FL^*$, particularly with reference to the figures with the chosen values of the tuning parameters.

#### Equation 10

$\frac{dF}{dx} = y(1 − p_t)^\gamma (\gamma p_t log(p_t) + p_t − 1)$

#### Equation 11

$\frac{dFL^{∗}}{dx} = y(p_t^{*} − 1)$

### Tables
#### Table 1
#### Table 2
#### Table 3

### Appendix
#### Appendix A
#### Appendix B

```python

```


# Getting started with the practicals

***These notebooks are best viewed in Jupyter. GitHub might not display all content of the notebook properly.***

## Goal of the practical exercises

The exercises have two goals:

1. Give you the opportunity to obtain 'hands-on' experience in implementing, training and evaluation machine learning models in Python. This experience will also help you better understand the theory covered during the lectures.

2. Occasionally demonstrate some 'exam-style' questions that you can use as a reference when studying for the exam. Note however that the example questions are (as the name suggests) only examples and do not constitute a complete and sufficient list of 'things that you have to learn for the exam'. You can recognize example questions as (parts of) exercises by this font color.

For each set of exercises (one Python notebook such as this one $==$ one set of exercises) you have to submit deliverables that will then be graded and constitute 25% of the final grade. Thus, the work that you do during the practicals has double contribution towards the final grade: as 30% direct contribution and as a preparation for the exam that will define the other 65% of the grade.

## Deliverables

For each set of exercises, you have to submit:
1. Python functions and/or classes (`.py` files) that implement basic functionalities (e.g. a $k$-NN classifier) and
2. A *single* Python notebook that contains the experiments, visualization and answer to the questions and math problems. *Do not submit your answers as Word or PDF documents (they will not be graded)*. The submitted code and notebook should run without errors and be able to fully reproduce the reported results.

We recommend that you clone the provided notebooks (such as this one) and write your code in them. The following rubric will be used when grading the practical work:

Component  | Insufficient | Satisfactory | Excellent
--- | --- | --- | ---
**Code** | Missing or incomplete code structure, runs with errors, lacks documentation | Self-contained, does not result in errors, contains some documentation, can be easily used to reproduce the reported results | User-friendly, well-structured (good separation of general functionality and experiments, i.e. between `.py` files and the Pyhthon notebook), detailed documentation, optimized for speed, use of a version control system (such as GitHub)
**Answers to questions** | Incorrect, does not convey understanding of the material, appears to be copied from another source | Correct, conveys good understanding of the material, description in own words | Correct, conveys excellent level of understanding, makes connections between topics

## A word on notation

When we refer to Python variables, we will use a monospace font. For example, `X` is a Python variable that contains the data matrix. When we refer to mathematical variables, we will use the de-facto standard notation: $a$ or $\lambda$ is a scalar variable, $\boldsymbol{\mathrm{w}}$ is a vector and $\boldsymbol{\mathrm{X}}$ is a matrix (e.g. a data matrix from the example above). You should use the same notation when writing your answers and solutions.

# Two simple machine learning models

## Preliminaries

Throughout the practical curriculum of this course, we will use the Python programming language and its ecosystem of libraries for scientific computing (such as `numpy`, `scipy`, `matplotlib`, `scikit-learn` etc). The practicals for the deep learning part of the course will use the `keras` deep learning framework. If you are not sufficiently familiar with this programming language and/or the listed libraries and packages, you are strongly advised to go over the corresponding tutorials from the ['Essential skills'](https://github.com/tueimage/essential-skills) module (the `scikit-learn` library is not covered by the tutorial, however, an extensive documentation is available [here](https://scikit-learn.org/stable/documentation.html).

In this first set of exercises, we will use two toy datasets that ship together with `scikit-learn`.

The first dataset is named `diabetes` and contains 442 patients described with 10 features: age, sex, body mass index, average blood pressure, and six blood serum measurements. The target variable is a continuous quantitative measure of the disease (diabetes) progression one year after the baseline measurements were recorded. More information is available [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/descr/diabetes.rst) and [here](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html).

The second dataset is named `breast_cancer` and is a copy of the UCI ML Breast Cancer Wisconsin (Diagnostic) datasets (more infortmation is available [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/descr/breast_cancer.rst) and [here](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)). The datasets contains of 569 instances represented with 30 features that are computed from a images of a fine needle aspirate of a breast mass. The features describe characteristics of the cell nuclei present in the image. Each instance is associated with a binary target variable ('malignant' or 'benign').

You can load the two datasets in the following way:

```python
import numpy as np
from sklearn.datasets import load_diabetes, load_breast_cancer

diabetes = load_diabetes()

breast_cancer = load_breast_cancer()

```

In the majority of the exercises in this course, we will use higher-level libraries and packages such as `scikit-learn` and `keras` to implement, train and evaluate machine learning models. However, the goal of this first set of exercises is to illustrate basic mathematical tools and machine learning concepts. Because of this, we will impose a restriction of only using basic `numpy` functionality. Furthermore, you should as much as possible restrict the use of for-loops (e.g. use a vector-to-matrix product instead of a for loop when appropriate).

If `X` is a 2D data matrix, we will use the convention that the rows of the matrix contain the samples (or instances) and the columns contain the features (inputs to the model). That means that a data matrix with a shape `(122, 13)` represents a dataset with 122 samples, each represented with 13 features. Similarly, if `Y` is a 2D matrix containing the targets, the rows correspond to the samples and the columns to the different targets (outputs of the model). Thus, if the shape of `Y` is `(122, 3)` that means that there are 122 samples and each sample is has 3 targets (note that in the majority of the examples we will only have a single target and thus the number of columns of `Y` will be 1).

You can obtain the data and target matrices from the two datasets in the following way:

```python
X = diabetes.data
Y = diabetes.target[:, np.newaxis]

print(X.shape)
print(Y.shape)
```

    (442, 10)
    (442, 1)

If you want to only use a subset of the available features, you can obtain a reduced data matrix in the following way:

```python
# use only the fourth feature
X = diabetes.data[:, np.newaxis, 3]
print(X.shape)

# use the third, and tenth features
X = diabetes.data[:, (3,9)]
print(X.shape)
```

    (442, 1)
    (442, 2)

***Question***: Why we need to use the `np.newaxis` expression in the examples above?

Note that in all your experiments in the exercises, you should use and independent training and testing sets. You can split the dataset into a training and testing subsets in the following way:

```python
# use the fourth feature
# use the first 300 training samples for training, and the rest for testing
X_train = diabetes.data[:300, np.newaxis, 3]
y_train = diabetes.target[:300, np.newaxis]
X_test = diabetes.data[300:, np.newaxis, 3]
y_test = diabetes.target[300:, np.newaxis]
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
```

    (300, 1)
    (300, 1)
    (142, 1)
    (142, 1)

## Exercises

### Linear regression

Implement training and evaluation of a linear regression model on the diabetes dataset using only matrix multiplication, inversion and transpose operations. Report the mean squared error of the model.

To get you started we have implemented the first part of this exercise (fitting of the model) as an example.

```python
# add subfolder that contains all the function implementations
# to the system path so we can import them
import sys
sys.path.append('code/')

# the actual implementation is in linear_regression.py,
# here we will just use it to fit a model
from linear_regression import *

from sklearn.datasets import load_diabetes, load_breast_cancer
diabetes = load_diabetes()
breast_cancer = load_breast_cancer()

# load the dataset
# same as before, but now we use all features
X_train = diabetes.data[:300, :]
y_train = diabetes.target[:300, np.newaxis]
X_test = diabetes.data[300:, :]
y_test = diabetes.target[300:, np.newaxis]

beta = lsq(X_train, y_train)

X_test2 = np.c_[np.ones(len(X_test)) ,X_test] #add column of ones
Y_pred = np.dot(X_test2,beta)
Err = np.subtract(y_test,Y_pred)
MSE = np.dot(Err.T, Err)/len(X_test)

print("MSE of test data =", MSE)
```

    MSE of test data = [[2794.5690145]]

### Weighted linear regression

Assume that in the dataset that you use to train a linear regression model, there are identical versions of some samples. This problem can be reformulated to a weighted linear regression problem where the matrices $\boldsymbol{\mathrm{X}}$ and $\boldsymbol{\mathrm{Y}}$ (or the vector $\boldsymbol{\mathrm{y}}$ if there is only a single target/output variable) contain only the unique data samples, and a vector $\boldsymbol{\mathrm{d}}$ is introduced that gives more weight to samples that appear multiple times in the original dataset (for example, the sample that appears 3 times has a corresponding weight of 3).

Derive the expression for the least-squares solution of a weighted linear regression model (note that in addition to the matrices $\boldsymbol{\mathrm{X}}$ and $\boldsymbol{\mathrm{Y}}$, the solution should include a vector of weights $\boldsymbol{\mathrm{d}}$).

\begin{align}
WRRS(\mathbf{w}) & = \sum_{i=1}^N \mathbf{d}(y_i - \mathbf{x}_i\mathbf{w})^2 \\
\end{align}

The matrix notation:
\begin{align}
WRRS(\mathbf{w}) & = (\mathbf{y} - \mathbf{Xw})^T\mathbf{d}(\mathbf{y}-\mathbf{Xw}) \\
\end{align}

Differentiate with respect to w
\begin{align}
-2\mathbf{dX}^T(\mathbf{y}-\mathbf{Xw}) \\
\end{align}

To ﬁnd the minimum our derivative must be 0, hence:
$$
\begin{align}
    \ -2\mathbf{dX}^T(\mathbf{y}-\mathbf{Xw}) = 0 \\
    \ \mathbf{dX}^T\mathbf{y}-\mathbf{dX}^T\mathbf{Xw} = 0 \\
    \ \mathbf{dX}^T\mathbf{y} = \mathbf{dX}^T\mathbf{Xw} \\
    \ \mathbf{w} = (\mathbf{dX}^T\mathbf{X})^{-1}\mathbf{dX}^T\mathbf{Y}
\end{align}
$$

### $k$-NN classification

Implement a $k$-Nearest neighbors classifier from scratch in Python using only basic matrix operations with `numpy` and `scipy`. Train and evaluate the classifier on the breast cancer dataset, using all features. Show the performance of the classifier for different values of $k$ (plot the results in a graph). Note that for optimal results, you should normalize the features (e.g. to the $[0, 1]$ range or to have a zero mean and unit standard deviation).

```python
import numpy as np
from sklearn.datasets import load_diabetes, load_breast_cancer
import operator
import matplotlib.pyplot as plt
import sys
sys.path.append('code/')
from def_of_week_1_v2 import *
diabetes = load_diabetes()
breast_cancer = load_breast_cancer()

x = breast_cancer.data[:]

# normalize the data
max_ft = np.zeros(x.shape[1])           #create vector max_ft
for ii in range (0,x.shape[1]):
    max_ft[ii] = np.max(x[:,ii])        #try to make this an array of all max values and compute the normalized data in a matrix division

x_norm = x*(1/max_ft)

X_train = x_norm[:350] #CHANGE BACK TO 350
y_train = breast_cancer.target[:350, np.newaxis]
X_test = x_norm[350:]
y_test = breast_cancer.target[350:, np.newaxis]

dict_of_acc ={}

# the value of k needs always to be an odd number
for k in range(1,X_train.shape[0]+1,2):        #len(X_train)+1,
    predicted_labels = kNN_test(X_train, X_test, y_train, y_test, k)
    #calculates the error of every k
    #knn_error = error_squared(y_test,predicted_labels)
    predictionmeasure = predicted_labels ==y_test.T #whuch predictions were true or false
    right = sum(sum(predictionmeasure))  #nr of correct predictions
    accuracy = right/len(y_test)
    dict_of_acc[k]=accuracy

# It plots all the errors (y-as) against the k value's (x-as)
plt.plot(list(dict_of_acc.keys()), list(dict_of_acc.values()))
plt.xlabel('value of k')
plt.ylabel('accuracy')
plt.title('Accuracy dependence on k')
plt.show()

# It will print the value of K with the lowest error
whichK = sorted(dict_of_acc.items(), key=operator.itemgetter(1), reverse=True)
bestKvalue = whichK[0][0]
print("the value of k with the best accuracy =", bestKvalue)
```

```python

```

It is important to choose an odd k-value.
As you can see in the image, as the k-value becomes too high, the error
will become higher. This process is also called overfitting.
The best value of k in this example is 31

### $k$-NN regression

Modify the $k$-NN implementation to do regression instead of classification. Compare the performance of the linear regression model and the $k$-NN regression model on the diabetes dataset for different values of $k$..

```python
import sys
sys.path.append('code/')
from knn_classifier1 import *
import numpy as np
from sklearn.datasets import load_diabetes
import operator
from scipy.special import expit
import matplotlib.pyplot as plt

diabetes = load_diabetes()
X_train_d = diabetes.data[:300, np.newaxis, 3]
y_train_d = diabetes.target[:300, np.newaxis]
X_test_d = diabetes.data[300:, np.newaxis, 3]
y_test_d = diabetes.target[300:, np.newaxis]

dict_of_reg_errors ={}
all_errors = 0

for k in range(1,len(X_train_d)+1,2):      # the value of k needs always to be an odd number
    prediction = kNN_test_reg(X_train_d, X_test_d, y_train_d, y_test_d, k)
    knn_error = error_squared(y_test_d,prediction) #calculates the error of every k
    all_errors = all_errors + knn_error
    if k in dict_of_reg_errors:
        print (k)
    # creates a dictionary with the k as key and the error as value
    else:
        dict_of_reg_errors[k]=knn_error
# It plots all the errors (y-as) against the k value's (x-as)
plt.figure()
plt.plot(list(dict_of_reg_errors.keys()), list(dict_of_reg_errors.values()))
plt.xlabel('value of k')
plt.ylabel('error')
plt.title('predicting which k is optimal for the lowest error')

# It will print the value of K with the lowest error
whichK = sorted(dict_of_reg_errors.items(), key=operator.itemgetter(1))
bestKvalue = whichK[0][0]
print ("the value of k with the lowest error =", bestKvalue)

#calculate the mean squared error
MSE = all_errors/len(y_test_d)
print("the value of the mean squared error =", MSE)
```

The order of the MSE with k-NN regression is the same order of the MSE calculated with linear regression. The data doesn't show a lineair relationship, thus the sum of squared errors are very high.

### Class-conditional probability

Compute and visualize the class-conditional probability (conditional probability where the class label is the conditional variable, i.e. $P(X = x \mid Y = y)$ for all features in the breast cancer dataset. Assume a Gaussian distribution.

Based on visual analysis of the plots, which individual feature can best discriminate between the two classes? Motivate your answer.

```python
import scipy.stats as stats
from scipy.stats import norm
import matplotlib.pyplot as plt

#1 is B_data(healthy), 0 is M_dataant
B_patients = np.where(breast_cancer.target==1) #which patients have benign breast cancer
B_data = breast_cancer.data[B_patients] #data of these patients

M_patients=np.where(breast_cancer.target==0)
M_data = breast_cancer.data[M_patients]

B_mean = np.average(B_data, axis=0)  #calcalate the mean
B_std = np.std(B_data, axis=0)  #calculate standard deviation

M_mean = np.average(M_data, axis=0)
M_std = np.std(M_data, axis=0)

#for ii in range(30):
ii=27
xB = np.linspace(B_mean[ii]-3*B_std[ii],B_mean[ii]+3*B_std[ii],100)
xM = np.linspace(M_mean[ii]-3*M_std[ii], M_mean[ii]+3*M_std[ii], 100)
plt.figure()
plt.plot(xB, stats.norm.pdf(xB,B_mean[ii],B_std[ii]),xM,stats.norm.pdf(xM,M_mean[ii],M_std[ii]))
plt.title(ii)

```

When the class-conditional probabilities for all 30 features were plotted, the 28th feature 'worst concave points' gave the most distinguishable classes.


# Chap_07_Nochapter
## Comparison of Forward and Futures Contracts
skipped...
## Hedging
skipped, still here though. ***Hedging***: *eliminate some or all of the risks*. We can hedge with **futures**, or even ***perfect hedge***, that if there's a future contract that trades:

- exactly the asset we want to hedge
- with the exact maturity we want to hedge

Otherwise, ***asset mismatch*** or ***maturity mismatch***. Then, the solution is **cross-hedging**.

## Cross-Hedging
also skipped, presented here though. Let
$
% color
% Aquamarine, black, blue, brown, cyan, darkgray, gray, green, lightgray, lime, magenta, olive, orange, pink, purple, red, teal, violet, white, yellow
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\bbspace}{\;\;\;\;\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\given}[1]{\left. #1 \right|}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N}\left( #1 \right)}
\newcommand{\Exp}[1]{\mathrm{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Avar}[1]{\mathrm{Avar}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left( #1 \right)}
\newcommand{\Corr}[1]{\mathrm{Corr}\left( #1 \right)}
\newcommand{\ExpH}{\mathrm{E}}
\newcommand{\VarH}{\mathrm{Var}}
\newcommand{\AVarH}{\mathrm{Avar}}
\newcommand{\CovH}{\mathrm{Cov}}
\newcommand{\CorrH}{\mathrm{Corr}}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\wp}{\text{with probability }}
\newcommand{\FSD}{\text{FSD}}
\newcommand{\SSD}{\text{SSD}} S_T^\P1$ be the **spot price** of the asset at which the hedger has to deliver at time $T$; and $F_2\P{t,U}$ be the **futures price** at time $t$ of the contract maturing at time $U$ on the asset $S^\P2$, which will be used for **hedging**.

The hedger takes a *long position* in $\delta$ units of the futures $F_2\P{t,U}$. Here $\delta$ is called the ***hedge ratio***. After hedging, we have the ***risk exposure*** $H$:

$$H = S_t^\P1 - S_T^\P1 + \delta\P{F_2\P{T,U} - F_2\P{t,U}}$$

- $S_t^\P1 - S_T^\P1$: the change in the value of the asset to be delivered at time $T$
- $\delta\P{F_2\P{T,U} - F_2\P{t,U}}$: the payoff of holding the *long position* in the futures contract between $t$ and $T$.

Ideally, $H=0$, meaning the perfect hedge.
## Optimal $\delta$
skipped, interesting though.

## Delta Neutral Strategy
Risk exposure $H$ per one call option is $H = c\P{0} - c\P{T} +y\P{S_T-S_0}$. Here the ***delta hedging*** means: *to hedge your position by holding the stock with the number of shares equal to the option's delta*.

Under the **single-period binomial tree** model, $H = x-xe^{rT}$, and with more periods, we can hedge dynamically. Then the **continuous-time** model, we define the ***Delta***:

$$\Delta_c \equiv \ffrac{\partial c\P{t,S_t}}{\partial S_t}$$

Commonly, an option written on an underlying asset $S$ is most sensitive to the changes in the value of $S$. Hence, the largest part of the risk comes from the price movements of asset $S$.

And the **first order approximation** for $c$:

$$c\P{t,S_t} \approx c\P{0,S_t} + \Delta_c\P{S_t-S_0}$$

Then the ***Delta*** of a portfolio: *the change in the value of the portfolio divided by the change in the value of the underlying asset*.

And the position with a **Delta** of $0$ is called ***delta neutral***, meaning that the portfolio doesn't change if the price of the underlying asset changes.

**e.g.**

We are short $c\P{t,S_t}$ and we want to hedge the associated risk with respect to the movements of $S_t$ by holding a position in $S_t$. We should find the number of shares of $S_t$ we should hold, namely $y$, in the complete portfolio $\Pi$ in order to make $\Delta_\Pi = 0$:

$$\Delta_\Pi = -\Delta_c + y \times 1 = 0 \Rightarrow y = \Delta_c$$
***

A portfolio is called ***long*** (***short***) ***Delta*** if its **Delta** is positive, $\Delta_\Pi > 0$ (negative with $\Delta_\Pi < 0$), in which case its value will *increase* (*decrease*) if the value of $S_t$ increases, all other variables being constant.

***Static Hedging***: The hedge is set up initially and *never adjusted*. That's exactly what we do under the **single-period binomial model**, where a perfect replication of a payoff $c\P{T,S_T}$ is achieved by holding exactly $\Delta_c$ shares of the underlying at the initial time. All the risk is eliminated keeping the portfolio **Delta-Neutral** under this model. This result is not necessarily true in other models. However in practice, $S_t$ changes frequently thus to neutralize the **Delta**, we need to adjust the number of shares of $S_t$ held in the portfolio. We called this the ***dynamic hedging*** strategy.

And for the **perfect hedge**, rebalancing must take *continuously*. This requires that the model and its parameters are exactly correct.

## Delta under the Black-Scholes Model
The **Black-Scholes-Mertion equation** is derived by setting up a **delta-neutral position** (*short* $1$ option and *long* $\Delta_c$ shares of the stock) and arguing that the return on the position should be the *risk-free interest rate*.

For the European call option, BS equation gives

$$c\P{t,S_t} = S_t\N{d_1} - Ke^{-r\P{T-t}}\N{d_2}\\
d_{1,2} = \ffrac{1}{\sigma\sqrt{T-t}}\P{\log\ffrac{S_t}{K}+\P{r\pm\ffrac{\sigma^2}{2}}\P{T-t}}$$

and we have $\Delta_c = \N{d_1}$ and then using the put-call parity we have $\Delta_p = \N{d_1} -1$. What's more

- $\Delta_c > 0$: the **replicating** portfolio of the call, never goes short in the stock
- $\Delta_c < 1$: holding one share or more of the stock is *more than enough* to **hedge** the call option payoff.
- $\Delta_p < 0$: the **replicating** portfolio, of the put, never goes long in the stock
- $\Delta_p > -1$: holding one share or more of the stock is *more than enough* to **hedge** the put option payoff.

## Option Sensitivities
*Partial derivatives* of the portfolio value with respect to parameters are called ***Greeks***. The five standard **Greeks** for a *portfolio* with value $V$ are

$$\begin{array}{c|ccccc}\hline
\text{name} & \text{Delta} & \text{Theta} & \text{Gamma} & \text{Vega} & \text{Rho}\\ \hline
\text{expression} & \Delta = \ffrac{\partial V}{\partial S_t} & \Theta = \ffrac{\partial V}{\partial t} & \Gamma = \ffrac{\partial^2 V}{\partial S_t^2} & {\large\nu} = \ffrac{\partial V}{\partial \sigma} & {\large\rho} = \ffrac{\partial r}{\partial S_t}\\\hline
\end{array}$$

### $\Theta$, derivative with respect to time
**Theta** is sometimes referred to as the ***time decay*** of the portfolio.

$$\begin{align}
\Theta_c &= S_t\mathcal{N}'\P{d_1}\ffrac{\partial d_1}{\partial t} - rKe^{-r\P{T-t}}\N{d_2}-Ke^{-r\P{T-t}}\mathcal N'\P{d_2}\ffrac{\partial d_2}{\partial t}\\
\Theta_p &= rKe^{-r\P{T-t}}\N{-d_2}-Ke^{-r\P{T-t}}\mathcal N'\P{-d_2}\ffrac{\partial d_2}{\partial t} + S_t\mathcal{N}'\P{-d_1}\ffrac{\partial d_1}{\partial t}
\end{align}$$

and using the facts that $S_t\mathcal{N}'\P{\pm d_1} = Ke^{-r\P{T-t}}\mathcal{N}'\P{\pm d_2}$ and $\ffrac{\partial d_1}{\partial t} = \ffrac{\partial d_2}{\partial t} - \ffrac{\sigma}{2\sqrt{T-t}}$ we have

$$\begin{align}
\Theta_c &= - \ffrac{\sigma}{2\sqrt{T-t}}S_t\mathcal{N}'\P{d_1} - rKe^{-r\P{T-t}}\N{d_2}\\
\Theta_p &= - \ffrac{\sigma}{2\sqrt{T-t}}S_t\mathcal{N}'\P{d_1} + rKe^{-r\P{T-t}}\N{-d_2}
\end{align}$$

Usually $\Theta$ is negative, since option tends to be less valuable.

### $\Gamma$, second order derivative with respect to asset price
If the *absolute value* of $\Gamma$ is large, $\Delta$ would be highly sensitive to the price of the underlying asset, meaning that adjustments to keep a portfolio **delta neutral** need to be made relatively frequently.

And since $\Delta_c = \N{d_1}$ and $\Delta_p = \N{d_1}-1$, we have, for European call/put

$$\Gamma = \ffrac{\partial \N{d_1}}{\partial S_t} = \mathcal{N}'\P{d_1} \ffrac{1}{\sigma\sqrt{T-t}}\ffrac{1}{S_t}>0$$

Up to this point, we can already draw two very useful conclusions.

### Approximation
Using **Taylor Expansion**:

$$\begin{align}
V\P{t+\Delta t,S_{t+\Delta t}} &= V\P{t,S_t} + \Delta t \ffrac{\partial V\P{t,S_t}}{\partial t} + O\P{\P{\Delta t}
^2} + \P{S_{t+\Delta t} - S_t}\ffrac{\partial V\P{t,S_t}}{\partial S_t} \\
&\bspace + \ffrac{1}{2}\P{S_{t+\Delta t} - S_t}^2 \ffrac{\partial^2 V\P{t,S_t}}{\partial S_t\partial S_t}+\cdots\\
&\approx V\P{t,S_t} + \color{brown}{\Theta}\Delta t + \color{brown}{\Delta} \P{S_{t+\Delta t} - S_t} + \ffrac{1}{2}\color{brown}{\Gamma} \P{S_{t+\Delta t} - S_t}^2
\end{align}$$

Further, if the portolio is **Delta Neutral**, then

$$V\P{t+\Delta t,S_{t+\Delta t}} \approx V\P{t,S_t} + \color{brown}{\Theta}\Delta t + \ffrac{1}{2}\color{brown}{\Gamma} \P{S_{t+\Delta t} - S_t}^2$$

### Reaching Gamma Neutral
Suppose a **Delta Neutral** portfolio has $\Gamma$ as its **Gamma**, and another traded opion has **Gamma** equal to $\Gamma_T$. If the number of traded options added to the portfolio is $w_T$, gamma of the new portfolio is $w_T \Gamma_T + \Gamma$ and thus buying $w_T = -\ffrac{\Gamma}{\Gamma_T}$ would lead to **Gamma Neutral**.

However this changes the $\Delta$ of the portfolio, we need to trade more shares to neutralize our $\Delta$ after that.

**e.g.**

Suppose that a portfolio is **Delta Neutral** and has a $\Gamma$ of $-3000$. The $\Delta_c$ and $\Gamma_c$ of a particular traded call option are $0.62$ and $1.50$, respectively. How to make the portfolio **Delta Neutral** and **Gamma Neutral**?

> The portfolio can be made **Gamma Neutral** by including in the portfolio a *long position* of $-\ffrac{\Gamma}{\Gamma_c} = -\ffrac{-3000}{1.5} = 2000$ call option. However, the **Delta** of the portfolio will then change fram $0$ to $\Delta_v = 2000\times 0.62 = 1240$. Thus, $1240$ units of the underlying asset must be *sold* from the portfolio to keep it **Delta Neutral**.
***

***delta hedging error***: when the stock price moves from $S_{1}$ to $S_{2}$ the option price moves from $C_1$ to $C_2$, while **delta hedging** assumes that it moves to $C_2'$. The difference between $C_2$ and $C_2'$ is the **delta hedging error**.

- **Delta neutrality** provides protection against *relatively small* stock price moves between rebalancing.
- **Gamma neutrality** provides protection against *larger* movements in this stock price between hedge rebalancing.

### $\large\nu$, derivative with respect to volatility

For a European call or put option, **Vega** is given by

$$\begin{align}{\large{\nu}} &= S_t \mathcal N'\P{d_1}\ffrac{\partial d_1}{\partial \sigma} - Ke^{-r\P{T-t}}\mathcal N'\P{d_2}\ffrac{\partial d_2}{\partial \sigma}\\
&= S_t \mathcal N'\P{d_1}\ffrac{\partial d_1}{\partial \sigma} - S_t \mathcal N'\P{d_1}\P{\ffrac{\partial d_1}{\partial \sigma} - \sqrt{T-t}}\\
&= S_t \sqrt{T-t} \mathcal N'\P{d_1}>0
\end{align}$$

Thus, both the European call and put prices *increase with increasing volatility*. And this doesn't violate our intuition. Higher volatility lead to a higher chance that the option may end up either **deeper in-the-money** or **deep out-of-the-money**, while there's no extra peanlty for the option to be deeper out-of-the-money but the payoff increases when the option expires deeper in the money.

### $\large\rho$, derivative with respect to interest rate

$$\begin{align}
{\large\rho}_c &= S_t \mathcal N'\P{d_1}\ffrac{\partial d_1}{\partial r} + K\P{T-t}e^{-r\P{T-t}}\N{d_2} - Ke^{-r\P{T-t}}\mathcal N'\P{d_2}\ffrac{\partial d_2}{\partial r}\\
&= K\P{T-t}e^{-r\P{T-t}}\N{d_2} + S_t \mathcal N'\P{d_1}\ffrac{\partial d_1}{\partial \sigma} - S_t \mathcal N'\P{d_1}\P{\ffrac{\partial d_1}{\partial \sigma} - 0}\\
&= K\P{T-t}e^{-r\P{T-t}}\N{d_2} > 0\\
{\large\rho}_p &= -K\P{T-t}e^{-r\P{T-t}} \N{-d_2} < 0
\end{align}$$

A *higher interest* rate lowers the present value of the cost of exercising the European call option at expiration (the effect is similar to the *lowering of the strike price*), in turn this increases the call price. Reverse effect holds for the put option price.

***

FinMath/Models and Pricing of Financial Derivatives/Chap_07_Anonymous.ipynb


```python
#format the book
%matplotlib inline
from __future__ import division, print_function
import sys;sys.path.insert(0,'..')
from  book_format import load_style;load_style('..')
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Lora');

//@import url('http://fonts.googleapis.com/css?family=Open+Sans');
//@import url('http://fonts.googleapis.com/css?family=Vollkorn');
//@import url('http://fonts.googleapis.com/css?family=Karla');
//@import url('http://fonts.googleapis.com/css?family=Poppins');
//@import url('http://fonts.googleapis.com/css?family=Arimo');
//@import url('http://fonts.googleapis.com/css?family=Roboto');
//@import url('http://fonts.googleapis.com/css?family=Lato');
//@import url('http://fonts.googleapis.com/css?family=Domine');
//@import url('http://fonts.googleapis.com/css?family=Chivo');
//@import url('http://fonts.googleapis.com/css?family=Cardo');
//@import url('http://fonts.googleapis.com/css?family=Arvo');
//@import url('http://fonts.googleapis.com/css?family=Crimson+Text');
//@import url('http://fonts.googleapis.com/css?family=Ubuntu');
//@import url('http://fonts.googleapis.com/css?family=Fontin');
//@import url('http://fonts.googleapis.com/css?family=Raleway');
//@import url('http://fonts.googleapis.com/css?family=Merriweather');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        width: 850px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell_render{
        font-family: 'Lora';
        //font-family: 'Open Sans';
        //font-family: 'Karla',verdana,arial,sans-serif;
        //font-family: 'Roboto',verdana,arial,sans-serif;
        //font-family: 'Lato',verdana,arial,sans-serif;
        //font-family: 'Domine',verdana,arial,sans-serif;
        //font-family: 'Chivo',verdana,arial,sans-serif;
        //font-family: 'Cardo',verdana,arial,sans-serif;
        //font-family: 'Arvo',verdana,arial,sans-serif;
        //font-family: 'Poppins',verdana,arial,sans-serif;
        //font-family: 'Ubuntu',verdana,arial,sans-serif;
        //font-family: 'Fontin',verdana,arial,sans-serif;
        //font-family: 'Raleway',verdana,arial,sans-serif;
        //font-family: 'Merriweather',verdana,arial,sans-serif;
        //font-family: 'Crimson Text', verdana,arial,sans-serif;
        //font-family: verdana,arial,sans-serif;
        //font-family: arial,sans-serif;
        line-height: 125%;
        font-size: 130%;
        text-align: justify;
        text-justify:inter-word;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 400;
        font-size: 12pt;
        //font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       font-weight: bold;
       color:#447adb;
    }
    a:visited{
       font-weight: bold;
       color: #1d3b84;
    }
    a:hover{
       font-weight: bold;
       color: #1d3b84;
    }
    a:focus{
       font-weight: bold;
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

# Computing and plotting PDFs of discrete data

So let's investigate how to compute and plot probability distributions.

First, let's make some data according to a normal distribution. We use `numpy.random.normal` for this. The parameters are not well named. `loc` is the mean of the distribution, and `scale` is the standard deviation. We can call this function to create an arbitrary number of data points that are distributed according to that mean and std.

```python
import numpy as np
import numpy.random as random

mean = 3
std = 2

data = random.normal(loc=mean, scale=std, size=50000)
print(len(data))
print(data.mean())
print(data.std())
```

    50000
    3.00621596517
    1.99696435477

As you can see from the print statements we got 5000 points that have a mean very close to 3, and a standard deviation close to 2.

We can plot this Gaussian by using `scipy.stats.norm` to create a frozen function that we will then use to compute the pdf (probability distribution function) of the Gaussian.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_normal(xs, mean, std, **kwargs):
    norm = stats.norm(mean, std)
    plt.plot(xs, norm.pdf(xs), **kwargs)

xs = np.linspace(-5, 15, num=200)
plot_normal(xs, mean, std, color='k')
```

But we really want to plot the PDF of the discrete data, not the idealized function.

There are a couple of ways of doing that. First, we can take advantage of `matplotlib`'s `hist` method, which computes a histogram of a collection of data. Normally `hist` computes the number of points that fall in a bin, like so:

```python
plt.hist(data, bins=200)
plt.show()
```

that is not very useful to us - we want the PDF, not bin counts. Fortunately `hist` includes a `density` parameter which will plot the PDF for us.

```python
plt.hist(data, bins=200, normed=True)
plt.show()
```

I may not want bars, so I can specify the `histtype` as 'step' to get a line.

```python
plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
plt.show()
```

To be sure it is working, let's also plot the idealized Gaussian in black.

```python
plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
norm = stats.norm(mean, std)
plt.plot(xs, norm.pdf(xs), color='k', lw=2)
plt.show()
```

There is another way to get the approximate distribution of a set of data. There is a technique called *kernel density estimate* that uses a kernel to estimate the probability distribution of a set of data. NumPy implements it with the function `gaussian_kde`. Do not be mislead by the name - Gaussian refers to the type of kernel used in the computation. This works for any distribution, not just Gaussians. In this section we have a Gaussian distribution, but soon we will not, and this same function will work.

```python
kde = stats.gaussian_kde(data)

xs = np.linspace(-5, 15, num=200)
plt.plot(xs, kde(xs))
plt.show()
```

## Monte Carlo Simulations

We (well I) want to do this sort of thing because I want to use monte carlo simulations to compute distributions. It is easy to compute Gaussians when they pass through linear functions, but difficult to impossible to compute them analytically when passed through nonlinear functions. Techniques like particle filtering handle this by taking a large sample of points, passing them through a nonlinear function, and then computing statistics on the transformed points. Let's do that.

We will start with the linear function $f(x) = 2x + 12$ just to prove to ourselves that the code is working. I will alter the mean and std of the data we are working with to help ensure the numbers that are output are unique It is easy to be fooled, for example, if the formula multipies x by 2, the mean is 2, and the std is 2. If the output of something is 4, is that due to the multication factor, the mean, the std, or a bug? It's hard to tell.

```python
def f(x):
    return 2*x + 12

mean = 1.
std = 1.4
data = random.normal(loc=mean, scale=std, size=50000)

d_t = f(data) # transform data through f(x)

plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)

plt.ylim(0, .35)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

This is what we expected. The input is the Gaussian $\mathcal{N}(\mu=1, \sigma=1.4)$, and the function is $f(x) = 2x+1$. Therefore we expect the mean to be shifted to $f(\mu) = 2*1+12=14$. We can see from the plot and the print statement that this is what happened.

Before I go on, can you explain what happened to the standard deviation? You may have thought that the new $\sigma$ should be passed through $f(x)$ like so $2(1.4) + 12=14.81$. But that is not correct - the standard deviation is only affected by the multiplicative factor, not the shift. If you think about that for a moment you will see it makes sense. We multiply our samples by 2, so they are twice as spread out as before. Standard deviation is a measure of how spread out things are, so it should also double. It doesn't matter if we then shift that distribution 12 places, or 12 million for that matter - the spread is still twice the input data.

## Nonlinear Functions

Now that we believe in our code, lets try it with nonlinear functions.

```python
def f2(x):
    return (np.cos((1.5*x + 2.1))) * np.sin(0.3*x) - 1.6*x

d_t = f2(data)
plt.subplot(121)
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)

plt.subplot(122)
kde = stats.gaussian_kde(d_t)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, kde(xs), 'k')
plot_normal(xs, d_t.mean(), d_t.std(), color='g', lw=3)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

Here I passed the data through the nonlinear function $f(x) = \cos(1.5x+2.1)\sin(\frac{x}{3}) - 1.6x$. That function is quite close to linear, but we can see how much it alters the pdf of the sampled data.

There is a lot of computation going on behind the scenes to transform 50,000 points and then compute their PDF. The Extended Kalman Filter (EKF) gets around this by linearizing the function at the mean and then passing the Gaussian through the linear equation. We saw above how easy it is to pass a Gaussian through a linear function. So lets try that.

We can linearize this by taking the derivative of the function at x. We can use sympy to get the derivative.

```python
import sympy
x = sympy.symbols('x')
f = sympy.cos(1.5*x+2.1) * sympy.sin(x/3) - 1.6*x
dfx = sympy.diff(f, x)
dfx
```

    -1.5*sin(x/3)*sin(1.5*x + 2.1) + cos(x/3)*cos(1.5*x + 2.1)/3 - 1.6

We can now compute the slope of the function by evaluating the derivative at the mean.

```python
m = dfx.subs(x, mean)
m
```

    -1.66528051815545

The equation of a line is $y=mx+b$, so the new standard deviation should be $~1.67$ times the input std. We can compute the new mean by passing it through the original function because the linearized function is just the slope of f(x) evaluated at the mean. The slope is a tangent that touches the function at $x$, so both will return the same result. So, let's plot this and compare it to the results from the monte carlo simulation.

```python
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)
plot_normal(xs, f2(mean), abs(float(m)*std), color='k', lw=3, label='EKF')
plot_normal(xs, d_t.mean(), d_t.std(), color='r', lw=3, label='MC')
plt.legend()
plt.show()
```

We can see from this that the estimate from the EKF (in red) is not exact, but it is not a bad approximation either.


「TransformerとTorchTextを用いたsequence-to-sequenceモデルの学習」
===============================================================
【原題】Sequence-to-Sequence Modeling with nn.Transformer and TorchText

【元URL】https://pytorch.org/tutorials/beginner/transformer_tutorial.html

【翻訳】電通国際情報サービスISID AIトランスフォーメーションセンター　御手洗 拓真

【日付】2020年10月13日

【チュトーリアル概要】

本チュートリアルでは、Attention機構を備えたTransformerモジュールを使用したsequence-to-sequenceモデルの実装手法と訓練方法を解説します。

WikiText2から取得した文章において、文章＝単語系列（sequence）を入力し、その次に来るであろう単語を予測するsequence-to-sequenceモデルを構築します。

---

本ノートブックは、[nn.Transformer](https://pytorch.org/docs/master/nn.html?highlight=nn%20transformer#torch.nn.Transformer)モジュールを使用した、sequence-to-sequenceモデル学習の解説チュートリアルです。

PyTorch 1.2 以降では、論文 [Attention is All You Need](https://arxiv.org/pdf/1706.03762.pdf)を基に実装された、標準的なTransformerのモジュールが用意されています。

Transformerモデルは、これまでのモデル（日本語訳注：RNN系）と比べてより並列化が容易であると同時に、多くのsequence-to-sequenceのタスクにおいて、より優れた結果が得られることが実証されています。

`nn.Transformer`モジュールは、Attensionメカニズム（最近、
[nn.MultiheadAttention](https://pytorch.org/docs/master/nn.html?highlight=multiheadattention#torch.nn.MultiheadAttention)として実装された別のモジュール）に完全に依存しています。このAttensionメカニズムにより、入力と出力の間の大域的な依存関係を捉えることができます。

`nn.Transformer`モジュールは、単一でも使用しやすいように、高度にモジュール化されており、簡単に、`nn.Transformer`モジュールをコンポーネント（本チュートリアルの[nn.TransformerEncoder](https://pytorch.org/docs/master/nn.html?highlight=nn%20transformerencoder#torch.nn.TransformerEncoder)など）に適用したり、あるいは`nn.Transformer`モジュールを使って、コンポーネントを作成したりすることができます。

モデルの定義
----------------

本チュートリアルでは、言語モデルタスクで`NN.TransformerEncoder`モデルを訓練します。

ここでの言語モデルタスク（language modeling task）とは、特定の単語（または単語のシーケンス）が、与えられた単語シーケンスの後に続けて登場する確率を当てるタスクです。

単語（トークン）のシーケンスは、最初にモデルの埋め込み層に渡され、その後、単語の順序の情報を得るための位置エンコーディング層（詳細は次の段落）へと渡されます。

`nn.TransformerEncoder`は、複数の[NN.TransformerEncoderLayer](https://pytorch.org/docs/master/nn.html?highlight=transformerencoderlayer#torch.nn.TransformerEncoderLaye)の層で構成されています。

言語モデルタスクでは、入力シーケンスと共に、アテンション・マスクが必要です。

なぜなら、`NN.TransformerEncoder`のSelf-Attention層では、シーケンス内でより前方に登場するトークンのみに着目することが許されているためです。

※日本語訳注：RNNでも単語は前から順番に入力され、後ろの単語は通常、考慮できないので、その状態を再現しています（後方からのシーケンスを利用する場合もありますが）。

普通の日常会話でも、会話最中に、会話の先の内容を事前に聞くことはできない点と同じ状況です。

そこで、言語モデルタスクでは、後方の位置にある単語（トークン）は未知のトークンとして扱うため、マスクする必要があります。

正解の単語を求めるために、`NN.TransformerEncoder`モデルの出力は、最終的に全結合層に送られ、その後にlog-Softmax関数で処理されます。

```python
# 日本語訳注：追加
%matplotlib inline
!pip3 install torchtext==0.4.0
```

    Collecting torchtext==0.4.0
      Downloading https://files.pythonhosted.org/packages/43/94/929d6bd236a4fb5c435982a7eb9730b78dcd8659acf328fd2ef9de85f483/torchtext-0.4.0-py3-none-any.whl (53kB)
         |████████████████████████████████| 61kB 8.1MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchtext==0.4.0) (1.18.5)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext==0.4.0) (4.41.1)
    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchtext==0.4.0) (1.7.0+cu101)
    Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext==0.4.0) (2.23.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchtext==0.4.0) (1.15.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch->torchtext==0.4.0) (3.7.4.3)
    Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch->torchtext==0.4.0) (0.8)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch->torchtext==0.4.0) (0.16.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext==0.4.0) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext==0.4.0) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext==0.4.0) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext==0.4.0) (2.10)
    Installing collected packages: torchtext
      Found existing installation: torchtext 0.3.1
        Uninstalling torchtext-0.3.1:
          Successfully uninstalled torchtext-0.3.1
    Successfully installed torchtext-0.4.0

```python
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerModel(nn.Module):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):
        super(TransformerModel, self).__init__()
        from torch.nn import TransformerEncoder, TransformerEncoderLayer
        self.model_type = 'Transformer'
        self.src_mask = None
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = nn.Linear(ninp, ntoken)

        self.init_weights()

    def _generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src):
        if self.src_mask is None or self.src_mask.size(0) != src.size(0):
            device = src.device
            mask = self._generate_square_subsequent_mask(src.size(0)).to(device)
            self.src_mask = mask

        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src, self.src_mask)
        output = self.decoder(output)
        return output
```

`PositionalEncoding` モジュールは、シーケンス内のトークンの相対的な位置、もしくは絶対的な位置に関する情報をモデルに与えます。

位置エンコーディング層（PositionalEncoding Layer）は、埋め込み層と同じ次元であり、位置エンコーディングの出力と、埋め込み層の出力は足し算することができます。

ここでは，異なる周波数のサイン波関数とコサイン波関数を使用し、位置エンコーディングを行っています。

```python
class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
```

データの読み込みとバッチ処理
-------------------

学習には`torchtext`のWikitext-2データセットを使用します。

vocabオブジェクトは訓練データセットを元に構築され、トークン（単語）をテンソル形式の数値に変換するために使用されます。

`batchify()`  関数は、シーケンス形式のデータ（トークンが左から右に一つずつ並んだ形）を、列が並んだ形式に変換します。変換する際には、データを``batch_size`` 変数のサイズで分割し、最後に余ったトークンは無視されます。

例えば、アルファベットをシーケンス (長さ26) とし、バッチサイズを 4 とすると、アルファベットを長さ 6 の 4 つのシーケンスに分割したのが以下の例です。

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

各列（=バッチ）は、モデル内では独立したものとして扱われます。そのため、例えば ``F`` と ``G`` の依存関係を学習することはできません。

（日本語訳注：FとGは連続したアルファベットという依存関係がありますが、それはこの学習では考慮できないことになります）

しかしながら、各バッチを独立したものとして扱うことで、より効率的なバッチ処理が可能になります。

```python
import torchtext
from torchtext.data.utils import get_tokenizer
TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"),
                            init_token='<sos>',
                            eos_token='<eos>',
                            lower=True)
train_txt, val_txt, test_txt = torchtext.datasets.WikiText2.splits(TEXT)
TEXT.build_vocab(train_txt)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def batchify(data, bsz):
    data = TEXT.numericalize([data.examples[0].text])
    # データセットをbszサイズに分割した際のバッチ数を求める
    nbatch = data.size(0) // bsz
    # Trim off any extra elements that wouldn't cleanly fit (remainders).
    data = data.narrow(0, 0, nbatch * bsz)
    # Evenly divide the data across the bsz batches.
    data = data.view(bsz, -1).t().contiguous()
    return data.to(device)

batch_size = 20
eval_batch_size = 10
train_data = batchify(train_txt, batch_size)
val_data = batchify(val_txt, eval_batch_size)
test_data = batchify(test_txt, eval_batch_size)
```

    wikitext-2-v1.zip: 100%|██████████| 4.48M/4.48M [00:00<00:00, 64.5MB/s]

    downloading wikitext-2-v1.zip
    extracting

```python
# 日本語訳注：追記（GPUの確認と訓練データの確認）

print("GPU環境：", device)  # GPU環境であれば、cuda と表示されます
print("訓練データのサンプル：", train_txt.examples[0].text[:100])
```

    GPU環境： cuda
    訓練データのサンプル： ['<eos>', '=', 'valkyria', 'chronicles', 'iii', '=', '<eos>', '<eos>', 'senjō', 'no', 'valkyria', '3', '<unk>', 'chronicles', '(', 'japanese', '戦場のヴァルキュリア3', ',', 'lit', '.', 'valkyria', 'of', 'the', 'battlefield', '3', ')', ',', 'commonly', 'referred', 'to', 'as', 'valkyria', 'chronicles', 'iii', 'outside', 'japan', ',', 'is', 'a', 'tactical', 'role', '@-@', 'playing', 'video', 'game', 'developed', 'by', 'sega', 'and', 'media', '.', 'vision', 'for', 'the', 'playstation', 'portable', '.', 'released', 'in', 'january', '2011', 'in', 'japan', ',', 'it', 'is', 'the', 'third', 'game', 'in', 'the', 'valkyria', 'series', '.', '<unk>', 'the', 'same', 'fusion', 'of', 'tactical', 'and', 'real', '@-@', 'time', 'gameplay', 'as', 'its', 'predecessors', ',', 'the', 'story', 'runs', 'parallel', 'to', 'the', 'first', 'game', 'and', 'follows', 'the']

**入力シーケンスとTargetシーケンスを生成するための関数**

``get_batch()``関数は，Transformerモデルの入力シーケンスと、Targetシーケンスを生成します。

この関数はソースデータを変数``bptt``の長さのチャンクデータに細分化します。

 言語モデルのタスクでは、入力シーケンスに後続する単語がTargetとして必要となります。

 例えば以下の図で、``bptt`` の値が 2 の場合、``i`` = 0 に後続する2 つの要素を取得します。

日本語訳注：bpttより2つの要素をinputに使用。

下の図では、モデルへの入力（とあるタイミングで来た単語）とモデルが予測して欲しい出力（次のタイミングで来るべき単語）を表示しています。

``get_batch()``関数の返り値``data``変数の 0 次元目 がチャンクの長さで、これはトランスフォーマーモデルの 次元``S``と一致していることに注意してください。

また、``data``変数の 1 次元目 はバッチサイズを示す次元数 ``N`` となっています。

```python
bptt = 35
def get_batch(source, i):
    seq_len = min(bptt, len(source) - 1 - i)
    data = source[i:i+seq_len]
    target = source[i+1:i+1+seq_len].view(-1)
    return data, target
```

インスタンスの初期化
--------------------

モデルは、以下のハイパーパラメータを使用して設定されます。vocabのサイズは、vocabオブジェクトの長さと同じです。

```python
ntokens = len(TEXT.vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 2 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value
model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout).to(device)
```

モデルの実行
-------------

[CrossEntropyLoss](https://pytorch.org/docs/master/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss)は損失値を計算するために適用され、[SGD](https://pytorch.org/docs/master/optim.html?highlight=sgd#torch.optim.SGD)はオプティマイザとして使われる確率的勾配降下法です。

初期の学習率は5.0に設定しています。

エポック単位で学習率を調整するために[StepLR](https://pytorch.org/docs/master/optim.html?highlight=steplr#torch.optim.lr_scheduler.StepLR)__を使用します。

訓練中は、勾配爆発を防ぐために、[nn.utils.clip_grad_norm](https://pytorch.org/docs/master/nn.html?highlight=nn%20utils%20clip_grad_norm#torch.nn.utils.clip_grad_norm_) 関数を用いて、全ての勾配をまとめた後、大きさをスケーリングしています。

（日本語訳注：ここでのスケーリングとは、clip_grad_normで勾配が大きすぎる値は設定した上限値に変更するスケーリング操作を示します）

```python
criterion = nn.CrossEntropyLoss()
lr = 5.0 # 学習率
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

import time
def train():
    model.train() # 訓練モードに
    total_loss = 0.
    start_time = time.time()
    ntokens = len(TEXT.vocab.stoi)
    for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
        data, targets = get_batch(train_data, i)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        log_interval = 200
        if batch % log_interval == 0 and batch > 0:
            cur_loss = total_loss / log_interval
            elapsed = time.time() - start_time
            print('| epoch {:3d} | {:5d}/{:5d} batches | '
                  'lr {:02.2f} | ms/batch {:5.2f} | '
                  'loss {:5.2f} | ppl {:8.2f}'.format(
                    epoch, batch, len(train_data) // bptt, scheduler.get_lr()[0],
                    elapsed * 1000 / log_interval,
                    cur_loss, math.exp(cur_loss)))
            total_loss = 0
            start_time = time.time()

def evaluate(eval_model, data_source):
    eval_model.eval() # 検証モードに
    total_loss = 0.
    ntokens = len(TEXT.vocab.stoi)
    with torch.no_grad():
        for i in range(0, data_source.size(0) - 1, bptt):
            data, targets = get_batch(data_source, i)
            output = eval_model(data)
            output_flat = output.view(-1, ntokens)
            total_loss += len(data) * criterion(output_flat, targets).item()
    return total_loss / (len(data_source) - 1)
```

エポックを繰り返します。

その最中に、検証データの損失がそれまでの実行のなかで最も良い（低い）場合はモデルを保存します。

そして、各エポックの後に学習率を調整し小さくします。

```python
best_val_loss = float("inf")
epochs = 3 # The number of epochs
best_model = None

for epoch in range(1, epochs + 1):
    epoch_start_time = time.time()
    train()
    val_loss = evaluate(model, val_data)
    print('-' * 89)
    print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '
          'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),
                                     val_loss, math.exp(val_loss)))
    print('-' * 89)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_model = model

    scheduler.step()
```

    /usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:370: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
      "please use `get_last_lr()`.", UserWarning)

    | epoch   1 |   200/ 2981 batches | lr 5.00 | ms/batch 18.01 | loss  7.99 | ppl  2946.49
    | epoch   1 |   400/ 2981 batches | lr 5.00 | ms/batch 16.62 | loss  6.79 | ppl   890.76
    | epoch   1 |   600/ 2981 batches | lr 5.00 | ms/batch 16.79 | loss  6.37 | ppl   582.28
    | epoch   1 |   800/ 2981 batches | lr 5.00 | ms/batch 16.73 | loss  6.24 | ppl   511.11
    | epoch   1 |  1000/ 2981 batches | lr 5.00 | ms/batch 16.73 | loss  6.12 | ppl   452.94
    | epoch   1 |  1200/ 2981 batches | lr 5.00 | ms/batch 16.74 | loss  6.09 | ppl   442.13
    | epoch   1 |  1400/ 2981 batches | lr 5.00 | ms/batch 16.81 | loss  6.05 | ppl   425.44
    | epoch   1 |  1600/ 2981 batches | lr 5.00 | ms/batch 16.81 | loss  6.04 | ppl   421.63
    | epoch   1 |  1800/ 2981 batches | lr 5.00 | ms/batch 16.86 | loss  5.95 | ppl   384.24
    | epoch   1 |  2000/ 2981 batches | lr 5.00 | ms/batch 16.83 | loss  5.96 | ppl   389.04
    | epoch   1 |  2200/ 2981 batches | lr 5.00 | ms/batch 16.87 | loss  5.85 | ppl   348.46
    | epoch   1 |  2400/ 2981 batches | lr 5.00 | ms/batch 16.87 | loss  5.89 | ppl   361.68
    | epoch   1 |  2600/ 2981 batches | lr 5.00 | ms/batch 16.88 | loss  5.90 | ppl   363.64
    | epoch   1 |  2800/ 2981 batches | lr 5.00 | ms/batch 16.91 | loss  5.81 | ppl   333.02
    -----------------------------------------------------------------------------------------
    | end of epoch   1 | time: 52.96s | valid loss  5.71 | valid ppl   301.92
    -----------------------------------------------------------------------------------------
    | epoch   2 |   200/ 2981 batches | lr 4.51 | ms/batch 17.01 | loss  5.81 | ppl   332.85
    | epoch   2 |   400/ 2981 batches | lr 4.51 | ms/batch 16.95 | loss  5.78 | ppl   322.99
    | epoch   2 |   600/ 2981 batches | lr 4.51 | ms/batch 16.99 | loss  5.60 | ppl   270.27
    | epoch   2 |   800/ 2981 batches | lr 4.51 | ms/batch 16.98 | loss  5.64 | ppl   281.97
    | epoch   2 |  1000/ 2981 batches | lr 4.51 | ms/batch 17.02 | loss  5.60 | ppl   269.70
    | epoch   2 |  1200/ 2981 batches | lr 4.51 | ms/batch 17.01 | loss  5.62 | ppl   276.25
    | epoch   2 |  1400/ 2981 batches | lr 4.51 | ms/batch 17.03 | loss  5.64 | ppl   281.10
    | epoch   2 |  1600/ 2981 batches | lr 4.51 | ms/batch 17.08 | loss  5.67 | ppl   289.51
    | epoch   2 |  1800/ 2981 batches | lr 4.51 | ms/batch 17.13 | loss  5.59 | ppl   268.94
    | epoch   2 |  2000/ 2981 batches | lr 4.51 | ms/batch 17.10 | loss  5.63 | ppl   277.96
    | epoch   2 |  2200/ 2981 batches | lr 4.51 | ms/batch 17.14 | loss  5.52 | ppl   249.62
    | epoch   2 |  2400/ 2981 batches | lr 4.51 | ms/batch 17.10 | loss  5.58 | ppl   265.00
    | epoch   2 |  2600/ 2981 batches | lr 4.51 | ms/batch 17.15 | loss  5.59 | ppl   268.33
    | epoch   2 |  2800/ 2981 batches | lr 4.51 | ms/batch 17.18 | loss  5.52 | ppl   248.84
    -----------------------------------------------------------------------------------------
    | end of epoch   2 | time: 53.46s | valid loss  5.56 | valid ppl   259.92
    -----------------------------------------------------------------------------------------
    | epoch   3 |   200/ 2981 batches | lr 4.29 | ms/batch 17.25 | loss  5.55 | ppl   256.28
    | epoch   3 |   400/ 2981 batches | lr 4.29 | ms/batch 17.22 | loss  5.56 | ppl   259.06
    | epoch   3 |   600/ 2981 batches | lr 4.29 | ms/batch 17.19 | loss  5.37 | ppl   215.03
    | epoch   3 |   800/ 2981 batches | lr 4.29 | ms/batch 17.21 | loss  5.42 | ppl   226.48
    | epoch   3 |  1000/ 2981 batches | lr 4.29 | ms/batch 17.19 | loss  5.38 | ppl   217.09
    | epoch   3 |  1200/ 2981 batches | lr 4.29 | ms/batch 17.20 | loss  5.42 | ppl   225.31
    | epoch   3 |  1400/ 2981 batches | lr 4.29 | ms/batch 17.19 | loss  5.44 | ppl   230.62
    | epoch   3 |  1600/ 2981 batches | lr 4.29 | ms/batch 17.15 | loss  5.48 | ppl   239.92
    | epoch   3 |  1800/ 2981 batches | lr 4.29 | ms/batch 17.20 | loss  5.41 | ppl   222.97
    | epoch   3 |  2000/ 2981 batches | lr 4.29 | ms/batch 17.21 | loss  5.44 | ppl   231.59
    | epoch   3 |  2200/ 2981 batches | lr 4.29 | ms/batch 17.22 | loss  5.32 | ppl   204.66
    | epoch   3 |  2400/ 2981 batches | lr 4.29 | ms/batch 17.20 | loss  5.39 | ppl   219.99
    | epoch   3 |  2600/ 2981 batches | lr 4.29 | ms/batch 17.24 | loss  5.42 | ppl   226.52
    | epoch   3 |  2800/ 2981 batches | lr 4.29 | ms/batch 17.37 | loss  5.34 | ppl   209.23
    -----------------------------------------------------------------------------------------
    | end of epoch   3 | time: 53.91s | valid loss  5.54 | valid ppl   254.24
    -----------------------------------------------------------------------------------------

テストデータセットでモデルを評価する
-------------------------------------

結果を確認するために、ベストモデルでテスト用データセットを評価してみます。

```python
test_loss = evaluate(best_model, test_data)
print('=' * 89)
print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(
    test_loss, math.exp(test_loss)))
print('=' * 89)
```

    =========================================================================================
    | End of training | test loss  5.45 | test ppl   232.03
    =========================================================================================