# CIS 519 Homework 2: Linear Classifiers

- Handed Out: October 5, 2020
- Due: October 19, 2020 at 11:59pm.

Although the solutions are my own, I consulted with the following people
while working on this homework:
Zhijie Qiao

## Preface

- Feel free to talk to other members of the class in doing the homework. I am more concerned that you learn how to solve the problem than that you demonstrate that you solved it entirely on your own. You should, however, **write down your solution yourself**. Please include here the list of people you consulted with in the course of working on the homework:

- While we encourage discussion within and outside the class, cheating and copying code is strictly not allowed. Copied code will result in the entire assignment being discarded at the very least.

- Please use Piazza if you have questions about the homework. Also, please come to the TAs recitations and to the office hours.

- The homework is due at 11:59 PM on the due date. We will be using Gradescope for collecting the homework assignments. You should have been automatically added to Gradescope. If not, please ask a TA for assistance. Post on Piazza and contact the TAs if you are having technical difficulties in submitting the assignment.

- Here are some resources you will need for this assignment (https://www.seas.upenn.edu/~cis519/fall2020/assets/HW/HW2/hw2-materials.zip)

# Overview

### About Jupyter Notebooks

In this homework assignment, we will use a Jupyter notebook to implement, analyze, and discuss ML classifiers.
Knowing and being comfortable with Jupyter notebooks is a must in every data scientist, ML engineer, researcher, etc. They are widely used in industry and are a standard form of communication in ML by intertwining text and code to "tell a story". There are many resources that can introduce you to Jupyter notebooks (they are pretty easy to understand!), and if you still need help any of the TAs are more than willing to help.

We will be using a local instance of Jupyter instead of Colab. You are of course free to use Colab, but you will need to understand how to hook your Colab instance with Google Drive to upload the datasets and to save images.

### About the Homework

You will experiment with several different linear classifiers and analyze
their performances in both real and synthetic datasets. The goal is to understand the differences and
similarities between the algorithms and the impact that the dataset characteristics have on the
algorithms' learning behaviors and performances.

In total, there are seven different learning algorithms which you will implement.
Six are variants of the Perceptron algorithm and the seventh is a support vector machine (SVM).
The details of these models is described in Section 1.

In order to evaluate the performances of these models, you will use several different datasets.
The first two datasets are synthetic datasets that have features and labels that were programatically
generated. They were generated using the same script but use different input parameters that produced
sparse and dense variants. The second two datasets are for the task of named-entity recognition (NER),
identifying the names of people, locations, and organizations within text.
One comes from news text and the other from a corpus of emails.
For these two datasets, you need to implement the feature extraction yourself.
All of the datasets and feature extraction information are described in Section 2.

Finally, you will run two sets of experiments, one on the synthetic data and one on the NER data.
The first set will analyze how the amount of training data impacts model performance.
The second will look at the consequences of having training and testing data that come from different domains.
The details of the experiments are described in Section 3.

### Distribution of Points

The homework has 4 sections for a total of 100 points + 10 extra credit points:
- Section 0: Warmup (5 points)
- Section 1: Linear Classifiers (30 points)
- Section 2: Datasets (0 points, just text)
- Section 3: Experiments (65 points)
    - Synthetic Experiment:
        - Parameter Tuning (10 points)
        - Learning Curves(10 points)
        - Final Test Accuracies (5 points)
        - Discussion Questions (5 points)
        - Noise Experiment (10 points **extra credit**)
    - NER Experiment:
        - Feature Extraction (25 points)
        - Final Test Accuracies (5 points)
        - $F_1$ Discussion Questions (5 points)

# Section 0: Warmup

###### Only For Colab

If you want to complete this homework in Colab, you are more than welcome to.
You will need a little bit more maneuvering since you will need to upload
the files of hw2 to your Google Drive and run the following two cells:

```python
# Uncomment if you want to use Colab for this homework.
# from google.colab import drive
# drive.mount('/content/drive', force_remount=True)
```

    Mounted at /content/drive

```python
# Uncomment if you want to use Colab for this homework.
# %cd /content/drive/My Drive/Colab Notebooks/YOUR_PATH_TO_HW_FOLDER
```

    /content/drive/My Drive/Colab Notebooks

###### Python Version

Python 3.6 or above is required for this homework. Make sure you have it installed.

```python
# Let's check.
import sys
if sys.version_info[:2] < (3, 6):
    raise Exception("You have Python version " + str(sys.version_info))
```

## Imports and Helper Functions (5 points total)

Let's import useful modules we will need throughout the homework
as well as implement helper functions for our experiment. **Read and remember** what each function is doing, as you will probably need some of them down the line.

```python
# Install necessary libraries for this homework. only need to run once i think
%pip install sklearn
%pip install matplotlib
%pip install numpy
```

    Requirement already satisfied: sklearn in c:\users\tongd\anaconda3\lib\site-packages (0.0)
    Requirement already satisfied: scikit-learn in c:\users\tongd\anaconda3\lib\site-packages (from sklearn) (0.23.1)
    Requirement already satisfied: joblib>=0.11 in c:\users\tongd\anaconda3\lib\site-packages (from scikit-learn->sklearn) (0.16.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\tongd\anaconda3\lib\site-packages (from scikit-learn->sklearn) (2.1.0)
    Requirement already satisfied: scipy>=0.19.1 in c:\users\tongd\anaconda3\lib\site-packages (from scikit-learn->sklearn) (1.5.0)
    Requirement already satisfied: numpy>=1.13.3 in c:\users\tongd\anaconda3\lib\site-packages (from scikit-learn->sklearn) (1.18.5)
    Note: you may need to restart the kernel to use updated packages.
    Requirement already satisfied: matplotlib in c:\users\tongd\anaconda3\lib\site-packages (3.2.2)
    Requirement already satisfied: python-dateutil>=2.1 in c:\users\tongd\anaconda3\lib\site-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: numpy>=1.11 in c:\users\tongd\anaconda3\lib\site-packages (from matplotlib) (1.18.5)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\tongd\anaconda3\lib\site-packages (from matplotlib) (1.2.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\tongd\anaconda3\lib\site-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in c:\users\tongd\anaconda3\lib\site-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: six>=1.5 in c:\users\tongd\anaconda3\lib\site-packages (from python-dateutil>=2.1->matplotlib) (1.15.0)
    Note: you may need to restart the kernel to use updated packages.
    Requirement already satisfied: numpy in c:\users\tongd\anaconda3\lib\site-packages (1.18.5)
    Note: you may need to restart the kernel to use updated packages.

```python
import json
import os

import numpy as np
import matplotlib.pylab as plt
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import accuracy_score

DATASETS_PATH = "datasets/"
NER_PATH = os.path.join(DATASETS_PATH, 'ner')
SYNTHETIC_PATH = os.path.join(DATASETS_PATH, 'synthetic')
```

```python
"""
Helper function that loads a synthetic dataset from the dataset root (e.g. "synthetic/sparse").

You should not need to edit this method.
"""
def load_synthetic_data(dataset_type):

    def load_jsonl(file_path):
        data = []
        with open(file_path, 'r') as f:
            for line in f:
                data.append(json.loads(line))
        return data

    def load_txt(file_path):
        data = []
        with open(file_path, 'r') as f:
            for line in f:
                data.append(int(line.strip()))
        return data

    def convert_to_sparse(X):
        sparse = []
        for x in X:
            data = {}
            for i, value in enumerate(x):
                if value != 0:
                    data[str(i)] = value
            sparse.append(data)
        return sparse

    path = os.path.join(SYNTHETIC_PATH, dataset_type)

    X_train = load_jsonl(os.path.join(path, 'train.X'))
    X_dev = load_jsonl(os.path.join(path, 'dev.X'))
    X_test = load_jsonl(os.path.join(path, 'test.X'))

    num_features = len(X_train[0])
    features = [str(i) for i in range(num_features)]

    X_train = convert_to_sparse(X_train)
    X_dev = convert_to_sparse(X_dev)
    X_test = convert_to_sparse(X_test)

    y_train = load_txt(os.path.join(path, 'train.y'))
    y_dev = load_txt(os.path.join(path, 'dev.y'))
    y_test = load_txt(os.path.join(path, 'test.y'))

    return X_train, y_train, X_dev, y_dev, X_test, y_test, features
```

```python
"""
Helper function that loads the NER data from a path (e.g. "ner/conll/train").

You should not need to edit this method.
"""
def load_ner_data(dataset=None, dataset_type=None):
    # List of tuples for each sentence
    data = []
    path = os.path.join(os.path.join(NER_PATH, dataset), dataset_type)
    for filename in os.listdir(path):
        with open(os.path.join(path, filename), 'r') as file:
            sentence = []
            for line in file:
                if line == '\n':
                    data.append(sentence)
                    sentence = []
                else:
                    sentence.append(tuple(line.split()))
    return data
```

```python
"""
A helper function that plots the relationship between number of examples
and accuracies for all the models.

You should not need to edit this method.
"""
def plot_learning_curves(
    perceptron_accs,
    winnow_accs,
    adagrad_accs,
    avg_perceptron_accs,
    avg_winnow_accs,
    avg_adagrad_accs,
    svm_accs
):
    """
    This function will plot the learning curve for the 7 different models.
    Pass the accuracies as lists of length 11 where each item corresponds
    to a point on the learning curve.
    """

    accuracies = [
        ('perceptron', perceptron_accs),
        ('winnow', winnow_accs),
        ('adagrad', adagrad_accs),
        ('avg-perceptron', avg_perceptron_accs),
        ('avg-winnow', avg_winnow_accs),
        ('avg-adagrad', avg_adagrad_accs),
        ('svm', svm_accs)
    ]

    x = [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 10000]
    plt.figure()
    f, (ax, ax2) = plt.subplots(1, 2, sharey=True, facecolor='w')

    for label, acc_list in accuracies:
        assert len(acc_list) == 11
        ax.plot(x, acc_list, label=label)
        ax2.plot(x, acc_list, label=label)

    ax.set_xlim(0, 5500)
    ax2.set_xlim(9500, 10000)
    ax2.set_xticks([10000])
    # hide the spines between ax and ax2
    ax.spines['right'].set_visible(False)
    ax2.spines['left'].set_visible(False)
    ax.yaxis.tick_left()
    ax.tick_params(labelright='off')
    ax2.yaxis.tick_right()
    ax2.legend()
    plt.show()
```

### F1 Score (5 points)

For some part of the homework, you will use the F1 score instead of accuracy to evaluate how well a model does. The F1 score is computed as the harmonic mean of the precision and recall of the classifier. Precision measures the number of correctly identified positive results by the total number of positive results. Recall, on the other hand, measures the number of correctly identified positive results divided by the number of all samples that should have been identified as positive. More formally, we have that

$$
\begin{align}
\text{Precision} &= \frac{TP}{TP + FP} \\
\text{Recall} &= \frac{TP}{TP + FN}
\end{align}
$$

where $TP$ is the number of true positives, $FP$ false positives and $FN$ false negatives. Combining these two we define F1 as

$$
\text{F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
$$

You now need to implement the calculation of F1 yourself using the provided function header. It will be unit tested on Gradescope.

```python
def calculate_f1(y_gold, y_model):
    """
    TODO: MODIFY

    Computes the F1 of the model predictions using the
    gold labels. Each of y_gold and y_model are lists with
    labels 1 or -1. The function should return the F1
    score as a number between 0 and 1.
    """
    import numpy as np

    diff = np.zeros(len(y_gold))
    for i in range(len(y_gold)):
        diff[i] = y_gold[i] - y_model[i]
    TP = 0
    FP = 0
    FN = 0

    for i in range(len(diff)):
        if diff[i] == 0 and y_gold[i] == 1:
            TP += 1
        if diff[i] > 0:
            FN += 1
        if diff[i] < 0:
            FP += 1

    precision = TP / (TP + FP)
    recall = TP / (TP + FN)
    F1 = 2 * precision * recall / (precision + recall)
    print('TP is',TP)
    print('FP is',FP)
    print('FN is',FN)
    return F1
```

Looking at the formula for the F1 score, what is the highest and lowest possible value?

```python
def highest_and_lowest_f1_score():
    """
    TODO: MODIFY

    Return the highest and lowest possible F1 score
    (ie one line solution returning the theoretical max and min)
    """

    maxscore = 1
    minscore = 0

    return maxscore, minscore
```

# Section 1. Linear Classifiers (30 points total)

This section details the seven different algorithms that you will use in the experiments. For each of the algorithms, we describe the initialization you should use to start training and the different parameter settings that you should use for the experiment on the synthetic datasets. Each of the update functions for the Perceptron, Winnow, and Perceptron with AdaGrad will be unittested on Gradescope, so please do not edit the function definitions.

### 1.1 Base Classifier

```python
class Classifier(object):
    """
    DO NOT MODIFY

    The Classifier class is the base class for all of the Perceptron-based
    algorithms. Your class should override the "process_example" and
    "predict_single" functions. Further, the averaged models should
    override the "finalize" method, where the final parameter values
    should be calculated.

    You should not need to edit this class any further.
    """

    ITERATIONS = 10

    def train(self, X, y):
        for iteration in range(self.ITERATIONS):
            for x_i, y_i in zip(X, y):
                self.process_example(x_i, y_i)
        self.finalize()

    def process_example(self, x, y):
        """
        Makes a prediction using the current parameter values for
        the features x and potentially updates the parameters based
        on the gradient. Note "x" is a dictionary which maps from the feature
        name to the feature value and y is either 1 or -1.
        """
        raise NotImplementedError

    def finalize(self):
        """Calculates the final parameter values for the averaged models."""
        pass

    def predict(self, X):
        """
        Predicts labels for all of the input examples. You should not need
        to override this method.
        """
        return [self.predict_single(x) for x in X]

    def predict_single(self, x):
        """
        Predicts a label, 1 or -1, for the input example. "x" is a dictionary
        which maps from the feature name to the feature value.
        """
        raise NotImplementedError
```

### 1.2 Basic Perceptron (2 points)

We do this classifier for you, so enjoy the two free points and pay attention to the techniques and code written.

#### 1.2.1 Description

This is the basic version of the Perceptron Algorithm.
    In this version, an update will be performed on the example $(\textbf{x}, y)$ if $y(\mathbf{w}^\intercal \mathbf{x} + \theta) \leq 0$. The Perceptron needs to learn both the bias term $\theta$ and the weight vector $\mathbf{w}$ parameters.
When the Perceptron makes a mistake on the example $(\textbf{x}, y)$, both $\mathbf{w}$ and $\theta$ need to be updated using the following update equations:
$$
\begin{align*}
    \mathbf{w}^\textrm{new} &\gets \mathbf{w} + \eta \cdot y \cdot \mathbf{x} \\
    \theta^\textrm{new} &\gets \theta + \eta \cdot y
\end{align*}
$$
where $\eta$ is the learning rate.

#### 1.2.2 Hyperparameters

We set $\eta$ to 1, so there are no hyperparameters to tune.

Note: If we assume that the order of the examples presented to the algorithm is fixed, we initialize $\mathbf{w} = \mathbf{0}$ and $\theta = 0$, and train both together, then the learning rate $\eta$ does not have any effect.
        In fact you can show that, if $\mathbf{w}_1$ and $\theta_1$ are the outputs of the Perceptron algorithm with learning rate $\eta_1$, then $\mathbf{w}_1/\eta_1$ and $\theta_1/\eta_1$ will be the result of the Perceptron with learning rate 1 (note that these two hyperplanes give identical predictions).

#### 1.2.3 Initialization

$\mathbf{w} = [0, 0, \dots, 0]$ and $\theta = 0$.

```python
class Perceptron(Classifier):
    """
    DO NOT MODIFY THIS CELL

    The Perceptron model. Note how we are subclassing `Classifier`.
    """

    def __init__(self, features):
        """
        Initializes the parameters for the Perceptron model. "features"
        is a list of all of the features of the model where each is
        represented by a string.
        """

        # NOTE: Do not change the names of these 3 data members because
        # they are used in the unit tests
        self.eta = 1
        self.theta = 0
        self.w = {feature: 0.0 for feature in features}

    def process_example(self, x, y):
        y_pred = self.predict_single(x)
        if y != y_pred:
            for feature, value in x.items():
                self.w[feature] += self.eta * y * value
            self.theta += self.eta * y

    def predict_single(self, x):
        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta
        if score <= 0:
            return -1
        return 1
```

For the rest of the Perceptron-based algorithms, you will have to implement the corresponding class like we have done for `Perceptron`.
Use the `Perceptron` class as a guide for how to implement the functions.

### 1.3 Winnow (5 points)

#### 1.3.1 Description
The Winnow algorithm is a variant of the Perceptron algorithm with multiplicative updates. Since the algorithm requires that the target function is monotonic, you will only use it on the synthetic datasets.

The Winnow algorithm only learns parameters $\mathbf{w}$.
We will fix $\theta = -n$, where $n$ is the number of features.
When the Winnow algorithm makes a mistake on the example $(\textbf{x}, y)$, the parameters are updated with the following equation:
$$
\begin{equation}
    w^\textrm{new}_i \gets w_i \cdot \alpha^{y \cdot x_i}
\end{equation}
$$
where $w_i$ and $x_i$ are the $i$th components of the corresponding vectors.
Here, $\alpha$ is a promotion/demotion hyperparameter.

#### 1.3.2 Hyperparameters

For the experiment, choose $\alpha \in \{1.1, 1.01, 1.005, 1.0005, 1.0001\}$.

#### 1.3.3 Initialization

$\mathbf{w} = [1, 1, \dots, 1]$ and $\theta = -n$ (constant).

```python
class Winnow(Classifier):
    def __init__(self, alpha, features):
        # DO NOT change the names of these 3 data members because
        # they are used in the unit tests
        self.alpha = alpha
        self.w = {feature: 1.0 for feature in features}
        self.theta = -len(features)

    def process_example(self, x, y):
        """ TODO: IMPLEMENT"""

        y_pred = self.predict_single(x)
        if y != y_pred:
            for feature, value in x.items():
                self.w[feature] = self.w[feature] * pow(self.alpha,y*value)

    def predict_single(self, x):
        """ TODO: IMPLEMENT"""

        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta

        if score <= 0:
            return -1
        return 1

```

### 1.4 AdaGrad (10 points)

#### 1.4.1 Description
AdaGrad is a variant of the Perceptron algorithm that adapts the learning rate for each parameter based on historical information.
    The idea is that frequently changing features get smaller learning rates and stable features higher ones.

To derive the update equations for this model, we first need to start with the loss function.
Instead of using the hinge loss with the elbow at 0 (like the basic Perceptron does), we will instead use the standard hinge loss with the elbow at 1:

$$
\begin{equation}
    \mathcal{L}(\mathbf{x}, y, \mathbf{w}, \theta) = \max\left\{0, 1 - y(\mathbf{w}^\intercal \mathbf{x} + \theta)\right\}
\end{equation}
$$

Then, by taking the partial derivative of $\mathcal{L}$ with respect to $\mathbf{w}$ and $\theta$, we can derive the respective graidents (make sure you understand how you could derive these gradients on your own):

$$
\begin{align}
    \frac{\partial\mathcal{L}}{\partial\mathbf{w}} &=
        \begin{cases}
        \mathbf{0} & \text{if $y(\mathbf{w}^\intercal \mathbf{x} + \theta) > 1$} \\
        -y\cdot \mathbf{x} & \textrm{otherwise}
        \end{cases} \\
    \frac{\partial\mathcal{L}}{\partial\theta} &=
        \begin{cases}
            0 & \text{if $y(\mathbf{w}^\intercal \mathbf{x} + \theta) > 1$} \\
            -y & \textrm{otherwise}
        \end{cases}
\end{align}
$$

Then for each parameter, we will keep track of the sum of the parameters' squared gradients.
In the following equations, the $k$ superscript refers to the $k$th non-zero gradient (i.e., the $k$th weight vector/misclassified example) and $t$ is the number of mistakes seen thus far.

$$
\begin{align}
    G^t_j &= \sum_{k=1}^t \left(\frac{\partial \mathcal{L}}{\partial w^k_j}\right)^2 \\
    H^t &= \sum_{k=1}^t \left(\frac{\partial \mathcal{L}}{\partial \theta^k}\right)^2
\end{align}
$$

For example, on the 3rd mistake ($t = 3$), $G^3_j$ is the sum of the squares of the first three non-zero gradients ($\left(\frac{\partial \mathcal{L}}{\partial w^1_j}\right)^2$, $\left(\frac{\partial \mathcal{L}}{\partial w^2_j}\right)^2$, and $\left(\frac{\partial \mathcal{L}}{\partial w^3_j}\right)^2$).
Then $\mathbf{G}^3$ is used to calculate the 4th value of the weight vector as follows.
On example $(\mathbf{x}, y)$, if $y(\mathbf{w}^\intercal \mathbf{x} + \theta) \leq 1$, then the parameters are updated with the following equations:

$$
\begin{align}
    \mathbf{w}^{t+1} &\gets \mathbf{w}^t + \eta \cdot \frac{y \cdot \mathbf{x}}{\sqrt{\mathbf{G}^t}} \\
    \theta^{t+1} &\gets \theta^t + \eta \frac{y}{\sqrt{H^t}}
\end{align}
$$

Note that, although we use the hinge loss with the elbow at 1 for training, you still make the prediction based on whether or not $y(\mathbf{w}^\intercal \mathbf{x} + \theta) \leq 0$ during testing.

#### 1.4.2 Hyperparameters

For the experiment, choose $\eta \in \{1.5, 0.25, 0.03, 0.005, 0.001\}$

#### 1.4.3 Initialization

$\mathbf{w} = [0, 0, \dots, 0]$ and $\theta = 0$.

```python
class AdaGrad(Classifier):
    def __init__(self, eta, features):
        # DO NOT change the names of these 3 data members because
        # they are used in the unit tests
        self.eta = eta
        self.w = {feature: 0.0 for feature in features}
        self.theta = 0
        self.G = {feature: 1e-5 for feature in features}  # 1e-5 prevents divide by 0 problems
        self.H = 0

    def process_example(self, x, y):
        """ TODO: IMPLEMENT"""
        import numpy as np

        y_pred = self.predict_single(x)
        dotpro = 0

        if y != y_pred:

#             calculate dot product
            for feature, value in x.items():
                dotpro += self.w[feature] * value

#             update w
            for feature, value in x.items():
                if(y * (dotpro + self.theta) > 1):
                    dldw = 0
                else:
                    dldw = -y * value
                self.G[feature] += dldw ** 2
                self.w[feature] += self.eta * y * value / np.sqrt(self.G[feature])

#             update theta
            if(y * (dotpro + self.theta) > 1):
                dldtheta = 0
            else:
                dldtheta = -y
            self.H += dldtheta ** 2
            self.theta += self.eta * y / np.sqrt(self.H)

    def predict_single(self, x):
        """ TODO: IMPLEMENT"""

        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta

        if score <= 0:
            return -1
        return 1

```

### 1.5 Averaged Models (15 points)

You will also implement the averaged version of the previous three algorithms.

During the course of training, each of the above algorithms will have $K + 1$ different parameter settings for the $K$ different updates it will make during training.
The regular implementation of these algorithms uses the parameter values after the $K$th update as the final ones.
Instead, the averaged version use the weighted average of the $K + 1$ parameter values as the final parameter values.
Let $m_k$ denote the number of correctly classified examples by the $k$th parameter values and $M$ the total number of correctly classified examples.
The final parameter values are

$$
\begin{align}
    M &= \sum_{k=1}^{K+1} m_k \\
    \mathbf{w} &\gets \frac{1}{M} \sum_{k=1}^{K+1} m_k \cdot \mathbf{w}^k \\
    \theta &\gets \frac{1}{M} \sum_{k=1}^{K+1} m_k \cdot \theta^k \\
\end{align}
$$

For each of the averaged versions of Perceptron, Winnow, and AdaGrad, use the same hyperparameters and initialization as before.

#### 1.5.1 Implementation Note
Implementing the averaged variants of these algorithms can be tricky.
    While the final parameter values are based on the sum of $K$ different vectors, there is no need to maintain *all* of these parameters.
    Instead, you should implement these algorithms by keeping only two vectors, one which maintains the cumulative sum and the current one.

Additionally, there are two ways of keeping track of these two vectors.
One is more straightforward but prohibitively slow.
The second requires some algebra to derive but is significantly faster to run.
Try to analyze how the final weight vector is a function of the intermediate updates and their corresponding weights.
It should take less than a minute or two for ten iterations for any of the averaged algorithms.
**You need to think about how to efficiently implement the averaged algorithms yourself.**

Further, the implementation for Winnow is slightly more complicated than the other two, so if you consistently have low accuracy for the averaged Winnow, take a closer look at the derivation.

```python
class AveragedPerceptron(Classifier):
    def __init__(self, features):
        self.eta = 1
        self.w = {feature: 0.0 for feature in features}
        self.theta = 0
        """TODO: You will need to add data members here"""
        self.M = 0
        self.lastM = 0
        self.weightedW = {feature: 0.0 for feature in features}
        self.weightedT = 0
        self.Mk = []
        self.yvec = []
        self.xvec = []

    def process_example(self, x, y):
        """ TODO: IMPLEMENT"""
        y_pred = self.predict_single(x)
        if y != y_pred:

            self.weightedT += (self.M - self.lastM) * self.theta

#             append variables for weighted W calculation
            self.xvec.append(x)
            self.yvec.append(y)
            self.Mk.append(self.M - self.lastM)

#             update weights
            for feature, value in x.items():
                self.w[feature] += self.eta * y * value
            self.theta += self.eta * y

#             reset lastM
            self.lastM = self.M

        else:
            self.M += 1

    def predict_single(self, x):
        """ TODO: IMPLEMENT"""

        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta
        if score <= 0:
            return -1
        return 1

    def finalize(self):
        """ TODO: IMPLEMENT"""

        self.M +=1
        self.Msum = self.M
        self.w = {feature: 0.0 for feature in features}

#         sum up rebuild weight vectors from saved x,y,mk values by just summing
#         1/M * (m1+m2+...mk)*y*x and (mi) terms reduces m[i] every iteration
        for i in range(len(self.yvec)):
            for feature, value in self.xvec[i].items():
                self.weightedW[feature] += 1/self.M * self.Msum * \
                                            self.eta * self.yvec[i] * value
#                 self.w[feature] += 1/self.M * self.Msum * self.eta * self.yvec[i] * value
            self.Msum -= self.Mk[i]
        self.theta = 1/self.M * self.weightedT

        self.w = self.weightedW
```

```python
# test the averaged perceptron model

X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')

AvgPerceptron = AveragedPerceptron(features)
iteration = 10

for iter in range(iteration):
    for i in range(len(X_train)):
        AvgPerceptron.process_example(X_train[i],y_train[i])
AvgPerceptron.finalize()

count = 0
for i in range(len(X_dev)):
    x = X_dev[i]
    y = AvgPerceptron.predict_single(x)
    if y_dev[i] == y:
        count += 1
print(count/len(X_dev))
```

    0.9445

```python
class AveragedWinnow(Classifier):
    def __init__(self, alpha, features):
        self.alpha = alpha
        self.w = {feature: 1.0 for feature in features}
        self.theta = -len(features)
        """TODO: You will need to add data members here"""
        self.M = 0
        self.lastM = 0
        self.weightedW = {feature: 1.0 for feature in features}

    def process_example(self, x, y):
        """ TODO: IMPLEMENT"""

        y_pred = self.predict_single(x)
        if y != y_pred:
            for feature, value in x.items():
                self.w[feature] = self.w[feature] * pow(self.alpha,y*value)

            for feature in self.weightedW:
                self.weightedW[feature] += (self.M - self.lastM) * self.w[feature]

#             self.M += 1
            self.lastM = self.M
        else:
            self.M +=1

    def predict_single(self, x):
        """ TODO: IMPLEMENT"""
        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta

        if score <= 0:
            return -1
        return 1

    def finalize(self):
        """ TODO: IMPLEMENT"""
        self.M += 1
        for feature in self.weightedW:
            self.w[feature] = 1/self.M * self.weightedW[feature]

```

```python
# test the averaged Winnow model

X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('sparse')

iteration = 10
alpha = 1.1
AvgWinnow = AveragedWinnow(alpha,features)
AvgWinnow.train(X_train,y_train)

# AvgWinnow = Winnow(alpha,features)
# AvgWinnow.train(X_train,y_train)

count = 0
for i in range(len(X_dev)):
    x = X_dev[i]
    y = AvgWinnow.predict_single(x)
    if y_dev[i] == y:
        count += 1
print(count/len(X_dev))
```

    0.939

```python
class AveragedAdaGrad(Classifier):
    def __init__(self, eta, features):
        self.eta = eta
        self.w = {feature: 0.0 for feature in features}
        self.theta = 0
        self.G = {feature: 1e-5 for feature in features}
        self.H = 0
        """TODO: You will need to add data members here"""
        self.M = 0
        self.lastM = 0
        self.weightedW = {feature: 1.0 for feature in features}
        self.weightedT = 0

    def process_example(self, x, y):
        """ TODO: IMPLEMENT"""
        import numpy as np

        y_pred = self.predict_single(x)
        dotpro = 0

        if y != y_pred:

            for feature in self.weightedW:
                self.weightedW[feature] += (self.M - self.lastM) * self.w[feature]
            self.weightedT += (self.M - self.lastM) * self.theta

#             append variables for weighted W calculation
            self.xvec.append(x)
            self.yvec.append(y)
            self.Mk.append(self.M - self.lastM)

#             calculate dot product
            for feature, value in x.items():
                dotpro += self.w[feature] * value

#             update w
            for feature, value in x.items():
                if(y * (dotpro + self.theta) > 1):
                    dldw = 0
                else:
                    dldw = -y * value
                self.G[feature] += dldw ** 2
                self.w[feature] += self.eta * y * value / np.sqrt(self.G[feature])

#             update theta
            if(y * (dotpro + self.theta) > 1):
                dldtheta = 0
            else:
                dldtheta = -y
            self.H += dldtheta ** 2
            self.theta += self.eta * y / np.sqrt(self.H)

            self.lastM = self.M

        else:
            self.M += 1

    def predict_single(self, x):
        """ TODO: IMPLEMENT"""

        score = 0
        for feature, value in x.items():
            score += self.w[feature] * value
        score += self.theta

        if score <= 0:
            return -1
        return 1

    def finalize(self):
        """ TODO: IMPLEMENT"""
        self.M += 1
        for feature in self.weightedW:
            self.w[feature] = 1/self.M * self.weightedW[feature]
        self.theta = 1/self.M * self.weightedT
```

```python
# test the averaged Adagrad model

X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')

# 𝜂∈{1.5,0.25,0.03,0.005,0.001}

eta = 1.5
AvgAdaGrad = AveragedAdaGrad(eta,features)
AvgAdaGrad.train(X_train,y_train)
# OrigAdaGrad = AdaGrad(eta,features)
# OrigAdaGrad.train(X_train,y_train)

count = 0
for i in range(len(X_dev)):
    x = X_dev[i]
    y = AvgAdaGrad.predict_single(x)
    if y_dev[i] == y:
        count += 1
print(count/len(X_dev))
```

    0.9445

### 1.6 Support Vector Machines

Although we have not yet covered SVMs in class, you can still train them using the `sklearn` library.
We will use a soft margin SVM for non-linearly separable data.
You should use the `sklearn` implementation as follows:
```
from sklearn.svm import LinearSVC
classifier = LinearSVC(loss='hinge')
classifier.fit(X, y)
```

`sklearn` requires a different feature representation than what we use for the Perceptron models.
The provided Python template code demonstrates how to convert to the require representation.

Given training samples $S = \{(\mathbf{x}^1, y^1), (\mathbf{x}^2, y^2), \dots, (\mathbf{x}^m, y^m)\}$, the objective for the SVM is the following:

$$
\begin{equation}
    \min_{\mathbf{w}, b, \boldsymbol{\xi}} \frac{1}{2} \vert\vert \mathbf{w}\vert\vert^2_2 + C \sum_{i=1}^m \xi_i
\end{equation}
$$

subject to the following constraints:

$$
\begin{align}
    y^i(\mathbf{w}^\intercal \mathbf{x}^i + b) \geq 1 - \xi_i \;\;&\textrm{for } i = 1, 2, \dots, m \\
    \xi_i \geq 0 \;\;& \textrm{for } i = 1, 2, \dots, m
\end{align}
$$

```python
class SVMClassifier(Classifier):

    def __init__(self):
        from sklearn.svm import LinearSVC
        self.local_classifier = LinearSVC(loss = 'hinge')
        self.vectorizer = DictVectorizer()

    def trainSVM(self,X_train,y_train):
        X_train_dict = self.vectorizer.fit_transform(X_train)
        self.local_classifier.fit(X_train_dict,y_train)

    def testSVM(self,X_test,y_test):
        X_test_dict = self.vectorizer.transform(X_test)
        return self.local_classifier.score(X_test_dict,y_test)
```

```python
"""TODO: Create an SVM classifier"""
X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')

# This is how you convert from the way we represent features in the
# Perceptron code to how you need to represent features for the SVM.
# You can then train with (X_train_dict, y_train) and test with
# (X_conll_test_dict, y_conll_test) and (X_enron_test_dict, y_enron_test)
vectorizer = DictVectorizer()
X_train_dict = vectorizer.fit_transform(X_train)
X_test_dict = vectorizer.transform(X_test)

from sklearn.svm import LinearSVC
classifier = LinearSVC(loss = 'hinge')
classifier.fit(X_train_dict,y_train)
classifier.score(X_test_dict,y_test)
```

    0.9405

```python

```

# Section 2. Datasets

In this section, we describe the synthetic and NER datasets that you will use for your experiments.
For the NER datasets, there is also an explanation for the features which you need to extract from the data.

### 2.1 Synthetic Data

#### 2.1.1 Introduction

The synthetic datasets have features and labels which are automatically generated from a python script.
Each instance will have $n$ binary features and are labeled according to a $l$-of-$m$-of-$n$ Boolean function.
Specifically, there is a set of $m$ features such that an example if positive if and only if at least $l$ of these $m$ features are active.
The set of $m$ features is the same for the dataset (i.e., it is not a separate set of $m$ features for each individual instance).

We provide two versions of the synthetic dataset called sparse and dense.
For both datasets, we set $l = 10$ and $m=20$.
We set $n = 200$ for the sparse data and $n = 40$ for the dense data.
Additionally, we add noise to the data as follows:
With probability $0.05$ the label assigned by the function is changed and with probability $0.001$ each feature value is changed.
Consequently, the data is not linearly separable.

We have provided you with three data splits for both sparse and dense with 10,000 training, 2,000 development, and 2,000 testing examples.
Section 3 describes the experiments that you need to run on these datasets.

#### 2.1.2 Feature Representation

The features of the synthetic data provided are vectors of 0s and 1s.
Storing these large matrices requires lots of memory so we use a sparse representation that stores them as dictionaries instead.
For example, the vector $[0,1,0,0,0,1]$ can be stored as `{"x2": 1,"x6": 1}` (using 1-based indexing).
We have provided you with the code for parsing and converting the data to this format.
You can use these for the all algorithms you develop except the SVM.
Since you will be using the implementation of SVM from sklearn, you will need to provide a vector to it. You can use `sklearn.feature_extraction.DictVectorizer` for converting feature-value dictionaries to vectors.

### 2.2 NER Data

In addition to the synthetic data, we have provided you two datasets for the task of named-entity recognition (NER).
The goal is to identify whether strings in text represent names of people, organizations, or locations.
An example instance looks like the following:

```
    [PER Wolff] , currently a journalist in [LOC Argentina] , played with
    [PER del Bosque] in the final years of the seventies in
    [ORG Real Madrid] .
```

In this problem, we will simplify the task to identifying whether a string is named entity or not (that is, you don't have to say which type of entity it is).
For each token in the input, we will use the tag $\texttt{I}$ to denote that token is an entity and $\texttt{O}$ otherwise.
For example, the full tagging for the above instace is as follows:

```
    [I Wolff] [O ,] [O currently] [a] [O journalist] [O in] [I Argentina]
    [O ,] [O played] [O with] [I del] [I Bosque] [O in] [O the] [O final]
    [O years] [O of] [O the] [O seventies] [O in] [I Real] [I Madrid] .
```

Given a sentence $S = w_1, w_2, \dots, w_n$, you need to predict the `I`, `O` tag for each word in the sentence.
That is, you will produce the sequence $Y = y_1, y_2, \dots, y_n$ where $y_i \in$ {`I`, `O`}.

#### 2.2.1 Datasets: CoNLL and Enron

We have provided two datasets, the CoNLL dataset which is text from news articles, and Enron, a corpus of emails.
The files contain one word and one tag per line.
For CoNLL, there are training, development, and testing files, whereas Enron only has a test dataset.
There are 14,987 training sentences (204,567 words), 336 development sentences (3,779 words), and 303 testing sentences (3,880 words) in CoNLL.
For Enron there are 368 sentences (11,852 words).

**Please note that the CoNLL dataset is available only for the purposes of this assignment.
It is copyrighted, and you are granted access because you are a Penn student, but please delete it when you are done with the homework.**

#### 2.2.2 Feature Extraction

The NER data is provided as raw text, and you are required to extract features for the classifier.
In this assignment, we will only consider binary features based on the context of the word that is supposed to be tagged.

Assume that there are $V$ unique words in the dataset and each word has been assigned a unique ID which is a number $\{1, 2, \dots, V\}$.
Further, $w_{-k}$ and $w_{+k}$ indicate the $k$th word before and after the target word.
The feature templates that you should use to generate features are as follows:

| Template             | Number of Features |
|----------------------|--------------------|
| $w_{-3}$             | $V$                |
| $w_{-2}$             | $V$                |
| $w_{-1}$             | $V$                |
| $w_{+1}$             | $V$                |
| $w_{+2}$             | $V$                |
| $w_{+3}$             | $V$                |
| $w_{-1}$ & $w_{-2}$  | $V \times V$       |
| $w_{+1}$ \& $w_{+2}$ | $V \times V$       |
| $w_{-1}$ \& $w_{+1}$ | $V \times V$       |

Each feature template corresponds to a set of features that you will compute (similar to the features you generated in problem 2 from the first homework assignment).
The $w_{-3}$ feature template corresponds to $V$ features where the $i$th feature is 1 if the third word to the left of the target word has ID $i$.
The $w_{-1} \& w_{+1}$ feature template corresponds to $V \times V$ features where there is one feature for every unique pair words.
For example, feature $(i - 1) \times V + j$ is a binary feature that is 1 if the word 1 to the left of the target has ID $i$ and the first word to the right of the target has ID $j$.
In practice, you will not need to keep track of the feature IDs.
Instead, each feature will be given a name such as "$w_{-1}=\textrm{the} \& w_{+1}=\textrm{cat}$".

In total, all of the above feature templates correspond to a very large number of features.
However, for each word, there will be exactly 9 features which are active (non-zero), so the feature vector is quite sparse.
You will represent this as a dictionary which maps from the feature name to the value.
In the provided Python template, we have implemented a couple of the features for you to demonstrate how to compute them and what the naming scheme should look like.

In order to deal with the first two words and the last two words in a sentence, we will add special symbol "SSS" and "EEE" to the vocabulary to represent the words before the first word and the words after the last word.
Notice that in the test data you may encounter a word that was not observed in training, and therefore is not in your dictionary.
In this case, you cannot generate a feature for it, resulting in less than 7 active features in some of the test examples.

# Section 3. Experiments (65 points total)

You will run two sets of experiments, one using the synthetic data and one using the NER data.

### 3.1 Synthetic Experiment (30 + 10 extra credit points)

This experiment will explore the impact that the amount of training data has on model performance.
First, you will do hyperparameter tuning for Winnow and Perceptron with AdaGrad (both standard and averaged versions).
Then you will generate learning curves that will plot the size of the training data against the performance.
Finally, for each of the models trained on all of the training data, you will find the test score.
You should use accuracy to compute the performance of the model.

In summary, the experiment consists of three parts
1. Parameter Tuning
2. Learning Curves
3. Final Evaluation

#### 3.1.1 Parameter Tuning (10 points)

For both the Winnow and Perceptron with AdaGrad (standard and averaged), there are hyperparameters that you need to choose.
(The same is true for SVM, but you should only use the default settings.)
Similarly to cross-validation from Homework 1, we will estimate how well each model will do on the true test data using the development dataset (we will not run cross-validation), and choose the hyperparameter settings based on these results.

For each hyperparameter value in Section 1, train a model using that value on the training data and compute the accuracy on the development dataset. Each model should be trained for 10 iterations (i.e., 10 passes over the entire dataset).

TODO: Fill in the table with the best hyperparameter values and the corresponding validation accuracies.
Repeat this for both the sparse and dense data.

# Winnow Sweep

| $\alpha$ | Sparse | Dense |
|----------|--------|-------|
| 1.1      | 0.8935 | 0.8995 |
| 1.01     | 0.9270    | 0.9215     |
| 1.005    | 0.9195    | 0.9080   |
| 1.0005   | 0.5630     | 0.8615     |
| 1.0001   | 0.5205      | 0.6140      |

##### Averaged Winnow Sweep

| $\alpha$ | Sparse | Dense |
|----------|--------|-------|
| 1.1      |  0.9390      | 0.9445      |
| 1.01     |  0.8980      | 0.9335      |
| 1.005    |  0.8405      | 0.9150      |
| 1.0005   |  0.5255      | 0.6700      |
| 1.0001   |  0.5095      | 0.5460      |

##### AdaGrad Sweep

| $\eta$   | Sparse | Dense |
|----------|--------|-------|
| 1.5      |  0.8745      | 0.9325      |
| 0.25     |  0.8745      | 0.9325      |
| 0.03     |  0.8745      | 0.9325      |
| 0.005    |  0.8745      | 0.9325      |
| 0.001    |  0.8745      | 0.9325      |

##### Averaged AdaGrad Sweep

| $\eta$   | Sparse | Dense |
|----------|--------|-------|
| 1.5      | 0.8935       | 0.9445      |
| 0.25     | 0.8935       | 0.9445      |
| 0.03     | 0.8935       | 0.9445      |
| 0.005    | 0.8935       | 0.9445      |
| 0.001    | 0.8935       | 0.9445      |

#### 3.1.2 Learning Curves (10 points)

Next, you will train all 7 models with different amounts of training data.
For Winnow and Perceptron with AdaGrad (standard and averaged), use the best hyperparameters from the parameter tuning experiment.

Each of the datasets contains 10,000 training examples.
You will train each model 11 times on varying amounts of training data.
The first 10 will increase by 500 examples: 500, 1k, 1.5k, 2k, ..., 5k.
The 11th model should use all 10k examples.
Each Perceptron-based model should be trained for 10 iterations (e.g., 10 passes over the total number of training examples available to that model).
The SVM can be run until convergence with the default parameters.

For each model, compute the accuracy on the development dataset and plot the results using the provided code.
There should be a separate plot for the sparse and dense data.

**Note** how we have included an image in markdown. You should do the same for both plots and include them in the output below by running your experiment, saving your plots to the images folder, and linking it to this cell.

##### Sparse Plot

##### Dense Plot

```python
# set up the learning curve variables and load the dataset

perceptron_accs = np.zeros(11)
winnow_accs = np.zeros(11)
adagrad_accs = np.zeros(11)
avg_perceptron_accs = np.zeros(11)
avg_winnow_accs = np.zeros(11)
avg_adagrad_accs = np.zeros(11)
svm_accs = np.zeros(11)

X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('sparse')

```

```python
# basic perceptron
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    eta = 1.5
    model = Perceptron(features)
    model.train(X_train_i,y_train_i)
    perceptron_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
# averaged perceptron
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    model = AveragedPerceptron(features)
    model.train(X_train_i,y_train_i)
    avg_perceptron_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
# winnow
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    alpha = 1.01
    model = Winnow(alpha,features)
    model.train(X_train_i,y_train_i)
    winnow_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
# averaged winnow
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    alpha = 1.1
    model = AveragedWinnow(alpha,features)
    model.train(X_train_i,y_train_i)
    avg_winnow_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
# adagrad
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    eta = 1.5
    model = AdaGrad(eta,features)
    model.train(X_train_i,y_train_i)
    adagrad_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
# averaged AdaGrad
for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    eta = 1.5
    model = AveragedAdaGrad(eta,features)
    model.train(X_train_i,y_train_i)
    avg_adagrad_accs[i-1] = compute_accuracy_313(X_dev,y_dev,model)
```

```python
from sklearn.svm import LinearSVC

for i in range(1,12):
    X_train_i = X_train[0:(i*500)]
    y_train_i = y_train[0:(i*500)]
    if i == 11:
        X_train_i = X_train
        y_train_i = y_train
    vectorizer = DictVectorizer()
    X_train_dict = vectorizer.fit_transform(X_train_i)
    X_dev_dict = vectorizer.transform(X_dev)
    classifier = LinearSVC(loss = 'hinge')
    classifier.fit(X_train_dict,y_train_i)
    svm_accs[i-1] = classifier.score(X_dev_dict,y_dev)

```

    C:\Users\tongd\anaconda3\lib\site-packages\sklearn\svm\_base.py:976: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      warnings.warn("Liblinear failed to converge, increase "

```python

plot_learning_curves(perceptron_accs,
                        winnow_accs,
                        adagrad_accs,
                        avg_perceptron_accs,
                        avg_winnow_accs,
                        avg_adagrad_accs,
                        svm_accs
                        )
```

```python

```

```python
def compute_accuracy_313(X_dev,y_dev,classifier):
    count = 0
    for i in range(len(X_dev)):
        x = X_dev[i]
        y = classifier.predict_single(x)
        if y_dev[i] == y:
            count += 1
    return (count/len(X_dev))
```

#### 3.1.3 Final Evaluation (5 points)

Finally, for each of the 7 models, train the models on all of the training data and compute the test accuracy.
For Winnow and Perceptron with AdaGrad, use the best hyperparameter settings you found.
Report these accuracies in a table.

We will run our models with [500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 10000] examples.

```python
sample_sizes = [500 * i for i in range(1, 11)] + [10_000]
```

```python

```

```python

```

```python
def run_synthetic_experiment(dataset_type='sparse'):
    """
    TODO: IMPLEMENT

    Runs the synthetic experiment on either the sparse or dense data
    depending on the data path (e.g. "data/sparse" or "data/dense").

    We have provided how to train the Perceptron on the training and
    test on the testing data (the last part of the experiment). You need
    to implement the hyperparameter sweep, the learning curves, and
    predicting on the test dataset for the other models.
    """

    X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data(dataset_type)

    # TODO: YOUR CODE HERE. Determine the best hyperparameters for the relevant models
    # report the validation accuracy after training using the best hyper parameters

#     i think i did all of these picked the best hyper parameters way above here near the hyper parameter tables

    # TOOD: YOUR CODE HERE. Downsample the dataset to the number of desired training
    # instances (e.g. 500, 1000), then train all of the models on the
    # sampled dataset. Compute the accuracy and add the accuracies to
    # the corresponding list. Use plot_learning_curves()

#     I did all of these near where the plot is, separated into 7 or 8 cells. please refer to that.

    # TODO: Train all 7 models on the training data and make predictions
    # for test data
    # We will show you how to do it for the basic Perceptron model.
    classifier = Perceptron(features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"Perceptron's accuracy is {acc}")

    # YOUR CODE HERE: Repeat for the other 6 models.

#     Averaged Perceptron
    classifier = AveragedPerceptron(features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"Averaged Perceptron's accuracy is {acc}")

#     Winnow
    alpha = 1.01
    classifier = Winnow(alpha,features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"Winnow's accuracy is {acc}")

#     Averaged Winnow
    alpha = 1.1
    classifier = AveragedWinnow(alpha,features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"Averged Winnow's accuracy is {acc}")

#     Averaged Winnow
    eta = 1.5
    classifier = AdaGrad(eta,features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"AdaGrad's accuracy is {acc}")

#     Averaged Winnow
    eta = 1.5
    classifier = AveragedAdaGrad(eta,features)
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"Averged AdaGrad's accuracy is {acc}")

    vectorizer = DictVectorizer()
    X_train_dict = vectorizer.fit_transform(X_train)
    X_test_dict = vectorizer.transform(X_test)
    classifier = LinearSVC(loss = 'hinge')
    classifier.fit(X_train_dict,y_train)
    acc = classifier.score(X_test_dict,y_test)
    print(f"SVM's accuracy is {acc}")
```

```python
"""
Run the synthetic experiment on the sparse dataset. For reference,
"synthetic/sparse" is the path to where the data is located.
Note: This experiment takes substantial time (around 15 minutes),
so don't worry if it's taking a long time to finish.
"""
run_synthetic_experiment('sparse')
```

    Perceptron's accuracy is 0.717
    Averaged Perceptron's accuracy is 0.9135
    Winnow's accuracy is 0.926
    Averged Winnow's accuracy is 0.936
    AdaGrad's accuracy is 0.878
    Averged AdaGrad's accuracy is 0.884
    SVM's accuracy is 0.936

```python
"""
Run the synthetic experiment on the dense dataset. For reference,
"synthetic/dense" is the path to where the data is located.
Note: this experiment should take much less time.
"""
run_synthetic_experiment('dense')
```

    Perceptron's accuracy is 0.9205
    Averaged Perceptron's accuracy is 0.9405
    Winnow's accuracy is 0.9255
    Averged Winnow's accuracy is 0.9405
    AdaGrad's accuracy is 0.9325
    Averged AdaGrad's accuracy is 0.9405
    SVM's accuracy is 0.9405

##### Questions (5 points)

Answer the following questions:

1. Discuss the trends that you see when comparing the standard version of an algorithm to the averaged version (e.g., Winnow versus Averaged Winnow). Is there an observable trend?

    Typically averaged version of the algorithm has a slightly higher accuracy than the standard version, only 0.01~0.02 improvement but quite consistent. Averaged version also tends to converge a faster than regular version

2. We provided you with 10,000 training examples.
   Were all 10,000 necessary to achieve the best performance for each classifier?
   If not, how many were necessary? (Rough estimates, no exact numbers required)

   Not all were necessary. Looking at the learning rate graph, accuracy for both dense and sparse datasets plateau around 6000-7000 training examples for even the worst-performing algorithm. Winnow and averaged winnow converged around 2000 and 4000 examples respectively. For the best-performing algorithm, SVM, it converge to the highest accuracy with only about 1000 training examples. Other algorithm converged between 5000-7000 examples. Further examples did not provide any improvement over accuracy.

3. Report your Final Test Accuracies

| Model               | Sparse | Dense |
|---------------------|--------|-------|
| Perceptron          |  0.7170      |   0.9205    |
| Winnow              |  0.9260      |  0.9255     |
| AdaGrad             |  0.8780      |  0.9325     |
| Averaged Perceptron |  0.9135      |  0.9405     |
| Averaged Winnow     |  0.9360      |  0.9405     |
| Averaged AdaGrad    |  0.8840      |  0.9405     |
| SVM                 |  0.9360      |  0.9405     |

#### 3.1.5 Extra Credit (10 points)

Included in the resources for this homework assignment is the code that we used to generate the synthetic data.
We used a small amount of noise to create the dataset which you ran the experiments on.
For extra credit, vary the amount of noise in either/both of the label and features.
Then, plot the models' performances as a function of the amount of noise.
Discuss your observations.

TODO: Extra Credit observations

I ran the experiment with varying random values using the code shown below. Key thing is the 100% random value means that I randomly switched len(y_train) amount of the y_train, but they could've been repeatedly flipped. In the figure below, we can see the perceptron and adagrad approaches 50% which is essentially guess randomly. However, winnow algorithm did not descend to that low of an accuracy because a wrong y_train for winnow means that the added weight goes from w^alpha*y*x to 1/w^alpha*y*x. While for perceptron and adagrad, a wrong y_train means altering the weight vector in the completely opposite direction. Therefore, winnow accuracy decreased but not to a meaningless value while adagrad and perceptron algorithms become obsolete with 100% noise. For some reason SVM did not change. I experiemnted with different ways to make noise but its accuracy end up being a step function not sure why.

```python
X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')
noise = np.linspace(0,1,21)
ec_winnowacc = np.zeros(len(noise))
ec_perceptronacc = np.zeros(len(noise))
ec_adagradacc = np.zeros(len(noise))
ec_svmacc = np.zeros(len(noise))

for i in range(len(noise)):
    X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')
    classifier = Perceptron(features)
    num = int(len(y_train) * noise[i])
    for j in range(num):
        tempRand = np.random.randint(0,len(y_train))
        y_train[tempRand] = -y_train[tempRand]
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    ec_perceptronacc[i] = accuracy_score(y_test, y_pred)

for i in range(len(noise)):
    X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')
    classifier = Winnow(1.1,features)
    num = int(len(y_train) * noise[i])
    for j in range(num):
        tempRand = np.random.randint(0,len(y_train))
        y_train[tempRand] = -y_train[tempRand]
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    ec_winnowacc[i] = accuracy_score(y_test, y_pred)

for i in range(len(noise)):
    X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')
    classifier = AdaGrad(1.5,features)
    num = int(len(y_train) * noise[i])
    for j in range(num):
        tempRand = np.random.randint(0,len(y_train))
        y_train[tempRand] = -y_train[tempRand]
    classifier.train(X_train, y_train)
    y_pred = classifier.predict(X_test)
    ec_adagradacc[i] = accuracy_score(y_test, y_pred)

for i in range(len(noise)):
    X_train, y_train, X_dev, y_dev, X_test, y_test, features = load_synthetic_data('dense')
    num = int(len(y_train) * noise[i])
    for j in range(num):
        tempRand = np.random.randint(0,len(y_train))
        y_train[tempRand] = -y_train[tempRand]
    vectorizer = DictVectorizer()
    X_train_dict = vectorizer.fit_transform(X_train)
    X_test_dict = vectorizer.transform(X_test)
    classifier = LinearSVC(loss = 'hinge')
    classifier.fit(X_train_dict,y_train)
    ec_svmacc[i] = classifier.score(X_test_dict,y_test)
```

```python
plot_EC_curves(ec_perceptronacc,ec_winnowacc,ec_adagradacc,ec_svmacc)
```

```python
"""
A helper function that plots the relationship between number of examples
and accuracies for all the models.

You should not need to edit this method.
"""
def plot_EC_curves(
    perceptron_accs,
    winnow_accs,
    adagrad_accs,
    svm_accs
):

    accuracies = [
        ('perceptron', perceptron_accs),
        ('winnow', winnow_accs),
        ('adagrad', adagrad_accs),
        ('svm', svm_accs)
    ]

    x = np.linspace(0,1,21)
    plt.figure()
#     f, (ax, ax2) = plt.subplots(1, 2, sharey=True, facecolor='w')

    for label, acc_list in accuracies:
        assert len(acc_list) == 21
        plt.plot(x, acc_list, label=label)
#         ax2.plot(x, acc_list, label=label)
#     plt.set_xlim(0,1)
#     ax.set_xlim(0, 1)
#     ax2.set_xlim(9500, 10000)
#     ax2.set_xticks([10000])
    # hide the spines between ax and ax2
#     plt.spines['right'].set_visible(False)
#     ax2.spines['left'].set_visible(False)
#     ax.yaxis.tick_left()
#     ax.tick_params(labelright='off')
#     ax2.yaxis.tick_right()
#     ax2.legend()
    plt.legend()
    plt.title('Extra Credit Noise Plot')
    plt.show()
```

```python
ec_winnowacc
```

    array([0.814 , 0.802 , 0.718 , 0.784 , 0.6285, 0.7525, 0.769 , 0.6545,
           0.5415, 0.6905, 0.707 , 0.6945, 0.688 , 0.718 , 0.656 , 0.7205,
           0.725 , 0.6525, 0.77  , 0.7515, 0.812 ])

### 3.2 NER Experiment: Welcome to the Real World (35 points)

The experiment with the NER data will analyze how changing the domain of the training and testing data can impact the performance of a model.

Instead of accuracy, you will use your $F_1$ score implementation in Section 0 to evaluate how well a model does.
Recall measures how many of the actual entities the model successfully tagged as an entity.

$$
\begin{align}
    \textrm{Precision} &= \frac{\#\textrm{(Actually Entity & Model Predicted Entity)}}{\#\textrm{(Model Predicted Entity)}} \\
    \textrm{Recall} &= \frac{\#\textrm{(Actually Entity & Model Predicted Entity)}}{\#\textrm{(Actually Entity)}} \\
    \textrm{F}_1 &= 2 \cdot \frac{\textrm{Precision} \times \textrm{Recall}}{\textrm{Precision} + \textrm{Recall}}
\end{align}
$$

For this experiment, you will only use the averaged basic Perceptron and SVM.
Hence, no parameter tuning is necessary.
Train both models on the CoNLL training data then compute the F$_1$ on the development and testing data of both CoNLL and Enron.
Note that the model which is used to predict labels for Enron is trained on CoNLL data, not Enron data.
Report the F$_1$ scores in a table.

#### 3.2.1 Extracting NER Features  (25 points)

Reread Section 2.2.2 to understand how to extract the features required to train the models
and translate it to the code below.

```python
def extract_ner_features_train(train):
    """
    Extracts feature dictionaries and labels from the data in "train"
    Additionally creates a list of all of the features which were created.
    We have implemented the w-1 and w+1 features for you to show you how
    to create them.

    TODO: You should add your additional featurization code here.
    (which might require adding and/or changing existing code)
    """
    y = []
    X = []
    features = set()
    for sentence in train:
        padded = [('SSS', None)] + [('SSS', None)] + [('SSS', None)] + sentence[:]\
                    + [('EEE', None)] + [('EEE', None)] + [('EEE', None)]
        for i in range(3, len(padded) - 3):
            y.append(1 if padded[i][1] == 'I' else -1)
            feat1 = 'w-1=' + str(padded[i - 1][0])
            feat2 = 'w+1=' + str(padded[i + 1][0])
            feat3 = 'w-3=' + str(padded[i - 3][0])
            feat4 = 'w-2=' + str(padded[i - 2][0])
            feat5 = 'w+2=' + str(padded[i + 2][0])
            feat6 = 'w+3=' + str(padded[i + 3][0])
            feat7 = 'w-1&w-2=' + str(padded[i - 1][0] + str(padded[i - 2][0]))
            feat8 = 'w+1&w+2=' + str(padded[i + 1][0] + str(padded[i + 2][0]))
            feat9 = 'w-1&w+1=' + str(padded[i - 1][0] + str(padded[i + 1][0]))

            feats = [feat1, feat2, feat3, feat4, feat5, feat6, feat7, feat8, feat9]
            features.update(feats)
            feats = {feature: 1 for feature in feats}
            X.append(feats)

    return features, X, y
```

Now, repeat the process of extracting features from the test data.
What is the difference between the code above and below?

```python
def extract_features_dev_or_test(data, features):
    """
    Extracts feature dictionaries and labels from "data". The only
    features which should be computed are those in "features". You
    should add your additional featurization code here.

    TODO: You should add your additional featurization code here.
    """
    y = []
    X = []
    for sentence in data:
        padded = [('SSS', None)] + [('SSS', None)] + [('SSS', None)] + sentence[:]\
                    + [('EEE', None)] + [('EEE', None)] + [('EEE', None)]
        for i in range(3, len(padded) - 3):
            y.append(1 if padded[i][1] == 'I' else -1)
            feat1 = 'w-1=' + str(padded[i - 1][0])
            feat2 = 'w+1=' + str(padded[i + 1][0])
            feat3 = 'w-3=' + str(padded[i - 3][0])
            feat4 = 'w-2=' + str(padded[i - 2][0])
            feat5 = 'w+2=' + str(padded[i + 2][0])
            feat6 = 'w+3=' + str(padded[i + 3][0])
            feat7 = 'w-1&w-2=' + str(padded[i - 1][0] + str(padded[i - 2][0]))
            feat8 = 'w+1&w+2=' + str(padded[i + 1][0] + str(padded[i + 2][0]))
            feat9 = 'w-1&w+1=' + str(padded[i - 1][0] + str(padded[i + 1][0]))
            feats = [feat1, feat2, feat3, feat4, feat5, feat6, feat7, feat8, feat9]
            feats = {feature: 1 for feature in feats if feature in features}
            X.append(feats)
    return X, y
```

#### 3.2.2 Running the NER Experiment

As stated previously, train both models on the CoNLL training data then compute the $F_1$  on the development and testing data of both CoNLL and Enron. Note that the model which is used to predict labels for Enron is trained on CoNLL data, not Enron data.

```python
def run_ner_experiment(data_path):
    """
    Runs the NER experiment using the path to the ner data
    (e.g. "ner" from the released resources). We have implemented
    the standard Perceptron below. You should do the same for
    the averaged version and the SVM.

    The SVM requires transforming the features into a different
    format. See the end of this function for how to do that.
    """
    train = load_ner_data(dataset='conll', dataset_type='train')
    conll_test = load_ner_data(dataset='conll', dataset_type='test')
    enron_test = load_ner_data(dataset='enron', dataset_type='test')

    features, X_train, y_train = extract_ner_features_train(train)
    X_conll_test, y_conll_test = extract_features_dev_or_test(conll_test, features)
    X_enron_test, y_enron_test = extract_features_dev_or_test(enron_test, features)

    # TODO: We show you how to do this for Perceptron.
    # You should do this for the Averaged Perceptron and SVM
    classifier = Perceptron(features)
    classifier.train(X_train, y_train)

    y_pred = classifier.predict(X_conll_test)
    conll_f1 = calculate_f1(y_conll_test, y_pred)

    y_pred = classifier.predict(X_enron_test)
    enron_f1 = calculate_f1(y_enron_test, y_pred)
    print('Perceptron on NER')
    print('  CoNLL', conll_f1)
    print('  Enron', enron_f1)
    print('Accuracy',accuracy_score(y_enron_test, y_pred))

#     Averaged Perceptron
    classifier = AveragedPerceptron(features)
    classifier.train(X_train, y_train)

    y_pred = classifier.predict(X_conll_test)
    conll_f1 = calculate_f1(y_conll_test, y_pred)

    y_pred = classifier.predict(X_enron_test)
    enron_f1 = calculate_f1(y_enron_test, y_pred)
    print('Averaged Perceptron on NER')
    print('  CoNLL', conll_f1)
    print('  Enron', enron_f1)
    print('Accuracy',accuracy_score(y_enron_test, y_pred))

#     SVM
    # This is how you convert from the way we represent features in the
    # Perceptron code to how you need to represent features for the SVM.
    # You can then train with (X_train_dict, y_train) and test with
    # (X_conll_test_dict, y_conll_test) and (X_enron_test_dict, y_enron_test)
    vectorizer = DictVectorizer()
    X_train_dict = vectorizer.fit_transform(X_train)
    X_conll_test_dict = vectorizer.transform(X_conll_test)
    X_enron_test_dict = vectorizer.transform(X_enron_test)
    classifier = LinearSVC(loss = 'hinge')
    classifier.fit(X_train_dict,y_train)

    y_pred = classifier.predict(X_conll_test_dict)
    conll_f1 = calculate_f1(y_conll_test,y_pred)

    y_pred = classifier.predict(X_enron_test_dict)
    enron_f1 = calculate_f1(y_enron_test, y_pred)
    print('SVM on NER')
    print('  CoNLL', conll_f1)
    print('  Enron', enron_f1)
    print('Accuracy',classifier.score(X_enron_test_dict,y_enron_test))
```

```python
# Run the NER experiment. "ner" is the path to where the data is located.
run_ner_experiment('ner')
```

    TP is 597
    FP is 235
    FN is 175
    TP is 182
    FP is 1021
    FN is 380
    Perceptron on NER
      CoNLL 0.7443890274314214
      Enron 0.20623229461756373
    Accuracy 0.8359676852827538
    TP is 574
    FP is 69
    FN is 198
    TP is 104
    FP is 285
    FN is 458
    Averaged Perceptron on NER
      CoNLL 0.8113074204946996
      Enron 0.21871713985278654
    Accuracy 0.9130078445146939
    TP is 601
    FP is 79
    FN is 171
    TP is 113
    FP is 262
    FN is 449
    SVM on NER
      CoNLL 0.827823691460055
      Enron 0.24119530416221988
    Accuracy 0.916754478398314

```python
conll_test = load_ner_data(dataset='conll', dataset_type='test')
enron_test = load_ner_data(dataset='enron', dataset_type='test')
X_conll_test, y_conll_test = extract_features_dev_or_test(conll_test, features)
X_enron_test, y_enron_test = extract_features_dev_or_test(enron_test, features)
print(np.shape(y_conll_test))
print(np.shape(y_enron_test))
```

    (3779,)
    (8541,)

##### F1 Scores Table
TODO: report your values:

| Model               | CoNLL Test F1 | Enron Test F1 |
|---------------------|---------------|---------------|
| Averaged Perceptron |   0.8131            |    0.2187           |
| SVM                 |   0.8278            |    0.2412           |

##### Questions (5 points)

Comment on the results:
1. Are the F$_1$ scores on CoNLL and Enron similar?

    No, F$_1$ scores on CoNLL and Enron are quite different. CoNLL has much higher ones than Enron.

2. If they are dissimilar, explain why you think the F$_1$ score increased/decreased on the Enron data.

    In CoNLL, TP is significantly higher than FN and FP. However, in Enron TP is lower than FN and FP which resulted in a much lower F1 score. I printed out the accuracy on Enron data which is very high (80~90 percent for both). And then I printed out the TP,FN,FP,TN (TN = total - TP - FN - FP) and found that true negative is an order of magnitude higher than all three other values. It seems like Enron has lots of true negative but algorithms have trouble identify the positive examples in Enron. It didn't have this problem with CoNLL.

## Submission Instructions

We will be using Gradescope to turn in both the Python code.
You should have been automatically added to Gradescope.
If you do not have access, please ask the TA staff on Piazza.

There are three parts to the submission on Gradescope:
* ipynb file: Submit this notebook (.ipynb). If you are using Google Colab, you will have to download it as an ipynb file.
* PDF: This notebook saved as a PDF. We will use this to see the overall structure of your homework and code, and to check manual questions like the tables, plots, and your discussions. How should I convert this notebook to PDF, you may ask? There are a few ways, but for simplicity print the Jupyter notebook and _Save as PDF_.
* Code: A `hw2.py` file. We will use this to unit test and automate grading some parts of the homework. We only need a few functions/classes from the whole notebook. In particular, to unit test we only need:
    - `calculate_f1`
    - `highest_and_lowest_f1_score`
    - `Perceptron`, `Winnow`, and `AdaGrad` classes.

There are two ways to submit these pieces of code. You can either manually copy and paste these to a Python file,
or you can use Jupyter's _Download as .py_, and delete all unnecessary code.


```python
%matplotlib inline
```

강화 학습 (DQN) 튜토리얼
=====================================
**Author**: `Adam Paszke <https://github.com/apaszke>`_
  **번역**: `황성수 <https://github.com/adonisues>`_

이 튜토리얼에서는 `OpenAI Gym <https://gym.openai.com/>`__
CartPole-v0 태스크의 DQN (Deep Q Learning) 에이전트를 학습하는데
PyTorch를 사용하는 방법을 보여드립니다.

**태스크**

에이전트는 연결된 막대가 똑바로 서 있도록 카트를 왼쪽이나 오른쪽으로
움직이는 두 가지 동작 중 하나를 선택해야합니다.
다양한 알고리즘과 시각화 기능을 갖춘 공식 순위표를
`Gym website <https://gym.openai.com/envs/CartPole-v0>`__ 에서 찾을 수 있습니다.

.. figure:: /_static/img/cartpole.gif
   :alt: cartpole

   cartpole

에이전트가 현재 환경 상태를 관찰하고 행동을 선택하면
환경이 새로운 상태로 *전환* 되고 작업의 결과를 나타내는 보상도 반환됩니다.
이 태스크에서는 막대가 지나치게 떨어지면 환경이 종료됩니다.

카트폴 태스크는 에이전트에 대한 입력이 환경 상태(위치, 속도 등)를 나타내는
4개의 실제 값이 되도록 설계되었습니다. 그러나 신경망은 순수하게 그 장면을 보고
태스크를 해결할 수 있습니다 따라서 카트 중심의 화면 패치를 입력으로 사용합니다.
이 때문에 우리의 결과는 공식 순위표의 결과와 직접적으로 비교할 수 없습니다.
우리의 태스크는 훨씬 더 어렵습니다.
불행히도 모든 프레임을 렌더링해야되므로 이것은 학습 속도를 늦추게됩니다.

엄밀히 말하면, 현재 스크린 패치와 이전 스크린 패치 사이의 차이로 상태를 표시 할 것입니다.
이렇게하면 에이전트가 막대의 속도를 한 이미지에서 고려할 수 있습니다.

**패키지**

먼저 필요한 패키지를 가져옵니다. 첫째, 환경을 위해
`gym <https://gym.openai.com/docs>`__ 이 필요합니다.
(`pip install gym` 을 사용하여 설치하십시오).
또한 PyTorch에서 다음을 사용합니다:

-  신경망 (``torch.nn``)
-  최적화 (``torch.optim``)
-  자동 미분 (``torch.autograd``)
-  시각 태스크를 위한 유틸리티들 (``torchvision`` - `a separate
   package <https://github.com/pytorch/vision>`__).

```python
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

env = gym.make('CartPole-v0').unwrapped

# matplotlib 설정
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# GPU를 사용할 경우
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

재현 메모리(Replay Memory)
-------------------------------

우리는 DQN 학습을 위해 경험 재현 메모리를 사용할 것입니다.
에이전트가 관찰한 전환(transition)을 저장해고 나중에 이 데이터를
재사용 할 수 있습니다. 무작위로 샘플링하면 배치를 구성한는 전환들이
비상관(decorrelated)하게 됩니다. 이것이 DQN 학습 절차를 크게 안정시키고
향상시키는 것으로 나타났습니다.

이를 위해서 두개의 클래스가 필요합니다:

-  ``Transition`` - 우리 환경에서 단일 전환을 나타내도록 명명된 튜플
-  ``ReplayMemory`` - 최근 관찰된 전이를 보관 유지하는 제한된 크기의 순환 버퍼.
   또한 학습을 위한 전환의 무작위 배치를 선택하기위한
   ``.sample ()`` 메소드를 구현합니다.

```python
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """전환 저장"""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
```

이제 모델을 정의합시다. 그러나 먼저 DQN이 무엇인지 간단히 요약해 보겠습니다.

DQN 알고리즘
-------------

우리의 환경은 결정론적이므로 여기에 제시된 모든 방정식은 단순화를 위해
결정론적으로 공식화됩니다. 강화 학습 자료은 환경에서 확률론적 전환에
대한 기대값(expectation)도 포함 할 것입니다.

우리의 목표는 할인된 누적 보상 (discounted cumulative reward)을
극대화하려는 정책(policy)을 학습하는 것입니다.
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, 여기서
$R_{t_0}$ 는 *반환(return)* 입니다. 할인 상수,
$\gamma$, 는 $0$ 과 $1$ 의 상수이고 합계가
수렴되도록 보장합니다. 에이전트에게 불확실한 먼 미래의 보상이
가까운 미래의 것에 비해 덜 중요하게 만들고, 이것은 상당히 합리적입니다.

Q-learning의 주요 아이디어는 만일 함수 $Q^*: State \times Action \rightarrow \mathbb{R}$ 를
가지고 있다면 반환이 어덯게 될지 알려줄 수 있고,
만약 주어진 상태(state)에서 행동(action)을 한다면, 보상을 최대화하는
정책을 쉽게 구축할 수 있습니다:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

그러나 세계(world)에 관한 모든 것을 알지 못하기 때문에,
$Q^*$ 에 도달할 수 없습니다. 그러나 신경망은
범용 함수 근사자(universal function approximator)이기 때문에
간단하게 생성하고 $Q^*$ 를 닮도록 학습 할 수 있습니다.

학습 업데이트 규칙으로, 일부 정책을 위한 모든 $Q$ 함수가
Bellman 방정식을 준수한다는 사실을 사용할 것입니다:

\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

평등(equality)의 두 측면 사이의 차이는
시간차 오류(temporal difference error), $\delta$ 입니다.:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

오류를 최소화하기 위해서 `Huber
loss <https://en.wikipedia.org/wiki/Huber_loss>`__ 를 사용합니다.
Huber loss 는 오류가 작으면 평균 제곱 오차( mean squared error)와 같이
동작하고 오류가 클 때는 평균 절대 오류와 유사합니다.
- 이것은 $Q$ 의 추정이 매우 혼란스러울 때 이상 값에 더 강건하게 합니다.
재현 메모리에서 샘플링한 전환 배치 $B$ 에서 이것을 계산합니다:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

Q-네트워크
^^^^^^^^^^^

우리 모델은 현재와 이전 스크린 패치의 차이를 취하는
CNN(convolutional neural network) 입니다. 두가지 출력 $Q(s, \mathrm{left})$ 와
$Q(s, \mathrm{right})$ 가 있습니다. (여기서 $s$ 는 네트워크의 입력입니다)
결과적으로 네트워크는 주어진 현재 입력에서 각 행동의 *품질* 을 예측하려고 합니다.

```python
class DQN(nn.Module):

    def __init__(self):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)
        self.head = nn.Linear(448, 2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
```

입력 추출
^^^^^^^^^^^^^^^^

아래 코드는 환경에서 렌더링 된 이미지를 추출하고 처리하는 유틸리티입니다.
이미지 변환을 쉽게 구성 할 수 있는 ``torchvision`` 패키지를 사용합니다.
셀(cell)을 실행하면 추출한 예제 패치가 표시됩니다.

```python
resize = T.Compose([T.ToPILImage(),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

# 이것은 gym 코드를 기반으로 합니다.
screen_width = 600

def get_cart_location():
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0)  # 카트의 중간

def get_screen():
    screen = env.render(mode='rgb_array').transpose(
        (2, 0, 1))  # transpose into torch order (CHW)
    # Strip off the top and bottom of the screen
    screen = screen[:, 160:320]
    view_width = 320
    cart_location = get_cart_location()
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width, None)
    else:
        slice_range = slice(cart_location - view_width // 2,
                            cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescare, convert to torch tensor
    # (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0).to(device)

env.reset()
plt.figure()
plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),
           interpolation='none')
plt.title('Example extracted screen')
plt.show()
```

학습
--------

하이퍼 파라미터와 유틸리티
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
이 셀은 모델과 최적화기를 인스턴스화하고 일부 유틸리티를 정의합니다:

-  ``select_action`` - Epsilon Greedy 정책에 따라 행동을 선택합니다.
   간단히 말해서, 가끔 모델을 사용하여 행동을 선택하고 때로는 단지 하나를
   균일하게 샘플링 할 것입니다. 임의의 액션을 선택할 확률은
   ``EPS_START`` 에서 시작해서 ``EPS_END`` 를 향해 지수적으로 감소 할 것입니다.
   ``EPS_DECAY``는 감쇠 속도를 제어합니다.
-  ``plot_durations`` - 지난 100개 에피소드의 평균(공식 평가에서 사용 된 수치)에 따른
   에피소드의 지속을 도표로 그리기 위한 헬퍼. 도표는 기본 훈련 루프가
   포함 된 셀 밑에 있으며, 매 에피소드마다 업데이트됩니다.

```python
BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 10

policy_net = DQN().to(device)
target_net = DQN().to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample > eps_threshold:
        with torch.no_grad():
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(2)]], device=device, dtype=torch.long)

episode_durations = []

def plot_durations():
    plt.figure(2)
    plt.clf()
    durations_t = torch.tensor(episode_durations, dtype=torch.float)
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(durations_t.numpy())
    # 100개의 에피소드 평균을 가져 와서 도표 그리기
    if len(durations_t) >= 100:
        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)
        means = torch.cat((torch.zeros(99), means))
        plt.plot(means.numpy())

    plt.pause(0.001)  # 도표가 업데이트되도록 잠시 멈춤
    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())
```

학습 루프
^^^^^^^^^^^^^

최종적으로 모델 학습을 위한 코드.

여기서, 최적화의 한 단계를 수행하는 ``optimize_model`` 함수를 찾을 수 있습니다.
먼저 배치 하나를 샘플링하고 모든 Tensor를 하나로 연결하고
$Q(s_t, a_t)$ 와  $V(s_{t+1}) = \max_a Q(s_{t+1}, a)$ 를 계산하고
그것들을 손실로 합칩니다. 우리가 설정한 정의를 따르면 만약 $s$ 가
마지막 상태라면 $V(s) = 0$ 이다.
또한 안정성 추가 위한 $V(s_{t+1})$ 계산을 위해 목표 네트워크를 사용합니다.
목표 네트워크는 대부분의 시간 동결 상태로 유지되지만, 가끔 정책
네트워크의 가중치로 업데이트됩니다.
이것은 대개 설정한 스텝 숫자이지만 단순화를 위해 에피소드를 사용합니다.

```python
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (자세한 설명을 위해 http://stackoverflow.com/a/19343/3343043 를 보십시오).
    batch = Transition(*zip(*transitions))

    # 최종 상태가 아닌 마스크를 계산하고 배치 요소를 연결합니다.
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.uint8)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Q(s_t, a) 계산 - 모델이 Q(s_t)를 계산하고, 취한 행동의 칼럼을 선택한다.
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # 모든 다음 상태를 위한 V(s_{t+1}) 계산
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # 기대 Q 값 계산
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Huber 손실 계산
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # 모델 최적화
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
```

아래에서 주요 학습 루프를 찾을 수 있습니다. 처음으로 환경을
재설정하고 ``상태`` Tensor를 초기화합니다. 그런 다음 행동을
샘플링하고, 그것을 실행하고, 다음 화면과 보상(항상 1)을 관찰하고,
모델을 한 번 최적화합니다. 에피소드가 끝나면 (모델이 실패)
루프를 다시 시작합니다.

아래에서 `num_episodes` 는 작게 설정됩니다. 노트북을 다운받고
더많은 에피소드를 실행해 보십시오

```python
num_episodes = 50
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    for t in count():
        # 행동 선택과 수행
        action = select_action(state)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # 새로운 상태 관찰
        last_screen = current_screen
        current_screen = get_screen()
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None

        # 메모리에 변이 저장
        memory.push(state, action, next_state, reward)

        # 다음 상태로 이동
        state = next_state

        # 최적화 한단계 수행(목표 네트워크에서)
        optimize_model()
        if done:
            episode_durations.append(t + 1)
            plot_durations()
            break
    # 목표 네트워크 업데이트
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
env.render()
env.close()
plt.ioff()
plt.show()
```