# **Introduction to PyTorch** - Logistic Regression, MLP, and MNIST

**Duke Machine Learning Summer School 2021**

**Code by Billy Carson**

In this notebook, we're going to gain familiarity with the PyTorch deep learning framework by building PyTorch models to perform handwritten digit classification.

## **1. Setting Up the Enivronment**

### 1.1. Import libraries, modules, and functions

First, we need to import necessary libraries, modules, and functions we'll use throughout this tutorial. To give ourselves access to the offerings of the PyTorch library, we'll import the [torch](https://pytorch.org/docs/stable/torch.html) package and the [torch.nn](https://pytorch.org/docs/stable/nn.html) module. Later, you can cut down on import bloat by only importing specific modules, functions, and objects that you'll need for your project. For now, we'll keep it simple by just using these two import statements.

```python
# Define random state
random_state = 0

# Import libraries, modules, and functions
import time
import numpy as np
import matplotlib.pyplot as plt

# PyTorch imports
from torchvision import datasets, transforms
import torch
import torch.nn as nn
```

### 1.2. GPU/CUDA availability

When setting up your coding environment for a project or experiment, it's important that we are aware of the available computing resources. The resources we have access to will determine what models we can or can't use and the amount of data we can load into memory, among other considerations. For this tutorial, since we are dealing with simpler machine learning models, we won't need access to a [Graphics Processing Unit](https://en.wikipedia.org/wiki/Graphics_processing_unit) (GPU). GPUs help speed up the training of deep learning models by parallelizing many computations intead of computing sequentially. This parallelization is critical when training more complex models like [Convolutional Neural Networks](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNNs), and without it these models would take prohibitively long to train.

We can check whether or not our environment has access to a GPU by calling the [`torch.cuda.is_available()`](https://pytorch.org/docs/stable/cuda.html) function. [CUDA](https://developer.nvidia.com/about-cuda) is a parallel computing platform and programming model developed by Nvidia for general computing on its own GPUs. This function will tell us if CUDA is available in this computing environment. Since we don't need/have access to a GPU, `torch.cuda.is_available()` should return `False`.

```python
# Display GPU/CUDA availability
print('\nCUDA availability:  %s\n' % (torch.cuda.is_available()))

```

## **2. Loading and Preparing the Data**

One of the first (and often most important) steps in a machine learning pipeline is preparing the data. Here, we will load the MNIST dataset of images of handwritten digits and format this data so it can be used with PyTorch models.

### 2.1. Load MNIST dataset

The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) is very popular machine learning dataset, consisting of 70,000 grayscale, 28x28 images of handwritten digits.  We'll be using it as our example dataset for this section of the tutorial, with the goal being to predict which digit is depicted in each image.

First, let's load the training and test sets of MNIST.  Normally, MNIST data comes formatted as images and need to be converted to a type that can be operated on by PyTorch models. For this tutorial, we already have the MNIST dataset formatted as [PyTorch tensors](https://pytorch.org/docs/stable/tensors.html).

```python
# Load MNIST dataset
mnist_train = datasets.MNIST(root="../datasets", train=True, transform=transforms.ToTensor(), download=True)
mnist_test = datasets.MNIST(root="../datasets", train=False, transform=transforms.ToTensor(), download=True)

X_raw_train, y_train = torch.load('../datasets/MNIST/processed/training.pt')
X_raw_test, y_test = torch.load('../datasets/MNIST/processed/test.pt')
```

### 2.2. Explore data/PyTorch tensors

Tensors are a specialized data structure very similar to arrays and matrices that house data in $N$ dimensions. Tensors can be thought of as generalizations of matrices (which are specifically 2-dimensional tensors) to $N$-dimensional space.

In PyTorch, tensors are used to encode data, model inputs, model outputs, and model parameters. [PyTorch tensors](https://pytorch.org/docs/stable/tensors.html) are objects similar to [NumPy ndarrays](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html), except that tensors can run on GPUs or other hardware accelerators. Tensors are also optimized for automatic differentiation with [`autograd`](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html), PyTorch’s automatic differentiation engine that powers neural network training. Here, where call the tensor method `shape()` to get an idea of the size and dimensionality of our data.

```python
# Display training data dimensionality
print('\nTraining data dimensionality:')
print('  Images:  (%d, %d, %d)' % (X_raw_train.shape))
print('  Label:  (%s,)' % (y_train.shape))

# Display test data dimensionality
print('\nTest data dimensionality:')
print('  Images:  (%d, %d, %d)' % (X_raw_test.shape))
print('  Label:  (%d,)\n' % (y_test.shape))

# Display different labels
print('Training labels:')
print(np.unique(y_train.numpy()))
print('\nTest labels:')
print(np.unique(y_test.numpy()))
print()

```

### 2.3. Format tensors

For our PyTorch models and loss functions to be able to operate on the tensors of MNIST data, they first need to be of a specific [tensor type](https://pytorch.org/docs/stable/tensors.html). Specifically, the image data tensors need to be 32-bit floating point tensors (tensor type `FloatTensor`) and the labels tensors need to be 64-bit signed integers (tensor type `LongTensor`).

We can display the tensor type by calling the tensor method `type()`, which will return the tensor type. As we'll see, our labels are already in the correct format but our image data tensors need to be converted from type `ByteTensor` to type `FloatTensor`. We can make this conversion by calling the tensor method `float()`.

```python
# Display original tensor type
print('\nTensor type (before conversion):')
print('  Images: %s' % (X_raw_train.type()))
print('  Labels: %s\n' % (y_train.type()))

```

```python
# Convert image tensor type to float
X_raw_train = X_raw_train.float()
X_raw_test = X_raw_test.float()

```

```python
# Display tensor type after conversion
print('\nTensor type (after conversion):')
print('  Images: %s\n' % (X_raw_train.type()))

```

### 2.4. Scale data

Often in many machine learning and data science applications, it helps to first scale the data. Scaling our data can ensure that all of our features are on approximately the same scale. Significant differences in the scales across input variables may increase the difficulty of our optimization problem [1].

For this modelling problem, we'll scale our data to be between 0 and 1. Since MNIST data are black and white images, data elements correspond to pixel intensities with values between 0 and 255. Therefore, to scale our data between 0 and 1 we just need to divide our image data tensors by 255.

```python
# Display data max and min
print('\nData range pre-scaling (min, max)):')
print('  Train images:  %.1f, %.1f' % (torch.min(X_raw_train), torch.max(X_raw_train)))
print('  Test images:  %.1f, %.1f\n' % (torch.min(X_raw_test), torch.max(X_raw_test)))

```

```python
# Scale image pixel intensities between 0 and 1
X_train = X_raw_train / 255.0
X_test = X_raw_test / 255.0

```

```python
# Display data max and min
print('\nData range post-scaling (min, max)):')
print('  Train images:  %.1f, %.1f' % (torch.min(X_train), torch.max(X_train)))
print('  Test images:  %.1f, %.1f\n' % (torch.min(X_test), torch.max(X_test)))

```

### 2.5. Reshape data

Sometimes we need to reshape the data so that our chosen model can operate on it. Computer vision models such as CNNs operate on directly 2-dimensional image data, so if we were using a CNN we would not need to reshape our MNIST data. However, for this tutorial we will be working with logistic regression and MLP models. Both of these models require input data to be a 1-dimensional vector, meanining that we'll have to "flatten" our image data.

When flattening the image data, we want to preserve the first dimension of our tensors, which represents the samples or observations dimension. We can reshape our training and test data tensors using the [`view()`](https://pytorch.org/docs/stable/tensor_view.html) method. Additionally, this method avoids explicit data copy, thus facilitating fast and memory efficient reshaping, slicing, and element-wise operations on tensors [2].

```python
# Reshape and "flatten" image data
# Hint: -1 tells view() to infer the size of a given dimension
X_train = X_train.view(X_train.shape[0], -1)
X_test = X_test.view(X_test.shape[0], -1)

```

```python
# Display training data dimensionality
print('\nTraining data dimensionality:')
print('  Images:  (%d, %d)' % (X_train.shape))

# Display test data dimensionality
print('\nTest data dimensionality:')
print('  Images:  (%d, %d)\n' % (X_test.shape))

```

### 2.6. Define PyTorch dataset object

[`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) is an abstract class representing a dataset. It allows us to treat the dataset as an object of a class, rather than a set of data and labels. Creating a custom dataset will allow us to have greater control over how our model loads and interacts with the data, as well as permit built-in batching, shuffling, and loading data in parallel later on [3]. A custom `Dataset` class must implement three functions: ` __init__`, `__len__`, and `__getitem__`:

* The `__init__` function is run once when instantiating the `Dataset` object.
* The `__len__` function returns the number of samples in our dataset.
* The `__getitem__` function loads and returns a sample from the dataset at the given index idx.

```python
# MNIST PyTorch dataset object
class MNISTDataset(torch.utils.data.Dataset):
    """
    PyTorch dataset object for the MNIST dataset.

    Parameters
    ----------
    X : torch.Tensor
        Flattened image data of size (n_samples x 784).
    y : torch.Tensor
        Image labels, integer values 0, 1, 2, ..., 9.

    Attributes
    ----------
    X : torch.Tensor
        Flattened image data of size (n_samples x 784).
    y : torch.Tensor
        Image labels, integer values 0, 1, 2, ..., 9.
    len : int
        Length of dataset/number of observations in dataset.
    """

    # MNIST PyTorch dataset instantiation method
    def __init__(self, X, y):
        """
        MNIST dataset instantiation method.

        Parameters
        ----------
        X : torch.Tensor
            Flattened image data of size (n_samples x 784).
        y : torch.Tensor
            Image labels, integer values 0, 1, 2, ..., 9.

        Attributes
        ----------
        X : torch.Tensor
            Flattened image data of size (n_samples x 784).
        y : torch.Tensor
            Image labels, integer values 0, 1, 2, ..., 9.
        len : int
            Length of dataset/number of observations in dataset.
        """

        # Assign attributes
        self.X = X         # flattened image data
        self.y = y         # image labels
        self.len = len(X)  # length of dataset

    # Dataset length method
    def __len__(self):
        """
        Method that returns length of dataset/number of observations in dataset.

        Returns
        -------
        self.len : int
            Length of dataset/number of observations in dataset.
        """

        return self.len

    # Dataset indexing method
    def __getitem__(self, idx):
        """
        Method that retrieves samples and corresponding labels from the dataset.

        Parameters
        ----------
        idx : int
            Integer index of dataset sample to be retrieved.

        Returns
        -------
        X_samp : torch.Tensor
            Image sample.
        y_samp : torch.Tensor
            Label corresponding to image sample.
        """

        # Retrieve sample of data indexed by idx
        X_samp = self.X[idx, :]
        y_samp = self.y[idx]

        # Return data sample
        return X_samp, y_samp

```

### 2.7. Instantiate PyTorch datasets

Here, we instantiate a training and test version of the `MNISTDataset` class we defined above. Instantiating a class creates a copy or instance of the class which inherits all class attributes and methods. To instantiate an object in Python, we simply call the class as if it were a function, passing the requisite arguments to the `__init__` method. The newly created object will then be assigned to whatever variable name we choose.

```python
# Instantiate MNIST training and test PyTorch dataset objects
train_dataset = MNISTDataset(X_train, y_train)
test_dataset = MNISTDataset(X_test, y_test)

```

### 2.8. Instantiate PyTorch data loader objects

While we could interface directly with the data as a PyTorch `torch.utils.data.Dataset` object, it's often easier to use a PyTorch [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) object. The PyTorch `DataLoader` is an iterator that automatically takes care of:
* batching the data
* shuffling the data
* loading the data in parallel using multiprocessing workers

Similar to how we created both training and test set instances of PyTorch `Dataset` objects, we will now create instances of training and test set `DataLoader` objects.

```python
# Instantiate MNIST training and test PyTorch data loader objects
batch_size = 100
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

```

## **3. Building a Logistic Regression Model**

### 3.1. Define logistic regression model

For the first part of this tutorial, we'll be building a logistic regression model, which is essentially a fully-connected neural network without any hidden layers. Although basic, logistic regression can perform quite well on many simple classification tasks.

PyTorch has higher level abstractions to help speed up implementation and improve model organization. While there are many ways to organize PyTorch code, one common paradigm is the `torch.nn.Module`. PyTorch uses modules to represent neural networks. Modules are:

* Building blocks of stateful computation. PyTorch provides a robust library of pre-defined modules and makes it simple to define new custom modules, facilitating construction of many different deep neural network architecturess.
* Tightly integrated with PyTorch’s autograd system. Modules make it simple to specify learnable parameters for PyTorch’s Optimizers to update.
* Easy to work with and transform. Modules are straightforward to save, restore, transfer between CPU/GPU/TPU devices, and more [4].

For more on `torch.nn.Module`, see [this](https://pytorch.org/tutorials/beginner/nn_tutorial.html) tutorial. To create our LogisticRegression model that inherits from `torch.nn.Module`, we need to define two methods: `__init__()` and `forward()`. `__init__()` initializes the model parameters and assigns layers to object attributes. The `forward()` method computes the forward pass of the model by passing minibatches of input data through the different layers of the model.

To get an idea of what types of PyTorch layers to include in our logistic regression models, let's take a look at the mathematical formulation of logistic regression. Multiclass logistic regression can be expressed in matrix form as:

\begin{align}
y = x W + b
\end{align}

To take advantage of parallel computation, we commonly process multiple inputs $x$ at once, in a minibatch. We can individual observations or samples of our data $x$ into a batch matrix of $X$. This can be expressed in matrix form as:

\begin{align}
Y = X W + b
\end{align}

In our specific example, the minibatch size $m$ is 100, the dimension of the data is 28 $\times$ 28 = 784, and the number of classes $c$ is 10. Below is an image depicting the transformation of a batch of data $X$ into a set of corresponding predictions $Y$:

PyTorch offers a pre-defined [`torch.nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer that computes a linear transformation of the input data. We can use this layer to define the parameters of our logistic regression model. Typically, after the `Linear` layer we would implement a softmax activation function that converts our output to a multinomial probability distribution. However, PyTorch has a way of building this calculation into its loss functions. This helps model stability during training. Therefore, we won't add a softmax activation function for our models and instead leave that calculation to PyTorch's loss functions.

```python
# Logistic regression model
class LogisticRegression(nn.Module):
    """
    PyTorch implementation of a logistic regression model.

    Parameters
    ----------
    input_dim : int
        Size/dimensionality of the model input data.
    output_dim : int
        Size/dimensionality of the model output.

    Attributes
    ----------
    input_dim : int
        Size/dimensionality of the model input data.
    output_dim : int
        Size/dimensionality of the model output.
    linear : torch.nn.Linear
        Fully-connected/linear layer.
    """

    # Logistic regression instantiation method
    def __init__(self, input_dim, output_dim):
        """
        Logistic regression model instantiation method.

        Parameters
        ----------
        input_dim : int
            Size/dimensionality of the model input data.
        output_dim : int
            Size/dimensionality of the model output.

        Attributes
        ----------
        input_dim : int
            Size/dimensionality of the model input data.
        output_dim : int
            Size/dimensionality of the model output.
        linear : torch.nn.Linear
            Fully-connected/linear layer.
        """

        # Inherit from torch.nn.Module
        super(LogisticRegression, self).__init__()

        # Assign logistic regression parameters to model attributes
        self.input_dim = input_dim
        self.output_dim = output_dim

        self.linear = nn.Linear(in_features=input_dim,
                                out_features=output_dim,
                                bias=True)

    # Logistic regression forward pass method
    def forward(self, x):
        """
        Logistic regression forward pass method.

        Parameters
        ----------
        x : torch.Tensor()
            Tensor of input data.

        Returns
        -------
        logits : torch.Tensor()
             Raw model predictions (not passed through sigmoid or softmax function).
        """

        # Calculate model predictions/logits
        logits = self.linear(x)

        # Return model output
        return logits

```

### 3.2. Instantiate model

To instantiate our logistic regression model, we need to pass two parameters to the instantiation method `__init__()`: the model `input_dim` and model `output_dim`. `input_dim` will be the dimensionality of our flattened image data, which is 784. `output_dim` corresponds to the number of digits we are trying to classify, which is 10.

```python
# Define model input and output dimensions from data
input_dim = X_train.shape[1]             # second dimension is data dimensionality
output_dim = len(np.unique(y_train.numpy()))  # output dim. is equal to number of unique labels

# Instantiate logistic regression model
lr_model = LogisticRegression(input_dim=input_dim, output_dim=output_dim)

```

## **4. Training the Logistic Regression Model**

### 4.1. Defining a loss function

We need a way to evaluate our model predictions. This evaluation is done via a loss function, which takes the model's predictions and returns a single number summarizing model performance. The magnitude of this loss will inform how much or how little we should update the model parameters. The loss we commonly use in classification is cross-entropy, a concept from information theory. Cross-entropy not only captures whether or not model predictions are correct, it also accounts for how *confident* these predictions are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers.

To define our loss function object, we can use PyTorch's implementation of cross entropy [`torch.nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss). In addition to computing the cross entropy loss, this loss function takes logits as inputs computes the log-softmax internally, which is why we did not have to add a softmax activation at the end of our logistic regression implementation.

```python
# Instantiate cross-entropy loss function
loss_func = nn.CrossEntropyLoss()

```

### 4.2. Defining optimizer and learning rate

Now that we've defined our loss function as a way of quantifying how good or bad our model predictions are, we can improve our model by changing the parameters in a way that minimizes the loss. For neural networks, this is done via [backpropagation](https://en.wikipedia.org/wiki/Backpropagation): we take the gradient of the loss with respect to model parameters and take a "step" in the direction that reduces our loss. If we were not using a deep learning framework like PyTorch, we would have to go through and derive all the gradients ourselves by hand, then code them into our training pipeline. Luckily, Pytorch has automated gradient calculation with [`torch.autograd`](https://pytorch.org/docs/stable/autograd.html): an "automatic differentiation engine that powers neural network training" [5].

Next, we define our optimizer and the learning rate we'll use for training the model. [`torch.optim`](https://pytorch.org/docs/stable/optim.html) is a package of various, commonly-used optimization algorithms implementations. A `torch.optim` optimizer object will keep track of the current state of model parameters and update the parameters based on the gradients computed by `torch.autograd`. To construct an optimizer you must provide an iterable containing the model parameters to optimize. This can be done by calling the `parameters()` method of our model (a method that is inherited from `torch.nn.Module`). Then, you can specify optimizer-specific options such as the learning rate and weight decay. For many deep learning applications, standard [stochastic gradient descent](https://scikit-learn.org/stable/modules/sgd.html) (SGD) optimizes models during training quite well. We can use PyTorch's `torch.optim` implementation of SGD as our optimizer.

```python
# Instantiate stochastic gradient descent (SGD) optimizer
lr = 0.1
optimizer = torch.optim.SGD(lr_model.parameters(), lr=lr)

```

### 4.3. Define number of training epochs

```python
# Number of training epochs (epochs = # of passes through data)
n_epochs = 10

```

### Training logistic regression on MNIST

To train our model, we need to repeat the following steps:

1. Draw a minibatch.
2. Reset all gradients to zero in the buffers.
3. Perform the forward pass (compute prediction, calculate loss).
4. Perform the backward pass (compute gradients with respects to loss, optimize parameters with SGD).

Performing these steps for the entirety of a dataset once is referred to as an epoch. We'll accomplish the above steps using the following PyTorch functionalities:

1. Draw a minibatches by iterating over our training `DataLoader` instance. Remember, our custom PyTorch data loader takes care of batching the data, shuffling the data, and loading the data in parallel.
2. Reset all gradients to zero in the buffers using the `zero_grad()` method of our optimizer. We need to set the gradients to zero before performing backpropragation because PyTorch accumulates the gradients on subsequent backward passes if `zero_grad()` is not called. Accumulation of gradients is convenient for models such as recurrent neural networks (RNNs); however, for this tutorial we'll zero out gradients before processing each minibatch of data.
3. Perform the forward pass by providing the minibatch of data to our model's `forward()` method.
4. Perform the backward pass by calling the `backward()` method of our model's computed loss and the `step()` method of our optimizer. The `backward()` method of our model computes the parameter gradients with respects to the calculated loss, and the `step()` method of our optimizer updates the parameters according to the calculated gradients.

```python
# Get weights
weights_before = lr_model.linear.weight.detach().numpy().copy()

# Time model training
print('\nTraining logistic regression model...\n')
time_start = time.time()

# Place model in training mode
# .train() method affects operations such as dropout and batch normalization
lr_model.train()

# Train model/iterate over epochs
for epoch in range(n_epochs):
    # Initialize epoch metrics variables
    n_obs = 0
    loss_sum = 0
    n_correct = 0

    # Iterate through training data mini-batches
    for images_batch, labels_batch in train_loader:
        # Zero out the gradients
        optimizer.zero_grad()

        # Forward pass
        y_pred = lr_model(images_batch)         # model predictions
        loss = loss_func(y_pred, labels_batch)  # loss function evaluation

        # Backward pass
        loss.backward()    # backpropagation
        optimizer.step()   # update parameters according to learning rate, gradients

        # Update epoch metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        loss_sum += n_batch * loss.item()
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

    # Calculate epoch training loss and training accuracy
    loss_train = loss_sum / n_obs
    acc_train = n_correct / n_obs

    # Display training progress
    prog_disp_freq = 1   # frequency of training progress display
    if (((epoch + 1) % prog_disp_freq) == 0) | (epoch == 0) | ((epoch + 1) == n_epochs):
        print('  E%02d | train loss: %s | train acc.: %s' %
              (epoch + 1, '{:.4f}'.format(loss_train), '{:.4f}'.format(acc_train)))

# Time model training
time_end = time.time()
print('\nLogistic regression model training complete.\n')
print('Time to train logistic regression model:  %.1f s\n' % (time_end - time_start))

# Get weights after training
weights_after = lr_model.linear.weight.detach().numpy().copy()

```

### 4.4. Visualize logistic regression weights

One of the nice things about logistic regression is that since it is a linear model we can interpret what the model has learned by plotting the weights before and after training.

Before training, the model weights for each digit/output look like a static TV screen. Since the weights were randomly initialized, it is expected that the model parameters will appear random before training on any data. After training, we can see the model learned a general template for each digit. Remember that logistic regression takes a dot product between the weights of each digit and input. Therefore, the more the input matches the template or weights for a given digit, the higher the value of the dot product for that digit will be which makes the model more likely to predict that digit.

```python
# Plot logistic regression weights before and after training
fig1, ax1 = plt.subplots(2, 10, figsize=(14, 2.6))
for digit in range(10):
    ax1[0, digit].imshow(np.reshape(weights_before[digit, :], newshape=(28, 28)), cmap='gray')
    ax1[1, digit].imshow(np.reshape(weights_after[digit, :], newshape=(28, 28)), cmap='gray')
    ax1[0, digit].axis('off')
    ax1[1, digit].axis('off')
plt.show()

```

## **5. Evaluating Logistic Regression Model on Test Set**

### 5.1. Evaluate model on test set

Once our model has been trained, we now need to evaluate it on a set of hold-out data, or a test set. Evaluating trained models on a test set is a critical step of machine learning pipelines, as this gives us an idea of how well our model will generalize in the real world to new, never-seen-before data.

First, we need to put our model in evaluation mode using the `eval()` method. This method affects operations with different training and test-time behavior such as dropout and batch normalization. Although we don't use any layers in this tutorial that would be affected by this method, it's a good thing to get into the habit of doing.

Additionally, since we aren't training our model any more via backpropagation, we don't need to compute a loss or calculate gradients. We can use the [`torch.no_grad()`](https://pytorch.org/docs/stable/generated/torch.no_grad.html) context-manager that disables gradient calculation. Disabling gradient calculation is useful for model inference, when you are sure that you will not call Tensor.backward(). Using `torch.no_grad()` will help reduce memory consumption for computations that would otherwise require gradients during training [6].

```python
# Place model in evaluation mode
# .eval() method affects operations such as dropout and batch normalization
lr_model.eval()

# Initialize test set metrics variables
n_obs = 0
n_correct = 0

# Disable gradient calculation
with torch.no_grad():
    # Iterate through test data mini-batches
    for images_batch, labels_batch in test_loader:
        # Forward pass
        y_pred = lr_model(images_batch)  # model predictions

        # Update test set metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

# Calculate test accuracy
acc_test = n_correct / n_obs

# Display test accuracy
print('\nLogistic regression model MNIST test acc.:  %.4f\n' % (acc_test))

```

## **6. Training and Evaluating Multi-Layer Perceptron on MNIST Dataset**

### 6.1. Defining MLP model

Similarly to how we previously defined our logistic regression model by using PyTorch's object-oriented style, we can implement a neural network/[multi-layer perceptron](https://machinelearningmastery.com/neural-networks-crash-course/) (MLP) with just a few tweaks. Instead of just one `Linear` layer, this time we'll use two. Additionally, after the first `Linear` layer we'll apply a non-linear [activation function](https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/). Activation functions are a critical part of neural network design, and help give our neural network model added expressivity for modelling complex, non-linear relationships. The de facto nonlinearity used in neural networks is the [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) (ReLU). We can implement this ReLU activation function as a layer in our MLP by using PyTorch's [`torch.nn.ReLU`](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) layer.

*Note: multi-layer perceptron and artifical neural network (ANN) are often used interchangeably to describe a feed-forward, fully-connected neural network with many layers.*

```python
# Multi-layer perceptron model
class MLP(nn.Module):
    """
    PyTorch implementation of a multi-layer perceptron model.

    Parameters
    ----------
    input_dim : int
        Size/dimensionality of the model input data.
    output_dim : int
        Size/dimensionality of the model output.
    hidden_dim : int
        Size/dimensionality of the hidden layer.

    Attributes
    ----------
    input_dim : int
        Size/dimensionality of the model input data.
    output_dim : int
        Size/dimensionality of the model output.
    hidden_dim : int
        Size/dimensionality of the hidden layer.
    linear_1 : torch.nn.Linear
        Fully-connected/linear neural network layer.
    linear_2 : torch.nn.Linear
        Fully-connected/linear neural network layer.
    relu : torch.nn.ReLU
        Rectified linear unit activation function.
    """

    # MLP instantiation method
    def __init__(self, input_dim, output_dim, hidden_dim=100):
        """
        Model instantiation method.

        Parameters
        ----------
        input_dim : int
            Size/dimensionality of the model input data.
        output_dim : int
            Size/dimensionality of the model output.
        hidden_dim : int
            Size/dimensionality of the hidden layer.

        Attributes
        ----------
        input_dim : int
            Size/dimensionality of the model input data.
        output_dim : int
            Size/dimensionality of the model output.
        hidden_dim : int
            Size/dimensionality of the hidden layer.
        linear_1 : torch.nn.Linear
            Fully-connected/linear neural network layer.
        linear_2 : torch.nn.Linear
            Fully-connected/linear neural network layer.
        relu : torch.nn.ReLU
            Rectified linear unit activation function.
        """

        # Inherit from torch.nn.Module
        super(MLP, self).__init__()

        # Assign MLP parameters to model attributes
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.output_dim = output_dim

        # Define MLP layers
        self.linear_1 = nn.Linear(in_features=input_dim,
                                  out_features=hidden_dim,
                                  bias=True)
        self.linear_2 = nn.Linear(in_features=hidden_dim,
                                  out_features=output_dim,
                                  bias=True)

        # Define MLP activation function
        self.relu = nn.ReLU()

    # MLP forward pass method
    def forward(self, x):
        """
        MLP forward pass method.

        Parameters
        ----------
        x : torch.Tensor()
            Tensor of input data.

        Returns
        -------
        logits : torch.Tensor()
             Raw model predictions (not passed through sigmoid or softmax function).
        """

        # Forward pass through model
        x = self.linear_1(x)       # first fully-connected layer
        x = self.relu(x)           # non-linear activation transformation
        logits = self.linear_2(x)  # second fully-connected layer

        # Return model output
        return logits

```

### 6.2. Instantiate MLP model

```python
# Define model input and output dimensions from data
input_dim = X_train.shape[1]                  # second dimension is data dimensionality
output_dim = len(np.unique(y_train.numpy()))  # output dim. is equal to number of unique labels

# Instantiate multi-layer perceptron model
mlp_model = MLP(input_dim=input_dim, output_dim=output_dim)

```

### 6.3. Re-define optimizer with MLP model parameters

Since a PyTorch optimizer is an object that keeps track of and updates model parameters, we need to re-define our optimizer with the parameters of our newly defined MLP model. We can keep the same learning rate of 0.1 as last time.

```python
# Instantiate stochastic gradient descent (SGD) optimizer
lr = 0.1
optimizer = torch.optim.SGD(mlp_model.parameters(), lr=lr)

```

### 6.4. Train MLP model on training data

Now we train our MLP model on MNIST. Ideally, since we have now defined a non-linear model with added expressivity by including a hidden layer and a non-linear activation function following this hidden layer, we would like to see better classification performance than that achieved by logistic regression. As we will see, these modifications do indeed result in better hand-written digit classification performance.

Even though we changed the type of model, we can keep the same exact training loop we developed for training logistic regression on MNIST.

```python
# Number of training epochs (epochs = # of passes through data)
n_epochs = 10

# Time model training
print('\nTraining MLP model...\n')
time_start = time.time()

# Place model in training mode
# .train() method affects operations such as dropout and batch normalization
mlp_model.train()

# Train model/iterate over epochs
for epoch in range(n_epochs):
    # Initialize epoch metrics variables
    n_obs = 0
    loss_sum = 0
    n_correct = 0

    # Iterate through training data mini-batches
    for images_batch, labels_batch in train_loader:
        # Zero out the gradients
        optimizer.zero_grad()

        # Forward pass
        y_pred = mlp_model(images_batch)         # model predictions
        loss = loss_func(y_pred, labels_batch)  # loss function evaluation

        # Backward pass
        loss.backward()    # backpropagation
        optimizer.step()   # update parameters according to learning rate, gradients

        # Update epoch metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        loss_sum += n_batch * loss.item()
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

    # Calculate epoch training loss and training accuracy
    loss_train = loss_sum / n_obs
    acc_train = n_correct / n_obs

    # Display training progress
    prog_disp_freq = 1   # frequency of training progress display
    if (((epoch + 1) % prog_disp_freq) == 0) | (epoch == 0) | ((epoch + 1) == n_epochs):
        print('  E%02d | train loss: %s | train acc.: %s' %
              (epoch + 1, '{:.4f}'.format(loss_train), '{:.4f}'.format(acc_train)))

# Time model training
time_end = time.time()
print('\nMLP model training complete.\n')
print('Time to train MLP model:  %.1f s\n' % (time_end - time_start))

```

### 6.5. Evaluate MLP model on test set

We'll now evaluate our MLP model on the test set. The added expressivity and complexity of our model can be a double-edged sword. While this added complexity gives our model the capacity to learn more complex, non-linear relationships, we must now also be weary of [overfitting](https://www.ibm.com/cloud/learn/overfitting). Overfitting occurs when a model learns the information represented in training data to the point where it's performance on hold-out data is negatively influenced. Deep learning models can have millions (sometimes [billions](https://developer.nvidia.com/blog/openai-presents-gpt-3-a-175-billion-parameters-language-model/)!) of trainable parameters, which means that the number of model parameters can be greater than the number of data points on which the model is trained. These "over-parametrized" models can be prone to overfitting, especially in cases of small data. Here, we're hoping that our test accuracy is similar to our training accuracy of around 96%-97%. As we'll see, our model did not overfit the training set and achieves a comparable test set accuracy.

```python
# Place model in evaluation mode
# .eval() method affects operations such as dropout and batch normalization
mlp_model.eval()

# Initialize test set metrics variables
n_obs = 0
n_correct = 0

# Disable gradient calculation
with torch.no_grad():
    # Iterate through test data mini-batches
    for images_batch, labels_batch in test_loader:
        # Forward pass
        y_pred = mlp_model(images_batch)  # model predictions

        # Update test set metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

# Calculate test accuracy
acc_test = n_correct / n_obs

# Display test accuracy
print('\nMLP model MNIST test acc.:  %.4f\n' % (acc_test))

```

## **7. Full Pipeline** - Training PyTorch Model on MNIST Dataset

Finally, we can combine our entire deep learning pipeline into a single cell.

In practice, when using Jupyter notebooks you shouldn't combine all of your code into a single cell, as this defeats the purpose of the modular nature of Jupyter notebooks. However, it can be helpful to see the entirety of a basic deep learning pipeline from beginning to end in code (without the extra bloat of my writing).

```python
# -----------------
# IMPORT STATEMENTS
# -----------------

# Define random state
random_state = 0

# Import libraries, modules, and functions
import time
import numpy as np
import matplotlib.pyplot as plt

# PyTorch imports
import torch
import torch.nn as nn

# -----------------
# CLASS DEFINITIONS
# -----------------

# MNIST PyTorch dataset object
class MNISTDataset(torch.utils.data.Dataset):
    # MNIST PyTorch dataset instantiation method
    def __init__(self, X, y):
        # Assign attributes
        self.X = X         # flattened image data
        self.y = y         # image labels
        self.len = len(X)  # length of dataset

    # Dataset length method
    def __len__(self):
        # Return length of dataset
        return self.len

    # Dataset indexing method
    def __getitem__(self, idx):
        # Retrieve sample of data indexed by idx
        X_samp = self.X[idx, :]
        y_samp = self.y[idx]

        # Return data sample
        return X_samp, y_samp

# Multi-layer perceptron model
class MLP(nn.Module):
    # MLP instantiation method
    def __init__(self, input_dim, output_dim, hidden_dim=100):
        # Inherit from torch.nn.Module
        super(MLP, self).__init__()

        # Assign MLP parameters to model attributes
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.output_dim = output_dim

        # Define MLP layers
        self.linear_1 = nn.Linear(in_features=input_dim,
                                  out_features=hidden_dim,
                                  bias=True)
        self.linear_2 = nn.Linear(in_features=hidden_dim,
                                  out_features=output_dim,
                                  bias=True)

        # Define MLP activation function
        self.relu = nn.ReLU()

    # MLP forward pass method
    def forward(self, x):
        # Forward pass through model
        x = self.linear_1(x)       # first fully-connected layer
        x = self.relu(x)           # non-linear activation transformation
        logits = self.linear_2(x)  # second fully-connected layer

        # Return model output
        return logits

# ---------------------------
# LOADING AND FORMATTING DATA
# ---------------------------

# Load MNIST dataset
X_raw_train, y_train = torch.load('../datasets/MNIST/processed/training.pt')
X_raw_test, y_test = torch.load('../datasets/MNIST/processed/test.pt')

# Convert image tensor type to float
X_raw_train = X_raw_train.float()
X_raw_test = X_raw_test.float()

# Scale image pixel intensities between 0 and 1
X_train = X_raw_train / 255.0
X_test = X_raw_test / 255.0

# Reshape and "flatten" image data
X_train = X_train.view(X_train.shape[0], -1)
X_test = X_test.view(X_test.shape[0], -1)

# -------------------------------------------
# INSTANTIATE DATASET AND DATA LOADER OBJECTS
# -------------------------------------------

# Instantiate MNIST training and test PyTorch dataset objects
train_dataset = MNISTDataset(X_train, y_train)
test_dataset = MNISTDataset(X_test, y_test)

# Load the data
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=100, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=100, shuffle=False)

# -----------------------------------------------
# INSTANTIATE MODEL, LOSS FUNCTION, AND OPTIMIZER
# -----------------------------------------------

# Define model input and output dimensions from data
input_dim = X_train.shape[1]                  # second dimension is data dimensionality
output_dim = len(np.unique(y_train.numpy()))  # output dim. is equal to number of unique labels

# Instantiate model
# model = LogisticRegression(input_dim=input_dim, output_dim=output_dim)
model = MLP(input_dim=input_dim, output_dim=output_dim)

# Loss and Optimizer
loss_func = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# ----------------------------
# TRAIN MODEL ON TRAINING DATA
# ----------------------------

# Define number of training epochs
n_epochs = 10

# Time model training
print('\nTraining model...\n')
time_start = time.time()

# Place model in training mode
# .train() method affects operations such as dropout and batch normalization
model.train()

# Train model/iterate over epochs
for epoch in range(n_epochs):
    # Initialize epoch metrics variables
    n_obs = 0
    loss_sum = 0
    n_correct = 0

    # Iterate through training data mini-batches
    for images_batch, labels_batch in train_loader:
        # Zero out the gradients
        optimizer.zero_grad()

        # Forward pass
        y_pred = model(images_batch)            # model predictions
        loss = loss_func(y_pred, labels_batch)  # loss function evaluation

        # Backward pass
        loss.backward()   # backpropagation
        optimizer.step()  # update parameters according to learning rate, gradients

        # Update epoch metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        loss_sum += n_batch * loss.item()
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

    # Calculate epoch training loss and training accuracy
    loss_train = loss_sum / n_obs
    acc_train = n_correct / n_obs

    # Display training progress
    prog_disp_freq = 1   # frequency of training progress display
    if (((epoch + 1) % prog_disp_freq) == 0) | (epoch == 0) | ((epoch + 1) == n_epochs):
        print('  E%02d | train loss: %s | train acc.: %s' %
              (epoch + 1, '{:.4f}'.format(loss_train), '{:.4f}'.format(acc_train)))

# Time model training
time_end = time.time()
print('\nModel training complete.\n')
print('Time to train model:  %.1f s\n' % (time_end - time_start))

# ---------------------------
# EVALUATE MODEL ON TEST DATA
# ---------------------------

# Place model in evaluation mode
# .eval() method affects operations such as dropout and batch normalization
model.eval()

# Initialize test set metrics variables
n_obs = 0
n_correct = 0

# Disable gradient calculation
with torch.no_grad():
    # Iterate through test data mini-batches
    for images_batch, labels_batch in test_loader:
        # Forward pass
        y_pred = model(images_batch)  # model predictions

        # Update test set metrics variables
        n_batch = len(labels_batch)
        n_obs += n_batch
        n_correct += (y_pred.argmax(1)==labels_batch.long()).float().sum().item()

# Calculate test accuracy
acc_test = n_correct / n_obs

# Display test accuracy
print('Model MNIST test acc.:  %.4f\n' % (acc_test))

```

## **References**

[1] https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/

[2] https://pytorch.org/docs/stable/tensor_view.html

[3] https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

[4] https://pytorch.org/docs/stable/notes/modules.html

[5] https://pytorch.org/docs/stable/autograd.html, https://pytorch.org/docs/stable/optim.html

[6] https://pytorch.org/docs/stable/generated/torch.no_grad.html


## Linear Regression

$$ \bar{y} = wx + b $$

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np
import theano
from theano import tensor as T
```

```python
train_X = np.linspace(-1, 1, 101)
train_Y = 2 * train_X + 1 + np.random.randn(train_X.size) * 0.33
plt.scatter(train_X, train_Y)
```

```python
X, Y = T.scalar(), T.scalar()
X.name, Y.name = 'x', 'y'

def linear_model(X, w, b):
    return X * w + b
```

损失函数： $ C = \left|\bar{y} - y \right|^2 $

我们可以使用梯度下降法来迭代参数 w,b 的值，为此，我们将 w 和 b 设成共享变量：

```python
w = theano.shared(np.asarray(0., dtype=theano.config.floatX))
w.name = 'w'
b = theano.shared(np.asarray(0., dtype=theano.config.floatX))
b.name = 'b'

Y_bar = linear_model(X, w, b)
theano.pp(Y_bar)
```

    '((x * w) + b)'

```python
cost = T.mean(T.sqr(Y_bar - Y))
grads = T.grad(cost=cost, wrt=[w, b])
```

```python
lr = 0.01
updates = [[w, w - grads[0] * lr],
           [b, b - grads[1] * lr]]

train_model = theano.function(inputs=[X,Y],
                              outputs=cost,
                              updates=updates,
                              allow_input_downcast=True)

for i in range(100):
    for x, y in zip(train_X, train_Y):
        train_model(x, y)
```

```python
print(w.get_value())
print(b.get_value())

plt.scatter(train_X, train_Y)
plt.plot(train_X, w.get_value() * train_X + b.get_value(), 'r')
```

## Logistic Regression

$$ s(x) = \frac{1}{1 + e^{-x}} = \frac{1 + tanh\left(\frac{x}{2}\right)}{2} $$

```python
rng = np.random

# 数据大小和规模
N = 400
feats = 784

# D = (X, Y)
D = (rng.randn(N, feats), rng.randint(size=N, low=0, high=2))
```

```python
x = T.matrix('x')
y = T.vector('y')

# 要更新的变量：
w = theano.shared(rng.randn(feats), name='w')
b = theano.shared(0., name='b')
```

```python
h = 1 / (1 + T.exp(-T.dot(x, w) - b))
prediction = h > 0.5
```

```python
cost = - T.mean(y * T.log(h) + (1 - y) * T.log(1 - h)) + 0.01 * T.sum(w ** 2)  # 正则项，防止过拟合
gw, gb = T.grad(cost, [w, b])
```

```python
train = theano.function(inputs=[x, y],
                        outputs=cost,
                        updates=[[w, w - 0.1 * gw], [b, b - 0.1 * gb]],
                        allow_input_downcast=True)

predict = theano.function(inputs=[x],
                          outputs=prediction,
                          allow_input_downcast=True)

for i in range(10001):
    err = train(D[0], D[1])
    if i % 1000 == 0:
        print('iter {0:5d}, error {1}'.format(i, err))
```

    iter     0, error nan
    iter  1000, error 0.20916386518319272
    iter  2000, error 0.12385585775631414
    iter  3000, error 0.12256679622266706
    iter  4000, error 0.12254240274449477
    iter  5000, error 0.12254162133721462
    iter  6000, error 0.12254152989301592
    iter  7000, error 0.12254151063149532
    iter  8000, error 0.12254150627087208
    iter  9000, error 0.12254150527768518
    iter 10000, error 0.12254150505136419

```python
D[1]
```

    array([0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1,
           0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1,
           0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0,
           1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0,
           1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1,
           0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0,
           1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1,
           0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1,
           0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0,
           0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0,
           0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1,
           0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
           0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1,
           1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0,
           0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0,
           1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0,
           1, 0, 1, 0, 1, 0, 0, 0, 0])

```python
predict(D[0])
```

    array([False,  True,  True, False,  True,  True,  True, False, False,
            True, False,  True,  True,  True,  True, False, False, False,
           False, False,  True, False,  True, False, False, False,  True,
           False,  True, False,  True,  True,  True, False, False,  True,
            True, False, False,  True, False, False, False, False, False,
           False, False, False,  True, False, False,  True,  True,  True,
           False, False, False,  True, False, False, False,  True, False,
            True, False,  True, False, False,  True, False,  True,  True,
            True, False,  True, False,  True,  True, False, False,  True,
            True, False,  True, False, False,  True, False, False,  True,
           False, False,  True,  True, False, False,  True, False,  True,
           False, False,  True, False, False,  True, False,  True, False,
            True,  True,  True,  True,  True,  True, False,  True, False,
            True, False, False,  True,  True, False,  True,  True, False,
            True, False, False, False, False,  True, False, False,  True,
           False,  True,  True, False,  True,  True, False, False, False,
           False,  True, False,  True,  True, False, False,  True,  True,
            True, False,  True,  True, False,  True, False, False,  True,
           False,  True,  True, False,  True, False, False,  True, False,
            True,  True,  True,  True, False,  True,  True,  True, False,
           False, False, False,  True, False,  True, False,  True, False,
            True,  True, False, False,  True, False, False,  True, False,
            True,  True, False,  True,  True,  True, False,  True,  True,
           False,  True, False, False,  True, False, False, False, False,
            True,  True, False,  True,  True,  True,  True,  True, False,
            True,  True, False,  True, False, False,  True,  True, False,
            True, False,  True, False, False,  True,  True,  True, False,
           False,  True,  True, False, False, False, False,  True, False,
           False, False,  True,  True, False, False,  True, False, False,
            True, False, False, False, False,  True, False,  True, False,
           False,  True,  True,  True,  True,  True, False,  True,  True,
           False, False, False,  True,  True,  True, False,  True,  True,
            True, False, False,  True,  True, False, False, False, False,
           False,  True, False,  True,  True, False, False,  True,  True,
            True, False, False, False, False, False,  True,  True,  True,
           False, False,  True,  True, False, False,  True,  True,  True,
           False,  True,  True,  True,  True, False,  True,  True,  True,
           False,  True,  True, False, False, False,  True, False, False,
           False,  True, False, False,  True, False, False,  True, False,
           False, False,  True,  True,  True, False, False, False,  True,
           False,  True,  True,  True,  True,  True,  True, False,  True,
           False,  True, False,  True, False,  True, False,  True, False,
           False,  True, False,  True, False, False, False, False,  True,
            True,  True,  True, False,  True, False,  True, False,  True,
           False, False, False, False], dtype=bool)

## Softmax Regression

[MNIST 数据集](http://yann.lecun.com/exdb/mnist/) 是一个手写数字组成的数据集，现在被当作一个机器学习算法评测的基准数据集。

这是一个下载并解压数据的脚本：

```python
%%file download_mnist.py
import os
import os.path
import urllib
import gzip
import shutil

if not os.path.exists('mnist'):
    os.mkdir('mnist')

def download_and_gzip(name):
    if not os.path.exists(name + '.gz'):
        urllib.request.urlretrieve('http://yann.lecun.com/exdb/' + name + '.gz', name + '.gz')
    if not os.path.exists(name):
        with gzip.open(name + '.gz', 'rb') as f_in, open(name, 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)

download_and_gzip('mnist/train-images-idx3-ubyte')
download_and_gzip('mnist/train-labels-idx1-ubyte')
download_and_gzip('mnist/t10k-images-idx3-ubyte')
download_and_gzip('mnist/t10k-labels-idx1-ubyte')
```

    Writing download_mnist.py

```python
%run download_mnist.py
```

```python
%%file load.py
import numpy as np
import os

datasets_dir = './'

def one_hot(x,n):
	if type(x) == list:
		x = np.array(x)
	x = x.flatten()
	o_h = np.zeros((len(x),n))
	o_h[np.arange(len(x)),x] = 1
	return o_h

def mnist(ntrain=60000,ntest=10000,onehot=True):
	data_dir = os.path.join(datasets_dir,'mnist/')
	fd = open(os.path.join(data_dir,'train-images-idx3-ubyte'))
	loaded = np.fromfile(file=fd,dtype=np.uint8)
	trX = loaded[16:].reshape((60000,28*28)).astype(float)

	fd = open(os.path.join(data_dir,'train-labels-idx1-ubyte'))
	loaded = np.fromfile(file=fd,dtype=np.uint8)
	trY = loaded[8:].reshape((60000))

	fd = open(os.path.join(data_dir,'t10k-images-idx3-ubyte'))
	loaded = np.fromfile(file=fd,dtype=np.uint8)
	teX = loaded[16:].reshape((10000,28*28)).astype(float)

	fd = open(os.path.join(data_dir,'t10k-labels-idx1-ubyte'))
	loaded = np.fromfile(file=fd,dtype=np.uint8)
	teY = loaded[8:].reshape((10000))

	trX = trX/255.
	teX = teX/255.

	trX = trX[:ntrain]
	trY = trY[:ntrain]

	teX = teX[:ntest]
	teY = teY[:ntest]

	if onehot:
		trY = one_hot(trY, 10)
		teY = one_hot(teY, 10)
	else:
		trY = np.asarray(trY)
		teY = np.asarray(teY)

	return trX,teX,trY,teY
```

    Writing load.py

```python
from load import mnist

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

A = T.matrix()

B = T.nnet.softmax(A)

test_softmax = theano.function([A], B)

a = floatX(np.random.rand(3, 4))

b = test_softmax(a)

print(b.shape)

# 行和
print(b.sum(1))
```

    (3, 4)
    [ 1.  1.  1.]

```python
def model(X, w):
    return T.nnet.softmax(T.dot(X, w))

trX, teX, trY, teY = mnist(onehot=True)

X = T.fmatrix()
Y = T.fmatrix()

w = init_weights((784, 10))

py_x = model(X, w)
y_pred = T.argmax(py_x, axis=1)

# 损失函数为多类的交叉熵，这个在 theano 中也被定义好了：
cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))
gradient = T.grad(cost=cost, wrt=w)
update = [[w, w - gradient * 0.05]]

train = theano.function(inputs=[X, Y], outputs=cost, updates=update, allow_input_downcast=True)
predict = theano.function(inputs=[X], outputs=y_pred, allow_input_downcast=True)
```

```python
for i in range(100):
    for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)):
        cost = train(trX[start:end], trY[start:end])
    print("{0:03d}".format(i), np.mean(np.argmax(teY, axis=1) == predict(teX)))
```

    000 0.8857
    001 0.8977
    002 0.9045
    003 0.9074
    004 0.9099
    005 0.9111
    006 0.9135
    007 0.9143
    008 0.9151
    009 0.9159
    010 0.9164
    011 0.9167
    012 0.9169
    013 0.9172
    014 0.9177
    015 0.9177
    016 0.9182
    017 0.9188
    018 0.919
    019 0.919
    020 0.9197
    021 0.9199
    022 0.9199
    023 0.9204
    024 0.9205
    025 0.9205
    026 0.9209
    027 0.9211
    028 0.9211
    029 0.9214
    030 0.9213
    031 0.9216
    032 0.9215
    033 0.9218
    034 0.9219
    035 0.9218
    036 0.9217
    037 0.9218
    038 0.9221
    039 0.9222
    040 0.9223
    041 0.9225
    042 0.9225
    043 0.9225
    044 0.9224
    045 0.9225
    046 0.9225
    047 0.9224
    048 0.923
    049 0.9231
    050 0.9233
    051 0.9235
    052 0.9236
    053 0.9236
    054 0.9236
    055 0.9235
    056 0.9236
    057 0.9239
    058 0.9241
    059 0.9241
    060 0.9242
    061 0.9242
    062 0.9242
    063 0.9242
    064 0.9245
    065 0.9245
    066 0.9245
    067 0.9243
    068 0.9244
    069 0.9243
    070 0.9245
    071 0.9244
    072 0.9245
    073 0.9244
    074 0.9245
    075 0.9244
    076 0.9243
    077 0.9244
    078 0.9244
    079 0.9243
    080 0.9243
    081 0.9247
    082 0.9248
    083 0.9248
    084 0.9248
    085 0.9248
    086 0.9246
    087 0.9246
    088 0.9246
    089 0.9246
    090 0.9247
    091 0.9248
    092 0.9248
    093 0.9251
    094 0.925
    095 0.925
    096 0.925
    097 0.925
    098 0.9249
    099 0.9248

## ANN(人工神经网络)

我们在这里使用一个简单的三层神经网络：输入 - 隐层 - 输出。

对于网络的激活函数，隐层用sigmoid函数，输出层用softmax函数，其模型如下：

$$
\begin{aligned}
    h &= \sigma (W_h X) \\
    o &= \text{softmax} (W_o h)
\end{aligned}
$$

```python
def model(X, w_h, w_o):
    """
    input:
        X: input data
        w_h: hidden unit weights
        w_o: output unit weights
    output:
        Y: probability of y given x
    """
    # 隐层
    h = T.nnet.sigmoid(T.dot(X, w_h))
    # 输出层
    pyx = T.nnet.softmax(T.dot(h, w_o))
    return pyx

def sgd(cost, params, lr=0.05):
    """
    input:
        cost: cost function
        params: parameters
        lr: learning rate
    output:
        update rules
    """
    grads = T.grad(cost=cost, wrt=params)
    updates = []
    for p, g in zip(params, grads):
        updates.append([p, p - g * lr])
    return updates

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

X = T.matrix()
Y = T.matrix()

w_h = init_weights((784, 625))
w_o = init_weights((625, 10))

py_x = model(X, w_h, w_o)
y_x = T.argmax(py_x, axis=1)

cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))
updates = sgd(cost, [w_h, w_o])

train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)
predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)

trX, teX, trY, teY = mnist(onehot=True)
```

```python
for i in range(100):
    for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)):
        cost = train(trX[start:end], trY[start:end])
    print("{0:03d}".format(i), np.mean(np.argmax(teY, axis=1) == predict(teX)))
```

    000 0.7079
    001 0.829
    002 0.8671
    003 0.8831
    004 0.8894
    005 0.8954
    006 0.8981
    007 0.9016
    008 0.9043
    009 0.9068
    010 0.9093
    011 0.9114
    012 0.9129
    013 0.9141
    014 0.9153
    015 0.9159
    016 0.9163
    017 0.917
    018 0.9176
    019 0.9184
    020 0.9188
    021 0.919
    022 0.9195
    023 0.9197
    024 0.9204
    025 0.9208
    026 0.921
    027 0.9217
    028 0.922
    029 0.9228
    030 0.9238
    031 0.9236
    032 0.9241
    033 0.9251
    034 0.9258
    035 0.9262
    036 0.9265
    037 0.9271
    038 0.9278
    039 0.9286
    040 0.9294
    041 0.9295
    042 0.9299
    043 0.93
    044 0.9304
    045 0.9307
    046 0.9317
    047 0.9322
    048 0.9331
    049 0.9334
    050 0.9338
    051 0.9352
    052 0.9359
    053 0.9366
    054 0.9374
    055 0.9379
    056 0.9385
    057 0.939
    058 0.9395
    059 0.9399
    060 0.9408
    061 0.9414
    062 0.9418
    063 0.9423
    064 0.9429
    065 0.9434
    066 0.944
    067 0.9447
    068 0.9451
    069 0.9457
    070 0.9464
    071 0.9464
    072 0.947
    073 0.9474
    074 0.9478
    075 0.9485
    076 0.9488
    077 0.9491
    078 0.95
    079 0.9507
    080 0.951
    081 0.9513
    082 0.9517
    083 0.952
    084 0.9527
    085 0.9535
    086 0.9536
    087 0.9543
    088 0.9545
    089 0.9548
    090 0.955
    091 0.9554
    092 0.9558
    093 0.9563
    094 0.9569
    095 0.9572
    096 0.9575
    097 0.9576
    098 0.9582
    099 0.9584

## 更复杂的神经网络

之前我们采用的的激活函数是 sigmoid，现在我们使用 rectify 激活函数。

这可以使用 `T.nnet.relu(x, alpha=0)` 来实现，它本质上相当于：`T.switch(x > 0, x, alpha * x)`，而 rectify 函数的定义为：

$$
\text{rectify}(x) = \left\{
\begin{aligned}
x, & \ x > 0 \\
0, & \ x < 0
\end{aligned}\right.
$$

之前我们构造的是一个单隐层的神经网络结构，现在我们构造一个双隐层的结构即“输入-隐层1-隐层2-输出”的全连接结构。

$$
\begin{aligned}
& h_1 =  \text{rectify}(W_{h_1} \ x) \\
& h_2 =  \text{rectify}(W_{h_2} \ h_1) \\
& o =  \text{softmax}(W_o h_2)
\end{aligned}
$$

Theano 自带的 `T.nnet.softmax()` 的 GPU 实现目前似乎有 bug 会导致梯度溢出的问题，因此自定义了 softmax 函数：

```python
def dropout(X, prob=0.):
    if prob > 0:
        X *= srng.binomial(X.shape, p=1-prob, dtype = theano.config.floatX)
        X /= 1 - prob
    return X

def softmax(X):
    e_x = T.exp(X - X.max(axis=1).dimshuffle(0, 'x'))
    return e_x / e_x.sum(axis=1).dimshuffle(0, 'x')

def model(X, w_h1, w_h2, w_o, p_drop_input, p_drop_hidden):
    """
    input:
        X:             input data
        w_h1:          weights input layer to hidden layer 1
        w_h2:          weights hidden layer 1 to hidden layer 2
        w_o:           weights hidden layer 2 to output layer
        p_drop_input:  dropout rate for input layer
        p_drop_hidden: dropout rate for hidden layer
    output:
        h1:    hidden layer 1
        h2:    hidden layer 2
        py_x:  output layer
    """
    X = dropout(X, p_drop_input)
    h1 = T.nnet.relu(T.dot(X, w_h1))

    h1 = dropout(h1, p_drop_hidden)
    h2 = T.nnet.relu(T.dot(h1, w_h2))

    h2 = dropout(h2, p_drop_hidden)
    py_x = softmax(T.dot(h2, w_o))
    return h1, h2, py_x

def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

w_h1 = init_weights((784, 625))
w_h2 = init_weights((625, 625))
w_o = init_weights((625, 10))

X = T.matrix()
Y = T.matrix()
```

定义更新的规则，之前我们使用的是简单的 SGD，这次我们使用 RMSprop 来更新，其规则为：
$$
\begin{align}
MS(w, t) &= \rho MS(w, t-1) + (1-\rho) \left(\left.\frac{\partial E}{\partial w}\right|_{w(t-1)}\right)^2 \\
w(t) &= w(t-1) - \alpha \left.\frac{\partial E}{\partial w}\right|_{w(t-1)} / \sqrt{MS(w, t)}
\end{align}
$$

```python
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
srng = RandomStreams()

def RMSprop(cost, params, accs, lr=0.001, rho=0.9, epsilon=1e-6):
    grads = T.grad(cost=cost, wrt=params)
    updates = []
    for p, g, acc in zip(params, grads, accs):
        acc_new = rho * acc + (1 - rho) * g ** 2
        gradient_scaling = T.sqrt(acc_new + epsilon)
        g = g / gradient_scaling
        updates.append((acc, acc_new))
        updates.append((p, p - lr * g))
    return updates

# 有 dropout，用来训练
noise_h1, noise_h2, noise_py_x = model(X, w_h1, w_h2, w_o, 0.2, 0.5)
cost = T.mean(T.nnet.categorical_crossentropy(noise_py_x, Y))
params = [w_h1, w_h2, w_o]
accs = [theano.shared(p.get_value() * 0.) for p in params]
updates = RMSprop(cost, params, accs, lr=0.001)
# 训练函数
train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)

# 没有 dropout，用来预测
h1, h2, py_x = model(X, w_h1, w_h2, w_o, 0., 0.)
# 预测的结果
y_x = T.argmax(py_x, axis=1)
predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)
```

```python
trX, teX, trY, teY = mnist(onehot=True)

for i in range(50):
    for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)):
        cost = train(trX[start:end], trY[start:end])
    print("iter {:03d} accuracy:".format(i + 1), np.mean(np.argmax(teY, axis=1) == predict(teX)))
```

    iter 001 accuracy: 0.9412
    iter 002 accuracy: 0.9633
    iter 003 accuracy: 0.9721
    iter 004 accuracy: 0.9745
    iter 005 accuracy: 0.9741
    iter 006 accuracy: 0.9775
    iter 007 accuracy: 0.9762
    iter 008 accuracy: 0.981
    iter 009 accuracy: 0.9808
    iter 010 accuracy: 0.9808
    iter 011 accuracy: 0.9818
    iter 012 accuracy: 0.9828
    iter 013 accuracy: 0.9832
    iter 014 accuracy: 0.9844
    iter 015 accuracy: 0.984
    iter 016 accuracy: 0.9826
    iter 017 accuracy: 0.984
    iter 018 accuracy: 0.9852
    iter 019 accuracy: 0.9842
    iter 020 accuracy: 0.9853
    iter 021 accuracy: 0.9844
    iter 022 accuracy: 0.9843
    iter 023 accuracy: 0.9846
    iter 024 accuracy: 0.9862
    iter 025 accuracy: 0.9846
    iter 026 accuracy: 0.9854
    iter 027 accuracy: 0.9855
    iter 028 accuracy: 0.9858
    iter 029 accuracy: 0.9869
    iter 030 accuracy: 0.9872
    iter 031 accuracy: 0.9854
    iter 032 accuracy: 0.9852
    iter 033 accuracy: 0.9851
    iter 034 accuracy: 0.9874
    iter 035 accuracy: 0.9857
    iter 036 accuracy: 0.9857
    iter 037 accuracy: 0.9851
    iter 038 accuracy: 0.986
    iter 039 accuracy: 0.986
    iter 040 accuracy: 0.9855
    iter 041 accuracy: 0.9865
    iter 042 accuracy: 0.9857
    iter 043 accuracy: 0.9862
    iter 044 accuracy: 0.9859
    iter 045 accuracy: 0.9873
    iter 046 accuracy: 0.9856
    iter 047 accuracy: 0.988
    iter 048 accuracy: 0.9878
    iter 049 accuracy: 0.987
    iter 050 accuracy: 0.988

清理文件

```python
import os
import shutil
os.remove("download_mnist.py")
os.remove("load.py")
shutil.rmtree("mnist")
```

```python
from theano.tensor.nnet import conv2d
from theano.tensor.signal import pool

trX, teX, trY, teY = mnist(onehot=True)

trX = trX.reshape(-1, 1, 28, 28)
teX = teX.reshape(-1, 1, 28, 28)

def model(X, w, w2, w3, w4, p_drop_conv, p_drop_hidden):

    # X:  128 * 1 * 28 * 28
    # w:  32 * 1 * 3 * 3
    # full mode
    # l1a: 128 * 32 * (28 + 3 - 1) * (28 + 3 - 1)
    l1a = rectify(conv2d(X, w, border_mode='full'))
    # l1a: 128 * 32 * 30 * 30
    # ignore_border False
    # l1:  128 * 32 * (30 / 2) * (30 / 2)
    l1 = pool.pool_2d(l1a, (2, 2), ignore_border=False)
    l1 = dropout(l1, p_drop_conv)

    # l1:  128 * 32 * 15 * 15
    # w2:  64 * 32 * 3 * 3
    # valid mode
    # l2a: 128 * 64 * (15 - 3 + 1) * (15 - 3 + 1)
    l2a = rectify(conv2d(l1, w2))
    # l2a: 128 * 64 * 13 * 13
    # l2:  128 * 64 * (13 / 2 + 1) * (13 / 2 + 1)
    l2 = pool.pool_2d(l2a, (2, 2), ignore_border=False)
    l2 = dropout(l2, p_drop_conv)

    # l2:  128 * 64 * 7 * 7
    # w3:  128 * 64 * 3 * 3
    # l3a: 128 * 128 * (7 - 3 + 1) * (7 - 3 + 1)
    l3a = rectify(conv2d(l2, w3))
    # l3a: 128 * 128 * 5 * 5
    # l3b: 128 * 128 * (5 / 2 + 1) * (5 / 2 + 1)
    l3b = pool.pool_2d(l3a, (2, 2), ignore_border=False)
    # l3b: 128 * 128 * 3 * 3
    # l3:  128 * (128 * 3 * 3)
    l3 = T.flatten(l3b, outdim=2)
    l3 = dropout(l3, p_drop_conv)

    # l3: 128 * (128 * 3 * 3)
    # w4: (128 * 3 * 3) * 625
    # l4: 128 * 625
    l4 = rectify(T.dot(l3, w4))
    l4 = dropout(l4, p_drop_hidden)

    # l5:  128 * 625
    # w5:  625 * 10
    # pyx: 128 * 10
    pyx = softmax(T.dot(l4, w_o))
    return l1, l2, l3, l4, pyx
```

```python
X = T.ftensor4()
Y = T.fmatrix()

w = init_weights((32, 1, 3, 3))
w2 = init_weights((64, 32, 3, 3))
w3 = init_weights((128, 64, 3, 3))
w4 = init_weights((128 * 3 * 3, 625))
w_o = init_weights((625, 10))
```

```python
print("Theano Examples")
```

    Theano Examples

```python

```


# Quadratic Programming

## 1. Introduction

### 1.1 Libraries Used

For Quadratic Programming, the packages [quadprog](https://pypi.python.org/pypi/quadprog/0.1.2) and [cvxopt](http://cvxopt.org) were installed:

```bash
pip install quadprog
pip install cvxopt
```

Help for the appropriate functions are available via

```python
help(quadprog.solve_qp)
help(cvxopt.solvers.qp)
```

The remaining libraries are loaded in the code below:

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy
import cvxopt
import quadprog
from numpy.random import permutation
from sklearn import linear_model
from sympy import var, diff, exp, latex, factor, log, simplify
from IPython.display import display, Math, Latex
np.set_printoptions(precision=4,threshold=400)
%matplotlib inline
```

### 1.2 Theory

#### 1.2.1 Lagrange Multipliers

The Lagrangian is given by:

$$\mathcal{L}\left(\mathbf{w},b,\mathbf{\alpha}\right) = \frac{1}{2}\mathbf{w^T w} - \sum\limits_{n=1}^N \alpha_n\left[y_n\left(\mathbf{w^T x_n + b}\right) - 1\right]$$

The Lagrangian may be simplified by making the following substitution:

$$\mathbf{w} = \sum\limits_{n=1}^N \alpha_n y_n \mathbf{x_n}, \quad \sum\limits_{n=1}^N \alpha_n y_n = 0$$

whereby we obtain:

$$\mathcal{L}\left(\mathbf{\alpha}\right) = \sum\limits_{n=1}^N \alpha_n - \frac{1}{2}\sum\limits_{n=1}^N \sum\limits_{m=1}^N y_n y_m \alpha_n \alpha_m \mathbf{x_n^T x_m}$$

We wish to maximize the Lagrangian with respect to $\mathbf{\alpha}$ subject to the conditions: $\alpha_n \ge 0$ for:

$$n = 1, \dots, N \quad\text{and}\quad \sum\limits_{n=1}^N \alpha_n y_n = 0$$

To do this, we convert the Lagrangian to match a form that can be used with quadratic programming software packages.

$$\min\limits_\alpha \frac{1}{2}\alpha^T \left[\begin{array}{cccc}
y_1 y_1 \mathbf{x_1^T x_1} & y_1 y_2 \mathbf{x_1^T x_2} & \cdots & y_1 y_N \mathbf{x_1^T x_N}\\
y_2 y_1 \mathbf{x_2^T x_1} & y_2 y_2 \mathbf{x_2^T x_2} & \cdots & y_2 y_N \mathbf{x_2^T x_N}\\
\vdots & \vdots & & \vdots\\
y_N y_1 \mathbf{x_N^T x_1} & y_N y_2 \mathbf{x_N^T x_2} & \cdots & y_N y_N \mathbf{x_N^T x_N}\end{array}\right]\alpha + \left(-\mathbf{1^T}\right)\mathbf{\alpha}$$

i.e.

$$\min\limits_\alpha \frac{1}{2}\alpha^T \mathbf{Q} \alpha + \left(-\mathbf{1^T}\right)\mathbf{\alpha}$$

Subject to the linear constraint: $\mathbf{y^T \alpha} = 0$ and $0 \le \alpha \le \infty$.

#### 1.2.2 Quadratic Programming

In [Quadratic Programming](https://en.wikipedia.org/wiki/Quadratic_programming), the objective is to find the value of $\mathbf{x}$ that minimizes the function:

$$\frac{1}{2}\mathbf{x^T Q x + c^T x}$$

subject to the constraint:

$$\mathbf{Ax \le b}$$

The support vectors are $\mathbf{x_n}$ where $\alpha_n > 0$.

The solution to the above is calculated using a subroutine such as `solve_qp(G, a, C, b)`,
which finds the $\alpha$'s that minimize:

$$\frac{1}{2}\mathbf{x^T G x} - \mathbf{a^T x}$$

subject to the condition:

$$\mathbf{C^T x} \ge \mathbf{b}$$

The quadratic programming solver is implemented in [`solve.QP.c`](https://github.com/rmcgibbo/quadprog/blob/master/quadprog/solve.QP.c), with a Cython wrapper [`quadprog.pyx`](https://github.com/rmcgibbo/quadprog/blob/master/quadprog/quadprog.pyx).  The unit tests are in [`test_1.py`](https://github.com/rmcgibbo/quadprog/blob/master/quadprog/tests/test_1.py) which compares the solution from quadprog's `solve_qp()` with that obtained from `scipy.optimize.minimize`, and [`test_factorized.py`](https://github.com/rmcgibbo/quadprog/blob/master/quadprog/tests/test_factorized.py).

## 2. Validation

### 2.1 Is there a Validation Bias when choosing the minimum of two random variables?

Let $\text{e}_1$ and $\text{e}_2$ be independent random variables, distributed uniformly over the interval [0, 1]. Let $\text{e} = \min\left(\text{e}_1, \text{e}_2\right)$. What is the expected values of $\left(\text{e}_1, \text{e}_2, \text{e}\right)$:

```python
n_samples = 1000
e1 = np.random.random(n_samples)
e2 = np.random.random(n_samples)
e = np.vstack((e1,e2))
e = np.min(e, axis=0)
print("E(e1) = {}".format(np.mean(e1)))
print("E(e2) = {}".format(np.mean(e2)))
print("E(e ) = {}".format(np.mean(e)))
```

    E(e1) = 0.5099531103020672
    E(e2) = 0.49670171083789605
    E(e ) = 0.3390777181905127

### 2.2 Leave-one-out Cross-Validation Example

```python
from sympy import Matrix, Rational, Eq, sqrt
var('x1 x2 x3 rho')
```

    (x1, x2, x3, rho)

The [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator for the weights is:

$$w = \left(\mathbf{X^T X}\right)^{-1}\mathbf{X^T y} = \mathbf{X^\dagger}y$$

When $\mathbf{X}$ is invertible, $\mathbf{X^\dagger} = \mathbf{X^{-1}}$, so:

$$w = \mathbf{X^{-1}}y$$

Lastly, the error is given by

$$e = \left[h(x) - y\right]^2 = \left|\mathbf{w^T x - y}\right|$$

Linear model

```python
def linear_model_cv_err(x1,y1,x2,y2,x3,y3):
    X_train1 = Matrix((x2,x3))
    X_train2 = Matrix((x1,x3))
    X_train3 = Matrix((x1,x2))
    display(Math('X_1^{train} = ' + latex(X_train1) + ', ' +
                 'X_2^{train} = ' + latex(X_train2) + ', ' +
                 'X_3^{train} = ' + latex(X_train3)))
    display(Math('(X_1^{train})^{-1} = ' + latex(X_train1.inv()) + ', ' +
                 '(X_2^{train})^{-1} = ' + latex(X_train2.inv()) + ', ' +
                 '(X_3^{train})^{-1} = ' + latex(X_train3.inv()) ))
    y_train1 = Matrix((y2,y3))
    y_train2 = Matrix((y1,y3))
    y_train3 = Matrix((y1,y2))
    display(Math('y_1^{train} = ' + latex(y_train1) + ', ' +
                 'y_2^{train} = ' + latex(y_train2) + ', ' +
                 'y_3^{train} = ' + latex(y_train3)))
    w1 = X_train1.inv() * y_train1
    w2 = X_train2.inv() * y_train2
    w3 = X_train3.inv() * y_train3
    display(Math('w_1 = ' + latex(w1) + ', ' +
                 'w_2 = ' + latex(w2) + ', ' +
                 'w_3 = ' + latex(w3)))
    y_pred1 = w1.T*Matrix(x1)
    y_pred2 = w2.T*Matrix(x2)
    y_pred3 = w3.T*Matrix(x3)
    display(Math('y_1^{pred} = ' + latex(y_pred1) + ', ' +
                 'y_2^{pred} = ' + latex(y_pred2) + ', ' +
                 'y_3^{pred} = ' + latex(y_pred3)))
    e1 = (y_pred1 - Matrix([y1])).norm()**2
    e2 = (y_pred2 - Matrix([y2])).norm()**2
    e3 = (y_pred3 - Matrix([y3])).norm()**2
    display(Math('e_1 = ' + latex(e1) + ', ' +
                 'e_2 = ' + latex(e2) + ', ' +
                 'e_3 = ' + latex(e3)))
    return (e1 + e2 + e3)/3
```

```python
x1 = 1,-1
x2 = 1,rho
x3 = 1,1
y1 = 0
y2 = 1
y3 = 0
e_linear = linear_model_cv_err(x1,y1,x2,y2,x3,y3)
display(Math('e_{linear\;model} = ' + latex(e_linear)))
```

$$X_1^{train} = \left[\begin{matrix}1 & \rho\\1 & 1\end{matrix}\right], X_2^{train} = \left[\begin{matrix}1 & -1\\1 & 1\end{matrix}\right], X_3^{train} = \left[\begin{matrix}1 & -1\\1 & \rho\end{matrix}\right]$$

$$(X_1^{train})^{-1} = \left[\begin{matrix}\frac{\rho}{- \rho + 1} + 1 & - \frac{\rho}{- \rho + 1}\\- \frac{1}{- \rho + 1} & \frac{1}{- \rho + 1}\end{matrix}\right], (X_2^{train})^{-1} = \left[\begin{matrix}\frac{1}{2} & \frac{1}{2}\\- \frac{1}{2} & \frac{1}{2}\end{matrix}\right], (X_3^{train})^{-1} = \left[\begin{matrix}1 - \frac{1}{\rho + 1} & \frac{1}{\rho + 1}\\- \frac{1}{\rho + 1} & \frac{1}{\rho + 1}\end{matrix}\right]$$

$$y_1^{train} = \left[\begin{matrix}1\\0\end{matrix}\right], y_2^{train} = \left[\begin{matrix}0\\0\end{matrix}\right], y_3^{train} = \left[\begin{matrix}0\\1\end{matrix}\right]$$

$$w_1 = \left[\begin{matrix}\frac{\rho}{- \rho + 1} + 1\\- \frac{1}{- \rho + 1}\end{matrix}\right], w_2 = \left[\begin{matrix}0\\0\end{matrix}\right], w_3 = \left[\begin{matrix}\frac{1}{\rho + 1}\\\frac{1}{\rho + 1}\end{matrix}\right]$$

$$y_1^{pred} = \left[\begin{matrix}\frac{\rho}{- \rho + 1} + 1 + \frac{1}{- \rho + 1}\end{matrix}\right], y_2^{pred} = \left[\begin{matrix}0\end{matrix}\right], y_3^{pred} = \left[\begin{matrix}\frac{2}{\rho + 1}\end{matrix}\right]$$

$$e_1 = \left\lvert{\frac{\rho}{\rho - 1} - 1 + \frac{1}{\rho - 1}}\right\rvert^{2}, e_2 = 1, e_3 = 4 \left\lvert{\frac{1}{\rho + 1}}\right\rvert^{2}$$

$$e_{linear\;model} = \frac{4}{3} \left\lvert{\frac{1}{\rho + 1}}\right\rvert^{2} + \frac{1}{3} \left\lvert{\frac{\rho}{\rho - 1} - 1 + \frac{1}{\rho - 1}}\right\rvert^{2} + \frac{1}{3}$$

Constant model (inverse does not work here as the matrix is not square)

```python
def const_model_cv_err(x1,y1,x2,y2,x3,y3):
    X_train1 = Matrix((x2,x3))
    X_train2 = Matrix((x1,x3))
    X_train3 = Matrix((x1,x2))
    y_train1 = Matrix((y2,y3))
    y_train2 = Matrix((y1,y3))
    y_train3 = Matrix((y1,y2))
    w1 = Rational(y2+y3,2)
    w2 = Rational(y1+y3,2)
    w3 = Rational(y1+y2,2)
    e1 = (w1 * Matrix([x1]) - Matrix([y1])).norm()**2
    e2 = (w2 * Matrix([x2]) - Matrix([y2])).norm()**2
    e3 = (w3 * Matrix([x3]) - Matrix([y3])).norm()**2
    return Rational(e1 + e2 + e3,3)
```

```python
x1 = 1
x2 = 1
x3 = 1
y1 = 0
y2 = 1
y3 = 0
e_const = const_model_cv_err(x1,y1,x2,y2,x3,y3)
display(Math('e_{constant\;model} = ' + latex(e_const)))
```

$$e_{constant\;model} = \frac{1}{2}$$

```python
rho1 = sqrt(sqrt(3)+4)
rho2 = sqrt(sqrt(3)-1)
rho3 = sqrt(9+4*sqrt(6))
rho4 = sqrt(9-sqrt(6))
ans1 = e_linear.subs(rho,rho1).simplify()
ans2 = e_linear.subs(rho,rho2).simplify()
ans3 = e_linear.subs(rho,rho3).simplify()
ans4 = e_linear.subs(rho,rho4).simplify()
display(Math(latex(ans1) + '=' + str(ans1.evalf())))
display(Math(latex(ans2) + '=' + str(ans2.evalf())))
display(Math(latex(ans3) + '=' + str(ans3.evalf())))
display(Math(latex(ans4) + '=' + str(ans4.evalf())))
```

$$- \frac{4 \sqrt{3}}{3} + \frac{31}{9}=1.13504336768594$$

$$\frac{56 \sqrt{3}}{3} + \frac{97}{3}=64.6649484079524$$

$$\frac{1}{2}=0.500000000000000$$

$$\frac{60 \sqrt{6}}{841} + \frac{683}{841}=0.986883929330548$$

Here, we can see that the 3rd expression gives the same leave-one-out cross validation error as the constant model.

```python
Math(latex(Eq(6*(e_linear-e_const),0)))
```

$$8 \left\lvert{\frac{1}{\rho + 1}}\right\rvert^{2} + 2 \left\lvert{\frac{\rho}{\rho - 1} - 1 + \frac{1}{\rho - 1}}\right\rvert^{2} - 1 = 0$$

## 3. Quadratic Programming

### 3.1 Background
In the notation of `help(solve_qp)`, we wish to minimize:

$$\frac{1}{2}\mathbf{x^T G x - a^T x}$$

subject to the constraint

$$\mathbf{C^T x} \ge \mathbf{b}$$

The matrix, `Q`, (also called `Dmat`) is:

$$G = \left[\begin{array}{cccc}
y_1 y_1 \mathbf{x_1^T x_1} & y_1 y_2 \mathbf{x_1^T x_2} & \cdots & y_1 y_N \mathbf{x_1^T x_N}\\
y_2 y_1 \mathbf{x_2^T x_1} & y_2 y_2 \mathbf{x_2^T x_2} & \cdots & y_2 y_N \mathbf{x_2^T x_N}\\
\vdots & \vdots & & \vdots\\
y_N y_1 \mathbf{x_N^T x_1} & y_N y_2 \mathbf{x_N^T x_2} & \cdots & y_N y_N \mathbf{x_N^T x_N}\end{array}\right]$$

The calculation of the above matrix is implemented in the code below:

```python
def get_Dmat(X,y):
    n = len(X)
    K = np.zeros(shape=(n,n))
    for i in range(n):
        for j in range(n):
            K[i,j] = np.dot(X[i], X[j])
    Q = np.outer(y,y)*K
    return(Q)
```

Calculation of `dvec`:

$$-\mathbf{a^T x} = \left(-\mathbf{1^T}\right)\mathbf{\alpha} = \begin{pmatrix} -1 & -1 & \dots & -1\end{pmatrix}\mathbf{\alpha}$$

is implemented as:

```python
a = np.ones(n)
```

Calculation of Inequality constraint:

$$\mathbf{C^T x} \ge \mathbf{b}$$

via

$$\mathbf{y^T x} \ge \mathbf{0}$$

$$\mathbf{\alpha} \ge \mathbf{0}$$

where the last two constraints are implemented as:

$$\mathbf{C^T} = \begin{pmatrix}y_1 & y_2 & \dots & y_n\\
1 & 0 & \cdots & 0\\
0 & 1 & \cdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & 1\end{pmatrix}$$

$$\mathbf{b} = \begin{pmatrix}0 \\ 0 \\ \vdots \\ 0\end{pmatrix}$$

```python
C = np.vstack([y,np.eye(n)])
b = np.zeros(1+n)
```

```python
def get_GaCb(X,y, verbose=False):
    n = len(X)
    assert n == len(y)
    G = get_Dmat(X,y)
    a = np.ones(n)
    C = np.vstack([y,np.eye(n)]).T
    b = np.zeros(1+n)
    I = np.eye(n, dtype=float)
    assert G.shape == (n,n)
    assert y.shape == (n,)
    assert a.shape == (n,)
    assert C.shape == (n,n+1)
    assert b.shape == (1+n,)
    assert I.shape == (n,n)
    if verbose is True:
        print(G)
        print(C.astype(int).T)
    return G,a,C,b,I
```

```python
def solve_cvxopt(P, q, G, h, A, b):
    P = cvxopt.matrix(P)
    q = cvxopt.matrix(q)
    G = cvxopt.matrix(G)
    h = cvxopt.matrix(h)
    A = cvxopt.matrix(A)
    b = cvxopt.matrix(b)
    solution = cvxopt.solvers.qp(P, q, G, h, A, b)
    return solution
```

```python
def create_toy_problem_1():
    X = np.array(
         [[ 1.0],
          [ 2.0],
          [ 3.0]])
    y = np.array([-1,-1,1], dtype=float)
    return X,y

def create_toy_problem_2():
    X = np.array(
         [[ 1.0, 0.0],
          [ 2.0, 0.0],
          [ 3.0, 0.0]])
    y = np.array([-1,-1,1], dtype=float)
    return X,y

def create_toy_problem_3():
    X = np.array(
         [[ 0.0, 0.0],
          [ 2.0, 2.0],
          [ 2.0, 0.0],
          [ 3.0, 0.0]])
    y = np.array([-1,-1,1,1], dtype=float)
    return X,y

def create_toy_problem_4():
    X = np.array(
         [[ 0.78683463, 0.44665934],
          [-0.16648517,-0.72218041],
          [ 0.94398266, 0.74900882],
          [ 0.45756412,-0.91334759],
          [ 0.15403063,-0.75459915],
          [-0.47632360, 0.02265701],
          [ 0.53992470,-0.25138609],
          [-0.73822772,-0.50766569],
          [ 0.92590792,-0.92529239],
          [ 0.08283211,-0.15199064]])
    y = np.array([-1,1,-1,1,1,-1,1,-1,1,1], dtype=float)
    G,a,C,b,I = get_GaCb(X,y)
    assert np.allclose(G[0,:],np.array([0.818613299,0.453564930,1.077310034,0.047927947,
        0.215852131,-0.364667935,-0.312547506,-0.807616753,-0.315245922,0.002712864]))
    assert np.allclose(G[n-1,:],np.array([0.002712864,0.095974341,0.035650250,0.176721283,
        0.127450687,0.042898544,0.082931435,-0.016011470,0.217330687,0.029962312]))
    return X,y
```

```python
def solve_quadratic_programming(X,y,tol=1.0e-8,method='solve_qp'):
    n = len(X)
    G,a,C,b,I = get_GaCb(X,y)
    eigs = np.linalg.eigvals(G + tol*I)
    pos_definite = np.all(eigs > 0)
    if pos_definite is False:
        print("Warning! Positive Definite(G+tol*I) = {}".format(pos_definite))

    if method=='solve_qp':
        try:
            alphas, f, xu, iters, lagr, iact = quadprog.solve_qp(G + tol*I,a,C,b,meq=1)
            print("solve_qp(): alphas = {} (f = {})".format(alphas,f))
            return alphas
        except:
            print("solve_qp() failed")
    else:
        #solution = cvxopt.solvers.qp(G, a, np.eye(n), np.zeros(n), np.diag(y), np.zeros(n))
        solution = solve_cvxopt(P=G, q=-np.ones(n),
                                G=-np.eye(n), h=np.zeros(n),
                                A=np.array([y]), b=np.zeros(1))  #A=np.diag(y), b=np.zeros(n))
        if solution['status'] != 'optimal':
            print("cvxopt.solvers.qp() failed")
            return None
        else:
            alphas = np.ravel(solution['x'])
            print("cvxopt.solvers.qp(): alphas = {}".format(alphas))
        #ssv = alphas > 1e-5
        #alphas = alphas[ssv]
        #print("alphas = {}".format(alphas))
        return alphas
```

Here, the quadratic programming code is tested on a handful of 'toy problems'

```python
#X, y = create_toy_problem_1()
X, y = create_toy_problem_3()
G,a,C,b,I = get_GaCb(X,y,verbose=True)
```

    [[ 0.  0. -0. -0.]
     [ 0.  8. -4. -6.]
     [-0. -4.  4.  6.]
     [-0. -6.  6.  9.]]
    [[-1 -1  1  1]
     [ 1  0  0  0]
     [ 0  1  0  0]
     [ 0  0  1  0]
     [ 0  0  0  1]]

```python
X, y = create_toy_problem_3()
alphas1 = solve_quadratic_programming(X,y,method='cvxopt')
alphas2 = solve_quadratic_programming(X,y,method='solve_qp')
```

         pcost       dcost       gap    pres   dres
     0: -8.1633e-01 -2.1224e+00  6e+00  2e+00  2e+00
     1: -8.5663e-01 -1.5796e+00  7e-01  3e-16  6e-16
     2: -9.9227e-01 -1.0195e+00  3e-02  2e-16  3e-16
     3: -9.9992e-01 -1.0002e+00  3e-04  2e-16  3e-16
     4: -1.0000e+00 -1.0000e+00  3e-06  2e-16  6e-16
     5: -1.0000e+00 -1.0000e+00  3e-08  2e-16  2e-16
    Optimal solution found.
    cvxopt.solvers.qp(): alphas = [  5.0000e-01   5.0000e-01   1.0000e+00   7.8099e-09]
    solve_qp(): alphas = [ 0.5  0.5  1.   0. ] (f = -0.9999999981373549)

```python
#h = np.hstack([
#    np.zeros(n),
#    np.ones(n) * 999999999.0])
#A = np.array([y])    #A = cvxopt.matrix(y, (1,n))
#b = np.array([0.0])  #b = cvxopt.matrix(0.0)
```

```python
def solve_cvxopt(n,P,y_output):
    # Generating all the matrices and vectors
    # P = cvxopt.matrix(np.outer(y_output, y_output) * K)
    q = cvxopt.matrix(np.ones(n) * -1)
    G = cvxopt.matrix(np.vstack([
        np.eye(n) * -1,
        np.eye(n)
        ]))
    h = cvxopt.matrix(np.hstack([
        np.zeros(n),
        np.ones(n) * 999999999.0
        ]))
    A = cvxopt.matrix(y_output, (1,n))
    b = cvxopt.matrix(0.0)
    solution = cvxopt.solvers.qp(P, q, G, h, A, b)
    return solution
```

```python
G = np.eye(3, 3)
a = np.array([0, 5, 0], dtype=np.double)
C = np.array([[-4, 2, 0], [-3, 1, -2], [0, 0, 1]], dtype=np.double)
b = np.array([-8, 2, 0], dtype=np.double)
xf, f, xu, iters, lagr, iact = quadprog.solve_qp(G, a, C, b)
```

```python
#https://github.com/rmcgibbo/quadprog/blob/master/quadprog/tests/test_1.py
def solve_qp_scipy(G, a, C, b, meq=0):
    # Minimize     1/2 x^T G x - a^T x
    # Subject to   C.T x >= b
    def f(x):
        return 0.5 * np.dot(x, G).dot(x) - np.dot(a, x)

    if C is not None and b is not None:
        constraints = [{
            'type': 'ineq',
            'fun': lambda x, C=C, b=b, i=i: (np.dot(C.T, x) - b)[i]
        } for i in range(C.shape[1])]
    else:
        constraints = []

    result = scipy.optimize.minimize(f, x0=np.zeros(len(G)), method='COBYLA',
        constraints=constraints, tol=1e-10)
    return result

def verify(G, a, C=None, b=None):
    xf, f, xu, iters, lagr, iact = quadprog.solve_qp(G, a, C, b)
    result = solve_qp_scipy(G, a, C, b)
    np.testing.assert_array_almost_equal(result.x, xf)
    np.testing.assert_array_almost_equal(result.fun, f)

def test_1():
    G = np.eye(3, 3)
    a = np.array([0, 5, 0], dtype=np.double)
    C = np.array([[-4, 2, 0], [-3, 1, -2], [0, 0, 1]], dtype=np.double)
    b = np.array([-8, 2, 0], dtype=np.double)
    xf, f, xu, iters, lagr, iact = quadprog.solve_qp(G, a, C, b)
    np.testing.assert_array_almost_equal(xf, [0.4761905, 1.0476190, 2.0952381])
    np.testing.assert_almost_equal(f, -2.380952380952381)
    np.testing.assert_almost_equal(xu, [0, 5, 0])
    np.testing.assert_array_equal(iters, [3, 0])
    np.testing.assert_array_almost_equal(lagr, [0.0000000, 0.2380952, 2.0952381])

    verify(G, a, C, b)

def test_2():
    G = np.eye(3, 3)
    a = np.array([0, 0, 0], dtype=np.double)
    C = np.ones((3, 1))
    b = -1000 * np.ones(1)
    verify(G, a, C, b)
    verify(G, a)

def test_3():
    random = np.random.RandomState(0)
    G = scipy.stats.wishart(scale=np.eye(3,3), seed=random).rvs()
    a = random.randn(3)
    C = random.randn(3, 2)
    b = random.randn(2)
    verify(G, a, C, b)
    verify(G, a)
```

```python
test_1()
test_2()
test_3()
```

```python
#https://gist.github.com/zibet/4f76b66feeb5aa24e124740081f241cb
from cvxopt import solvers
from cvxopt import matrix

def toysvm():
    def to_matrix(a):
        return matrix(a, tc='d')
    X = np.array([
        [0,2],
        [2,2],
        [2,0],
        [3,0]], dtype=float)
    y = np.array([-1,-1,1,1], dtype=float)
    Qd = np.array([
        [0,0,0,0],
        [0,8,-4,-6],
        [0,-4,4,6],
        [0,-6,6,9]], dtype=float)
    Ad = np.array([
        [-1,-1,1,1],
        [1,1,-1,-1],
        [1,0,0,0],
        [0,1,0,0],
        [0,0,1,0],
        [0,0,0,1]], dtype=float)
    N = len(y)
    P = to_matrix(Qd)
    q = to_matrix(-(np.ones((N))))
    G = to_matrix(-Ad)
    h = to_matrix(np.array(np.zeros(N+2)))
    sol = solvers.qp(P,q,G,h)
    print(sol['x'])

    #xf, f, xu, iters, lagr, iact = solve_qp(Qd, y, Ad, X)
```

```python
toysvm()
```

         pcost       dcost       gap    pres   dres
     0: -1.0249e+00 -2.3063e+00  1e+01  3e+00  2e+00
     1: -8.2851e-01 -1.7889e+00  1e+00  1e-01  6e-02
     2: -9.8330e-01 -1.0720e+00  1e-01  4e-03  2e-03
     3: -9.9979e-01 -1.0008e+00  1e-03  4e-05  2e-05
     4: -1.0000e+00 -1.0000e+00  1e-05  4e-07  2e-07
     5: -1.0000e+00 -1.0000e+00  1e-07  4e-09  2e-09
    Optimal solution found.
    [ 5.00e-01]
    [ 5.00e-01]
    [ 1.00e+00]
    [ 2.20e-08]