```python
%matplotlib inline
```

Adversarial Example Generation
==============================

**Author:** `Nathan Inkawhich <https://github.com/inkawhich>`__

**Updated:** `Gharib Gharibi`

If you are reading this, hopefully you can appreciate how effective some
machine learning models are. Research is constantly pushing ML models to
be faster, more accurate, and more efficient. However, an often
overlooked aspect of designing and training models is security and
robustness, especially in the face of an adversary who wishes to fool
the model.

This tutorial will raise your awareness to the security vulnerabilities
of ML models, and will give insight into the hot topic of adversarial
machine learning. You may be surprised to find that adding imperceptible
perturbations to an image *can* cause drastically different model
performance. Given that this is a tutorial, we will explore the topic
via example on an image classifier. Specifically we will use one of the
first and most popular attack methods, the Fast Gradient Sign Attack
(FGSM), to fool an MNIST classifier.

Threat Model
------------

For context, there are many categories of adversarial attacks, each with
a different goal and assumption of the attacker’s knowledge. However, in
general the overarching goal is to add the least amount of perturbation
to the input data to cause the desired misclassification. There are
several kinds of assumptions of the attacker’s knowledge, two of which
are: **white-box** and **black-box**. A *white-box* attack assumes the
attacker has full knowledge and access to the model, including
architecture, inputs, outputs, and weights. A *black-box* attack assumes
the attacker only has access to the inputs and outputs of the model, and
knows nothing about the underlying architecture or weights. There are
also several types of goals, including **misclassification** and
**source/target misclassification**. A goal of *misclassification* means
the adversary only wants the output classification to be wrong but does
not care what the new classification is. A *source/target
misclassification* means the adversary wants to alter an image that is
originally of a specific source class so that it is classified as a
specific target class.

In this case, the FGSM attack is a *white-box* attack with the goal of
*misclassification*. With this background information, we can now
discuss the attack in detail.

Fast Gradient Sign Attack
-------------------------

One of the first and most popular adversarial attacks to date is
referred to as the *Fast Gradient Sign Attack (FGSM)* and is described
by Goodfellow et. al. in `Explaining and Harnessing Adversarial
Examples <https://arxiv.org/abs/1412.6572>`__.

The attack is remarkablym powerful, and yet intuitive. It is designed to attack neural networks by
leveraging the way they learn--*gradients*. The idea is simple, rather
than working to minimize the loss by adjusting the weights based on the
backpropagated gradients, the attack *adjusts the input data to maximize
the loss* based on the same backpropagated gradients. In other words,
the attack uses the gradient of the loss w.r.t the input data, then
adjusts the input data to maximize the loss.

Before we jump into the code, let’s look at the famous
`FGSM <https://arxiv.org/abs/1412.6572>`__ panda example and extract
some notation.

From the figure, $\mathbf{x}$ is the original input image
correctly classified as a “panda”, $y$ is the ground truth label
for $\mathbf{x}$, $\mathbf{\theta}$ represents the model
parameters, and $J(\mathbf{\theta}, \mathbf{x}, y)$ is the loss
that is used to train the network. The attack backpropagates the
gradient back to the input data to calculate
$\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y)$. Then, it adjusts
the input data by a small step ($\epsilon$ or $0.007$ in the
picture) in the direction (i.e.
$sign(\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y))$) that will
maximize the loss. The resulting perturbed image, $x'$, is then
*misclassified* by the target network as a “gibbon” when it is still
clearly a “panda”.

Hopefully now the motivation for this tutorial is clear, so lets jump
into the implementation.

```python
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
import numpy as np
import matplotlib.pyplot as plt
```

Implementation
--------------

In this section, we will discuss the input parameters for the tutorial,
define the model under attack, then code the attack and run some tests.

There are only three inputs for this tutorial, and are defined as
follows:

-  **epsilons** - List of epsilon values to use for the run. It is
   important to keep 0 in the list because it represents the model
   performance on the original test set. Also, intuitively we would
   expect the larger the epsilon, the more noticeable the perturbations
   but the more effective the attack in terms of degrading model
   accuracy. Since the data range here is $[0,1]$, no epsilon
   value should exceed 1.

-  **pretrained_model** - path to the pretrained MNIST model which was
   trained with
   `pytorch/examples/mnist <https://github.com/pytorch/examples/tree/master/mnist>`__.
   For simplicity, download the pretrained model `here <https://drive.google.com/drive/folders/1fn83DF14tWmit0RTKWRhPq5uVXt73e0h?usp=sharing>`__.

-  **use_cuda** - boolean flag to use CUDA if desired and available.
   Note, a GPU with CUDA is not critical for this tutorial as a CPU will
   not take much time.

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code

    Enter your authorization code:
    ··········
    Mounted at /content/drive

```python
epsilons = [0, .05, .1, .15, .2, .25, .3]
pretrained_model = "/content/drive/My Drive/lenet_mnist_model.pth"
use_cuda=True
```

Model Under Attack
---

As mentioned, the model under attack is the same MNIST model from
`pytorch/examples/mnist <https://github.com/pytorch/examples/tree/master/mnist>`__.
You may train and save your own MNIST model or you can download and use
the provided model. The *Net* definition and test dataloader here have
been copied from the MNIST example. The purpose of this section is to
define the model and dataloader, then initialize the model and load the
pretrained weights.

```python
# LeNet Model definition
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

```

```python
# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),
            ])),
        batch_size=1, shuffle=True)

# Define what device we are using
print("CUDA Available: ",torch.cuda.is_available())

# Initialize the network
model = Net()

# Load the pretrained model
model.load_state_dict(torch.load(pretrained_model))

# Set the model in evaluation mode. In this case this is for the Dropout layers
model.eval()
```

    0it [00:00, ?it/s]

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ../data/MNIST/raw/train-images-idx3-ubyte.gz

    9920512it [00:01, 6589746.54it/s]

    Extracting ../data/MNIST/raw/train-images-idx3-ubyte.gz

    32768it [00:00, 327435.18it/s]
    0it [00:00, ?it/s]

    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ../data/MNIST/raw/train-labels-idx1-ubyte.gz
    Extracting ../data/MNIST/raw/train-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ../data/MNIST/raw/t10k-images-idx3-ubyte.gz

    1654784it [00:00, 5331617.10it/s]
    8192it [00:00, 130809.53it/s]

    Extracting ../data/MNIST/raw/t10k-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz
    Extracting ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz
    Processing...
    Done!
    CUDA Available:  True

    Net(
      (conv1): Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1))
      (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
      (conv2_drop): Dropout2d(p=0.5)
      (fc1): Linear(in_features=320, out_features=50, bias=True)
      (fc2): Linear(in_features=50, out_features=10, bias=True)
    )

FGSM Attack
-------

Now, we can define the function that creates the adversarial examples by
perturbing the original inputs. The ``fgsm_attack`` function takes three
inputs, *image* is the original clean image ($x$), *epsilon* is
the pixel-wise perturbation amount ($\epsilon$), and *data_grad*
is gradient of the loss w.r.t the input image
($\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y)$). The function
then creates perturbed image as

\begin{align}perturbed\_image = image + epsilon*sign(data\_grad) = x + \epsilon * sign(\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y))\end{align}

Finally, in order to maintain the original range of the data, the
perturbed image is clipped to range $[0,1]$.

```python
# FGSM attack code
def fgsm_attack(image, epsilon, data_grad):
    # Collect the element-wise sign of the data gradient
    sign_data_grad = data_grad.sign()
    # Create the perturbed image by adjusting each pixel of the input image
    perturbed_image = image + epsilon*sign_data_grad
    # Adding clipping to maintain [0,1] range
    perturbed_image = torch.clamp(perturbed_image, 0, 1)
    # Return the perturbed image
    return perturbed_image
```

Testing Function
----

Finally, the central result of this tutorial comes from the ``test``
function. Each call to this test function performs a full test step on
the MNIST test set and reports a final accuracy. However, notice that
this function also takes an *epsilon* input. This is because the
``test`` function reports the accuracy of a model that is under attack
from an adversary with strength $\epsilon$. More specifically, for
each sample in the test set, the function computes the gradient of the
loss w.r.t the input data ($data\_grad$), creates a perturbed
image with ``fgsm_attack`` ($perturbed\_data$), then checks to see
if the perturbed example is adversarial. In addition to testing the
accuracy of the model, the function also saves and returns some
successful adversarial examples to be visualized later.

```python
def test(model, test_loader, epsilon):

    # Accuracy counter
    correct = 0
    adv_examples = []

    # Loop over all examples in test set
    for data, target in test_loader:

        # Set requires_grad attribute of tensor. Important for Attack
        data.requires_grad = True

        # Forward pass the data through the model
        output = model(data)
        init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability

        # If the initial prediction is wrong, dont bother attacking, just move on
        if init_pred.item() != target.item():
            continue

        # Calculate the loss
        loss = F.nll_loss(output, target)

        # Zero all existing gradients
        model.zero_grad()

        # Calculate gradients of model in backward pass
        loss.backward()

        # Collect datagrad
        data_grad = data.grad.data

        # Call FGSM Attack
        perturbed_data = fgsm_attack(data, epsilon, data_grad)

        # Re-classify the perturbed image
        output = model(perturbed_data)

        # Check for success
        final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
        if final_pred.item() == target.item():
            correct += 1
            # Special case for saving 0 epsilon examples
            if (epsilon == 0) and (len(adv_examples) < 5):
                adv_ex = perturbed_data.squeeze().detach().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )
        else:
            # Save some adv examples for visualization later
            if len(adv_examples) < 5:
                adv_ex = perturbed_data.squeeze().detach().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )

    # Calculate final accuracy for this epsilon
    final_acc = correct/float(len(test_loader))
    print("Epsilon: {}\tTest Accuracy = {} / {} = {}".format(epsilon, correct, len(test_loader), final_acc))

    # Return the accuracy and an adversarial example
    return final_acc, adv_examples
```

Run Attack
---

The last part of the implementation is to actually run the attack. Here,
we run a full test step for each epsilon value in the *epsilons* input.
For each epsilon we also save the final accuracy and some successful
adversarial examples to be plotted in the coming sections. Notice how
the printed accuracies decrease as the epsilon value increases. Also,
note the $\epsilon=0$ case represents the original test accuracy,
with no attack.

```python
accuracies = []
examples = []

# Run test for each epsilon
for eps in epsilons:
    acc, ex = test(model, test_loader, eps)
    accuracies.append(acc)
    examples.append(ex)
```

    Epsilon: 0	Test Accuracy = 9810 / 10000 = 0.981
    Epsilon: 0.05	Test Accuracy = 9426 / 10000 = 0.9426
    Epsilon: 0.1	Test Accuracy = 8510 / 10000 = 0.851
    Epsilon: 0.15	Test Accuracy = 6826 / 10000 = 0.6826
    Epsilon: 0.2	Test Accuracy = 4301 / 10000 = 0.4301
    Epsilon: 0.25	Test Accuracy = 2082 / 10000 = 0.2082
    Epsilon: 0.3	Test Accuracy = 869 / 10000 = 0.0869

Results
-------

Accuracy vs Epsilon
---

The first result is the accuracy versus epsilon plot. As alluded to
earlier, as epsilon increases we expect the test accuracy to decrease.
This is because larger epsilons mean we take a larger step in the
direction that will maximize the loss. Notice the trend in the curve is
not linear even though the epsilon values are linearly spaced. For
example, the accuracy at $\epsilon=0.05$ is only about 4% lower
than $\epsilon=0$, but the accuracy at $\epsilon=0.2$ is 25%
lower than $\epsilon=0.15$. Also, notice the accuracy of the model
hits random accuracy for a 10-class classifier between
$\epsilon=0.25$ and $\epsilon=0.3$.

```python
plt.figure(figsize=(5,5))
plt.plot(epsilons, accuracies, "*-")
plt.yticks(np.arange(0, 1.1, step=0.1))
plt.xticks(np.arange(0, .35, step=0.05))
plt.title("Accuracy vs Epsilon")
plt.xlabel("Epsilon")
plt.ylabel("Accuracy")
plt.show()
```

Sample Adversarial Examples
---

Remember the idea of no free lunch? In this case, as epsilon increases
the test accuracy decreases **BUT** the perturbations become more easily
perceptible. In reality, there is a tradeoff between accuracy
degredation and perceptibility that an attacker must consider. Here, we
show some examples of successful adversarial examples at each epsilon
value. Each row of the plot shows a different epsilon value. The first
row is the $\epsilon=0$ examples which represent the original
“clean” images with no perturbation. The title of each image shows the
“original classification -> adversarial classification.” Notice, the
perturbations start to become evident at $\epsilon=0.15$ and are
quite evident at $\epsilon=0.3$. However, in all cases humans are
still capable of identifying the correct class despite the added noise.

```python
# Plot several examples of adversarial samples at each epsilon
cnt = 0
plt.figure(figsize=(8,10))
for i in range(len(epsilons)):
    for j in range(len(examples[i])):
        cnt += 1
        plt.subplot(len(epsilons),len(examples[0]),cnt)
        plt.xticks([], [])
        plt.yticks([], [])
        if j == 0:
            plt.ylabel("Eps: {}".format(epsilons[i]), fontsize=14)
        orig,adv,ex = examples[i][j]
        plt.title("{} -> {}".format(orig, adv))
        plt.imshow(ex, cmap="gray")
plt.tight_layout()
plt.show()
```

Where to go next?
-----------------

Hopefully this tutorial gives some insight into the topic of adversarial
machine learning. There are many potential directions to go from here.
This attack represents the very beginning of adversarial attack research
and since there have been many subsequent ideas for how to attack and
defend ML models from an adversary.

In fact, at NIPS 2017 there was an
adversarial attack and defense competition and many of the methods used
in the competition are described in this paper: `Adversarial Attacks and
Defences Competition <https://arxiv.org/pdf/1804.00097.pdf>`__. The work
on defense also leads into the idea of making machine learning models
more *robust* in general, to both naturally perturbed and adversarially
crafted inputs.

Another direction to go is adversarial attacks and defense in different
domains. Adversarial research is not limited to the image domain, check
out `this <https://arxiv.org/pdf/1801.01944.pdf>`__ attack on
speech-to-text models.

But perhaps the best way to learn more about
adversarial machine learning is to get your hands dirty. Try to
implement a different attack from the NIPS 2017 competition, and see how
it differs from FGSM. Then, try to defend the model from your own
attacks.

ICP 77
---

*   Replicate the above attack using a different model. E.g., try to mimic this notebok to attack the SqueezeNet in ModelHub.AI
*   EXTRA GRADES (2% in total): implement the MI-FGSM and test its recall and precision on several networks. Try it on Clarifai.com and Modelhub.ai

```python
import torch.nn.init as init
class SqueezeNet(nn.Module):

    def __init__(self, version='1_0', num_classes=1000):
        super(SqueezeNet, self).__init__()
        self.num_classes = num_classes
        if version == '1_0':
            self.features = nn.Sequential(
                nn.Conv2d(3, 96, kernel_size=7, stride=2),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(96, 16, 64, 64),
                Fire(128, 16, 64, 64),
                Fire(128, 32, 128, 128),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(256, 32, 128, 128),
                Fire(256, 48, 192, 192),
                Fire(384, 48, 192, 192),
                Fire(384, 64, 256, 256),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(512, 64, 256, 256),
            )
        elif version == '1_1':
            self.features = nn.Sequential(
                nn.Conv2d(3, 64, kernel_size=3, stride=2),
                nn.ReLU(inplace=True),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(64, 16, 64, 64),
                Fire(128, 16, 64, 64),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(128, 32, 128, 128),
                Fire(256, 32, 128, 128),
                nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True),
                Fire(256, 48, 192, 192),
                Fire(384, 48, 192, 192),
                Fire(384, 64, 256, 256),
                Fire(512, 64, 256, 256),
            )
        else:
            # FIXME: Is this needed? SqueezeNet should only be called from the
            # FIXME: squeezenet1_x() functions
            # FIXME: This checking is not done for the other models
            raise ValueError("Unsupported SqueezeNet version {version}:"
                             "1_0 or 1_1 expected".format(version=version))

        # Final convolution is initialized differently from the rest
        final_conv = nn.Conv2d(512, self.num_classes, kernel_size=1)
        self.classifier = nn.Sequential(
            nn.Dropout(p=0.5),
            final_conv,
            nn.ReLU(inplace=True),
            nn.AdaptiveAvgPool2d((1, 1))
        )

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                if m is final_conv:
                    init.normal_(m.weight, mean=0.0, std=0.01)
                else:
                    init.kaiming_uniform_(m.weight)
                if m.bias is not None:
                    init.constant_(m.bias, 0)

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x.view(x.size(0), -1)

```

```python
class Fire(nn.Module):

    def __init__(self, inplanes, squeeze_planes,
                 expand1x1_planes, expand3x3_planes):
        super(Fire, self).__init__()
        self.inplanes = inplanes
        self.squeeze = nn.Conv2d(inplanes, squeeze_planes, kernel_size=1)
        self.squeeze_activation = nn.ReLU(inplace=True)
        self.expand1x1 = nn.Conv2d(squeeze_planes, expand1x1_planes,
                                   kernel_size=1)
        self.expand1x1_activation = nn.ReLU(inplace=True)
        self.expand3x3 = nn.Conv2d(squeeze_planes, expand3x3_planes,
                                   kernel_size=3, padding=1)
        self.expand3x3_activation = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.squeeze_activation(self.squeeze(x))
        return torch.cat([
            self.expand1x1_activation(self.expand1x1(x)),
            self.expand3x3_activation(self.expand3x3(x))
        ], 1)

```

```python
epsilons = [0, .05, .1, .15, .2, .25, .3]
pretrained_model = "/content/drive/My Drive/CS490/ICP77/squeezenet1_0.pth"
use_cuda=True
```

```python
# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(
    datasets.ImageNet('../data', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),
            ])),
        batch_size=3, shuffle=True)

# Define what device we are using
print("CUDA Available: ",torch.cuda.is_available())

# Initialize the network
model = SqueezeNet()

# Load the pretrained model
model.load_state_dict(torch.load(pretrained_model))

# Set the model in evaluation mode. In this case this is for the Dropout layers
model.eval()
```

```python
def squeezetest(model, test_loader, epsilon):

    # Accuracy counter
    correct = 0
    adv_examples = []

    # Loop over all examples in test set
    for data, target in test_loader:

        # Set requires_grad attribute of tensor. Important for Attack
        data.requires_grad = True

        # Forward pass the data through the model
        output = model(data)
        init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability

        # If the initial prediction is wrong, dont bother attacking, just move on
        if init_pred.item() != target.item():
            continue

        # Calculate the loss
        loss = F.nll_loss(output, target)

        # Zero all existing gradients
        model.zero_grad()

        # Calculate gradients of model in backward pass
        loss.backward()

        # Collect datagrad
        data_grad = data.grad.data

        # Call FGSM Attack
        perturbed_data = fgsm_attack(data, epsilon, data_grad)

        # Re-classify the perturbed image
        output = model(perturbed_data)

        # Check for success
        final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
        if final_pred.item() == target.item():
            correct += 1
            # Special case for saving 0 epsilon examples
            if (epsilon == 0) and (len(adv_examples) < 5):
                adv_ex = perturbed_data.squeeze().detach().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )
        else:
            # Save some adv examples for visualization later
            if len(adv_examples) < 5:
                adv_ex = perturbed_data.squeeze().detach().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )

    # Calculate final accuracy for this epsilon
    final_acc = correct/float(len(test_loader))
    print("Epsilon: {}\tTest Accuracy = {} / {} = {}".format(epsilon, correct, len(test_loader), final_acc))

    # Return the accuracy and an adversarial example
    return final_acc, adv_examples
```

```python
squeezetest(model, test_loader, .01)

```

```python
accuracies = []
examples = []

# Run test for each epsilon
for eps in epsilons:
    acc, ex = test(model, test_loader, eps)
    accuracies.append(acc)
    examples.append(ex)
```


# Facial Keypoints Detection

```python
import torch
import torchvision
import torch.nn as nn
import torch.optim as optim
from torchvision import models
from torchvision import transforms
from torch.utils.data import DataLoader, SequentialSampler
from data_loader import FacialKeypointsDataset
import transform
import matplotlib.pyplot as plt
import time
import copy
import numpy as np
import cv2 as cv

print('Torch version: {}'.format(torch.__version__))
print('Torchvision version: {}'.format(torchvision.__version__))
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('Using GPU: {}'.format(device))
```

## Load the dataset with the following transforms:
* Normalization in the range (a, b)
\begin{equation}
sample^* = (b - a) * \frac{(sample - min(sample))}{max(sample) - min(sample)} + a
\end{equation}

     \begin{equation} Image = (1-0) * \frac{Image - 0}{255 - 0} + 0 = Image/255 \end{equation}
     \begin{equation} Keypoints = (1 - (-1))* \frac{Keypoints - 0}{96 - 0} + (-1) = \frac{Keypoints - 48}{48} \end{equation}

* Numpy array to Tensor conversion

```python
data_transforms = transforms.Compose([
    transform.Normalize(),
    transform.ToTensor(),
])
train_set = FacialKeypointsDataset('training.csv', transform=data_transforms)
val_set = FacialKeypointsDataset('val.csv', transform=data_transforms)
print(len(train_set), len(val_set))
```

## Use PyTorch DataLoader to load streams of train and validation

```python
bs = 20
train_loader = DataLoader(train_set, batch_size=bs, shuffle=True, num_workers=4)
val_loader = DataLoader(val_set, batch_size=bs, shuffle=False, num_workers=4, drop_last=True)
```

### Plot some samples

```python
num_samples = 4
fig = plt.figure(figsize=(15,15))
for idx, sample in enumerate(train_loader):
    if idx == num_samples:
        break
    image, key_pts = sample['image'], sample['keypoints']
    image = torch.squeeze(image[0]) * 255.
    key_pts = (key_pts[0].view(-1, 2) * 48) + 48
    plt.subplot(1, num_samples ,idx+1)
    plt.title('Sample {}'.format(idx))
    plt.axis('off')
    plt.imshow(image, cmap='gray')
    plt.scatter(key_pts[:,0], key_pts[:,1], s=100,  marker='.', c='r')
plt.show()
```

## Define Network Architecture
```python
features{(Layer Segment 1) = Conv2d(32, 96, 96),
                             BatchNorm2d(32, 96, 96),
                             MaxPool2d(32, 48, 48),
                             ReLU(32, 48, 48),
                             Dropout2d(32, 48, 48)

         (Layer Segment 2) = Conv2d(64, 48, 48),
                             BatchNorm2d(64, 48, 48),
                             MaxPool2d(64, 24, 24),
                             ReLU(64, 24, 24),
                             Dropout2d(64, 24, 24)

         (Layer Segment 3) = Conv2d(128, 24, 24),
                             BatchNorm2d(128, 24, 24),
                             MaxPool2d(128, 12, 12),
                             ReLU(128, 12, 12),
                             Dropout2d(128, 12, 12)

         (Layer Segment 4) = Conv2d(256, 12, 12),
                             BatchNorm2d(256, 12, 12),
                             MaxPool2d(256, 6, 6),
                             ReLU(256, 6, 6),
                             Dropout2d(256, 6, 6)
         }

classifier{(Layer Segment 1) = Linear(256),
                               ReLU(256),
                               Dropout(256),
                               Linear(30)
          }
```

```python
class KeypointsNet(nn.Module):

    def __init__(self):
        super(KeypointsNet, self).__init__()
        self.reset()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2, 2),
            nn.ReLU(),
            nn.Dropout2d(0.1),

            nn.Conv2d(32, 64, 3, padding=1),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2, 2),
            nn.ReLU(),
            nn.Dropout2d(0.2),

            nn.Conv2d(64, 128, 3, padding=1),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(2, 2),
            nn.ReLU(),
            nn.Dropout2d(0.3),

            nn.Conv2d(128, 256, 3, padding=1),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(2, 2),
            nn.ReLU(),
            nn.Dropout2d(0.4),
        )
        self.classifier = nn.Sequential(
            nn.Linear(256*6*6, 256),
            nn.ReLU(),
            nn.Dropout(0.4),
            nn.Linear(256, 30)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, self.num_flat_features(x))
        x = self.classifier(x)
        return x

    def num_flat_features(self, x):
        n_feats = 1
        for s in x.size()[1:]:
            n_feats *= s

        return n_feats

    def reset(self):
        self.train_loss_history = []
        self.train_acc_history = []
        self.val_loss_history = []
        self.val_acc_history = []

    def print_params(self):
        total_params = sum(param.numel() for param in self.parameters())
        print('Total params: {}'.format(total_params))
```

### Display Model

```python
model = KeypointsNet()
print(model)
model.print_params()
model = model.to(device)
```

### Training Loop

```python
def train_net(model, train_loader, val_loader, use_cuda, print_every, n_epochs):
    print('\nStarted training...\n')
    best_score = 140
    best_model_wts = copy.deepcopy(model.state_dict())
    for epoch in range(n_epochs):
        model.train
        running_loss = 0.0
        running_corrects = 0
        for batch, data in enumerate(train_loader):
            images, key_pts = data['image'], data['keypoints']
            key_pts = key_pts.view(key_pts.size(0), -1)
            if use_cuda:
                images = images.to(device)
                key_pts = key_pts.to(device)

            output = model(images)
            train_loss = criterion(output, key_pts)
            model.train_loss_history.append(train_loss.item())

            optimizer.zero_grad()
            train_loss.backward()
            optimizer.step()

            if (batch+1) % print_every == 0:
                print('[Iteration {}/{}] Training loss: {:.3f}'.format(batch+1, len(train_loader), train_loss.item()))

        with torch.no_grad():
            running_loss = 0.0

            model.eval
            for batch, data in enumerate(val_loader):
                image_val, key_pts_val = data['image'], data['keypoints']
                key_pts_val = key_pts_val.view(key_pts_val.size(0), -1)
                if use_cuda:
                    image_val = image_val.to(device)
                    key_pts_val = key_pts_val.to(device)

                o_val = model(image_val)
                val_loss = criterion(o_val, key_pts_val)
                running_loss += val_loss.item()
                model.val_loss_history.append(val_loss.item())

        metric = 1.0 / (2 * (running_loss/len(val_loader)))
        print('[Epoch {}/{}] Training loss: {:.3f}, Validation score: {:.3f}\n'.format(epoch+1, n_epochs, model.train_loss_history[-1], metric))

        if metric > best_score:
            best_score = metric
            best_model_wts = copy.deepcopy(model.state_dict())
            print('Achieved better score. Saving state...')

    print('\nFinished training...\n')
    model.load_state_dict(best_model_wts)
    return model
```

```python
since = time.time()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01, betas=(0.9, 0.999), eps=1e-8, weight_decay=1e-6)
best_model = train_net(model, train_loader, val_loader, True, 10, 25)
torch.save(best_model, 'bestModel2.pt')
time_elapsed = time.time() - since
print('Total training time: {}m {}s'.format(time_elapsed // 60, time_elapsed % 60))
```

## Testing

* Detect faces using OpenCV's Haar Cascade
* Resize detected faces to (96, 96)
* Make a Torch Tensor from the face sample
* Evaluate the model on the sample
* Plot the output keypoints back on the sample

```python
face_cascade = cv.CascadeClassifier('/home/hashir/anaconda3/lib/python3.7/site-packages/cv2/data/haarcascade_frontalface_default.xml')

def face_detect(frame):
    gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.3, 5)
    face_gray = None
    for (x,y,w,h) in faces:
        cv.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
        face_gray = gray[y:y+h, x:x+w]

    if face_gray is not None:
        sample = cv.resize(face_gray, (96, 96))
        sample_np = np.asarray(sample).reshape(1,96,96).astype(np.float32)
        sample_np /= 255.
        sample_tensor = torch.from_numpy(sample_np).unsqueeze(0).to(device)
        return sample_tensor
    else:
        print('No face found!')

saved_model = torch.load('Saved_Models/bestModel2.pt')
saved_model = saved_model.to(device)
saved_model.eval()
```

## Test in real-time using OpenCV's Video Capture

```python
cap = cv.VideoCapture(0)
time.sleep(2.0)

# fourcc = cv.VideoWriter_fourcc(*'MJPG')
# out = cv.VideoWriter('output.avi',fourcc, 20.0)

while cap.isOpened():
    ret, frame = cap.read()
#     frame = cv.resize(frame, (1024, 600))
    gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.3, 5)
    for (x,y,w,h) in faces:
        cv.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
        face_gray = gray[y:y+h, x:x+w]
        original_shape = face_gray.shape
        sample = cv.resize(face_gray, (96, 96))
        sample = sample.astype('float32')/255
        sample = np.asarray(sample).reshape(1,96,96)
        sample = torch.from_numpy(sample).unsqueeze(0).to(device)
        output = saved_model(sample)
        output = output.view(-1, 2).detach()
        output = (output * 48) + 48
        output = output.cpu().numpy()
        for i in range(15):
            x_pt = (output[i][0] * (original_shape[1]/96)) + x
            y_pt = (output[i][1] * (original_shape[0]/96)) + y
            cv.circle(frame, (int(x_pt), int(y_pt)), 3, (0, 0, 255), -1)
#     out.write(frame)
    cv.imshow("Frame", frame)
    key = cv.waitKey(1) & 0xFF

    if key == ord('q'):
        break

cap.release()
# out.release()
cv.destroyAllWindows()
cv.waitKey(1)
```

```python

```


# Time Evolution of Maxwell's Equations & Constraints in Flat Spacetime and Cartesian Coordinates

## Authors: Terrence Pierre Jacques, Zachariah Etienne and Ian Ruchlin

### This module constructs the evolution equations for Maxwell's equations as symbolic (SymPy) expressions, for an electromagnetic field in vacuum, as defined in [Tutorial-VacuumMaxwell_formulation_Cartesian](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb).

**Notebook Status:**  Validated

**Validation Notes:** All expressions generated in this module have been validated.

### NRPy+ Source Code for this module: [Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py](../edit/Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py)

[comment]: <> (Introduction: TODO)

This tutorial takes the expressions defined in [Tutorial-VacuumMaxwell_formulation_Cartesian](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb), and constructs them using NRPy+ in SymPy expressions.

# Table of Contents
$$\label{toc}$$

1. [Step 1](#initializenrpy): Initialize needed Python/NRPy+ modules
1. [Step 2](#system1): System 1
    1. [Step 2.a](#system1_rhs): Construct evolution equations
    1. [Step 2.b](#system1_val): Code Validation
1. [Step 3](#system2): System 2
    1. [Step 3.a](#system2_rhs): Construct evolution equations
    1. [Step 3.b](#system2_val): Code Validation
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize needed Python/NRPy+ modules \[Back to [top](#toc)\]

$$\label{initializenrpy}$$

```python
# Import needed Python modules
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import grid as gri               # NRPy+: Functions having to do with numerical grids

#Step 0: Set the spatial dimension parameter to 3.
par.set_parval_from_str("grid::DIM", 3)
DIM = par.parval_from_str("grid::DIM")
```

# Step 2: System I \[Back to [top](#toc)\]
$$\label{system1}$$

## Step 2.a: Construct evolution equations \[Back to [top](#toc)\]
$$\label{system1_rhs}$$

Following our derivations in [Tutorial-VacuumMaxwell_formulation_Cartesian](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb), we construct the system of equations for System I, in flat space and cartesian coordinates:

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \partial_j A^j - \partial_j \partial^j A^i, \\
\partial_t \varphi &= -\partial_i A^i,
\end{align}

```python
# Register gridfunctions that are needed as input.

#  Declare the rank-1 indexed expressions E_{i}, A_{i},
#  and \partial_{i} \psi, that are to be evolved in time.
#  Derivative variables like these must have an underscore
#  in them, so the finite difference module can parse
#  the variable name properly.

# E_i
EU = ixp.register_gridfunctions_for_single_rank1("EVOL", "EU")

# A_i, _AD is unused
_AU = ixp.register_gridfunctions_for_single_rank1("EVOL", "AU")

# \psi is a scalar function that is time evolved
# _psi is unused
_psi = gri.register_gridfunctions("EVOL", ["psi"])

# \partial_i \psi
psi_dD = ixp.declarerank1("psi_dD")

# \partial_k ( A_i ) --> rank two tensor
AU_dD = ixp.declarerank2("AU_dD", "nosym")

# \partial_k partial_m ( A_i ) --> rank three tensor
AU_dDD = ixp.declarerank3("AU_dDD", "sym12")
```

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \partial_j A^j - \partial_j \partial^j A^i, \\
\partial_t \varphi &= -\partial_i A^i,
\end{align}

```python
# \partial_t \psi = -\partial_i A_i
psi_rhs = 0

# \partial_t A_i = E_i - \partial_i \psi
ArhsU = ixp.zerorank1()

# \partial_t E_i = -\partial_j^2 A_i + \partial_j \partial_i A_j
ErhsU = ixp.zerorank1()

# RHSs - equations 10 and 11 from https://arxiv.org/abs/gr-qc/0201051
for i in range(DIM):
    ArhsU[i] = -EU[i] - psi_dD[i]
    psi_rhs += -AU_dD[i][i]
    for j in range(DIM):
        ErhsU[i] += -AU_dDD[i][j][j] + AU_dDD[j][j][i]
```

Constraint:

\begin{align}
\mathcal{C} \equiv \partial_i E^i
\end{align}

```python
# \partial_k ( E^i ) --> rank two tensor
EU_dD = ixp.declarerank2("EU_dD", "nosym")

# C = \partial_i E^i
C = EU_dD[0][0] + EU_dD[1][1] + EU_dD[2][2]
```

```python
print('C = ', C)
print('ArhsU[0] = ', ArhsU[0])
print('ArhsU[1] = ', ArhsU[1])
print('ArhsU[2] = ', ArhsU[2])
print('ErhsU[0] = ', ErhsU[0])
print('ErhsU[1] = ', ErhsU[1])
print('ErhsU[2] = ', ErhsU[2])
print('psi_rhs  = ', psi_rhs)
```

    C =  EU_dD00 + EU_dD11 + EU_dD22
    ArhsU[0] =  -EU0 - psi_dD0
    ArhsU[1] =  -EU1 - psi_dD1
    ArhsU[2] =  -EU2 - psi_dD2
    ErhsU[0] =  -AU_dDD011 - AU_dDD022 + AU_dDD101 + AU_dDD202
    ErhsU[1] =  AU_dDD001 - AU_dDD100 - AU_dDD122 + AU_dDD212
    ErhsU[2] =  AU_dDD002 + AU_dDD112 - AU_dDD200 - AU_dDD211
    psi_rhs  =  -AU_dD00 - AU_dD11 - AU_dD22

## Step 2.b: NRPy+ Module Code Validation \[Back to [top](#toc)\]
$$\label{system1_val}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the RHSs of Maxwell's equations (in System I) between
1. this tutorial and
2. the NRPy+ [Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py](../edit/Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py) module.

```python
# Reset the list of gridfunctions, as registering a gridfunction
#   twice will spawn an error.
gri.glb_gridfcs_list = []

#          Call the VacuumMaxwellRHSs() function from within the
#          Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py module,
#          which should do exactly the same as the steps above.

# Set which system to use, which are defined in Maxwell/InitialData.py
par.initialize_param(par.glb_param("char", "Maxwell.InitialData","System_to_use","System_I"))

import Maxwell.VacuumMaxwell_Flat_Evol_Cartesian as mwevol
mwevol.VacuumMaxwellRHSs()

print("Consistency check between Tutorial-VacuumMaxwell_Cartesian_RHSs tutorial and NRPy+ module: ALL SHOULD BE ZERO.")

print("C - mwevol.C = " + str(C - mwevol.C))
print("psi_rhs - mwevol.psi_rhs = " + str(psi_rhs - mwevol.psi_rhs))
for i in range(DIM):

    print("ArhsU["+str(i)+"] - mwevol.ArhsU["+str(i)+"] = " + str(ArhsU[i] - mwevol.ArhsU[i]))
    print("ErhsU["+str(i)+"] - mwevol.ErhsU["+str(i)+"] = " + str(ErhsU[i] - mwevol.ErhsU[i]))
```

    Currently using System_I RHSs

    Consistency check between Tutorial-VacuumMaxwell_Cartesian_RHSs tutorial and NRPy+ module: ALL SHOULD BE ZERO.
    C - mwevol.C = 0
    psi_rhs - mwevol.psi_rhs = 0
    ArhsU[0] - mwevol.ArhsU[0] = 0
    ErhsU[0] - mwevol.ErhsU[0] = 0
    ArhsU[1] - mwevol.ArhsU[1] = 0
    ErhsU[1] - mwevol.ErhsU[1] = 0
    ArhsU[2] - mwevol.ArhsU[2] = 0
    ErhsU[2] - mwevol.ErhsU[2] = 0

# Step 3: System II \[Back to [top](#toc)\]
$$\label{system2}$$

## Step 3.a: Construct evolution equations \[Back to [top](#toc)\]
$$\label{system2_rhs}$$

Next, again following [Tutorial-VacuumMaxwell_formulation_Cartesian](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb), we construct the system of equations for System II, in flat space and cartesian coordinates:

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \Gamma - \partial_j \partial^j A^i, \\
\partial_t \Gamma &= - \partial_i \partial^i \varphi, \\
\partial_t \varphi &= -\Gamma,
\end{align}

```python
# We inherit here all of the definitions from System I, above

# Register the scalar auxiliary variable \Gamma
Gamma = gri.register_gridfunctions("EVOL", ["Gamma"])

# Declare the ordinary gradient \partial_{i} \Gamma
Gamma_dD = ixp.declarerank1("Gamma_dD")

# partial_i \partial_j \psi
psi_dDD = ixp.declarerank2("psi_dDD", "sym01")

psi_rhs = -Gamma
```

\begin{align}
\partial_t A^i &= -E^i - \partial^i \varphi, \\
\partial_t E^i &= \partial^i \Gamma - \partial_j \partial^j A^i, \\
\partial_t \Gamma &= - \partial_i \partial^i \varphi, \\
\partial_t \varphi &= -\Gamma,
\end{align}

```python
# Equation 15 https://arxiv.org/abs/gr-qc/0201051
# \partial_t \Gamma = -\partial_i^2 \psi
Gamma_rhs = 0

# \partial_t A_i = -E_i - \partial_i \psi
ArhsU = ixp.zerorank1()

# \partial_t E_i = -\partial_j^2 A_i + \partial_i \Gamma
ErhsU = ixp.zerorank1()

# RHSs - equations 10 and 14 https://arxiv.org/abs/gr-qc/0201051
for i in range(DIM):
    ArhsU[i] = -EU[i] - psi_dD[i]
    Gamma_rhs -= psi_dDD[i][i]
    ErhsU[i] += Gamma_dD[i]
    for j in range(DIM):
        ErhsU[i] -= AU_dDD[i][j][j]
```

Constraints:

\begin{align}
\mathcal{G} &\equiv \Gamma - \partial_i A^i, \\
\mathcal{C} &\equiv \partial_i E^i.
\end{align}

```python
# \partial_k ( E^i ) --> rank two tensor
EU_dD = ixp.declarerank2("EU_dD", "nosym")

# C = \partial_i E^i
C = EU_dD[0][0] + EU_dD[1][1] + EU_dD[2][2]

# G = \Gamma - \partial_i A^i
G = Gamma - AU_dD[0][0] - AU_dD[1][1] - AU_dD[2][2]
```

```python
print('C = ', C)
print('G = ', G)
print('ArhsU[0]  = ', ArhsU[0])
print('ArhsU[1]  = ', ArhsU[1])
print('ArhsU[2]  = ', ArhsU[2])
print('ErhsU[0]  = ', ErhsU[0])
print('ErhsU[1]  = ', ErhsU[1])
print('ErhsU[2]  = ', ErhsU[2])
print('psi_rhs   = ', psi_rhs)
print('Gamma_rhs =', Gamma_rhs)
```

    C =  EU_dD00 + EU_dD11 + EU_dD22
    G =  -AU_dD00 - AU_dD11 - AU_dD22 + Gamma
    ArhsU[0]  =  -EU0 - psi_dD0
    ArhsU[1]  =  -EU1 - psi_dD1
    ArhsU[2]  =  -EU2 - psi_dD2
    ErhsU[0]  =  -AU_dDD000 - AU_dDD011 - AU_dDD022 + Gamma_dD0
    ErhsU[1]  =  -AU_dDD100 - AU_dDD111 - AU_dDD122 + Gamma_dD1
    ErhsU[2]  =  -AU_dDD200 - AU_dDD211 - AU_dDD222 + Gamma_dD2
    psi_rhs   =  -Gamma
    Gamma_rhs = -psi_dDD00 - psi_dDD11 - psi_dDD22

## Step 3.b: NRPy+ Module Code Validation \[Back to [top](#toc)\]
$$\label{system2_val}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the RHSs of Maxwell's equations (in System II) between
1. this tutorial and
2. the NRPy+ [Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py](../edit/Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py) module.

```python
# Reset the list of gridfunctions, as registering a gridfunction
#   twice will spawn an error.
gri.glb_gridfcs_list = []

#          Call the VacuumMaxwellRHSs() function from within the
#          Maxwell/VacuumMaxwell_Flat_Evol_Cartesian.py module,
#          which should do exactly the same as the steps above.

par.set_paramsvals_value("Maxwell.InitialData::System_to_use=System_II")
mwevol.VacuumMaxwellRHSs()

print("Consistency check between Tutorial-VacuumMaxwell_Cartesian_RHSs tutorial and NRPy+ module: ALL SHOULD BE ZERO.")

print("C - mwevol.C = " + str(C - mwevol.C))
print("G - mwevol.G = " + str(G - mwevol.G))
print("psi_rhs - mwevol.psi_rhs = " + str(psi_rhs - mwevol.psi_rhs))
print("Gamma_rhs - mwevol.Gamma_rhs = " + str(Gamma_rhs - mwevol.Gamma_rhs))
for i in range(DIM):

    print("ArhsU["+str(i)+"] - mwevol.ArhsU["+str(i)+"] = " + str(ArhsU[i] - mwevol.ArhsU[i]))
    print("ErhsU["+str(i)+"] - mwevol.ErhsU["+str(i)+"] = " + str(ErhsU[i] - mwevol.ErhsU[i]))
```

    Currently using System_II RHSs

    Consistency check between Tutorial-VacuumMaxwell_Cartesian_RHSs tutorial and NRPy+ module: ALL SHOULD BE ZERO.
    C - mwevol.C = 0
    G - mwevol.G = 0
    psi_rhs - mwevol.psi_rhs = 0
    Gamma_rhs - mwevol.Gamma_rhs = 0
    ArhsU[0] - mwevol.ArhsU[0] = 0
    ErhsU[0] - mwevol.ErhsU[0] = 0
    ArhsU[1] - mwevol.ArhsU[1] = 0
    ErhsU[1] - mwevol.ErhsU[1] = 0
    ArhsU[2] - mwevol.ArhsU[2] = 0
    ErhsU[2] - mwevol.ErhsU[2] = 0

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-VacuumMaxwell_Cartesian_RHSs.pdf](Tutorial-VacuumMaxwell_Cartesian_RHSs.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-VacuumMaxwell_Cartesian_RHSs")
```

    Created Tutorial-VacuumMaxwell_Cartesian_RHSs.tex, and compiled LaTeX file
        to PDF file Tutorial-VacuumMaxwell_Cartesian_RHSs.pdf


XRF quantification
========
The fluorescence/scattered intensity $I_f$ due to irradiation with flux $I_s$

$$
\begin{equation}
I_f=I_s(Hz)\Delta t(s)\Omega(sr)\sum_{i,j}\epsilon_{i,j} c_{i,j}(sr^{-1})
\end{equation}
$$

where $I$ the detected intensity (sum of selected fluorescence and or scattering lines), $\Delta t$ the exposure time, $\Omega$ the solid angle of the detector, $\epsilon_{i,j}$ a product of filter transmission and detector absorbance and $c_{i,j}$ the rate of line $j$ with energy $E_j$ due to source line $i$ with energy $E_i$ (depending on sample composition).

As an example, the fluorescence rate of a flat-multilayer sample can be written as (only primary interactions)

$$
\begin{equation}
\begin{split}
c_{i,j}(sr^{-1})=&\frac{d\mu_{i,j}}{d\Omega}\sum_k w_{j,k}\rho_k t_k^\prime(E_i,E_j)\\
\frac{d\mu_{i,j}^{fluo}}{d\Omega} =& \frac{\mu_j(E_i)}{4\pi}\\
\frac{d\mu_{i,j}^R}{d\Omega} =& r_e^2 K_R(\phi,\theta) \frac{N_A}{M_j}f_j^2(E_i,\theta)\\
\frac{d\mu_{i,j}^C}{d\Omega} =& r_e^2 K_C(\phi,\theta) \frac{N_A}{M_j}S_j(E_i,\theta)
\end{split}
\end{equation}
$$

where $k$ loops over the layers. Note that $j$ refers to a particular interaction type (fluorescence of element $Z$, elastic or inelastic scattering). See [polarization](polarization.ipynb) for the definition of the differential scattering cross-sections (SI units $cm^2/g/sr$ and $M_j$ the molar mass of the atom ($g/mol$), $N_A$ the Avogadro constant ($1/mol$), $f$ the atomic form factor and $S$ the incoherent scattering function of the atom).

The corrected layer thickness $t_k^\prime$ takes attenuation of primary X-rays and fluorescence/scattering into account. For a single layer in reflection geometry it can be written as

$$
\begin{equation}
\begin{split}
t^\prime(E_i,E_j) =& \frac{e^{\chi(E_i,E_j) t}-1}{\chi(E_i,E_j)\cos\alpha_{in}}\\
\chi(E_i,E_j) =& \rho\left(\frac{\mu(E_j)}{\cos\alpha_{out}}-\frac{\mu(E_i)}{\cos\alpha_{in}}\right)
\end{split}
\end{equation}
$$

where $\alpha$ the angle between the sample surface normal (pointing away from the source) and the incident(in) or fluorescence/scattering(out) direction ($\alpha_{out}>90^\circ$ in reflection geometry). Note that $\lim_{\chi\to\infty}t^\prime=\frac{t}{\cos\alpha_{in}}$.

## Geometry calibration

### Solid-angle parameterization (without standard)
See the notebook on [diodes](diodes.ipynb) on how $I_s$ is measured. We will assume the detector has a centric-cone geometry with solid angle

$$
\begin{equation}
\Omega=2\pi\left(1-\frac{x+d_0}{\sqrt{\frac{A}{\pi}+\left(x+d_0\right)^2}}\right)
\end{equation}
$$

where $A(mm^2)$ the active area of the detector, $x(mm)$ the position of the detector and $d_0(mm)$ the distance to the sample for $x=0$. To determine $A$ and $d_0$ we can measure the fluorescence of any sample as function of $x$:

$$
\begin{equation}
I_f(x,c,d_0,A)=c\Omega(x,d_0,A)
\end{equation}
$$

As an illustration we will define a detector geometry and multilayer sample. A thin-film standard is used here but any other material can be considered:

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline

from spectrocrunch.materials import xrfstandards
from spectrocrunch.detectors import xrf as xrfdetectors
from spectrocrunch.geometries import xrf as xrfgeometries
from spectrocrunch.sources import xray as xraysources

source = xraysources.factory("synchrotron")
detector = xrfdetectors.factory("leia")
geometry = xrfgeometries.factory("sxm120",detectorposition=-10,positionunits="mm",\
                                 detector=detector,source=source)

addnoise = True # add noise to simulations done below
method = "fisx" # XRF simulation method

realsample = xrfstandards.factory("RF7-200-S2371-03",geometry=geometry,\
                                             filmthickness=10e-7) # 10 nm
print(realsample)
```

    Multilayer (ordered top-bottom):
     Layer 0. 0.01 um (RF7-200-S2371-03)
     Layer 1. 0.2 um (silicon nitride)

Simulate a detector scan $I_f(x,c,d_0,A)$:

```python
from spectrocrunch.utils import units
from spectrocrunch.math import noisepropagation
from spectrocrunch.materials import pymca

# Geometry at which the data is collected
geometry.zerodistance = units.Quantity(5.,"cm")
detector.activearea = units.Quantity(70.,"mm^2")
print("\nTheoretical geometry:")
print(" Zero-distance: {:~}".format(geometry.zerodistance.to("cm")))
print(" Active area: {:~}".format(detector.activearea.to("mm^2")))

# Simulate measurement at current distance
energy = 7.3
flux = 1e9
time = 5
pymcahandle = pymca.PymcaHandle(sample=realsample,energy=energy,flux=flux,time=time,\
                                linear=True,escape=False,continuum=False,scatter=False)
mcaref = pymcahandle.mca(histogram=True,scattering=False,method=method)

# Simulate detector scan
n = 100
x = units.Quantity(np.linspace(-20,60,n),"mm")
I0  = np.full(n,flux*time)
solidangle = geometry.detector.solidangle_calc(activearea=detector.activearea,distance=x+geometry.zerodistance)
fluo = mcaref.sum()/geometry.solidangle*solidangle

if addnoise:
    I0 = np.random.poisson(np.round(I0).astype(int))
    fluo = np.random.poisson(np.round(fluo).astype(int))

fig,axs = plt.subplots(1,2,figsize=(12,5))

u = x.units
plt.sca(axs[0])
plt.plot(x,fluo/I0.astype(float))
xref = geometry.detectorposition.to(u).magnitude
iref = mcaref.sum()/(flux*time)
lines = plt.plot([xref,xref,x[0].magnitude],[0,iref,iref])
color = lines[0].get_color()
plt.ylabel("Normalized fluorescence")
plt.xlabel("Motor position ({:~})".format(u))

plt.sca(axs[1])
plt.plot(mcaref,color=color)
plt.gca().set_yscale('log', base=10)
plt.xlim([0,len(mcaref)-1])
plt.ylim([1,np.max(mcaref)*1.1])
plt.ylabel("ph/channel")
plt.xlabel("MCA channels")
plt.title("\nSpectrum at x ={:~}:".format(geometry.detectorposition.to("mm")))
plt.show()
```

Calibrate the geometry (starting from different values as the ones used to simulate the data):

```python
# Calibration resources
intensities = noisepropagation.poisson(fluo)/noisepropagation.poisson(I0)

calibrc = {"signal":noisepropagation.E(intensities),\
            "var":noisepropagation.VAR(intensities),\
            "detectorposition":x.magnitude,\
            "positionunits":x.units}

# Calibrate the geometry (starting from wrong values)
geometry.zerodistance += units.Quantity(-5.,"cm")
detector.activearea += units.Quantity(10.,"mm^2")

print("\nInitial geometry:")
print("Zero-distance: {:~}".format(geometry.zerodistance.to("cm")))
print("Active area: {:f~}".format(detector.activearea.to("mm^2")))

geometry.calibrate(calibrc=calibrc,plot=True,fit=True,fixedactivearea=False)
plt.show()

print("Calibrated geometry:")
print("Zero-distance: {:~}".format(geometry.zerodistance_rv.to("cm")))
print("Active area: {:f~}".format(detector.activearea_rv.to("mm^2")))
```

The correlation between $c$, $A$ and $d_0$ is too high to provide a usable result (also when fixing the active area).

### Solid-angle parameterization (with standard)

To solve the correlation issue, we determine $\Omega_{ref}$ at a particular motor position $x=x_{ref}$ by fitting the fluorescence spectrum of a standard measured with the detector at this position:

$$
\begin{equation}
\Omega_{ref}=\frac{I_{f}}{I_s(Hz)\Delta t(s)\sum_{i,j}\epsilon_{i,j} c_{i,j}(sr^{-1})}
\end{equation}
$$

where $I_s$, $\Delta t$, $\epsilon_{i,j}$ and $c_{i,j}$ are assumed to be known (flux measured by calibrated diodes, known sample and filter composition).

This provides a fixed relationship between $A$ and $d_0$ which can be substituted in the expression used for calibrating the geometry

$$
\begin{equation}
\begin{split}
I_f(x,c,d_0)=&c\Omega(x,d_0,A(d_0))\\
A(d_0)=&\pi\left(\frac{\left(x_{ref}+d_0\right)^2}{\left(1-\frac{\Omega_{ref}}{2\pi}\right)^2}-\left(x_{ref}+d_0\right)^2\right)
\end{split}
\end{equation}
$$

When using a thin-film standard, the thickness and density of the film are unknown but the areal densities of the elements in the film are known. For elements only present in the film and assuming absorption and other secondary effects are negligible, we can write

$$
\begin{equation}
\begin{split}
c_{i,j}^{film}=&\frac{d\mu_{i,j}}{d\Omega} w_{j}^{film}\rho_{film} t_{film}^{\prime}(E_i,E_j)\\
\approx&\frac{1}{\cos\alpha_{in}}\frac{d\mu_{i,j}}{d\Omega} w_{Z}^{film}\rho_{film} t_{film}\\
=&\frac{1}{\cos\alpha_{in}}\frac{d\mu_{i,j}}{d\Omega}\rho_{Z,A}^{film}
\end{split}
\end{equation}
$$

Hence for elements in the thin-film, it is enough to know their areal densities. In practice however we use mass fractions calculated from the areal densities using the density and the thickness of the substrate. The mass fractions obtained are physically meaningless but valid for the purpose of calculating $\Omega_{ref}$.

For elements in the substrate, density and thickness need to be known if self-absorption is not non-negligible:

$$
\begin{equation}
t_{subs}^{\prime}(E_i,E_j)\neq \frac{t_{subs}}{\cos\alpha_{in}}
\end{equation}
$$

Simulate and fit an XRF spectrum of a thin-film standard (simulation and fit are done with a different $d_0$ and $A$; scattering, escape and sum peaks are omitted):

```python
# Thin film standards have an unknown film thickness and density,
# only the areal densities of the different elements and the
# composition and thickness of the substrate are known.
thinfilmapprox = True

# Geometry at which the data is collected
geometry.zerodistance = units.Quantity(5.,"cm")
detector.activearea = units.Quantity(70.,"mm^2")
print("\nTheoretical geometry:")
print(" Zero-distance: {:~}".format(geometry.zerodistance_rv.to("cm")))
print(" Active area: {:~}".format(detector.activearea_rv.to("mm^2")))

# Simulate measurement (use the sample with known film thickness)
energy = 7.3
flux = 1e9
time = 1000 # the sum spectrum of a 2D map
pymcahandle = pymca.PymcaHandle(sample=realsample,energy=energy,flux=flux,time=time,\
                                linear=True,escape=False,continuum=False,scatter=False)
mca = pymcahandle.mca(histogram=True,scattering=False,method=method)
if method=="fisx":
    mca1 = mca
    mca2 = pymcahandle.mca(histogram=True,scattering=False,method="analytical")
if addnoise:
    mca = np.random.poisson(np.round(mca).astype(int))

# Calibrate with unknown film thickness
if thinfilmapprox:
    thinfilmsample = xrfstandards.factory("RF7-200-S2371-03",geometry=geometry)
    pymcahandle.sample = thinfilmsample

# Initialize fit with the wrong geometry
pymcahandle.setdata(mca)
geometry.zerodistance += units.Quantity(5.,"cm")
detector.activearea += units.Quantity(-10.,"mm^2")
pymcahandle.addtopymca(fresh=True)

# Adapt config manually if needed:
#config = pymcahandle.mcafit.getConfiguration()
#config["fit"]["stripflag"] = 0
#...
#pymcahandle.mcafit.configure(config)

# Perform fit
fitresult = pymcahandle.fit()

# Print errors
def strwerror(e,wfrac,exwfrac):
    error = (wfrac-exwfrac)/exwfrac
    return " {}: {:6.02f} wt%     (expected: {:6.02f} wt%,     error: {:.02f}%)".\
                  format(e,wfrac*100,exwfrac*100,error*100)

def straderror(e,ad,exad):
    error = (ad-exad)/exad
    return " {}: {:6.02f} ng/mm^2 (expected: {:6.02f} ng/mm^2, error: {:.02f}%)".\
                  format(e,ad*1e7,exad*1e7,error*100)

def printerrors(fitresult,sample):
    out = {}
    if thinfilmapprox:
        exarealdensities = sample.arealdensity()

        rho = sample[0].density
        t = sample[0].thickness

        for k,wfrac in fitresult["massfractions"].items():
            element = k.element
            ad = wfrac*rho*t
            if element in exarealdensities:
                exad = exarealdensities[element]
                exwfrac = exad/(rho*t)
                out[element] = {"ad":straderror(element,ad,exad),\
                                "wfrac":strwerror(element,wfrac,exwfrac)}

    else:
        exarealdensities = sample.arealdensity()

        arealdensities = {}
        massfractions = {}
        exmassfractions = {}
        exarealdensities = {}
        for layer,wfracs in zip(sample,fitresult["lmassfractions"]):
            rho = layer.density
            t = layer.thickness
            exwfracs = layer.elemental_massfractions()
            exad = layer.arealdensity()
            for k,wfrac in wfracs.items():
                if wfrac!=0:
                    element = k.element
                    arealdensities[k] = wfrac*rho*t
                    massfractions[k] = wfrac
                    exmassfractions[k] = exwfracs[element]
                    exarealdensities[k] = exad[element]

        for k,wfrac in massfractions.items():
            if k in exmassfractions:
                element = k.element
                exwfrac = exmassfractions[k]
                exad = exarealdensities[k]
                ad = arealdensities[k]
                out[element] = {"ad":straderror(element,ad,exad),\
                                "wfrac":strwerror(element,wfrac,exwfrac)}
    print(" Mass fractions and areal densities (within one layer):")
    for k in out:
        print(out[k]["wfrac"])
        print(out[k]["ad"])

print("\nFitted vs. theory (before geometry calibration):")
printerrors(fitresult,pymcahandle.sample)

# Plot fit
def plotfit(fitresult):
    plt.plot(fitresult["energy"],fitresult["y"],label='data')
    plt.plot(fitresult["energy"],fitresult["yfit"],label='pymca fit')
    backfunc = fitresult["interpol_energy"](fitresult["yback"])
    plt.plot(fitresult["energy"],backfunc(fitresult["energy"]),label='background')
    plt.gca().set_yscale('log', base=10)
    plt.ylim([1,np.max(fitresult["y"])*1.1])
    plt.ylabel("ph/channel")
    plt.xlabel("Energy (keV)")

plotfit(fitresult)
plt.show()

if method=="fisx":
    plt.plot(mca1,label="fisx")
    plt.plot(mca2,label="xraylib")
    plt.gca().set_yscale('log', base=10)
    plt.ylim([1,np.max(mca)*1.1])
    plt.ylabel("ph/ch")
    plt.xlabel("Channels")
    plt.legend()
    plt.show()
```

Determine $\Omega_{ref}$ by comparing the fitted and theoretical fluorescence intensities:

```python
caliblines = ["Ca"]
useline = lambda k: any(str(k).startswith(e) for e in caliblines)

# rate = Ifluo/I0 with I0 = flux * time
Rfit = {k:v for k,v in fitresult["fitrates"].items() if useline(k)}
Rinit = {k:v for k,v in fitresult["rates"].items() if useline(k)}
if thinfilmapprox:
    # for an element within the film:
    # - pymca mass fraction = 1
    # - substrate density and thicknes
    rho = pymcahandle.sample[0].density
    t = pymcahandle.sample[0].thickness
    arealdensities = pymcahandle.sample.arealdensity()
    substrate = pymcahandle.sample[0].elements
    for k in Rinit:
        el = k.element
        if el not in substrate:
            Rinit[k] *= arealdensities[el]/(rho*t)

solidangleref = geometry.solidangle * sum(Rfit.values())/sum(Rinit.values())
```

Calibrate the geometry ($d_0$ and $A$) with a known $[\Omega_{ref},x_{ref}]$ pair as constraint:

```python
geometry.calibrate(calibrc=calibrc,solidanglecalib=solidangleref,\
                   plot=True,fit=True,fixedactivearea=False)

# Force to real values for testing:
#geometry.zerodistance = units.Quantity(5,"cm")
#detector.activearea = units.Quantity(70,"mm^2")

print("\nCalibrate geometry using {}:".format(caliblines))
print(" Zero-distance: {:~}".format(geometry.zerodistance_rv.to("cm")))
print(" Active area: {:~}".format(detector.activearea_rv.to("mm^2")))

print("\nCurrent distance:")
print(geometry.detectorposition)
print(" Motor position = {:~}".format(geometry.detectorposition))
print(" Distance: {:~}".format(geometry.distance_rv.to("cm")))
```

The correlation between the two unknowns $c$ and $d_0$ is low enough to provide estimates of $d_0$ and $A$ with acceptable uncertainty. Known and fitted areal densities should be the same:

```python
pymcahandle.addtopymca(fresh=False)
fitresult = pymcahandle.fit()
print("\nFitted vs. theory (after geometry calibration):")
printerrors(fitresult,pymcahandle.sample)

plt.figure(figsize=(12,5))
plotfit(fitresult)

spectrum = realsample.xrayspectrum(energy,emin=1,emax = energy+0.5,scattering=False,method=method)

matplotlib.rcParams.update({'font.size': 15})
spectrum.plot(histogram=True,decompose=True,fluxtime=pymcahandle.I0,\
              legend=False,forcelines=True)
matplotlib.rcParams.update({'font.size': 14})

plt.show()
```

Errors originate from least-squares fitting, the thin-film approximation (if enabled) and discrepancies between the cross-sections used to similate fluorescence and the ones used for fitting (if method not "fisx").


#
EE 380L: Data Mining

#
Assignment 4

##
Total points: 70

##
Due: Thursday, April 5th, submitted via Canvas by 11:59 pm

Your homework should be written in a **Jupyter notebook**. You may work in groups of two if you wish. Only one student per team needs to submit the assignment on Canvas.  But be sure to include name and UTID for both students.  Homework groups will be created and managed through Canvas, so please do not arbitrarily change your homework group.

Also, please make sure your code runs and the graphics (and anything else) are displayed in your notebook before submitting. (%matplotlib inline)

## Name: Chia-Ju Chen
## UT EID: cc65542

## Name: Minke Cheng
## UT EID: mc65568

### Note: Q1 and Q3 may take a long time to run (a few hours). It will be more convenient if you run them on a powerful machine. Of course, this is also a good opportunity to use the Azure cloud.

# Question 1:  2 class Classification (30 pts)

In this problem, you will evaluate the performance of 6 classifiers in classifying signal vs background in the MAGIC Gamma Telescope Data Set. The data has been split into training and test as well as standardized for you.
You will train the following models on this dataset:

1. Gaussian Naive Bayes (http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html)
2. Linear Discriminant Analysis (http://scikit-learn.org/stable/modules/lda_qda.html)
3. Quadratic Discriminant Analysis
4. Logistic Regression (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html)
5. Linear SVM (http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC)
6. Radial Basis Function SVM.

For Logistic Regression, you will have to run kFold cross validation with 5 folds to get the best value of C. C is the inverse of the regularization hyper-parameter.
For SVM (linear and rbf), you will use GridSearchCV (http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html) with cv set to 5, to identify the correct value of C. Use `Cs = 10**np.linspace(-2, 2, 10)` as the range of C for SVM gridsearch

Note: Maintain all other parameters at their default settings.

The deliverables for this Problem are the following:
1. Use cross validation to find the best C for logistic regression, linear SVM and rbf SVM. (C means different things for each of the models. Please refer the documentation of each API in sklearn for this) (9 points - 3 points per model)
2. Report the Mean Error Rate = (1 - model_score) for each of these 3 models. (9 points - 3 points per model)
3. Train all 6 models using the best C for logistic regression, linear SVM and rbf SVM. Plot the ROC curve for each of these in one figure. Report the area under the curve (AUC) and accuracy for all 6. (12 points)

Note: SVM by default only learns a decision function. In order to obtain ROC curve, you will need prediction probabilities as well. This will require you to set probability=True when training SVM models.

Note: It is a good idea to measure the time taken to run the fit function for each hyperparameter setting to estimate the time for gridSearchCv on a range of hyperparameters.

```python
%matplotlib inline
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn import preprocessing
from sklearn.model_selection import KFold
from sklearn.linear_model import LogisticRegressionCV, LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
import numpy as np
import matplotlib.pyplot as plt
```

```python
# load data
# train
train = pd.read_csv('magic_train.csv', header=None)
y_train = train.values[:,10]
y_train[y_train == 'g'] = 0
y_train[y_train == 'h'] = 1
y_train = y_train.astype(float)
X_train = train.values[:,:10]

# test
test = pd.read_csv('magic_test.csv', header=None)
y_test = test.values[:,10]
y_test[y_test == 'g'] = 0
y_test[y_test == 'h'] = 1
y_test = y_test.astype(float)
X_test = test.values[:,:10]

# standardize the data
X_train = preprocessing.scale(X_train)
X_test = preprocessing.scale(X_test)
```

    /home/cmk13/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:475: DataConversionWarning: Data with input dtype object was converted to float64 by the scale function.
      warnings.warn(msg, DataConversionWarning)

### 1

```python
# To find the best C for logistc regression
kf = KFold(n_splits = 5)
LR = LogisticRegressionCV(cv=kf)

LR.fit(X_train, y_train)

print("Best C for logistic regression: " + str(LR.C_[0]))
```

    Best C for logistic regression: 2.782559402207126

```python
parameters = {'C':10**np.linspace(-2, 2, 10)}
svc = SVC(kernel='linear')
clf = GridSearchCV(svc, parameters, cv=5)
clf.fit(X_train, y_train)
```

    GridSearchCV(cv=5, error_score='raise',
           estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False),
           fit_params=None, iid=True, n_jobs=1,
           param_grid={'C': array([1.00000e-02, 2.78256e-02, 7.74264e-02, 2.15443e-01, 5.99484e-01,
           1.66810e+00, 4.64159e+00, 1.29155e+01, 3.59381e+01, 1.00000e+02])},
           pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',
           scoring=None, verbose=0)

```python
print("Best C for linear SVM: " + str(clf.best_params_['C']))
```

    Best C for linear SVM: 0.027825594022071243

```python
parameters_rbf = {'C':10**np.linspace(-2, 2, 10)}
svc_rbf = SVC(kernel='rbf')
clf_rbf = GridSearchCV(svc_rbf, parameters_rbf, cv=5)
clf_rbf.fit(X_train, y_train)
```

    GridSearchCV(cv=5, error_score='raise',
           estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False),
           fit_params=None, iid=True, n_jobs=1,
           param_grid={'C': array([1.00000e-02, 2.78256e-02, 7.74264e-02, 2.15443e-01, 5.99484e-01,
           1.66810e+00, 4.64159e+00, 1.29155e+01, 3.59381e+01, 1.00000e+02])},
           pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',
           scoring=None, verbose=0)

```python
print("Best C for rbf SVM: " + str(clf_rbf.best_params_['C']))
```

    Best C for rbf SVM: 100.0

### 2

```python
print("Mean Error Rate for logistic regression is: " + str(1 - LR.score(X_test, y_test)))

svc_Linear = SVC(kernel='linear', C=clf.best_params_['C'])
svc_Linear.fit(X_train, y_train)
print("Mean Error Rate for linear SVM is: " + str(1 - svc_Linear.score(X_test, y_test)))

svc_RBF = SVC(kernel='rbf', C=clf_rbf.best_params_['C'])
svc_RBF.fit(X_train, y_train)
print("Mean Error Rate for rbf SVM is: " + str(1 - svc_RBF.score(X_test, y_test)))
```

    Mean Error Rate for logistic regression is: 0.20376831821353802
    Mean Error Rate for linear SVM is: 0.20219818562456382
    Mean Error Rate for rbf SVM is: 0.1292742498255408

### 3

```python
clf_GNB = GaussianNB()
clf_GNB.fit(X_train, y_train)
y_GNB_predict = clf_GNB.predict_proba(X_test)
fpr_GNB, tpr_GNB, thresholds_GNB = roc_curve(y_test, y_GNB_predict[:,1])
AUC_GNB = roc_auc_score(y_test, y_GNB_predict[:,1])
```

```python
clf_LDA = LDA()
clf_LDA.fit(X_train, y_train)
y_LDA_predict = clf_LDA.predict_proba(X_test)
fpr_LDA, tpr_LDA, thresholds_LDA = roc_curve(y_test, y_LDA_predict[:,1])
AUC_LDA = roc_auc_score(y_test, y_LDA_predict[:,1])
```

```python
clf_QDA = QDA()
clf_QDA.fit(X_train, y_train)
y_QDA_predict = clf_QDA.predict_proba(X_test)
fpr_QDA, tpr_QDA, thresholds_QDA = roc_curve(y_test, y_QDA_predict[:,1])
AUC_QDA = roc_auc_score(y_test, y_QDA_predict[:,1])
```

```python
y_LR_predict = LR.predict_proba(X_test)
fpr_LR, tpr_LR, thresholds_LR = roc_curve(y_test, y_LR_predict[:,1])
AUC_LR = roc_auc_score(y_test, y_LR_predict[:,1])
```

```python
clf_Linear_SVC = SVC(kernel='linear', C=clf.best_params_['C'], probability=True)
clf_Linear_SVC.fit(X_train, y_train)
y_Linear_SVC_predict = clf_Linear_SVC.predict_proba(X_test)
fpr_Linear_SVC, tpr_Linear_SVC, thresholds_Linear_SVC = roc_curve(y_test, y_Linear_SVC_predict[:,1])
AUC_Linear_SVC = roc_auc_score(y_test, y_Linear_SVC_predict[:,1])
```

```python
clf_RBF_SVC = SVC(kernel='rbf', C=clf_rbf.best_params_['C'], probability=True)
clf_RBF_SVC.fit(X_train, y_train)
y_RBF_SVC_predict = clf_RBF_SVC.predict_proba(X_test)
fpr_RBF_SVC, tpr_RBF_SVC, thresholds_RBF_SVC = roc_curve(y_test, y_RBF_SVC_predict[:,1])
AUC_RBF_SVC = roc_auc_score(y_test, y_RBF_SVC_predict[:,1])
```

```python
GNB_plot, = plt.plot(fpr_GNB, tpr_GNB, 'r', label="GNB: AUC = " + str(AUC_GNB))
LDA_plot, = plt.plot(fpr_LDA, tpr_LDA, 'b', label="LDA: AUC = " + str(AUC_LDA))
QDA_plot, = plt.plot(fpr_QDA, tpr_QDA, 'y', label="QDA: AUC = " + str(AUC_QDA))
LR_plot, = plt.plot(fpr_LR, tpr_LR, 'g', label="Logistic: AUC = " + str(AUC_LR))
Linear_plot, = plt.plot(fpr_Linear_SVC, tpr_Linear_SVC, 'c', label="Linear SVM: AUC = " + str(AUC_Linear_SVC))
RBF_plot, = plt.plot(fpr_RBF_SVC, tpr_RBF_SVC, 'm', label="RBF SVM: AUC = " + str(AUC_RBF_SVC))

plt.legend(handles=[GNB_plot, LDA_plot, QDA_plot, LR_plot, Linear_plot, RBF_plot])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC curves')
plt.show()

print("AUC for GNB model: ", AUC_GNB)
print("AUC for LDA model: ", AUC_LDA)
print("AUC for QDA model: ", AUC_QDA)
print("AUC for Logistic Regression model: ", AUC_LR)
print("AUC for linear SVM model: ", AUC_Linear_SVC)
print("AUC for RBF SVM model: ", AUC_RBF_SVC)
print("\n")
print("Accuracy for GNB model: ", clf_GNB.score(X_test, y_test))
print("Accuracy for LDA model: ", clf_LDA.score(X_test, y_test))
print("Accuracy for QDA model: ", clf_QDA.score(X_test, y_test))
print("Accuracy for Logistic Regression model: ", LR.score(X_test, y_test))
print("Accuracy for linear SVM model: ", clf_Linear_SVC.score(X_test, y_test))
print("Accuracy for RBF SVM model: ", clf_RBF_SVC.score(X_test, y_test))
```

# Question 2: Bayesian Networks (10 pts)
In the Bayesian Network shown below, the nodes represent the following random variables: S stands for "Smoking", LC for "Lung Cancer", B for "Bronchitis", T for "Tuberculosis", D for "Dyspnea". The necessary conditional probabilities are provided alongside.

```python
from IPython.display import Image
Image(filename='BayesianNetworks.png')
```

$$Pr(S = yes) = 0.1$$
$$Pr(LC = yes | S = yes) = 0.03$$
$$Pr(LC = yes | S = no) = 0.01$$
$$Pr(B = yes | S = yes) = 0.06$$
$$Pr(B = yes | S = no) = 0.02$$
$$Pr(T = yes | LC = yes, B = yes) = 0.8$$
$$Pr(T = yes | LC = yes, B = no) = 0.5$$
$$Pr(T = yes | LC = no, B = yes) = 0.1$$
$$Pr(T = yes | LC = no, B = no) = 0.04$$
$$Pr(D = yes | T = yes, B = yes) = 0.9$$
$$Pr(D = yes | T = yes, B = no) = 0.7$$
$$Pr(D = yes | T = no, B = yes) = 0.65$$
$$Pr(D = yes | T = no, B = no) = 0.015$$

Compute the following probabilities:

(a) (2 pts) $Pr(D = yes)$

(b) (2 pts) $Pr(LC = no, B = yes, D = yes)$

(c) (3 pts) Given that a person has Bronchitis (B = yes) and no Lung Cancer (LC = no), the probability he will has symptoms of Dyspnea (D = yes).

(d) (3 pts) Given that a person is a smoker (S = yes), the probability he will get infected with Tuberculosis (T = yes).

# Answer
### (a)
Firstly, we have:
$$
\begin{align}
\\
Pr(B = yes) & = Pr(B = yes | S = yes) \cdot Pr(S = yes) + Pr(B = yes | S = no) \cdot Pr(S = no) \\
& = 0.06 * 0.1 + 0.02 * 0.9 \\
& = 0.024
\end{align}
\\
$$
$$
\begin{align}
Pr(LC = yes) & = Pr(LC = yes | S = yes) \cdot Pr(S = yes) + Pr(LC = yes | S = no) \cdot Pr(S = no) \\
& = 0.03 * 0.1 + 0.01 * 0.9 \\
& = 0.012
\end{align}
$$

Then, we can get:
$$
\begin{align}
\\
Pr(T = yes | B = yes) & = Pr(T = yes | LC = yes, B = yes) \cdot Pr(LC = yes) + Pr(T = yes | LC = no, B = yes) \cdot Pr(LC = no)\\
& = 0.8 * 0.012 + 0.1 * 0.988 \\
& = 0.1084
\end{align} \\
$$
$$
\begin{align}
Pr(T = yes | B = no) & = Pr(T = yes | LC = yes, B = no) \cdot Pr(LC = yes) + Pr(T = yes | LC = no, B = no) \cdot Pr(LC = no)\\
& = 0.5 * 0.012 + 0.04 * 0.988 \\
& = 0.04552
\end{align} \\
$$
$$
\begin{align}
Pr(T = no | B = yes) & = Pr(T = no | LC = yes, B = yes) \cdot Pr(LC = yes) + Pr(T = no | LC = no, B = yes) \cdot Pr(LC = no)\\
& = 0.2 * 0.012 + 0.9 * 0.988 \\
& = 0.8916
\end{align} \\
$$
$$
\begin{align}
Pr(T = no | B = no) & = Pr(T = no | LC = yes, B = no) \cdot Pr(LC = yes) + Pr(T = no | LC = no, B = no) \cdot Pr(LC = no)\\
& = 0.5 * 0.012 + 0.96 * 0.988 \\
& = 0.95448
\end{align} \\
$$

Based on the results above, we have:
$$
\begin{align}
\\
Pr(T = yes , B = yes) & = Pr(T = yes | B = yes) \cdot Pr(B = yes) \\
& = 0.1084 * 0.024 \\
& = 2.6016 \times 10^{-3}
\end{align} \\
$$
$$
\begin{align}
Pr(T = yes , B = no) & = Pr(T = yes | B = no) \cdot Pr(B = no) \\
& = 0.04552 * 0.976 \\
& = 0.04442752
\end{align} \\
$$
$$
\begin{align}
Pr(T = no , B = yes) & = Pr(T = no | B = yes) \cdot Pr(B = yes) \\
& = 0.8916 * 0.024 \\
& = 0.0213984
\end{align} \\
$$
$$
\begin{align}
Pr(T = no , B = no) & = Pr(T = no | B = no) \cdot Pr(B = no) \\
& = 0.95448 * 0.976 \\
& = 0.93157248
\end{align} \\
$$

So, we can get:
$$
\begin{align}
Pr(D = yes) = \ & Pr(D = yes | T = yes, B = yes) * Pr(T = yes, B = yes) + \\
& Pr(D = yes | T = yes, B = no) * Pr(T = yes, B = no) + \\
& Pr(D = yes | T = no, B = yes) * Pr(T = no, B = yes) + \\
& Pr(D = yes | T = no, B = no) * Pr(T = no, B = no) \\
= \ & 0.9 * 0.0026016 + 0.7 * 0.04442752 + 0.65 * 0.0213984 + 0.015 * 0.93157248 \\
= \ & 0.0613232512
\end{align}
$$

### (b)
Firstly, we know LC is independent with B with condition S, so we have:
$$
\begin{align}
\\
Pr(LC = no, B = yes) = & \ Pr(LC = no | S = yes) \cdot Pr(B = yes | S = yes) \cdot Pr(S = yes) +\\
& \ Pr(LC = no | S = no) \cdot Pr(B = yes | S = no) \cdot Pr(S = no) \\
= & \ 0.97 * 0.06 * 0.1 + 0.99 * 0.02 * 0.9 \\
= & \ 0.02364
\end{align} \\
$$

Then we calculate the following:
$$
\begin{align}
\\
Pr(D = yes | LC = no, B = yes) = & \ Pr(D = yes | T = yes, B = yes) * Pr(T = yes| LC = no, B = yes) + \\
& \ Pr(D = yes | T = no, B = yes) * Pr(T = no| LC = no, B = yes) \\
= & \ 0.9 * 0.1 + 0.65 * 0.9 \\
= & \ 0.675
\end{align}
$$

Now, we calculate the final result of the target probability:
$$
\begin{align}
\\
Pr(LC = no, B = yes, D = yes) = & \ Pr(D = yes | LC = no, B = yes) \cdot Pr(LC = no, B = yes) \\
= & \ 0.675 * 0.02364 \\
= & \ 0.015957
\end{align}
$$

### (C)
The target probability can be written as:
$$Pr(D = yes | B = yes, LC = no)$$

The value of it has been calaulated above in (b), which is 0.675.

### (d)
The target probability can be written as:
$$Pr(T = yes | S = yes)$$

LC and B are independent with each other when given S. Based on what we are given, we get:
$$
\begin{align}\\
Pr(LC = yes, B = yes | S = yes) = & \ Pr(LC = yes | S = yes) \cdot Pr(B = yes | S = yes) \\
= & \ 0.03 * 0.06 \\
= & \ 0.0018
\end{align} \\
$$
$$
\begin{align}
Pr(LC = yes, B = no | S = yes) = & \ Pr(LC = yes | S = yes) \cdot Pr(B = no | S = yes) \\
= & \ 0.03 * 0.94 \\
= & \ 0.0282
\end{align} \\
$$
$$
\begin{align}
Pr(LC = no, B = yes | S = yes) = & \ Pr(LC = no | S = yes) \cdot Pr(B = yes | S = yes) \\
= & \ 0.97 * 0.06 \\
= & \ 0.0582
\end{align} \\
$$
$$
\begin{align}
Pr(LC = no, B = no | S = yes) = & \ Pr(LC = no | S = yes) \cdot Pr(B = no | S = yes) \\
= & \ 0.97 * 0.94 \\
= & \ 0.9118
\end{align} \\
$$

So the target probability should be:
$$
\begin{align} \\
Pr(T = yes | S = yes) = & \ Pr(T = yes | LC = yes, B = yes) \cdot Pr(LC = yes, B = yes | S = yes) + \\
& \ Pr(T = yes | LC = yes, B = no) \cdot Pr(LC = yes, B = no | S = yes) + \\
& \ Pr(T = yes | LC = no, B = yes) \cdot Pr(LC = no, B = yes | S = yes) + \\
& \ Pr(T = yes | LC = no, B = no) \cdot Pr(LC = no, B = no | S = yes) \\
= & \ 0.8 * 0.0018 + 0.5 * 0.0282 + 0.1 * 0.0582 + 0.04 * 0.9118 \\
= & \ 0.057832
\end{align}
$$

# Question 3: (15 pts)

In this problem we will be classifying data in the CIFAR-10 dataset which contains 60000 images classified in 10 mutually exclusive classes. Please download and extract the data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz

For parts 1-3 we will build a classifier to determine whether an image is in class 0 or class 1. In part 4, we will classify the imges into any of 10 classes. We've included starter code to load the data from the unzipped folder.

1. Train a logistic regression with almost no regularization (pass l2 (ridge) to penalty and 1,000,000 to the C parameter which is the inverse of regularization strength lambda. This essentially does l2 regularization but applies very little weight to the penalty term) and report the [confusion matrix](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) on the test data. Also report the accuracy for the "y = 0" class, the "y = 1" class, and the average per-class accuracy on the test data. Average per-class accuracy is described in this [post](http://rasbt.github.io/mlxtend/user_guide/evaluate/scoring/). You can use your confusion matrix to calculate this.  (4pts)
2. Repeat step 1 except use l2 penalty with Cs of [0.01, 0.1, 1, 10 ,100]. You will want to use 3-fold cross validation to select the best parameter. To evaluate which parameter is best, maximize the average per-class accuracy. To help with this task, check out [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV) and how to make your own [custom scorer](http://scikit-learn.org/stable/modules/model_evaluation.html).  (4pts)
3. Repeat question 2 except use l1 instead of l2 as the penalty type, use Cs of  [0.01, ..., 100].  Compare the performances of the models (no regularization, l2, l1) using the average per-class accuracy.  (3pts)
4. Using the optimal regularization parameter obtained in step 2, train a logistic regression classifier with ridge penalty to clasify $\textbf{all 10 digits}$ (digits 0,1,2,3,...9) of the MNIST dataset.  Use 'newton-cg' and 'multinomial' options for solver and multi_class parameters, respectively. Report the average per-class accuracy and confusion matrix (10 X 10) on the test data. (4pts)

```python
import numpy as np
import numpy.linalg as la
import matplotlib.pyplot as plt
import _pickle as cPickle
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import mean_squared_error, confusion_matrix
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score

def unpickle(path):
    with open(path, 'rb') as fo:
      data = cPickle.load(fo, encoding='latin1')
    return data

def load_data(path, test=False):
    data = []
    labels = []
    if test:
        batch = unpickle(path + '/test_batch')
        data = batch['data']
        labels = batch['labels']
    else:
        for batch_num in range(1, 5):
            batch = unpickle(path + '/data_batch_' + str(batch_num))
            if len(data) == 0:
                data = batch['data']
            else:
                data = np.concatenate([data, batch['data']], axis=0)
            labels += batch['labels']
    data = np.array(data)
    labels = np.array(labels)
    return data, labels
```

```python
all_train_data, all_train_labels = load_data('./cifar-10-batches-py')
all_test_data, all_test_labels = load_data('./cifar-10-batches-py', test=True)

two_class_train_data = all_train_data[(all_train_labels == 0) | (all_train_labels == 1)]
two_class_train_labels = all_train_labels[(all_train_labels == 0) | (all_train_labels == 1)]

two_class_test_data = all_test_data[(all_test_labels == 0) | (all_test_labels == 1)]
two_class_test_labels = all_test_labels[(all_test_labels == 0) | (all_test_labels == 1)]

X_train = (two_class_train_data / 255.)
y_train = two_class_train_labels

X_test = (two_class_test_data / 255.)
y_test = two_class_test_labels
```

```python
y_test.shape
```

    (2000,)

```python
X_train = X_train[np.array(range(0, 7972, 10))]
y_train = y_train[np.array(range(0, 7972, 10))]

X_test = X_test[np.array(range(0, 2000, 10))]
y_test = y_test[np.array(range(0, 2000, 10))]
```

# Answer
### 1

```python
logistic_model = LogisticRegression(penalty='l2',C=1000000)
logistic_model.fit(X_train, y_train)

y_predict = logistic_model.predict(X_test)
con_matrix = confusion_matrix(y_test, y_predict)

accuracy_y0 = np.sum([(y_predict[i] == 0) & (y_test[i] == 0) for i in range(len(y_test))]) / np.sum(y_test == 0)
accuracy_y1 = np.sum([(y_predict[i] == 1) & (y_test[i] == 1) for i in range(len(y_test))]) / np.sum(y_test == 1)

ERR = (con_matrix[0][1] + con_matrix[1][0]) / np.sum(con_matrix)
ACC = 1 - ERR

print("The confusion matrix is: ")
print(con_matrix)
print("Accuracy for the \"y = 0\" class: ", accuracy_y0)
print("Accuracy for the \"y = 1\" class: ", accuracy_y1)
print("Average per-class accuracy on the test data is: ", ACC)
```

    The confusion matrix is:
    [[78 17]
     [30 75]]
    Accuracy for the "y = 0" class:  0.8210526315789474
    Accuracy for the "y = 1" class:  0.7142857142857143
    Average per-class accuracy on the test data is:  0.765

### 2

```python
from sklearn.metrics import make_scorer

def my_custom_scorer(y_test, y_predict):
    con_matrix = confusion_matrix(y_test, y_predict)
    ERR = (con_matrix[0][1] + con_matrix[1][0]) / np.sum(con_matrix)
    ACC = 1 - ERR
    return ACC

custom_score = make_scorer(my_custom_scorer, greater_is_better=True)
parameters_LR = {'C':[0.01, 0.1, 1, 10 ,100]}

logistic_model2 = LogisticRegression(penalty='l2')
clf_LR_model2 = GridSearchCV(logistic_model2, parameters_LR, cv=3, scoring=custom_score)
clf_LR_model2.fit(X_train, y_train)
```

    GridSearchCV(cv=3, error_score='raise',
           estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False),
           fit_params=None, iid=True, n_jobs=1,
           param_grid={'C': [0.01, 0.1, 1, 10, 100]}, pre_dispatch='2*n_jobs',
           refit=True, return_train_score='warn',
           scoring=make_scorer(my_custom_scorer), verbose=0)

```python
print("Best C using l2 penalty to maximize the average per-class accuracy: ", clf_LR_model2.best_params_['C'])
```

    Best C using l2 penalty to maximize the average per-class accuracy:  0.01

```python
LR_l2_BestC = LogisticRegression(penalty='l2', C=clf_LR_model2.best_params_['C'])
LR_l2_BestC.fit(X_train, y_train)

y_LR_l2_predict = LR_l2_BestC.predict(X_test)
con_LR_l2_matrix = confusion_matrix(y_test, y_LR_l2_predict)

accuracy_y0_l2 = np.sum([(y_LR_l2_predict[i] == 0) & (y_test[i] == 0) for i in range(len(y_test))]) / np.sum(y_test == 0)
accuracy_y1_l2 = np.sum([(y_LR_l2_predict[i] == 1) & (y_test[i] == 1) for i in range(len(y_test))]) / np.sum(y_test == 1)

ERR_l2 = (con_LR_l2_matrix[0][1] + con_LR_l2_matrix[1][0]) / np.sum(con_LR_l2_matrix)
ACC_l2 = 1 - ERR_l2

print("The confusion matrix for l2 penalty is: ")
print(con_LR_l2_matrix)
print("Accuracy for the \"y = 0\" class: ", accuracy_y0_l2)
print("Accuracy for the \"y = 1\" class: ", accuracy_y1_l2)
print("Average per-class accuracy on the test data is: ", ACC_l2)
```

    The confusion matrix for l2 penalty is:
    [[78 17]
     [27 78]]
    Accuracy for the "y = 0" class:  0.8210526315789474
    Accuracy for the "y = 1" class:  0.7428571428571429
    Average per-class accuracy on the test data is:  0.78

### 3

```python
parameters_LR3 = {'C':[0.01, 0.1, 1, 10 ,100]}

logistic_model3 = LogisticRegression(penalty='l1')
clf_LR_model3 = GridSearchCV(logistic_model3, parameters_LR3, cv=3, scoring=custom_score)
clf_LR_model3.fit(X_train, y_train)
```

    GridSearchCV(cv=3, error_score='raise',
           estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l1', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False),
           fit_params=None, iid=True, n_jobs=1,
           param_grid={'C': [0.01, 0.1, 1, 10, 100]}, pre_dispatch='2*n_jobs',
           refit=True, return_train_score='warn',
           scoring=make_scorer(my_custom_scorer), verbose=0)

```python
print("Best C using l1 penalty to maximize the average per-class accuracy: ", clf_LR_model3.best_params_['C'])
```

    Best C using l1 penalty to maximize the average per-class accuracy:  1

```python
LR_l1_BestC = LogisticRegression(penalty='l1', C=clf_LR_model3.best_params_['C'])
LR_l1_BestC.fit(X_train, y_train)

y_LR_l1_predict = LR_l1_BestC.predict(X_test)
con_LR_l1_matrix = confusion_matrix(y_test, y_LR_l1_predict)

accuracy_y0_l1 = np.sum([(y_LR_l1_predict[i] == 0) & (y_test[i] == 0) for i in range(len(y_test))]) / np.sum(y_test == 0)
accuracy_y1_l1 = np.sum([(y_LR_l1_predict[i] == 1) & (y_test[i] == 1) for i in range(len(y_test))]) / np.sum(y_test == 1)

ERR_l1 = (con_LR_l1_matrix[0][1] + con_LR_l1_matrix[1][0]) / np.sum(con_LR_l1_matrix)
ACC_l1 = 1 - ERR_l1

print("The confusion matrix for l1 penalty is: ")
print(con_LR_l1_matrix)
print("Accuracy for the \"y = 0\" class: ", accuracy_y0_l1)
print("Accuracy for the \"y = 1\" class: ", accuracy_y1_l1)
print("Average per-class accuracy on the test data is: ", ACC_l1)
```

    The confusion matrix for l1 penalty is:
    [[75 20]
     [23 82]]
    Accuracy for the "y = 0" class:  0.7894736842105263
    Accuracy for the "y = 1" class:  0.780952380952381
    Average per-class accuracy on the test data is:  0.785

According to the result above, the average per-class accuracy of each model is shown here:
No regularization: 0.765
l2 penalty: 0.78
l1 penalty: 0.785
We can find that lasso performs the best while no regularization performs the worst. Ridge performs little worse than lasso. The part of the reason can be, we defined our own scorer when using ridge and lasso, which aims at finding the maximum of the average per-class accuracy. Therefore, it is reasonable that lasso and ridge both perform better than no regularization. Besides, over-fit can be reduced with penalty, which also accounts for the result above.

### 4

```python
X_train = (all_train_data / 255.)
y_train = all_train_labels

X_test = (all_test_data / 255.)
y_test = all_test_labels

LR_all_data = LogisticRegression(penalty='l2', C=clf_LR_model2.best_params_['C'], solver='newton-cg', multi_class='multinomial')
LR_all_data.fit(X_train, y_train)

y_mnist_predict = LR_all_data.predict(X_test)
con_matrix_mnist = confusion_matrix(y_test, y_mnist_predict)
```

```python
total_num = np.sum(con_matrix_mnist)

APC = 0
for i in range(10):
    TP = con_matrix_mnist[i][i]
    FP = 0
    FN = 0
    for j in range(10):
        if(j != i):
            FP = FP + con_matrix_mnist[j][i]
            FN = FN + con_matrix_mnist[i][j]
    TN = total_num - TP - FP - FN
    APC = APC + (TP + TN) / total_num
APC = APC / 10

print("The average per-class accuracy is: ", APC)
print("The confusion matrix is: ")
print(con_matrix_mnist)
```

    The average per-class accuracy is:  0.8827399999999999
    The confusion matrix is:
    [[489  43  56  38  21  32  22  47 177  75]
     [ 55 498  33  35  21  35  41  50  73 159]
     [ 98  44 286  85 118  86 152  67  43  21]
     [ 40  58 102 263  62 187 136  49  39  64]
     [ 55  30 134  71 297  81 165 113  27  27]
     [ 39  48  93 172  74 344  89  70  50  21]
     [ 14  38  78 125  90  77 497  37  14  30]
     [ 42  50  65  63  84  78  48 460  35  75]
     [157  70  23  23   8  51   8  19 537 104]
     [ 74 187  18  29  17  28  43  51  87 466]]

# Question 4: (10 pts)
In this problem, we will perform some very basic image recovery using linear regression. The main purpose of this example is to set up the Azure cloud and get acquainted with the environment. Download the dataset on the remote machine into the same folder as this notebook.

This part of the homework must be run on the azure cloud machine. We will doublecheck that everyone has used at least a portion of their allocated resources to ensure that everyone has successfully set up the VM.

### Using Linear Regression for Image Recovery
Given a set of uncorrupted images, $X$, and a single corrupted image $y$, we can find a linear combination of the images in the dataset which best reproduce the corrupted image. This sounds like a regression problem!

Obviously, if we perfectly reproduce the corrupted image, then we haven't done a good job recovering the original image. However, if we introduce a penalty on the number of images used in the solution, then we can reproduce a decent approximation of the original image.

After finding a linear combination of the images $\beta$, $X\beta$ will give us our recovered image (this is done for you in the `recover` function below). Once again, for the result to be an acceptable recovery of the image $X\beta$ should not exactly equal $y$ since $y$ is the corrupted image.

Each image is flattened into a vector of lengh 4096 (each image is 64x64) and the columns of $X$ correspond to the uncorrupted images in the dataset. You can think of the uncorrupted images in the dataset as the features.

The starter code we have provided below does most of the work for us, all that is left to do is to:
1. Modify the `ols` and `ridge` functions to solve OLS and ridge, respectively on the arguments provided to the functions ($X$ and $y$). (6pts)
2. Try different values of the regularization coefficient for the ridge problem. You should be able to get a decent reproduction of the original image. (2pts)
3. Explain why OLS does a bad job recovering the image, but ridge does an acceptable job with a good choice of alpha (2pts)

```python
from scipy.io import loadmat

from sklearn.linear_model import Lasso
from sklearn.linear_model import Ridge
from sklearn.linear_model import LinearRegression
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import imshow

import numpy as np
import numpy.linalg as la
import matplotlib.pyplot as plt

import PIL.Image
import glob
import re

def read_image_array(image_path):
    im = np.asarray(PIL.Image.open(image_path)).ravel()
    return im

def read_images(ignore=[]):
    frame_path = 'faces/'
    frames = []
    file_list = glob.glob(frame_path + "*.png")
    for file_name in set(file_list) - set(ignore):
        frames.append(read_image_array(file_name))
    frames_array = np.array(frames).T
    normalized = frames_array / 255.0 - np.mean(frames_array / 255.0, axis=0)
    return normalized

def ols(X, y):

    ols_model = LinearRegression()
    ols_model.fit(X, y)
    model = ols_model.coef_
    print(len(model))
    return model

def ridge(X, y):
  #UPDATE THIS FUNCTION
    model = Ridge()
    model.fit(X, y)
    return model.coef_

def ridge2(X, y, param_alpha):
  #UPDATE THIS FUNCTION
    model = Ridge(alpha = param_alpha)
    model.fit(X, y)
    return model.coef_

def get_reference_image_name(corrupted_image_name):
    parts = corrupted_image_name.split("_")
    return parts[1]

def recover(method, name):
    print("Recovering with", name)
    plt.figure()
    corrupted_image_name = "cor_585.png"
    print("recovering", corrupted_image_name)
    corrupted_image_path = corrupted_image_name
    original_image_dimensions = np.asarray(PIL.Image.open(corrupted_image_path)).shape
    reference_image_name = get_reference_image_name(corrupted_image_name)
    reference_image = read_image_array('faces/' + reference_image_name)
    reference_image_normalized = reference_image / 255.0 - np.mean(reference_image / 255.0)
    corrupted_image = read_image_array(corrupted_image_path)
    corrupted_image_normalized = corrupted_image / 255.0 - np.mean(corrupted_image / 255.0)
    image_database = read_images()
    model = method(image_database, corrupted_image_normalized)
    recovered_image = np.dot(image_database, model)
    recovered_image_restored = (np.dot(image_database, model) + np.mean(corrupted_image / 255.0)) * 255
    return imshow(PIL.Image.fromarray(np.int32(recovered_image_restored.reshape(original_image_dimensions))).convert("L"), cmap='gray')

def recover_tuning(method, name, alpha):
    print("Recovering with", name)
    plt.figure()
    corrupted_image_name = "cor_585.png"
    print("recovering", corrupted_image_name)
    corrupted_image_path = corrupted_image_name
    original_image_dimensions = np.asarray(PIL.Image.open(corrupted_image_path)).shape
    reference_image_name = get_reference_image_name(corrupted_image_name)
    reference_image = read_image_array('faces/' + reference_image_name)
    reference_image_normalized = reference_image / 255.0 - np.mean(reference_image / 255.0)
    corrupted_image = read_image_array(corrupted_image_path)
    corrupted_image_normalized = corrupted_image / 255.0 - np.mean(corrupted_image / 255.0)
    image_database = read_images()
    model = method(image_database, corrupted_image_normalized, alpha)
    recovered_image = np.dot(image_database, model)
    recovered_image_restored = (np.dot(image_database, model) + np.mean(corrupted_image / 255.0)) * 255
    return imshow(PIL.Image.fromarray(np.int32(recovered_image_restored.reshape(original_image_dimensions))).convert("L"), cmap='gray')
```

The code below will then save the restored images to the filesystem.

### 1

```python
%matplotlib inline
recover(ols, 'OLS')
```

```python
%matplotlib inline
recover(ridge, 'Ridge')
```

### 2

```python
for param in range(0, 500, 50):
    %matplotlib inline
    plt.figure()
    recover_tuning(ridge2, 'Ridge', param)
    plt.show()

```

### 3

From the answers above, it is easy to find that ridge performs better than ols when the regularization coefficients of ridge is set to a proper value. With no regularization, the model tries to reproduce the corrupted image as accurately as possible, and thus it is expected that we got an image almost the same as the corrupted one. We can regard it as over-fitting in regular regression. However, with regularization (ridge), coefficients won't be too large, which means it tries to avoid over-fitting. In this case, the model tries to reduce the error of gray-scale values between each pair of pixels as much as possible. Since the corrupted part isn't too big, the model may tend to sacrifice that part and thus we can get a better recovered image.

# Question 5: (5 pts)
What are two properties of Naïve Bayes that make it so suitable for parallel/distributed computation and for analyzing "streaming" data (e.g. each document is processed only once and need not be further stored or revisited)? (5 pts)

# Answer

+ Independent properties. Naive bayes network holds the assumption that the conditional probability could be split down and does not need wait until the all the previous term finalized the answer. Under this circumstances, the probability computation and be split, calculated and finally return the answer.
+ since the bayes network could calculate the probility by parts, sum together and don't need to re-compute each time, in that sense, we can deal with streaming data well.