# Data Augmentation in image classification models

**By Neuromatch Academy**

__Content creators:__ [Jama Hussein Mohamud](https://engmubarak48.github.io/jmohamud/index.html), [Alex Hernandez-Garcia](https://alexhernandezgarcia.github.io/)

__Production editors:__ Spiros Chavlis, Saeed Salehi

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

----
# Objective

Data augmentation refers to synthetically increasing the amount of training data by transforming the existing training examples. Data augmentation has been shown to be a very useful technique, especially in computer vision applications. However, there are multiple ways of performing data augmentation and it is yet to be understood which transformations are more effective and why, and how data augmentation interacts with other techniques. In fact, it is common to see different augmentation schemes and setups in different papers. For example, there are perceptually possible image transformations (related to human visual perception), simple synthetic transformations such as cutout, more artificial transformations such as mixup that even transform the class labels, among many others.

In this notebook, we will show how to train deep neural networks for image classification with data augmentation and analyse the results.

---
# Setup

```python
# @title Install dependencies
!pip install pandas --quiet
```

```python
# imports
import os
import csv
import multiprocessing

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.backends.cudnn as cudnn
from torch.autograd import Variable

import torchvision
import torchvision.transforms as transforms
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU)

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
set_seed(seed=2021)
set_device()
```

    Random seed 2021 has been set.
    GPU is enabled in this notebook.

    'cuda'

### Training hyperparameters

**Note:** We have reduced the number of epochs, `end_epochs`. The value was set to 200. Please, change it back and run the code.

```python
# hyper-parameters
use_cuda = torch.cuda.is_available()
alpha = 1  # alpha for mixup augmentation
best_acc = 0  # best test accuracy
start_epoch = 0  # start from epoch 0 or last checkpoint epoch
batch_size = 128
end_apochs = 15  # Please change this to 200
base_learning_rate = 0.1
cutout = True  # True/False if you want to use cutout augmentation
mixup = False  # True/False if you want to use mixup augmentation
n_holes = 1  # number of holes to cut out from image for cutout
length = 16  # length of the holes for cutout augmentation
torchvision_transforms = False  # True/False if you want use torchvision augmentations
```

---
#  Augmentation

## Cutout
Randomly mask out one or more patches from an image.

```python
# @markdown `Cutout` Augmentation class

class Cutout(object):
  """
  code from: https://github.com/uoguelph-mlrg/Cutout

  Randomly mask out one or more patches from an image.
  Args:
      n_holes (int): Number of patches to cut out of each image.
      length (int): The length (in pixels) of each square patch.
  """
  def __init__(self, n_holes, length):
    self.n_holes = n_holes
    self.length = length

  def __call__(self, img):
    """
    Args:
        img (Tensor): Tensor image of size (C, H, W).
    Returns:
        Tensor: Image with n_holes of dimension length x length cut out of it.
    """
    h = img.size(1)
    w = img.size(2)

    mask = np.ones((h, w), np.float32)

    for n in range(self.n_holes):
      y = np.random.randint(h)
      x = np.random.randint(w)

      y1 = np.clip(y - self.length // 2, 0, h)
      y2 = np.clip(y + self.length // 2, 0, h)
      x1 = np.clip(x - self.length // 2, 0, w)
      x2 = np.clip(x + self.length // 2, 0, w)

      mask[y1: y2, x1: x2] = 0.

    mask = torch.from_numpy(mask)
    mask = mask.expand_as(img)
    img = img * mask

    return img
```

## Mixup

Mixup is a data augmentation technique that combines pairs of examples via a convex combination of the images and the labels. Given images $x_i$ and $x_j$ with labels $y_i$ and $y_j$, respectively, and $\lambda \in [0, 1]$, mixup creates a new image $\hat{x}$ with label $\hat{y}$ the following way:

\begin{align}
\hat{x} &= \lambda x_i + (1 - \lambda) x_j \\
\hat{y} &= \lambda y_i + (1 - \lambda) y_j
\end{align}

You may check the [original paper](https://arxiv.org/abs/1710.09412) and [code repository](https://github.com/hongyi-zhang/mixup).

```python
# @markdown `mixup_data` Augmentation function

def mixup_data(x, y, alpha=1.0, use_cuda=True):
  '''Compute the mixup data. Return mixed inputs, pairs of targets, and lambda
      - https://github.com/hongyi-zhang/mixup
  '''
  if alpha > 0.:
    lam = np.random.beta(alpha, alpha)
  else:
    lam = 1.
  batch_size = x.size()[0]
  if use_cuda:
    index = torch.randperm(batch_size).cuda()
  else:
    index = torch.randperm(batch_size)

  mixed_x = lam * x + (1 - lam) * x[index, :]
  y_a, y_b = y, y[index]

  return mixed_x, y_a, y_b, lam
```

---
# Data

## Datasets

We will start using CIFAR-10 data set from PyTorch, but with small tweaks we can get any other data we are interested in.

```python
# @markdown Download and prepare Data
print('==> Preparing data...')
def percentageSplit(full_dataset, percent=0.0):
  set1_size = int(percent * len(full_dataset))
  set2_size = len(full_dataset) - set1_size
  final_dataset, _ = torch.utils.data.random_split(full_dataset,
                                                   [set1_size, set2_size])
  return final_dataset

# CIFAR100 normalizing
# mean = [0.5071, 0.4866, 0.4409]
# std = [0.2673, 0.2564, 0.2762]

# CIFAR10 normalizing
mean = (0.4914, 0.4822, 0.4465)
std = (0.2023, 0.1994, 0.2010)

# torchvision transforms
transform_train = transforms.Compose([])
if torchvision_transforms:
  transform_train.transforms.append(transforms.RandomCrop(32, padding=4))
  transform_train.transforms.append(transforms.RandomHorizontalFlip())

transform_train.transforms.append(transforms.ToTensor())
transform_train.transforms.append(transforms.Normalize(mean, std))
if cutout:
  transform_train.transforms.append(Cutout(n_holes=n_holes, length=length))

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean, std),
])

trainset = torchvision.datasets.CIFAR10(
    root='./CIFAR10', train=True, download=True,
    transform=transform_train)

testset = torchvision.datasets.CIFAR10(
    root='./CIFAR10', train=False, download=True,
    transform=transform_test)
```

    ==> Preparing data...
    Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./CIFAR10/cifar-10-python.tar.gz

    HBox(children=(FloatProgress(value=0.0, max=170498071.0), HTML(value='')))

    Extracting ./CIFAR10/cifar-10-python.tar.gz to ./CIFAR10
    Files already downloaded and verified

#### CIFAR-10

CIFAR-10 is a data set of 50,000 colour (RGB) training images and 10,000 test images, of size 32 x 32 pixels. Each image is labelled as 1 of 10 possible classes:
```
'plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'
```
The data set is stored as a custom `torchvision.datasets.cifar.CIFAR` object. You can check some of its properties with the following code:

```python
print(f"Object type: {type(trainset)}")
print(f"Training data shape: {trainset.data.shape}")
print(f"Test data shape: {testset.data.shape}")
print(f"Number of classes: {np.unique(trainset.targets).shape[0]}")
```

    Object type: <class 'torchvision.datasets.cifar.CIFAR10'>
    Training data shape: (50000, 32, 32, 3)
    Test data shape: (10000, 32, 32, 3)
    Number of classes: 10

```python
# choose percentage from the trainset. set percent = 1.0 to use the whole train data
percent = 1.0
trainset = percentageSplit(trainset, percent = percent)
print(f"size of the new trainset: {len(trainset)}")
```

    size of the new trainset: 50000

## Data loaders

A dataloader is an optimized data iterator that provides functionality for efficient shuffling, transformation and batching of the data.

```python
# Dataloader
num_workers = multiprocessing.cpu_count()

print(f'----> number of workers: {num_workers}')

trainloader = torch.utils.data.DataLoader(
    trainset, batch_size=batch_size, shuffle=True, num_workers=num_workers)
testloader = torch.utils.data.DataLoader(
    testset, batch_size=batch_size, shuffle=False, num_workers=num_workers)
```

    ----> number of workers: 4

## Visualization

To visualize some of the augmentations, make sure you set to ```True``` their corresponding flags in the hyperparameters section

```python
# get batch of data
batch_X, batch_Y =  next(iter(trainloader))
```

```python
def plot_mixed_images(images):
  inv_normalize = transforms.Normalize(
                      mean= [-m/s for m, s in zip(mean, std)],
                      std= [1/s for s in std]
                      )
  inv_PIL = transforms.ToPILImage()
  fig = plt.figure(figsize=(10, 8))
  for i in range(1, len(images) + 1):
    image = images[i-1]
    ax = fig.add_subplot(1, 4, i)
    inv_tensor = inv_normalize(image).cpu()
    ax.imshow(inv_PIL(inv_tensor))
  plt.show()
```

```python
# Mixup Visualization
if mixup:
  alpha = 0.9
  mixed_x, y_a, y_b, lam = mixup_data(batch_X, batch_Y,
                                      alpha=alpha, use_cuda=use_cuda)
  plot_mixed_images(mixed_x[:4])
```

```python
# Mixup Visualization
if cutout:
  plot_mixed_images(batch_X[:4])
```

```python
# Torchvision Visualization
if torchvision_transforms:
  plot_mixed_images(batch_X[:4])
```

---
# Model

## Architecture: ResNet

ResNet is a family of network architectures whose main property is that the network is organised as a stack of _residual blocks_. Residual blocks consist of a stack of layers whose output is added the input, making a _shortcut connection_.

See the [original paper](https://arxiv.org/abs/1512.03385) for more details.

ResNet is just a popular choice out of many others, but data augmentation works well in general. We just picked ResNet for illustration purposes.

```python
# @markdown ResNet model in PyTorch

class BasicBlock(nn.Module):
  """ResNet in PyTorch.
        Reference:
        [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
        Deep Residual Learning for Image Recognition.
        arXiv:1512.03385
  """
  expansion = 1

  def __init__(self, in_planes, planes, stride=1):
    super(BasicBlock, self).__init__()
    self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
    self.bn1 = nn.BatchNorm2d(planes)
    self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
    self.bn2 = nn.BatchNorm2d(planes)

    self.shortcut = nn.Sequential()
    if stride != 1 or in_planes != self.expansion*planes:
        self.shortcut = nn.Sequential(
            nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
            nn.BatchNorm2d(self.expansion*planes)
        )

  def forward(self, x):
    out = F.relu(self.bn1(self.conv1(x)))
    out = self.bn2(self.conv2(out))
    out += self.shortcut(x)
    out = F.relu(out)
    return out

class Bottleneck(nn.Module):
  expansion = 4

  def __init__(self, in_planes, planes, stride=1):
    super(Bottleneck, self).__init__()
    self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
    self.bn1 = nn.BatchNorm2d(planes)
    self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
    self.bn2 = nn.BatchNorm2d(planes)
    self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)
    self.bn3 = nn.BatchNorm2d(self.expansion*planes)

    self.shortcut = nn.Sequential()
    if stride != 1 or in_planes != self.expansion*planes:
      self.shortcut = nn.Sequential(
          nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
          nn.BatchNorm2d(self.expansion*planes)
      )

  def forward(self, x):
    out = F.relu(self.bn1(self.conv1(x)))
    out = F.relu(self.bn2(self.conv2(out)))
    out = self.bn3(self.conv3(out))
    out += self.shortcut(x)
    out = F.relu(out)
    return out

class ResNet(nn.Module):
  def __init__(self, block, num_blocks, num_classes=10):
    super(ResNet, self).__init__()
    self.in_planes = 64

    self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
    self.bn1 = nn.BatchNorm2d(64)
    self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
    self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
    self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
    self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
    self.linear = nn.Linear(512*block.expansion, num_classes)

  def _make_layer(self, block, planes, num_blocks, stride):
    strides = [stride] + [1]*(num_blocks-1)
    layers = []
    for stride in strides:
      layers.append(block(self.in_planes, planes, stride))
      self.in_planes = planes * block.expansion
    return nn.Sequential(*layers)

  def forward(self, x):
    out = F.relu(self.bn1(self.conv1(x)))
    out = self.layer1(out)
    out = self.layer2(out)
    out = self.layer3(out)
    out = self.layer4(out)
    out = F.avg_pool2d(out, 4)
    out = out.view(out.size(0), -1)
    out = self.linear(out)
    return out

def ResNet18():
  return ResNet(BasicBlock, [2, 2, 2, 2])

def ResNet34():
  return ResNet(BasicBlock, [3, 4, 6, 3])

def ResNet50():
  return ResNet(Bottleneck, [3, 4, 6, 3])
```

## Model setup and test

```python
# load the Model
net = ResNet18()
print('-----> verify if model is run on random data')
y = net(Variable(torch.randn(1,3,32,32)))
print('model loaded')

result_folder = './results/'
if not os.path.exists(result_folder):
    os.makedirs(result_folder)

logname = result_folder + net.__class__.__name__ + '_' + '.csv'

if use_cuda:
  net.cuda()
  net = torch.nn.DataParallel(net)
  print('Using', torch.cuda.device_count(), 'GPUs.')
  cudnn.benchmark = True
  print('Using CUDA..')
```

    -----> verify if model is run on random data
    model loaded
    Using 1 GPUs.
    Using CUDA..

---
# Training

## Loss function and Optimizer

We use the cross entropy loss, commonly used for classification, and stochastic gradient descent (SGD) as optimizer, with momentum and weight decay.

```python
# optimizer and criterion

def mixup_criterion(y_a, y_b, lam):
  '''
    - Mixup criterion
    - https://github.com/hongyi-zhang/mixup
  '''
  return lambda criterion, pred: lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)

criterion = nn.CrossEntropyLoss()  # only for test data
optimizer = optim.SGD(net.parameters(), lr=base_learning_rate, momentum=0.9, weight_decay=1e-4)
```

## Train and test loops

```python
# Training & Test functions
def train(epoch, alpha, use_cuda=False):
  print('\nEpoch: %d' % epoch)
  net.train()
  train_loss = 0
  correct = 0
  total = 0
  for batch_idx, (inputs, targets) in enumerate(trainloader):
    if use_cuda:
      inputs, targets = inputs.cuda(), targets.cuda()
    optimizer.zero_grad()
    if mixup:
      # generate mixed inputs, two one-hot label vectors and mixing coefficient
      inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, alpha, use_cuda)
      inputs, targets_a, targets_b = Variable(inputs), Variable(targets_a), Variable(targets_b)
      outputs = net(inputs)
      loss_func = mixup_criterion(targets_a, targets_b, lam)
      loss = loss_func(criterion, outputs)
    else:
      inputs, targets = Variable(inputs), Variable(targets)
      outputs = net(inputs)
      loss = criterion(outputs, targets)

    loss.backward()
    optimizer.step()

    train_loss += loss.item()
    _, predicted = torch.max(outputs.data, 1)
    total += targets.size(0)
    if mixup:
      correct += lam * predicted.eq(targets_a.data).cpu().sum() + (1 - lam) * predicted.eq(targets_b.data).cpu().sum()
    else:
      correct += predicted.eq(targets.data).cpu().sum()

    if batch_idx % 500 == 0:
      print(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
          % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))
  return (train_loss/batch_idx, 100.*correct/total)

def test(epoch, use_cuda=False):
  global best_acc
  net.eval()
  test_loss = 0
  correct = 0
  total = 0
  with torch.no_grad():
    for batch_idx, (inputs, targets) in enumerate(testloader):
      if use_cuda:
        inputs, targets = inputs.cuda(), targets.cuda()
      # inputs, targets = Variable(inputs, volatile=True), Variable(targets)
      outputs = net(inputs)
      loss = criterion(outputs, targets)

      test_loss += loss.item()
      _, predicted = torch.max(outputs.data, 1)
      total += targets.size(0)
      correct += predicted.eq(targets.data).cpu().sum()

      if batch_idx % 200 == 0:
        print(batch_idx, len(testloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
            % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))

  # Save checkpoint.
  acc = 100.*correct/total
  if acc > best_acc:
    best_acc = acc
    checkpoint(acc, epoch)
  return (test_loss/batch_idx, 100.*correct/total)
```

### Auxiliary functions

* `checkpoint()`: Store checkpoints of the model
* `adjust_learning_rate()`: Decreases the learning rate (learning rate decay) at certain epochs of training.

```python
# @markdown `checkpoint` and `adjust_learning_rate` functions
def checkpoint(acc, epoch):
  # Save checkpoint.
  print('Saving..')
  state = {
      'net': net.state_dict(),
      'acc': acc,
      'epoch': epoch,
      'rng_state': torch.get_rng_state()
      }
  if not os.path.isdir('checkpoint'):
    os.mkdir('checkpoint')
  torch.save(state, './checkpoint/ckpt.t7')

def adjust_learning_rate(optimizer, epoch):
  """decrease the learning rate at 100 and 150 epoch"""
  lr = base_learning_rate
  if epoch <= 9 and lr > 0.1:
    # warm-up training for large minibatch
    lr = 0.1 + (base_learning_rate - 0.1) * epoch / 10.
  if epoch >= 100:
    lr /= 10
  if epoch >= 150:
    lr /= 10
  for param_group in optimizer.param_groups:
    param_group['lr'] = lr
```

```python
# start training
if not os.path.exists(logname):
  with open(logname, 'w') as logfile:
    logwriter = csv.writer(logfile, delimiter=',')
    logwriter.writerow(['epoch', 'train loss', 'train acc',
                        'test loss', 'test acc'])

for epoch in range(start_epoch, end_apochs):
  adjust_learning_rate(optimizer, epoch)
  train_loss, train_acc = train(epoch, alpha, use_cuda=use_cuda)
  test_loss, test_acc = test(epoch, use_cuda=use_cuda)
  with open(logname, 'a') as logfile:
    logwriter = csv.writer(logfile, delimiter=',')
    logwriter.writerow([epoch, train_loss, train_acc.item(),
                        test_loss, test_acc.item()])
  print(f'Epoch: {epoch} | train acc: {train_acc} | test acc: {test_acc}')
```

    Epoch: 0
    0 391 Loss: 2.443 | Acc: 10.938% (14/128)
    0 79 Loss: 1.531 | Acc: 46.094% (59/128)
    Saving..
    Epoch: 0 | train acc: 31.604000091552734 | test acc: 44.2599983215332

    Epoch: 1
    0 391 Loss: 1.619 | Acc: 39.844% (51/128)
    0 79 Loss: 1.199 | Acc: 60.156% (77/128)
    Saving..
    Epoch: 1 | train acc: 47.03200149536133 | test acc: 54.41999816894531

    Epoch: 2
    0 391 Loss: 1.301 | Acc: 53.906% (69/128)
    0 79 Loss: 1.013 | Acc: 61.719% (79/128)
    Saving..
    Epoch: 2 | train acc: 56.257999420166016 | test acc: 62.599998474121094

    Epoch: 3
    0 391 Loss: 1.036 | Acc: 64.062% (82/128)
    0 79 Loss: 0.909 | Acc: 69.531% (89/128)
    Saving..
    Epoch: 3 | train acc: 62.43199920654297 | test acc: 65.6500015258789

    Epoch: 4
    0 391 Loss: 0.839 | Acc: 68.750% (88/128)
    0 79 Loss: 0.859 | Acc: 70.312% (90/128)
    Saving..
    Epoch: 4 | train acc: 67.0 | test acc: 69.08999633789062

    Epoch: 5
    0 391 Loss: 0.922 | Acc: 64.844% (83/128)
    0 79 Loss: 0.660 | Acc: 76.562% (98/128)
    Saving..
    Epoch: 5 | train acc: 70.1259994506836 | test acc: 72.52999877929688

    Epoch: 6
    0 391 Loss: 0.833 | Acc: 65.625% (84/128)
    0 79 Loss: 0.616 | Acc: 78.125% (100/128)
    Saving..
    Epoch: 6 | train acc: 73.45999908447266 | test acc: 73.45999908447266

    Epoch: 7
    0 391 Loss: 0.686 | Acc: 75.000% (96/128)
    0 79 Loss: 0.533 | Acc: 81.250% (104/128)
    Saving..
    Epoch: 7 | train acc: 75.99600219726562 | test acc: 75.91000366210938

    Epoch: 8
    0 391 Loss: 0.626 | Acc: 78.125% (100/128)
    0 79 Loss: 0.458 | Acc: 82.031% (105/128)
    Saving..
    Epoch: 8 | train acc: 78.42400360107422 | test acc: 79.11000061035156

    Epoch: 9
    0 391 Loss: 0.465 | Acc: 85.938% (110/128)
    0 79 Loss: 0.465 | Acc: 87.500% (112/128)
    Saving..
    Epoch: 9 | train acc: 80.72599792480469 | test acc: 80.37000274658203

    Epoch: 10
    0 391 Loss: 0.509 | Acc: 81.250% (104/128)
    0 79 Loss: 0.523 | Acc: 79.688% (102/128)
    Epoch: 10 | train acc: 82.16400146484375 | test acc: 79.25

    Epoch: 11
    0 391 Loss: 0.423 | Acc: 82.031% (105/128)
    0 79 Loss: 0.610 | Acc: 78.125% (100/128)
    Epoch: 11 | train acc: 83.96199798583984 | test acc: 79.68000030517578

    Epoch: 12
    0 391 Loss: 0.221 | Acc: 89.844% (115/128)
    0 79 Loss: 0.467 | Acc: 82.812% (106/128)
    Saving..
    Epoch: 12 | train acc: 85.61799621582031 | test acc: 80.88999938964844

    Epoch: 13
    0 391 Loss: 0.427 | Acc: 85.938% (110/128)
    0 79 Loss: 0.522 | Acc: 82.812% (106/128)
    Saving..
    Epoch: 13 | train acc: 87.21199798583984 | test acc: 81.54000091552734

    Epoch: 14
    0 391 Loss: 0.216 | Acc: 93.750% (120/128)
    0 79 Loss: 0.386 | Acc: 86.719% (111/128)
    Epoch: 14 | train acc: 88.08000183105469 | test acc: 81.44999694824219

```python
# plot results
results = pd.read_csv('/content/results/ResNet_.csv', sep=',')
results.head()
```

      epoch
      train loss
      train acc
      test loss
      test acc

      0
      0
      1.932130
      31.604000
      1.535233
      44.259998

      1
      1
      1.446863
      47.032001
      1.262779
      54.419998

      2
      2
      1.212518
      56.257999
      1.069593
      62.599998

      3
      3
      1.051850
      62.431999
      0.996476
      65.650002

      4
      4
      0.928131
      67.000000
      0.898354
      69.089996

```python
train_accuracy = results['train acc'].values
test_accuracy = results['test acc'].values

print(f"Average test Accuracy over {end_apochs} epochs: {sum(test_accuracy)//len(test_accuracy)}")
print(f"best test accuraccy over {end_apochs} epochs: {max(test_accuracy)}")
```

    Average test Accuracy over 15 epochs: 72.0
    best test accuraccy over 15 epochs: 81.54000091552734

```python
figureName = 'WithMixUp' # change figure name

plt.figure(figsize=(9, 6))
plt.plot(results['epoch'].values, train_accuracy, label='train')
plt.plot(results['epoch'].values, test_accuracy, label='test')
plt.xlabel('Number of epochs')
plt.ylabel('Accuracy')
plt.title(f'Train/Test Accuracy curve for {end_apochs} epochs')
plt.savefig(f'/content/results/{figureName}.png')
plt.legend()
plt.show()
```