# NBA Team Ratings, Pt. 1

For each game, we observe the number of points, possessions, home court, and the teams involved.
Then, for each side of the court, we model the points scored (normalized by possessions) as a normal distribution with mean determined by the teams along with home court.

\begin{align}
x_i &\thicksim \mathcal{N}\left(\frac{\text{poss}_i}{100} \left( \mu + \text{off}_i - \text{def}_i + \delta_\text{home} \text{home} \right), \frac{\text{poss}_i}{100}\sigma_{\text{game}}^2 \right) \notag \\
\text{where } x_i &\equiv \text{points scored in the }i^\text{th} \text{ game} \notag \\
\text{poss}_i &\equiv \text{possessions in the }i^\text{th} \text{ game} \notag \\
\delta_\text{home} &\equiv \begin{cases} 1 & \text{if home team is on offense} \\ -1 & \text{otherwise} \end{cases} \notag \\
\mu &\equiv \text{league average scoring rate per 100 possessions} \notag
\end{align}

This model has four parameters of interest:
1. $ \overrightarrow{\text{off}} $, the points scored per 100 possessions above league average for each team
2. $ \overrightarrow{\text{def}} $, the points allowed per 100 possessions above league average for each team
3. $ \text{home} $, the points scored per 100 possessions above league average by the home team
4. $ \sigma_{\text{game}} $, the variance for the number of points scored per 100 possessions

These paremeters each have a prior distribution (since we're being Bayesian about it):

\begin{align}
    \overrightarrow{\text{off}} &\thicksim \mathcal{N}(0, \sigma_{\text{off}} ^ 2) \notag \\
    \overrightarrow{\text{def}} &\thicksim \mathcal{N}(0, \sigma_{\text{def}} ^ 2) \notag \\
    \text{home} &\thicksim \text{Gamma}(1.5, 0.5) \notag \\
    \sigma_{\text{game}} &\thicksim \text{Gamma}(5, 2) \notag
\end{align}

Note that each prior has its own set of parameters, aka hyperparameters, and specifically, that the priors for $ \overrightarrow{\text{off}} $ and $ \overrightarrow{\text{def}} $ have parameters with their own priors, aka hyperpriors.
The $ \text{home} $ and $ \sigma_{\text{game}} $ parameters use a Gamma prior.
This distribution is conveniently continuous and positive, making it a common one for the prior of a positive parameter like the standard deviation.
For that reason, I've also used it for the two hyperpriors:

\begin{align}
    \sigma_{\text{off}} &\thicksim \text{Gamma}(2, 0.5) \notag \\
    \sigma_{\text{def}} &\thicksim \text{Gamma}(1.5, 0.4) \notag \\
\end{align}

I haven't been too careful with setting these priors since this model is mostly for demonstration, but I've run it on the 2019 NBA Regular Season:

```python
import pymc3 as pm
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np

from pynba import halfgames_from_file, team_id_to_abb, plot_ratings, use_blackontrans_style

league = "nba"
year = 2019
season_type = "Regular Season"
```

```python
halfgames = halfgames_from_file(league, year, season_type)
```

```python
n_teams = halfgames["off_team_id"].unique().shape[0]
team_id_to_team_ind = {
    team_id: team_ind
    for team_ind, team_id in enumerate(halfgames["off_team_id"].unique())
}
team_id_to_team_abb = team_id_to_abb(league, year)
team_abb_to_team_id = {
    team_abb: team_id
    for team_id, team_abb in team_id_to_team_abb.items()
}
team_ind_to_team_abb = {
    team_ind: team_id_to_team_abb[team_id]
    for team_id, team_ind in team_id_to_team_ind.items()
}
points_mu = halfgames["points_scored"].sum() / halfgames["possession_num"].sum() * 100
off_index = (
    halfgames["off_team_id"].map(team_id_to_team_ind).to_numpy()
)
def_index = (
    halfgames["def_team_id"].map(team_id_to_team_ind).to_numpy()
)
home_index = (
    (halfgames["off_team_id"] != halfgames["home_team_id"])
    .to_numpy()
    .astype(int)
) * 2 - 1

with pm.Model() as model:
    sigma_game = pm.Gamma("sigma_game", mu=5, sigma=2)
    sigma_off = pm.Gamma("sigma_off", mu=2, sigma=0.5)
    sigma_def = pm.Gamma("sigma_def", mu=1.5, sigma=0.4)
    home = pm.Gamma("home", mu=1.5, sigma=0.5)
    teams_off = pm.Normal("teams_off", mu=0, sigma=sigma_off, shape=n_teams)
    teams_def = pm.Normal("teams_def", mu=0, sigma=sigma_def, shape=n_teams)
    poss_norm = halfgames["possession_num"].to_numpy() / 100
    pm.Normal(
        "points",
        mu=poss_norm * (points_mu + teams_off[off_index] - teams_def[def_index] + home * home_index),
        sigma=poss_norm * sigma_game,
        observed=halfgames["points_scored"].to_numpy()
    )
```

```python
with model:
    trace = pm.sample(
        draws=10000,
        init="adapt_diag",
        chains=4,
        random_seed=42,
        return_inferencedata=False,
    )
```

    2021-05-17 21:47:06 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 495 | INFO | Auto-assigning NUTS sampler...
    2021-05-17 21:47:06 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:init_nuts @ 2081 | INFO | Initializing NUTS using adapt_diag...
    2021-05-17 21:47:14 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 556 | INFO | Multiprocess sampling (4 chains in 4 jobs)
    2021-05-17 21:47:14 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:_print_step_hierarchy @ 232 | INFO | NUTS: [teams_def, teams_off, home, sigma_def, sigma_off, sigma_game]

  <progress value='44000' class='' max='44000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [44000/44000 00:45<00:00 Sampling 4 chains, 0 divergences]

    2021-05-17 21:48:00 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 628 | INFO | Sampling 4 chains for 1_000 tune and 10_000 draw iterations (4_000 + 40_000 draws total) took 46 seconds.

```python
results = pd.DataFrame({
    "team": [team_ind_to_team_abb[ind] for ind in range(n_teams)],
    "net_scoring_above_average": trace["teams_off"].mean(0) + trace["teams_def"].mean(0),
    "off_scoring_above_average": trace["teams_off"].mean(0),
    "def_scoring_above_average": trace["teams_def"].mean(0),
    "league": league,
    "year": year,
    "season_type": season_type,
})

results.sort_values("net_scoring_above_average", ascending=False)
```

      team
      net_scoring_above_average
      off_scoring_above_average
      def_scoring_above_average
      league
      year
      season_type

      4
      MIL
      6.210715
      2.802680
      3.408035
      nba
      2019
      Regular Season

      2
      GSW
      5.181949
      4.292881
      0.889067
      nba
      2019
      Regular Season

      15
      TOR
      4.262034
      2.190613
      2.071421
      nba
      2019
      Regular Season

      21
      UTA
      3.961912
      0.298230
      3.663683
      nba
      2019
      Regular Season

      17
      HOU
      3.784661
      4.048864
      -0.264203
      nba
      2019
      Regular Season

      29
      POR
      3.508986
      3.373137
      0.135849
      nba
      2019
      Regular Season

      22
      DEN
      3.437770
      2.023928
      1.413842
      nba
      2019
      Regular Season

      0
      BOS
      2.932447
      1.154435
      1.778012
      nba
      2019
      Regular Season

      3
      OKC
      2.686181
      -0.140427
      2.826608
      nba
      2019
      Regular Season

      8
      IND
      2.030188
      -0.486166
      2.516354
      nba
      2019
      Regular Season

      1
      PHI
      1.689404
      1.467895
      0.221509
      nba
      2019
      Regular Season

      19
      SAS
      1.419684
      1.933909
      -0.514225
      nba
      2019
      Regular Season

      23
      LAC
      0.864585
      1.477342
      -0.612757
      nba
      2019
      Regular Season

      11
      ORL
      0.280951
      -1.289209
      1.570159
      nba
      2019
      Regular Season

      6
      BKN
      -0.223051
      -0.804554
      0.581502
      nba
      2019
      Regular Season

      10
      MIA
      -0.498958
      -2.427690
      1.928733
      nba
      2019
      Regular Season

      7
      DET
      -0.698599
      -1.298317
      0.599718
      nba
      2019
      Regular Season

      20
      SAC
      -0.713353
      0.041432
      -0.754785
      nba
      2019
      Regular Season

      5
      CHA
      -0.867620
      1.065196
      -1.932815
      nba
      2019
      Regular Season

      24
      DAL
      -0.871049
      -0.698385
      -0.172665
      nba
      2019
      Regular Season

      16
      NOP
      -0.956585
      0.852863
      -1.809448
      nba
      2019
      Regular Season

      28
      LAL
      -0.965957
      -1.885082
      0.919124
      nba
      2019
      Regular Season

      18
      MIN
      -1.034573
      0.836243
      -1.870816
      nba
      2019
      Regular Season

      9
      MEM
      -1.803026
      -3.420818
      1.617792
      nba
      2019
      Regular Season

      27
      WAS
      -2.018878
      0.512290
      -2.531167
      nba
      2019
      Regular Season

      12
      ATL
      -4.065261
      -1.511523
      -2.553738
      nba
      2019
      Regular Season

      25
      PHX
      -6.408621
      -3.452818
      -2.955804
      nba
      2019
      Regular Season

      26
      CHI
      -6.488680
      -4.154362
      -2.334318
      nba
      2019
      Regular Season

      13
      NYK
      -6.939783
      -4.419421
      -2.520363
      nba
      2019
      Regular Season

      14
      CLE
      -7.786211
      -2.140490
      -5.645721
      nba
      2019
      Regular Season

In addition to plotting the offensive and defensive ratings of each team, I've annotated the difference between each team's "raw" ratings and their modeled ones.
We can see that although the model is factoring in strength of schedule and home-court advantage, it's primary effect is "shrinking" the raw ratings.
This is due to the priors for each team rating, which make extreme values less likely.

```python
use_blackontrans_style()

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(1, 1, 1)
for team, off_aa, def_aa in zip(results["team"], results["off_scoring_above_average"], results["def_scoring_above_average"]):
    team_id = team_abb_to_team_id[team]
    filt = halfgames["off_team_id"] == team_id
    raw_off = halfgames.loc[filt, "points_scored"].sum() / halfgames.loc[filt, "possession_num"].sum() * 100 - points_mu
    filt = halfgames["def_team_id"] == team_id
    raw_def = points_mu - halfgames.loc[filt, "points_scored"].sum() / halfgames.loc[filt, "possession_num"].sum() * 100
    ax.annotate("", xy=(off_aa, def_aa), xytext=(raw_off, raw_def), arrowprops={"arrowstyle": "->", "color": "white"})
plot_ratings(results, ax)

```


# Anomaly Detection on the MNIST Dataset
### By Shashank Swaminathan and Chris Lee
## Introduction

Our project was centered around anomaly detection, a method of detecting outliers in a given dataset. It is especially useful in situations where early detection of items differing from the norm is important. For example, in food production facilities, broken conveyor belts could lead to contamination. However, detecting problems early is difficult, as there are not many signs of issues prior to the belt completely breaks. With anomaly detection, the defects could be found earlier on, reducing the risk of contamination.

Anomaly detection can be used in a multitude of scenarios, from finding anomalies in astronomical data to recognition of oddities in medical images. We chose to focus on the image recognition aspect of anomaly detection - specifically on the MNIST dataset. This was for multiple reasons - ease of access to data, the amount of data available, and the relative uniformity of the training data (most arabic numerals look similar to each other, even with moderately bad handwriting). Our goal was to be able to identify if a provided image of a number was anomalous - far off from the standard form of a number.

We approached this problem in two ways, using PCA and Autoencoders. In fact, both methods come from the same angle, and so first we'll address how the overall anomaly detection algorithm works.

## How do we detect anomalies?
A good way to detect anomalies is to simply compare the input image against a "average image" for the given type, i.e. when checking if a given number is a "7", we compare the input image against an average image for a seven. This method is quite useful, but has a significant drawback - we need to know what the input image is trying to represent ahead of time. If the image is already misidentified, then we might improperly identify anomalies when there are none. Even worse, we might miss anomalies. Hence, we look towards the second method, the one that we adopted.

In this method, we will train a model of some kind on a given set of good training images. The model will identify major features within all the images. We'll then use this model to try and recreate a given input image, using those identified major features. The differences between the input image and the recreated image will then represent parts of the image that aren't common to the given dataset - the anomalies in the image. We'll use these anomalies (the loss between the input image and the recreated image) to determine if the image is an anomaly.

Finally, we'll use the assumption that given a set of good training data, the losses between input and recreated images will form a standard distribution. We can then put a threshold at the higher end of this distribution - if the loss between a given input and its recreation is higher than this bar, we'll consider it an anomaly.

There are some points to make about this method:
 - First, the type of training data. Since we are training a model to identify key features within the training data, the classes of data we are training on should all be related, so that the key features across images are also similar. Training across a bunch of single digit numbers would satisfy this - since they are all single digits, and of the same style. However, applying this method on inconsistent datasets, like one of cars and cats would definitely not be appropriate.
 - The second point is that we are making a pretty big assumption when stating that the losses we'll find will become something like a standard distribution curve. While this was not disproven by the training that we did, it is important to remember that this might not always hold, and that it is a point of caution.

## Implementation of Anomaly Detection Algorithm
Now, we move onto discussing how we implemented the algorithm. Just to reiterate, here are the steps we need to implement for our anomaly detection to succeed:
 - Train a model to identify the key features in a set of training images.
 - Use this model to reconstruct a set of training images, and find the losses between input images and reconstructed images.
 - Find a suitable upper limit on the loss values based on the distribution of losses from training.
 - For a given input image, find the loss between input and the recreation. If above the threshold, it is an anomaly.

The first few steps - training a model to identify key features and reconstruct images - fall within the category of **image compression**. In image compression, we identify key features of the input data, and then try to compress the data into a lower dimension by only using these key features. This is essentially the same process as recreating an image from a set of key features. Only here, after determining the key features and stripping it down to a construct of those key features (reducing the image's dimensionality to only these key features), we recreate the image based on the key features (transform the compressed image back to the original dimension).

There are a few methods to do image compression - two common methods are using **Principal Component Analysis (PCA)** and using **Autoencoders**. In PCA, we use the covariance matrix of the data to determine a set of "basis vectors" that represent the input data's key features. Autoencoders are completely different - to be concise, they are unsupervised neural nets that iteratively learn how to compress the given data in the best way possible (reducing the loss of information before and after compression).

PCA is a linear process - it takes the input data, finds the covariance matrix, and then finds a set of key feature vectors such that a linear combination of those vectors can approximate the input data. It does a linear mapping between the data and its key features. This is good as an approximation of the data - it is does the job to a good degree, and is quite fast. However, it is quite poor at identifying non-linear relationships. The more complex the input data, the worse it is at identifying feature relationships. In a situation such as anomaly detection, where misidentified cases can have serious consequences, this may not be useful. It's also important to note here that the PCA method is not one that involves neural networks, and is essentially a linear model.

Autoencoders are a very nonlinear process. They follow the standard mechanisms of an ANN - when used in image processing, they include convolutional layers, pooling layers, ReLU layers, the whole deal. Because they also use gradient descent to optimize its encoding process (the compression process), it can compress data quite well. However, because it is a neural net, it also falls to the trappings of neural nets. One particularly problematic point that shows up while using it in anomaly detection is overfitting. Autoencoders tend to overfit to its input data - which is understandable, given that it is incentivized to reduce losses in compression for its training data. However, overfitting means that the encoder is not only extracting key features, but also common anomalous patterns in the input data, which is troublesome.

Because both have drawbacks, we decided to use both approaches to anomaly detection. First we implemented the PCA approach, and then the Autoencoder approach. Since it is hard to get a good set of 'anomalous data', we don't have an actual accuracy score of how good the detector is. Instead, we were able to provide example cases of the detector in action, for both PCA and Autoencoding.

We didn't put docstrings in our functions since we're already documenting the work using the text option of the notebook. Also, the reason we spend some time discussing the PCA approach in detail, though it is does not involve a neural network, is to provide some basis intuition on what we are doing for anomaly detection. The process we use for PCA will be extended for the Autoencoder process.

### Using PCA

First up, we download our data. Because `sklearn` has a lot of cool PCA-related functions, we used `sklearn` for the PCA method.

```
# from sklearn.datasets import fetch_mldata
# mnist = fetch_mldata('MNIST original')
import matplotlib.pyplot as plt
import numpy as np # we always love numpy
import time
import seaborn as sns

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.metrics import mean_squared_error
from sklearn.datasets import fetch_openml

# Load data from https://www.openml.org/d/554
mnist = fetch_openml('mnist_784', version=1, return_X_y=False)

from sklearn.model_selection import train_test_split
# test_size: what proportion of original data is used for test set
train_img, test_img, train_lbl, test_lbl = train_test_split(mnist.data,
                                                            mnist.target,
                                                            test_size=1/7.0,
                                                            random_state=0)
```

Next, we define the helper functions used for the PCA algorithm:
 - First up is the ever-useful image display function, useful for validating if the image data we have makes sense.
 - Second is the preprocessing function. Because PCA is a linearly based process, larger base values will scale the losses up, and might skew the results. Hence, we center our data around the mean of the data, and normalize all the values.
 - The third function is the actual PCA function. Here, we make use of `sklearn`'s PCA function to create a PCA model for us.
 - The fourth function will take a set of input training images, use the PCA model to reconstruct approximations of those images, and then find the losses between each image in the input dataset.
 - This is a useful way of inspecting the losses between input images and reconstructed images. We especially use this function to check if the losses appear to fall in a standard distribution curve.

```
def disp_image_pca(image):
    # need to reorder the tensor dimensions to work properly with imshow
    plt.imshow(image, 'gray')
    plt.axis('off')
    plt.show()

def preprocess_img_pca(train_img, test_img):
  scaler = StandardScaler()# Fit on training set only.
  scaler.fit(train_img)# Apply transform to both the training set and the test set.
  train_img = scaler.transform(train_img)
  test_img = scaler.transform(test_img)
  return train_img, test_img

def my_pca(train_img, amt):
  mpca = PCA(amt)
  mpca.fit(train_img)
  return mpca

def get_losses_pca(imgs, pca):
  post_encoding = mpca.inverse_transform(mpca.transform(imgs))
  losses = []
  for i in np.arange(imgs.shape[0]):
    losses.append(mean_squared_error(imgs[i,:], post_encoding[i,:]))
  return losses

def plot_losses_pca(losses, set_name):
  sns.distplot(losses, kde=False)
  plt.title("Loss Distribution: " + set_name)
  plt.xlabel("Loss Value")
  plt.ylabel("Quanitity")
  plt.show()
```

Just to inspect the loaded data, we'll use the display image function to see what we're looking at.

```
disp_image_pca(train_img[2,:].reshape(28,28))
```

Now we begin actually training our PCA model. We preprocess our data, creating mean-centered and normalized versions of the training and test datasets. We then train a PCA model on the training images. The way the PCA model works is that it preserves a certain percent of variation within the dataset - here, we've selected for 90% of variance within the dataset. The resultant number of dimensions that it retains after compressions is shown as the output of `mpca.n_components`.

```
n_train_img, n_test_img = preprocess_img_pca(train_img, test_img)
mpca = my_pca(n_train_img, 0.90)
mpca.n_components_
```

With the PCA model trained, it's time to evaluate the losses across the data! We evaluate the losses across both the training and test datasets. This way, we can do three things at once:
 - We can see the losses across the training and test sets, between the input and reconstructed images.
 - We can see if the losses across the datasets fall into a standard distribution curve (thereby validating our initial assumption).
 - We can finally see if the curves for the training and test set are similar - if they are, that means that the PCA model is not overfit for the training set, as compared to the test set.

It is worth pointing out that of the losses graphed, there is a certain amount cut from the graph - losses that were greater than 0.5. This was because there were a small amount of loss values that were significant outliers, enough to warp the graph display. In order to preserve the view of the graph, we select only for the losses within a reasonable range (and ignore the outliers). This is a reasonable choice, since there weren't many outliers to begin with - for example, in the training dataset, there were only around 2000 outliers, as compared to 60000 images.

```
# Training set
all_losses_train = np.array(get_losses_pca(n_train_img, mpca))
plot_losses_pca(all_losses_train[np.where(all_losses_train < 0.5)], "Training Set")

# Testing set
all_losses_test = np.array(get_losses_pca(n_test_img, mpca))
plot_losses_pca(all_losses_test[np.where(all_losses_test < 0.5)], "Test Set")
```

From the above, we can see that all three of the points we were looking for have been found. The losses have been evaluated; the curves do appear to be like standard distribution curves; and the curves between the training and test sets appear to be the same. This means the model has performed as we expected.

From the plot of losses, we can infer a reasonable threshold to consider an input an anomaly. We based our threshold off of z-score analysis, and placed a cutoff at 0.15. With that defined, we can create an 'anomaly identifier' function, that would guess if the image is an anomaly.

```
def is_anomaly_pca(img, pca = mpca, threshold = 0.15):
  loss = get_losses_pca(img, mpca)
  if (loss[0] > threshold):
    print("This is an anomaly with loss: %s" % loss[0])
  return loss[0]
```

Though the function is done, we still lack true anomalous data. To help create some, we defined a couple of helper functions that would essentially mash images together and force create an anomalous image. In addition to that, we added a helper function that would plot both the image and the reconstruction, to show the effects of the PCA based analysis.

```
# Combines images from dataset to create anomalies.
# Assumes input images of shape (1, 784)
def make_anomaly_pca(img1, img2):
    return np.max((img1, img2), axis=0)

#Plots an input and output of the model
def show_in_out(img, pca_model):
    # Plots anomaly
    plt.title("Input")
    disp_image_pca(img.reshape(28,28))

    # Passes image into Model
    img = img.reshape(1,img.shape[0])
    encoded = pca_model.inverse_transform(pca_model.transform(img))

    # Plots output
    plt.title("Output")
    disp_image_pca(encoded.reshape(28,28))

    loss = is_anomaly_pca(img)
    if loss < 0.15:
      print("Anomaly not detected.")
    print(F"LOSS: {loss}")
```

With those defined, we'll now present two examples of anomaly detection using PCA - one where an anomaly was not detected, and one where an anomaly was detected.

#### No Anomaly Detected
Here, the image is clearly very poor - it would almost certainly be considered an actual anomaly. However, the detector did not register this as an anomaly. This could be attributed to multiple things, but we think the most likely explanation is because PCA cannot handle non-linear relationships well, and so was unable to deal with such a messy image well.

```
anomaly_img = make_anomaly_pca(n_test_img[1,:], n_test_img[0,:])
show_in_out(anomaly_img, mpca)
```

#### Anomaly Detected
Here, an anomaly is detected. However, in order to do so, we had to mash multiple images together, and thereby accumulate a bunch of error before getting the final 'anomalous' image. This clearly shows the limitation of PCA - while it can definitely do reconstruction to a certain degree, it fails at identifying anything than the most anomalous images.

```
anomaly_img_1 = make_anomaly_pca(n_test_img[1,:], n_test_img[0,:])
anomaly_img_2 = make_anomaly_pca(n_test_img[2,:], anomaly_img_1)
anomaly_img_3 = make_anomaly_pca(n_test_img[3,:], anomaly_img_2)
anomaly_img_4 = make_anomaly_pca(n_test_img[4,:], anomaly_img_3)
anomaly_img_5 = make_anomaly_pca(n_test_img[5,:], anomaly_img_4)
show_in_out(anomaly_img_5, mpca)
```

With that, we have covered the PCA approach to anomaly detection. Note that this did not involve any neural networks! Instead, we focused on using a linear relationship between image data and variance. However, we also realized that there were quite a few limitations with this method. This in turn motivates the next section - using an Autoencoder.

### Using an Autoencoder
To reiterate what an Autoencoder based approach is, it is a nonlinear process involving reinforcement learning that trains an unsupervised neural network model to encode and decode data to lower dimensions while minizing information loss. We will use this encoding process the same way we did with PCA - to isolate key features of the dataset, and then use the difference between the input and reconstructed image to determine the presence of an anomaly.

In this approach, we will be using PyTorch to do our neural network implementation. Because the model is quite massive, we mount a Google Drive account to the system, in order to save the model parameters for future sessions.

The first step, as always, is to import the necessary libraries and the actual data we will be using.

```
# imports all necessary libraries

!pip install torchviz

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchviz import make_dot
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST

import time
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

from google.colab import drive
drive.mount('/content/gdrive')
```

    Requirement already satisfied: torchviz in /usr/local/lib/python3.6/dist-packages (0.0.1)
    Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (from torchviz) (0.10.1)
    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchviz) (1.2.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch->torchviz) (1.16.5)
    Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount("/content/gdrive", force_remount=True).

### Helper Functions
The functions below help to visualize different parts of the model. 'to_image' converts an output of the model into an image easily shown using 'disp_model', which uses matplotlib to plot and show an image. 'visualize_network' is a function that shows the structure of the model used.

```
def to_img(x):
    """Converts x into an image compatible with matplotlib"""
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

def disp_image(image, predicted=None):

    plt.imshow(image, 'gray')
    plt.axis('off')
    plt.show()

def visualize_network(net, device='cuda', image_dims=(1,28,28)):
    # Visualize the architecture of the model
    # We need to give the net a fake input for this library to visualize the architecture
    fake_input = Variable(torch.zeros((1,image_dims[0], image_dims[1], image_dims[2]))).to(device)
    outputs = net(fake_input)
    # Plot the DAG (Directed Acyclic Graph) of the model
    return make_dot(outputs, dict(net.named_parameters()))
```

### Hyperparameters
After some testing, we found that 20 epochs with a batch size of 128 and a learning rate of 0.001, our model became reasonably accurate.

```
epochs = 20
batch_size = 128
learning_rate = 1e-3
```

### Loading the Data

```
img_transform = transforms.Compose([
    transforms.ToTensor(), # Transforms data to tensor
    transforms.Normalize((.5,), (.5,)) # Normalizes data
])

# Loads data and formats based on transform above
dataset = MNIST('./data', download=True, transform=img_transform)

# Creates a dataloader for pulling random sets of data for training
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
```

### The Model
The model below is an autoencoder, a type of unsupervised learning model. It has two primary parts: the encoder and decoder. The encoder learns non-linear patterns for reducing the dimensionality of the image, and the decoder learns how to convert an encoded image back to a similar image to the original input. The layer structure of the model was based off the code seen here: https://github.com/L1aoXingyu/pytorch-beginner/blob/master/08-AutoEncoder/conv_autoencoder.py.

```
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=3, padding=1), # Convolution 1 input, 16 outputs, 3x3 kernel
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2), # Maxpooling 2x2 kernel
            nn.Conv2d(16, 8, 3, stride=2, padding=1), # Convolution 16 inputs, 8 outputs, 3x3 kernel
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=1) # Maxpooling 2x2 kernel
        )

        # The decoder reverses the steps accomplished in the encoder.
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 3, stride=2),
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 5, stride=3, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 1, 2, stride=2, padding=1),
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
```

### Training the Model
The model was trained over 20 epochs using a batch size of 128 and learning rate of 0.001. Any more epochs only seemed to overfit the data and only minimally decrease the loss. Because we have no labels, our loss was calculated as the mean squared error between the input and output image of the model. By optimizing this value, the autoencoder could learn to encode and decode the data.

```
def train_model(model, num_epochs, find_loss, model_save='autoencoder.pt'):
    batch_loss = []
    batch_num = []
    curr_batch = 0

    print_every = 16

    # Attempts to load model from google drive
    try:
        path = F"/content/gdrive/My Drive/{model_save}"
        model.load_state_dict(torch.load(path))
        print("Model Loaded")

    # If model is not found in drive, model is trained
    except:
        print("Model Training")

        # Iterates through num_epochs epochs
        for epoch in range(num_epochs):
            # Total loss of a particular batch
            running_loss = 0.0

            # The start time of an epoch
            start_time = time.time()

            # Iterates through a batch
            for i, data in enumerate(dataloader, 0):
                # Loads image and converts is to a variable in graphics memory
                img, _ = data
                img = Variable(img).cuda()
                # ===================forward=====================
                # Plugs input into the model and finds loss
                output = model(img)
                loss = find_loss(output, img)
                running_loss += loss.data.item()
                # ===================backward====================
                # Backpropagates values to tweak weights
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                # Prints epoch every print_every and appends loss data to list
                if (i % print_every) == print_every-1:
                    print("Epoch {}, Iteration {}\t train_loss: {:.2f} took: {:.2f}s".format(
                          epoch + 1, i+1, running_loss / print_every, time.time() - start_time))
                    batch_loss.append(running_loss / print_every)
                    batch_num.append(curr_batch)
                    running_loss = 0.0

                curr_batch+=1
            # ===================log========================
            # Prints when epoch is finished
            print('\nEpoch [{}/{}], loss:{:.4f}'
                .format(epoch+1, num_epochs, loss.data))

        # Saves model to
        model_save_name = 'autoencoder.pt'
        path = F"/content/gdrive/My Drive/{model_save_name}"
        torch.save(model.state_dict(), path)

    return batch_loss, batch_num
```

```
model = Autoencoder().cuda()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

visualize_network(model)
```

```
batch_loss, batch_num = train_model(model, epochs, criterion)
```

    Model Training
    Epoch 1, Iteration 16	 train_loss: 0.75 took: 0.51s
    Epoch 1, Iteration 32	 train_loss: 0.67 took: 1.01s
    Epoch 1, Iteration 48	 train_loss: 0.52 took: 1.49s
    Epoch 1, Iteration 64	 train_loss: 0.43 took: 1.94s
    Epoch 1, Iteration 80	 train_loss: 0.40 took: 2.41s
    Epoch 1, Iteration 96	 train_loss: 0.38 took: 2.89s
    Epoch 1, Iteration 112	 train_loss: 0.37 took: 3.36s
    Epoch 1, Iteration 128	 train_loss: 0.36 took: 3.82s
    Epoch 1, Iteration 144	 train_loss: 0.35 took: 4.31s
    Epoch 1, Iteration 160	 train_loss: 0.35 took: 4.76s
    Epoch 1, Iteration 176	 train_loss: 0.34 took: 5.23s
    Epoch 1, Iteration 192	 train_loss: 0.33 took: 5.71s
    Epoch 1, Iteration 208	 train_loss: 0.33 took: 6.24s
    Epoch 1, Iteration 224	 train_loss: 0.33 took: 6.71s
    Epoch 1, Iteration 240	 train_loss: 0.33 took: 7.19s
    Epoch 1, Iteration 256	 train_loss: 0.33 took: 7.64s
    Epoch 1, Iteration 272	 train_loss: 0.32 took: 8.10s
    Epoch 1, Iteration 288	 train_loss: 0.32 took: 8.59s
    Epoch 1, Iteration 304	 train_loss: 0.32 took: 9.05s
    Epoch 1, Iteration 320	 train_loss: 0.32 took: 9.53s
    Epoch 1, Iteration 336	 train_loss: 0.32 took: 10.00s
    Epoch 1, Iteration 352	 train_loss: 0.31 took: 10.49s
    Epoch 1, Iteration 368	 train_loss: 0.31 took: 10.96s
    Epoch 1, Iteration 384	 train_loss: 0.31 took: 11.42s
    Epoch 1, Iteration 400	 train_loss: 0.31 took: 11.91s
    Epoch 1, Iteration 416	 train_loss: 0.31 took: 12.39s
    Epoch 1, Iteration 432	 train_loss: 0.30 took: 12.84s
    Epoch 1, Iteration 448	 train_loss: 0.30 took: 13.30s
    Epoch 1, Iteration 464	 train_loss: 0.29 took: 13.78s

    Epoch [1/20], loss:0.2851
    Epoch 2, Iteration 16	 train_loss: 0.28 took: 0.48s
    Epoch 2, Iteration 32	 train_loss: 0.26 took: 0.94s
    Epoch 2, Iteration 48	 train_loss: 0.25 took: 1.41s
    Epoch 2, Iteration 64	 train_loss: 0.24 took: 1.89s
    Epoch 2, Iteration 80	 train_loss: 0.23 took: 2.38s
    Epoch 2, Iteration 96	 train_loss: 0.22 took: 2.84s
    Epoch 2, Iteration 112	 train_loss: 0.22 took: 3.29s
    Epoch 2, Iteration 128	 train_loss: 0.22 took: 3.80s
    Epoch 2, Iteration 144	 train_loss: 0.21 took: 4.29s
    Epoch 2, Iteration 160	 train_loss: 0.21 took: 4.74s
    Epoch 2, Iteration 176	 train_loss: 0.20 took: 5.20s
    Epoch 2, Iteration 192	 train_loss: 0.20 took: 5.66s
    Epoch 2, Iteration 208	 train_loss: 0.20 took: 6.12s
    Epoch 2, Iteration 224	 train_loss: 0.20 took: 6.61s
    Epoch 2, Iteration 240	 train_loss: 0.19 took: 7.09s
    Epoch 2, Iteration 256	 train_loss: 0.19 took: 7.57s
    Epoch 2, Iteration 272	 train_loss: 0.19 took: 8.05s
    Epoch 2, Iteration 288	 train_loss: 0.19 took: 8.51s
    Epoch 2, Iteration 304	 train_loss: 0.19 took: 8.98s
    Epoch 2, Iteration 320	 train_loss: 0.19 took: 9.46s
    Epoch 2, Iteration 336	 train_loss: 0.18 took: 9.94s
    Epoch 2, Iteration 352	 train_loss: 0.18 took: 10.41s
    Epoch 2, Iteration 368	 train_loss: 0.18 took: 10.86s
    Epoch 2, Iteration 384	 train_loss: 0.18 took: 11.32s
    Epoch 2, Iteration 400	 train_loss: 0.18 took: 11.80s
    Epoch 2, Iteration 416	 train_loss: 0.18 took: 12.29s
    Epoch 2, Iteration 432	 train_loss: 0.18 took: 12.77s
    Epoch 2, Iteration 448	 train_loss: 0.18 took: 13.24s
    Epoch 2, Iteration 464	 train_loss: 0.18 took: 13.71s

    Epoch [2/20], loss:0.1749
    Epoch 3, Iteration 16	 train_loss: 0.17 took: 0.48s
    Epoch 3, Iteration 32	 train_loss: 0.17 took: 0.97s
    Epoch 3, Iteration 48	 train_loss: 0.17 took: 1.44s
    Epoch 3, Iteration 64	 train_loss: 0.17 took: 1.89s
    Epoch 3, Iteration 80	 train_loss: 0.17 took: 2.37s
    Epoch 3, Iteration 96	 train_loss: 0.17 took: 2.82s
    Epoch 3, Iteration 112	 train_loss: 0.17 took: 3.30s
    Epoch 3, Iteration 128	 train_loss: 0.17 took: 3.74s
    Epoch 3, Iteration 144	 train_loss: 0.17 took: 4.21s
    Epoch 3, Iteration 160	 train_loss: 0.16 took: 4.66s
    Epoch 3, Iteration 176	 train_loss: 0.16 took: 5.12s
    Epoch 3, Iteration 192	 train_loss: 0.16 took: 5.56s
    Epoch 3, Iteration 208	 train_loss: 0.16 took: 6.02s
    Epoch 3, Iteration 224	 train_loss: 0.16 took: 6.50s
    Epoch 3, Iteration 240	 train_loss: 0.16 took: 6.96s
    Epoch 3, Iteration 256	 train_loss: 0.16 took: 7.45s
    Epoch 3, Iteration 272	 train_loss: 0.16 took: 7.92s
    Epoch 3, Iteration 288	 train_loss: 0.15 took: 8.41s
    Epoch 3, Iteration 304	 train_loss: 0.15 took: 8.91s
    Epoch 3, Iteration 320	 train_loss: 0.15 took: 9.38s
    Epoch 3, Iteration 336	 train_loss: 0.15 took: 9.85s
    Epoch 3, Iteration 352	 train_loss: 0.15 took: 10.35s
    Epoch 3, Iteration 368	 train_loss: 0.15 took: 10.81s
    Epoch 3, Iteration 384	 train_loss: 0.15 took: 11.29s
    Epoch 3, Iteration 400	 train_loss: 0.15 took: 11.74s
    Epoch 3, Iteration 416	 train_loss: 0.15 took: 12.21s
    Epoch 3, Iteration 432	 train_loss: 0.15 took: 12.68s
    Epoch 3, Iteration 448	 train_loss: 0.15 took: 13.16s
    Epoch 3, Iteration 464	 train_loss: 0.15 took: 13.64s

    Epoch [3/20], loss:0.1478
    Epoch 4, Iteration 16	 train_loss: 0.15 took: 0.48s
    Epoch 4, Iteration 32	 train_loss: 0.15 took: 0.96s
    Epoch 4, Iteration 48	 train_loss: 0.15 took: 1.43s
    Epoch 4, Iteration 64	 train_loss: 0.15 took: 1.90s
    Epoch 4, Iteration 80	 train_loss: 0.15 took: 2.35s
    Epoch 4, Iteration 96	 train_loss: 0.15 took: 2.85s
    Epoch 4, Iteration 112	 train_loss: 0.14 took: 3.32s
    Epoch 4, Iteration 128	 train_loss: 0.15 took: 3.83s
    Epoch 4, Iteration 144	 train_loss: 0.14 took: 4.30s
    Epoch 4, Iteration 160	 train_loss: 0.14 took: 4.77s
    Epoch 4, Iteration 176	 train_loss: 0.14 took: 5.25s
    Epoch 4, Iteration 192	 train_loss: 0.14 took: 5.72s
    Epoch 4, Iteration 208	 train_loss: 0.14 took: 6.17s
    Epoch 4, Iteration 224	 train_loss: 0.14 took: 6.65s
    Epoch 4, Iteration 240	 train_loss: 0.14 took: 7.12s
    Epoch 4, Iteration 256	 train_loss: 0.14 took: 7.59s
    Epoch 4, Iteration 272	 train_loss: 0.15 took: 8.05s
    Epoch 4, Iteration 288	 train_loss: 0.14 took: 8.50s
    Epoch 4, Iteration 304	 train_loss: 0.14 took: 8.98s
    Epoch 4, Iteration 320	 train_loss: 0.14 took: 9.45s
    Epoch 4, Iteration 336	 train_loss: 0.14 took: 9.93s
    Epoch 4, Iteration 352	 train_loss: 0.14 took: 10.39s
    Epoch 4, Iteration 368	 train_loss: 0.14 took: 10.87s
    Epoch 4, Iteration 384	 train_loss: 0.14 took: 11.35s
    Epoch 4, Iteration 400	 train_loss: 0.14 took: 11.82s
    Epoch 4, Iteration 416	 train_loss: 0.14 took: 12.27s
    Epoch 4, Iteration 432	 train_loss: 0.14 took: 12.74s
    Epoch 4, Iteration 448	 train_loss: 0.14 took: 13.19s
    Epoch 4, Iteration 464	 train_loss: 0.14 took: 13.64s

    Epoch [4/20], loss:0.1456
    Epoch 5, Iteration 16	 train_loss: 0.14 took: 0.46s
    Epoch 5, Iteration 32	 train_loss: 0.14 took: 0.94s
    Epoch 5, Iteration 48	 train_loss: 0.14 took: 1.44s
    Epoch 5, Iteration 64	 train_loss: 0.14 took: 1.91s
    Epoch 5, Iteration 80	 train_loss: 0.14 took: 2.40s
    Epoch 5, Iteration 96	 train_loss: 0.14 took: 2.87s
    Epoch 5, Iteration 112	 train_loss: 0.14 took: 3.37s
    Epoch 5, Iteration 128	 train_loss: 0.14 took: 3.83s
    Epoch 5, Iteration 144	 train_loss: 0.14 took: 4.31s
    Epoch 5, Iteration 160	 train_loss: 0.14 took: 4.77s
    Epoch 5, Iteration 176	 train_loss: 0.14 took: 5.25s
    Epoch 5, Iteration 192	 train_loss: 0.14 took: 5.73s
    Epoch 5, Iteration 208	 train_loss: 0.14 took: 6.18s
    Epoch 5, Iteration 224	 train_loss: 0.14 took: 6.65s
    Epoch 5, Iteration 240	 train_loss: 0.14 took: 7.10s
    Epoch 5, Iteration 256	 train_loss: 0.14 took: 7.55s
    Epoch 5, Iteration 272	 train_loss: 0.14 took: 8.01s
    Epoch 5, Iteration 288	 train_loss: 0.14 took: 8.51s
    Epoch 5, Iteration 304	 train_loss: 0.14 took: 8.99s
    Epoch 5, Iteration 320	 train_loss: 0.14 took: 9.45s
    Epoch 5, Iteration 336	 train_loss: 0.13 took: 9.90s
    Epoch 5, Iteration 352	 train_loss: 0.14 took: 10.36s
    Epoch 5, Iteration 368	 train_loss: 0.14 took: 10.81s
    Epoch 5, Iteration 384	 train_loss: 0.14 took: 11.27s
    Epoch 5, Iteration 400	 train_loss: 0.14 took: 11.76s
    Epoch 5, Iteration 416	 train_loss: 0.14 took: 12.23s
    Epoch 5, Iteration 432	 train_loss: 0.13 took: 12.70s
    Epoch 5, Iteration 448	 train_loss: 0.13 took: 13.15s
    Epoch 5, Iteration 464	 train_loss: 0.14 took: 13.61s

    Epoch [5/20], loss:0.1359
    Epoch 6, Iteration 16	 train_loss: 0.14 took: 0.48s
    Epoch 6, Iteration 32	 train_loss: 0.14 took: 0.96s
    Epoch 6, Iteration 48	 train_loss: 0.13 took: 1.42s
    Epoch 6, Iteration 64	 train_loss: 0.13 took: 1.92s
    Epoch 6, Iteration 80	 train_loss: 0.14 took: 2.37s
    Epoch 6, Iteration 96	 train_loss: 0.14 took: 2.84s
    Epoch 6, Iteration 112	 train_loss: 0.13 took: 3.34s
    Epoch 6, Iteration 128	 train_loss: 0.13 took: 3.81s
    Epoch 6, Iteration 144	 train_loss: 0.14 took: 4.29s
    Epoch 6, Iteration 160	 train_loss: 0.13 took: 4.76s
    Epoch 6, Iteration 176	 train_loss: 0.13 took: 5.26s
    Epoch 6, Iteration 192	 train_loss: 0.13 took: 5.73s
    Epoch 6, Iteration 208	 train_loss: 0.14 took: 6.20s
    Epoch 6, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 6, Iteration 240	 train_loss: 0.13 took: 7.13s
    Epoch 6, Iteration 256	 train_loss: 0.14 took: 7.60s
    Epoch 6, Iteration 272	 train_loss: 0.13 took: 8.11s
    Epoch 6, Iteration 288	 train_loss: 0.13 took: 8.59s
    Epoch 6, Iteration 304	 train_loss: 0.13 took: 9.07s
    Epoch 6, Iteration 320	 train_loss: 0.13 took: 9.55s
    Epoch 6, Iteration 336	 train_loss: 0.13 took: 10.03s
    Epoch 6, Iteration 352	 train_loss: 0.13 took: 10.49s
    Epoch 6, Iteration 368	 train_loss: 0.13 took: 10.98s
    Epoch 6, Iteration 384	 train_loss: 0.13 took: 11.49s
    Epoch 6, Iteration 400	 train_loss: 0.13 took: 11.99s
    Epoch 6, Iteration 416	 train_loss: 0.13 took: 12.48s
    Epoch 6, Iteration 432	 train_loss: 0.13 took: 12.95s
    Epoch 6, Iteration 448	 train_loss: 0.13 took: 13.41s
    Epoch 6, Iteration 464	 train_loss: 0.13 took: 13.87s

    Epoch [6/20], loss:0.1376
    Epoch 7, Iteration 16	 train_loss: 0.13 took: 0.47s
    Epoch 7, Iteration 32	 train_loss: 0.13 took: 0.97s
    Epoch 7, Iteration 48	 train_loss: 0.13 took: 1.42s
    Epoch 7, Iteration 64	 train_loss: 0.13 took: 1.87s
    Epoch 7, Iteration 80	 train_loss: 0.13 took: 2.36s
    Epoch 7, Iteration 96	 train_loss: 0.13 took: 2.84s
    Epoch 7, Iteration 112	 train_loss: 0.13 took: 3.30s
    Epoch 7, Iteration 128	 train_loss: 0.13 took: 3.75s
    Epoch 7, Iteration 144	 train_loss: 0.13 took: 4.28s
    Epoch 7, Iteration 160	 train_loss: 0.13 took: 4.75s
    Epoch 7, Iteration 176	 train_loss: 0.13 took: 5.23s
    Epoch 7, Iteration 192	 train_loss: 0.13 took: 5.70s
    Epoch 7, Iteration 208	 train_loss: 0.13 took: 6.19s
    Epoch 7, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 7, Iteration 240	 train_loss: 0.13 took: 7.13s
    Epoch 7, Iteration 256	 train_loss: 0.13 took: 7.62s
    Epoch 7, Iteration 272	 train_loss: 0.13 took: 8.10s
    Epoch 7, Iteration 288	 train_loss: 0.13 took: 8.56s
    Epoch 7, Iteration 304	 train_loss: 0.13 took: 9.02s
    Epoch 7, Iteration 320	 train_loss: 0.13 took: 9.49s
    Epoch 7, Iteration 336	 train_loss: 0.13 took: 9.97s
    Epoch 7, Iteration 352	 train_loss: 0.13 took: 10.44s
    Epoch 7, Iteration 368	 train_loss: 0.13 took: 10.92s
    Epoch 7, Iteration 384	 train_loss: 0.13 took: 11.40s
    Epoch 7, Iteration 400	 train_loss: 0.13 took: 11.87s
    Epoch 7, Iteration 416	 train_loss: 0.13 took: 12.35s
    Epoch 7, Iteration 432	 train_loss: 0.13 took: 12.84s
    Epoch 7, Iteration 448	 train_loss: 0.13 took: 13.33s
    Epoch 7, Iteration 464	 train_loss: 0.13 took: 13.81s

    Epoch [7/20], loss:0.1277
    Epoch 8, Iteration 16	 train_loss: 0.13 took: 0.50s
    Epoch 8, Iteration 32	 train_loss: 0.13 took: 0.98s
    Epoch 8, Iteration 48	 train_loss: 0.13 took: 1.44s
    Epoch 8, Iteration 64	 train_loss: 0.13 took: 1.90s
    Epoch 8, Iteration 80	 train_loss: 0.13 took: 2.38s
    Epoch 8, Iteration 96	 train_loss: 0.13 took: 2.86s
    Epoch 8, Iteration 112	 train_loss: 0.13 took: 3.35s
    Epoch 8, Iteration 128	 train_loss: 0.13 took: 3.83s
    Epoch 8, Iteration 144	 train_loss: 0.13 took: 4.30s
    Epoch 8, Iteration 160	 train_loss: 0.13 took: 4.78s
    Epoch 8, Iteration 176	 train_loss: 0.13 took: 5.25s
    Epoch 8, Iteration 192	 train_loss: 0.13 took: 5.71s
    Epoch 8, Iteration 208	 train_loss: 0.13 took: 6.17s
    Epoch 8, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 8, Iteration 240	 train_loss: 0.13 took: 7.11s
    Epoch 8, Iteration 256	 train_loss: 0.13 took: 7.56s
    Epoch 8, Iteration 272	 train_loss: 0.13 took: 8.04s
    Epoch 8, Iteration 288	 train_loss: 0.13 took: 8.49s
    Epoch 8, Iteration 304	 train_loss: 0.13 took: 8.94s
    Epoch 8, Iteration 320	 train_loss: 0.13 took: 9.42s
    Epoch 8, Iteration 336	 train_loss: 0.13 took: 9.91s
    Epoch 8, Iteration 352	 train_loss: 0.13 took: 10.40s
    Epoch 8, Iteration 368	 train_loss: 0.13 took: 10.88s
    Epoch 8, Iteration 384	 train_loss: 0.13 took: 11.35s
    Epoch 8, Iteration 400	 train_loss: 0.13 took: 11.85s
    Epoch 8, Iteration 416	 train_loss: 0.13 took: 12.32s
    Epoch 8, Iteration 432	 train_loss: 0.13 took: 12.80s
    Epoch 8, Iteration 448	 train_loss: 0.13 took: 13.27s
    Epoch 8, Iteration 464	 train_loss: 0.13 took: 13.72s

    Epoch [8/20], loss:0.1196
    Epoch 9, Iteration 16	 train_loss: 0.13 took: 0.49s
    Epoch 9, Iteration 32	 train_loss: 0.13 took: 0.97s
    Epoch 9, Iteration 48	 train_loss: 0.13 took: 1.43s
    Epoch 9, Iteration 64	 train_loss: 0.13 took: 1.89s
    Epoch 9, Iteration 80	 train_loss: 0.13 took: 2.35s
    Epoch 9, Iteration 96	 train_loss: 0.13 took: 2.79s
    Epoch 9, Iteration 112	 train_loss: 0.13 took: 3.25s
    Epoch 9, Iteration 128	 train_loss: 0.13 took: 3.70s
    Epoch 9, Iteration 144	 train_loss: 0.13 took: 4.18s
    Epoch 9, Iteration 160	 train_loss: 0.13 took: 4.63s
    Epoch 9, Iteration 176	 train_loss: 0.13 took: 5.09s
    Epoch 9, Iteration 192	 train_loss: 0.13 took: 5.54s
    Epoch 9, Iteration 208	 train_loss: 0.13 took: 6.01s
    Epoch 9, Iteration 224	 train_loss: 0.13 took: 6.47s
    Epoch 9, Iteration 240	 train_loss: 0.13 took: 6.94s
    Epoch 9, Iteration 256	 train_loss: 0.13 took: 7.40s
    Epoch 9, Iteration 272	 train_loss: 0.13 took: 7.86s
    Epoch 9, Iteration 288	 train_loss: 0.13 took: 8.32s
    Epoch 9, Iteration 304	 train_loss: 0.13 took: 8.77s
    Epoch 9, Iteration 320	 train_loss: 0.13 took: 9.23s
    Epoch 9, Iteration 336	 train_loss: 0.13 took: 9.67s
    Epoch 9, Iteration 352	 train_loss: 0.13 took: 10.13s
    Epoch 9, Iteration 368	 train_loss: 0.12 took: 10.61s
    Epoch 9, Iteration 384	 train_loss: 0.13 took: 11.07s
    Epoch 9, Iteration 400	 train_loss: 0.13 took: 11.54s
    Epoch 9, Iteration 416	 train_loss: 0.13 took: 11.99s
    Epoch 9, Iteration 432	 train_loss: 0.13 took: 12.47s
    Epoch 9, Iteration 448	 train_loss: 0.13 took: 12.93s
    Epoch 9, Iteration 464	 train_loss: 0.12 took: 13.45s

    Epoch [9/20], loss:0.1304
    Epoch 10, Iteration 16	 train_loss: 0.13 took: 0.47s
    Epoch 10, Iteration 32	 train_loss: 0.13 took: 0.93s
    Epoch 10, Iteration 48	 train_loss: 0.13 took: 1.38s
    Epoch 10, Iteration 64	 train_loss: 0.12 took: 1.85s
    Epoch 10, Iteration 80	 train_loss: 0.12 took: 2.30s
    Epoch 10, Iteration 96	 train_loss: 0.12 took: 2.79s
    Epoch 10, Iteration 112	 train_loss: 0.12 took: 3.26s
    Epoch 10, Iteration 128	 train_loss: 0.13 took: 3.72s
    Epoch 10, Iteration 144	 train_loss: 0.13 took: 4.21s
    Epoch 10, Iteration 160	 train_loss: 0.13 took: 4.67s
    Epoch 10, Iteration 176	 train_loss: 0.13 took: 5.17s
    Epoch 10, Iteration 192	 train_loss: 0.13 took: 5.65s
    Epoch 10, Iteration 208	 train_loss: 0.12 took: 6.13s
    Epoch 10, Iteration 224	 train_loss: 0.13 took: 6.61s
    Epoch 10, Iteration 240	 train_loss: 0.13 took: 7.10s
    Epoch 10, Iteration 256	 train_loss: 0.12 took: 7.61s
    Epoch 10, Iteration 272	 train_loss: 0.12 took: 8.10s
    Epoch 10, Iteration 288	 train_loss: 0.13 took: 8.59s
    Epoch 10, Iteration 304	 train_loss: 0.12 took: 9.09s
    Epoch 10, Iteration 320	 train_loss: 0.13 took: 9.58s
    Epoch 10, Iteration 336	 train_loss: 0.12 took: 10.05s
    Epoch 10, Iteration 352	 train_loss: 0.13 took: 10.51s
    Epoch 10, Iteration 368	 train_loss: 0.13 took: 11.02s
    Epoch 10, Iteration 384	 train_loss: 0.13 took: 11.51s
    Epoch 10, Iteration 400	 train_loss: 0.12 took: 12.00s
    Epoch 10, Iteration 416	 train_loss: 0.12 took: 12.48s
    Epoch 10, Iteration 432	 train_loss: 0.13 took: 12.96s
    Epoch 10, Iteration 448	 train_loss: 0.13 took: 13.48s
    Epoch 10, Iteration 464	 train_loss: 0.12 took: 13.98s

    Epoch [10/20], loss:0.1279
    Epoch 11, Iteration 16	 train_loss: 0.13 took: 0.49s
    Epoch 11, Iteration 32	 train_loss: 0.12 took: 0.98s
    Epoch 11, Iteration 48	 train_loss: 0.13 took: 1.46s
    Epoch 11, Iteration 64	 train_loss: 0.12 took: 1.91s
    Epoch 11, Iteration 80	 train_loss: 0.13 took: 2.38s
    Epoch 11, Iteration 96	 train_loss: 0.12 took: 2.87s
    Epoch 11, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 11, Iteration 128	 train_loss: 0.12 took: 3.83s
    Epoch 11, Iteration 144	 train_loss: 0.13 took: 4.31s
    Epoch 11, Iteration 160	 train_loss: 0.13 took: 4.78s
    Epoch 11, Iteration 176	 train_loss: 0.12 took: 5.26s
    Epoch 11, Iteration 192	 train_loss: 0.12 took: 5.76s
    Epoch 11, Iteration 208	 train_loss: 0.13 took: 6.24s
    Epoch 11, Iteration 224	 train_loss: 0.12 took: 6.70s
    Epoch 11, Iteration 240	 train_loss: 0.13 took: 7.18s
    Epoch 11, Iteration 256	 train_loss: 0.12 took: 7.66s
    Epoch 11, Iteration 272	 train_loss: 0.12 took: 8.14s
    Epoch 11, Iteration 288	 train_loss: 0.12 took: 8.62s
    Epoch 11, Iteration 304	 train_loss: 0.12 took: 9.09s
    Epoch 11, Iteration 320	 train_loss: 0.12 took: 9.59s
    Epoch 11, Iteration 336	 train_loss: 0.12 took: 10.04s
    Epoch 11, Iteration 352	 train_loss: 0.12 took: 10.52s
    Epoch 11, Iteration 368	 train_loss: 0.12 took: 10.99s
    Epoch 11, Iteration 384	 train_loss: 0.13 took: 11.48s
    Epoch 11, Iteration 400	 train_loss: 0.12 took: 11.92s
    Epoch 11, Iteration 416	 train_loss: 0.12 took: 12.39s
    Epoch 11, Iteration 432	 train_loss: 0.12 took: 12.86s
    Epoch 11, Iteration 448	 train_loss: 0.12 took: 13.35s
    Epoch 11, Iteration 464	 train_loss: 0.12 took: 13.83s

    Epoch [11/20], loss:0.1313
    Epoch 12, Iteration 16	 train_loss: 0.12 took: 0.47s
    Epoch 12, Iteration 32	 train_loss: 0.12 took: 0.93s
    Epoch 12, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 12, Iteration 64	 train_loss: 0.13 took: 1.86s
    Epoch 12, Iteration 80	 train_loss: 0.13 took: 2.33s
    Epoch 12, Iteration 96	 train_loss: 0.12 took: 2.82s
    Epoch 12, Iteration 112	 train_loss: 0.12 took: 3.27s
    Epoch 12, Iteration 128	 train_loss: 0.12 took: 3.74s
    Epoch 12, Iteration 144	 train_loss: 0.12 took: 4.19s
    Epoch 12, Iteration 160	 train_loss: 0.12 took: 4.70s
    Epoch 12, Iteration 176	 train_loss: 0.12 took: 5.17s
    Epoch 12, Iteration 192	 train_loss: 0.12 took: 5.66s
    Epoch 12, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 12, Iteration 224	 train_loss: 0.12 took: 6.60s
    Epoch 12, Iteration 240	 train_loss: 0.12 took: 7.07s
    Epoch 12, Iteration 256	 train_loss: 0.12 took: 7.56s
    Epoch 12, Iteration 272	 train_loss: 0.12 took: 8.02s
    Epoch 12, Iteration 288	 train_loss: 0.12 took: 8.49s
    Epoch 12, Iteration 304	 train_loss: 0.12 took: 8.96s
    Epoch 12, Iteration 320	 train_loss: 0.12 took: 9.41s
    Epoch 12, Iteration 336	 train_loss: 0.12 took: 9.88s
    Epoch 12, Iteration 352	 train_loss: 0.12 took: 10.39s
    Epoch 12, Iteration 368	 train_loss: 0.12 took: 10.85s
    Epoch 12, Iteration 384	 train_loss: 0.12 took: 11.30s
    Epoch 12, Iteration 400	 train_loss: 0.12 took: 11.80s
    Epoch 12, Iteration 416	 train_loss: 0.12 took: 12.26s
    Epoch 12, Iteration 432	 train_loss: 0.12 took: 12.75s
    Epoch 12, Iteration 448	 train_loss: 0.12 took: 13.22s
    Epoch 12, Iteration 464	 train_loss: 0.12 took: 13.69s

    Epoch [12/20], loss:0.1206
    Epoch 13, Iteration 16	 train_loss: 0.12 took: 0.45s
    Epoch 13, Iteration 32	 train_loss: 0.12 took: 0.94s
    Epoch 13, Iteration 48	 train_loss: 0.12 took: 1.50s
    Epoch 13, Iteration 64	 train_loss: 0.12 took: 2.03s
    Epoch 13, Iteration 80	 train_loss: 0.12 took: 2.49s
    Epoch 13, Iteration 96	 train_loss: 0.12 took: 2.98s
    Epoch 13, Iteration 112	 train_loss: 0.12 took: 3.46s
    Epoch 13, Iteration 128	 train_loss: 0.12 took: 3.96s
    Epoch 13, Iteration 144	 train_loss: 0.12 took: 4.45s
    Epoch 13, Iteration 160	 train_loss: 0.12 took: 4.93s
    Epoch 13, Iteration 176	 train_loss: 0.12 took: 5.41s
    Epoch 13, Iteration 192	 train_loss: 0.12 took: 5.86s
    Epoch 13, Iteration 208	 train_loss: 0.12 took: 6.31s
    Epoch 13, Iteration 224	 train_loss: 0.12 took: 6.79s
    Epoch 13, Iteration 240	 train_loss: 0.12 took: 7.26s
    Epoch 13, Iteration 256	 train_loss: 0.12 took: 7.75s
    Epoch 13, Iteration 272	 train_loss: 0.12 took: 8.29s
    Epoch 13, Iteration 288	 train_loss: 0.12 took: 8.76s
    Epoch 13, Iteration 304	 train_loss: 0.12 took: 9.24s
    Epoch 13, Iteration 320	 train_loss: 0.12 took: 9.73s
    Epoch 13, Iteration 336	 train_loss: 0.12 took: 10.21s
    Epoch 13, Iteration 352	 train_loss: 0.12 took: 10.67s
    Epoch 13, Iteration 368	 train_loss: 0.12 took: 11.13s
    Epoch 13, Iteration 384	 train_loss: 0.12 took: 11.57s
    Epoch 13, Iteration 400	 train_loss: 0.12 took: 12.04s
    Epoch 13, Iteration 416	 train_loss: 0.12 took: 12.50s
    Epoch 13, Iteration 432	 train_loss: 0.12 took: 12.99s
    Epoch 13, Iteration 448	 train_loss: 0.12 took: 13.46s
    Epoch 13, Iteration 464	 train_loss: 0.12 took: 13.92s

    Epoch [13/20], loss:0.1162
    Epoch 14, Iteration 16	 train_loss: 0.12 took: 0.54s
    Epoch 14, Iteration 32	 train_loss: 0.12 took: 1.01s
    Epoch 14, Iteration 48	 train_loss: 0.12 took: 1.48s
    Epoch 14, Iteration 64	 train_loss: 0.12 took: 1.98s
    Epoch 14, Iteration 80	 train_loss: 0.12 took: 2.44s
    Epoch 14, Iteration 96	 train_loss: 0.12 took: 2.92s
    Epoch 14, Iteration 112	 train_loss: 0.12 took: 3.43s
    Epoch 14, Iteration 128	 train_loss: 0.12 took: 3.94s
    Epoch 14, Iteration 144	 train_loss: 0.12 took: 4.42s
    Epoch 14, Iteration 160	 train_loss: 0.12 took: 4.90s
    Epoch 14, Iteration 176	 train_loss: 0.12 took: 5.38s
    Epoch 14, Iteration 192	 train_loss: 0.12 took: 5.85s
    Epoch 14, Iteration 208	 train_loss: 0.12 took: 6.31s
    Epoch 14, Iteration 224	 train_loss: 0.12 took: 6.78s
    Epoch 14, Iteration 240	 train_loss: 0.12 took: 7.22s
    Epoch 14, Iteration 256	 train_loss: 0.12 took: 7.71s
    Epoch 14, Iteration 272	 train_loss: 0.12 took: 8.19s
    Epoch 14, Iteration 288	 train_loss: 0.12 took: 8.66s
    Epoch 14, Iteration 304	 train_loss: 0.12 took: 9.12s
    Epoch 14, Iteration 320	 train_loss: 0.12 took: 9.59s
    Epoch 14, Iteration 336	 train_loss: 0.12 took: 10.03s
    Epoch 14, Iteration 352	 train_loss: 0.12 took: 10.50s
    Epoch 14, Iteration 368	 train_loss: 0.12 took: 10.96s
    Epoch 14, Iteration 384	 train_loss: 0.12 took: 11.42s
    Epoch 14, Iteration 400	 train_loss: 0.12 took: 11.87s
    Epoch 14, Iteration 416	 train_loss: 0.12 took: 12.33s
    Epoch 14, Iteration 432	 train_loss: 0.12 took: 12.80s
    Epoch 14, Iteration 448	 train_loss: 0.12 took: 13.25s
    Epoch 14, Iteration 464	 train_loss: 0.12 took: 13.73s

    Epoch [14/20], loss:0.1277
    Epoch 15, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 15, Iteration 32	 train_loss: 0.12 took: 0.95s
    Epoch 15, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 15, Iteration 64	 train_loss: 0.12 took: 1.88s
    Epoch 15, Iteration 80	 train_loss: 0.12 took: 2.34s
    Epoch 15, Iteration 96	 train_loss: 0.12 took: 2.82s
    Epoch 15, Iteration 112	 train_loss: 0.12 took: 3.27s
    Epoch 15, Iteration 128	 train_loss: 0.12 took: 3.73s
    Epoch 15, Iteration 144	 train_loss: 0.12 took: 4.18s
    Epoch 15, Iteration 160	 train_loss: 0.12 took: 4.67s
    Epoch 15, Iteration 176	 train_loss: 0.12 took: 5.16s
    Epoch 15, Iteration 192	 train_loss: 0.12 took: 5.63s
    Epoch 15, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 15, Iteration 224	 train_loss: 0.12 took: 6.59s
    Epoch 15, Iteration 240	 train_loss: 0.12 took: 7.10s
    Epoch 15, Iteration 256	 train_loss: 0.12 took: 7.58s
    Epoch 15, Iteration 272	 train_loss: 0.12 took: 8.08s
    Epoch 15, Iteration 288	 train_loss: 0.12 took: 8.56s
    Epoch 15, Iteration 304	 train_loss: 0.12 took: 9.04s
    Epoch 15, Iteration 320	 train_loss: 0.12 took: 9.51s
    Epoch 15, Iteration 336	 train_loss: 0.12 took: 9.99s
    Epoch 15, Iteration 352	 train_loss: 0.12 took: 10.45s
    Epoch 15, Iteration 368	 train_loss: 0.12 took: 10.94s
    Epoch 15, Iteration 384	 train_loss: 0.12 took: 11.41s
    Epoch 15, Iteration 400	 train_loss: 0.12 took: 11.88s
    Epoch 15, Iteration 416	 train_loss: 0.12 took: 12.35s
    Epoch 15, Iteration 432	 train_loss: 0.12 took: 12.83s
    Epoch 15, Iteration 448	 train_loss: 0.12 took: 13.33s
    Epoch 15, Iteration 464	 train_loss: 0.12 took: 13.80s

    Epoch [15/20], loss:0.1163
    Epoch 16, Iteration 16	 train_loss: 0.12 took: 0.51s
    Epoch 16, Iteration 32	 train_loss: 0.12 took: 0.97s
    Epoch 16, Iteration 48	 train_loss: 0.12 took: 1.48s
    Epoch 16, Iteration 64	 train_loss: 0.12 took: 1.97s
    Epoch 16, Iteration 80	 train_loss: 0.12 took: 2.45s
    Epoch 16, Iteration 96	 train_loss: 0.12 took: 2.92s
    Epoch 16, Iteration 112	 train_loss: 0.12 took: 3.41s
    Epoch 16, Iteration 128	 train_loss: 0.12 took: 3.89s
    Epoch 16, Iteration 144	 train_loss: 0.12 took: 4.36s
    Epoch 16, Iteration 160	 train_loss: 0.12 took: 4.82s
    Epoch 16, Iteration 176	 train_loss: 0.12 took: 5.29s
    Epoch 16, Iteration 192	 train_loss: 0.12 took: 5.78s
    Epoch 16, Iteration 208	 train_loss: 0.12 took: 6.26s
    Epoch 16, Iteration 224	 train_loss: 0.12 took: 6.73s
    Epoch 16, Iteration 240	 train_loss: 0.12 took: 7.20s
    Epoch 16, Iteration 256	 train_loss: 0.12 took: 7.69s
    Epoch 16, Iteration 272	 train_loss: 0.12 took: 8.15s
    Epoch 16, Iteration 288	 train_loss: 0.12 took: 8.62s
    Epoch 16, Iteration 304	 train_loss: 0.12 took: 9.10s
    Epoch 16, Iteration 320	 train_loss: 0.12 took: 9.58s
    Epoch 16, Iteration 336	 train_loss: 0.12 took: 10.04s
    Epoch 16, Iteration 352	 train_loss: 0.12 took: 10.51s
    Epoch 16, Iteration 368	 train_loss: 0.12 took: 11.02s
    Epoch 16, Iteration 384	 train_loss: 0.12 took: 11.50s
    Epoch 16, Iteration 400	 train_loss: 0.12 took: 11.98s
    Epoch 16, Iteration 416	 train_loss: 0.12 took: 12.46s
    Epoch 16, Iteration 432	 train_loss: 0.12 took: 12.93s
    Epoch 16, Iteration 448	 train_loss: 0.12 took: 13.41s
    Epoch 16, Iteration 464	 train_loss: 0.12 took: 13.89s

    Epoch [16/20], loss:0.1290
    Epoch 17, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 17, Iteration 32	 train_loss: 0.12 took: 0.97s
    Epoch 17, Iteration 48	 train_loss: 0.12 took: 1.46s
    Epoch 17, Iteration 64	 train_loss: 0.12 took: 1.95s
    Epoch 17, Iteration 80	 train_loss: 0.12 took: 2.44s
    Epoch 17, Iteration 96	 train_loss: 0.12 took: 2.89s
    Epoch 17, Iteration 112	 train_loss: 0.12 took: 3.33s
    Epoch 17, Iteration 128	 train_loss: 0.12 took: 3.79s
    Epoch 17, Iteration 144	 train_loss: 0.12 took: 4.24s
    Epoch 17, Iteration 160	 train_loss: 0.12 took: 4.70s
    Epoch 17, Iteration 176	 train_loss: 0.12 took: 5.15s
    Epoch 17, Iteration 192	 train_loss: 0.12 took: 5.64s
    Epoch 17, Iteration 208	 train_loss: 0.12 took: 6.08s
    Epoch 17, Iteration 224	 train_loss: 0.12 took: 6.55s
    Epoch 17, Iteration 240	 train_loss: 0.12 took: 7.05s
    Epoch 17, Iteration 256	 train_loss: 0.12 took: 7.51s
    Epoch 17, Iteration 272	 train_loss: 0.12 took: 7.97s
    Epoch 17, Iteration 288	 train_loss: 0.12 took: 8.42s
    Epoch 17, Iteration 304	 train_loss: 0.12 took: 8.90s
    Epoch 17, Iteration 320	 train_loss: 0.12 took: 9.36s
    Epoch 17, Iteration 336	 train_loss: 0.12 took: 9.82s
    Epoch 17, Iteration 352	 train_loss: 0.12 took: 10.27s
    Epoch 17, Iteration 368	 train_loss: 0.12 took: 10.75s
    Epoch 17, Iteration 384	 train_loss: 0.12 took: 11.21s
    Epoch 17, Iteration 400	 train_loss: 0.12 took: 11.70s
    Epoch 17, Iteration 416	 train_loss: 0.12 took: 12.19s
    Epoch 17, Iteration 432	 train_loss: 0.12 took: 12.68s
    Epoch 17, Iteration 448	 train_loss: 0.12 took: 13.16s
    Epoch 17, Iteration 464	 train_loss: 0.12 took: 13.64s

    Epoch [17/20], loss:0.1098
    Epoch 18, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 18, Iteration 32	 train_loss: 0.12 took: 0.96s
    Epoch 18, Iteration 48	 train_loss: 0.12 took: 1.43s
    Epoch 18, Iteration 64	 train_loss: 0.12 took: 1.91s
    Epoch 18, Iteration 80	 train_loss: 0.12 took: 2.38s
    Epoch 18, Iteration 96	 train_loss: 0.12 took: 2.86s
    Epoch 18, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 18, Iteration 128	 train_loss: 0.12 took: 3.79s
    Epoch 18, Iteration 144	 train_loss: 0.12 took: 4.27s
    Epoch 18, Iteration 160	 train_loss: 0.12 took: 4.72s
    Epoch 18, Iteration 176	 train_loss: 0.12 took: 5.19s
    Epoch 18, Iteration 192	 train_loss: 0.12 took: 5.64s
    Epoch 18, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 18, Iteration 224	 train_loss: 0.12 took: 6.56s
    Epoch 18, Iteration 240	 train_loss: 0.12 took: 7.01s
    Epoch 18, Iteration 256	 train_loss: 0.12 took: 7.46s
    Epoch 18, Iteration 272	 train_loss: 0.12 took: 7.94s
    Epoch 18, Iteration 288	 train_loss: 0.12 took: 8.40s
    Epoch 18, Iteration 304	 train_loss: 0.11 took: 8.87s
    Epoch 18, Iteration 320	 train_loss: 0.12 took: 9.33s
    Epoch 18, Iteration 336	 train_loss: 0.12 took: 9.78s
    Epoch 18, Iteration 352	 train_loss: 0.12 took: 10.26s
    Epoch 18, Iteration 368	 train_loss: 0.11 took: 10.71s
    Epoch 18, Iteration 384	 train_loss: 0.12 took: 11.19s
    Epoch 18, Iteration 400	 train_loss: 0.12 took: 11.68s
    Epoch 18, Iteration 416	 train_loss: 0.12 took: 12.15s
    Epoch 18, Iteration 432	 train_loss: 0.12 took: 12.63s
    Epoch 18, Iteration 448	 train_loss: 0.12 took: 13.11s
    Epoch 18, Iteration 464	 train_loss: 0.12 took: 13.63s

    Epoch [18/20], loss:0.1211
    Epoch 19, Iteration 16	 train_loss: 0.12 took: 0.47s
    Epoch 19, Iteration 32	 train_loss: 0.12 took: 0.93s
    Epoch 19, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 19, Iteration 64	 train_loss: 0.12 took: 1.85s
    Epoch 19, Iteration 80	 train_loss: 0.12 took: 2.30s
    Epoch 19, Iteration 96	 train_loss: 0.12 took: 2.76s
    Epoch 19, Iteration 112	 train_loss: 0.12 took: 3.23s
    Epoch 19, Iteration 128	 train_loss: 0.12 took: 3.72s
    Epoch 19, Iteration 144	 train_loss: 0.12 took: 4.20s
    Epoch 19, Iteration 160	 train_loss: 0.12 took: 4.66s
    Epoch 19, Iteration 176	 train_loss: 0.12 took: 5.11s
    Epoch 19, Iteration 192	 train_loss: 0.12 took: 5.57s
    Epoch 19, Iteration 208	 train_loss: 0.12 took: 6.05s
    Epoch 19, Iteration 224	 train_loss: 0.12 took: 6.51s
    Epoch 19, Iteration 240	 train_loss: 0.12 took: 6.97s
    Epoch 19, Iteration 256	 train_loss: 0.12 took: 7.45s
    Epoch 19, Iteration 272	 train_loss: 0.11 took: 7.93s
    Epoch 19, Iteration 288	 train_loss: 0.12 took: 8.40s
    Epoch 19, Iteration 304	 train_loss: 0.11 took: 8.87s
    Epoch 19, Iteration 320	 train_loss: 0.12 took: 9.33s
    Epoch 19, Iteration 336	 train_loss: 0.12 took: 9.81s
    Epoch 19, Iteration 352	 train_loss: 0.11 took: 10.29s
    Epoch 19, Iteration 368	 train_loss: 0.12 took: 10.74s
    Epoch 19, Iteration 384	 train_loss: 0.11 took: 11.22s
    Epoch 19, Iteration 400	 train_loss: 0.11 took: 11.71s
    Epoch 19, Iteration 416	 train_loss: 0.12 took: 12.21s
    Epoch 19, Iteration 432	 train_loss: 0.12 took: 12.67s
    Epoch 19, Iteration 448	 train_loss: 0.12 took: 13.15s
    Epoch 19, Iteration 464	 train_loss: 0.11 took: 13.63s

    Epoch [19/20], loss:0.1236
    Epoch 20, Iteration 16	 train_loss: 0.11 took: 0.49s
    Epoch 20, Iteration 32	 train_loss: 0.11 took: 0.94s
    Epoch 20, Iteration 48	 train_loss: 0.12 took: 1.42s
    Epoch 20, Iteration 64	 train_loss: 0.12 took: 1.90s
    Epoch 20, Iteration 80	 train_loss: 0.12 took: 2.38s
    Epoch 20, Iteration 96	 train_loss: 0.12 took: 2.87s
    Epoch 20, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 20, Iteration 128	 train_loss: 0.12 took: 3.83s
    Epoch 20, Iteration 144	 train_loss: 0.12 took: 4.32s
    Epoch 20, Iteration 160	 train_loss: 0.11 took: 4.77s
    Epoch 20, Iteration 176	 train_loss: 0.11 took: 5.22s
    Epoch 20, Iteration 192	 train_loss: 0.11 took: 5.72s
    Epoch 20, Iteration 208	 train_loss: 0.12 took: 6.22s
    Epoch 20, Iteration 224	 train_loss: 0.12 took: 6.70s
    Epoch 20, Iteration 240	 train_loss: 0.12 took: 7.16s
    Epoch 20, Iteration 256	 train_loss: 0.11 took: 7.62s
    Epoch 20, Iteration 272	 train_loss: 0.11 took: 8.06s
    Epoch 20, Iteration 288	 train_loss: 0.11 took: 8.52s
    Epoch 20, Iteration 304	 train_loss: 0.11 took: 8.96s
    Epoch 20, Iteration 320	 train_loss: 0.11 took: 9.43s
    Epoch 20, Iteration 336	 train_loss: 0.12 took: 9.90s
    Epoch 20, Iteration 352	 train_loss: 0.12 took: 10.40s
    Epoch 20, Iteration 368	 train_loss: 0.12 took: 10.87s
    Epoch 20, Iteration 384	 train_loss: 0.11 took: 11.35s
    Epoch 20, Iteration 400	 train_loss: 0.11 took: 11.83s
    Epoch 20, Iteration 416	 train_loss: 0.12 took: 12.30s
    Epoch 20, Iteration 432	 train_loss: 0.12 took: 12.77s
    Epoch 20, Iteration 448	 train_loss: 0.11 took: 13.25s
    Epoch 20, Iteration 464	 train_loss: 0.12 took: 13.71s

    Epoch [20/20], loss:0.1182

```
# Plots Loss over batches
plt.plot(batch_num,batch_loss)
plt.title('Loss of Autoencoder')
plt.legend(['train loss', 'validation loss'])
plt.xlabel('Batch number')
plt.ylabel('Loss')
plt.show()
```

### Anomaly or Not?
To determine, whether an image was an anomaly or not, it was necessary to look at the loss distribution of our non-anomalous dataset, and because it appeared like a relatively normal distribution, we decided that using a point's Z score was an anomaly. Z score can be calculated using:

\begin{align}
\ Z = \frac {x + \mu}{\sigma}
\end{align}

If a Z score of an image is found to be greater than 2.5, then the image is an outlier, which gave resonable accuracy.

```
all_losses = []

# Finds the loss for every image in the database to be plotted as a histogram
for i in range(len(dataset)):
    img, _ = dataset[i]
    img = Variable(img.unsqueeze(1)).cuda()

    output = model(img)

    all_losses.append(criterion(output, img).item())

# Plots Loss distribution
sns.distplot(all_losses, kde=False)
plt.title("Loss Distribution")
plt.xlabel("Loss Value")
plt.ylabel("Quanitity")
plt.show()

# Converts list of losses to Float Tensor type
all_losses = torch.FloatTensor(all_losses)
std = all_losses.std().item() # Finds standard deviation
mean = all_losses.mean().item() # Finds mean

# Finds if an image is an anomaly based on its z score value. 2.5 is the threshold.
def is_anomaly(value, z_thresh = 2.5):
    zscore = (value-mean)/std # Calculates z score for value

    if zscore > z_thresh:
        return True
    return False
```

### Detecting Anomalies
For detecting anomalies, anomalous images had to be created. Using the function below, images in the dataset are combined to create images which clearly deviated from the rest. The show_in_out function plots an image before and after being passed through the model. It also prints its loss and whether or not it is an anomaly.

```
# Combines images from dataset to create anomalies.
def make_anomaly(img1, img2, img_dim = (28,28)):
    # Matrix to store combined image
    output = np.ones((img_dim[0]*img_dim[1],1))

    # Reshapes images for consistency
    img1 = img1.reshape(img_dim[0]*img_dim[1], 1).numpy()
    img2 = img2.reshape(img_dim[0]*img_dim[1], 1).numpy()

    # Iterates through each pixel and selects largest
    for i in range(len(img1)):
        output[i] = max([img1[i], img2[i]])

    # Converts numpy array to tensor
    output = torch.from_numpy(output).type(torch.FloatTensor)

    return output

#Plots an input and output of the model
def show_in_out(img, model, loss_func):
    # Plots anomaly
    plt.title("Input")
    disp_image(img.reshape(28,28))

    # Passes image into Model
    img = img.reshape((1,1,28,28))
    img = img.to("cuda")
    output = model(Variable(img).cuda())
    loss = loss_func(output, img).item()

    encoded = to_img(output.cpu().data)

    # Plots output
    plt.title("Output")
    disp_image(encoded.reshape(28,28))

    print(F"LOSS: {loss}")
    print(F"Outlier: {is_anomaly(loss)}")
```

#### Non Anomalous Image
Below is an image from the dataset, non-anomalous, being passed through the model. The input and output are relatively similar, and the image is not classified as an anomaly.

```
# Pulls image from MNist dataset and displays it
img, label = dataset[1]

show_in_out(img, model, criterion)
```

#### Anomalous Image
Below is an anomalous image created by combining different non-anomalous images. After the image was passed through the model, differences were accentuated leading to a greater loss. As a result, the model was able to predict that the image was an anomaly.

```
img1, label = dataset[1]
img2, label = dataset[2]
img3, label = dataset[10]

# Combines images at index 1, 2 and 10 in the dataset
img = make_anomaly(make_anomaly(img1,img2), img3)

show_in_out(img, model, criterion)

```

#### Anomalous Image Inappropriately Classified.
Below is an image created by combining different non-anomalous images. Though the image is considered an anomaly, the model did not classify the image as an anomaly. This is likely because the image is close enough to a badly drawn zero.

```
img1, label = dataset[1]
img2, label = dataset[10]

# Combines images at index 1 and 10 in the dataset
img = make_anomaly(img1,img2)

show_in_out(img, model, criterion)
```

#### Large Anomaly Appropriately Classified.
Below is generated noise passed, which was passed through the model and classified. The input and output are very different, leading to a large loss and ultimately, a classification as an anomaly.

```
# Generates random noisy image
noisy = np.random.random((1,1,28,28))

# Converts noisy image to tensor
noisy = torch.from_numpy(noisy).type(torch.FloatTensor)

show_in_out(noisy, model, criterion)
```


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python
result = x.replace("DiClOrod  IFeNi  lTRicLOr oETaNo  ", "").lower().replace(" ", "").title()
if result == "Dichlorodiphenyltrichloroethane":
    print(result)
```

    Dichlorodiphenyltrichloroethane

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
A, B, C, D = "word", 15, 3.14, "list"
print(f'{A} {B} {C} {D}')
```

    word 15 3.14 list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
def bmi(response = 0):
    response = input('Enter your weight and height')
    response1 = list(response.split())
    w = response1[0]
    h = response1[1]
    if len(response1) > 2 or not str(w).isdigit() or not str(h).isdigit():
        print('Invlid input')
        bmi()
        return
    else:
        BMI = float(w)/((float(h)/100)**2)
        print('Your body mass index (BMI) is {}'.format(BMI))
```

```python
bmi()
```

    Enter your weight and height 62 120

    Your body mass index (BMI) is 43.05555555555556 - Obesity class III

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
def bmi(response = 0):
    response = input('Enter your weight and height')
    response1 = list(response.split())
    w = response1[0]
    h = response1[1]
    if len(response1) > 2 or not str(w).isdigit() or not str(h).isdigit():
        print('Invlid input')
        bmi()
        return
    else:
        BMI = float(w)/((float(h)/100)**2)
        if BMI < 18.5:
            print('Your body mass index (BMI) is {} - underweight'.format(BMI))
        elif BMI >= 18.5 and  BMI <= 24.9:
            print('Your body mass index (BMI) is {} - Normal weight'.format(BMI))
        elif BMI >= 25.0 and  BMI <= 29.9:
            print('Your body mass index (BMI) is {} - Pre-obesity'.format(BMI))
        elif BMI >= 30.0 and  BMI <= 34.9:
            print('Your body mass index (BMI) is {} - Obesity class I'.format(BMI))
        elif BMI >= 35.0 and  BMI <= 39.9:
            print('Your body mass index (BMI) is {} - Obesity class II'.format(BMI))
        elif BMI >= 40.0:
            print('Your body mass index (BMI) is {} - Obesity class III'.format(BMI))
```

```python
bmi()
```

    Enter your weight and height 45 175

    Your body mass index (BMI) is 14.693877551020408 - underweight

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
def factorial(x = 0):
    x = int(input('Enter a number'))
    y = 1
    for i in range(2, x + 1):
        y *= i
    return y
```

```python
factorial()
```

    Enter a number 6

    720

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
def wow(my_number=0):
    y = 0
    my_number = int(input('Enter a number'))
    while my_number != -1:
        y += my_number
        my_number = int(input('Enter a number'))
    print(y)
```

```python
wow()
```

    Enter a number 5
    Enter a number 4
    Enter a number -1

    9

```python
## that is probably NOT what the task is, but in any case...
def aha(summa=0):
    my_number = int(input('Enter a number: '))
    all_numbers = [*range(-1, my_number+1)]
    summa = sum(all_numbers[1:])
    print(all_numbers)
    print(summa)
```

```python
aha()
```

    Enter a number:  13

    [-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
    91

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python

name, working_hours, salary_per_hour = 'John', 40, 15.56
salary = working_hours * salary_per_hour
salary_decimal = "{:.2f}".format(salary)
print(salary_decimal)
```

    622.40

```python
data = input ('Enter name, amount_of_hours_worked, salary_ per_hour')
data_1 = data.replace(" ", "")
data_2 = data_1.split(',')
print(data_2)
name = data_2[0]
working_hours = float(data_2[1])
salary_per_hour = float(data_2[2])
salary = working_hours * salary_per_hour
salary_decimal = "{:.2f}".format(salary)
print(salary_decimal)
```

    Enter name, amount_of_hours_worked, salary_ per_hour peter, 30    , 15

    ['peter', '30', '15']
    450.00

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate items a, b, c, d and e:

```python
A = float(input('Enter 1st floating point'))
B = float(input('Enter 2nd floating point'))
C = float(input('Enter 3rd floating point'))
print(A, B, C)
```

    Enter 1st floating point 13
    Enter 2nd floating point 12
    Enter 3rd floating point 23

    13.0 12.0 23.0

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
s = (A * C) / 2
print(s)
```

    149.5

    b) the area of the circle of radius C. (pi = 3.14159)

```python
pi = 3.14159
surface_circle = pi * C **2
print(surface_circle)
```

    38.4844775

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
area_tra = ((A + B) * C) / 2
print(area_tra)
```

    17.5

    d) the area of the square that has side B.

```python
area_square = B * B
print(area_square)
```

    30.25

    e) the area of the rectangle that has sides A and B.

```python
area_rectangle = A * B
print(area_rectangle)
```

    24.75

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
### That is too much math for me, sorry.
```

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
locus1 = input('enter lat and lon of place 1: ')
locus2 = input('enter lat and lon of place 2: ')
coordinates_1 = locus1.split(' ')
coordinates_2 = locus2.split(' ')
coordinates1_float = [float(i) for i in coordinates_1]
coordinates2_float = [float(i) for i in coordinates_2]

from math import sin, cos, sqrt, atan2, radians

R = 6373.0 # radius of earth in km

lat_1 = radians(coordinates1_float[0])
lon_1 = radians(coordinates1_float[1])
lat_2 = radians(coordinates2_float[0])
lon_2 = radians(coordinates2_float[1])

dlon = lon_2 - lon_1
dlat = lat_2 - lat_1

a = sin(dlat / 2)**2 + cos(lat_1) * cos(lat_2) * sin(dlon / 2)**2
c = 2 * atan2(sqrt(a), sqrt(1 - a))

distance = R * c

print("Result:", distance)
```

    enter lat and lon of place 1 42 23
    enter lat and lon of place 2 48 16

    42 23
    Result: 864.4150003885633

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python

```

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)

```

      Country
      Country calling code

      0
      Austria
      43

      1
      Belgium
      32

      2
      Bulgaria
      359

      3
      Croatia
      385

      4
      Cyprus
      357

      5
      Czech Republic
      420

      6
      Denmark
      45

      7
      Estonia
      372

      8
      Finland
      358

      9
      France
      33

      10
      Germany
      49

      11
      Greece
      30

      12
      Hungary
      36

      13
      Iceland
      354

      14
      Ireland
      353

      15
      Italy
      39

      16
      Latvia
      371

      17
      Liechtenstein
      423

      18
      Lithuania
      370

      19
      Luxembourg
      352

```python
codes = {'Austria' : 43, 'Belgium' : 32, 'Bulgaria' : 359, 'Croatia' : 385, 'Cyprus' : 357, 'Czech' : 420, 'Denmark' : 45}
response = int(input('What is your code?'))
for key, value in codes.items():
    if value == response:
        print(key)
```

    What is your code? 43

    Austria

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
def numbers(n1 = 0):
    n1 = input('Enter a number')
    n2 = input('Enter a number')
    n3 = input('Enter a number')
    n4 = input('Enter a number')
    n5 = input('Enter a number')
    n6 = input('Enter a number')
    response1 = [n1, n2, n3, n4, n5, n6]
    response2 = [i for i in response1 if i.isdigit()]
    import numpy as np
    x = np.array(response1)
    y = x.astype(np.float)
    z = [k for k in y if k>0]
    w = sum(y) / len(y)
    j = sum(z) / len(z)
    print('All positive values:', z)
    print('The average of all values is: ', w)
    print('The average of all positive values is: ', j)
```

```python
numbers()
```

    Enter a number -96
    Enter a number -20
    Enter a number 4
    Enter a number 2
    Enter a number 3
    Enter a number 1

    All positive values: [4.0, 2.0, 3.0, 1.0]
    The average of all values is:  -17.666666666666668
    The average of all positive values is:  2.5

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python

```

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
def number(x=0):
    x = input('Enter a number')
    if len(x.split())!=1 or not x.isdigit:
        print('Bad input')
        number()
        return
    else:
        y = int(x)
        print('Your number is:', y)
        if (y % 2) == 0 and y !=0 and y < 0:
            print('even negative')
        elif (y % 2) == 0 and y !=0 and y>0:
            print('even negative')
        elif (y % 2) != 0 and y !=0 and y < 0:
            print('odd negative')
        elif (y % 2) !=0  and y !=0 and y > 0:
            print('odd positive')
        elif (y == 0):
            print('null')
```

```python
number()
```

    Enter a number -5

    Your number is: -5
    odd negative

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python

```