# SageMaker Factorization Machine(FM)으로 추천 시스템 구축하기

*본 노트북 예제는 AWS 머신 러닝 블로그에 기고된 글들에 기반하여 SageMaker의 Factorization Machine(FM)으로 추천 시스템을 구축하는 방법을 설명합니다.*

References
- [Build a movie recommender with factorization machines on Amazon SageMaker](https://aws.amazon.com/ko/blogs/machine-learning/build-a-movie-recommender-with-factorization-machines-on-amazon-sagemaker/)
- [Amazon SageMaker Factorization Machines 알고리즘을 확장하여 추천 시스템 구현하기](https://aws.amazon.com/ko/blogs/korea/extending-amazon-sagemaker-factorization-machines-algorithm-to-predict-top-x-recommendations/)
- [Factorization Machine 논문](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)

## 1. Factorization Machine
---

### 개요

일반적인 추천 문제들은 user가 행, item이 열, rating이 값으로 이루어진 행렬을 데이터셋으로 하여 Matrix Factorization 기법을 활용하는데, real-world의 다양한 메타데이터 피처(feature)들을 그대로 적용하기에는 어려움이 있습니다. Factoriztion Machine(이하 FM) 알고리즘은 Matrix Factorization의 개념을 확장하여 메타데이터 피처들을 같이 고려하고 피처 간의 상호 관계(interaction)를 선형 계산 복잡도로 자동으로 모델링할 수 있기에, 피처 엔지니어링에 들어가는 노력을 크게 줄일 수 있습니다.

### 설명

다양한 메타데이터 피처를 고려하기 위해 아래 그림처럼 user와 item을 원-핫 인코딩으로 변환하고 추가 피처들을 그대로 concatenate하여 `f(user, item, additional features) = rating` 형태의 선형 회귀(Linear Regression) 문제로 변환하여 풀 수 있습니다.

하지만, 추천 문제를 선형 회귀로만 풀려고 하면 피처 간의 상호 관계를 고려할 수 없기에 아래 수식처럼 피처 간의 상호 관계를 모델링하는 항을 추가하여 다항 회귀(Polynomial Regression)로 변환해야 합니다.

$$
\begin{align} \hat{y}(\mathbf{x}) = w_{0} + \sum_{i=1}^{d} w_{i} x_{i} + \sum_{i=1}^d \sum_{j=i+1}^d x_{i} x_{j} w_{ij}, \;\; x \in \mathbb{R}^d \tag {1}
\end{align}
$$
$d$는 피처 갯수로, $x \in \mathbb{R}^d$는 단일 샘플의 피처 벡터를 나타냅니다.

하지만 대부분의 추천 시스템 데이터셋은 희소하기에(sparse) cold-start 문제가 있으며, 추가적으로 고려해야 하는 피처들이 많아질 수록 계산이 매우 복잡해집니다. (예: user가 6만명, item 갯수가 5천개, 추가 피처가 5천개일 경우 70,000x70,000 행렬을 예측해야 합니다.)

FM은 이러한 문제들을 행렬 분해 기법을 활용하여 feature 쌍(예: user, item) 간의 상호 관계를 내적(dot product)으로 변환하고
수식을 재구성하여 계산 복잡도를 $O(kd^2)$에서 $O(kd)$로 감소시켰습니다. (수식 (2)에서 추가적인 계산을 거치면 계산 복잡도를 선형으로 감소할 수 있습니다. 자세한 내용은 논문을 참조하세요.)

$$
\begin{align}
\hat{y}(\mathbf{x}) = w_{0} + \sum_{i=1}^{d} w_i x_i + \sum_{i=1}^d\sum_{j=i+1}^d  x_{i} x_{j} \langle\mathbf{v}_i, \mathbf{v}_j\rangle \tag{2}
\end{align}
$$

$$
\begin{align}
\langle \textbf{v}_i , \textbf{v}_{j} \rangle = \sum_{f=1}^k v_{i,f} v_{j,f},\; k: \text{dimension of latent feature}  \tag{3}
\end{align}
$$

위의 모델을 2-way(degree = 2) FM이라고 하며, 이를 일반화한 d-way FM도 있지만, 보통 2-way FM를 많이 사용합니다. SageMaker의 FM 또한 2-way FM입니다.

FM이 훈련하는 파라메터 튜플은 ($w_{0}, \mathbf{w}, \mathbf{V}$) 이며, 의미는 아래와 같습니다.
- $w_{0} \in \mathbb{R}$: global bias
- $\mathbf{w} \in \mathbb{R}^d$: 피처 벡터 $\mathbf{x}_i$의 가중치
- $\mathbf{V} \in \mathbb{R}^{n \times k}$: 피처 임베딩 행렬로 i번째 행은 $\mathbf{v}_i$

FM은 위의 수식에서 알 수 있듯이 closed form이며 시간 복잡도가 선형이기 때문에, 다수의 user & item과 메타데이터들이 많은 추천 문제에 적합합니다.
훈련 방법은 대표적으로 Gradient Descent, ALS(Alternating Least Square), MCMC(Markov Chain Monte Carlo)가 있으며, AWS에서는 이 중 딥러닝 아키텍처에 기반한 Gradient Descent를 MXNet 프레임워크를 이용하여 훈련합니다.

## 2. MovieLens 데이터셋으로 FM 모델 훈련 및 배포하기
---

딥러닝의 Hello World가 MNIST 데이터셋이라면, 추천 시스템의 Hello World는 MovieLens 데이터셋입니다.
이 데이터셋은 여러 크기로 제공되며, 본 예제에서는 943명의 사용자와 1,682개의 영화에 대해 10만개의 등급이 부여된 ml100k를 사용합니다.

```python
import sagemaker
import sagemaker.amazon.common as smac
from sagemaker import get_execution_role
from sagemaker.predictor import json_deserializer
from sagemaker.amazon.amazon_estimator import get_image_uri
import numpy as np
from scipy.sparse import lil_matrix
import pandas as pd
import boto3, io, os, csv, json
```

### MovieLens 데이터셋 다운로드

```python
!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip
!unzip -o ml-100k.zip
```

    --2020-03-12 23:40:30--  http://files.grouplens.org/datasets/movielens/ml-100k.zip
    Resolving files.grouplens.org (files.grouplens.org)... 128.101.65.152
    Connecting to files.grouplens.org (files.grouplens.org)|128.101.65.152|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 4924029 (4.7M) [application/zip]
    Saving to: ‘ml-100k.zip’

    ml-100k.zip         100%[===================>]   4.70M  14.6MB/s    in 0.3s

    2020-03-12 23:40:31 (14.6 MB/s) - ‘ml-100k.zip’ saved [4924029/4924029]

    Archive:  ml-100k.zip
       creating: ml-100k/
      inflating: ml-100k/allbut.pl
      inflating: ml-100k/mku.sh
      inflating: ml-100k/README
      inflating: ml-100k/u.data
      inflating: ml-100k/u.genre
      inflating: ml-100k/u.info
      inflating: ml-100k/u.item
      inflating: ml-100k/u.occupation
      inflating: ml-100k/u.user
      inflating: ml-100k/u1.base
      inflating: ml-100k/u1.test
      inflating: ml-100k/u2.base
      inflating: ml-100k/u2.test
      inflating: ml-100k/u3.base
      inflating: ml-100k/u3.test
      inflating: ml-100k/u4.base
      inflating: ml-100k/u4.test
      inflating: ml-100k/u5.base
      inflating: ml-100k/u5.test
      inflating: ml-100k/ua.base
      inflating: ml-100k/ua.test
      inflating: ml-100k/ub.base
      inflating: ml-100k/ub.test

### 데이터 셔플링

```python
!shuf ml-100k/ua.base -o ml-100k/ua.base.shuffled
```

### 훈련 데이터 로드

```python
user_movie_ratings_train = pd.read_csv('ml-100k/ua.base.shuffled', sep='\t', index_col=False,
                 names=['user_id' , 'movie_id' , 'rating'])
user_movie_ratings_train.head(5)
```

      user_id
      movie_id
      rating

      0
      416
      762
      3

      1
      574
      332
      3

      2
      854
      483
      4

      3
      280
      1217
      5

      4
      476
      210
      4

### 테스트 데이터 로드

```python
user_movie_ratings_test = pd.read_csv('ml-100k/ua.test', sep='\t', index_col=False,
                 names=['user_id' , 'movie_id' , 'rating'])
user_movie_ratings_test.head(5)
```

      user_id
      movie_id
      rating

      0
      1
      20
      4

      1
      1
      33
      4

      2
      1
      61
      4

      3
      1
      117
      3

      4
      1
      155
      2

10만건의 등급 데이터가 왜 희소한지 궁금하실 수 있습니다. 943명의 사용자와 1682개의 영화를 모두 고려하면 가능한 rating의 총 개수는
943 * 1,682 = 1,586,126개로 이 중 6.3%의 등급만 보유하게 됩니다.

```python
nb_users = user_movie_ratings_train['user_id'].max()
nb_movies = user_movie_ratings_train['movie_id'].max()
nb_features = nb_users + nb_movies
total_ratings = nb_users * nb_movies

nb_ratings_test = len(user_movie_ratings_test.index)
nb_ratings_train = len(user_movie_ratings_train.index)

print("# of users: {}".format(nb_users))
print("# of movies: {}".format(nb_movies))
print("Training Count: {}".format(nb_ratings_train))
print("Test Count: {}".format(nb_ratings_test))
print("Features (# of users + # of movies): {}".format(nb_features))
print("Sparsity: {}%".format(((nb_ratings_test+nb_ratings_train)/total_ratings)*100))
```

    # of users: 943
    # of movies: 1682
    Training Count: 90570
    Test Count: 9430
    Features (# of users + # of movies): 2625
    Sparsity: 6.304669364224531%

### 원-핫 인코딩 희소 행렬 변환

FM의 입력 데이터 포맷인 원-핫 인코딩 희소 행렬로 변환하겠습니다. 물론 희소 행렬이 아닌 밀집(dense) 행렬도 가능하지만, 데이터가 많아질수록 계산 속도가 느려지므로, 희소 행렬을 추천합니다.

참고로, MovieLens 데이터셋은 별도의 메타데이터 피처가 존재하지 않아 943명의 사용자와 1,682개 영화에 대해서만 원-핫 인코딩 변환을 수행하므로 변환 후 피처의 차원은 943+1,682=2,625입니다.

또한, 본 예시에서는 rating이 4 이상인 영화들에 대한 이진 분류 문제로 간소화합니다. (즉, rating이 4 이상일 경우 $y = 1$, 4 미만일 경우 $y = 0$ 입니다.)

아래 셀은 약 20초 소요되며, 변환 후 데이터셋의 차원은 rating 개수 x 피쳐 개수 입니다.

```python
%%time
def loadDataset(df, lines, columns):

    # 피처는 원-핫 인코딩 희소 행렬로 변환합니다.
    X = lil_matrix((lines, columns)).astype('float32')
    Y = []
    line = 0
    for line, (index, row) in enumerate(df.iterrows()):
            X[line,row['user_id']-1] = 1
            X[line, nb_users+(row['movie_id']-1)] = 1

            if int(row['rating']) >= 4:
                Y.append(1)
            else:
                Y.append(0)

    Y = np.array(Y).astype('float32')
    return X,Y

X_train, Y_train = loadDataset(user_movie_ratings_train, nb_ratings_train, nb_features)
X_test, Y_test = loadDataset(user_movie_ratings_test, nb_ratings_test, nb_features)
```

    CPU times: user 17.6 s, sys: 33.4 ms, total: 17.6 s
    Wall time: 17.6 s

```python
print(X_train.shape)
print(Y_train.shape)
assert X_train.shape == (nb_ratings_train, nb_features)
assert Y_train.shape == (nb_ratings_train, )
zero_labels = np.count_nonzero(Y_train)
print("Training labels: {} zeros, {} ones".format(zero_labels, nb_ratings_train-zero_labels))

print(X_test.shape)
print(Y_test.shape)
assert X_test.shape  == (nb_ratings_test, nb_features)
assert Y_test.shape  == (nb_ratings_test, )
zero_labels = np.count_nonzero(Y_test)
print("Test labels: {} zeros, {} ones".format(zero_labels, nb_ratings_test-zero_labels))
```

    (90570, 2625)
    (90570,)
    Training labels: 49906 zeros, 40664 ones
    (9430, 2625)
    (9430,)
    Test labels: 5469 zeros, 3961 ones

### Protobuf 포맷 변환 후 S3에 저장

```python
import sagemaker
bucket = sagemaker.Session().default_bucket()
#bucket = '[YOUR-BUCKET]'
prefix = 'fm-hol'

if bucket.strip() == '':
    raise RuntimeError("bucket name is empty.")

train_key      = 'train.protobuf'
train_prefix   = '{}/{}'.format(prefix, 'train')

test_key       = 'test.protobuf'
test_prefix    = '{}/{}'.format(prefix, 'test')

output_prefix  = 's3://{}/{}/output'.format(bucket, prefix)
```

아래 셀은 약 15초 소요됩니다.

```python
%%time
def writeDatasetToProtobuf(X, bucket, prefix, key, d_type, Y=None):
    buf = io.BytesIO()
    if d_type == "sparse":
        smac.write_spmatrix_to_sparse_tensor(buf, X, labels=Y)
    else:
        smac.write_numpy_to_dense_tensor(buf, X, labels=Y)

    buf.seek(0)
    obj = '{}/{}'.format(prefix, key)
    boto3.resource('s3').Bucket(bucket).Object(obj).upload_fileobj(buf)
    return 's3://{}/{}'.format(bucket,obj)

fm_train_data_path = writeDatasetToProtobuf(X_train, bucket, train_prefix, train_key, "sparse", Y_train)
fm_test_data_path  = writeDatasetToProtobuf(X_test, bucket, test_prefix, test_key, "sparse", Y_test)

print("Training data S3 path: ", fm_train_data_path)
print("Test data S3 path: ", fm_test_data_path)
print("FM model output S3 path: ", output_prefix)
```

    Training data S3 path:  s3://sagemaker-us-east-1-143656149352/fm-hol/train/train.protobuf
    Test data S3 path:  s3://sagemaker-us-east-1-143656149352/fm-hol/test/test.protobuf
    FM model output S3 path:  s3://sagemaker-us-east-1-143656149352/fm-hol/output
    CPU times: user 14.6 s, sys: 202 ms, total: 14.8 s
    Wall time: 14.9 s

### 훈련

본 핸즈온은 하이퍼파라메터 튜닝 없이 휴리스틱한 하이퍼파라메터들을 사용합니다.

- `feature_dim`: 피처 개수로 본 핸즈온에서는 2,625으로 설정해야 합니다.
- `mini_batch_size`: 본 핸즈온에서는 1,000으로 설정합니다.
- `num_factors`: latent factor 차원으로 본 핸즈온에서는 64차원으로 설정합니다.
- `epochs`: 본 핸즈온에서는 100 에폭으로 설정합니다.

```python
instance_type_training = 'ml.c4.xlarge'
fm = sagemaker.estimator.Estimator(get_image_uri(boto3.Session().region_name, "factorization-machines"),
                                   get_execution_role(),
                                   train_instance_count=1,
                                   train_instance_type=instance_type_training,
                                   output_path=output_prefix,
                                   sagemaker_session=sagemaker.Session())

fm.set_hyperparameters(feature_dim=nb_features,
                      predictor_type='binary_classifier',
                      mini_batch_size=1000,
                      num_factors=64,
                      epochs=100)
```

이제 훈련을 시작하기 위한 모든 준비가 완료되었으며, 여러분께서는 `fit()` 메소드만 호출하면 됩니다.

훈련은 약 4분에서 5분이 소요되며(순수 훈련에 소요되는 시간은 훨씬 짧지만, 훈련 인스턴스를 프로비저닝하는 시간이 고정적으로 소요됩니다), 검증 데이터셋에 대한 accuracy는 약 70%에 F1 스코어는 약 0.73~0.74입니다. (아래 output 메세지 참조)

```
[03/12/2020 09:35:42 INFO 139967441712960] #test_score (algo-1) : ('binary_classification_accuracy', 0.6928950159066808)
[03/12/2020 09:35:42 INFO 139967441712960] #test_score (algo-1) : ('binary_classification_cross_entropy', 0.5799107152103493)
[03/12/2020 09:35:42 INFO 139967441712960] #test_score (algo-1) : ('binary_f_1.000', 0.7331859222406486)
```

```python
%%time
fm.fit({'train': fm_train_data_path, 'test': fm_test_data_path})
```

    2020-03-12 23:41:11 Starting - Starting the training job...
    2020-03-12 23:41:12 Starting - Launching requested ML instances......
    2020-03-12 23:42:17 Starting - Preparing the instances for training......
    2020-03-12 23:43:18 Downloading - Downloading input data...
    2020-03-12 23:43:57 Training - Training image download completed. Training in progress.Docker entrypoint called with argument(s): train
    Running default environment configuration script
    /opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.
      from numpy.testing import nosetester
    [03/12/2020 23:44:00 INFO 140601087346496] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}
    [03/12/2020 23:44:00 INFO 140601087346496] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'2625', u'mini_batch_size': u'1000', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}
    [03/12/2020 23:44:00 INFO 140601087346496] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'2625', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}
    [03/12/2020 23:44:00 WARNING 140601087346496] Loggers have already been setup.
    Process 1 is a worker.
    [03/12/2020 23:44:00 INFO 140601087346496] Using default worker.
    [2020-03-12 23:44:00.857] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.
    [2020-03-12 23:44:00.865] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 0, "duration": 12, "num_examples": 1, "num_bytes": 64000}
    [03/12/2020 23:44:00 INFO 140601087346496] nvidia-smi took: 0.0252029895782 secs to identify 0 gpus
    [03/12/2020 23:44:00 INFO 140601087346496] Number of GPUs being used: 0
    [03/12/2020 23:44:00 INFO 140601087346496] [Sparse network] Building a sparse network.
    [03/12/2020 23:44:00 INFO 140601087346496] Create Store: local
    #metrics {"Metrics": {"initialize.time": {"count": 1, "max": 39.54601287841797, "sum": 39.54601287841797, "min": 39.54601287841797}}, "EndTime": 1584056640.902592, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056640.852945}

    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Total Records Seen": {"count": 1, "max": 1000, "sum": 1000.0, "min": 1000}, "Max Records Seen Between Resets": {"count": 1, "max": 1000, "sum": 1000.0, "min": 1000}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1584056640.902793, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056640.902739}

    [23:44:00] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.202804.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.
    [23:44:00] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.202804.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.543
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692450073242
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_f_1.000 <score>=0.692255892256
    [2020-03-12 23:44:01.636] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 2, "duration": 700, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.548395604396
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.684067218487
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=0, train binary_f_1.000 <score>=0.708199608055
    #metrics {"Metrics": {"epochs": {"count": 1, "max": 100, "sum": 100.0, "min": 100}, "update.time": {"count": 1, "max": 733.983039855957, "sum": 733.983039855957, "min": 733.983039855957}}, "EndTime": 1584056641.637007, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056640.902666}

    [03/12/2020 23:44:01 INFO 140601087346496] #progress_metric: host=algo-1, completed 1 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 92, "sum": 92.0, "min": 92}, "Total Records Seen": {"count": 1, "max": 91570, "sum": 91570.0, "min": 91570}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 2, "sum": 2.0, "min": 2}}, "EndTime": 1584056641.637216, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 0}, "StartTime": 1584056640.902993}

    [03/12/2020 23:44:01 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=123337.249989 records/second
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.549
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.678646972656
    [03/12/2020 23:44:01 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_f_1.000 <score>=0.708844415752
    [2020-03-12 23:44:02.387] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 4, "duration": 748, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.561087912088
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.67371950858
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=1, train binary_f_1.000 <score>=0.708236239454
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 751.4450550079346, "sum": 751.4450550079346, "min": 751.4450550079346}}, "EndTime": 1584056642.389225, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056641.637081}

    [03/12/2020 23:44:02 INFO 140601087346496] #progress_metric: host=algo-1, completed 2 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 183, "sum": 183.0, "min": 183}, "Total Records Seen": {"count": 1, "max": 182140, "sum": 182140.0, "min": 182140}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 3, "sum": 3.0, "min": 3}}, "EndTime": 1584056642.389826, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 1}, "StartTime": 1584056641.637716}

    [03/12/2020 23:44:02 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=120397.055309 records/second
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.588
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.670379760742
    [03/12/2020 23:44:02 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_f_1.000 <score>=0.721621621622
    [2020-03-12 23:44:03.116] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 6, "duration": 723, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.594098901099
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.66426916638
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=2, train binary_f_1.000 <score>=0.723249941933
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 726.2499332427979, "sum": 726.2499332427979, "min": 726.2499332427979}}, "EndTime": 1584056643.116806, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056642.389538}

    [03/12/2020 23:44:03 INFO 140601087346496] #progress_metric: host=algo-1, completed 3 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 274, "sum": 274.0, "min": 274}, "Total Records Seen": {"count": 1, "max": 272710, "sum": 272710.0, "min": 272710}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 4, "sum": 4.0, "min": 4}}, "EndTime": 1584056643.117039, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 2}, "StartTime": 1584056642.390525}

    [03/12/2020 23:44:03 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=124640.146309 records/second
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.628
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.662557250977
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_f_1.000 <score>=0.73275862069
    [2020-03-12 23:44:03.823] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 8, "duration": 703, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, train binary_classification_accuracy <score>=0.630384615385
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, train binary_classification_cross_entropy <loss>=0.655455263913
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=3, train binary_f_1.000 <score>=0.737859386325
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 706.2439918518066, "sum": 706.2439918518066, "min": 706.2439918518066}}, "EndTime": 1584056643.823933, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056643.116882}

    [03/12/2020 23:44:03 INFO 140601087346496] #progress_metric: host=algo-1, completed 4 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 365, "sum": 365.0, "min": 365}, "Total Records Seen": {"count": 1, "max": 363280, "sum": 363280.0, "min": 363280}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 5, "sum": 5.0, "min": 5}}, "EndTime": 1584056643.824186, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 3}, "StartTime": 1584056643.117655}

    [03/12/2020 23:44:03 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=128163.96259 records/second
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_accuracy <score>=0.656
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_cross_entropy <loss>=0.655179931641
    [03/12/2020 23:44:03 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_f_1.000 <score>=0.739788199697
    [2020-03-12 23:44:04.421] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 10, "duration": 595, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, train binary_classification_accuracy <score>=0.655846153846
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, train binary_classification_cross_entropy <loss>=0.647187304151
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=4, train binary_f_1.000 <score>=0.74730913844
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 597.649097442627, "sum": 597.649097442627, "min": 597.649097442627}}, "EndTime": 1584056644.422451, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056643.824014}

    [03/12/2020 23:44:04 INFO 140601087346496] #progress_metric: host=algo-1, completed 5 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 456, "sum": 456.0, "min": 456}, "Total Records Seen": {"count": 1, "max": 453850, "sum": 453850.0, "min": 453850}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 6, "sum": 6.0, "min": 6}}, "EndTime": 1584056644.422714, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 4}, "StartTime": 1584056643.824771}

    [03/12/2020 23:44:04 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151440.165043 records/second
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_accuracy <score>=0.666
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_cross_entropy <loss>=0.648295288086
    [03/12/2020 23:44:04 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_f_1.000 <score>=0.738654147105
    [2020-03-12 23:44:05.024] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 12, "duration": 599, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, train binary_classification_accuracy <score>=0.673934065934
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, train binary_classification_cross_entropy <loss>=0.639498197115
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=5, train binary_f_1.000 <score>=0.753742987086
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 602.1349430084229, "sum": 602.1349430084229, "min": 602.1349430084229}}, "EndTime": 1584056645.025437, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056644.422553}

    [03/12/2020 23:44:05 INFO 140601087346496] #progress_metric: host=algo-1, completed 6 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 547, "sum": 547.0, "min": 547}, "Total Records Seen": {"count": 1, "max": 544420, "sum": 544420.0, "min": 544420}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 7, "sum": 7.0, "min": 7}}, "EndTime": 1584056645.02569, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 5}, "StartTime": 1584056644.423273}

    [03/12/2020 23:44:05 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150315.155038 records/second
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_accuracy <score>=0.67
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_cross_entropy <loss>=0.641921508789
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_f_1.000 <score>=0.735576923077
    [2020-03-12 23:44:05.637] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 14, "duration": 609, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, train binary_classification_accuracy <score>=0.687296703297
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, train binary_classification_cross_entropy <loss>=0.632389056866
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=6, train binary_f_1.000 <score>=0.758569198398
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 611.8409633636475, "sum": 611.8409633636475, "min": 611.8409633636475}}, "EndTime": 1584056645.63814, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056645.02552}

    [03/12/2020 23:44:05 INFO 140601087346496] #progress_metric: host=algo-1, completed 7 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 638, "sum": 638.0, "min": 638}, "Total Records Seen": {"count": 1, "max": 634990, "sum": 634990.0, "min": 634990}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 8, "sum": 8.0, "min": 8}}, "EndTime": 1584056645.638404, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 6}, "StartTime": 1584056645.026267}

    [03/12/2020 23:44:05 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=147925.655688 records/second
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_accuracy <score>=0.684
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_cross_entropy <loss>=0.636046020508
    [03/12/2020 23:44:05 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_f_1.000 <score>=0.741407528642
    [2020-03-12 23:44:06.226] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 16, "duration": 585, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, train binary_classification_accuracy <score>=0.697043956044
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, train binary_classification_cross_entropy <loss>=0.625836983817
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=7, train binary_f_1.000 <score>=0.761796141251
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 588.1578922271729, "sum": 588.1578922271729, "min": 588.1578922271729}}, "EndTime": 1584056646.227203, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056645.638225}

    [03/12/2020 23:44:06 INFO 140601087346496] #progress_metric: host=algo-1, completed 8 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 729, "sum": 729.0, "min": 729}, "Total Records Seen": {"count": 1, "max": 725560, "sum": 725560.0, "min": 725560}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 9, "sum": 9.0, "min": 9}}, "EndTime": 1584056646.227443, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 7}, "StartTime": 1584056645.639013}

    [03/12/2020 23:44:06 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153885.337791 records/second
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_accuracy <score>=0.687
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_cross_entropy <loss>=0.630640869141
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_f_1.000 <score>=0.741108354012
    [2020-03-12 23:44:06.818] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 18, "duration": 589, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, train binary_classification_accuracy <score>=0.703230769231
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, train binary_classification_cross_entropy <loss>=0.61980730128
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=8, train binary_f_1.000 <score>=0.76336680511
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 591.4878845214844, "sum": 591.4878845214844, "min": 591.4878845214844}}, "EndTime": 1584056646.819485, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056646.227285}

    [03/12/2020 23:44:06 INFO 140601087346496] #progress_metric: host=algo-1, completed 9 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 820, "sum": 820.0, "min": 820}, "Total Records Seen": {"count": 1, "max": 816130, "sum": 816130.0, "min": 816130}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 10, "sum": 10.0, "min": 10}}, "EndTime": 1584056646.819726, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 8}, "StartTime": 1584056646.227971}

    [03/12/2020 23:44:06 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153024.251363 records/second
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_accuracy <score>=0.696
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_cross_entropy <loss>=0.625671691895
    [03/12/2020 23:44:06 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_f_1.000 <score>=0.745393634841
    [2020-03-12 23:44:07.450] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 20, "duration": 628, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, train binary_classification_accuracy <score>=0.708406593407
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, train binary_classification_cross_entropy <loss>=0.614260453109
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=9, train binary_f_1.000 <score>=0.764829438196
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 630.7199001312256, "sum": 630.7199001312256, "min": 630.7199001312256}}, "EndTime": 1584056647.451027, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056646.819557}

    [03/12/2020 23:44:07 INFO 140601087346496] #progress_metric: host=algo-1, completed 10 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 911, "sum": 911.0, "min": 911}, "Total Records Seen": {"count": 1, "max": 906700, "sum": 906700.0, "min": 906700}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 11, "sum": 11.0, "min": 11}}, "EndTime": 1584056647.451288, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 9}, "StartTime": 1584056646.820276}

    [03/12/2020 23:44:07 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=143500.663443 records/second
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_classification_accuracy <score>=0.699
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_classification_cross_entropy <loss>=0.621101928711
    [03/12/2020 23:44:07 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_f_1.000 <score>=0.745131244708
    [2020-03-12 23:44:08.054] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 22, "duration": 600, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, train binary_classification_accuracy <score>=0.71210989011
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, train binary_classification_cross_entropy <loss>=0.609155807998
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=10, train binary_f_1.000 <score>=0.765607944887
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 602.72216796875, "sum": 602.72216796875, "min": 602.72216796875}}, "EndTime": 1584056648.054686, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056647.451113}

    [03/12/2020 23:44:08 INFO 140601087346496] #progress_metric: host=algo-1, completed 11 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1002, "sum": 1002.0, "min": 1002}, "Total Records Seen": {"count": 1, "max": 997270, "sum": 997270.0, "min": 997270}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 12, "sum": 12.0, "min": 12}}, "EndTime": 1584056648.054917, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 10}, "StartTime": 1584056647.451935}

    [03/12/2020 23:44:08 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150173.372786 records/second
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_classification_accuracy <score>=0.703
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_classification_cross_entropy <loss>=0.616895996094
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_f_1.000 <score>=0.745936698033
    [2020-03-12 23:44:08.658] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 24, "duration": 595, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, train binary_classification_accuracy <score>=0.715065934066
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, train binary_classification_cross_entropy <loss>=0.604453672304
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=11, train binary_f_1.000 <score>=0.76614205186
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 602.9999256134033, "sum": 602.9999256134033, "min": 602.9999256134033}}, "EndTime": 1584056648.658487, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056648.054763}

    [03/12/2020 23:44:08 INFO 140601087346496] #progress_metric: host=algo-1, completed 12 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1093, "sum": 1093.0, "min": 1093}, "Total Records Seen": {"count": 1, "max": 1087840, "sum": 1087840.0, "min": 1087840}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 13, "sum": 13.0, "min": 13}}, "EndTime": 1584056648.65872, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 11}, "StartTime": 1584056648.055461}

    [03/12/2020 23:44:08 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150105.191623 records/second
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_classification_accuracy <score>=0.704
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_classification_cross_entropy <loss>=0.613019897461
    [03/12/2020 23:44:08 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_f_1.000 <score>=0.746140651801
    [2020-03-12 23:44:09.240] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 26, "duration": 580, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, train binary_classification_accuracy <score>=0.717659340659
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, train binary_classification_cross_entropy <loss>=0.600116470505
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=12, train binary_f_1.000 <score>=0.766726287214
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.9258689880371, "sum": 581.9258689880371, "min": 581.9258689880371}}, "EndTime": 1584056649.241221, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056648.658555}

    [03/12/2020 23:44:09 INFO 140601087346496] #progress_metric: host=algo-1, completed 13 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1184, "sum": 1184.0, "min": 1184}, "Total Records Seen": {"count": 1, "max": 1178410, "sum": 1178410.0, "min": 1178410}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 14, "sum": 14.0, "min": 14}}, "EndTime": 1584056649.241422, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 12}, "StartTime": 1584056648.659268}

    [03/12/2020 23:44:09 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155549.032207 records/second
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_classification_accuracy <score>=0.704
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_classification_cross_entropy <loss>=0.609442260742
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_f_1.000 <score>=0.745704467354
    [2020-03-12 23:44:09.823] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 28, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, train binary_classification_accuracy <score>=0.719802197802
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, train binary_classification_cross_entropy <loss>=0.596109260308
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=13, train binary_f_1.000 <score>=0.767281821003
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.9811820983887, "sum": 581.9811820983887, "min": 581.9811820983887}}, "EndTime": 1584056649.824017, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056649.241283}

    [03/12/2020 23:44:09 INFO 140601087346496] #progress_metric: host=algo-1, completed 14 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1275, "sum": 1275.0, "min": 1275}, "Total Records Seen": {"count": 1, "max": 1268980, "sum": 1268980.0, "min": 1268980}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 15, "sum": 15.0, "min": 15}}, "EndTime": 1584056649.82427, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 13}, "StartTime": 1584056649.242002}

    [03/12/2020 23:44:09 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155511.398869 records/second
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_classification_accuracy <score>=0.707
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_classification_cross_entropy <loss>=0.606134277344
    [03/12/2020 23:44:09 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_f_1.000 <score>=0.747631352283
    [2020-03-12 23:44:10.414] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 30, "duration": 587, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, train binary_classification_accuracy <score>=0.721637362637
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, train binary_classification_cross_entropy <loss>=0.592400017707
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=14, train binary_f_1.000 <score>=0.767680102719
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 589.8468494415283, "sum": 589.8468494415283, "min": 589.8468494415283}}, "EndTime": 1584056650.414687, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056649.8241}

    [03/12/2020 23:44:10 INFO 140601087346496] #progress_metric: host=algo-1, completed 15 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1366, "sum": 1366.0, "min": 1366}, "Total Records Seen": {"count": 1, "max": 1359550, "sum": 1359550.0, "min": 1359550}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 16, "sum": 16.0, "min": 16}}, "EndTime": 1584056650.414911, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 14}, "StartTime": 1584056649.824813}

    [03/12/2020 23:44:10 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153454.113518 records/second
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_classification_accuracy <score>=0.708
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_classification_cross_entropy <loss>=0.603069824219
    [03/12/2020 23:44:10 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_f_1.000 <score>=0.746527777778
    [2020-03-12 23:44:11.016] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 32, "duration": 599, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, train binary_classification_accuracy <score>=0.72356043956
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, train binary_classification_cross_entropy <loss>=0.588959610824
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=15, train binary_f_1.000 <score>=0.768305487502
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 601.7978191375732, "sum": 601.7978191375732, "min": 601.7978191375732}}, "EndTime": 1584056651.017289, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056650.414755}

    [03/12/2020 23:44:11 INFO 140601087346496] #progress_metric: host=algo-1, completed 16 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1457, "sum": 1457.0, "min": 1457}, "Total Records Seen": {"count": 1, "max": 1450120, "sum": 1450120.0, "min": 1450120}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 17, "sum": 17.0, "min": 17}}, "EndTime": 1584056651.017522, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 15}, "StartTime": 1584056650.415464}

    [03/12/2020 23:44:11 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150405.557479 records/second
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_classification_accuracy <score>=0.71
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_classification_cross_entropy <loss>=0.600225585937
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_f_1.000 <score>=0.747826086957
    [2020-03-12 23:44:11.594] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 34, "duration": 575, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, train binary_classification_accuracy <score>=0.724637362637
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, train binary_classification_cross_entropy <loss>=0.585761714055
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=16, train binary_f_1.000 <score>=0.768406070353
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 577.0840644836426, "sum": 577.0840644836426, "min": 577.0840644836426}}, "EndTime": 1584056651.59516, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056651.017368}

    [03/12/2020 23:44:11 INFO 140601087346496] #progress_metric: host=algo-1, completed 17 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1548, "sum": 1548.0, "min": 1548}, "Total Records Seen": {"count": 1, "max": 1540690, "sum": 1540690.0, "min": 1540690}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 18, "sum": 18.0, "min": 18}}, "EndTime": 1584056651.59536, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 16}, "StartTime": 1584056651.018048}

    [03/12/2020 23:44:11 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156852.380268 records/second
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_classification_accuracy <score>=0.713
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_classification_cross_entropy <loss>=0.597580444336
    [03/12/2020 23:44:11 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_f_1.000 <score>=0.749782040105
    [2020-03-12 23:44:12.170] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 36, "duration": 573, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, train binary_classification_accuracy <score>=0.725659340659
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, train binary_classification_cross_entropy <loss>=0.582782614907
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=17, train binary_f_1.000 <score>=0.768415877404
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 575.477123260498, "sum": 575.477123260498, "min": 575.477123260498}}, "EndTime": 1584056652.171431, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056651.595229}

    [03/12/2020 23:44:12 INFO 140601087346496] #progress_metric: host=algo-1, completed 18 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1639, "sum": 1639.0, "min": 1639}, "Total Records Seen": {"count": 1, "max": 1631260, "sum": 1631260.0, "min": 1631260}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 19, "sum": 19.0, "min": 19}}, "EndTime": 1584056652.171677, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 17}, "StartTime": 1584056651.595921}

    [03/12/2020 23:44:12 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157269.599919 records/second
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_classification_accuracy <score>=0.715
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_classification_cross_entropy <loss>=0.595115600586
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_f_1.000 <score>=0.750219106047
    [2020-03-12 23:44:12.767] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 38, "duration": 593, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, train binary_classification_accuracy <score>=0.726527472527
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, train binary_classification_cross_entropy <loss>=0.580001071133
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=18, train binary_f_1.000 <score>=0.768437703545
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 595.9248542785645, "sum": 595.9248542785645, "min": 595.9248542785645}}, "EndTime": 1584056652.768195, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056652.171511}

    [03/12/2020 23:44:12 INFO 140601087346496] #progress_metric: host=algo-1, completed 19 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1730, "sum": 1730.0, "min": 1730}, "Total Records Seen": {"count": 1, "max": 1721830, "sum": 1721830.0, "min": 1721830}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 20, "sum": 20.0, "min": 20}}, "EndTime": 1584056652.768396, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 18}, "StartTime": 1584056652.172243}

    [03/12/2020 23:44:12 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151896.805497 records/second
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_classification_accuracy <score>=0.716
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_classification_cross_entropy <loss>=0.592814208984
    [03/12/2020 23:44:12 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_f_1.000 <score>=0.750877192982
    [2020-03-12 23:44:13.352] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 40, "duration": 582, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, train binary_classification_accuracy <score>=0.727120879121
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, train binary_classification_cross_entropy <loss>=0.577398022327
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=19, train binary_f_1.000 <score>=0.768289041505
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 584.2070579528809, "sum": 584.2070579528809, "min": 584.2070579528809}}, "EndTime": 1584056653.353177, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056652.768265}

    [03/12/2020 23:44:13 INFO 140601087346496] #progress_metric: host=algo-1, completed 20 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1821, "sum": 1821.0, "min": 1821}, "Total Records Seen": {"count": 1, "max": 1812400, "sum": 1812400.0, "min": 1812400}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 21, "sum": 21.0, "min": 21}}, "EndTime": 1584056653.353403, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 19}, "StartTime": 1584056652.768943}

    [03/12/2020 23:44:13 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154934.414503 records/second
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_classification_accuracy <score>=0.721
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_classification_cross_entropy <loss>=0.590661376953
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_f_1.000 <score>=0.755048287972
    [2020-03-12 23:44:13.925] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 42, "duration": 569, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, train binary_classification_accuracy <score>=0.727923076923
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, train binary_classification_cross_entropy <loss>=0.574956415533
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=20, train binary_f_1.000 <score>=0.768436509198
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 572.1209049224854, "sum": 572.1209049224854, "min": 572.1209049224854}}, "EndTime": 1584056653.926071, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056653.353243}

    [03/12/2020 23:44:13 INFO 140601087346496] #progress_metric: host=algo-1, completed 21 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 1912, "sum": 1912.0, "min": 1912}, "Total Records Seen": {"count": 1, "max": 1902970, "sum": 1902970.0, "min": 1902970}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 22, "sum": 22.0, "min": 22}}, "EndTime": 1584056653.926295, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 20}, "StartTime": 1584056653.353918}

    [03/12/2020 23:44:13 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=158203.050167 records/second
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_classification_accuracy <score>=0.723
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_classification_cross_entropy <loss>=0.588643554687
    [03/12/2020 23:44:13 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_f_1.000 <score>=0.756804214223
    [2020-03-12 23:44:14.516] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 44, "duration": 585, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, train binary_classification_accuracy <score>=0.728626373626
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, train binary_classification_cross_entropy <loss>=0.572661030193
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=21, train binary_f_1.000 <score>=0.768615251998
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 589.8439884185791, "sum": 589.8439884185791, "min": 589.8439884185791}}, "EndTime": 1584056654.51681, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056653.92614}

    [03/12/2020 23:44:14 INFO 140601087346496] #progress_metric: host=algo-1, completed 22 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2003, "sum": 2003.0, "min": 2003}, "Total Records Seen": {"count": 1, "max": 1993540, "sum": 1993540.0, "min": 1993540}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 23, "sum": 23.0, "min": 23}}, "EndTime": 1584056654.517058, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 21}, "StartTime": 1584056653.926931}

    [03/12/2020 23:44:14 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153433.597989 records/second
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_classification_accuracy <score>=0.723
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_classification_cross_entropy <loss>=0.586748901367
    [03/12/2020 23:44:14 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_f_1.000 <score>=0.756804214223
    [2020-03-12 23:44:15.116] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 46, "duration": 597, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, train binary_classification_accuracy <score>=0.729296703297
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, train binary_classification_cross_entropy <loss>=0.570498263516
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=22, train binary_f_1.000 <score>=0.768699179358
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 599.8380184173584, "sum": 599.8380184173584, "min": 599.8380184173584}}, "EndTime": 1584056655.117477, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056654.516888}

    [03/12/2020 23:44:15 INFO 140601087346496] #progress_metric: host=algo-1, completed 23 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2094, "sum": 2094.0, "min": 2094}, "Total Records Seen": {"count": 1, "max": 2084110, "sum": 2084110.0, "min": 2084110}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 24, "sum": 24.0, "min": 24}}, "EndTime": 1584056655.117671, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 22}, "StartTime": 1584056654.517611}

    [03/12/2020 23:44:15 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150908.947394 records/second
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_classification_accuracy <score>=0.723
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_classification_cross_entropy <loss>=0.584966430664
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_f_1.000 <score>=0.756804214223
    [2020-03-12 23:44:15.690] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 48, "duration": 571, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, train binary_classification_accuracy <score>=0.730142857143
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, train binary_classification_cross_entropy <loss>=0.568455958104
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=23, train binary_f_1.000 <score>=0.768960099352
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 573.462963104248, "sum": 573.462963104248, "min": 573.462963104248}}, "EndTime": 1584056655.691729, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056655.11754}

    [03/12/2020 23:44:15 INFO 140601087346496] #progress_metric: host=algo-1, completed 24 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2185, "sum": 2185.0, "min": 2185}, "Total Records Seen": {"count": 1, "max": 2174680, "sum": 2174680.0, "min": 2174680}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 25, "sum": 25.0, "min": 25}}, "EndTime": 1584056655.691954, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 23}, "StartTime": 1584056655.118232}

    [03/12/2020 23:44:15 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157830.363209 records/second
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_classification_accuracy <score>=0.725
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_classification_cross_entropy <loss>=0.583286621094
    [03/12/2020 23:44:15 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_f_1.000 <score>=0.758135444151
    [2020-03-12 23:44:16.267] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 50, "duration": 573, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, train binary_classification_accuracy <score>=0.730912087912
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, train binary_classification_cross_entropy <loss>=0.566523286589
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=24, train binary_f_1.000 <score>=0.769240917872
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 575.4449367523193, "sum": 575.4449367523193, "min": 575.4449367523193}}, "EndTime": 1584056656.267966, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056655.69179}

    [03/12/2020 23:44:16 INFO 140601087346496] #progress_metric: host=algo-1, completed 25 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2276, "sum": 2276.0, "min": 2276}, "Total Records Seen": {"count": 1, "max": 2265250, "sum": 2265250.0, "min": 2265250}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 26, "sum": 26.0, "min": 26}}, "EndTime": 1584056656.268168, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 24}, "StartTime": 1584056655.692494}

    [03/12/2020 23:44:16 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157300.728486 records/second
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_classification_accuracy <score>=0.725
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_classification_cross_entropy <loss>=0.581700927734
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_f_1.000 <score>=0.757709251101
    [2020-03-12 23:44:16.844] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 52, "duration": 574, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, train binary_classification_accuracy <score>=0.731483516484
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, train binary_classification_cross_entropy <loss>=0.564690573891
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=25, train binary_f_1.000 <score>=0.769344044102
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 576.1971473693848, "sum": 576.1971473693848, "min": 576.1971473693848}}, "EndTime": 1584056656.844955, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056656.268035}

    [03/12/2020 23:44:16 INFO 140601087346496] #progress_metric: host=algo-1, completed 26 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2367, "sum": 2367.0, "min": 2367}, "Total Records Seen": {"count": 1, "max": 2355820, "sum": 2355820.0, "min": 2355820}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 27, "sum": 27.0, "min": 27}}, "EndTime": 1584056656.845158, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 25}, "StartTime": 1584056656.268728}

    [03/12/2020 23:44:16 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157092.570975 records/second
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_classification_accuracy <score>=0.727
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_classification_cross_entropy <loss>=0.580201660156
    [03/12/2020 23:44:16 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_f_1.000 <score>=0.759471365639
    [2020-03-12 23:44:17.419] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 54, "duration": 572, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, train binary_classification_accuracy <score>=0.731967032967
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, train binary_classification_cross_entropy <loss>=0.562949179178
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=26, train binary_f_1.000 <score>=0.769463426622
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 574.131965637207, "sum": 574.131965637207, "min": 574.131965637207}}, "EndTime": 1584056657.419871, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056656.845028}

    [03/12/2020 23:44:17 INFO 140601087346496] #progress_metric: host=algo-1, completed 27 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2458, "sum": 2458.0, "min": 2458}, "Total Records Seen": {"count": 1, "max": 2446390, "sum": 2446390.0, "min": 2446390}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 28, "sum": 28.0, "min": 28}}, "EndTime": 1584056657.420128, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 26}, "StartTime": 1584056656.845713}

    [03/12/2020 23:44:17 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157638.070223 records/second
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_classification_accuracy <score>=0.726
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_classification_cross_entropy <loss>=0.578781616211
    [03/12/2020 23:44:17 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_f_1.000 <score>=0.758802816901
    [2020-03-12 23:44:18.049] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 56, "duration": 627, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, train binary_classification_accuracy <score>=0.732263736264
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, train binary_classification_cross_entropy <loss>=0.561291421408
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=27, train binary_f_1.000 <score>=0.769446231879
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 629.0180683135986, "sum": 629.0180683135986, "min": 629.0180683135986}}, "EndTime": 1584056658.049761, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056657.419944}

    [03/12/2020 23:44:18 INFO 140601087346496] #progress_metric: host=algo-1, completed 28 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2549, "sum": 2549.0, "min": 2549}, "Total Records Seen": {"count": 1, "max": 2536960, "sum": 2536960.0, "min": 2536960}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 29, "sum": 29.0, "min": 29}}, "EndTime": 1584056658.050008, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 27}, "StartTime": 1584056657.420711}

    [03/12/2020 23:44:18 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=143894.369342 records/second
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_classification_accuracy <score>=0.724
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_classification_cross_entropy <loss>=0.577434814453
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_f_1.000 <score>=0.756613756614
    [2020-03-12 23:44:18.623] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 58, "duration": 571, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, train binary_classification_accuracy <score>=0.732692307692
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, train binary_classification_cross_entropy <loss>=0.559710399586
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=28, train binary_f_1.000 <score>=0.769573248709
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 573.6351013183594, "sum": 573.6351013183594, "min": 573.6351013183594}}, "EndTime": 1584056658.624247, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056658.049838}

    [03/12/2020 23:44:18 INFO 140601087346496] #progress_metric: host=algo-1, completed 29 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2640, "sum": 2640.0, "min": 2640}, "Total Records Seen": {"count": 1, "max": 2627530, "sum": 2627530.0, "min": 2627530}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 30, "sum": 30.0, "min": 30}}, "EndTime": 1584056658.624444, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 28}, "StartTime": 1584056658.050585}

    [03/12/2020 23:44:18 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157797.32044 records/second
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_classification_accuracy <score>=0.724
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_classification_cross_entropy <loss>=0.576155517578
    [03/12/2020 23:44:18 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_f_1.000 <score>=0.756183745583
    [2020-03-12 23:44:19.211] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 60, "duration": 585, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, train binary_classification_accuracy <score>=0.732967032967
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, train binary_classification_cross_entropy <loss>=0.558199994098
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=29, train binary_f_1.000 <score>=0.769493454752
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 586.906909942627, "sum": 586.906909942627, "min": 586.906909942627}}, "EndTime": 1584056659.211917, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056658.624307}

    [03/12/2020 23:44:19 INFO 140601087346496] #progress_metric: host=algo-1, completed 30 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2731, "sum": 2731.0, "min": 2731}, "Total Records Seen": {"count": 1, "max": 2718100, "sum": 2718100.0, "min": 2718100}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 31, "sum": 31.0, "min": 31}}, "EndTime": 1584056659.212118, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 29}, "StartTime": 1584056658.624982}

    [03/12/2020 23:44:19 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154229.148138 records/second
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_classification_accuracy <score>=0.725
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_classification_cross_entropy <loss>=0.574938659668
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_f_1.000 <score>=0.756852343059
    [2020-03-12 23:44:19.811] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 62, "duration": 597, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, train binary_classification_accuracy <score>=0.733340659341
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, train binary_classification_cross_entropy <loss>=0.556754714463
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=30, train binary_f_1.000 <score>=0.769562409785
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 599.729061126709, "sum": 599.729061126709, "min": 599.729061126709}}, "EndTime": 1584056659.812416, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056659.211984}

    [03/12/2020 23:44:19 INFO 140601087346496] #progress_metric: host=algo-1, completed 31 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2822, "sum": 2822.0, "min": 2822}, "Total Records Seen": {"count": 1, "max": 2808670, "sum": 2808670.0, "min": 2808670}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 32, "sum": 32.0, "min": 32}}, "EndTime": 1584056659.812618, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 30}, "StartTime": 1584056659.212658}

    [03/12/2020 23:44:19 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150931.131977 records/second
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_classification_cross_entropy <loss>=0.573779663086
    [03/12/2020 23:44:19 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_f_1.000 <score>=0.759538598048
    [2020-03-12 23:44:20.393] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 64, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, train binary_classification_accuracy <score>=0.733824175824
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, train binary_classification_cross_entropy <loss>=0.555369665796
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=31, train binary_f_1.000 <score>=0.769770359668
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.1619758605957, "sum": 581.1619758605957, "min": 581.1619758605957}}, "EndTime": 1584056660.394405, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056659.812486}

    [03/12/2020 23:44:20 INFO 140601087346496] #progress_metric: host=algo-1, completed 32 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 2913, "sum": 2913.0, "min": 2913}, "Total Records Seen": {"count": 1, "max": 2899240, "sum": 2899240.0, "min": 2899240}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 33, "sum": 33.0, "min": 33}}, "EndTime": 1584056660.394628, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 31}, "StartTime": 1584056659.81321}

    [03/12/2020 23:44:20 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155743.793578 records/second
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_classification_accuracy <score>=0.727
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_classification_cross_entropy <loss>=0.572674377441
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_f_1.000 <score>=0.757333333333
    [2020-03-12 23:44:20.979] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 66, "duration": 583, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, train binary_classification_accuracy <score>=0.73410989011
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, train binary_classification_cross_entropy <loss>=0.554040452894
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=32, train binary_f_1.000 <score>=0.769824961948
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 585.0510597229004, "sum": 585.0510597229004, "min": 585.0510597229004}}, "EndTime": 1584056660.980252, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056660.394473}

    [03/12/2020 23:44:20 INFO 140601087346496] #progress_metric: host=algo-1, completed 33 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3004, "sum": 3004.0, "min": 3004}, "Total Records Seen": {"count": 1, "max": 2989810, "sum": 2989810.0, "min": 2989810}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 34, "sum": 34.0, "min": 34}}, "EndTime": 1584056660.980482, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 32}, "StartTime": 1584056660.395169}

    [03/12/2020 23:44:20 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154707.640399 records/second
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_classification_accuracy <score>=0.727
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_classification_cross_entropy <loss>=0.571619140625
    [03/12/2020 23:44:20 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_f_1.000 <score>=0.757333333333
    [2020-03-12 23:44:21.573] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 68, "duration": 590, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, train binary_classification_accuracy <score>=0.734362637363
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, train binary_classification_cross_entropy <loss>=0.552763150058
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=33, train binary_f_1.000 <score>=0.769800683751
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 592.8421020507812, "sum": 592.8421020507812, "min": 592.8421020507812}}, "EndTime": 1584056661.573889, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056660.980323}

    [03/12/2020 23:44:21 INFO 140601087346496] #progress_metric: host=algo-1, completed 34 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3095, "sum": 3095.0, "min": 3095}, "Total Records Seen": {"count": 1, "max": 3080380, "sum": 3080380.0, "min": 3080380}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 35, "sum": 35.0, "min": 35}}, "EndTime": 1584056661.574093, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 33}, "StartTime": 1584056660.981019}

    [03/12/2020 23:44:21 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152685.049661 records/second
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_classification_cross_entropy <loss>=0.570610473633
    [03/12/2020 23:44:21 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_f_1.000 <score>=0.758007117438
    [2020-03-12 23:44:22.151] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 70, "duration": 575, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, train binary_classification_accuracy <score>=0.734791208791
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, train binary_classification_cross_entropy <loss>=0.55153422865
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=34, train binary_f_1.000 <score>=0.76996397049
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 577.1880149841309, "sum": 577.1880149841309, "min": 577.1880149841309}}, "EndTime": 1584056662.151862, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056661.573952}

    [03/12/2020 23:44:22 INFO 140601087346496] #progress_metric: host=algo-1, completed 35 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3186, "sum": 3186.0, "min": 3186}, "Total Records Seen": {"count": 1, "max": 3170950, "sum": 3170950.0, "min": 3170950}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 36, "sum": 36.0, "min": 36}}, "EndTime": 1584056662.152064, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 34}, "StartTime": 1584056661.574645}

    [03/12/2020 23:44:22 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156823.565298 records/second
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_classification_accuracy <score>=0.727
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_classification_cross_entropy <loss>=0.569645385742
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_f_1.000 <score>=0.756467439786
    [2020-03-12 23:44:22.781] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 72, "duration": 627, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, train binary_classification_accuracy <score>=0.735076923077
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, train binary_classification_cross_entropy <loss>=0.550350505452
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=35, train binary_f_1.000 <score>=0.770058372439
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 629.4388771057129, "sum": 629.4388771057129, "min": 629.4388771057129}}, "EndTime": 1584056662.782099, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056662.151925}

    [03/12/2020 23:44:22 INFO 140601087346496] #progress_metric: host=algo-1, completed 36 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3277, "sum": 3277.0, "min": 3277}, "Total Records Seen": {"count": 1, "max": 3261520, "sum": 3261520.0, "min": 3261520}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 37, "sum": 37.0, "min": 37}}, "EndTime": 1584056662.782329, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 35}, "StartTime": 1584056662.152629}

    [03/12/2020 23:44:22 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=143803.238761 records/second
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_classification_cross_entropy <loss>=0.568721008301
    [03/12/2020 23:44:22 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_f_1.000 <score>=0.757575757576
    [2020-03-12 23:44:23.367] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 74, "duration": 583, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, train binary_classification_accuracy <score>=0.735604395604
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, train binary_classification_cross_entropy <loss>=0.54920913864
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=36, train binary_f_1.000 <score>=0.770393556514
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 585.7760906219482, "sum": 585.7760906219482, "min": 585.7760906219482}}, "EndTime": 1584056663.368697, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056662.78217}

    [03/12/2020 23:44:23 INFO 140601087346496] #progress_metric: host=algo-1, completed 37 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3368, "sum": 3368.0, "min": 3368}, "Total Records Seen": {"count": 1, "max": 3352090, "sum": 3352090.0, "min": 3352090}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 38, "sum": 38.0, "min": 38}}, "EndTime": 1584056663.368906, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 36}, "StartTime": 1584056662.782892}

    [03/12/2020 23:44:23 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154519.419878 records/second
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_classification_cross_entropy <loss>=0.567834777832
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_f_1.000 <score>=0.75935828877
    [2020-03-12 23:44:23.945] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 76, "duration": 574, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, train binary_classification_accuracy <score>=0.735978021978
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, train binary_classification_cross_entropy <loss>=0.548107528519
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=37, train binary_f_1.000 <score>=0.770577900004
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 576.4539241790771, "sum": 576.4539241790771, "min": 576.4539241790771}}, "EndTime": 1584056663.945919, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056663.368766}

    [03/12/2020 23:44:23 INFO 140601087346496] #progress_metric: host=algo-1, completed 38 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3459, "sum": 3459.0, "min": 3459}, "Total Records Seen": {"count": 1, "max": 3442660, "sum": 3442660.0, "min": 3442660}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 39, "sum": 39.0, "min": 39}}, "EndTime": 1584056663.946142, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 37}, "StartTime": 1584056663.369434}

    [03/12/2020 23:44:23 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157014.134682 records/second
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_classification_cross_entropy <loss>=0.566984375
    [03/12/2020 23:44:23 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_f_1.000 <score>=0.75935828877
    [2020-03-12 23:44:24.545] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 78, "duration": 596, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, train binary_classification_accuracy <score>=0.736285714286
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, train binary_classification_cross_entropy <loss>=0.547043359107
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=38, train binary_f_1.000 <score>=0.770705140455
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 599.092960357666, "sum": 599.092960357666, "min": 599.092960357666}}, "EndTime": 1584056664.545794, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056663.945992}

    [03/12/2020 23:44:24 INFO 140601087346496] #progress_metric: host=algo-1, completed 39 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3550, "sum": 3550.0, "min": 3550}, "Total Records Seen": {"count": 1, "max": 3533230, "sum": 3533230.0, "min": 3533230}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 40, "sum": 40.0, "min": 40}}, "EndTime": 1584056664.546028, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 38}, "StartTime": 1584056663.946673}

    [03/12/2020 23:44:24 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151086.005973 records/second
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_classification_cross_entropy <loss>=0.566167724609
    [03/12/2020 23:44:24 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_f_1.000 <score>=0.75935828877
    [2020-03-12 23:44:25.125] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 80, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, train binary_classification_accuracy <score>=0.736758241758
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, train binary_classification_cross_entropy <loss>=0.546014502892
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=39, train binary_f_1.000 <score>=0.771008784927
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 579.1850090026855, "sum": 579.1850090026855, "min": 579.1850090026855}}, "EndTime": 1584056665.125791, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056664.545885}

    [03/12/2020 23:44:25 INFO 140601087346496] #progress_metric: host=algo-1, completed 40 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3641, "sum": 3641.0, "min": 3641}, "Total Records Seen": {"count": 1, "max": 3623800, "sum": 3623800.0, "min": 3623800}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 41, "sum": 41.0, "min": 41}}, "EndTime": 1584056665.126021, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 39}, "StartTime": 1584056664.546573}

    [03/12/2020 23:44:25 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156270.499407 records/second
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_classification_cross_entropy <loss>=0.5653828125
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_f_1.000 <score>=0.760035682426
    [2020-03-12 23:44:25.743] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 82, "duration": 615, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, train binary_classification_accuracy <score>=0.737186813187
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, train binary_classification_cross_entropy <loss>=0.54501902553
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=40, train binary_f_1.000 <score>=0.771261333639
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 617.4421310424805, "sum": 617.4421310424805, "min": 617.4421310424805}}, "EndTime": 1584056665.744039, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056665.125857}

    [03/12/2020 23:44:25 INFO 140601087346496] #progress_metric: host=algo-1, completed 41 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3732, "sum": 3732.0, "min": 3732}, "Total Records Seen": {"count": 1, "max": 3714370, "sum": 3714370.0, "min": 3714370}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 42, "sum": 42.0, "min": 42}}, "EndTime": 1584056665.744288, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 40}, "StartTime": 1584056665.126564}

    [03/12/2020 23:44:25 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=146589.414963 records/second
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_classification_cross_entropy <loss>=0.564627807617
    [03/12/2020 23:44:25 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_f_1.000 <score>=0.758928571429
    [2020-03-12 23:44:26.319] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 84, "duration": 573, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, train binary_classification_accuracy <score>=0.737450549451
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, train binary_classification_cross_entropy <loss>=0.544055167062
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=41, train binary_f_1.000 <score>=0.771372796693
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 575.4530429840088, "sum": 575.4530429840088, "min": 575.4530429840088}}, "EndTime": 1584056666.320318, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056665.74412}

    [03/12/2020 23:44:26 INFO 140601087346496] #progress_metric: host=algo-1, completed 42 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3823, "sum": 3823.0, "min": 3823}, "Total Records Seen": {"count": 1, "max": 3804940, "sum": 3804940.0, "min": 3804940}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 43, "sum": 43.0, "min": 43}}, "EndTime": 1584056666.320563, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 41}, "StartTime": 1584056665.744832}

    [03/12/2020 23:44:26 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157280.278624 records/second
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_classification_accuracy <score>=0.733
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_classification_cross_entropy <loss>=0.563901062012
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_f_1.000 <score>=0.76096687556
    [2020-03-12 23:44:26.936] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 86, "duration": 613, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, train binary_classification_accuracy <score>=0.737692307692
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, train binary_classification_cross_entropy <loss>=0.543121328501
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=42, train binary_f_1.000 <score>=0.771452097815
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 615.5998706817627, "sum": 615.5998706817627, "min": 615.5998706817627}}, "EndTime": 1584056666.936755, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056666.320393}

    [03/12/2020 23:44:26 INFO 140601087346496] #progress_metric: host=algo-1, completed 43 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 3914, "sum": 3914.0, "min": 3914}, "Total Records Seen": {"count": 1, "max": 3895510, "sum": 3895510.0, "min": 3895510}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 44, "sum": 44.0, "min": 44}}, "EndTime": 1584056666.936946, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 42}, "StartTime": 1584056666.321129}

    [03/12/2020 23:44:26 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=147047.846637 records/second
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_classification_cross_entropy <loss>=0.563201171875
    [03/12/2020 23:44:26 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:27.515] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 88, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, train binary_classification_accuracy <score>=0.738
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, train binary_classification_cross_entropy <loss>=0.542216007442
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=43, train binary_f_1.000 <score>=0.77162835249
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 579.0359973907471, "sum": 579.0359973907471, "min": 579.0359973907471}}, "EndTime": 1584056667.516574, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056666.936818}

    [03/12/2020 23:44:27 INFO 140601087346496] #progress_metric: host=algo-1, completed 44 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4005, "sum": 4005.0, "min": 4005}, "Total Records Seen": {"count": 1, "max": 3986080, "sum": 3986080.0, "min": 3986080}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 45, "sum": 45.0, "min": 45}}, "EndTime": 1584056667.516822, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 43}, "StartTime": 1584056666.937502}

    [03/12/2020 23:44:27 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156306.37868 records/second
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_classification_cross_entropy <loss>=0.562526489258
    [03/12/2020 23:44:27 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:28.150] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 90, "duration": 631, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, train binary_classification_accuracy <score>=0.738098901099
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, train binary_classification_cross_entropy <loss>=0.541337841663
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=44, train binary_f_1.000 <score>=0.771646753346
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 633.3580017089844, "sum": 633.3580017089844, "min": 633.3580017089844}}, "EndTime": 1584056668.150745, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056667.516656}

    [03/12/2020 23:44:28 INFO 140601087346496] #progress_metric: host=algo-1, completed 45 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4096, "sum": 4096.0, "min": 4096}, "Total Records Seen": {"count": 1, "max": 4076650, "sum": 4076650.0, "min": 4076650}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 46, "sum": 46.0, "min": 46}}, "EndTime": 1584056668.151001, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 44}, "StartTime": 1584056667.517345}

    [03/12/2020 23:44:28 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=142907.649133 records/second
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_classification_cross_entropy <loss>=0.561875854492
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:28.721] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 92, "duration": 568, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, train binary_classification_accuracy <score>=0.738417582418
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, train binary_classification_cross_entropy <loss>=0.540485577594
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=45, train binary_f_1.000 <score>=0.771817484663
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 570.4739093780518, "sum": 570.4739093780518, "min": 570.4739093780518}}, "EndTime": 1584056668.72207, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056668.15082}

    [03/12/2020 23:44:28 INFO 140601087346496] #progress_metric: host=algo-1, completed 46 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4187, "sum": 4187.0, "min": 4187}, "Total Records Seen": {"count": 1, "max": 4167220, "sum": 4167220.0, "min": 4167220}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 47, "sum": 47.0, "min": 47}}, "EndTime": 1584056668.722276, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 45}, "StartTime": 1584056668.151569}

    [03/12/2020 23:44:28 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=158667.911891 records/second
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_classification_cross_entropy <loss>=0.561248046875
    [03/12/2020 23:44:28 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:29.303] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 94, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, train binary_classification_accuracy <score>=0.738582417582
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, train binary_classification_cross_entropy <loss>=0.539658030751
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=46, train binary_f_1.000 <score>=0.771840982113
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.4459323883057, "sum": 581.4459323883057, "min": 581.4459323883057}}, "EndTime": 1584056669.304308, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056668.722138}

    [03/12/2020 23:44:29 INFO 140601087346496] #progress_metric: host=algo-1, completed 47 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4278, "sum": 4278.0, "min": 4278}, "Total Records Seen": {"count": 1, "max": 4257790, "sum": 4257790.0, "min": 4257790}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 48, "sum": 48.0, "min": 48}}, "EndTime": 1584056669.304499, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 46}, "StartTime": 1584056668.722834}

    [03/12/2020 23:44:29 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155677.415385 records/second
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_classification_accuracy <score>=0.733
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_classification_cross_entropy <loss>=0.560641845703
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_f_1.000 <score>=0.76096687556
    [2020-03-12 23:44:29.900] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 96, "duration": 594, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, train binary_classification_accuracy <score>=0.738626373626
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, train binary_classification_cross_entropy <loss>=0.538854108538
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=47, train binary_f_1.000 <score>=0.771778658402
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 596.4610576629639, "sum": 596.4610576629639, "min": 596.4610576629639}}, "EndTime": 1584056669.901544, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056669.304371}

    [03/12/2020 23:44:29 INFO 140601087346496] #progress_metric: host=algo-1, completed 48 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4369, "sum": 4369.0, "min": 4369}, "Total Records Seen": {"count": 1, "max": 4348360, "sum": 4348360.0, "min": 4348360}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 49, "sum": 49.0, "min": 49}}, "EndTime": 1584056669.901779, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 47}, "StartTime": 1584056669.305052}

    [03/12/2020 23:44:29 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151748.691159 records/second
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_classification_accuracy <score>=0.733
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_classification_cross_entropy <loss>=0.560056152344
    [03/12/2020 23:44:29 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_f_1.000 <score>=0.76096687556
    [2020-03-12 23:44:30.492] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 98, "duration": 588, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, train binary_classification_accuracy <score>=0.738802197802
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, train binary_classification_cross_entropy <loss>=0.538072793478
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=48, train binary_f_1.000 <score>=0.771875269932
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 590.2559757232666, "sum": 590.2559757232666, "min": 590.2559757232666}}, "EndTime": 1584056670.492633, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056669.901616}

    [03/12/2020 23:44:30 INFO 140601087346496] #progress_metric: host=algo-1, completed 49 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4460, "sum": 4460.0, "min": 4460}, "Total Records Seen": {"count": 1, "max": 4438930, "sum": 4438930.0, "min": 4438930}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 50, "sum": 50.0, "min": 50}}, "EndTime": 1584056670.492838, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 48}, "StartTime": 1584056669.90235}

    [03/12/2020 23:44:30 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153343.419549 records/second
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_classification_cross_entropy <loss>=0.559489990234
    [03/12/2020 23:44:30 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:31.082] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 100, "duration": 587, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, train binary_classification_accuracy <score>=0.739032967033
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, train binary_classification_cross_entropy <loss>=0.537313125778
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=49, train binary_f_1.000 <score>=0.771995852375
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 589.4179344177246, "sum": 589.4179344177246, "min": 589.4179344177246}}, "EndTime": 1584056671.082913, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056670.4927}

    [03/12/2020 23:44:31 INFO 140601087346496] #progress_metric: host=algo-1, completed 50 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4551, "sum": 4551.0, "min": 4551}, "Total Records Seen": {"count": 1, "max": 4529500, "sum": 4529500.0, "min": 4529500}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 51, "sum": 51.0, "min": 51}}, "EndTime": 1584056671.083172, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 49}, "StartTime": 1584056670.493435}

    [03/12/2020 23:44:31 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153542.808649 records/second
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_classification_cross_entropy <loss>=0.558942504883
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:31.678] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 102, "duration": 592, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, train binary_classification_accuracy <score>=0.739186813187
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, train binary_classification_cross_entropy <loss>=0.536574205336
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=50, train binary_f_1.000 <score>=0.77204272158
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 594.7620868682861, "sum": 594.7620868682861, "min": 594.7620868682861}}, "EndTime": 1584056671.678574, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056671.082993}

    [03/12/2020 23:44:31 INFO 140601087346496] #progress_metric: host=algo-1, completed 51 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4642, "sum": 4642.0, "min": 4642}, "Total Records Seen": {"count": 1, "max": 4620070, "sum": 4620070.0, "min": 4620070}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 52, "sum": 52.0, "min": 52}}, "EndTime": 1584056671.678775, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 50}, "StartTime": 1584056671.083785}

    [03/12/2020 23:44:31 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152195.183753 records/second
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_classification_cross_entropy <loss>=0.558412597656
    [03/12/2020 23:44:31 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:32.263] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 104, "duration": 582, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, train binary_classification_accuracy <score>=0.739252747253
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, train binary_classification_cross_entropy <loss>=0.535855181684
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=51, train binary_f_1.000 <score>=0.77204342396
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 585.0701332092285, "sum": 585.0701332092285, "min": 585.0701332092285}}, "EndTime": 1584056672.264451, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056671.678642}

    [03/12/2020 23:44:32 INFO 140601087346496] #progress_metric: host=algo-1, completed 52 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4733, "sum": 4733.0, "min": 4733}, "Total Records Seen": {"count": 1, "max": 4710640, "sum": 4710640.0, "min": 4710640}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 53, "sum": 53.0, "min": 53}}, "EndTime": 1584056672.264833, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 51}, "StartTime": 1584056671.679346}

    [03/12/2020 23:44:32 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154650.515408 records/second
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_classification_cross_entropy <loss>=0.557899597168
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:32.912] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 106, "duration": 645, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, train binary_classification_accuracy <score>=0.744307692308
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, train binary_classification_cross_entropy <loss>=0.535155260694
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=52, train binary_f_1.000 <score>=0.775483422749
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 647.6149559020996, "sum": 647.6149559020996, "min": 647.6149559020996}}, "EndTime": 1584056672.913144, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056672.264572}

    [03/12/2020 23:44:32 INFO 140601087346496] #progress_metric: host=algo-1, completed 53 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4824, "sum": 4824.0, "min": 4824}, "Total Records Seen": {"count": 1, "max": 4801210, "sum": 4801210.0, "min": 4801210}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 54, "sum": 54.0, "min": 54}}, "EndTime": 1584056672.913343, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 52}, "StartTime": 1584056672.265501}

    [03/12/2020 23:44:32 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=139779.72966 records/second
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_classification_cross_entropy <loss>=0.557402587891
    [03/12/2020 23:44:32 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:33.474] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 108, "duration": 559, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, train binary_classification_accuracy <score>=0.744373626374
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, train binary_classification_cross_entropy <loss>=0.534473667019
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=53, train binary_f_1.000 <score>=0.775519657229
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 561.2778663635254, "sum": 561.2778663635254, "min": 561.2778663635254}}, "EndTime": 1584056673.475211, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056672.913211}

    [03/12/2020 23:44:33 INFO 140601087346496] #progress_metric: host=algo-1, completed 54 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 4915, "sum": 4915.0, "min": 4915}, "Total Records Seen": {"count": 1, "max": 4891780, "sum": 4891780.0, "min": 4891780}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 55, "sum": 55.0, "min": 55}}, "EndTime": 1584056673.475403, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 53}, "StartTime": 1584056672.9139}

    [03/12/2020 23:44:33 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=161270.212274 records/second
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_classification_accuracy <score>=0.733
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_classification_cross_entropy <loss>=0.556921020508
    [03/12/2020 23:44:33 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_f_1.000 <score>=0.76096687556
    [2020-03-12 23:44:34.063] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 110, "duration": 586, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, train binary_classification_accuracy <score>=0.744692307692
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, train binary_classification_cross_entropy <loss>=0.533809693724
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=54, train binary_f_1.000 <score>=0.775754065923
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 588.6600017547607, "sum": 588.6600017547607, "min": 588.6600017547607}}, "EndTime": 1584056674.064661, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056673.475283}

    [03/12/2020 23:44:34 INFO 140601087346496] #progress_metric: host=algo-1, completed 55 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5006, "sum": 5006.0, "min": 5006}, "Total Records Seen": {"count": 1, "max": 4982350, "sum": 4982350.0, "min": 4982350}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 56, "sum": 56.0, "min": 56}}, "EndTime": 1584056674.064884, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 54}, "StartTime": 1584056673.47597}

    [03/12/2020 23:44:34 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153763.626424 records/second
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_classification_cross_entropy <loss>=0.556453857422
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:34.693] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 112, "duration": 626, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, train binary_classification_accuracy <score>=0.744758241758
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, train binary_classification_cross_entropy <loss>=0.533162658021
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=55, train binary_f_1.000 <score>=0.775803322362
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 628.8661956787109, "sum": 628.8661956787109, "min": 628.8661956787109}}, "EndTime": 1584056674.694317, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056674.064736}

    [03/12/2020 23:44:34 INFO 140601087346496] #progress_metric: host=algo-1, completed 56 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5097, "sum": 5097.0, "min": 5097}, "Total Records Seen": {"count": 1, "max": 5072920, "sum": 5072920.0, "min": 5072920}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 57, "sum": 57.0, "min": 57}}, "EndTime": 1584056674.69453, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 55}, "StartTime": 1584056674.065419}

    [03/12/2020 23:44:34 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=143939.896148 records/second
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_classification_cross_entropy <loss>=0.556000610352
    [03/12/2020 23:44:34 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:35.276] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 114, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, train binary_classification_accuracy <score>=0.744923076923
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, train binary_classification_cross_entropy <loss>=0.532531883156
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=56, train binary_f_1.000 <score>=0.775911337658
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.7420482635498, "sum": 581.7420482635498, "min": 581.7420482635498}}, "EndTime": 1584056675.276894, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056674.694389}

    [03/12/2020 23:44:35 INFO 140601087346496] #progress_metric: host=algo-1, completed 57 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5188, "sum": 5188.0, "min": 5188}, "Total Records Seen": {"count": 1, "max": 5163490, "sum": 5163490.0, "min": 5163490}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 58, "sum": 58.0, "min": 58}}, "EndTime": 1584056675.277083, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 56}, "StartTime": 1584056674.695126}

    [03/12/2020 23:44:35 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155600.321819 records/second
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_classification_cross_entropy <loss>=0.555560668945
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_f_1.000 <score>=0.759856630824
    [2020-03-12 23:44:35.876] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 116, "duration": 597, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, train binary_classification_accuracy <score>=0.745120879121
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, train binary_classification_cross_entropy <loss>=0.531916758443
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=57, train binary_f_1.000 <score>=0.77605916657
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 599.1261005401611, "sum": 599.1261005401611, "min": 599.1261005401611}}, "EndTime": 1584056675.876851, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056675.276956}

    [03/12/2020 23:44:35 INFO 140601087346496] #progress_metric: host=algo-1, completed 58 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5279, "sum": 5279.0, "min": 5279}, "Total Records Seen": {"count": 1, "max": 5254060, "sum": 5254060.0, "min": 5254060}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 59, "sum": 59.0, "min": 59}}, "EndTime": 1584056675.877054, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 57}, "StartTime": 1584056675.277697}

    [03/12/2020 23:44:35 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151084.984445 records/second
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_classification_cross_entropy <loss>=0.555133300781
    [03/12/2020 23:44:35 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:36.469] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 118, "duration": 589, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, train binary_classification_accuracy <score>=0.745153846154
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, train binary_classification_cross_entropy <loss>=0.531316687951
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=58, train binary_f_1.000 <score>=0.776060023755
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 591.9759273529053, "sum": 591.9759273529053, "min": 591.9759273529053}}, "EndTime": 1584056676.46962, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056675.876919}

    [03/12/2020 23:44:36 INFO 140601087346496] #progress_metric: host=algo-1, completed 59 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5370, "sum": 5370.0, "min": 5370}, "Total Records Seen": {"count": 1, "max": 5344630, "sum": 5344630.0, "min": 5344630}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 60, "sum": 60.0, "min": 60}}, "EndTime": 1584056676.469869, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 58}, "StartTime": 1584056675.877611}

    [03/12/2020 23:44:36 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152889.436047 records/second
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_classification_cross_entropy <loss>=0.554717895508
    [03/12/2020 23:44:36 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:37.067] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 120, "duration": 595, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, train binary_classification_accuracy <score>=0.745351648352
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, train binary_classification_cross_entropy <loss>=0.530731080108
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=59, train binary_f_1.000 <score>=0.776207905589
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 598.0300903320312, "sum": 598.0300903320312, "min": 598.0300903320312}}, "EndTime": 1584056677.068495, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056676.469702}

    [03/12/2020 23:44:37 INFO 140601087346496] #progress_metric: host=algo-1, completed 60 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5461, "sum": 5461.0, "min": 5461}, "Total Records Seen": {"count": 1, "max": 5435200, "sum": 5435200.0, "min": 5435200}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 61, "sum": 61.0, "min": 61}}, "EndTime": 1584056677.068726, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 59}, "StartTime": 1584056676.470422}

    [03/12/2020 23:44:37 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151351.831518 records/second
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_classification_cross_entropy <loss>=0.554313964844
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:37.637] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 122, "duration": 567, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, train binary_classification_accuracy <score>=0.745373626374
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, train binary_classification_cross_entropy <loss>=0.530159396329
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=60, train binary_f_1.000 <score>=0.77618399065
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 569.1308975219727, "sum": 569.1308975219727, "min": 569.1308975219727}}, "EndTime": 1584056677.638434, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056677.068563}

    [03/12/2020 23:44:37 INFO 140601087346496] #progress_metric: host=algo-1, completed 61 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5552, "sum": 5552.0, "min": 5552}, "Total Records Seen": {"count": 1, "max": 5525770, "sum": 5525770.0, "min": 5525770}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 62, "sum": 62.0, "min": 62}}, "EndTime": 1584056677.638632, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 60}, "StartTime": 1584056677.069276}

    [03/12/2020 23:44:37 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=159047.099863 records/second
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_classification_cross_entropy <loss>=0.553920898438
    [03/12/2020 23:44:37 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:38.251] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 124, "duration": 610, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, train binary_classification_accuracy <score>=0.745538461538
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, train binary_classification_cross_entropy <loss>=0.529601105742
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=61, train binary_f_1.000 <score>=0.776292145686
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 612.497091293335, "sum": 612.497091293335, "min": 612.497091293335}}, "EndTime": 1584056678.251731, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056677.6385}

    [03/12/2020 23:44:38 INFO 140601087346496] #progress_metric: host=algo-1, completed 62 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5643, "sum": 5643.0, "min": 5643}, "Total Records Seen": {"count": 1, "max": 5616340, "sum": 5616340.0, "min": 5616340}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 63, "sum": 63.0, "min": 63}}, "EndTime": 1584056678.251926, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 61}, "StartTime": 1584056677.639205}

    [03/12/2020 23:44:38 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=147790.298142 records/second
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_classification_cross_entropy <loss>=0.553538269043
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:38.833] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 126, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, train binary_classification_accuracy <score>=0.745703296703
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, train binary_classification_cross_entropy <loss>=0.529055700617
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=62, train binary_f_1.000 <score>=0.77640465723
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.0799598693848, "sum": 581.0799598693848, "min": 581.0799598693848}}, "EndTime": 1584056678.83359, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056678.251797}

    [03/12/2020 23:44:38 INFO 140601087346496] #progress_metric: host=algo-1, completed 63 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5734, "sum": 5734.0, "min": 5734}, "Total Records Seen": {"count": 1, "max": 5706910, "sum": 5706910.0, "min": 5706910}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 64, "sum": 64.0, "min": 64}}, "EndTime": 1584056678.833845, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 62}, "StartTime": 1584056678.252478}

    [03/12/2020 23:44:38 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155760.396972 records/second
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_classification_cross_entropy <loss>=0.553165527344
    [03/12/2020 23:44:38 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_f_1.000 <score>=0.756950672646
    [2020-03-12 23:44:39.418] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 128, "duration": 582, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, train binary_classification_accuracy <score>=0.745802197802
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, train binary_classification_cross_entropy <loss>=0.528522686969
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=63, train binary_f_1.000 <score>=0.776467859766
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 584.6011638641357, "sum": 584.6011638641357, "min": 584.6011638641357}}, "EndTime": 1584056679.419013, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056678.833668}

    [03/12/2020 23:44:39 INFO 140601087346496] #progress_metric: host=algo-1, completed 64 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5825, "sum": 5825.0, "min": 5825}, "Total Records Seen": {"count": 1, "max": 5797480, "sum": 5797480.0, "min": 5797480}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 65, "sum": 65.0, "min": 65}}, "EndTime": 1584056679.419215, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 63}, "StartTime": 1584056678.834384}

    [03/12/2020 23:44:39 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154838.235174 records/second
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_classification_cross_entropy <loss>=0.552802001953
    [03/12/2020 23:44:39 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_f_1.000 <score>=0.75763016158
    [2020-03-12 23:44:40.005] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 130, "duration": 584, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, train binary_classification_accuracy <score>=0.745703296703
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, train binary_classification_cross_entropy <loss>=0.52800158792
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=64, train binary_f_1.000 <score>=0.776348471523
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 586.3621234893799, "sum": 586.3621234893799, "min": 586.3621234893799}}, "EndTime": 1584056680.006154, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056679.419081}

    [03/12/2020 23:44:40 INFO 140601087346496] #progress_metric: host=algo-1, completed 65 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 5916, "sum": 5916.0, "min": 5916}, "Total Records Seen": {"count": 1, "max": 5888050, "sum": 5888050.0, "min": 5888050}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 66, "sum": 66.0, "min": 66}}, "EndTime": 1584056680.006402, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 64}, "StartTime": 1584056679.419759}

    [03/12/2020 23:44:40 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154353.981124 records/second
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_classification_cross_entropy <loss>=0.552447631836
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_f_1.000 <score>=0.758744394619
    [2020-03-12 23:44:40.585] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 132, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, train binary_classification_accuracy <score>=0.745747252747
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, train binary_classification_cross_entropy <loss>=0.527491954091
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=65, train binary_f_1.000 <score>=0.776391452678
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 579.1549682617188, "sum": 579.1549682617188, "min": 579.1549682617188}}, "EndTime": 1584056680.586173, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056680.006231}

    [03/12/2020 23:44:40 INFO 140601087346496] #progress_metric: host=algo-1, completed 66 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6007, "sum": 6007.0, "min": 6007}, "Total Records Seen": {"count": 1, "max": 5978620, "sum": 5978620.0, "min": 5978620}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 67, "sum": 67.0, "min": 67}}, "EndTime": 1584056680.586372, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 65}, "StartTime": 1584056680.006992}

    [03/12/2020 23:44:40 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156295.831808 records/second
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_classification_cross_entropy <loss>=0.552101623535
    [03/12/2020 23:44:40 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_f_1.000 <score>=0.758064516129
    [2020-03-12 23:44:41.203] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 134, "duration": 614, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, train binary_classification_accuracy <score>=0.745934065934
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, train binary_classification_cross_entropy <loss>=0.526993347168
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=66, train binary_f_1.000 <score>=0.776544951965
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 616.6179180145264, "sum": 616.6179180145264, "min": 616.6179180145264}}, "EndTime": 1584056681.20364, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056680.586237}

    [03/12/2020 23:44:41 INFO 140601087346496] #progress_metric: host=algo-1, completed 67 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6098, "sum": 6098.0, "min": 6098}, "Total Records Seen": {"count": 1, "max": 6069190, "sum": 6069190.0, "min": 6069190}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 68, "sum": 68.0, "min": 68}}, "EndTime": 1584056681.203839, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 66}, "StartTime": 1584056680.586994}

    [03/12/2020 23:44:41 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=146800.712938 records/second
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_classification_cross_entropy <loss>=0.551763671875
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:41.773] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 136, "duration": 567, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, train binary_classification_accuracy <score>=0.746252747253
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, train binary_classification_cross_entropy <loss>=0.526505340576
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=67, train binary_f_1.000 <score>=0.776784247006
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 569.2732334136963, "sum": 569.2732334136963, "min": 569.2732334136963}}, "EndTime": 1584056681.773703, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056681.203702}

    [03/12/2020 23:44:41 INFO 140601087346496] #progress_metric: host=algo-1, completed 68 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6189, "sum": 6189.0, "min": 6189}, "Total Records Seen": {"count": 1, "max": 6159760, "sum": 6159760.0, "min": 6159760}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 69, "sum": 69.0, "min": 69}}, "EndTime": 1584056681.773894, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 67}, "StartTime": 1584056681.204401}

    [03/12/2020 23:44:41 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=159006.623612 records/second
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_classification_cross_entropy <loss>=0.55143347168
    [03/12/2020 23:44:41 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:42.353] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 138, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, train binary_classification_accuracy <score>=0.746263736264
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, train binary_classification_cross_entropy <loss>=0.52602752652
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=68, train binary_f_1.000 <score>=0.776796071456
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 579.164981842041, "sum": 579.164981842041, "min": 579.164981842041}}, "EndTime": 1584056682.353658, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056681.773765}

    [03/12/2020 23:44:42 INFO 140601087346496] #progress_metric: host=algo-1, completed 69 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6280, "sum": 6280.0, "min": 6280}, "Total Records Seen": {"count": 1, "max": 6250330, "sum": 6250330.0, "min": 6250330}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 70, "sum": 70.0, "min": 70}}, "EndTime": 1584056682.353921, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 68}, "StartTime": 1584056681.77446}

    [03/12/2020 23:44:42 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156264.52113 records/second
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_classification_cross_entropy <loss>=0.551110412598
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:42.957] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 140, "duration": 601, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, train binary_classification_accuracy <score>=0.74632967033
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, train binary_classification_cross_entropy <loss>=0.525559509948
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=69, train binary_f_1.000 <score>=0.776832498695
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 603.2660007476807, "sum": 603.2660007476807, "min": 603.2660007476807}}, "EndTime": 1584056682.957844, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056682.353759}

    [03/12/2020 23:44:42 INFO 140601087346496] #progress_metric: host=algo-1, completed 70 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6371, "sum": 6371.0, "min": 6371}, "Total Records Seen": {"count": 1, "max": 6340900, "sum": 6340900.0, "min": 6340900}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 71, "sum": 71.0, "min": 71}}, "EndTime": 1584056682.958053, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 69}, "StartTime": 1584056682.354546}

    [03/12/2020 23:44:42 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150042.465088 records/second
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_classification_cross_entropy <loss>=0.550794311523
    [03/12/2020 23:44:42 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:43.537] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 142, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, train binary_classification_accuracy <score>=0.746494505495
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, train binary_classification_cross_entropy <loss>=0.525100904192
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=70, train binary_f_1.000 <score>=0.776971044617
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 579.7929763793945, "sum": 579.7929763793945, "min": 579.7929763793945}}, "EndTime": 1584056683.538427, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056682.957925}

    [03/12/2020 23:44:43 INFO 140601087346496] #progress_metric: host=algo-1, completed 71 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6462, "sum": 6462.0, "min": 6462}, "Total Records Seen": {"count": 1, "max": 6431470, "sum": 6431470.0, "min": 6431470}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 72, "sum": 72.0, "min": 72}}, "EndTime": 1584056683.538648, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 70}, "StartTime": 1584056682.958606}

    [03/12/2020 23:44:43 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156115.918445 records/second
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_classification_accuracy <score>=0.732
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_classification_cross_entropy <loss>=0.550484741211
    [03/12/2020 23:44:43 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_f_1.000 <score>=0.760286225403
    [2020-03-12 23:44:44.121] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 144, "duration": 581, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, train binary_classification_accuracy <score>=0.746538461538
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, train binary_classification_cross_entropy <loss>=0.524651349414
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=71, train binary_f_1.000 <score>=0.776988155668
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 583.057165145874, "sum": 583.057165145874, "min": 583.057165145874}}, "EndTime": 1584056684.122332, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056683.53851}

    [03/12/2020 23:44:44 INFO 140601087346496] #progress_metric: host=algo-1, completed 72 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6553, "sum": 6553.0, "min": 6553}, "Total Records Seen": {"count": 1, "max": 6522040, "sum": 6522040.0, "min": 6522040}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 73, "sum": 73.0, "min": 73}}, "EndTime": 1584056684.122564, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 71}, "StartTime": 1584056683.539243}

    [03/12/2020 23:44:44 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155232.868145 records/second
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_classification_cross_entropy <loss>=0.550181274414
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:44.710] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 146, "duration": 585, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, train binary_classification_accuracy <score>=0.746637362637
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, train binary_classification_cross_entropy <loss>=0.524210472023
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=72, train binary_f_1.000 <score>=0.777047151201
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 587.9120826721191, "sum": 587.9120826721191, "min": 587.9120826721191}}, "EndTime": 1584056684.711137, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056684.122407}

    [03/12/2020 23:44:44 INFO 140601087346496] #progress_metric: host=algo-1, completed 73 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6644, "sum": 6644.0, "min": 6644}, "Total Records Seen": {"count": 1, "max": 6612610, "sum": 6612610.0, "min": 6612610}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 74, "sum": 74.0, "min": 74}}, "EndTime": 1584056684.711378, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 72}, "StartTime": 1584056684.123192}

    [03/12/2020 23:44:44 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153955.93635 records/second
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_classification_cross_entropy <loss>=0.549883728027
    [03/12/2020 23:44:44 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:45.289] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 148, "duration": 576, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, train binary_classification_accuracy <score>=0.746736263736
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, train binary_classification_cross_entropy <loss>=0.523777950413
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=73, train binary_f_1.000 <score>=0.777097538566
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 578.3469676971436, "sum": 578.3469676971436, "min": 578.3469676971436}}, "EndTime": 1584056685.290287, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056684.711208}

    [03/12/2020 23:44:45 INFO 140601087346496] #progress_metric: host=algo-1, completed 74 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6735, "sum": 6735.0, "min": 6735}, "Total Records Seen": {"count": 1, "max": 6703180, "sum": 6703180.0, "min": 6703180}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 75, "sum": 75.0, "min": 75}}, "EndTime": 1584056685.29048, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 73}, "StartTime": 1584056684.711913}

    [03/12/2020 23:44:45 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156514.906969 records/second
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_classification_cross_entropy <loss>=0.549591674805
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:45.868] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 150, "duration": 576, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, train binary_classification_accuracy <score>=0.74689010989
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, train binary_classification_cross_entropy <loss>=0.52335343145
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=74, train binary_f_1.000 <score>=0.777211394303
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 578.3929824829102, "sum": 578.3929824829102, "min": 578.3929824829102}}, "EndTime": 1584056685.86948, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056685.29035}

    [03/12/2020 23:44:45 INFO 140601087346496] #progress_metric: host=algo-1, completed 75 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6826, "sum": 6826.0, "min": 6826}, "Total Records Seen": {"count": 1, "max": 6793750, "sum": 6793750.0, "min": 6793750}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 76, "sum": 76.0, "min": 76}}, "EndTime": 1584056685.869697, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 74}, "StartTime": 1584056685.291059}

    [03/12/2020 23:44:45 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156494.016212 records/second
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_classification_cross_entropy <loss>=0.54930480957
    [03/12/2020 23:44:45 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:46.440] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 152, "duration": 569, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, train binary_classification_accuracy <score>=0.746912087912
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, train binary_classification_cross_entropy <loss>=0.522936604594
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=75, train binary_f_1.000 <score>=0.777209189843
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 571.0761547088623, "sum": 571.0761547088623, "min": 571.0761547088623}}, "EndTime": 1584056686.441405, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056685.869549}

    [03/12/2020 23:44:46 INFO 140601087346496] #progress_metric: host=algo-1, completed 76 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 6917, "sum": 6917.0, "min": 6917}, "Total Records Seen": {"count": 1, "max": 6884320, "sum": 6884320.0, "min": 6884320}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 77, "sum": 77.0, "min": 77}}, "EndTime": 1584056686.441634, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 75}, "StartTime": 1584056685.870295}

    [03/12/2020 23:44:46 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=158490.169322 records/second
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_classification_cross_entropy <loss>=0.549022888184
    [03/12/2020 23:44:46 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:47.032] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 154, "duration": 589, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, train binary_classification_accuracy <score>=0.747010989011
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, train binary_classification_cross_entropy <loss>=0.522527152261
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=76, train binary_f_1.000 <score>=0.777255311738
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 590.9149646759033, "sum": 590.9149646759033, "min": 590.9149646759033}}, "EndTime": 1584056687.03311, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056686.441473}

    [03/12/2020 23:44:47 INFO 140601087346496] #progress_metric: host=algo-1, completed 77 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7008, "sum": 7008.0, "min": 7008}, "Total Records Seen": {"count": 1, "max": 6974890, "sum": 6974890.0, "min": 6974890}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 78, "sum": 78.0, "min": 78}}, "EndTime": 1584056687.033364, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 76}, "StartTime": 1584056686.442162}

    [03/12/2020 23:44:47 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=153163.750829 records/second
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_classification_cross_entropy <loss>=0.548745544434
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:44:47.600] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 156, "duration": 564, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, train binary_classification_accuracy <score>=0.747098901099
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, train binary_classification_cross_entropy <loss>=0.52212477531
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=77, train binary_f_1.000 <score>=0.777337022775
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 566.735029220581, "sum": 566.735029220581, "min": 566.735029220581}}, "EndTime": 1584056687.600702, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056687.03319}

    [03/12/2020 23:44:47 INFO 140601087346496] #progress_metric: host=algo-1, completed 78 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7099, "sum": 7099.0, "min": 7099}, "Total Records Seen": {"count": 1, "max": 7065460, "sum": 7065460.0, "min": 7065460}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 79, "sum": 79.0, "min": 79}}, "EndTime": 1584056687.600938, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 77}, "StartTime": 1584056687.033939}

    [03/12/2020 23:44:47 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=159705.321987 records/second
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_classification_cross_entropy <loss>=0.54847253418
    [03/12/2020 23:44:47 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_f_1.000 <score>=0.758064516129
    [2020-03-12 23:44:48.209] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 158, "duration": 606, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, train binary_classification_accuracy <score>=0.747208791209
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, train binary_classification_cross_entropy <loss>=0.521729186006
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=78, train binary_f_1.000 <score>=0.777438080495
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 608.630895614624, "sum": 608.630895614624, "min": 608.630895614624}}, "EndTime": 1584056688.210164, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056687.600765}

    [03/12/2020 23:44:48 INFO 140601087346496] #progress_metric: host=algo-1, completed 79 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7190, "sum": 7190.0, "min": 7190}, "Total Records Seen": {"count": 1, "max": 7156030, "sum": 7156030.0, "min": 7156030}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 80, "sum": 80.0, "min": 80}}, "EndTime": 1584056688.210411, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 78}, "StartTime": 1584056687.601504}

    [03/12/2020 23:44:48 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=148710.428682 records/second
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_classification_cross_entropy <loss>=0.548203674316
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_f_1.000 <score>=0.758064516129
    [2020-03-12 23:44:48.807] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 160, "duration": 595, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, train binary_classification_accuracy <score>=0.747164835165
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, train binary_classification_cross_entropy <loss>=0.521340093927
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=79, train binary_f_1.000 <score>=0.777369225708
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 597.1648693084717, "sum": 597.1648693084717, "min": 597.1648693084717}}, "EndTime": 1584056688.80818, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056688.210233}

    [03/12/2020 23:44:48 INFO 140601087346496] #progress_metric: host=algo-1, completed 80 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7281, "sum": 7281.0, "min": 7281}, "Total Records Seen": {"count": 1, "max": 7246600, "sum": 7246600.0, "min": 7246600}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 81, "sum": 81.0, "min": 81}}, "EndTime": 1584056688.808436, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 79}, "StartTime": 1584056688.210982}

    [03/12/2020 23:44:48 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151564.634243 records/second
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_classification_cross_entropy <loss>=0.547938598633
    [03/12/2020 23:44:48 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_f_1.000 <score>=0.756950672646
    [2020-03-12 23:44:49.393] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 162, "duration": 582, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, train binary_classification_accuracy <score>=0.747395604396
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, train binary_classification_cross_entropy <loss>=0.520957222739
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=80, train binary_f_1.000 <score>=0.77751860706
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 584.8410129547119, "sum": 584.8410129547119, "min": 584.8410129547119}}, "EndTime": 1584056689.393863, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056688.808267}

    [03/12/2020 23:44:49 INFO 140601087346496] #progress_metric: host=algo-1, completed 81 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7372, "sum": 7372.0, "min": 7372}, "Total Records Seen": {"count": 1, "max": 7337170, "sum": 7337170.0, "min": 7337170}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 82, "sum": 82.0, "min": 82}}, "EndTime": 1584056689.394112, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 80}, "StartTime": 1584056688.808994}

    [03/12/2020 23:44:49 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154757.304762 records/second
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_classification_cross_entropy <loss>=0.547677062988
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:49.967] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 164, "duration": 571, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, train binary_classification_accuracy <score>=0.747505494505
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, train binary_classification_cross_entropy <loss>=0.520580322601
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=81, train binary_f_1.000 <score>=0.777598172544
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 573.2541084289551, "sum": 573.2541084289551, "min": 573.2541084289551}}, "EndTime": 1584056689.967961, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056689.393938}

    [03/12/2020 23:44:49 INFO 140601087346496] #progress_metric: host=algo-1, completed 82 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7463, "sum": 7463.0, "min": 7463}, "Total Records Seen": {"count": 1, "max": 7427740, "sum": 7427740.0, "min": 7427740}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 83, "sum": 83.0, "min": 83}}, "EndTime": 1584056689.968152, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 81}, "StartTime": 1584056689.39468}

    [03/12/2020 23:44:49 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157903.316069 records/second
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_classification_cross_entropy <loss>=0.547418945313
    [03/12/2020 23:44:49 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:50.565] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 166, "duration": 595, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, train binary_classification_accuracy <score>=0.747692307692
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, train binary_classification_cross_entropy <loss>=0.520209131262
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=82, train binary_f_1.000 <score>=0.777739056359
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 597.0628261566162, "sum": 597.0628261566162, "min": 597.0628261566162}}, "EndTime": 1584056690.565816, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056689.968022}

    [03/12/2020 23:44:50 INFO 140601087346496] #progress_metric: host=algo-1, completed 83 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7554, "sum": 7554.0, "min": 7554}, "Total Records Seen": {"count": 1, "max": 7518310, "sum": 7518310.0, "min": 7518310}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 84, "sum": 84.0, "min": 84}}, "EndTime": 1584056690.566112, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 82}, "StartTime": 1584056689.968721}

    [03/12/2020 23:44:50 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151577.273862 records/second
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_classification_cross_entropy <loss>=0.547163818359
    [03/12/2020 23:44:50 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:51.158] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 168, "duration": 590, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, train binary_classification_accuracy <score>=0.747714285714
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, train binary_classification_cross_entropy <loss>=0.519843400557
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=83, train binary_f_1.000 <score>=0.777736901212
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 592.724084854126, "sum": 592.724084854126, "min": 592.724084854126}}, "EndTime": 1584056691.159428, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056690.565931}

    [03/12/2020 23:44:51 INFO 140601087346496] #progress_metric: host=algo-1, completed 84 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7645, "sum": 7645.0, "min": 7645}, "Total Records Seen": {"count": 1, "max": 7608880, "sum": 7608880.0, "min": 7608880}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 85, "sum": 85.0, "min": 85}}, "EndTime": 1584056691.159674, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 83}, "StartTime": 1584056690.566674}

    [03/12/2020 23:44:51 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152701.498314 records/second
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_classification_cross_entropy <loss>=0.546911743164
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:51.733] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 170, "duration": 571, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, train binary_classification_accuracy <score>=0.747912087912
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, train binary_classification_cross_entropy <loss>=0.519482893724
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=84, train binary_f_1.000 <score>=0.777893962279
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 573.4641551971436, "sum": 573.4641551971436, "min": 573.4641551971436}}, "EndTime": 1584056691.73374, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056691.159501}

    [03/12/2020 23:44:51 INFO 140601087346496] #progress_metric: host=algo-1, completed 85 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7736, "sum": 7736.0, "min": 7736}, "Total Records Seen": {"count": 1, "max": 7699450, "sum": 7699450.0, "min": 7699450}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 86, "sum": 86.0, "min": 86}}, "EndTime": 1584056691.733936, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 84}, "StartTime": 1584056691.160249}

    [03/12/2020 23:44:51 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157846.889784 records/second
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_classification_cross_entropy <loss>=0.546662231445
    [03/12/2020 23:44:51 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:52.306] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 172, "duration": 570, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, train binary_classification_accuracy <score>=0.748098901099
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, train binary_classification_cross_entropy <loss>=0.519127388084
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=85, train binary_f_1.000 <score>=0.778052110262
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 572.5510120391846, "sum": 572.5510120391846, "min": 572.5510120391846}}, "EndTime": 1584056692.307078, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056691.733806}

    [03/12/2020 23:44:52 INFO 140601087346496] #progress_metric: host=algo-1, completed 86 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7827, "sum": 7827.0, "min": 7827}, "Total Records Seen": {"count": 1, "max": 7790020, "sum": 7790020.0, "min": 7790020}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 87, "sum": 87.0, "min": 87}}, "EndTime": 1584056692.307544, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 85}, "StartTime": 1584056691.734494}

    [03/12/2020 23:44:52 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=158015.106632 records/second
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_classification_accuracy <score>=0.727
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_classification_cross_entropy <loss>=0.546415283203
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_f_1.000 <score>=0.755156950673
    [2020-03-12 23:44:52.876] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 174, "duration": 566, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, train binary_classification_accuracy <score>=0.748175824176
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, train binary_classification_cross_entropy <loss>=0.518776638492
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=86, train binary_f_1.000 <score>=0.778122034817
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 568.9258575439453, "sum": 568.9258575439453, "min": 568.9258575439453}}, "EndTime": 1584056692.87706, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056692.307151}

    [03/12/2020 23:44:52 INFO 140601087346496] #progress_metric: host=algo-1, completed 87 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 7918, "sum": 7918.0, "min": 7918}, "Total Records Seen": {"count": 1, "max": 7880590, "sum": 7880590.0, "min": 7880590}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 88, "sum": 88.0, "min": 88}}, "EndTime": 1584056692.87724, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 86}, "StartTime": 1584056692.308108}

    [03/12/2020 23:44:52 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=159109.652209 records/second
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_classification_cross_entropy <loss>=0.546170532227
    [03/12/2020 23:44:52 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_f_1.000 <score>=0.756272401434
    [2020-03-12 23:44:53.473] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 176, "duration": 594, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, train binary_classification_accuracy <score>=0.748318681319
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, train binary_classification_cross_entropy <loss>=0.518430443397
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=87, train binary_f_1.000 <score>=0.778241462446
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 596.4639186859131, "sum": 596.4639186859131, "min": 596.4639186859131}}, "EndTime": 1584056693.474239, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056692.877109}

    [03/12/2020 23:44:53 INFO 140601087346496] #progress_metric: host=algo-1, completed 88 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8009, "sum": 8009.0, "min": 8009}, "Total Records Seen": {"count": 1, "max": 7971160, "sum": 7971160.0, "min": 7971160}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 89, "sum": 89.0, "min": 89}}, "EndTime": 1584056693.474461, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 87}, "StartTime": 1584056692.877746}

    [03/12/2020 23:44:53 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=151754.328895 records/second
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_classification_cross_entropy <loss>=0.545927978516
    [03/12/2020 23:44:53 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_f_1.000 <score>=0.756950672646
    [2020-03-12 23:44:54.054] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 178, "duration": 577, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, train binary_classification_accuracy <score>=0.748395604396
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, train binary_classification_cross_entropy <loss>=0.518088602925
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=88, train binary_f_1.000 <score>=0.778311386522
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 580.1999568939209, "sum": 580.1999568939209, "min": 580.1999568939209}}, "EndTime": 1584056694.055255, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056693.474309}

    [03/12/2020 23:44:54 INFO 140601087346496] #progress_metric: host=algo-1, completed 89 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8100, "sum": 8100.0, "min": 8100}, "Total Records Seen": {"count": 1, "max": 8061730, "sum": 8061730.0, "min": 8061730}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 90, "sum": 90.0, "min": 90}}, "EndTime": 1584056694.055489, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 88}, "StartTime": 1584056693.475022}

    [03/12/2020 23:44:54 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155998.853977 records/second
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_classification_cross_entropy <loss>=0.545687316895
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_f_1.000 <score>=0.756950672646
    [2020-03-12 23:44:54.629] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 180, "duration": 572, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, train binary_classification_accuracy <score>=0.748395604396
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, train binary_classification_cross_entropy <loss>=0.517750888363
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=89, train binary_f_1.000 <score>=0.778315679402
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 574.181079864502, "sum": 574.181079864502, "min": 574.181079864502}}, "EndTime": 1584056694.630243, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056694.055313}

    [03/12/2020 23:44:54 INFO 140601087346496] #progress_metric: host=algo-1, completed 90 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8191, "sum": 8191.0, "min": 8191}, "Total Records Seen": {"count": 1, "max": 8152300, "sum": 8152300.0, "min": 8152300}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 91, "sum": 91.0, "min": 91}}, "EndTime": 1584056694.630444, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 89}, "StartTime": 1584056694.056035}

    [03/12/2020 23:44:54 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=157646.574643 records/second
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_classification_cross_entropy <loss>=0.545448425293
    [03/12/2020 23:44:54 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_f_1.000 <score>=0.756950672646
    [2020-03-12 23:44:55.270] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 182, "duration": 637, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, train binary_classification_accuracy <score>=0.748604395604
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, train binary_classification_cross_entropy <loss>=0.517417129014
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=90, train binary_f_1.000 <score>=0.778488918152
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 639.5080089569092, "sum": 639.5080089569092, "min": 639.5080089569092}}, "EndTime": 1584056695.270604, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056694.630317}

    [03/12/2020 23:44:55 INFO 140601087346496] #progress_metric: host=algo-1, completed 91 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8282, "sum": 8282.0, "min": 8282}, "Total Records Seen": {"count": 1, "max": 8242870, "sum": 8242870.0, "min": 8242870}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 92, "sum": 92.0, "min": 92}}, "EndTime": 1584056695.270817, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 90}, "StartTime": 1584056694.63107}

    [03/12/2020 23:44:55 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=141546.185235 records/second
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_classification_cross_entropy <loss>=0.54521105957
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:55.870] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 184, "duration": 597, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, train binary_classification_accuracy <score>=0.748703296703
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, train binary_classification_cross_entropy <loss>=0.517087121314
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=91, train binary_f_1.000 <score>=0.778586781821
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 599.9929904937744, "sum": 599.9929904937744, "min": 599.9929904937744}}, "EndTime": 1584056695.871397, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056695.270665}

    [03/12/2020 23:44:55 INFO 140601087346496] #progress_metric: host=algo-1, completed 92 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8373, "sum": 8373.0, "min": 8373}, "Total Records Seen": {"count": 1, "max": 8333440, "sum": 8333440.0, "min": 8333440}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 93, "sum": 93.0, "min": 93}}, "EndTime": 1584056695.871652, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 91}, "StartTime": 1584056695.271371}

    [03/12/2020 23:44:55 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=150851.956931 records/second
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_classification_cross_entropy <loss>=0.544975219727
    [03/12/2020 23:44:55 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:56.478] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 186, "duration": 605, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, train binary_classification_accuracy <score>=0.748791208791
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, train binary_classification_cross_entropy <loss>=0.516760687524
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=92, train binary_f_1.000 <score>=0.778659953524
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 606.8220138549805, "sum": 606.8220138549805, "min": 606.8220138549805}}, "EndTime": 1584056696.479066, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056695.871479}

    [03/12/2020 23:44:56 INFO 140601087346496] #progress_metric: host=algo-1, completed 93 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8464, "sum": 8464.0, "min": 8464}, "Total Records Seen": {"count": 1, "max": 8424010, "sum": 8424010.0, "min": 8424010}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 94, "sum": 94.0, "min": 94}}, "EndTime": 1584056696.479316, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 92}, "StartTime": 1584056695.872212}

    [03/12/2020 23:44:56 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=149156.646385 records/second
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_classification_accuracy <score>=0.728
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_classification_cross_entropy <loss>=0.544740600586
    [03/12/2020 23:44:56 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_f_1.000 <score>=0.755834829443
    [2020-03-12 23:44:57.074] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 188, "duration": 593, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, train binary_classification_accuracy <score>=0.748879120879
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, train binary_classification_cross_entropy <loss>=0.516437652588
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=93, train binary_f_1.000 <score>=0.778754550383
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 594.8388576507568, "sum": 594.8388576507568, "min": 594.8388576507568}}, "EndTime": 1584056697.074744, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056696.479146}

    [03/12/2020 23:44:57 INFO 140601087346496] #progress_metric: host=algo-1, completed 94 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8555, "sum": 8555.0, "min": 8555}, "Total Records Seen": {"count": 1, "max": 8514580, "sum": 8514580.0, "min": 8514580}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 95, "sum": 95.0, "min": 95}}, "EndTime": 1584056697.074968, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 93}, "StartTime": 1584056696.479879}

    [03/12/2020 23:44:57 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152169.395357 records/second
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_classification_accuracy <score>=0.729
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_classification_cross_entropy <loss>=0.544507080078
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_f_1.000 <score>=0.756513926325
    [2020-03-12 23:44:57.656] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 190, "duration": 579, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, train binary_classification_accuracy <score>=0.749098901099
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, train binary_classification_cross_entropy <loss>=0.516117834741
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=94, train binary_f_1.000 <score>=0.778931061193
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 581.7470550537109, "sum": 581.7470550537109, "min": 581.7470550537109}}, "EndTime": 1584056697.65734, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056697.074813}

    [03/12/2020 23:44:57 INFO 140601087346496] #progress_metric: host=algo-1, completed 95 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8646, "sum": 8646.0, "min": 8646}, "Total Records Seen": {"count": 1, "max": 8605150, "sum": 8605150.0, "min": 8605150}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 96, "sum": 96.0, "min": 96}}, "EndTime": 1584056697.657543, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 94}, "StartTime": 1584056697.075564}

    [03/12/2020 23:44:57 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=155595.86051 records/second
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_classification_cross_entropy <loss>=0.544274597168
    [03/12/2020 23:44:57 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_f_1.000 <score>=0.75763016158
    [2020-03-12 23:44:58.269] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 192, "duration": 610, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, train binary_classification_accuracy <score>=0.749164835165
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, train binary_classification_cross_entropy <loss>=0.515801076365
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=95, train binary_f_1.000 <score>=0.778976315433
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 612.2238636016846, "sum": 612.2238636016846, "min": 612.2238636016846}}, "EndTime": 1584056698.270367, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056697.657407}

    [03/12/2020 23:44:58 INFO 140601087346496] #progress_metric: host=algo-1, completed 96 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8737, "sum": 8737.0, "min": 8737}, "Total Records Seen": {"count": 1, "max": 8695720, "sum": 8695720.0, "min": 8695720}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 97, "sum": 97.0, "min": 97}}, "EndTime": 1584056698.270554, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 95}, "StartTime": 1584056697.658115}

    [03/12/2020 23:44:58 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=147859.269598 records/second
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_classification_accuracy <score>=0.73
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_classification_cross_entropy <loss>=0.544042907715
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_f_1.000 <score>=0.75763016158

    2020-03-12 23:45:10 Uploading - Uploading generated training model
    2020-03-12 23:45:10 Completed - Training job completed
    [2020-03-12 23:44:58.857] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 194, "duration": 584, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, train binary_classification_accuracy <score>=0.749252747253
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, train binary_classification_cross_entropy <loss>=0.515487213135
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=96, train binary_f_1.000 <score>=0.779045221265
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 586.4729881286621, "sum": 586.4729881286621, "min": 586.4729881286621}}, "EndTime": 1584056698.857648, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056698.270427}

    [03/12/2020 23:44:58 INFO 140601087346496] #progress_metric: host=algo-1, completed 97 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8828, "sum": 8828.0, "min": 8828}, "Total Records Seen": {"count": 1, "max": 8786290, "sum": 8786290.0, "min": 8786290}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 98, "sum": 98.0, "min": 98}}, "EndTime": 1584056698.857896, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 96}, "StartTime": 1584056698.271131}

    [03/12/2020 23:44:58 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154322.691244 records/second
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_classification_cross_entropy <loss>=0.543812011719
    [03/12/2020 23:44:58 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_f_1.000 <score>=0.758744394619
    [2020-03-12 23:44:59.450] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 196, "duration": 590, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, train binary_classification_accuracy <score>=0.749395604396
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, train binary_classification_cross_entropy <loss>=0.515176092798
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=97, train binary_f_1.000 <score>=0.779177519778
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 592.0751094818115, "sum": 592.0751094818115, "min": 592.0751094818115}}, "EndTime": 1584056699.45054, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056698.857722}

    [03/12/2020 23:44:59 INFO 140601087346496] #progress_metric: host=algo-1, completed 98 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 8919, "sum": 8919.0, "min": 8919}, "Total Records Seen": {"count": 1, "max": 8876860, "sum": 8876860.0, "min": 8876860}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 99, "sum": 99.0, "min": 99}}, "EndTime": 1584056699.450729, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 97}, "StartTime": 1584056698.858439}

    [03/12/2020 23:44:59 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=152889.128381 records/second
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_classification_cross_entropy <loss>=0.543581665039
    [03/12/2020 23:44:59 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_f_1.000 <score>=0.758744394619
    [2020-03-12 23:45:00.027] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 198, "duration": 574, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, train binary_classification_accuracy <score>=0.749538461538
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, train binary_classification_cross_entropy <loss>=0.514867560083
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=98, train binary_f_1.000 <score>=0.779284164859
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 576.624870300293, "sum": 576.624870300293, "min": 576.624870300293}}, "EndTime": 1584056700.027959, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056699.450601}

    [03/12/2020 23:45:00 INFO 140601087346496] #progress_metric: host=algo-1, completed 99 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 9010, "sum": 9010.0, "min": 9010}, "Total Records Seen": {"count": 1, "max": 8967430, "sum": 8967430.0, "min": 8967430}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 100, "sum": 100.0, "min": 100}}, "EndTime": 1584056700.028188, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 98}, "StartTime": 1584056699.451301}

    [03/12/2020 23:45:00 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=156966.773223 records/second
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_classification_accuracy <score>=0.731
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_classification_cross_entropy <loss>=0.543351806641
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_f_1.000 <score>=0.759176365264
    [2020-03-12 23:45:00.613] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 200, "duration": 583, "num_examples": 91, "num_bytes": 5796480}
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, train binary_classification_accuracy <score>=0.74967032967
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, train binary_classification_cross_entropy <loss>=0.51456147414
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, epoch=99, train binary_f_1.000 <score>=0.779400371862
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, train binary_classification_accuracy <score>=0.74967032967
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, train binary_classification_cross_entropy <loss>=0.51456147414
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, train binary_f_1.000 <score>=0.779400371862
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 586.0419273376465, "sum": 586.0419273376465, "min": 586.0419273376465}}, "EndTime": 1584056700.614928, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056700.02803}

    [03/12/2020 23:45:00 INFO 140601087346496] #progress_metric: host=algo-1, completed 100 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Batches Since Last Reset": {"count": 1, "max": 91, "sum": 91.0, "min": 91}, "Number of Records Since Last Reset": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Total Batches Seen": {"count": 1, "max": 9101, "sum": 9101.0, "min": 9101}, "Total Records Seen": {"count": 1, "max": 9058000, "sum": 9058000.0, "min": 9058000}, "Max Records Seen Between Resets": {"count": 1, "max": 90570, "sum": 90570.0, "min": 90570}, "Reset Count": {"count": 1, "max": 101, "sum": 101.0, "min": 101}}, "EndTime": 1584056700.615169, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 99}, "StartTime": 1584056700.028834}

    [03/12/2020 23:45:00 INFO 140601087346496] #throughput_metric: host=algo-1, train throughput=154434.804057 records/second
    [03/12/2020 23:45:00 WARNING 140601087346496] wait_for_all_workers will not sync workers since the kv store is not running distributed
    [03/12/2020 23:45:00 INFO 140601087346496] Pulling entire model from kvstore to finalize
    #metrics {"Metrics": {"finalize.time": {"count": 1, "max": 2.2830963134765625, "sum": 2.2830963134765625, "min": 2.2830963134765625}}, "EndTime": 1584056700.617767, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056700.615006}

    [03/12/2020 23:45:00 INFO 140601087346496] Saved checkpoint to "/tmp/tmpXdIgpm/state-0001.params"
    [2020-03-12 23:45:00.632] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/test", "epoch": 0, "duration": 59774, "num_examples": 1, "num_bytes": 64000}
    [2020-03-12 23:45:00.674] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/test", "epoch": 1, "duration": 41, "num_examples": 10, "num_bytes": 603520}
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 10, "sum": 10.0, "min": 10}, "Number of Batches Since Last Reset": {"count": 1, "max": 10, "sum": 10.0, "min": 10}, "Number of Records Since Last Reset": {"count": 1, "max": 9430, "sum": 9430.0, "min": 9430}, "Total Batches Seen": {"count": 1, "max": 10, "sum": 10.0, "min": 10}, "Total Records Seen": {"count": 1, "max": 9430, "sum": 9430.0, "min": 9430}, "Max Records Seen Between Resets": {"count": 1, "max": 9430, "sum": 9430.0, "min": 9430}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1584056700.674291, "Dimensions": {"Host": "algo-1", "Meta": "test_data_iter", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056700.632219}

    [03/12/2020 23:45:00 INFO 140601087346496] #test_score (algo-1) : ('binary_classification_accuracy', 0.6963944856839873)
    [03/12/2020 23:45:00 INFO 140601087346496] #test_score (algo-1) : ('binary_classification_cross_entropy', 0.5776413530206124)
    [03/12/2020 23:45:00 INFO 140601087346496] #test_score (algo-1) : ('binary_f_1.000', 0.740599800670472)
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, test binary_classification_accuracy <score>=0.696394485684
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, test binary_classification_cross_entropy <loss>=0.577641353021
    [03/12/2020 23:45:00 INFO 140601087346496] #quality_metric: host=algo-1, test binary_f_1.000 <score>=0.74059980067
    #metrics {"Metrics": {"totaltime": {"count": 1, "max": 59878.23796272278, "sum": 59878.23796272278, "min": 59878.23796272278}, "setuptime": {"count": 1, "max": 50.91285705566406, "sum": 50.91285705566406, "min": 50.91285705566406}}, "EndTime": 1584056700.675261, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1584056700.617836}

    Training seconds: 112
    Billable seconds: 112
    CPU times: user 655 ms, sys: 44.9 ms, total: 700 ms
    Wall time: 4min 12s

### 배포
배포 또한, 매우 간단하게 `deploy()` 메소드로 수행하실 수 있습니다. 배포는 약 5분에서 10분이 소요됩니다.

```python
%%time
instance_type_inference = 'ml.m5.large'
fm_predictor = fm.deploy(instance_type=instance_type_inference, initial_instance_count=1)
```

    -----------!CPU times: user 204 ms, sys: 8.26 ms, total: 213 ms
    Wall time: 5min 31s

```python
def fm_serializer(data):
    js = {'instances': []}
    for row in data:
        js['instances'].append({'features': row.tolist()})
    #print js
    return json.dumps(js)

fm_predictor.content_type = 'application/json'
fm_predictor.serializer = fm_serializer
fm_predictor.deserializer = json_deserializer
```

```python
result = fm_predictor.predict(X_test[1000:1010].toarray())
print(result)
print (Y_test[1000:1010])
```

    {'predictions': [{'score': 0.6734210848808289, 'predicted_label': 1.0}, {'score': 0.19272500276565552, 'predicted_label': 0.0}, {'score': 0.23256689310073853, 'predicted_label': 0.0}, {'score': 0.608762264251709, 'predicted_label': 1.0}, {'score': 0.5399172306060791, 'predicted_label': 1.0}, {'score': 0.15502098202705383, 'predicted_label': 0.0}, {'score': 0.38825932145118713, 'predicted_label': 0.0}, {'score': 0.5133833289146423, 'predicted_label': 1.0}, {'score': 0.3564694821834564, 'predicted_label': 0.0}, {'score': 0.11903449147939682, 'predicted_label': 0.0}]}
    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]

#### 지금까지 기본적인 사용법을 알아보았으며, 여기에서 실습을 종료하셔도 됩니다. 실습을 일찍 끝내셨거나, 좀 더 깊은 내용을 원하신다면 아래 셀들을 순차적으로 실행해 주세요. ####

#### [주의] 실시간 예측을 제공하기 위해 엔드포인트를 계속 실행할 필요가 없는 경우, 과금을 막기 위해 엔드포인트를 삭제해 주세요. ####

## 3. (Optional) top-k 추천을 위하여 FM 모델의 모델 파라메터를 사용하여 knn으로 훈련 및 배포하기
---

이제 SageMaker에 모델을 작성하고 저장 했으므로 동일한 FM 모델을 다운로드하여 KNN 모델에 맞게 다시 패키지 할 수 있습니다.

### 모델 아티팩트 다운로드

```python
#!pip install mxnet # 필요한 경우 주석을 해제하여 mxnet을 설치해 주세요
import mxnet as mx
model_file_name = "model.tar.gz"
model_full_path = fm.output_path + "/" + fm.latest_training_job.job_name + "/output/" + model_file_name
print("Model Path: ", model_full_path)

# FM 모델 아티팩트(model.tar.gz) 다운로드
os.system("aws s3 cp " + model_full_path+ " .")

# 모델 아티팩트 압축 해제
os.system("tar xzvf " + model_file_name)
os.system("unzip -o model_algo-1")
os.system("mv symbol.json model-symbol.json")
os.system("mv params model-0000.params")
```

    Model Path:  s3://sagemaker-us-east-1-143656149352/fm-hol/output/factorization-machines-2020-03-12-23-41-11-305/output/model.tar.gz

    0

### 모델 데이터 분리

FM에서 훈련한 파라메터 튜플 ($w_{0}, \mathbf{w}, \mathbf{V}$)을 가져옵니다.

```python
# 모델 추출
m = mx.module.Module.load('./model', 0, False, label_names=['out_label'])
V = m._arg_params['v'].asnumpy() # 2625 x 64
w = m._arg_params['w1_weight'].asnumpy() # 2625 x1
b = m._arg_params['w0_weight'].asnumpy() # 1
print(V.shape, w.shape, b.shape)
```

    (2625, 64) (2625, 1) (1,)

### 데이터셋 재가공

이제 FM 모델에서 추출한 모델 파라메터를 다시 패키지하여 k-NN 모델을 훈련하기 위한 준비를 수행해 보겠습니다. W
이 프로세스는 두 개의 데이터셋을 생성합니다.

- Item latent 행렬: k-NN 모델 학습에 사용; $a_i = concat(V, \; w)$
- User latent 행렬: 추론에 사용; $a_u = concat(V, \; 1)$

참고로, 본 핸즈온 코드는 user 및 item ID가 있는 시나리오에만 적용됩니다. 그러나 실제 데이터에는 추가 메타데이터(예: user의 경우 나이, 우편번호, 성별이 포함되고 영화의 경우 영화 장르, 주요 키워드)들이 포함될 수 있습니다. 이러한 경우에는 아래 방법으로 user 및 item 벡터를 추출할 수 있습니다.

- item과 item feature를 $x_i$로 인코딩 후 $\mathbf{V}, \mathbf{w}$에 내적; $a_i = concat(V^T \cdot x_i , \; w^T \cdot x_i)$
- user와 user feature를 $x_u$로 인코딩 후 $\mathbf{V}$에 내적; $a_u = concat(V^T \cdot x_u, \; , 1)$

$a_i$를 사용하여 k-NN 모델을 훈련하고 $a_u$를 사용하여 추론을 수행하시면 됩니다.

```python
# item latent matrix - concat(V[i], w[i]).
knn_item_matrix = np.concatenate((V[nb_users:], w[nb_users:]), axis=1) # 1682 x 65
knn_train_label = np.arange(1,nb_movies+1) # [1, 2, 3, ..., 1681, 1682]

# user latent matrix - concat (V[u], 1)
ones = np.ones(nb_users).reshape((nb_users, 1)) # 943x1
knn_user_matrix = np.concatenate((V[:nb_users], ones), axis=1) # 943 x 65
```

### k-NN 모델 훈련

k-NN 모델은 기본 index_type (faiss.Flat)을 사용합니다.  대규모 데이터셋의 경우 속도가 느려지기에, 이런 경우 더 빠른 훈련을 위해 다른 index_type 매개 변수를 사용할 수 있습니다. index 유형에 대한 자세한 내용은 k-NN 설명서를 참조해 주세요.

```python
print('KNN train features shape = ', knn_item_matrix.shape)
knn_prefix = 'knn'
knn_output_prefix  = 's3://{}/{}/output'.format(bucket, knn_prefix)
knn_train_data_path = writeDatasetToProtobuf(knn_item_matrix, bucket, knn_prefix, train_key, "dense", knn_train_label)
print('Uploaded KNN train data: {}'.format(knn_train_data_path))
```

    KNN train features shape =  (1682, 65)
    Uploaded KNN train data: s3://sagemaker-us-east-1-143656149352/knn/train.protobuf

```python
nb_recommendations = 100

knn = sagemaker.estimator.Estimator(get_image_uri(boto3.Session().region_name, "knn"),
    get_execution_role(),
    train_instance_count=1,
    train_instance_type=instance_type_training,
    output_path=knn_output_prefix,
    sagemaker_session=sagemaker.Session())

knn.set_hyperparameters(feature_dim=knn_item_matrix.shape[1], k=nb_recommendations,
                        index_metric="INNER_PRODUCT", predictor_type='classifier', sample_size=200000)
fit_input = {'train': knn_train_data_path}
```

훈련을 시작합니다. 아래 셀의 수행 시간은 약 4분에서 5분이 소요됩니다.

```python
%%time
knn.fit(fit_input)
knn_model_name =  knn.latest_training_job.job_name
print("Created model: ", knn_model_name)
```

    2020-03-12 23:57:36 Starting - Starting the training job...
    2020-03-12 23:57:38 Starting - Launching requested ML instances......
    2020-03-12 23:59:06 Starting - Preparing the instances for training.........
    2020-03-13 00:00:13 Downloading - Downloading input data...
    2020-03-13 00:01:06 Training - Training image download completed. Training in progress.
    2020-03-13 00:01:06 Uploading - Uploading generated training modelDocker entrypoint called with argument(s): train
    [03/13/2020 00:00:59 INFO 139648235816768] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'index_metric': u'L2', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'_log_level': u'info', u'faiss_index_ivf_nlists': u'auto', u'epochs': u'1', u'index_type': u'faiss.Flat', u'_faiss_index_nprobe': u'5', u'_kvstore': u'dist_async', u'_num_kv_servers': u'1', u'mini_batch_size': u'5000'}
    [03/13/2020 00:00:59 INFO 139648235816768] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'sample_size': u'200000', u'feature_dim': u'65', u'index_metric': u'INNER_PRODUCT', u'predictor_type': u'classifier', u'k': u'100'}
    [03/13/2020 00:00:59 INFO 139648235816768] Final configuration: {u'index_metric': u'INNER_PRODUCT', u'predictor_type': u'classifier', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'_log_level': u'info', u'feature_dim': u'65', u'faiss_index_ivf_nlists': u'auto', u'sample_size': u'200000', u'epochs': u'1', u'index_type': u'faiss.Flat', u'_faiss_index_nprobe': u'5', u'_kvstore': u'dist_async', u'_num_kv_servers': u'1', u'mini_batch_size': u'5000', u'k': u'100'}
    [03/13/2020 00:00:59 WARNING 139648235816768] Loggers have already been setup.
    [03/13/2020 00:00:59 INFO 139648235816768] Launching parameter server for role scheduler
    [03/13/2020 00:00:59 INFO 139648235816768] {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/670be1d0-5bba-46dc-a26a-d225b9d41dd1', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-03-12-23-57-36-380', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-151-112.ec2.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/81c7b043-acb4-4822-9777-e5e116db6d91', 'PWD': '/', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:us-east-1:143656149352:training-job/knn-2020-03-12-23-57-36-380', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [03/13/2020 00:00:59 INFO 139648235816768] envs={'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/670be1d0-5bba-46dc-a26a-d225b9d41dd1', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '1', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.151.112', 'AWS_REGION': 'us-east-1', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-03-12-23-57-36-380', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-151-112.ec2.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/81c7b043-acb4-4822-9777-e5e116db6d91', 'DMLC_ROLE': 'scheduler', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:us-east-1:143656149352:training-job/knn-2020-03-12-23-57-36-380', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [03/13/2020 00:00:59 INFO 139648235816768] Launching parameter server for role server
    [03/13/2020 00:00:59 INFO 139648235816768] {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/670be1d0-5bba-46dc-a26a-d225b9d41dd1', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-03-12-23-57-36-380', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-151-112.ec2.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/81c7b043-acb4-4822-9777-e5e116db6d91', 'PWD': '/', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:us-east-1:143656149352:training-job/knn-2020-03-12-23-57-36-380', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [03/13/2020 00:00:59 INFO 139648235816768] envs={'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/670be1d0-5bba-46dc-a26a-d225b9d41dd1', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '1', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.151.112', 'AWS_REGION': 'us-east-1', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-03-12-23-57-36-380', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-151-112.ec2.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/81c7b043-acb4-4822-9777-e5e116db6d91', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:us-east-1:143656149352:training-job/knn-2020-03-12-23-57-36-380', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [03/13/2020 00:00:59 INFO 139648235816768] Environment: {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/670be1d0-5bba-46dc-a26a-d225b9d41dd1', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '1', 'SAGEMAKER_HTTP_PORT': '8080', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.151.112', 'AWS_REGION': 'us-east-1', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-03-12-23-57-36-380', 'HOME': '/root', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-151-112.ec2.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/81c7b043-acb4-4822-9777-e5e116db6d91', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:us-east-1:143656149352:training-job/knn-2020-03-12-23-57-36-380', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    Process 59 is a shell:scheduler.
    Process 68 is a shell:server.
    Process 1 is a worker.
    [03/13/2020 00:00:59 INFO 139648235816768] Using default worker.
    [03/13/2020 00:00:59 INFO 139648235816768] nvidia-smi took: 0.0252020359039 secs to identify 0 gpus
    [03/13/2020 00:00:59 INFO 139648235816768] Create Store: dist_async
    [03/13/2020 00:01:00 ERROR 139648235816768] nvidia-smi: failed to run (127): /bin/sh: nvidia-smi: command not found
    [03/13/2020 00:01:00 INFO 139648235816768] Using per-worker sample size = 200000 (Available virtual memory = 6242377728 bytes, GPU free memory = 0 bytes, number of workers = 1). If an out-of-memory error occurs, choose a larger instance type, use dimension reduction, decrease sample_size, and/or decrease mini_batch_size.
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Records Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Max Records Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Reset Count": {"count": 1, "max": 0, "sum": 0.0, "min": 0}}, "EndTime": 1584057660.947634, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1584057660.947584}

    [2020-03-13 00:01:00.947] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 0, "duration": 1043, "num_examples": 1, "num_bytes": 518056}
    [2020-03-13 00:01:00.972] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 1, "duration": 24, "num_examples": 1, "num_bytes": 518056}
    [03/13/2020 00:01:00 INFO 139648235816768] push reservoir to kv... 1 num_workers 0 rank
    [03/13/2020 00:01:00 INFO 139648235816768] ...done (1682)
    [03/13/2020 00:01:00 INFO 139648235816768] #progress_metric: host=algo-1, completed 100 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 1682, "sum": 1682.0, "min": 1682}, "Total Batches Seen": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Total Records Seen": {"count": 1, "max": 1682, "sum": 1682.0, "min": 1682}, "Max Records Seen Between Resets": {"count": 1, "max": 1682, "sum": 1682.0, "min": 1682}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1584057660.993015, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/KNN", "epoch": 0}, "StartTime": 1584057660.948006}

    [03/13/2020 00:01:00 INFO 139648235816768] #throughput_metric: host=algo-1, train throughput=37196.7991901 records/second
    [03/13/2020 00:01:01 INFO 139648235816768] pulled row count... worker 0 rows 1682
    [03/13/2020 00:01:01 INFO 139648235816768] pulled... worker 0 data (1682, 65) labels (1682, 1) nans 0
    [03/13/2020 00:01:01 INFO 139648235816768] calling index.train...
    [03/13/2020 00:01:01 INFO 139648235816768] ...done calling index.train
    [03/13/2020 00:01:01 INFO 139648235816768] calling index.add...
    [03/13/2020 00:01:01 INFO 139648235816768] ...done calling index.add
    #metrics {"Metrics": {"epochs": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "model.serialize.time": {"count": 1, "max": 108.03389549255371, "sum": 108.03389549255371, "min": 108.03389549255371}, "finalize.time": {"count": 1, "max": 149.07002449035645, "sum": 149.07002449035645, "min": 149.07002449035645}, "initialize.time": {"count": 1, "max": 1033.7038040161133, "sum": 1033.7038040161133, "min": 1033.7038040161133}, "update.time": {"count": 1, "max": 44.6629524230957, "sum": 44.6629524230957, "min": 44.6629524230957}}, "EndTime": 1584057661.250632, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1584057659.903659}

    [03/13/2020 00:01:01 INFO 139648235816768] Test data is not provided.
    #metrics {"Metrics": {"totaltime": {"count": 1, "max": 1811.3501071929932, "sum": 1811.3501071929932, "min": 1811.3501071929932}, "setuptime": {"count": 1, "max": 21.845102310180664, "sum": 21.845102310180664, "min": 21.845102310180664}}, "EndTime": 1584057661.253665, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1584057661.250723}

    2020-03-13 00:01:12 Completed - Training job completed
    Training seconds: 59
    Billable seconds: 59
    Created model:  knn-2020-03-12-23-57-36-380
    CPU times: user 513 ms, sys: 21.1 ms, total: 534 ms
    Wall time: 4min 12s

배치 추론에서 참조할 수 있도록 모델을 생성합니다.

```python
# 다음 단계에서 배치 추론 중에 참조할 수 있도록 모델 저장
sm = boto3.client(service_name='sagemaker')
primary_container = {
    'Image': knn.image_name,
    'ModelDataUrl': knn.model_data,
}

knn_model = sm.create_model(
        ModelName = knn.latest_training_job.job_name,
        ExecutionRoleArn = knn.role,
        PrimaryContainer = primary_container)
```

### Batch Transform

Amazon SageMaker의 Batch Transform 기능을 사용하면 대규모로 배치 추론 결과를 생성할 수 있습니다.

아래 셀의 실행이 완료되기까지는 약 4분 소요됩니다.

```python
%%time
# 추론 데이터 S3에 업로드
knn_batch_data_path = writeDatasetToProtobuf(knn_user_matrix, bucket, knn_prefix, train_key, "dense")
print("Batch inference data path: ", knn_batch_data_path)

# Transformer 객체 초기화
transformer = sagemaker.transformer.Transformer(
    base_transform_job_name="knn",
    model_name=knn_model_name,
    instance_count=1,
    instance_type=instance_type_inference,
    output_path=knn_output_prefix,
    accept="application/jsonlines; verbose=true"
)

# 변환 작업 시작
transformer.transform(knn_batch_data_path, content_type='application/x-recordio-protobuf')
transformer.wait()

# S3에서 출력 파일 다운로드
results_file_name = "inference_output"
inference_output_file = "knn/output/train.protobuf.out"
s3_client = boto3.client('s3')
s3_client.download_file(bucket, inference_output_file, results_file_name)
with open(results_file_name) as f:
    results = f.readlines()
```

    Batch inference data path:  s3://sagemaker-us-east-1-143656149352/knn/train.protobuf
    ..................Docker entrypoint called with argument(s): serve
    [03/13/2020 00:04:37 INFO 140135089923904] loaded entry point class algorithm.serve.server_config:config_api
    [03/13/2020 00:04:37 INFO 140135089923904] loading entry points
    [03/13/2020 00:04:37 INFO 140135089923904] loaded request iterator text/csv
    [03/13/2020 00:04:37 INFO 140135089923904] loaded request iterator application/x-recordio-protobuf
    [03/13/2020 00:04:37 INFO 140135089923904] loaded request iterator application/json
    [03/13/2020 00:04:37 INFO 140135089923904] loaded request iterator application/jsonlines
    [03/13/2020 00:04:37 INFO 140135089923904] loaded response encoder application/x-recordio-protobuf
    [03/13/2020 00:04:37 INFO 140135089923904] loaded response encoder application/json
    [03/13/2020 00:04:37 INFO 140135089923904] loaded response encoder application/jsonlines
    [03/13/2020 00:04:38 INFO 140135089923904] loaded entry point class algorithm:model
    [03/13/2020 00:04:38 INFO 140135089923904] Number of server workers: 1
    [2020-03-13 00:04:38 +0000] [1] [INFO] Starting gunicorn 19.7.1
    [2020-03-13 00:04:38 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
    [2020-03-13 00:04:38 +0000] [1] [INFO] Using worker: sync
    [2020-03-13 00:04:38 +0000] [66] [INFO] Booting worker with pid: 66
    [03/13/2020 00:04:38 INFO 140135089923904] loading model...
    [03/13/2020 00:04:38 INFO 140135089923904] nvidia-smi took: 0.02516913414 secs to identify 0 gpus
    [03/13/2020 00:04:38 INFO 140135089923904] ...model loaded.
    #metrics {"Metrics": {"execution_parameters.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1584057902.590777, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1584057878.075725}

    [2020-03-13 00:05:02.964] [tensorio] [info] epoch_stats={"data_pipeline": "/tmp/tmpAsBJks/tmpmGPuw6", "epoch": 0, "duration": 154, "num_examples": 1, "num_bytes": 516764}
    #metrics {"Metrics": {"model.evaluate.time": {"count": 1, "max": 153.21803092956543, "sum": 153.21803092956543, "min": 153.21803092956543}, "invocations.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1584057903.118452, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1584057902.590928}

    2020-03-13T00:05:02.596:[sagemaker logs]: MaxConcurrentTransforms=1, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD
    CPU times: user 607 ms, sys: 36.1 ms, total: 643 ms
    Wall time: 3min 42s

### top-k 추론 예시
배치 추론 결과에서 90번 사용자의 추천 영화를 확인해 보겠습니다. 결과 데이터프레임의 1번째 행은 영화 id, 2번째 행은 영화 제목, 3번째 행은
유사도입니다.

```python
def get_movie_title(movie_id):
    movie_id = int(movie_id)
    return items.iloc[movie_id]['TITLE']

import json
test_user_idx = 89 # 인덱스는 0부터 시작하므로 90번 사용자의 인덱스는 89입니다.
u_one_json = json.loads(results[test_user_idx])

items = pd.read_csv('./ml-100k/u.item', sep='|', usecols=[0,1], encoding='latin-1', names=['ITEM_ID', 'TITLE'], index_col='ITEM_ID')

movie_id_list = [int(movie_id) for movie_id in u_one_json['labels']]
movie_dist_list = [round(distance, 4) for distance in u_one_json['distances']]
movie_title_list = [get_movie_title(movie_id) for movie_id in movie_id_list]

recommend_df = pd.DataFrame({'movie_id': movie_id_list,
                             'movie_title': movie_title_list,
                             'movie_dist': movie_dist_list})
print("Recommendations for user: ", test_user_idx)
recommend_df.head(30)
```

    Recommendations for user:  89

      movie_id
      movie_title
      movie_dist

      0
      423
      Children of the Corn: The Gathering (1996)
      2.5400

      1
      165
      Manon of the Spring (Manon des sources) (1986)
      2.5569

      2
      505
      Rebel Without a Cause (1955)
      2.5701

      3
      87
      Sleepless in Seattle (1993)
      2.5762

      4
      966
      Little Lord Fauntleroy (1936)
      2.5800

      5
      429
      Duck Soup (1933)
      2.5837

      6
      655
      M (1931)
      2.5857

      7
      89
      So I Married an Axe Murderer (1993)
      2.5864

      8
      69
      Four Weddings and a Funeral (1994)
      2.5917

      9
      23
      Rumble in the Bronx (1995)
      2.5937

      10
      193
      Sting, The (1973)
      2.6060

      11
      28
      Batman Forever (1995)
      2.6110

      12
      527
      Killing Fields, The (1984)
      2.6517

      13
      208
      This Is Spinal Tap (1984)
      2.6655

      14
      166
      Private Benjamin (1980)
      2.6795

      15
      705
      Bad Moon (1996)
      2.6868

      16
      509
      Magnificent Seven, The (1954)
      2.7184

      17
      923
      White Squall (1996)
      2.7192

      18
      180
      Return of the Jedi (1983)
      2.7219

      19
      268
      Full Monty, The (1997)
      2.7302

      20
      482
      Casablanca (1942)
      2.7364

      21
      176
      Good, The Bad and The Ugly, The (1966)
      2.7397

      22
      183
      Army of Darkness (1993)
      2.7447

      23
      83
      Robert A. Heinlein's The Puppet Masters (1994)
      2.7554

      24
      269
      Gattaca (1997)
      2.7628

      25
      173
      Raiders of the Lost Ark (1981)
      2.7714

      26
      216
      Bram Stoker's Dracula (1992)
      2.8293

      27
      124
      Phenomenon (1996)
      2.8682

      28
      520
      Deer Hunter, The (1978)
      2.8879

      29
      659
      Fried Green Tomatoes (1991)
      2.8886