# SageMaker Factorization Machine(FM) 및 KNN으로 추천 시스템 구축하기

**본 노트북은 김대근님의 노트북 내용을 많이 가지고 왔습니다.**

기존 movielens 데이타를 에어라인 리뷰 데이터로 교체하고 이를 가공하는 부분을 추가 하였습니다.
알고리즘 설명 및 코드는 거의 유사합니다.
- https://github.com/daekeun-ml/recommendation-workshop/blob/master/0.Recommendation-System-FM-KNN.ipynb

*본 노트북 예제는 AWS 머신 러닝 블로그에 기고된 글들에 기반하여 SageMaker의 Factorization Machine(FM)으로 추천 시스템을 구축하는 방법을 설명합니다.*

References
- [Build a movie recommender with factorization machines on Amazon SageMaker](https://aws.amazon.com/ko/blogs/machine-learning/build-a-movie-recommender-with-factorization-machines-on-amazon-sagemaker/)
- [Amazon SageMaker Factorization Machines 알고리즘을 확장하여 추천 시스템 구현하기](https://aws.amazon.com/ko/blogs/korea/extending-amazon-sagemaker-factorization-machines-algorithm-to-predict-top-x-recommendations/)
- [Factorization Machine 논문](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)

#### 이 노트북은 아래와 같은 작업을 합니다.
1. FM 알고리즘 학습, 배포 및 추론
- 에어라인 리뷰 데이터 다운로드
- 데이터 전처리
    - 필요한 컬럼을 추출하여 interaction data set 형태로 만들기 (timestamp, user_id, item_id, rating)
    - 상호작용이 5개 이상인 유저만 추출
    - 유저, 아이템을 문자열에서 숫자로 변환
    - 데이터를 유저 기준으로 시간순으로 정렬한 후에 학습, 검증으로 8:2 로 분리
    - 학습의 인스턴스 * 컬럼의 갯수 대비 실제 사용한 데이터 영역을 알아 봅니다. (희소성, Sparsity)
    - 원핫 인코딩으로 희소 행렬 변환
        - 점수가 8점 이상이면 1, 미만이면 0으로 하여 y 레이블 생성
    - Protobuf 포맷으로 변환 후에 S3에 저장
- FM 학습 (이진 분류 문제)
    - Python SDK SageMaker2.0에 맞게 변경
- 모델 엔드포인트 배포
- 커스텀 시리얼라이즈 구현 (Python SDK SageMaker2.0에 맞게 변경맞게 신규 구현)

2.FM 모델 파라미터를 사용하여 KNN 학습, 배포 및 배치 추론
- FM 모델 아티펙트 다운로드
- 모델 파리미터 ($w_{0}, \mathbf{w}, \mathbf{V}$)를 추출
- 학습/추론 데이터를 아래와 같이 새롭게 구성
    - Item latent 행렬: k-NN 모델 학습에 사용; $a_i = concat(V, \; w)$
    - User latent 행렬: 추론에 사용; $a_u = concat(V, \; 1)$
- KNN 모델 학습
- 배치 추론
- 에어라인 Top-K 추천

# 1. 에어라인 데이터셋으로 FM 모델 훈련 및 배포하기
---

MovieLens 데이터의 user_id, item_id가 숫자로 이미 학습에 바로 사용가능한 데이터입니다.
하지만, 에어라인 데이터는 모두 문자열로 데이타가 주어졌습니다.

이 데이타를 labelencoder를 통해서 숫자로 바꾸어서 여기서 사용합니다.

본 예제에서는 사용자가 5번 이상의 리뷰를 남긴 사용자만을 대상으로 하여 총 748명의 사용자와 293개의 에어라인 대해 1부터 10까지의 등급이 부여된 데이터를 사용합니다.

### Factorization Machine
---

### 개요

일반적인 추천 문제들은 user가 행, item이 열, rating이 값으로 이루어진 행렬을 데이터셋으로 하여 Matrix Factorization 기법을 활용하는데, real-world의 다양한 메타데이터 피처(feature)들을 그대로 적용하기에는 어려움이 있습니다. Factoriztion Machine(이하 FM) 알고리즘은 Matrix Factorization의 개념을 확장하여 메타데이터 피처들을 같이 고려하고 피처 간의 상호 관계(interaction)를 선형 계산 복잡도로 자동으로 모델링할 수 있기에, 피처 엔지니어링에 들어가는 노력을 크게 줄일 수 있습니다.

### 설명

다양한 메타데이터 피처를 고려하기 위해 아래 그림처럼 user와 item을 원-핫 인코딩으로 변환하고 추가 피처들을 그대로 concatenate하여 `f(user, item, additional features) = rating` 형태의 선형 회귀(Linear Regression) 문제로 변환하여 풀 수 있습니다.

하지만, 추천 문제를 선형 회귀로만 풀려고 하면 피처 간의 상호 관계를 고려할 수 없기에 아래 수식처럼 피처 간의 상호 관계를 모델링하는 항을 추가하여 다항 회귀(Polynomial Regression)로 변환해야 합니다.

$$
\begin{align} \hat{y}(\mathbf{x}) = w_{0} + \sum_{i=1}^{d} w_{i} x_{i} + \sum_{i=1}^d \sum_{j=i+1}^d x_{i} x_{j} w_{ij}, \;\; x \in \mathbb{R}^d \tag {1}
\end{align}
$$
$d$는 피처 갯수로, $x \in \mathbb{R}^d$는 단일 샘플의 피처 벡터를 나타냅니다.

하지만 대부분의 추천 시스템 데이터셋은 희소하기에(sparse) cold-start 문제가 있으며, 추가적으로 고려해야 하는 피처들이 많아질 수록 계산이 매우 복잡해집니다. (예: user가 6만명, item 갯수가 5천개, 추가 피처가 5천개일 경우 70,000x70,000 행렬을 예측해야 합니다.)

FM은 이러한 문제들을 행렬 분해 기법을 활용하여 feature 쌍(예: user, item) 간의 상호 관계를 내적(dot product)으로 변환하고
수식을 재구성하여 계산 복잡도를 $O(kd^2)$에서 $O(kd)$로 감소시켰습니다. (수식 (2)에서 추가적인 계산을 거치면 계산 복잡도를 선형으로 감소할 수 있습니다. 자세한 내용은 논문을 참조하세요.)

$$
\begin{align}
\hat{y}(\mathbf{x}) = w_{0} + \sum_{i=1}^{d} w_i x_i + \sum_{i=1}^d\sum_{j=i+1}^d  x_{i} x_{j} \langle\mathbf{v}_i, \mathbf{v}_j\rangle \tag{2}
\end{align}
$$

$$
\begin{align}
\langle \textbf{v}_i , \textbf{v}_{j} \rangle = \sum_{f=1}^k v_{i,f} v_{j,f},\; k: \text{dimension of latent feature}  \tag{3}
\end{align}
$$

위의 모델을 2-way(degree = 2) FM이라고 하며, 이를 일반화한 d-way FM도 있지만, 보통 2-way FM를 많이 사용합니다. SageMaker의 FM 또한 2-way FM입니다.

FM이 훈련하는 파라메터 튜플은 ($w_{0}, \mathbf{w}, \mathbf{V}$) 이며, 의미는 아래와 같습니다.
- $w_{0} \in \mathbb{R}$: global bias
- $\mathbf{w} \in \mathbb{R}^d$: 피처 벡터 $\mathbf{x}_i$의 가중치
- $\mathbf{V} \in \mathbb{R}^{n \times k}$: 피처 임베딩 행렬로 i번째 행은 $\mathbf{v}_i$

FM은 위의 수식에서 알 수 있듯이 closed form이며 시간 복잡도가 선형이기 때문에, 다수의 user & item과 메타데이터들이 많은 추천 문제에 적합합니다.
훈련 방법은 대표적으로 Gradient Descent, ALS(Alternating Least Square), MCMC(Markov Chain Monte Carlo)가 있으며, AWS에서는 이 중 딥러닝 아키텍처에 기반한 Gradient Descent를 MXNet 프레임워크를 이용하여 훈련합니다.

```python
import sagemaker
import sagemaker.amazon.common as smac
from sagemaker import get_execution_role
# from sagemaker.predictor import json_deserializer
from sagemaker.deserializers import JSONDeserializer
# from sagemaker.amazon.amazon_estimator import get_image_uri
import numpy as np
from scipy.sparse import lil_matrix
import pandas as pd
import boto3, io, os, csv, json
```

## 에어라인 리뷰 데이터 다운로드

데이터는 비행기 에어라인의 리뷰에 대한 데이타 셋 입니다.

평가에 대한 데이터가 안락함, 청결, 음료, 음식, 화장실, 직원 서비스, 종합에 대한 평가가 있습니다. 여기서는 "종합"에 대한 평가 점수를 사용합니다.
- Skytrax User Reviews Dataset (August 2nd, 2015)
    - https://github.com/quankiquanki/skytrax-reviews-dataset

```python
import os
data_dir = "airlines_data"
os.makedirs(data_dir, exist_ok=True)
!cd $data_dir && wget https://raw.githubusercontent.com/quankiquanki/skytrax-reviews-dataset/master/data/airline.csv
```

    --2020-10-03 09:21:10--  https://raw.githubusercontent.com/quankiquanki/skytrax-reviews-dataset/master/data/airline.csv
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.228.133
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.228.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 34752262 (33M) [text/plain]
    Saving to: ‘airline.csv.3’

    airline.csv.3       100%[===================>]  33.14M  16.9MB/s    in 2.0s

    2020-10-03 09:21:15 (16.9 MB/s) - ‘airline.csv.3’ saved [34752262/34752262]

```python
import pandas as pd
airline_df = pd.read_csv(data_dir + '/airline.csv', parse_dates=['date'])
print("airline_df: ", airline_df.shape)
airline_df.head()
```

    airline_df:  (41396, 20)

      airline_name
      link
      title
      author
      author_country
      date
      content
      aircraft
      type_traveller
      cabin_flown
      route
      overall_rating
      seat_comfort_rating
      cabin_staff_rating
      food_beverages_rating
      inflight_entertainment_rating
      ground_service_rating
      wifi_connectivity_rating
      value_money_rating
      recommended

      0
      adria-airways
      /airline-reviews/adria-airways
      Adria Airways customer review
      D Ito
      Germany
      2015-04-10
      Outbound flight FRA/PRN A319. 2 hours 10 min f...
      NaN
      NaN
      Economy
      NaN
      7.0
      4.0
      4.0
      4.0
      0.0
      NaN
      NaN
      4.0
      1

      1
      adria-airways
      /airline-reviews/adria-airways
      Adria Airways customer review
      Ron Kuhlmann
      United States
      2015-01-05
      Two short hops ZRH-LJU and LJU-VIE. Very fast ...
      NaN
      NaN
      Business Class
      NaN
      10.0
      4.0
      5.0
      4.0
      1.0
      NaN
      NaN
      5.0
      1

      2
      adria-airways
      /airline-reviews/adria-airways
      Adria Airways customer review
      E Albin
      Switzerland
      2014-09-14
      Flew Zurich-Ljubljana on JP365 newish CRJ900. ...
      NaN
      NaN
      Economy
      NaN
      9.0
      5.0
      5.0
      4.0
      0.0
      NaN
      NaN
      5.0
      1

      3
      adria-airways
      /airline-reviews/adria-airways
      Adria Airways customer review
      Tercon Bojan
      Singapore
      2014-09-06
      Adria serves this 100 min flight from Ljubljan...
      NaN
      NaN
      Business Class
      NaN
      8.0
      4.0
      4.0
      3.0
      1.0
      NaN
      NaN
      4.0
      1

      4
      adria-airways
      /airline-reviews/adria-airways
      Adria Airways customer review
      L James
      Poland
      2014-06-16
      WAW-SKJ Economy. No free snacks or drinks on t...
      NaN
      NaN
      Economy
      NaN
      4.0
      4.0
      2.0
      1.0
      2.0
      NaN
      NaN
      2.0
      0

### 필요한 데이타 컬럼을 추출하여 Interaction Data Set으로 사용하기

다운로드 받은 데이터에서 'date','airline_name', 'author', 'overall_rating' 만을 선택하여 사용 합니다.
또한 아래와 같이 컬럼 이름을 변경 합니다.
- airline_name --> item_id
- author --> user_id
- overall_rating --> rating

```python
def get_interaction_data(airline_df):
    '''
    필요한 컬럼만 선별하고, 컬럼 이름 변경
    '''
    a_interactions_df = airline_df.copy()
    a_interactions_df = a_interactions_df[['date','airline_name', 'author', 'overall_rating']]
    a_interactions_df['author'] = a_interactions_df['author'].str.replace(" ","")
    a_interactions_df.rename(columns = {'airline_name':'item_id', 'author':'user_id',
                                   'overall_rating': 'rating'}, inplace = True)
    print("a_interactions_df: ",a_interactions_df.shape)
    return a_interactions_df

a_interactions_df = get_interaction_data(airline_df)
a_interactions_df.head()
```

    a_interactions_df:  (41396, 4)

      date
      item_id
      user_id
      rating

      0
      2015-04-10
      adria-airways
      DIto
      7.0

      1
      2015-01-05
      adria-airways
      RonKuhlmann
      10.0

      2
      2014-09-14
      adria-airways
      EAlbin
      9.0

      3
      2014-09-06
      adria-airways
      TerconBojan
      8.0

      4
      2014-06-16
      adria-airways
      LJames
      4.0

#### 유저 선별
널 데이터 제거, 유저별로 5개 이상의 인터렉션이 있는 유저만 선별 합니다.

```python
def clean_data(air_df, user_limit=5):
    '''
    널 데이터 제거, 유저별로 5개 이상의 인터렉션이 있는 유저만 선별
    '''
    df = air_df.copy()
    df = df.dropna() # 널인 데이타는 제거
    user_g = df.groupby('user_id').count() # 유저 별로 집계

    u5_list = user_g[user_g.item_id >= user_limit].index # 5개의 상호작용 유저만 선택
    df = df[df.user_id.isin(u5_list)] # 5명의 유저만 선택
    # air_df.groupby('user_id').count().sort_values(by='item_id', ascending=False)

    return df

air_df = clean_data(a_interactions_df)

```

#### 유저 및 아이템에 대해서 숫자로 변환

- 레이블 인코더를 이용하여 숫자로 변환
- 사용된 레이블 인코더는 추후에 id-->실제 이름 으로 변경하기 위해서 다시 사용 됩니다.

```python
from sklearn import preprocessing
def make_labelencoder(a_interactions_df):
    '''
    user, item에 대해서 레이블 인코더 생성
    '''
    le_user = preprocessing.LabelEncoder()
    le_item = preprocessing.LabelEncoder()
    le_user.fit(a_interactions_df.user_id)
    le_item.fit(a_interactions_df.item_id)
    print(f"num_user: {len(le_user.classes_)}")
    print(f"num_item: {len(le_item.classes_)}")
    return le_user, le_item

le_user, le_item = make_labelencoder(air_df)

def format_interaction_tb(air_df,le_user,le_item ):
    df = air_df.copy()
    user_id = le_user.transform(df.user_id)
    item_id = le_item.transform(df.item_id)
    en_df = pd.DataFrame({'date': air_df.date, 'user_id':user_id, 'item_id':item_id,
                          'rating':air_df.rating.astype(int)})

    return en_df

encode_df = format_interaction_tb(air_df,le_user,le_item )

```

    num_user: 748
    num_item: 293

#### 데이타를 Train(학습), Test (검증) 를 시간순으로 8:2 로 구분

한 명의 유저에 대해서 Train, Test를 시간순으로 8:2 를 구분 합니다.

예로서 특정 유저가 5개의 추천이 있으면, 시간순으로 소팅하여 앞의 4개는 Train, 뒤의 1개는 Test에 속합니다.

Train, Test의 Unique User ID는 동일하게 748명 입니다.

```python
pd.options.display.max_rows = 5
def split_holdout(data, pct):
    '''
    시간순으로 pct 비율 만큼 학습,테스트 데이터 셋으로 구분
    '''
    df = data.copy()
    # Rank per each subgroup, 'USER_ID'
    ranks = df.groupby('user_id').date.rank(pct=True, method='first')
    df = df.join((ranks> pct).to_frame('holdout'))

    df = df.drop('date', axis=1)

    holdout = df[df['holdout']].drop('holdout', axis=1)
    holdout = holdout.sample(frac=1, random_state = 100)
    train = df[~df['holdout']].drop('holdout', axis=1)
    train = train.sample(frac=1, random_state = 100)

    return train, holdout

user_airline_ratings_train, user_aireline_ratings_test = split_holdout(encode_df, pct=0.8)
```

```python
print("train unique user_id # : ", user_airline_ratings_train.user_id.nunique())
print("test unique user_id # :", user_aireline_ratings_test.user_id.nunique())
print(f"train shape: {user_airline_ratings_train.shape}")
print(f"test shape: {user_aireline_ratings_test.shape}")
```

    train unique user_id # :  748
    test unique user_id # : 748
    train shape: (4968, 3)
    test shape: (1577, 3)

### Sparsity(희소성) 비율 구하기

학습 데이터는 총 4,968 개 입니다. 이 데이터가 왜 희소한지 궁금할 수 있습니다. 전체 매트릭스가 5,161,752 에서 9,936 만을 사용하므로 희소성은 99,8% 입니다.
- 피쳐수: 1,039
    - 사용자와 292개의 에어라인 회사 (747 + 292 = 1,039)
- 학습 인스터스 수: 4,968
- 총 매트릭스 수: 1,039 * 4,968 = 5,161,752
- 총 매트릭스 중에 사용된 셀 수: 4,968 * 2 = 9,936 (2는 사용자에 1개, 아이템에 1개를 사용하기에 2 입니다.)
- 매트릭스 희소성(Sparsity): 9,936 / 5,161,752 = 0.0019 (99.8%)

```python
nb_users = user_airline_ratings_train['user_id'].max()
nb_movies = user_airline_ratings_train['item_id'].max()
nb_features = nb_users + nb_movies

nb_ratings_train = len(user_airline_ratings_train.index)
nb_ratings_test = len(user_aireline_ratings_test.index)

total_matrix_size = nb_features * nb_ratings_train
filled_matrix_size =  nb_ratings_train * 2 # 2 means that 1 is for user and 1 is for item

print("# of users: {}".format(nb_users))
print("# of airlines: {}".format(nb_movies))
print("Training Count: {}".format(nb_ratings_train))
print("Test Count: {}".format(nb_ratings_test))
print("Features (# of users + # of movies): {}".format(nb_features))
print("Sparsity: {:.6}%".format(str((1 - filled_matrix_size / total_matrix_size) * 100)))
```

    # of users: 747
    # of airlines: 292
    Training Count: 4968
    Test Count: 1577
    Features (# of users + # of movies): 1039
    Sparsity: 99.807%

#### 원-핫 인코딩 희소 행렬 변환

FM의 입력 데이터 포맷인 원-핫 인코딩 희소 행렬로 변환하겠습니다. 물론 희소 행렬이 아닌 밀집(dense) 행렬도 가능하지만, 데이터가 많아질수록 계산 속도가 느려지므로, 희소 행렬을 추천합니다.

참고로, 에어라인 데이터셋은 별도의 메타데이터 피처가 존재하지 않아 747명의 사용자와 292개 에어라인에 대해서만 원-핫 인코딩 변환을 수행하므로 변환 후 피처의 차원은 747+292=1,039입니다.

**또한, 본 예시에서는 rating이 8 이상인 에어라인들에 대한 이진 분류 문제로 간소화합니다. (즉, rating이 8 이상일 경우 y = 1, 8 미만일 경우 y = 0 입니다.)**

아래 셀은 약 20초 소요되며, 변환 후 데이터셋의 차원은 rating 개수 x 피쳐 개수 입니다.

```python
pd.options.display.max_rows = 10
user_airline_ratings_train.rating.describe() # 중간값이 8 임. 8 이상을 긍정, 이하를 부정으로 간주 함
```

    count    4968.000000
    mean        7.090781
    std         2.485475
    min         1.000000
    25%         6.000000
    50%         8.000000
    75%         9.000000
    max        10.000000
    Name: rating, dtype: float64

```python
%%time
def loadDataset(df, lines, columns):

    # 피처는 원-핫 인코딩 희소 행렬로 변환합니다.
    X = lil_matrix((lines, columns)).astype('float32')
    Y = []
    line = 0
    for line, (index, row) in enumerate(df.iterrows()):
            X[line,row['user_id']-1] = 1
            X[line, nb_users+(row['item_id']-1)] = 1

            # Y lable 생성
            if int(row['rating']) >= 8:
                Y.append(1) # 긍정
            else:
                Y.append(0) # 부정

    Y = np.array(Y).astype('float32')
    return X,Y

X_train, Y_train = loadDataset(user_airline_ratings_train, nb_ratings_train, nb_features)
X_test, Y_test = loadDataset(user_aireline_ratings_test, nb_ratings_test, nb_features)
```

    CPU times: user 745 ms, sys: 0 ns, total: 745 ms
    Wall time: 744 ms

학습 및 검증의 전체 Shape를 확인하고, y의 1 과 0 분포를 확인 합니다.

```python
print(X_train.shape)
print(Y_train.shape)
assert X_train.shape == (nb_ratings_train, nb_features)
assert Y_train.shape == (nb_ratings_train, )
zero_labels = np.count_nonzero(Y_train)
print("Training labels: {} zeros, {} ones".format(zero_labels, nb_ratings_train-zero_labels))

print(X_test.shape)
print(Y_test.shape)
assert X_test.shape  == (nb_ratings_test, nb_features)
assert Y_test.shape  == (nb_ratings_test, )
zero_labels = np.count_nonzero(Y_test)
print("Test labels: {} zeros, {} ones".format(zero_labels, nb_ratings_test-zero_labels))
```

    (4968, 1039)
    (4968,)
    Training labels: 2718 zeros, 2250 ones
    (1577, 1039)
    (1577,)
    Test labels: 869 zeros, 708 ones

#### Protobuf 포맷 변환 후 S3에 저장

```python
import sagemaker
bucket = sagemaker.Session().default_bucket()
#bucket = '[YOUR-BUCKET]'
prefix = 'fm-hol'

if bucket.strip() == '':
    raise RuntimeError("bucket name is empty.")

train_key      = 'train.protobuf'
train_prefix   = '{}/{}'.format(prefix, 'train')

test_key       = 'test.protobuf'
test_prefix    = '{}/{}'.format(prefix, 'test')

output_prefix  = 's3://{}/{}/output'.format(bucket, prefix)
```

아래 셀은 약 15초 소요됩니다.

```python
%%time
def writeDatasetToProtobuf(X, bucket, prefix, key, d_type, Y=None):
    buf = io.BytesIO()
    if d_type == "sparse":
        smac.write_spmatrix_to_sparse_tensor(buf, X, labels=Y)
    else:
        smac.write_numpy_to_dense_tensor(buf, X, labels=Y)

    buf.seek(0)
    obj = '{}/{}'.format(prefix, key)
    boto3.resource('s3').Bucket(bucket).Object(obj).upload_fileobj(buf)
    return 's3://{}/{}'.format(bucket,obj)

fm_train_data_path = writeDatasetToProtobuf(X_train, bucket, train_prefix, train_key, "sparse", Y_train)
fm_test_data_path  = writeDatasetToProtobuf(X_test, bucket, test_prefix, test_key, "sparse", Y_test)

print("Training data S3 path: ", fm_train_data_path)
print("Test data S3 path: ", fm_test_data_path)
print("FM model output S3 path: ", output_prefix)
```

    Training data S3 path:  s3://sagemaker-ap-northeast-2-057716757052/fm-hol/train/train.protobuf
    Test data S3 path:  s3://sagemaker-ap-northeast-2-057716757052/fm-hol/test/test.protobuf
    FM model output S3 path:  s3://sagemaker-ap-northeast-2-057716757052/fm-hol/output
    CPU times: user 577 ms, sys: 34.2 ms, total: 611 ms
    Wall time: 1.77 s

## FM 학습

본 핸즈온은 하이퍼파라메터 튜닝 없이 휴리스틱한 하이퍼파라메터들을 사용합니다.

- `feature_dim`: 피처 개수로 본 핸즈온에서는 1.039으로 설정해야 합니다.
- `mini_batch_size`: 본 핸즈온에서는 256 으로 설정합니다.
- `num_factors`: latent factor 차원으로 본 핸즈온에서는 64차원으로 설정합니다.
- `epochs`: 본 핸즈온에서는 100 에폭으로 설정합니다.

```python
# 세이지 메이커 신규 버전 다운로드
!pip install --upgrade sagemaker
```

    Requirement already up-to-date: sagemaker in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (2.13.0)
    Requirement already satisfied, skipping upgrade: importlib-metadata>=1.4.0 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (1.5.0)
    Requirement already satisfied, skipping upgrade: packaging>=20.0 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (20.3)
    Requirement already satisfied, skipping upgrade: protobuf3-to-dict>=0.1.5 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (0.1.5)
    Requirement already satisfied, skipping upgrade: boto3>=1.14.12 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (1.14.59)
    Requirement already satisfied, skipping upgrade: numpy>=1.9.0 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (1.18.1)
    Requirement already satisfied, skipping upgrade: google-pasta in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (0.2.0)
    Requirement already satisfied, skipping upgrade: protobuf>=3.1 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (3.13.0)
    Requirement already satisfied, skipping upgrade: smdebug-rulesconfig==0.1.5 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from sagemaker) (0.1.5)
    Requirement already satisfied, skipping upgrade: zipp>=0.5 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker) (2.2.0)
    Requirement already satisfied, skipping upgrade: six in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (1.14.0)
    Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (2.4.6)
    Requirement already satisfied, skipping upgrade: botocore<1.18.0,>=1.17.59 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from boto3>=1.14.12->sagemaker) (1.17.59)
    Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from boto3>=1.14.12->sagemaker) (0.3.3)
    Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from boto3>=1.14.12->sagemaker) (0.9.4)
    Requirement already satisfied, skipping upgrade: setuptools in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from protobuf>=3.1->sagemaker) (46.1.3.post20200330)
    Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from botocore<1.18.0,>=1.17.59->boto3>=1.14.12->sagemaker) (0.15.2)
    Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from botocore<1.18.0,>=1.17.59->boto3>=1.14.12->sagemaker) (2.8.1)
    Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version != "3.4" in /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages (from botocore<1.18.0,>=1.17.59->boto3>=1.14.12->sagemaker) (1.25.8)
    WARNING: You are using pip version 20.0.2; however, version 20.2.3 is available.
    You should consider upgrading via the '/home/ec2-user/anaconda3/envs/mxnet_p36/bin/python -m pip install --upgrade pip' command.

```python
from sagemaker import image_uris, session
fm_image = image_uris.retrieve("factorization-machines", session.Session().boto_region_name, version="latest")
```

    Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: latest.

이제 훈련을 시작하기 위한 모든 준비가 완료되었으며, 여러분께서는 `fit()` 메소드만 호출하면 됩니다.

훈련은 약 4분에서 5분이 소요되며(순수 훈련에 소요되는 시간은 훨씬 짧지만, 훈련 인스턴스를 프로비저닝하는 시간이 고정적으로 소요됩니다),

검증 데이터셋에 대한 accuracy는 약 66%에 F1 스코어는 약 0.71입니다. (아래 output 메세지 참조)

```
[10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_classification_accuracy <score>=0.667089410273
[10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_classification_cross_entropy <loss>=0.61242604422
[10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_f_1.000 <score>=0.711696869852
```

```python
instance_type_training = 'ml.c4.xlarge'
fm = sagemaker.estimator.Estimator(image_uri = fm_image,
                                   role = get_execution_role(),
                                   instance_count=1,
                                   instance_type=instance_type_training,
                                   output_path=output_prefix,
                                   sagemaker_session=sagemaker.Session(),

                                  )

fm.set_hyperparameters(feature_dim=nb_features,
                      predictor_type='binary_classifier',
                      mini_batch_size=256,
                      num_factors=64,
                      epochs=100)
```

```python
%%time
fm.fit({'train': fm_train_data_path, 'test': fm_test_data_path})
```

    2020-10-03 09:21:22 Starting - Starting the training job...
    2020-10-03 09:21:23 Starting - Launching requested ML instances...
    2020-10-03 09:22:21 Starting - Preparing the instances for training......
    2020-10-03 09:23:18 Downloading - Downloading input data
    2020-10-03 09:23:18 Training - Downloading the training image...
    2020-10-03 09:23:51 Training - Training image download completed. Training in progress..Docker entrypoint called with argument(s): train
    Running default environment configuration script
    /opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.
      from numpy.testing import nosetester
    [10/03/2020 09:23:53 INFO 140262587922240] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}
    [10/03/2020 09:23:53 INFO 140262587922240] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'1039', u'mini_batch_size': u'256', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}
    [10/03/2020 09:23:53 INFO 140262587922240] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'1039', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'256', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}
    [10/03/2020 09:23:53 WARNING 140262587922240] Loggers have already been setup.
    Process 1 is a worker.
    [10/03/2020 09:23:53 INFO 140262587922240] Using default worker.
    [2020-10-03 09:23:53.920] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.
    [2020-10-03 09:23:53.921] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 0, "duration": 3, "num_examples": 1, "num_bytes": 16384}
    [10/03/2020 09:23:53 INFO 140262587922240] nvidia-smi took: 0.0251958370209 secs to identify 0 gpus
    [10/03/2020 09:23:53 INFO 140262587922240] Number of GPUs being used: 0
    [10/03/2020 09:23:53 INFO 140262587922240] [Sparse network] Building a sparse network.
    [10/03/2020 09:23:53 INFO 140262587922240] Create Store: local
    #metrics {"Metrics": {"initialize.time": {"count": 1, "max": 33.8749885559082, "sum": 33.8749885559082, "min": 33.8749885559082}}, "EndTime": 1601717033.955284, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717033.918091}

    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Total Records Seen": {"count": 1, "max": 256, "sum": 256.0, "min": 256}, "Max Records Seen Between Resets": {"count": 1, "max": 256, "sum": 256.0, "min": 256}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601717033.955481, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717033.955431}

    [09:23:53] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.203343.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.
    [09:23:53] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.203343.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.50390625
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692751169205
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_f_1.000 <score>=0.644257703081
    [2020-10-03 09:23:54.072] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 2, "duration": 96, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.5240234375
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.691170358658
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=0, train binary_f_1.000 <score>=0.655839570682
    #metrics {"Metrics": {"epochs": {"count": 1, "max": 100, "sum": 100.0, "min": 100}, "update.time": {"count": 1, "max": 117.68102645874023, "sum": 117.68102645874023, "min": 117.68102645874023}}, "EndTime": 1601717034.073378, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717033.955359}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 1 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 21, "sum": 21.0, "min": 21}, "Total Records Seen": {"count": 1, "max": 5224, "sum": 5224.0, "min": 5224}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 2, "sum": 2.0, "min": 2}}, "EndTime": 1601717034.073656, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 0}, "StartTime": 1601717033.95566}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=42039.6770999 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.692551374435
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.149] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 4, "duration": 73, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.4927734375
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.695379242301
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=1, train binary_f_1.000 <score>=0.545422720112
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 76.16901397705078, "sum": 76.16901397705078, "min": 76.16901397705078}}, "EndTime": 1601717034.150192, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.073468}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 2 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 41, "sum": 41.0, "min": 41}, "Total Records Seen": {"count": 1, "max": 10192, "sum": 10192.0, "min": 10192}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 3, "sum": 3.0, "min": 3}}, "EndTime": 1601717034.150534, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 1}, "StartTime": 1601717034.073986}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=64774.4793808 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.702881336212
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.233] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 6, "duration": 81, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.53125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.688527747989
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=2, train binary_f_1.000 <score>=0.693721286371
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 83.60815048217773, "sum": 83.60815048217773, "min": 83.60815048217773}}, "EndTime": 1601717034.234546, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.150275}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 3 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 61, "sum": 61.0, "min": 61}, "Total Records Seen": {"count": 1, "max": 15160, "sum": 15160.0, "min": 15160}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 4, "sum": 4.0, "min": 4}}, "EndTime": 1601717034.234802, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 2}, "StartTime": 1601717034.150904}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=59090.1163579 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.691893577576
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.315] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 8, "duration": 77, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, train binary_classification_accuracy <score>=0.5513671875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, train binary_classification_cross_entropy <loss>=0.686648726463
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=3, train binary_f_1.000 <score>=0.700950397084
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 80.67798614501953, "sum": 80.67798614501953, "min": 80.67798614501953}}, "EndTime": 1601717034.315773, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.23462}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 4 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 81, "sum": 81.0, "min": 81}, "Total Records Seen": {"count": 1, "max": 20128, "sum": 20128.0, "min": 20128}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 5, "sum": 5.0, "min": 5}}, "EndTime": 1601717034.316064, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 3}, "StartTime": 1601717034.235061}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=61225.8576755 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_classification_cross_entropy <loss>=0.691996693611
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.389] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 10, "duration": 69, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, train binary_classification_accuracy <score>=0.5580078125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, train binary_classification_cross_entropy <loss>=0.684000715613
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=4, train binary_f_1.000 <score>=0.703989535644
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 73.3330249786377, "sum": 73.3330249786377, "min": 73.3330249786377}}, "EndTime": 1601717034.389746, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.315861}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 5 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 101, "sum": 101.0, "min": 101}, "Total Records Seen": {"count": 1, "max": 25096, "sum": 25096.0, "min": 25096}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 6, "sum": 6.0, "min": 6}}, "EndTime": 1601717034.390008, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 4}, "StartTime": 1601717034.316356}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=67316.993836 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_classification_cross_entropy <loss>=0.688360571861
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.460] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 12, "duration": 68, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, train binary_classification_accuracy <score>=0.574609375
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, train binary_classification_cross_entropy <loss>=0.681511378288
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=5, train binary_f_1.000 <score>=0.71083377589
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 70.91307640075684, "sum": 70.91307640075684, "min": 70.91307640075684}}, "EndTime": 1601717034.461247, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.389833}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 6 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 121, "sum": 121.0, "min": 121}, "Total Records Seen": {"count": 1, "max": 30064, "sum": 30064.0, "min": 30064}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 7, "sum": 7.0, "min": 7}}, "EndTime": 1601717034.461506, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 5}, "StartTime": 1601717034.390303}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=69630.5902057 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_classification_cross_entropy <loss>=0.686040282249
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.527] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 14, "duration": 64, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, train binary_classification_accuracy <score>=0.5888671875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, train binary_classification_cross_entropy <loss>=0.678996151686
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=6, train binary_f_1.000 <score>=0.716192530673
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 66.54095649719238, "sum": 66.54095649719238, "min": 66.54095649719238}}, "EndTime": 1601717034.528384, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.461345}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 7 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 141, "sum": 141.0, "min": 141}, "Total Records Seen": {"count": 1, "max": 35032, "sum": 35032.0, "min": 35032}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 8, "sum": 8.0, "min": 8}}, "EndTime": 1601717034.528796, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 6}, "StartTime": 1601717034.461812}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=74006.8769672 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_classification_cross_entropy <loss>=0.683289408684
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.597] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 16, "duration": 66, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, train binary_classification_accuracy <score>=0.6
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, train binary_classification_cross_entropy <loss>=0.67645213306
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=7, train binary_f_1.000 <score>=0.719682452779
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 68.86887550354004, "sum": 68.86887550354004, "min": 68.86887550354004}}, "EndTime": 1601717034.598052, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.528513}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 8 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 161, "sum": 161.0, "min": 161}, "Total Records Seen": {"count": 1, "max": 40000, "sum": 40000.0, "min": 40000}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 9, "sum": 9.0, "min": 9}}, "EndTime": 1601717034.598311, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 7}, "StartTime": 1601717034.529151}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=71697.7509729 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_accuracy <score>=0.48046875
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_classification_cross_entropy <loss>=0.680659770966
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, batch=0 train binary_f_1.000 <score>=0.64907651715
    [2020-10-03 09:23:54.685] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 18, "duration": 79, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, train binary_classification_accuracy <score>=0.6095703125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, train binary_classification_cross_entropy <loss>=0.673912924528
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=8, train binary_f_1.000 <score>=0.7224767458
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 87.14890480041504, "sum": 87.14890480041504, "min": 87.14890480041504}}, "EndTime": 1601717034.685785, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.598136}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 9 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 181, "sum": 181.0, "min": 181}, "Total Records Seen": {"count": 1, "max": 44968, "sum": 44968.0, "min": 44968}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 10, "sum": 10.0, "min": 10}}, "EndTime": 1601717034.686056, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 8}, "StartTime": 1601717034.598598}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=56718.1994594 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_accuracy <score>=0.48828125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_classification_cross_entropy <loss>=0.677988171577
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, batch=0 train binary_f_1.000 <score>=0.652519893899
    [2020-10-03 09:23:54.755] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 20, "duration": 66, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, train binary_classification_accuracy <score>=0.623828125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, train binary_classification_cross_entropy <loss>=0.67136297822
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=9, train binary_f_1.000 <score>=0.728502960248
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 69.97299194335938, "sum": 69.97299194335938, "min": 69.97299194335938}}, "EndTime": 1601717034.75634, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.685874}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 10 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 201, "sum": 201.0, "min": 201}, "Total Records Seen": {"count": 1, "max": 49936, "sum": 49936.0, "min": 49936}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 11, "sum": 11.0, "min": 11}}, "EndTime": 1601717034.756825, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 9}, "StartTime": 1601717034.686335}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=70300.5781723 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_classification_accuracy <score>=0.5
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_classification_cross_entropy <loss>=0.675319075584
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, batch=0 train binary_f_1.000 <score>=0.657754010695
    [2020-10-03 09:23:54.846] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 22, "duration": 87, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, train binary_classification_accuracy <score>=0.63203125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, train binary_classification_cross_entropy <loss>=0.66881326139
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=10, train binary_f_1.000 <score>=0.731164383562
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 90.09122848510742, "sum": 90.09122848510742, "min": 90.09122848510742}}, "EndTime": 1601717034.847301, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.756493}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 11 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 221, "sum": 221.0, "min": 221}, "Total Records Seen": {"count": 1, "max": 54904, "sum": 54904.0, "min": 54904}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 12, "sum": 12.0, "min": 12}}, "EndTime": 1601717034.847906, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 10}, "StartTime": 1601717034.757177}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=54517.370562 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_classification_accuracy <score>=0.51953125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_classification_cross_entropy <loss>=0.672642409801
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, batch=0 train binary_f_1.000 <score>=0.666666666667
    [2020-10-03 09:23:54.931] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 24, "duration": 76, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, train binary_classification_accuracy <score>=0.6392578125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, train binary_classification_cross_entropy <loss>=0.666260898113
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=11, train binary_f_1.000 <score>=0.73374657633
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 83.85896682739258, "sum": 83.85896682739258, "min": 83.85896682739258}}, "EndTime": 1601717034.93239, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.847451}

    [10/03/2020 09:23:54 INFO 140262587922240] #progress_metric: host=algo-1, completed 12 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 241, "sum": 241.0, "min": 241}, "Total Records Seen": {"count": 1, "max": 59872, "sum": 59872.0, "min": 59872}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 13, "sum": 13.0, "min": 13}}, "EndTime": 1601717034.932656, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 11}, "StartTime": 1601717034.848436}

    [10/03/2020 09:23:54 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=58871.5825114 records/second
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_classification_accuracy <score>=0.55078125
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_classification_cross_entropy <loss>=0.669959664345
    [10/03/2020 09:23:54 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, batch=0 train binary_f_1.000 <score>=0.679665738162
    [2020-10-03 09:23:55.006] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 26, "duration": 70, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, train binary_classification_accuracy <score>=0.64921875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, train binary_classification_cross_entropy <loss>=0.663708221912
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=12, train binary_f_1.000 <score>=0.737580362361
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 74.17893409729004, "sum": 74.17893409729004, "min": 74.17893409729004}}, "EndTime": 1601717035.007186, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717034.932478}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 13 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 261, "sum": 261.0, "min": 261}, "Total Records Seen": {"count": 1, "max": 64840, "sum": 64840.0, "min": 64840}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 14, "sum": 14.0, "min": 14}}, "EndTime": 1601717035.007617, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 12}, "StartTime": 1601717034.93297}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=66377.9582376 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_classification_accuracy <score>=0.578125
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_classification_cross_entropy <loss>=0.667271375656
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, batch=0 train binary_f_1.000 <score>=0.693181818182
    [2020-10-03 09:23:55.110] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 28, "duration": 100, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, train binary_classification_accuracy <score>=0.658984375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, train binary_classification_cross_entropy <loss>=0.661155185103
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=13, train binary_f_1.000 <score>=0.742249778565
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 103.38211059570312, "sum": 103.38211059570312, "min": 103.38211059570312}}, "EndTime": 1601717035.111474, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.007297}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 14 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 281, "sum": 281.0, "min": 281}, "Total Records Seen": {"count": 1, "max": 69808, "sum": 69808.0, "min": 69808}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 15, "sum": 15.0, "min": 15}}, "EndTime": 1601717035.111738, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 13}, "StartTime": 1601717035.008027}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=47833.6675818 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_classification_accuracy <score>=0.5859375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_classification_cross_entropy <loss>=0.664577245712
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, batch=0 train binary_f_1.000 <score>=0.695402298851
    [2020-10-03 09:23:55.190] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 30, "duration": 77, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, train binary_classification_accuracy <score>=0.6654296875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, train binary_classification_cross_entropy <loss>=0.658602416515
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=14, train binary_f_1.000 <score>=0.74451901566
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 79.43415641784668, "sum": 79.43415641784668, "min": 79.43415641784668}}, "EndTime": 1601717035.191489, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.111571}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 15 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 301, "sum": 301.0, "min": 301}, "Total Records Seen": {"count": 1, "max": 74776, "sum": 74776.0, "min": 74776}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 16, "sum": 16.0, "min": 16}}, "EndTime": 1601717035.191746, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 14}, "StartTime": 1601717035.112025}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=62199.6026113 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_classification_accuracy <score>=0.59375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_classification_cross_entropy <loss>=0.661877691746
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, batch=0 train binary_f_1.000 <score>=0.697674418605
    [2020-10-03 09:23:55.269] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 32, "duration": 76, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, train binary_classification_accuracy <score>=0.6697265625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, train binary_classification_cross_entropy <loss>=0.656050252914
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=15, train binary_f_1.000 <score>=0.745522949586
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 78.16600799560547, "sum": 78.16600799560547, "min": 78.16600799560547}}, "EndTime": 1601717035.27024, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.191581}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 16 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 321, "sum": 321.0, "min": 321}, "Total Records Seen": {"count": 1, "max": 79744, "sum": 79744.0, "min": 79744}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 17, "sum": 17.0, "min": 17}}, "EndTime": 1601717035.270512, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 15}, "StartTime": 1601717035.192043}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=63219.0114592 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_classification_accuracy <score>=0.59375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_classification_cross_entropy <loss>=0.659172952175
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, batch=0 train binary_f_1.000 <score>=0.695906432749
    [2020-10-03 09:23:55.343] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 34, "duration": 70, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, train binary_classification_accuracy <score>=0.67421875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, train binary_classification_cross_entropy <loss>=0.65349906683
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=16, train binary_f_1.000 <score>=0.746966019417
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 72.92795181274414, "sum": 72.92795181274414, "min": 72.92795181274414}}, "EndTime": 1601717035.343761, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.270312}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 17 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 341, "sum": 341.0, "min": 341}, "Total Records Seen": {"count": 1, "max": 84712, "sum": 84712.0, "min": 84712}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 18, "sum": 18.0, "min": 18}}, "EndTime": 1601717035.344006, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 16}, "StartTime": 1601717035.270803}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=67762.2624339 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_classification_accuracy <score>=0.62109375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_classification_cross_entropy <loss>=0.656463503838
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, batch=0 train binary_f_1.000 <score>=0.708708708709
    [2020-10-03 09:23:55.409] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 36, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, train binary_classification_accuracy <score>=0.6822265625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, train binary_classification_cross_entropy <loss>=0.650949198008
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=17, train binary_f_1.000 <score>=0.750727746285
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.7958984375, "sum": 65.7958984375, "min": 65.7958984375}}, "EndTime": 1601717035.41009, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.343826}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 18 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 361, "sum": 361.0, "min": 361}, "Total Records Seen": {"count": 1, "max": 89680, "sum": 89680.0, "min": 89680}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 19, "sum": 19.0, "min": 19}}, "EndTime": 1601717035.410288, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 17}, "StartTime": 1601717035.344266}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75119.1545189 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_classification_accuracy <score>=0.62890625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_classification_cross_entropy <loss>=0.653749704361
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, batch=0 train binary_f_1.000 <score>=0.711246200608
    [2020-10-03 09:23:55.483] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 38, "duration": 71, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, train binary_classification_accuracy <score>=0.68671875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, train binary_classification_cross_entropy <loss>=0.648400983214
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=18, train binary_f_1.000 <score>=0.752316244595
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 73.94886016845703, "sum": 73.94886016845703, "min": 73.94886016845703}}, "EndTime": 1601717035.484522, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.410151}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 19 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 381, "sum": 381.0, "min": 381}, "Total Records Seen": {"count": 1, "max": 94648, "sum": 94648.0, "min": 94648}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 20, "sum": 20.0, "min": 20}}, "EndTime": 1601717035.484821, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 18}, "StartTime": 1601717035.410509}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=66704.3413758 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_classification_accuracy <score>=0.65625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_classification_cross_entropy <loss>=0.651031970978
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, batch=0 train binary_f_1.000 <score>=0.725
    [2020-10-03 09:23:55.558] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 40, "duration": 71, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, train binary_classification_accuracy <score>=0.6921875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, train binary_classification_cross_entropy <loss>=0.645854780078
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=19, train binary_f_1.000 <score>=0.754822650902
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 74.40590858459473, "sum": 74.40590858459473, "min": 74.40590858459473}}, "EndTime": 1601717035.559747, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.484606}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 20 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 401, "sum": 401.0, "min": 401}, "Total Records Seen": {"count": 1, "max": 99616, "sum": 99616.0, "min": 99616}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 21, "sum": 21.0, "min": 21}}, "EndTime": 1601717035.560126, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 19}, "StartTime": 1601717035.485233}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=66095.4011819 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_classification_accuracy <score>=0.6640625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_classification_cross_entropy <loss>=0.648310661316
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, batch=0 train binary_f_1.000 <score>=0.729559748428
    [2020-10-03 09:23:55.649] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 42, "duration": 87, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, train binary_classification_accuracy <score>=0.6966796875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, train binary_classification_cross_entropy <loss>=0.643310925364
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=20, train binary_f_1.000 <score>=0.757078054122
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 90.05093574523926, "sum": 90.05093574523926, "min": 90.05093574523926}}, "EndTime": 1601717035.650581, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.559837}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 21 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 421, "sum": 421.0, "min": 421}, "Total Records Seen": {"count": 1, "max": 104584, "sum": 104584.0, "min": 104584}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 22, "sum": 22.0, "min": 22}}, "EndTime": 1601717035.650904, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 20}, "StartTime": 1601717035.560493}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=54847.1300814 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_classification_accuracy <score>=0.6875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_classification_cross_entropy <loss>=0.645586431026
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, batch=0 train binary_f_1.000 <score>=0.74358974359
    [2020-10-03 09:23:55.727] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 44, "duration": 72, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, train binary_classification_accuracy <score>=0.7013671875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, train binary_classification_cross_entropy <loss>=0.640769779682
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=21, train binary_f_1.000 <score>=0.75925051173
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 77.8658390045166, "sum": 77.8658390045166, "min": 77.8658390045166}}, "EndTime": 1601717035.729151, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.650663}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 22 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 441, "sum": 441.0, "min": 441}, "Total Records Seen": {"count": 1, "max": 109552, "sum": 109552.0, "min": 109552}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 23, "sum": 23.0, "min": 23}}, "EndTime": 1601717035.729619, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 21}, "StartTime": 1601717035.651252}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=63227.0681842 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_classification_accuracy <score>=0.6875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_classification_cross_entropy <loss>=0.642859697342
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, batch=0 train binary_f_1.000 <score>=0.74358974359
    [2020-10-03 09:23:55.805] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 46, "duration": 72, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, train binary_classification_accuracy <score>=0.704296875
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, train binary_classification_cross_entropy <loss>=0.638231682777
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=22, train binary_f_1.000 <score>=0.760443037975
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 76.26199722290039, "sum": 76.26199722290039, "min": 76.26199722290039}}, "EndTime": 1601717035.806279, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.729417}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 23 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 461, "sum": 461.0, "min": 461}, "Total Records Seen": {"count": 1, "max": 114520, "sum": 114520.0, "min": 114520}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 24, "sum": 24.0, "min": 24}}, "EndTime": 1601717035.806545, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 22}, "StartTime": 1601717035.729982}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=64766.8287249 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_classification_accuracy <score>=0.7109375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_classification_cross_entropy <loss>=0.640130877495
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, batch=0 train binary_f_1.000 <score>=0.758169934641
    [2020-10-03 09:23:55.882] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 48, "duration": 72, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, train binary_classification_accuracy <score>=0.71015625
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, train binary_classification_cross_entropy <loss>=0.635696968436
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=23, train binary_f_1.000 <score>=0.763844684914
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 76.06291770935059, "sum": 76.06291770935059, "min": 76.06291770935059}}, "EndTime": 1601717035.882925, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.806373}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 24 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 481, "sum": 481.0, "min": 481}, "Total Records Seen": {"count": 1, "max": 119488, "sum": 119488.0, "min": 119488}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 25, "sum": 25.0, "min": 25}}, "EndTime": 1601717035.883258, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 23}, "StartTime": 1601717035.806823}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=64875.7184951 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_classification_accuracy <score>=0.7109375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_classification_cross_entropy <loss>=0.637400627136
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, batch=0 train binary_f_1.000 <score>=0.756578947368
    [2020-10-03 09:23:55.957] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 50, "duration": 70, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, train binary_classification_accuracy <score>=0.71328125
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, train binary_classification_cross_entropy <loss>=0.633165997267
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=24, train binary_f_1.000 <score>=0.765345268542
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 74.47600364685059, "sum": 74.47600364685059, "min": 74.47600364685059}}, "EndTime": 1601717035.958139, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.883005}

    [10/03/2020 09:23:55 INFO 140262587922240] #progress_metric: host=algo-1, completed 25 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 501, "sum": 501.0, "min": 501}, "Total Records Seen": {"count": 1, "max": 124456, "sum": 124456.0, "min": 124456}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 26, "sum": 26.0, "min": 26}}, "EndTime": 1601717035.958386, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 24}, "StartTime": 1601717035.883628}

    [10/03/2020 09:23:55 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=66333.583566 records/second
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_classification_accuracy <score>=0.7109375
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_classification_cross_entropy <loss>=0.634669303894
    [10/03/2020 09:23:55 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, batch=0 train binary_f_1.000 <score>=0.754966887417
    [2020-10-03 09:23:56.047] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 52, "duration": 80, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, train binary_classification_accuracy <score>=0.7138671875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, train binary_classification_cross_entropy <loss>=0.630639088154
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=25, train binary_f_1.000 <score>=0.764885251164
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 89.21003341674805, "sum": 89.21003341674805, "min": 89.21003341674805}}, "EndTime": 1601717036.047901, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717035.958218}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 26 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 521, "sum": 521.0, "min": 521}, "Total Records Seen": {"count": 1, "max": 129424, "sum": 129424.0, "min": 129424}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 27, "sum": 27.0, "min": 27}}, "EndTime": 1601717036.04811, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 25}, "StartTime": 1601717035.95866}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=55463.0520659 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_classification_accuracy <score>=0.71875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_classification_cross_entropy <loss>=0.631937503815
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, batch=0 train binary_f_1.000 <score>=0.76
    [2020-10-03 09:23:56.133] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 54, "duration": 84, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, train binary_classification_accuracy <score>=0.7166015625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, train binary_classification_cross_entropy <loss>=0.628116601706
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=26, train binary_f_1.000 <score>=0.766231673917
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 86.20500564575195, "sum": 86.20500564575195, "min": 86.20500564575195}}, "EndTime": 1601717036.134566, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.047971}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 27 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 541, "sum": 541.0, "min": 541}, "Total Records Seen": {"count": 1, "max": 134392, "sum": 134392.0, "min": 134392}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 28, "sum": 28.0, "min": 28}}, "EndTime": 1601717036.134839, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 26}, "StartTime": 1601717036.048328}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=57327.3896759 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_classification_accuracy <score>=0.72265625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_classification_cross_entropy <loss>=0.629205644131
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, batch=0 train binary_f_1.000 <score>=0.76254180602
    [2020-10-03 09:23:56.203] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 56, "duration": 66, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, train binary_classification_accuracy <score>=0.7203125
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, train binary_classification_cross_entropy <loss>=0.625598827004
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=27, train binary_f_1.000 <score>=0.768359754125
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 68.76492500305176, "sum": 68.76492500305176, "min": 68.76492500305176}}, "EndTime": 1601717036.203947, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.134635}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 28 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 561, "sum": 561.0, "min": 561}, "Total Records Seen": {"count": 1, "max": 139360, "sum": 139360.0, "min": 139360}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 29, "sum": 29.0, "min": 29}}, "EndTime": 1601717036.20416, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 27}, "StartTime": 1601717036.135151}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=71877.055943 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_classification_accuracy <score>=0.72265625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_classification_cross_entropy <loss>=0.626474261284
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, batch=0 train binary_f_1.000 <score>=0.76254180602
    [2020-10-03 09:23:56.287] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 58, "duration": 81, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, train binary_classification_accuracy <score>=0.7228515625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, train binary_classification_cross_entropy <loss>=0.623086124659
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=28, train binary_f_1.000 <score>=0.769978926893
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 84.27095413208008, "sum": 84.27095413208008, "min": 84.27095413208008}}, "EndTime": 1601717036.288694, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.204023}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 29 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 581, "sum": 581.0, "min": 581}, "Total Records Seen": {"count": 1, "max": 144328, "sum": 144328.0, "min": 144328}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 30, "sum": 30.0, "min": 30}}, "EndTime": 1601717036.289031, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 28}, "StartTime": 1601717036.204387}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=58568.2708655 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_classification_accuracy <score>=0.7265625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_classification_cross_entropy <loss>=0.623743772507
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, batch=0 train binary_f_1.000 <score>=0.765100671141
    [2020-10-03 09:23:56.358] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 60, "duration": 67, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, train binary_classification_accuracy <score>=0.726171875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, train binary_classification_cross_entropy <loss>=0.620578765869
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=29, train binary_f_1.000 <score>=0.771809895833
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 69.27299499511719, "sum": 69.27299499511719, "min": 69.27299499511719}}, "EndTime": 1601717036.358767, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.288756}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 30 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 601, "sum": 601.0, "min": 601}, "Total Records Seen": {"count": 1, "max": 149296, "sum": 149296.0, "min": 149296}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 31, "sum": 31.0, "min": 31}}, "EndTime": 1601717036.358967, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 29}, "StartTime": 1601717036.289466}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=71319.8351354 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_classification_accuracy <score>=0.73046875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_classification_cross_entropy <loss>=0.62101483345
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, batch=0 train binary_f_1.000 <score>=0.767676767677
    [2020-10-03 09:23:56.421] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 62, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, train binary_classification_accuracy <score>=0.7298828125
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, train binary_classification_cross_entropy <loss>=0.618077093363
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=30, train binary_f_1.000 <score>=0.774204081633
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.470197677612305, "sum": 62.470197677612305, "min": 62.470197677612305}}, "EndTime": 1601717036.421678, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.358837}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 31 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 621, "sum": 621.0, "min": 621}, "Total Records Seen": {"count": 1, "max": 154264, "sum": 154264.0, "min": 154264}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 32, "sum": 32.0, "min": 32}}, "EndTime": 1601717036.421907, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 30}, "StartTime": 1601717036.35918}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79056.7441098 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_classification_accuracy <score>=0.7421875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_classification_cross_entropy <loss>=0.618287622929
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, batch=0 train binary_f_1.000 <score>=0.775510204082
    [2020-10-03 09:23:56.483] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 64, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, train binary_classification_accuracy <score>=0.7330078125
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, train binary_classification_cross_entropy <loss>=0.615581381321
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=31, train binary_f_1.000 <score>=0.776231789164
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.44492530822754, "sum": 62.44492530822754, "min": 62.44492530822754}}, "EndTime": 1601717036.484601, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.421773}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 32 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 641, "sum": 641.0, "min": 641}, "Total Records Seen": {"count": 1, "max": 159232, "sum": 159232.0, "min": 159232}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 33, "sum": 33.0, "min": 33}}, "EndTime": 1601717036.485007, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 31}, "StartTime": 1601717036.422127}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78855.9945202 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_classification_accuracy <score>=0.7421875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_classification_cross_entropy <loss>=0.615562796593
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, batch=0 train binary_f_1.000 <score>=0.775510204082
    [2020-10-03 09:23:56.546] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 66, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, train binary_classification_accuracy <score>=0.73515625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, train binary_classification_cross_entropy <loss>=0.613091906905
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=32, train binary_f_1.000 <score>=0.777413000657
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.466217041015625, "sum": 61.466217041015625, "min": 61.466217041015625}}, "EndTime": 1601717036.546832, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.484737}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 33 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 661, "sum": 661.0, "min": 661}, "Total Records Seen": {"count": 1, "max": 164200, "sum": 164200.0, "min": 164200}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 34, "sum": 34.0, "min": 34}}, "EndTime": 1601717036.547028, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 32}, "StartTime": 1601717036.485339}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80389.8962281 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_classification_accuracy <score>=0.7421875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_classification_cross_entropy <loss>=0.612840771675
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, batch=0 train binary_f_1.000 <score>=0.775510204082
    [2020-10-03 09:23:56.611] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 68, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, train binary_classification_accuracy <score>=0.7373046875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, train binary_classification_cross_entropy <loss>=0.610608968139
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=33, train binary_f_1.000 <score>=0.778673687675
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.0169849395752, "sum": 65.0169849395752, "min": 65.0169849395752}}, "EndTime": 1601717036.612288, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.5469}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 34 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 681, "sum": 681.0, "min": 681}, "Total Records Seen": {"count": 1, "max": 169168, "sum": 169168.0, "min": 169168}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 35, "sum": 35.0, "min": 35}}, "EndTime": 1601717036.612539, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 33}, "StartTime": 1601717036.547239}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75928.5591456 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_classification_accuracy <score>=0.74609375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_classification_cross_entropy <loss>=0.610121846199
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, batch=0 train binary_f_1.000 <score>=0.776632302405
    [2020-10-03 09:23:56.675] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 70, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, train binary_classification_accuracy <score>=0.7392578125
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, train binary_classification_cross_entropy <loss>=0.608132833242
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=34, train binary_f_1.000 <score>=0.780029658922
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.517093658447266, "sum": 63.517093658447266, "min": 63.517093658447266}}, "EndTime": 1601717036.676356, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.612365}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 35 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 701, "sum": 701.0, "min": 701}, "Total Records Seen": {"count": 1, "max": 174136, "sum": 174136.0, "min": 174136}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 36, "sum": 36.0, "min": 36}}, "EndTime": 1601717036.676565, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 34}, "StartTime": 1601717036.612809}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77778.1180484 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_classification_accuracy <score>=0.74609375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_classification_cross_entropy <loss>=0.607406616211
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, batch=0 train binary_f_1.000 <score>=0.77508650519
    [2020-10-03 09:23:56.742] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 72, "duration": 64, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, train binary_classification_accuracy <score>=0.74140625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, train binary_classification_cross_entropy <loss>=0.605663749576
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=35, train binary_f_1.000 <score>=0.781301618764
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 66.03312492370605, "sum": 66.03312492370605, "min": 66.03312492370605}}, "EndTime": 1601717036.742848, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.676426}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 36 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 721, "sum": 721.0, "min": 721}, "Total Records Seen": {"count": 1, "max": 179104, "sum": 179104.0, "min": 179104}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 37, "sum": 37.0, "min": 37}}, "EndTime": 1601717036.743014, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 35}, "StartTime": 1601717036.676783}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=74889.4026114 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_classification_accuracy <score>=0.74609375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_classification_cross_entropy <loss>=0.604695320129
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, batch=0 train binary_f_1.000 <score>=0.77508650519
    [2020-10-03 09:23:56.803] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 74, "duration": 58, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, train binary_classification_accuracy <score>=0.74375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, train binary_classification_cross_entropy <loss>=0.603201976418
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=36, train binary_f_1.000 <score>=0.783140495868
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 60.804128646850586, "sum": 60.804128646850586, "min": 60.804128646850586}}, "EndTime": 1601717036.80405, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.742912}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 37 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 741, "sum": 741.0, "min": 741}, "Total Records Seen": {"count": 1, "max": 184072, "sum": 184072.0, "min": 184072}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 38, "sum": 38.0, "min": 38}}, "EndTime": 1601717036.804289, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 36}, "StartTime": 1601717036.743218}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=81191.1483645 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_classification_accuracy <score>=0.74609375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_classification_cross_entropy <loss>=0.601988494396
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, batch=0 train binary_f_1.000 <score>=0.77508650519
    [2020-10-03 09:23:56.869] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 76, "duration": 64, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, train binary_classification_accuracy <score>=0.7458984375
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, train binary_classification_cross_entropy <loss>=0.600747770071
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=37, train binary_f_1.000 <score>=0.784638304916
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.92392921447754, "sum": 65.92392921447754, "min": 65.92392921447754}}, "EndTime": 1601717036.870537, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.804118}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 38 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 761, "sum": 761.0, "min": 761}, "Total Records Seen": {"count": 1, "max": 189040, "sum": 189040.0, "min": 189040}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 39, "sum": 39.0, "min": 39}}, "EndTime": 1601717036.870764, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 37}, "StartTime": 1601717036.804582}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=74879.9834409 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_classification_accuracy <score>=0.75
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_classification_cross_entropy <loss>=0.599286496639
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, batch=0 train binary_f_1.000 <score>=0.777777777778
    [2020-10-03 09:23:56.940] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 78, "duration": 67, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, train binary_classification_accuracy <score>=0.7482421875
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, train binary_classification_cross_entropy <loss>=0.598301330209
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=38, train binary_f_1.000 <score>=0.786058091286
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 69.91791725158691, "sum": 69.91791725158691, "min": 69.91791725158691}}, "EndTime": 1601717036.94099, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.87061}

    [10/03/2020 09:23:56 INFO 140262587922240] #progress_metric: host=algo-1, completed 39 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 781, "sum": 781.0, "min": 781}, "Total Records Seen": {"count": 1, "max": 194008, "sum": 194008.0, "min": 194008}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 40, "sum": 40.0, "min": 40}}, "EndTime": 1601717036.941183, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 38}, "StartTime": 1601717036.871042}

    [10/03/2020 09:23:56 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=70715.9466507 records/second
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_classification_accuracy <score>=0.75390625
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_classification_cross_entropy <loss>=0.596589565277
    [10/03/2020 09:23:56 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, batch=0 train binary_f_1.000 <score>=0.780487804878
    [2020-10-03 09:23:57.004] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 80, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, train binary_classification_accuracy <score>=0.750390625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, train binary_classification_cross_entropy <loss>=0.595862871408
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=39, train binary_f_1.000 <score>=0.787495843033
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.200950622558594, "sum": 63.200950622558594, "min": 63.200950622558594}}, "EndTime": 1601717037.004615, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717036.941053}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 40 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 801, "sum": 801.0, "min": 801}, "Total Records Seen": {"count": 1, "max": 198976, "sum": 198976.0, "min": 198976}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 41, "sum": 41.0, "min": 41}}, "EndTime": 1601717037.004804, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 39}, "StartTime": 1601717036.941389}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78211.6427022 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_classification_accuracy <score>=0.75390625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_classification_cross_entropy <loss>=0.593898296356
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, batch=0 train binary_f_1.000 <score>=0.780487804878
    [2020-10-03 09:23:57.066] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 82, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, train binary_classification_accuracy <score>=0.7517578125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, train binary_classification_cross_entropy <loss>=0.593432664871
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=40, train binary_f_1.000 <score>=0.788131355226
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.9959831237793, "sum": 61.9959831237793, "min": 61.9959831237793}}, "EndTime": 1601717037.067035, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.00467}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 41 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 821, "sum": 821.0, "min": 821}, "Total Records Seen": {"count": 1, "max": 203944, "sum": 203944.0, "min": 203944}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 42, "sum": 42.0, "min": 42}}, "EndTime": 1601717037.067235, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 40}, "StartTime": 1601717037.00501}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79702.3484331 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_classification_accuracy <score>=0.7578125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_classification_cross_entropy <loss>=0.591212749481
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, batch=0 train binary_f_1.000 <score>=0.783216783217
    [2020-10-03 09:23:57.169] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 84, "duration": 99, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, train binary_classification_accuracy <score>=0.7548828125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, train binary_classification_cross_entropy <loss>=0.591010868549
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=41, train binary_f_1.000 <score>=0.790309106099
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 102.12016105651855, "sum": 102.12016105651855, "min": 102.12016105651855}}, "EndTime": 1601717037.169599, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.0671}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 42 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 841, "sum": 841.0, "min": 841}, "Total Records Seen": {"count": 1, "max": 208912, "sum": 208912.0, "min": 208912}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 43, "sum": 43.0, "min": 43}}, "EndTime": 1601717037.16983, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 41}, "StartTime": 1601717037.06745}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=48474.5121609 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_classification_accuracy <score>=0.7578125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_classification_cross_entropy <loss>=0.588533520699
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, batch=0 train binary_f_1.000 <score>=0.783216783217
    [2020-10-03 09:23:57.244] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 86, "duration": 72, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, train binary_classification_accuracy <score>=0.7572265625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, train binary_classification_cross_entropy <loss>=0.588597682118
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=42, train binary_f_1.000 <score>=0.791896869245
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 74.98002052307129, "sum": 74.98002052307129, "min": 74.98002052307129}}, "EndTime": 1601717037.24508, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.169663}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 43 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 861, "sum": 861.0, "min": 861}, "Total Records Seen": {"count": 1, "max": 213880, "sum": 213880.0, "min": 213880}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 44, "sum": 44.0, "min": 44}}, "EndTime": 1601717037.24546, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 42}, "StartTime": 1601717037.170044}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=65718.7173477 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_classification_accuracy <score>=0.765625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_classification_cross_entropy <loss>=0.585860848427
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, batch=0 train binary_f_1.000 <score>=0.788732394366
    [2020-10-03 09:23:57.318] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 88, "duration": 68, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, train binary_classification_accuracy <score>=0.76015625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, train binary_classification_cross_entropy <loss>=0.586193320155
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=43, train binary_f_1.000 <score>=0.794028849379
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 73.3788013458252, "sum": 73.3788013458252, "min": 73.3788013458252}}, "EndTime": 1601717037.319265, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.245242}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 44 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 881, "sum": 881.0, "min": 881}, "Total Records Seen": {"count": 1, "max": 218848, "sum": 218848.0, "min": 218848}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 45, "sum": 45.0, "min": 45}}, "EndTime": 1601717037.31947, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 43}, "StartTime": 1601717037.245855}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=67374.6739051 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_classification_accuracy <score>=0.7734375
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_classification_cross_entropy <loss>=0.583194971085
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, batch=0 train binary_f_1.000 <score>=0.795774647887
    [2020-10-03 09:23:57.387] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 90, "duration": 66, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, train binary_classification_accuracy <score>=0.76328125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, train binary_classification_cross_entropy <loss>=0.583797961473
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=44, train binary_f_1.000 <score>=0.796370967742
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 68.73297691345215, "sum": 68.73297691345215, "min": 68.73297691345215}}, "EndTime": 1601717037.388449, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.319333}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 45 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 901, "sum": 901.0, "min": 901}, "Total Records Seen": {"count": 1, "max": 223816, "sum": 223816.0, "min": 223816}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 46, "sum": 46.0, "min": 46}}, "EndTime": 1601717037.388649, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 44}, "StartTime": 1601717037.319687}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=71928.1670165 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_classification_accuracy <score>=0.78125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_classification_cross_entropy <loss>=0.580536305904
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, batch=0 train binary_f_1.000 <score>=0.8
    [2020-10-03 09:23:57.450] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 92, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, train binary_classification_accuracy <score>=0.765625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, train binary_classification_cross_entropy <loss>=0.581411746144
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=45, train binary_f_1.000 <score>=0.798047795355
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.235116958618164, "sum": 62.235116958618164, "min": 62.235116958618164}}, "EndTime": 1601717037.451123, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.38852}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 46 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 921, "sum": 921.0, "min": 921}, "Total Records Seen": {"count": 1, "max": 228784, "sum": 228784.0, "min": 228784}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 47, "sum": 47.0, "min": 47}}, "EndTime": 1601717037.451296, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 45}, "StartTime": 1601717037.388862}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79410.4507317 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_classification_accuracy <score>=0.79296875
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_classification_cross_entropy <loss>=0.5778850317
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, batch=0 train binary_f_1.000 <score>=0.808664259928
    [2020-10-03 09:23:57.513] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 94, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, train binary_classification_accuracy <score>=0.7671875
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, train binary_classification_cross_entropy <loss>=0.579034873843
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=46, train binary_f_1.000 <score>=0.799123693967
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.150001525878906, "sum": 62.150001525878906, "min": 62.150001525878906}}, "EndTime": 1601717037.513705, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.451179}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 47 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 941, "sum": 941.0, "min": 941}, "Total Records Seen": {"count": 1, "max": 233752, "sum": 233752.0, "min": 233752}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 48, "sum": 48.0, "min": 48}}, "EndTime": 1601717037.513958, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 46}, "StartTime": 1601717037.451522}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79428.3099935 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_classification_accuracy <score>=0.80078125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_classification_cross_entropy <loss>=0.575241565704
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, batch=0 train binary_f_1.000 <score>=0.814545454545
    [2020-10-03 09:23:57.576] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 96, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, train binary_classification_accuracy <score>=0.7703125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, train binary_classification_cross_entropy <loss>=0.576667487621
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=47, train binary_f_1.000 <score>=0.801418439716
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.87813186645508, "sum": 62.87813186645508, "min": 62.87813186645508}}, "EndTime": 1601717037.577169, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.513795}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 48 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 961, "sum": 961.0, "min": 961}, "Total Records Seen": {"count": 1, "max": 238720, "sum": 238720.0, "min": 238720}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 49, "sum": 49.0, "min": 49}}, "EndTime": 1601717037.577325, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 47}, "StartTime": 1601717037.514267}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78633.999917 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_classification_accuracy <score>=0.80078125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_classification_cross_entropy <loss>=0.572606086731
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, batch=0 train binary_f_1.000 <score>=0.814545454545
    [2020-10-03 09:23:57.637] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 98, "duration": 58, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, train binary_classification_accuracy <score>=0.7724609375
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, train binary_classification_cross_entropy <loss>=0.574309718609
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=48, train binary_f_1.000 <score>=0.802909829132
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 60.228824615478516, "sum": 60.228824615478516, "min": 60.228824615478516}}, "EndTime": 1601717037.637791, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.577215}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 49 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 981, "sum": 981.0, "min": 981}, "Total Records Seen": {"count": 1, "max": 243688, "sum": 243688.0, "min": 243688}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 50, "sum": 50.0, "min": 50}}, "EndTime": 1601717037.637964, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 48}, "StartTime": 1601717037.577536}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=82083.158452 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_classification_accuracy <score>=0.8046875
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_classification_cross_entropy <loss>=0.569978892803
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, batch=0 train binary_f_1.000 <score>=0.817518248175
    [2020-10-03 09:23:57.703] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 100, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, train binary_classification_accuracy <score>=0.774609375
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, train binary_classification_cross_entropy <loss>=0.57196174264
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=49, train binary_f_1.000 <score>=0.804340454391
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.66691398620605, "sum": 65.66691398620605, "min": 65.66691398620605}}, "EndTime": 1601717037.703861, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.637841}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 50 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1001, "sum": 1001.0, "min": 1001}, "Total Records Seen": {"count": 1, "max": 248656, "sum": 248656.0, "min": 248656}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 51, "sum": 51.0, "min": 51}}, "EndTime": 1601717037.7041, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 49}, "StartTime": 1601717037.638161}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75208.8987255 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_classification_accuracy <score>=0.8125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_classification_cross_entropy <loss>=0.56736022234
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, batch=0 train binary_f_1.000 <score>=0.823529411765
    [2020-10-03 09:23:57.768] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 102, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, train binary_classification_accuracy <score>=0.777734375
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, train binary_classification_cross_entropy <loss>=0.569623652101
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=50, train binary_f_1.000 <score>=0.806725543478
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.63193893432617, "sum": 64.63193893432617, "min": 64.63193893432617}}, "EndTime": 1601717037.769032, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.703937}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 51 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1021, "sum": 1021.0, "min": 1021}, "Total Records Seen": {"count": 1, "max": 253624, "sum": 253624.0, "min": 253624}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 52, "sum": 52.0, "min": 52}}, "EndTime": 1601717037.769228, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 50}, "StartTime": 1601717037.704371}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=76464.9194592 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_classification_accuracy <score>=0.8125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_classification_cross_entropy <loss>=0.564750313759
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, batch=0 train binary_f_1.000 <score>=0.823529411765
    [2020-10-03 09:23:57.831] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 104, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, train binary_classification_accuracy <score>=0.7794921875
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, train binary_classification_cross_entropy <loss>=0.567295628786
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=51, train binary_f_1.000 <score>=0.808156329652
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.88909912109375, "sum": 62.88909912109375, "min": 62.88909912109375}}, "EndTime": 1601717037.832409, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.7691}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 52 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1041, "sum": 1041.0, "min": 1041}, "Total Records Seen": {"count": 1, "max": 258592, "sum": 258592.0, "min": 258592}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 53, "sum": 53.0, "min": 53}}, "EndTime": 1601717037.832653, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 51}, "StartTime": 1601717037.769487}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78499.5093974 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_classification_accuracy <score>=0.81640625
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_classification_cross_entropy <loss>=0.562149524689
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, batch=0 train binary_f_1.000 <score>=0.826568265683
    [2020-10-03 09:23:57.896] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 106, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, train binary_classification_accuracy <score>=0.78125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, train binary_classification_cross_entropy <loss>=0.564977744222
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=52, train binary_f_1.000 <score>=0.80945899966
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.41712379455566, "sum": 64.41712379455566, "min": 64.41712379455566}}, "EndTime": 1601717037.897419, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.832483}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 53 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1061, "sum": 1061.0, "min": 1061}, "Total Records Seen": {"count": 1, "max": 263560, "sum": 263560.0, "min": 263560}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 54, "sum": 54.0, "min": 54}}, "EndTime": 1601717037.897879, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 52}, "StartTime": 1601717037.832924}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=76291.3442489 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_classification_accuracy <score>=0.8203125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_classification_cross_entropy <loss>=0.559557914734
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, batch=0 train binary_f_1.000 <score>=0.82962962963
    [2020-10-03 09:23:57.962] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 108, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, train binary_classification_accuracy <score>=0.782421875
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, train binary_classification_cross_entropy <loss>=0.5626701653
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=53, train binary_f_1.000 <score>=0.810350697991
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.65387344360352, "sum": 64.65387344360352, "min": 64.65387344360352}}, "EndTime": 1601717037.962859, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.897524}

    [10/03/2020 09:23:57 INFO 140262587922240] #progress_metric: host=algo-1, completed 54 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1081, "sum": 1081.0, "min": 1081}, "Total Records Seen": {"count": 1, "max": 268528, "sum": 268528.0, "min": 268528}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 55, "sum": 55.0, "min": 55}}, "EndTime": 1601717037.963052, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 53}, "StartTime": 1601717037.898176}

    [10/03/2020 09:23:57 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=76443.8804768 records/second
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_classification_accuracy <score>=0.8203125
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_classification_cross_entropy <loss>=0.556975722313
    [10/03/2020 09:23:57 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, batch=0 train binary_f_1.000 <score>=0.82962962963
    [2020-10-03 09:23:58.024] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 110, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, train binary_classification_accuracy <score>=0.784375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, train binary_classification_cross_entropy <loss>=0.560372939706
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=54, train binary_f_1.000 <score>=0.81179679509
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.87081336975098, "sum": 61.87081336975098, "min": 61.87081336975098}}, "EndTime": 1601717038.025154, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717037.962921}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 55 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1101, "sum": 1101.0, "min": 1101}, "Total Records Seen": {"count": 1, "max": 273496, "sum": 273496.0, "min": 273496}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 56, "sum": 56.0, "min": 56}}, "EndTime": 1601717038.025296, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 54}, "StartTime": 1601717037.963259}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79949.7458926 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_classification_accuracy <score>=0.82421875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_classification_cross_entropy <loss>=0.554403185844
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, batch=0 train binary_f_1.000 <score>=0.832713754647
    [2020-10-03 09:23:58.087] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 112, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, train binary_classification_accuracy <score>=0.78671875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, train binary_classification_cross_entropy <loss>=0.558086207509
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=55, train binary_f_1.000 <score>=0.813524590164
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.03889846801758, "sum": 62.03889846801758, "min": 62.03889846801758}}, "EndTime": 1601717038.087566, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.025198}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 56 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1121, "sum": 1121.0, "min": 1121}, "Total Records Seen": {"count": 1, "max": 278464, "sum": 278464.0, "min": 278464}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 57, "sum": 57.0, "min": 57}}, "EndTime": 1601717038.087803, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 55}, "StartTime": 1601717038.02549}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79557.8024542 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_classification_accuracy <score>=0.8359375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_classification_cross_entropy <loss>=0.551840543747
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, batch=0 train binary_f_1.000 <score>=0.842105263158
    [2020-10-03 09:23:58.149] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 114, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, train binary_classification_accuracy <score>=0.78828125
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, train binary_classification_cross_entropy <loss>=0.555810058117
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=56, train binary_f_1.000 <score>=0.814447107155
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.50197982788086, "sum": 61.50197982788086, "min": 61.50197982788086}}, "EndTime": 1601717038.149608, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.087633}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 57 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1141, "sum": 1141.0, "min": 1141}, "Total Records Seen": {"count": 1, "max": 283432, "sum": 283432.0, "min": 283432}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 58, "sum": 58.0, "min": 58}}, "EndTime": 1601717038.149841, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 56}, "StartTime": 1601717038.088078}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80291.0823437 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_classification_accuracy <score>=0.83984375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_classification_cross_entropy <loss>=0.549288034439
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, batch=0 train binary_f_1.000 <score>=0.846441947566
    [2020-10-03 09:23:58.214] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 116, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, train binary_classification_accuracy <score>=0.790234375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, train binary_classification_cross_entropy <loss>=0.553544589877
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=57, train binary_f_1.000 <score>=0.816095890411
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.95285034179688, "sum": 64.95285034179688, "min": 64.95285034179688}}, "EndTime": 1601717038.21506, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.149672}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 58 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1161, "sum": 1161.0, "min": 1161}, "Total Records Seen": {"count": 1, "max": 288400, "sum": 288400.0, "min": 288400}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 59, "sum": 59.0, "min": 59}}, "EndTime": 1601717038.215306, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 57}, "StartTime": 1601717038.150074}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75988.922094 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_classification_accuracy <score>=0.83984375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_classification_cross_entropy <loss>=0.546745657921
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, batch=0 train binary_f_1.000 <score>=0.846441947566
    [2020-10-03 09:23:58.291] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 118, "duration": 74, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, train binary_classification_accuracy <score>=0.793359375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, train binary_classification_cross_entropy <loss>=0.551289877295
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=58, train binary_f_1.000 <score>=0.818524871355
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 76.83396339416504, "sum": 76.83396339416504, "min": 76.83396339416504}}, "EndTime": 1601717038.292465, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.215139}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 59 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1181, "sum": 1181.0, "min": 1181}, "Total Records Seen": {"count": 1, "max": 293368, "sum": 293368.0, "min": 293368}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 60, "sum": 60.0, "min": 60}}, "EndTime": 1601717038.292708, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 58}, "StartTime": 1601717038.215596}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=64313.0583275 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_classification_accuracy <score>=0.84765625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_classification_cross_entropy <loss>=0.544213712215
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, batch=0 train binary_f_1.000 <score>=0.852830188679
    [2020-10-03 09:23:58.359] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 120, "duration": 64, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, train binary_classification_accuracy <score>=0.7966796875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, train binary_classification_cross_entropy <loss>=0.549045994878
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=59, train binary_f_1.000 <score>=0.821103282351
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 66.49494171142578, "sum": 66.49494171142578, "min": 66.49494171142578}}, "EndTime": 1601717038.359581, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.292535}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 60 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1201, "sum": 1201.0, "min": 1201}, "Total Records Seen": {"count": 1, "max": 298336, "sum": 298336.0, "min": 298336}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 61, "sum": 61.0, "min": 61}}, "EndTime": 1601717038.359784, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 59}, "StartTime": 1601717038.293055}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=74243.4040661 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_classification_accuracy <score>=0.8515625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_classification_cross_entropy <loss>=0.541692376137
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, batch=0 train binary_f_1.000 <score>=0.856060606061
    [2020-10-03 09:23:58.423] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 122, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, train binary_classification_accuracy <score>=0.798046875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, train binary_classification_cross_entropy <loss>=0.546813002229
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=60, train binary_f_1.000 <score>=0.822214580468
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.5838508605957, "sum": 63.5838508605957, "min": 63.5838508605957}}, "EndTime": 1601717038.423606, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.359651}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 61 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1221, "sum": 1221.0, "min": 1221}, "Total Records Seen": {"count": 1, "max": 303304, "sum": 303304.0, "min": 303304}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 62, "sum": 62.0, "min": 62}}, "EndTime": 1601717038.423862, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 60}, "StartTime": 1601717038.359988}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77625.9994039 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_classification_cross_entropy <loss>=0.53918170929
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.485] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 124, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, train binary_classification_accuracy <score>=0.800390625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, train binary_classification_cross_entropy <loss>=0.544591024518
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=61, train binary_f_1.000 <score>=0.824035812672
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.396121978759766, "sum": 61.396121978759766, "min": 61.396121978759766}}, "EndTime": 1601717038.48557, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.423685}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 62 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1241, "sum": 1241.0, "min": 1241}, "Total Records Seen": {"count": 1, "max": 308272, "sum": 308272.0, "min": 308272}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 63, "sum": 63.0, "min": 63}}, "EndTime": 1601717038.485846, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 61}, "StartTime": 1601717038.424138}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80331.0122941 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_classification_cross_entropy <loss>=0.536681950092
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.551] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 126, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, train binary_classification_accuracy <score>=0.8017578125
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, train binary_classification_cross_entropy <loss>=0.542380085588
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=62, train binary_f_1.000 <score>=0.825150732127
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.81711769104004, "sum": 65.81711769104004, "min": 65.81711769104004}}, "EndTime": 1601717038.551981, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.48565}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 63 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1261, "sum": 1261.0, "min": 1261}, "Total Records Seen": {"count": 1, "max": 313240, "sum": 313240.0, "min": 313240}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 64, "sum": 64.0, "min": 64}}, "EndTime": 1601717038.552224, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 62}, "StartTime": 1601717038.486137}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75014.7682737 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_classification_cross_entropy <loss>=0.534193217754
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.615] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 128, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, train binary_classification_accuracy <score>=0.8025390625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, train binary_classification_cross_entropy <loss>=0.540180203319
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=63, train binary_f_1.000 <score>=0.825719703499
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.369035720825195, "sum": 63.369035720825195, "min": 63.369035720825195}}, "EndTime": 1601717038.615893, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.552074}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 64 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1281, "sum": 1281.0, "min": 1281}, "Total Records Seen": {"count": 1, "max": 318208, "sum": 318208.0, "min": 318208}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 65, "sum": 65.0, "min": 65}}, "EndTime": 1601717038.616087, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 63}, "StartTime": 1601717038.552491}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77970.7843814 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_classification_cross_entropy <loss>=0.531715631485
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.677] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 130, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, train binary_classification_accuracy <score>=0.80390625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, train binary_classification_cross_entropy <loss>=0.537991517782
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=64, train binary_f_1.000 <score>=0.826777087647
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.34819984436035, "sum": 61.34819984436035, "min": 61.34819984436035}}, "EndTime": 1601717038.677688, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.615951}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 65 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1301, "sum": 1301.0, "min": 1301}, "Total Records Seen": {"count": 1, "max": 323176, "sum": 323176.0, "min": 323176}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 66, "sum": 66.0, "min": 66}}, "EndTime": 1601717038.677949, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 64}, "StartTime": 1601717038.616314}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80470.3035096 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_classification_cross_entropy <loss>=0.529249429703
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.741] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 132, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, train binary_classification_accuracy <score>=0.8052734375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, train binary_classification_cross_entropy <loss>=0.535814011097
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=65, train binary_f_1.000 <score>=0.827895736233
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.11910057067871, "sum": 64.11910057067871, "min": 64.11910057067871}}, "EndTime": 1601717038.742338, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.677784}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 66 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1321, "sum": 1321.0, "min": 1321}, "Total Records Seen": {"count": 1, "max": 328144, "sum": 328144.0, "min": 328144}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 67, "sum": 67.0, "min": 67}}, "EndTime": 1601717038.742556, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 65}, "StartTime": 1601717038.67818}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77024.2830602 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_classification_cross_entropy <loss>=0.526794493198
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.810] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 134, "duration": 65, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, train binary_classification_accuracy <score>=0.80625
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, train binary_classification_cross_entropy <loss>=0.533647760749
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=66, train binary_f_1.000 <score>=0.828729281768
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 67.8567886352539, "sum": 67.8567886352539, "min": 67.8567886352539}}, "EndTime": 1601717038.810722, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.742398}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 67 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1341, "sum": 1341.0, "min": 1341}, "Total Records Seen": {"count": 1, "max": 333112, "sum": 333112.0, "min": 333112}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 68, "sum": 68.0, "min": 68}}, "EndTime": 1601717038.81101, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 66}, "StartTime": 1601717038.742829}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=72720.903587 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_classification_cross_entropy <loss>=0.524351119995
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, batch=0 train binary_f_1.000 <score>=0.862595419847
    [2020-10-03 09:23:58.879] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 136, "duration": 66, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, train binary_classification_accuracy <score>=0.8078125
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, train binary_classification_cross_entropy <loss>=0.53149279654
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=67, train binary_f_1.000 <score>=0.829875518672
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 68.4199333190918, "sum": 68.4199333190918, "min": 68.4199333190918}}, "EndTime": 1601717038.879737, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.810841}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 68 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1361, "sum": 1361.0, "min": 1361}, "Total Records Seen": {"count": 1, "max": 338080, "sum": 338080.0, "min": 338080}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 69, "sum": 69.0, "min": 69}}, "EndTime": 1601717038.880024, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 67}, "StartTime": 1601717038.811287}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=72117.1129762 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_classification_accuracy <score>=0.85546875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_classification_cross_entropy <loss>=0.521919369698
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, batch=0 train binary_f_1.000 <score>=0.857142857143
    [2020-10-03 09:23:58.959] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 138, "duration": 77, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, train binary_classification_accuracy <score>=0.808984375
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, train binary_classification_cross_entropy <loss>=0.529349157214
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=68, train binary_f_1.000 <score>=0.83067867036
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 79.45394515991211, "sum": 79.45394515991211, "min": 79.45394515991211}}, "EndTime": 1601717038.959786, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.879861}

    [10/03/2020 09:23:58 INFO 140262587922240] #progress_metric: host=algo-1, completed 69 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1381, "sum": 1381.0, "min": 1381}, "Total Records Seen": {"count": 1, "max": 343048, "sum": 343048.0, "min": 343048}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 70, "sum": 70.0, "min": 70}}, "EndTime": 1601717038.96, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 68}, "StartTime": 1601717038.880299}

    [10/03/2020 09:23:58 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=62238.4311495 records/second
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_classification_accuracy <score>=0.85546875
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_classification_cross_entropy <loss>=0.519499361515
    [10/03/2020 09:23:58 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, batch=0 train binary_f_1.000 <score>=0.857142857143
    [2020-10-03 09:23:59.020] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 140, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, train binary_classification_accuracy <score>=0.8111328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, train binary_classification_cross_entropy <loss>=0.527216872573
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=69, train binary_f_1.000 <score>=0.832263660017
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 60.77098846435547, "sum": 60.77098846435547, "min": 60.77098846435547}}, "EndTime": 1601717039.021031, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717038.959857}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 70 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1401, "sum": 1401.0, "min": 1401}, "Total Records Seen": {"count": 1, "max": 348016, "sum": 348016.0, "min": 348016}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 71, "sum": 71.0, "min": 71}}, "EndTime": 1601717039.021254, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 69}, "StartTime": 1601717038.960237}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=81261.4401672 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_classification_cross_entropy <loss>=0.517091155052
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, batch=0 train binary_f_1.000 <score>=0.860465116279
    [2020-10-03 09:23:59.083] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 142, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, train binary_classification_accuracy <score>=0.8126953125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, train binary_classification_cross_entropy <loss>=0.525095973909
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=70, train binary_f_1.000 <score>=0.833478034381
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.10613250732422, "sum": 62.10613250732422, "min": 62.10613250732422}}, "EndTime": 1601717039.083659, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.021102}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 71 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1421, "sum": 1421.0, "min": 1421}, "Total Records Seen": {"count": 1, "max": 352984, "sum": 352984.0, "min": 352984}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 72, "sum": 72.0, "min": 72}}, "EndTime": 1601717039.083843, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 70}, "StartTime": 1601717039.021529}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79604.0000917 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_classification_accuracy <score>=0.859375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_classification_cross_entropy <loss>=0.514694929123
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, batch=0 train binary_f_1.000 <score>=0.860465116279
    [2020-10-03 09:23:59.145] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 144, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, train binary_classification_accuracy <score>=0.8150390625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, train binary_classification_cross_entropy <loss>=0.522986488044
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=71, train binary_f_1.000 <score>=0.835504603092
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.383174896240234, "sum": 62.383174896240234, "min": 62.383174896240234}}, "EndTime": 1601717039.146456, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.083716}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 72 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1441, "sum": 1441.0, "min": 1441}, "Total Records Seen": {"count": 1, "max": 357952, "sum": 357952.0, "min": 357952}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 73, "sum": 73.0, "min": 73}}, "EndTime": 1601717039.146683, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 71}, "StartTime": 1601717039.084042}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79171.1871972 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_classification_accuracy <score>=0.86328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_classification_cross_entropy <loss>=0.512310564518
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, batch=0 train binary_f_1.000 <score>=0.863813229572
    [2020-10-03 09:23:59.209] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 146, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, train binary_classification_accuracy <score>=0.816796875
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, train binary_classification_cross_entropy <loss>=0.5208884269
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=72, train binary_f_1.000 <score>=0.83698296837
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.163042068481445, "sum": 63.163042068481445, "min": 63.163042068481445}}, "EndTime": 1601717039.210114, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.146522}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 73 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1461, "sum": 1461.0, "min": 1461}, "Total Records Seen": {"count": 1, "max": 362920, "sum": 362920.0, "min": 362920}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 74, "sum": 74.0, "min": 74}}, "EndTime": 1601717039.210327, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 72}, "StartTime": 1601717039.146925}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78223.9742924 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_classification_accuracy <score>=0.86328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_classification_cross_entropy <loss>=0.509938299656
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, batch=0 train binary_f_1.000 <score>=0.863813229572
    [2020-10-03 09:23:59.283] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 148, "duration": 70, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, train binary_classification_accuracy <score>=0.8173828125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, train binary_classification_cross_entropy <loss>=0.518801812828
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=73, train binary_f_1.000 <score>=0.837532580365
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 72.75986671447754, "sum": 72.75986671447754, "min": 72.75986671447754}}, "EndTime": 1601717039.284822, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.210177}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 74 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1481, "sum": 1481.0, "min": 1481}, "Total Records Seen": {"count": 1, "max": 367888, "sum": 367888.0, "min": 367888}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 75, "sum": 75.0, "min": 75}}, "EndTime": 1601717039.285212, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 73}, "StartTime": 1601717039.211993}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=66420.6985022 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_classification_accuracy <score>=0.86328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_classification_cross_entropy <loss>=0.507578194141
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, batch=0 train binary_f_1.000 <score>=0.863813229572
    [2020-10-03 09:23:59.355] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 150, "duration": 68, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, train binary_classification_accuracy <score>=0.81796875
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, train binary_classification_cross_entropy <loss>=0.516726651788
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=74, train binary_f_1.000 <score>=0.838025721237
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 70.57905197143555, "sum": 70.57905197143555, "min": 70.57905197143555}}, "EndTime": 1601717039.356092, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.284912}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 75 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1501, "sum": 1501.0, "min": 1501}, "Total Records Seen": {"count": 1, "max": 372856, "sum": 372856.0, "min": 372856}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 76, "sum": 76.0, "min": 76}}, "EndTime": 1601717039.356284, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 74}, "StartTime": 1601717039.285481}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=70054.7744341 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_classification_accuracy <score>=0.86328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_classification_cross_entropy <loss>=0.505230247974
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, batch=0 train binary_f_1.000 <score>=0.863813229572
    [2020-10-03 09:23:59.419] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 152, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, train binary_classification_accuracy <score>=0.819140625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, train binary_classification_cross_entropy <loss>=0.514662954211
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=75, train binary_f_1.000 <score>=0.838900487126
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.78006935119629, "sum": 63.78006935119629, "min": 63.78006935119629}}, "EndTime": 1601717039.420303, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.356155}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 76 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1521, "sum": 1521.0, "min": 1521}, "Total Records Seen": {"count": 1, "max": 377824, "sum": 377824.0, "min": 377824}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 77, "sum": 77.0, "min": 77}}, "EndTime": 1601717039.420541, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 75}, "StartTime": 1601717039.35649}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77413.1674109 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_classification_accuracy <score>=0.86328125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_classification_cross_entropy <loss>=0.502894580364
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, batch=0 train binary_f_1.000 <score>=0.863813229572
    [2020-10-03 09:23:59.482] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 154, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, train binary_classification_accuracy <score>=0.8216796875
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, train binary_classification_cross_entropy <loss>=0.51261074543
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=76, train binary_f_1.000 <score>=0.840857591075
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.60204315185547, "sum": 62.60204315185547, "min": 62.60204315185547}}, "EndTime": 1601717039.483445, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.420366}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 77 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1541, "sum": 1541.0, "min": 1541}, "Total Records Seen": {"count": 1, "max": 382792, "sum": 382792.0, "min": 382792}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 78, "sum": 78.0, "min": 78}}, "EndTime": 1601717039.483635, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 76}, "StartTime": 1601717039.420813}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78927.3814686 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_classification_accuracy <score>=0.87109375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_classification_cross_entropy <loss>=0.500571191311
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, batch=0 train binary_f_1.000 <score>=0.871595330739
    [2020-10-03 09:23:59.545] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 156, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, train binary_classification_accuracy <score>=0.8240234375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, train binary_classification_cross_entropy <loss>=0.510570012033
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=77, train binary_f_1.000 <score>=0.842839699983
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.07394599914551, "sum": 62.07394599914551, "min": 62.07394599914551}}, "EndTime": 1601717039.545956, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.483506}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 78 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1561, "sum": 1561.0, "min": 1561}, "Total Records Seen": {"count": 1, "max": 387760, "sum": 387760.0, "min": 387760}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 79, "sum": 79.0, "min": 79}}, "EndTime": 1601717039.546234, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 77}, "StartTime": 1601717039.483851}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79493.1551195 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_classification_accuracy <score>=0.87109375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_classification_cross_entropy <loss>=0.498260200024
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, batch=0 train binary_f_1.000 <score>=0.871595330739
    [2020-10-03 09:23:59.612] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 158, "duration": 64, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, train binary_classification_accuracy <score>=0.825
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, train binary_classification_cross_entropy <loss>=0.508540757
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=78, train binary_f_1.000 <score>=0.843520782396
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 66.54906272888184, "sum": 66.54906272888184, "min": 66.54906272888184}}, "EndTime": 1601717039.613055, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.546018}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 79 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1581, "sum": 1581.0, "min": 1581}, "Total Records Seen": {"count": 1, "max": 392728, "sum": 392728.0, "min": 392728}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 80, "sum": 80.0, "min": 80}}, "EndTime": 1601717039.613285, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 78}, "StartTime": 1601717039.546475}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=74232.2954856 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_classification_accuracy <score>=0.87109375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_classification_cross_entropy <loss>=0.4959615767
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, batch=0 train binary_f_1.000 <score>=0.871595330739
    [2020-10-03 09:23:59.674] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 160, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, train binary_classification_accuracy <score>=0.82578125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, train binary_classification_cross_entropy <loss>=0.506522983313
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=79, train binary_f_1.000 <score>=0.844164919637
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.94710731506348, "sum": 61.94710731506348, "min": 61.94710731506348}}, "EndTime": 1601717039.675503, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.613129}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 80 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1601, "sum": 1601.0, "min": 1601}, "Total Records Seen": {"count": 1, "max": 397696, "sum": 397696.0, "min": 397696}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 81, "sum": 81.0, "min": 81}}, "EndTime": 1601717039.675708, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 79}, "StartTime": 1601717039.613524}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=79736.812533 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_classification_accuracy <score>=0.875
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_classification_cross_entropy <loss>=0.493675351143
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, batch=0 train binary_f_1.000 <score>=0.875
    [2020-10-03 09:23:59.738] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 162, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, train binary_classification_accuracy <score>=0.826953125
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, train binary_classification_cross_entropy <loss>=0.504516680539
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=80, train binary_f_1.000 <score>=0.845213137666
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.02499771118164, "sum": 63.02499771118164, "min": 63.02499771118164}}, "EndTime": 1601717039.738993, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.675572}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 81 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1621, "sum": 1621.0, "min": 1621}, "Total Records Seen": {"count": 1, "max": 402664, "sum": 402664.0, "min": 402664}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 82, "sum": 82.0, "min": 82}}, "EndTime": 1601717039.739247, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 80}, "StartTime": 1601717039.675933}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78311.2873502 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_classification_accuracy <score>=0.87890625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_classification_cross_entropy <loss>=0.491401612759
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, batch=0 train binary_f_1.000 <score>=0.878431372549
    [2020-10-03 09:23:59.800] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 164, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, train binary_classification_accuracy <score>=0.8287109375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, train binary_classification_cross_entropy <loss>=0.502521833777
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=81, train binary_f_1.000 <score>=0.846544181977
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.43617630004883, "sum": 61.43617630004883, "min": 61.43617630004883}}, "EndTime": 1601717039.801006, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.739071}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 82 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1641, "sum": 1641.0, "min": 1641}, "Total Records Seen": {"count": 1, "max": 407632, "sum": 407632.0, "min": 407632}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 83, "sum": 83.0, "min": 83}}, "EndTime": 1601717039.801235, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 81}, "StartTime": 1601717039.739523}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80330.0832395 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_classification_accuracy <score>=0.87890625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_classification_cross_entropy <loss>=0.489140361547
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, batch=0 train binary_f_1.000 <score>=0.878431372549
    [2020-10-03 09:23:59.864] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 166, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, train binary_classification_accuracy <score>=0.8291015625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, train binary_classification_cross_entropy <loss>=0.5005384624
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=82, train binary_f_1.000 <score>=0.846840539121
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.96603584289551, "sum": 63.96603584289551, "min": 63.96603584289551}}, "EndTime": 1601717039.865469, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.801082}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 83 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1661, "sum": 1661.0, "min": 1661}, "Total Records Seen": {"count": 1, "max": 412600, "sum": 412600.0, "min": 412600}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 84, "sum": 84.0, "min": 84}}, "EndTime": 1601717039.865708, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 82}, "StartTime": 1601717039.80147}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77162.0474734 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_classification_accuracy <score>=0.87890625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_classification_cross_entropy <loss>=0.486891627312
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, batch=0 train binary_f_1.000 <score>=0.878431372549
    [2020-10-03 09:23:59.928] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 168, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, train binary_classification_accuracy <score>=0.8302734375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, train binary_classification_cross_entropy <loss>=0.498566512764
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=83, train binary_f_1.000 <score>=0.847837506566
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.48705291748047, "sum": 63.48705291748047, "min": 63.48705291748047}}, "EndTime": 1601717039.929531, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.865541}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 84 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1681, "sum": 1681.0, "min": 1681}, "Total Records Seen": {"count": 1, "max": 417568, "sum": 417568.0, "min": 417568}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 85, "sum": 85.0, "min": 85}}, "EndTime": 1601717039.929801, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 83}, "StartTime": 1601717039.866009}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77709.9532039 records/second
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_classification_accuracy <score>=0.87890625
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_classification_cross_entropy <loss>=0.484655439854
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, batch=0 train binary_f_1.000 <score>=0.878431372549
    [2020-10-03 09:23:59.997] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 170, "duration": 65, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, train binary_classification_accuracy <score>=0.8318359375
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, train binary_classification_cross_entropy <loss>=0.496606004238
    [10/03/2020 09:23:59 INFO 140262587922240] #quality_metric: host=algo-1, epoch=84, train binary_f_1.000 <score>=0.849238312029
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 67.56711006164551, "sum": 67.56711006164551, "min": 67.56711006164551}}, "EndTime": 1601717039.997693, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.929599}

    [10/03/2020 09:23:59 INFO 140262587922240] #progress_metric: host=algo-1, completed 85 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1701, "sum": 1701.0, "min": 1701}, "Total Records Seen": {"count": 1, "max": 422536, "sum": 422536.0, "min": 422536}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 86, "sum": 86.0, "min": 86}}, "EndTime": 1601717039.997963, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 84}, "StartTime": 1601717039.93009}

    [10/03/2020 09:23:59 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=73045.9340048 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_classification_accuracy <score>=0.88671875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_classification_cross_entropy <loss>=0.482431799173
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, batch=0 train binary_f_1.000 <score>=0.885375494071
    [2020-10-03 09:24:00.063] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 172, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, train binary_classification_accuracy <score>=0.834375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, train binary_classification_cross_entropy <loss>=0.494656902552
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=85, train binary_f_1.000 <score>=0.851436580238
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.5660629272461, "sum": 65.5660629272461, "min": 65.5660629272461}}, "EndTime": 1601717040.06385, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717039.99779}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 86 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1721, "sum": 1721.0, "min": 1721}, "Total Records Seen": {"count": 1, "max": 427504, "sum": 427504.0, "min": 427504}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 87, "sum": 87.0, "min": 87}}, "EndTime": 1601717040.064096, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 85}, "StartTime": 1601717039.998247}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75289.8792532 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_classification_accuracy <score>=0.88671875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_classification_cross_entropy <loss>=0.480220705271
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, batch=0 train binary_f_1.000 <score>=0.885375494071
    [2020-10-03 09:24:00.128] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 174, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, train binary_classification_accuracy <score>=0.835546875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, train binary_classification_cross_entropy <loss>=0.492719183862
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=86, train binary_f_1.000 <score>=0.852384291725
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 64.7439956665039, "sum": 64.7439956665039, "min": 64.7439956665039}}, "EndTime": 1601717040.129149, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.063925}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 87 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1741, "sum": 1741.0, "min": 1741}, "Total Records Seen": {"count": 1, "max": 432472, "sum": 432472.0, "min": 432472}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 88, "sum": 88.0, "min": 88}}, "EndTime": 1601717040.12935, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 86}, "StartTime": 1601717040.064375}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=76317.8893101 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_classification_accuracy <score>=0.88671875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_classification_cross_entropy <loss>=0.478022158146
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, batch=0 train binary_f_1.000 <score>=0.885375494071
    [2020-10-03 09:24:00.193] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 176, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, train binary_classification_accuracy <score>=0.83671875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, train binary_classification_cross_entropy <loss>=0.490792843699
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=87, train binary_f_1.000 <score>=0.853333333333
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.12093544006348, "sum": 65.12093544006348, "min": 65.12093544006348}}, "EndTime": 1601717040.194728, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.129218}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 88 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1761, "sum": 1761.0, "min": 1761}, "Total Records Seen": {"count": 1, "max": 437440, "sum": 437440.0, "min": 437440}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 89, "sum": 89.0, "min": 89}}, "EndTime": 1601717040.195014, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 87}, "StartTime": 1601717040.129573}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75764.2940791 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_classification_accuracy <score>=0.88671875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_classification_cross_entropy <loss>=0.475836277008
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, batch=0 train binary_f_1.000 <score>=0.885375494071
    [2020-10-03 09:24:00.259] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 178, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, train binary_classification_accuracy <score>=0.837890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, train binary_classification_cross_entropy <loss>=0.48887783885
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=88, train binary_f_1.000 <score>=0.854385964912
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.09900093078613, "sum": 65.09900093078613, "min": 65.09900093078613}}, "EndTime": 1601717040.260419, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.194806}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 89 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1781, "sum": 1781.0, "min": 1781}, "Total Records Seen": {"count": 1, "max": 442408, "sum": 442408.0, "min": 442408}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 90, "sum": 90.0, "min": 90}}, "EndTime": 1601717040.260669, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 88}, "StartTime": 1601717040.195288}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75832.3984264 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_classification_cross_entropy <loss>=0.473662853241
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.336] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 180, "duration": 73, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, train binary_classification_accuracy <score>=0.8400390625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, train binary_classification_cross_entropy <loss>=0.486974161863
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=89, train binary_f_1.000 <score>=0.856189640035
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 75.81400871276855, "sum": 75.81400871276855, "min": 75.81400871276855}}, "EndTime": 1601717040.336784, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.260499}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 90 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1801, "sum": 1801.0, "min": 1801}, "Total Records Seen": {"count": 1, "max": 447376, "sum": 447376.0, "min": 447376}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 91, "sum": 91.0, "min": 91}}, "EndTime": 1601717040.337041, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 89}, "StartTime": 1601717040.260942}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=65167.0740824 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_classification_cross_entropy <loss>=0.471502065659
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.402] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 182, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, train binary_classification_accuracy <score>=0.841015625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, train binary_classification_cross_entropy <loss>=0.485081762075
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=90, train binary_f_1.000 <score>=0.857042500878
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.3998851776123, "sum": 65.3998851776123, "min": 65.3998851776123}}, "EndTime": 1601717040.402757, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.336863}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 91 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1821, "sum": 1821.0, "min": 1821}, "Total Records Seen": {"count": 1, "max": 452344, "sum": 452344.0, "min": 452344}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 92, "sum": 92.0, "min": 92}}, "EndTime": 1601717040.402984, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 90}, "StartTime": 1601717040.337323}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75519.6353713 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_classification_cross_entropy <loss>=0.469353795052
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.466] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 184, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, train binary_classification_accuracy <score>=0.842578125
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, train binary_classification_cross_entropy <loss>=0.483200639486
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=91, train binary_f_1.000 <score>=0.85824832923
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.72404098510742, "sum": 63.72404098510742, "min": 63.72404098510742}}, "EndTime": 1601717040.466957, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.402845}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 92 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1841, "sum": 1841.0, "min": 1841}, "Total Records Seen": {"count": 1, "max": 457312, "sum": 457312.0, "min": 457312}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 93, "sum": 93.0, "min": 93}}, "EndTime": 1601717040.467157, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 91}, "StartTime": 1601717040.403202}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77547.4212219 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_classification_cross_entropy <loss>=0.467218101025
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.529] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 186, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, train binary_classification_accuracy <score>=0.84375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, train binary_classification_cross_entropy <loss>=0.48133072257
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=92, train binary_f_1.000 <score>=0.859204505456
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.13490867614746, "sum": 63.13490867614746, "min": 63.13490867614746}}, "EndTime": 1601717040.530555, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.467025}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 93 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1861, "sum": 1861.0, "min": 1861}, "Total Records Seen": {"count": 1, "max": 462280, "sum": 462280.0, "min": 462280}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 94, "sum": 94.0, "min": 94}}, "EndTime": 1601717040.530808, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 92}, "StartTime": 1601717040.467386}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78173.4981242 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_classification_cross_entropy <loss>=0.465094953775
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.601] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 188, "duration": 68, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, train binary_classification_accuracy <score>=0.844140625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, train binary_classification_cross_entropy <loss>=0.479472020268
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=93, train binary_f_1.000 <score>=0.85960591133
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 70.74403762817383, "sum": 70.74403762817383, "min": 70.74403762817383}}, "EndTime": 1601717040.601869, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.53063}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 94 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1881, "sum": 1881.0, "min": 1881}, "Total Records Seen": {"count": 1, "max": 467248, "sum": 467248.0, "min": 467248}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 95, "sum": 95.0, "min": 95}}, "EndTime": 1601717040.602126, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 93}, "StartTime": 1601717040.53109}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=69803.6677532 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_classification_cross_entropy <loss>=0.462984323502
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.665] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 190, "duration": 61, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, train binary_classification_accuracy <score>=0.8453125
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, train binary_classification_cross_entropy <loss>=0.477624480426
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=94, train binary_f_1.000 <score>=0.860612460401
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 63.65394592285156, "sum": 63.65394592285156, "min": 63.65394592285156}}, "EndTime": 1601717040.666142, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.601951}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 95 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1901, "sum": 1901.0, "min": 1901}, "Total Records Seen": {"count": 1, "max": 472216, "sum": 472216.0, "min": 472216}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 96, "sum": 96.0, "min": 96}}, "EndTime": 1601717040.666339, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 94}, "StartTime": 1601717040.602459}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=77629.7589664 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_classification_cross_entropy <loss>=0.460886180401
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.731] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 192, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, train binary_classification_accuracy <score>=0.8466796875
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, train binary_classification_cross_entropy <loss>=0.475788059831
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=95, train binary_f_1.000 <score>=0.861722740884
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.5210018157959, "sum": 65.5210018157959, "min": 65.5210018157959}}, "EndTime": 1601717040.732094, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.666208}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 96 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1921, "sum": 1921.0, "min": 1921}, "Total Records Seen": {"count": 1, "max": 477184, "sum": 477184.0, "min": 477184}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 97, "sum": 97.0, "min": 97}}, "EndTime": 1601717040.732339, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 95}, "StartTime": 1601717040.666544}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75362.8564629 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_classification_cross_entropy <loss>=0.458800524473
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.793] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 194, "duration": 59, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, train binary_classification_accuracy <score>=0.8474609375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, train binary_classification_cross_entropy <loss>=0.473962713778
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=96, train binary_f_1.000 <score>=0.862330336683
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 61.26713752746582, "sum": 61.26713752746582, "min": 61.26713752746582}}, "EndTime": 1601717040.793911, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.732172}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 97 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1941, "sum": 1941.0, "min": 1941}, "Total Records Seen": {"count": 1, "max": 482152, "sum": 482152.0, "min": 482152}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 98, "sum": 98.0, "min": 98}}, "EndTime": 1601717040.794152, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 96}, "StartTime": 1601717040.732612}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=80560.5259227 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_classification_cross_entropy <loss>=0.456727325916
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.859] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 196, "duration": 62, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, train binary_classification_accuracy <score>=0.8484375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, train binary_classification_cross_entropy <loss>=0.472148431838
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=97, train binary_f_1.000 <score>=0.863235812478
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.2010440826416, "sum": 65.2010440826416, "min": 65.2010440826416}}, "EndTime": 1601717040.859662, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.793987}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 98 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1961, "sum": 1961.0, "min": 1961}, "Total Records Seen": {"count": 1, "max": 487120, "sum": 487120.0, "min": 487120}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 99, "sum": 99.0, "min": 99}}, "EndTime": 1601717040.859898, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 97}, "StartTime": 1601717040.794427}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75734.0035618 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_classification_cross_entropy <loss>=0.454666554928
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.922] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 198, "duration": 60, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, train binary_classification_accuracy <score>=0.8494140625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, train binary_classification_cross_entropy <loss>=0.470345155895
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=98, train binary_f_1.000 <score>=0.864140969163
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 62.76583671569824, "sum": 62.76583671569824, "min": 62.76583671569824}}, "EndTime": 1601717040.922958, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.859732}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 99 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 1981, "sum": 1981.0, "min": 1981}, "Total Records Seen": {"count": 1, "max": 492088, "sum": 492088.0, "min": 492088}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 100, "sum": 100.0, "min": 100}}, "EndTime": 1601717040.923164, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 98}, "StartTime": 1601717040.860163}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=78708.5528141 records/second
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_classification_accuracy <score>=0.890625
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_classification_cross_entropy <loss>=0.452618181705
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, batch=0 train binary_f_1.000 <score>=0.888888888889
    [2020-10-03 09:24:00.987] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 200, "duration": 63, "num_examples": 20, "num_bytes": 317952}
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, train binary_classification_accuracy <score>=0.849609375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, train binary_classification_cross_entropy <loss>=0.468552854657
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, epoch=99, train binary_f_1.000 <score>=0.864341085271
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, train binary_classification_accuracy <score>=0.849609375
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, train binary_classification_cross_entropy <loss>=0.468552854657
    [10/03/2020 09:24:00 INFO 140262587922240] #quality_metric: host=algo-1, train binary_f_1.000 <score>=0.864341085271
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 65.38009643554688, "sum": 65.38009643554688, "min": 65.38009643554688}}, "EndTime": 1601717040.9888, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.923023}

    [10/03/2020 09:24:00 INFO 140262587922240] #progress_metric: host=algo-1, completed 100 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Batches Since Last Reset": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "Number of Records Since Last Reset": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Total Batches Seen": {"count": 1, "max": 2001, "sum": 2001.0, "min": 2001}, "Total Records Seen": {"count": 1, "max": 497056, "sum": 497056.0, "min": 497056}, "Max Records Seen Between Resets": {"count": 1, "max": 4968, "sum": 4968.0, "min": 4968}, "Reset Count": {"count": 1, "max": 101, "sum": 101.0, "min": 101}}, "EndTime": 1601717040.989046, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "factorization-machines", "epoch": 99}, "StartTime": 1601717040.923393}

    [10/03/2020 09:24:00 INFO 140262587922240] #throughput_metric: host=algo-1, train throughput=75521.003907 records/second
    [10/03/2020 09:24:00 WARNING 140262587922240] wait_for_all_workers will not sync workers since the kv store is not running distributed
    [10/03/2020 09:24:00 INFO 140262587922240] Pulling entire model from kvstore to finalize
    #metrics {"Metrics": {"finalize.time": {"count": 1, "max": 2.2118091583251953, "sum": 2.2118091583251953, "min": 2.2118091583251953}}, "EndTime": 1601717040.991615, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.988888}

    [10/03/2020 09:24:00 INFO 140262587922240] Saved checkpoint to "/tmp/tmpKl3xLW/state-0001.params"
    [2020-10-03 09:24:00.998] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/test", "epoch": 0, "duration": 7078, "num_examples": 1, "num_bytes": 16384}
    [2020-10-03 09:24:01.020] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/test", "epoch": 1, "duration": 22, "num_examples": 7, "num_bytes": 100928}
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 7, "sum": 7.0, "min": 7}, "Number of Batches Since Last Reset": {"count": 1, "max": 7, "sum": 7.0, "min": 7}, "Number of Records Since Last Reset": {"count": 1, "max": 1577, "sum": 1577.0, "min": 1577}, "Total Batches Seen": {"count": 1, "max": 7, "sum": 7.0, "min": 7}, "Total Records Seen": {"count": 1, "max": 1577, "sum": 1577.0, "min": 1577}, "Max Records Seen Between Resets": {"count": 1, "max": 1577, "sum": 1577.0, "min": 1577}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601717041.021083, "Dimensions": {"Host": "algo-1", "Meta": "test_data_iter", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.998637}

    [10/03/2020 09:24:01 INFO 140262587922240] #test_score (algo-1) : ('binary_classification_accuracy', 0.6670894102726697)
    [10/03/2020 09:24:01 INFO 140262587922240] #test_score (algo-1) : ('binary_classification_cross_entropy', 0.6124260442197965)
    [10/03/2020 09:24:01 INFO 140262587922240] #test_score (algo-1) : ('binary_f_1.000', 0.7116968698517299)
    [10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_classification_accuracy <score>=0.667089410273
    [10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_classification_cross_entropy <loss>=0.61242604422
    [10/03/2020 09:24:01 INFO 140262587922240] #quality_metric: host=algo-1, test binary_f_1.000 <score>=0.711696869852
    #metrics {"Metrics": {"totaltime": {"count": 1, "max": 7155.694007873535, "sum": 7155.694007873535, "min": 7155.694007873535}, "setuptime": {"count": 1, "max": 47.18303680419922, "sum": 47.18303680419922, "min": 47.18303680419922}}, "EndTime": 1601717041.022119, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "factorization-machines"}, "StartTime": 1601717040.991675}

    2020-10-03 09:24:09 Uploading - Uploading generated training model
    2020-10-03 09:24:09 Completed - Training job completed
    Training seconds: 58
    Billable seconds: 58
    CPU times: user 417 ms, sys: 43 ms, total: 460 ms
    Wall time: 3min 11s

## 모델 엔드포인트 배포
배포 또한, 매우 간단하게 `deploy()` 메소드로 수행하실 수 있습니다. 배포는 약 5분에서 10분이 소요됩니다.

```python
%%time
instance_type_inference = 'ml.m5.large'
fm_predictor = fm.deploy(instance_type=instance_type_inference, initial_instance_count=1)
```

    -----------!CPU times: user 178 ms, sys: 5.19 ms, total: 183 ms
    Wall time: 5min 31s

## 엔드포인트 추론

일부 테스트로 10개의 인스턴스에 대해서 Rating이 1 인지 0인지를 예측 해보았습니다.
예제로 아래의 결과를 보면 10개 중에 8개의 예측 정확도를 보입니다.
아래의 결과는 실행시에 하이퍼 파라미터의 값에 따라서 다르게 나올 수 있습니다.
```
prediction labels:
 [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
true list:
 [0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0]
Prediction Accuracy:
 [True, False, True, True, True, True, True, False, True, True]
```

### 커스텀 시리얼라이져 구현

아래 URL을 참조하여 구현 하였습니다.
- https://github.com/aws/sagemaker-python-sdk/blob/8fff159389aa63941ccf0c59567c3eb7936a2c62/src/sagemaker/serializers.py

```python

class CustomJSONSerializer(sagemaker.serializers.BaseSerializer):
    """
    # Sample Code:
    # https://github.com/aws/sagemaker-python-sdk/blob/8fff159389aa63941ccf0c59567c3eb7936a2c62/src/sagemaker/serializers.py
    # How to test
        js = CustomJSONSerializer()
        sample = X_test[1000:1001].toarray()
        print(js.serialize(sample))
    """

    CONTENT_TYPE = "application/json"

    def serialize(self, data):
        """Serialize data of various formats to a JSON formatted string.
        Args:
            data (object): Data to be serialized.
        Returns:
            str: The data serialized as a JSON string.
        """

        if isinstance(data, np.ndarray):
            js = {'instances': []}
            for row in data:
                js['instances'].append({'features': row.tolist()})

            return json.dumps(js)
        else:
            print("Not np.ndarray type")
            return json.dumps(data)

fm_predictor.serializer = CustomJSONSerializer()
JSONDeserializer.ACCEPT = 'application/json'
fm_predictor.deserializer = JSONDeserializer()

```

```python
result = fm_predictor.predict(X_test[1000:1010].toarray())
pred_labels = list()
for pred in result['predictions']:
    label = pred['predicted_label']
    pred_labels.append(label)

true_list = Y_test[1000:1010].tolist()
judge = [p == t for p, t in zip (pred_labels,true_list)]

print("true list:\n", true_list)
print("prediction labels:\n", pred_labels)
print("Prediction Accuracy:\n", judge)
```

    true list:
     [0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0]
    prediction labels:
     [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
    Prediction Accuracy:
     [True, False, True, True, True, True, True, False, True, True]

**지금까지 기본적인 사용법을 알아보았으며, 여기에서 실습을 종료하셔도 됩니다. 실습을 일찍 끝내셨거나, 좀 더 깊은 내용을 원하신다면 아래 셀들을 순차적으로 실행해 주세요.
[주의] 실시간 예측을 제공하기 위해 엔드포인트를 계속 실행할 필요가 없는 경우, 과금을 막기 위해 엔드포인트를 삭제해 주세요.**

# 2. (Optional) top-k 추천을 위하여 FM 모델의 모델 파라메터를 사용하여 knn으로 훈련 및 배포하기
---

이제 SageMaker에 모델을 작성하고 저장 했으므로 동일한 FM 모델을 다운로드하여 KNN 모델에 맞게 다시 패키지 할 수 있습니다.

### 모델 아티팩트 다운로드

```python
#!pip install mxnet # 필요한 경우 주석을 해제하여 mxnet을 설치해 주세요
import mxnet as mx
model_file_name = "model.tar.gz"
model_full_path = fm.output_path + "/" + fm.latest_training_job.job_name + "/output/" + model_file_name
print("Model Path: ", model_full_path)

# FM 모델 아티팩트(model.tar.gz) 다운로드
os.system("aws s3 cp " + model_full_path+ " .")

# 모델 아티팩트 압축 해제
os.system("tar xzvf " + model_file_name)
os.system("unzip -o model_algo-1")
os.system("mv symbol.json model-symbol.json")
os.system("mv params model-0000.params")
```

    Model Path:  s3://sagemaker-ap-northeast-2-057716757052/fm-hol/output/factorization-machines-2020-10-03-09-21-21-934/output/model.tar.gz

    0

### 모델 데이터 분리

FM에서 훈련한 파라메터 튜플 ($w_{0}, \mathbf{w}, \mathbf{V}$)을 가져옵니다.

```python
# 모델 추출
m = mx.module.Module.load('./model', 0, False, label_names=['out_label'])
V = m._arg_params['v'].asnumpy() # 2625 x 64
w = m._arg_params['w1_weight'].asnumpy() # 2625 x1
b = m._arg_params['w0_weight'].asnumpy() # 1
print(V.shape, w.shape, b.shape)
```

    (1039, 64) (1039, 1) (1,)

### 데이터셋 재가공

이제 FM 모델에서 추출한 모델 파라메터를 다시 패키지하여 k-NN 모델을 훈련하기 위한 준비를 수행해 보겠습니다.
이 프로세스는 두 개의 데이터셋을 생성합니다.

- Item latent 행렬: k-NN 모델 학습에 사용; $a_i = concat(V, \; w)$
- User latent 행렬: 추론에 사용; $a_u = concat(V, \; 1)$

참고로, 본 핸즈온 코드는 user 및 item ID가 있는 시나리오에만 적용됩니다. 그러나 실제 데이터에는 추가 메타데이터(예: user의 경우 나이, 우편번호, 성별이 포함되고 영화의 경우 영화 장르, 주요 키워드)들이 포함될 수 있습니다. 이러한 경우에는 아래 방법으로 user 및 item 벡터를 추출할 수 있습니다.

- item과 item feature를 $x_i$로 인코딩 후 $\mathbf{V}, \mathbf{w}$에 내적; $a_i = concat(V^T \cdot x_i , \; w^T \cdot x_i)$
- user와 user feature를 $x_u$로 인코딩 후 $\mathbf{V}$에 내적; $a_u = concat(V^T \cdot x_u, \; , 1)$

$a_i$를 사용하여 k-NN 모델을 훈련하고 $a_u$를 사용하여 추론을 수행하시면 됩니다.

```python
# item latent matrix - concat(V[i], w[i]).
knn_item_matrix = np.concatenate((V[nb_users:], w[nb_users:]), axis=1) # 292 x 65
knn_train_label = np.arange(1,nb_movies+1) # [1, 2, 3, ..., 291, 292]

# user latent matrix - concat (V[u], 1)
ones = np.ones(nb_users).reshape((nb_users, 1)) # 747x1
knn_user_matrix = np.concatenate((V[:nb_users], ones), axis=1) # 747 x 65
```

### k-NN 모델 훈련

k-NN 모델은 기본 index_type (faiss.Flat)을 사용합니다.  대규모 데이터셋의 경우 속도가 느려지기에, 이런 경우 더 빠른 훈련을 위해 다른 index_type 매개 변수를 사용할 수 있습니다. index 유형에 대한 자세한 내용은 k-NN 설명서를 참조해 주세요.

```python
print('KNN train features shape = ', knn_item_matrix.shape)
knn_prefix = 'knn'
knn_output_prefix  = 's3://{}/{}/output'.format(bucket, knn_prefix)
knn_train_data_path = writeDatasetToProtobuf(knn_item_matrix, bucket, knn_prefix, train_key, "dense", knn_train_label)
print('Uploaded KNN train data: {}'.format(knn_train_data_path))
```

    KNN train features shape =  (292, 65)
    Uploaded KNN train data: s3://sagemaker-ap-northeast-2-057716757052/knn/train.protobuf

```python
nb_recommendations = 100
knn_image = image_uris.retrieve("knn", session.Session().boto_region_name, version="latest")

knn = sagemaker.estimator.Estimator(knn_image,
    get_execution_role(),
    instance_count=1,
    instance_type=instance_type_training,
    output_path=knn_output_prefix,
    sagemaker_session=sagemaker.Session())

knn.set_hyperparameters(feature_dim=knn_item_matrix.shape[1], k=nb_recommendations,
                        index_metric="INNER_PRODUCT", predictor_type='classifier', sample_size=200000)
fit_input = {'train': knn_train_data_path}
```

    Defaulting to the only supported framework/algorithm version: 1. Ignoring framework/algorithm version: latest.

훈련을 시작합니다. 아래 셀의 수행 시간은 약 4분에서 5분이 소요됩니다.

```python
%%time
knn.fit(fit_input)
knn_model_name =  knn.latest_training_job.job_name
print("Created model: ", knn_model_name)
```

    2020-10-03 09:46:33 Starting - Starting the training job...
    2020-10-03 09:46:35 Starting - Launching requested ML instances...
    2020-10-03 09:47:32 Starting - Preparing the instances for training......
    2020-10-03 09:48:15 Downloading - Downloading input data...
    2020-10-03 09:48:37 Training - Downloading the training image..Docker entrypoint called with argument(s): train
    Running default environment configuration script
    [10/03/2020 09:49:13 INFO 139974427436864] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'index_metric': u'L2', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'_log_level': u'info', u'feature_dim': u'auto', u'faiss_index_ivf_nlists': u'auto', u'epochs': u'1', u'index_type': u'faiss.Flat', u'_faiss_index_nprobe': u'5', u'_kvstore': u'dist_async', u'_num_kv_servers': u'1', u'mini_batch_size': u'5000'}
    [10/03/2020 09:49:13 INFO 139974427436864] Merging with provided configuration from /opt/ml/input/config/hyperparameters.json: {u'sample_size': u'200000', u'feature_dim': u'65', u'index_metric': u'INNER_PRODUCT', u'predictor_type': u'classifier', u'k': u'100'}
    [10/03/2020 09:49:13 INFO 139974427436864] Final configuration: {u'index_metric': u'INNER_PRODUCT', u'predictor_type': u'classifier', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'_log_level': u'info', u'feature_dim': u'65', u'faiss_index_ivf_nlists': u'auto', u'sample_size': u'200000', u'epochs': u'1', u'index_type': u'faiss.Flat', u'_faiss_index_nprobe': u'5', u'_kvstore': u'dist_async', u'_num_kv_servers': u'1', u'mini_batch_size': u'5000', u'k': u'100'}
    [10/03/2020 09:49:13 WARNING 139974427436864] Loggers have already been setup.
    [10/03/2020 09:49:13 INFO 139974427436864] Launching parameter server for role scheduler
    [10/03/2020 09:49:13 INFO 139974427436864] {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/7a005467-d037-4108-97ce-286f8302093b', 'ECS_CONTAINER_METADATA_URI_V4': 'http://169.254.170.2/v4/7a005467-d037-4108-97ce-286f8302093b', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'AWS_REGION': 'ap-northeast-2', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-10-03-09-46-33-691', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-171-204.ap-northeast-2.compute.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/5796ccc5-823a-40a7-add4-1d82e7289c88', 'PWD': '/', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:ap-northeast-2:057716757052:training-job/knn-2020-10-03-09-46-33-691', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [10/03/2020 09:49:13 INFO 139974427436864] envs={'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/7a005467-d037-4108-97ce-286f8302093b', 'ECS_CONTAINER_METADATA_URI_V4': 'http://169.254.170.2/v4/7a005467-d037-4108-97ce-286f8302093b', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '1', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.171.204', 'AWS_REGION': 'ap-northeast-2', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-10-03-09-46-33-691', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-171-204.ap-northeast-2.compute.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/5796ccc5-823a-40a7-add4-1d82e7289c88', 'DMLC_ROLE': 'scheduler', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:ap-northeast-2:057716757052:training-job/knn-2020-10-03-09-46-33-691', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [10/03/2020 09:49:13 INFO 139974427436864] Launching parameter server for role server
    [10/03/2020 09:49:13 INFO 139974427436864] {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/7a005467-d037-4108-97ce-286f8302093b', 'ECS_CONTAINER_METADATA_URI_V4': 'http://169.254.170.2/v4/7a005467-d037-4108-97ce-286f8302093b', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'AWS_REGION': 'ap-northeast-2', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-10-03-09-46-33-691', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-171-204.ap-northeast-2.compute.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/5796ccc5-823a-40a7-add4-1d82e7289c88', 'PWD': '/', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:ap-northeast-2:057716757052:training-job/knn-2020-10-03-09-46-33-691', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [10/03/2020 09:49:13 INFO 139974427436864] envs={'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/7a005467-d037-4108-97ce-286f8302093b', 'ECS_CONTAINER_METADATA_URI_V4': 'http://169.254.170.2/v4/7a005467-d037-4108-97ce-286f8302093b', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '1', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.171.204', 'AWS_REGION': 'ap-northeast-2', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-10-03-09-46-33-691', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-171-204.ap-northeast-2.compute.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/5796ccc5-823a-40a7-add4-1d82e7289c88', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:ap-northeast-2:057716757052:training-job/knn-2020-10-03-09-46-33-691', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    [10/03/2020 09:49:13 INFO 139974427436864] Environment: {'ECS_CONTAINER_METADATA_URI': 'http://169.254.170.2/v3/7a005467-d037-4108-97ce-286f8302093b', 'ECS_CONTAINER_METADATA_URI_V4': 'http://169.254.170.2/v4/7a005467-d037-4108-97ce-286f8302093b', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '1', 'SAGEMAKER_HTTP_PORT': '8080', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/opt/amazon/lib/python2.7/site-packages/cv2/../../../../lib:/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'eth0', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.0.171.204', 'AWS_REGION': 'ap-northeast-2', 'SAGEMAKER_METRICS_DIRECTORY': '/opt/ml/output/metrics/sagemaker', 'NVIDIA_VISIBLE_DEVICES': 'void', 'TRAINING_JOB_NAME': 'knn-2020-10-03-09-46-33-691', 'HOME': '/root', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '2', 'HOSTNAME': 'ip-10-0-171-204.ap-northeast-2.compute.internal', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/5796ccc5-823a-40a7-add4-1d82e7289c88', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '1', 'TRAINING_JOB_ARN': 'arn:aws:sagemaker:ap-northeast-2:057716757052:training-job/knn-2020-10-03-09-46-33-691', 'AWS_EXECUTION_ENV': 'AWS_ECS_EC2'}
    Process 60 is a shell:scheduler.
    Process 69 is a shell:server.
    Process 1 is a worker.
    [10/03/2020 09:49:13 INFO 139974427436864] Using default worker.
    [10/03/2020 09:49:13 INFO 139974427436864] Checkpoint loading and saving are disabled.
    [10/03/2020 09:49:13 INFO 139974427436864] nvidia-smi took: 0.0251090526581 secs to identify 0 gpus
    [10/03/2020 09:49:13 INFO 139974427436864] Create Store: dist_async
    [10/03/2020 09:49:14 ERROR 139974427436864] nvidia-smi: failed to run (127): /bin/sh: nvidia-smi: command not found
    [10/03/2020 09:49:14 INFO 139974427436864] Using per-worker sample size = 200000 (Available virtual memory = 6216159232 bytes, GPU free memory = 0 bytes, number of workers = 1). If an out-of-memory error occurs, choose a larger instance type, use dimension reduction, decrease sample_size, and/or decrease mini_batch_size.
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Records Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Max Records Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Reset Count": {"count": 1, "max": 0, "sum": 0.0, "min": 0}}, "EndTime": 1601718554.87305, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1601718554.873006}

    [2020-10-03 09:49:14.873] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 0, "duration": 928, "num_examples": 1, "num_bytes": 89936}
    [2020-10-03 09:49:14.908] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 1, "duration": 34, "num_examples": 1, "num_bytes": 89936}
    [10/03/2020 09:49:14 INFO 139974427436864] push reservoir to kv... 1 num_workers 0 rank
    [10/03/2020 09:49:14 INFO 139974427436864] ...done (292)
    [10/03/2020 09:49:14 INFO 139974427436864] #progress_metric: host=algo-1, completed 100 % of epochs
    #metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 292, "sum": 292.0, "min": 292}, "Total Batches Seen": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Total Records Seen": {"count": 1, "max": 292, "sum": 292.0, "min": 292}, "Max Records Seen Between Resets": {"count": 1, "max": 292, "sum": 292.0, "min": 292}, "Reset Count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718554.928338, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/KNN", "epoch": 0}, "StartTime": 1601718554.87358}

    [10/03/2020 09:49:14 INFO 139974427436864] #throughput_metric: host=algo-1, train throughput=5311.22565201 records/second
    [10/03/2020 09:49:14 INFO 139974427436864] pulled row count... worker 0 rows 292
    [10/03/2020 09:49:15 INFO 139974427436864] pulled... worker 0 data (292, 65) labels (292,) nans 0
    [10/03/2020 09:49:15 INFO 139974427436864] calling index.train...
    [10/03/2020 09:49:15 INFO 139974427436864] ...done calling index.train
    [10/03/2020 09:49:15 INFO 139974427436864] calling index.add...
    [10/03/2020 09:49:15 INFO 139974427436864] ...done calling index.add
    #metrics {"Metrics": {"epochs": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "model.serialize.time": {"count": 1, "max": 2.6509761810302734, "sum": 2.6509761810302734, "min": 2.6509761810302734}, "finalize.time": {"count": 1, "max": 114.50791358947754, "sum": 114.50791358947754, "min": 114.50791358947754}, "initialize.time": {"count": 1, "max": 923.2299327850342, "sum": 923.2299327850342, "min": 923.2299327850342}, "update.time": {"count": 1, "max": 54.33797836303711, "sum": 54.33797836303711, "min": 54.33797836303711}}, "EndTime": 1601718555.046006, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1601718553.944434}

    [10/03/2020 09:49:15 INFO 139974427436864] Test data is not provided.
    #metrics {"Metrics": {"totaltime": {"count": 1, "max": 1755.47194480896, "sum": 1755.47194480896, "min": 1755.47194480896}, "setuptime": {"count": 1, "max": 32.621145248413086, "sum": 32.621145248413086, "min": 32.621145248413086}}, "EndTime": 1601718555.047265, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/KNN"}, "StartTime": 1601718555.046087}

    2020-10-03 09:49:26 Uploading - Uploading generated training model
    2020-10-03 09:49:26 Completed - Training job completed
    Training seconds: 71
    Billable seconds: 71
    Created model:  knn-2020-10-03-09-46-33-691
    CPU times: user 446 ms, sys: 13.6 ms, total: 460 ms
    Wall time: 3min 11s

배치 추론에서 참조할 수 있도록 모델을 생성합니다.

```python
# 다음 단계에서 배치 추론 중에 참조할 수 있도록 모델 저장
sm = boto3.client(service_name='sagemaker')
primary_container = {
    'Image': knn.image_uri,
    'ModelDataUrl': knn.model_data,
}

knn_model = sm.create_model(
        ModelName = knn.latest_training_job.job_name,
        ExecutionRoleArn = knn.role,
        PrimaryContainer = primary_container)
```

### Batch Transform

Amazon SageMaker의 Batch Transform 기능을 사용하면 대규모로 배치 추론 결과를 생성할 수 있습니다.

아래 셀의 실행이 완료되기까지는 약 4분 소요됩니다.

```python
%%time
# 추론 데이터 S3에 업로드
knn_batch_data_path = writeDatasetToProtobuf(knn_user_matrix, bucket, knn_prefix, train_key, "dense")
print("Batch inference data path: ", knn_batch_data_path)

# Transformer 객체 초기화
transformer = sagemaker.transformer.Transformer(
    base_transform_job_name="knn",
    model_name=knn_model_name,
    instance_count=1,
    instance_type=instance_type_inference,
    output_path=knn_output_prefix,
    accept="application/jsonlines; verbose=true"
)

# 변환 작업 시작
transformer.transform(knn_batch_data_path, content_type='application/x-recordio-protobuf')
transformer.wait()

# S3에서 출력 파일 다운로드
results_file_name = "inference_output"
inference_output_file = "knn/output/train.protobuf.out"
s3_client = boto3.client('s3')
s3_client.download_file(bucket, inference_output_file, results_file_name)
with open(results_file_name) as f:
    results = f.readlines()
```

    Batch inference data path:  s3://sagemaker-ap-northeast-2-057716757052/knn/train.protobuf
    ............................2020-10-03T09:54:16.127:[sagemaker logs]: MaxConcurrentTransforms=1, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD
    Docker entrypoint called with argument(s): serve
    Running default environment configuration script
    [10/03/2020 09:54:15 INFO 140511654475584] loaded entry point class algorithm.serve.server_config:config_api
    [10/03/2020 09:54:15 INFO 140511654475584] loading entry points
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator text/csv
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/jsonlines
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/jsonlines
    Docker entrypoint called with argument(s): serve
    Running default environment configuration script
    [10/03/2020 09:54:15 INFO 140511654475584] loaded entry point class algorithm.serve.server_config:config_api
    [10/03/2020 09:54:15 INFO 140511654475584] loading entry points
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator text/csv
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/jsonlines
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/jsonlines
    [10/03/2020 09:54:16 INFO 140511654475584] loaded entry point class algorithm:model
    [10/03/2020 09:54:16 INFO 140511654475584] Number of server workers: 1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Starting gunicorn 19.7.1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
    [2020-10-03 09:54:16 +0000] [1] [INFO] Using worker: sync
    [2020-10-03 09:54:16 +0000] [76] [INFO] Booting worker with pid: 76
    [10/03/2020 09:54:16 INFO 140511654475584] loading model...
    [10/03/2020 09:54:16 INFO 140511654475584] nvidia-smi took: 0.0251379013062 secs to identify 0 gpus
    [10/03/2020 09:54:16 INFO 140511654475584] ...model loaded.
    #metrics {"Metrics": {"execution_parameters.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.114884, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.107669}

    [2020-10-03 09:54:16.243] [tensorio] [info] epoch_stats={"data_pipeline": "/tmp/tmpgCgluv/tmpa3TWtn", "epoch": 0, "duration": 65, "num_examples": 1, "num_bytes": 409356}
    #metrics {"Metrics": {"model.evaluate.time": {"count": 1, "max": 62.840938568115234, "sum": 62.840938568115234, "min": 62.840938568115234}, "invocations.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.369231, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.114966}

    [10/03/2020 09:54:16 INFO 140511654475584] loaded entry point class algorithm:model
    [10/03/2020 09:54:16 INFO 140511654475584] Number of server workers: 1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Starting gunicorn 19.7.1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
    [2020-10-03 09:54:16 +0000] [1] [INFO] Using worker: sync
    [2020-10-03 09:54:16 +0000] [76] [INFO] Booting worker with pid: 76
    [10/03/2020 09:54:16 INFO 140511654475584] loading model...
    [10/03/2020 09:54:16 INFO 140511654475584] nvidia-smi took: 0.0251379013062 secs to identify 0 gpus
    [10/03/2020 09:54:16 INFO 140511654475584] ...model loaded.
    #metrics {"Metrics": {"execution_parameters.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.114884, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.107669}

    [2020-10-03 09:54:16.243] [tensorio] [info] epoch_stats={"data_pipeline": "/tmp/tmpgCgluv/tmpa3TWtn", "epoch": 0, "duration": 65, "num_examples": 1, "num_bytes": 409356}
    #metrics {"Metrics": {"model.evaluate.time": {"count": 1, "max": 62.840938568115234, "sum": 62.840938568115234, "min": 62.840938568115234}, "invocations.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.369231, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.114966}

    2020-10-03T09:54:16.127:[sagemaker logs]: MaxConcurrentTransforms=1, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD
    Docker entrypoint called with argument(s): serve
    Running default environment configuration script
    [10/03/2020 09:54:15 INFO 140511654475584] loaded entry point class algorithm.serve.server_config:config_api
    [10/03/2020 09:54:15 INFO 140511654475584] loading entry points
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator text/csv
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/jsonlines
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/jsonlines
    Docker entrypoint called with argument(s): serve
    Running default environment configuration script
    [10/03/2020 09:54:15 INFO 140511654475584] loaded entry point class algorithm.serve.server_config:config_api
    [10/03/2020 09:54:15 INFO 140511654475584] loading entry points
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator text/csv
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded request iterator application/jsonlines
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/x-recordio-protobuf
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/json
    [10/03/2020 09:54:15 INFO 140511654475584] loaded response encoder application/jsonlines
    [10/03/2020 09:54:16 INFO 140511654475584] loaded entry point class algorithm:model
    [10/03/2020 09:54:16 INFO 140511654475584] Number of server workers: 1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Starting gunicorn 19.7.1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
    [2020-10-03 09:54:16 +0000] [1] [INFO] Using worker: sync
    [2020-10-03 09:54:16 +0000] [76] [INFO] Booting worker with pid: 76
    [10/03/2020 09:54:16 INFO 140511654475584] loading model...
    [10/03/2020 09:54:16 INFO 140511654475584] nvidia-smi took: 0.0251379013062 secs to identify 0 gpus
    [10/03/2020 09:54:16 INFO 140511654475584] ...model loaded.
    #metrics {"Metrics": {"execution_parameters.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.114884, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.107669}

    [2020-10-03 09:54:16.243] [tensorio] [info] epoch_stats={"data_pipeline": "/tmp/tmpgCgluv/tmpa3TWtn", "epoch": 0, "duration": 65, "num_examples": 1, "num_bytes": 409356}
    #metrics {"Metrics": {"model.evaluate.time": {"count": 1, "max": 62.840938568115234, "sum": 62.840938568115234, "min": 62.840938568115234}, "invocations.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.369231, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.114966}

    [10/03/2020 09:54:16 INFO 140511654475584] loaded entry point class algorithm:model
    [10/03/2020 09:54:16 INFO 140511654475584] Number of server workers: 1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Starting gunicorn 19.7.1
    [2020-10-03 09:54:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
    [2020-10-03 09:54:16 +0000] [1] [INFO] Using worker: sync
    [2020-10-03 09:54:16 +0000] [76] [INFO] Booting worker with pid: 76
    [10/03/2020 09:54:16 INFO 140511654475584] loading model...
    [10/03/2020 09:54:16 INFO 140511654475584] nvidia-smi took: 0.0251379013062 secs to identify 0 gpus
    [10/03/2020 09:54:16 INFO 140511654475584] ...model loaded.
    #metrics {"Metrics": {"execution_parameters.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.114884, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.107669}

    [2020-10-03 09:54:16.243] [tensorio] [info] epoch_stats={"data_pipeline": "/tmp/tmpgCgluv/tmpa3TWtn", "epoch": 0, "duration": 65, "num_examples": 1, "num_bytes": 409356}
    #metrics {"Metrics": {"model.evaluate.time": {"count": 1, "max": 62.840938568115234, "sum": 62.840938568115234, "min": 62.840938568115234}, "invocations.count": {"count": 1, "max": 1, "sum": 1.0, "min": 1}}, "EndTime": 1601718856.369231, "Dimensions": {"Host": "UNKNOWN", "Operation": "scoring", "Algorithm": "KNNModel"}, "StartTime": 1601718856.114966}

    CPU times: user 654 ms, sys: 33.5 ms, total: 688 ms
    Wall time: 5min 13s

## 에어라인 추천 (top-k 추론 예시)

아래는 90번 사용자인 BobMotto에 대해서 추천된 에어라인 이름을 보여주고 있습니다.
'airline_dist'는 airline의 유사도를 의미합니다.

```python
import json
pd.options.display.max_rows = 20
test_user_idx = 89 # 인덱스는 0부터 시작하므로 90번 사용자의 인덱스는 89입니다.
u_one_json = json.loads(results[test_user_idx])

airline_id_list = [int(airline_id) for airline_id in u_one_json['labels']]
airline_name_list = [le_item.inverse_transform([airline_name])[0] for airline_name in airline_id_list]
airline_dist_list = [round(distance, 4) for distance in u_one_json['distances']]

recommend_df = pd.DataFrame({'airline_id': airline_id_list,
                             'airline_name': airline_name_list,
                             'airline_dist': airline_dist_list})
user_name = le_user.inverse_transform([test_user_idx])[0]
print("Recommendations for user: ", user_name)
recommend_df.head(30)
```

    Recommendations for user:  BobMotto

      airline_id
      airline_name
      airline_dist

      0
      45
      aircalin
      0.1268

      1
      241
      sun-express
      0.1386

      2
      133
      hawaiian-airlines
      0.1576

      3
      42
      airasia-x
      0.1673

      4
      179
      mango
      0.1757

      ...
      ...
      ...
      ...

      25
      291
      yangon-airways
      0.2760

      26
      192
      olympic-air
      0.2770

      27
      10
      air-arabia
      0.2820

      28
      244
      swiss-international-air-lines
      0.2833

      29
      267
      turkish-airlines
      0.2845

30 rows × 3 columns

```python

```

```python

```

```python

```