# SageMaker/DeepAR demo on household electricity consumption dataset

This notebook complements the following two notebooks:
* [DeepAR introduction notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_synthetic/deepar_synthetic.ipynb).
* [Individual household electric power consumption dataset](https://github.com/amirrezaeian/Individual-household-electric-power-consumption-Data-Set-/blob/master/data_e_power.ipynb).

The household electric power consumption dataset is available at:

http://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption

In summary, the dataset consists of the following attributes:
* `date`: date (dd/mm/yyyy)
* `time`: time (hh:mm:ss)
* `global_active_power`: household global minute-averaged active power (in Kilowatt)
* `global_reactive_power`: household global minute-averaged reactive power (in Kilowatt)
* `voltage`: minute-averaged voltage (in Volt)
* `global_intensity`: household global minute-averaged current intensity (in Ampere)
* `sub_metering_1`: energy sub-metering No.1 (in Watt-per-hour of active energy). It corresponds to the kitchen, containing mainly a dishwasher, an oven and a microwave (hot plates are not electric but gas powered).
* `sub_metering_2`: energy sub-metering No.2 (in Watt-per-hour of active energy). It corresponds to the laundry room, containing a washing-machine, a tumble-drier, a refrigerator and a light.
* `sub_metering_3`: energy sub-metering No.3 (in Watt-per-hour of active energy). It corresponds to an electric water-heater and an air-conditioner.

In particular, we will see how to:
* Prepare the dataset
* Use the SageMaker Python SDK to train a DeepAR model and deploy it
* Make requests to the deployed model to obtain forecasts interactively
* Illustrate advanced features of DeepAR: missing values, additional time features, non-regular frequencies and category information

For more information about DeepAR, see the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html)

```python
%matplotlib inline

import sys
import os
import json
import zipfile
import random
import datetime
from urllib.request import urlretrieve
from dateutil.parser import parse
from random import shuffle

import boto3
import s3fs
import sagemaker
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
from scipy.stats import randint
import seaborn as sns # used for plot interactive graph.

from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from ipywidgets import IntSlider, FloatSlider, Checkbox

from sklearn.model_selection import train_test_split # to split the data into two parts
from sklearn.cross_validation import KFold # use for cross validation
from sklearn.preprocessing import StandardScaler # for normalization
from sklearn.preprocessing import MinMaxScaler
from sklearn.pipeline import Pipeline # pipeline making
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import SelectFromModel
from sklearn import metrics # for the check the error and accuracy of the model
from sklearn.metrics import mean_squared_error,r2_score

## for Deep-learing:
import keras
from keras.layers import Dense
from keras.models import Sequential
# from keras.utils import to_categorical
from keras.optimizers import SGD
from keras.callbacks import EarlyStopping
from keras.utils import np_utils
import itertools
from keras.layers import LSTM
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D
from keras.layers import Dropout

```

    /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)
    Using MXNet backend.

```python
# set random seeds for reproducibility
np.random.seed(42)
random.seed(42)
```

```python
sagemaker_session = sagemaker.Session()
```

Before starting, we can override the default values for the following:
- The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.
- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.

```python
s3_bucket = sagemaker.Session().default_bucket()  # replace with an existing bucket if needed
s3_prefix = 'deepar-household-electricity-notebook'    # prefix used for all data stored within the bucket

role = sagemaker.get_execution_role()             # IAM role to use by SageMaker
```

```python
region = sagemaker_session.boto_region_name

s3_data_path = "s3://{}/{}/data".format(s3_bucket, s3_prefix)
s3_output_path = "s3://{}/{}/output".format(s3_bucket, s3_prefix)
```

Next, we configure the container image to be used for the region that we are running in.

```python
image_name = sagemaker.amazon.amazon_estimator.get_image_uri(region, "forecasting-deepar", "latest")
```

### Importing household electricity consumption dataset

After downloading the original dataset from the UCI ML repository, we load and parse the dataset. In addition, we modify dataset into a time-series formation supported by Pandas, so that we can utilize many features to handle time-series dataset (e.g. datetime, resampling, aggregation, basic statistics, etc.).

* https://www.allaboutcircuits.com/textbook/alternating-current/chpt-11/true-reactive-and-apparent-power/
* https://circuitglobe.com/what-is-power-triangle.html
* https://en.wikipedia.org/wiki/AC_power

### White noise
In discrete time, white noise is a discrete signal whose samples are regarded as a sequence of serially uncorrelated random variables with zero mean and finite variance. Depending on the context, one may also require that the samples be independent and having identical probability distribution (a.k.a. _i.i.d_). In particular, if each sample has a normal distribution with zero mean, the signal is said to be **Gaussian white noise**.

Some properties of white noise:
* White noise is the simplest example of a **stationary process**.

* if the lag is 0, auto-covariance will be a variance of probability distribution. Otherwise, auto-covariance will be 0. That is:

\begin{equation}
   \gamma_l =
    \begin{cases}
        Var[e_t] & \text{for $l = 0$} \\
        0 & \text{for $l \neq 0$}
    \end{cases}
\end{equation}

* if the lag is 0, auto-correlation will be 1. Otherwise, auto-correlation will be 0. That is:

\begin{equation}
   \rho_l =
    \begin{cases}
        1 & \text{for $l = 0$} \\
        0 & \text{for $l \neq 0$}
    \end{cases}
\end{equation}

* Gaussian white noise can be expressed by:

\begin{equation}
e_t \sim \text{ $i.i.d$ } N(\mu,\sigma^2) \text{ for all $t$}
\end{equation}

**Prewhitening:**
A technique to process a time series data to make it behave statistically like white noise. (The 'pre' means that whitening precedes some other analytical approaches enabling to work better if the additive noise is white).

https://datascienceschool.net/view-notebook/6b963e771dc54f8c8cb23437274a86d6/

http://hosting.astro.cornell.edu/~cordes/A6523/Prewhitening.pdf

```python
FILE_NAME = './household_power_consumption.csv'
```

* Note that dataset includes `nan` and `?` as a `string`. They need to be converted to numpy `nan` in importing stage and treated both of them the same.
* Two columns `Date` and `Time` can be merged into one column `Date_Time`.
* The index of dataset need to be reset (with `Date_Time`).

* Data can be downloaded from: http://archive.ics.uci.edu/ml/machine-learning-databases/00235/
* Just open the zip file and grab the file 'household_power_consumption.txt' put it in the directory that you would like to run the code.
* `infer_datetime_format`: to allow speedups for homogeneously formatted datetimes. `pd.read_csv` and `pd.to_datetime` learned a new `infer_datetime_format` keyword which greatly improves parsing perf in many cases. (http://pandas.pydata.org/pandas-docs/version/0.17.1/whatsnew.html#id55)
* `low_memory`: Please refer the following link: https://stackoverflow.com/questions/24251219/pandas-read-csv-low-memory-and-dtype-options)

```python
data = pd.read_csv(FILE_NAME, sep=",", parse_dates={'Date_Time': ['Date', 'Time']},
                  infer_datetime_format=True, na_values=['nan','?'],
                  low_memory=False, index_col='Date_Time')
data.head()
```

    /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/dateutil/parser/__init__.py:46: DeprecationWarning: _timelex is a private class and may break without warning, it will be moved and or renamed in future versions.
      warnings.warn(msg, DeprecationWarning)

      Global_active_power
      Global_reactive_power
      Voltage
      Global_intensity
      Sub_metering_1
      Sub_metering_2
      Sub_metering_3

      Date_Time

      2006-12-16 17:24:00
      4.216
      0.418
      234.84
      18.4
      0.0
      1.0
      17.0

      2006-12-16 17:25:00
      5.360
      0.436
      233.63
      23.0
      0.0
      1.0
      16.0

      2006-12-16 17:26:00
      5.374
      0.498
      233.29
      23.0
      0.0
      2.0
      17.0

      2006-12-16 17:27:00
      5.388
      0.502
      233.74
      23.0
      0.0
      1.0
      17.0

      2006-12-16 17:28:00
      3.666
      0.528
      235.68
      15.8
      0.0
      1.0
      17.0

We need to check the type of data for each column. If some of them has `object` type, they should be converted into the numerical format (e.g. `float64`, `int64`).
For example, we can use the following codes for the above tasks:

```
data = data.convert_objects(convert_numeric=True)
data.info()
```

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 2075259 entries, 2006-12-16 17:24:00 to 2010-11-26 21:02:00
    Data columns (total 7 columns):
    Global_active_power      float64
    Global_reactive_power    float64
    Voltage                  float64
    Global_intensity         float64
    Sub_metering_1           float64
    Sub_metering_2           float64
    Sub_metering_3           float64
    dtypes: float64(7)
    memory usage: 126.7 MB

When we want to build an ML model, we need to understand dataset in the following perspectives:
* The meaning of data for each feature (column)
* Summarized information from the basic statistics
* Relaionship or Association between features
* Features having similar patterns
* Trend or Periodicity
* Outliers, Noisy data, Missing values
* Data type (categorical data, numerical data, etc.)
...

```python
data.describe()
```

      Global_active_power
      Global_reactive_power
      Voltage
      Global_intensity
      Sub_metering_1
      Sub_metering_2
      Sub_metering_3

      count
      2.049280e+06
      2.049280e+06
      2.049280e+06
      2.049280e+06
      2.049280e+06
      2.049280e+06
      2.049280e+06

      mean
      1.091615e+00
      1.237145e-01
      2.408399e+02
      4.627759e+00
      1.121923e+00
      1.298520e+00
      6.458447e+00

      std
      1.057294e+00
      1.127220e-01
      3.239987e+00
      4.444396e+00
      6.153031e+00
      5.822026e+00
      8.437154e+00

      min
      7.600000e-02
      0.000000e+00
      2.232000e+02
      2.000000e-01
      0.000000e+00
      0.000000e+00
      0.000000e+00

      25%
      3.080000e-01
      4.800000e-02
      2.389900e+02
      1.400000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00

      50%
      6.020000e-01
      1.000000e-01
      2.410100e+02
      2.600000e+00
      0.000000e+00
      0.000000e+00
      1.000000e+00

      75%
      1.528000e+00
      1.940000e-01
      2.428900e+02
      6.400000e+00
      0.000000e+00
      1.000000e+00
      1.700000e+01

      max
      1.112200e+01
      1.390000e+00
      2.541500e+02
      4.840000e+01
      8.800000e+01
      8.000000e+01
      3.100000e+01

## Dealing with missing values 'nan' with a test statistic

```python
## finding all columns that have nan:

droping_list_all=[]
for j in range(0,7):
    if not data.iloc[:, j].notnull().all():
        droping_list_all.append(j)
        #print(df.iloc[:,j].unique())
# droping_list_all
```

```python
# filling nan with mean in any columns

for j in range(0,7):
        data.iloc[:,j]=data.iloc[:,j].fillna(data.iloc[:,j].mean())
```

```python
# another sanity check to make sure that there are not more any nan
data.isnull().sum()
data.describe()
```

      Global_active_power
      Global_reactive_power
      Voltage
      Global_intensity
      Sub_metering_1
      Sub_metering_2
      Sub_metering_3

      count
      2.075259e+06
      2.075259e+06
      2.075259e+06
      2.075259e+06
      2.075259e+06
      2.075259e+06
      2.075259e+06

      mean
      1.091615e+00
      1.237145e-01
      2.408399e+02
      4.627759e+00
      1.121923e+00
      1.298520e+00
      6.458447e+00

      std
      1.050655e+00
      1.120142e-01
      3.219643e+00
      4.416490e+00
      6.114397e+00
      5.785470e+00
      8.384178e+00

      min
      7.600000e-02
      0.000000e+00
      2.232000e+02
      2.000000e-01
      0.000000e+00
      0.000000e+00
      0.000000e+00

      25%
      3.100000e-01
      4.800000e-02
      2.390200e+02
      1.400000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00

      50%
      6.300000e-01
      1.020000e-01
      2.409600e+02
      2.800000e+00
      0.000000e+00
      0.000000e+00
      1.000000e+00

      75%
      1.520000e+00
      1.920000e-01
      2.428600e+02
      6.400000e+00
      0.000000e+00
      1.000000e+00
      1.700000e+01

      max
      1.112200e+01
      1.390000e+00
      2.541500e+02
      4.840000e+01
      8.800000e+01
      8.000000e+01
      3.100000e+01

We can make a transformed dataset with different frequency by using `resample()`.

**down-sampling:**

* To transform the original dataset to a lower frequencey
* Summarize or aggregate the higher frequency data points (i.e. original dataset)
* Example: 1 minute-based timestamps → 5 minute-based timestamps

**up-sampling:**

* To transform the original dataset to a higher frequencey
* the lower frequency data points (i.e. original dataset)
* Example: 1 minute-based timestamps → 0.5 minute-based (i.e. 30 second-based) timestamps

https://machinelearningmastery.com/resample-interpolate-time-series-data-python/

**Resampling**

Resampling involves changing the frequency of your time series observations.

Two types of resampling are:

1. Upsampling: Where you increase the frequency of the samples, such as from minutes to seconds. For upsampling, `ffill()` (i.e. forward filling) or `bfill()` (i.e. backward filling) can be required to fill the newly created data points that was not available.
1. Downsampling: Where you decrease the frequency of the samples, such as from days to months. For downsampling, some kind of aggregation operation can be needed. (e.g. `mean()`, `first()`, etc.)

In both cases, data must be invented.

In the case of upsampling, care may be needed in determining how the fine-grained observations are calculated using interpolation. In the case of downsampling, care may be needed in selecting the summary statistics used to calculate the new aggregated values.

There are perhaps two main reasons why you may be interested in resampling your time series data:

1. Problem Framing: Resampling may be required if your data is available at the same frequency that you want to make predictions.
1. Feature Engineering: Resampling can also be used to provide additional structure or insight into the learning problem for supervised learning models.
There is a lot of overlap between these two cases.

For example, you may have daily data and want to predict a monthly problem. You could use the daily data directly or you could downsample it to monthly data and develop your model.

A feature engineering perspective may use observations and summaries of observations from both time scales and more in developing a model.

```python
data['Global_active_power'].resample('M').sum()
```

    Date_Time
    2006-12-31    41817.648460
    2007-01-31    69014.045230
    2007-02-28    56491.069230
    2007-03-31    58863.283615
    2007-04-30    39245.548781
    2007-05-31    44008.872000
    2007-06-30    35729.767447
    2007-07-31    29846.831570
    2007-08-31    34120.475531
    2007-09-30    41874.789230
    2007-10-31    49278.553230
    2007-11-30    55920.827230
    2007-12-31    72605.261615
    2008-01-31    65170.473615
    2008-02-29    49334.346845
    2008-03-31    55591.685615
    2008-04-30    48209.992000
    2008-05-31    45724.043230
    2008-06-30    42945.063615
    2008-07-31    35479.601230
    2008-08-31    12344.063230
    2008-09-30    42667.792000
    2008-10-31    50743.399447
    2008-11-30    59918.584535
    2008-12-31    56911.416668
    2009-01-31    62951.099615
    2009-02-28    50291.953362
    2009-03-31    54761.169230
    2009-04-30    49277.707230
    2009-05-31    45214.196460
    2009-06-30    37149.767696
    2009-07-31    27594.810460
    2009-08-31    30049.032998
    2009-09-30    42631.838845
    2009-10-31    51089.811615
    2009-11-30    55068.733615
    2009-12-31    60907.189230
    2010-01-31    62797.504679
    2010-02-28    55473.889230
    2010-03-31    50368.601679
    2010-04-30    44379.215615
    2010-05-31    48893.491615
    2010-06-30    41887.607230
    2010-07-31    32188.843615
    2010-08-31    29991.384254
    2010-09-30    42026.211946
    2010-10-31    51934.045615
    2010-11-30    44598.388000
    Freq: M, Name: Global_active_power, dtype: float64

## Data Visualization

Below I resample over day, and show the sum and mean of Global_active_power. It is seen that mean and sum of resampled data set, have similar structure.

```python
data.Global_active_power.resample('D').sum().plot(title='Global_active_power resampled over day for sum')
#df.Global_active_power.resample('D').mean().plot(title='Global_active_power resampled over day', color='red')
plt.tight_layout()
plt.show()

data.Global_active_power.resample('D').mean().plot(title='Global_active_power resampled over day for mean', color='red')
plt.tight_layout()
plt.show()
```

Below I show mean and standard deviation (std) of 'Global_intensity' resampled over day.

```python
r = data.Global_intensity.resample('D').agg(['mean', 'std'])
r.plot(subplots = True, title='Global_intensity resampled over day')
plt.show()
```

Below I show mean and standard deviation (std) of 'Global_reactive_power' resampled over day.

```python
r2 = data.Global_reactive_power.resample('D').agg(['mean', 'std'])
r2.plot(subplots = True, title='Global_reactive_power resampled over day', color='red')
plt.show()
```

Below I show sum of 'global_active_power' resampled over day.

```python
# Sum of 'Global_active_power' resampled over month
data['Global_active_power'].resample('M').mean().plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('Global_active_power')
plt.title('Global_active_power per month (averaged over month)')
plt.show()
```

Below I show mean of 'global_active_power' resampled over day.

```python
data['Global_active_power'].resample('Q').mean().plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('Global_active_power')
plt.title('Global_active_power per quarter (averaged over quarter)')
plt.show()
```

It is very important to note from above two plots that resampling over larger time inteval, will diminish the periodicity of system as we expect. This is important for machine learning feature engineering.

mean of 'Voltage' resampled over month

```python
data['Voltage'].resample('M').mean().plot(kind='bar', color='red')
plt.xticks(rotation=60)
plt.ylabel('Voltage')
plt.title('Voltage per quarter (summed over quarter)')
plt.show()
```

```python
data['Sub_metering_1'].resample('M').mean().plot(kind='bar', color='brown')
plt.xticks(rotation=60)
plt.ylabel('Sub_metering_1')
plt.title('Sub_metering_1 per quarter (summed over quarter)')
plt.show()

```

It is seen from the above plots that the mean of 'Volage' over month is pretty much constant compared to other features. This is important again in feature selection.

### Comparison of the mean of different features resampled over day

```python
# Below I compare the mean of different featuresresampled over day.
# specify columns to plot
cols = [0, 1, 2, 3, 5, 6]
i = 1
values = data.resample('D').mean().values
values
```

    array([[  3.05347475e+00,   8.81868687e-02,   2.36243763e+02, ...,
              0.00000000e+00,   1.37878788e+00,   1.24393939e+01],
           [  2.35448611e+00,   1.56948611e-01,   2.40087028e+02, ...,
              1.41180556e+00,   2.90763889e+00,   9.26458333e+00],
           [  1.53043472e+00,   1.12355556e-01,   2.41231694e+02, ...,
              7.38194444e-01,   1.82013889e+00,   9.73472222e+00],
           ...,
           [  1.24739444e+00,   9.19861111e-02,   2.40030965e+02, ...,
              7.61111111e-01,   1.97777778e+00,   8.48888889e+00],
           [  9.93863889e-01,   8.06444444e-02,   2.41536257e+02, ...,
              7.47222222e-01,   2.95833333e-01,   3.52222222e+00],
           [  1.17822961e+00,   9.56658749e-02,   2.40291029e+02, ...,
              8.55106888e-01,   3.04829770e-01,   7.90894695e+00]])

```python
# plot each column
groups = list(data.columns.values)
plt.figure(figsize=(15, 10))
for group in cols:
    plt.subplot(len(cols), 1, i)
    plt.plot(values[:, group])
    plt.title(data.columns[group], y=0.75, loc='right')
    i += 1
plt.show()
```

```python
## resampling over week and computing mean
data.Global_reactive_power.resample('W').mean().plot(color='y', legend=True)
data.Global_active_power.resample('W').mean().plot(color='r', legend=True)
data.Sub_metering_1.resample('W').mean().plot(color='b', legend=True)
data.Global_intensity.resample('W').mean().plot(color='g', legend=True)
plt.show()
```

```python
# Below I show hist plot of the mean of different feature resampled over month
data.Global_active_power.resample('M').mean().plot(kind='hist', alpha=0.3, legend=True )
data.Global_reactive_power.resample('M').mean().plot(kind='hist', alpha=0.3, legend=True)
#df.Voltage.resample('M').sum().plot(kind='hist',color='g', legend=True)
data.Global_intensity.resample('M').mean().plot(kind='hist', alpha=0.3, legend=True)
data.Sub_metering_1.resample('M').mean().plot(kind='hist', alpha=0.3, legend=True)
plt.show()
```

```python
# Below I show hist plot of the mean of different feature resampled over month
data.Global_active_power.resample('M').mean().plot(kind='hist', color='r', legend=True )
#from pyqt_fit import kde
plt.show()
```

### Comparison of the mean of different features resampled over day

```python
## The correlations between 'Global_intensity', 'Global_active_power'
# data_returns = data.pct_change()
sns.jointplot(x='Global_intensity', y='Global_active_power', data=data)
plt.show()
```

```python
## The correlations between 'Voltage' and  'Global_active_power'
sns.jointplot(x='Voltage', y='Global_active_power', data=data)
plt.show()
```

From above two plots it is seen that 'Global_intensity' and 'Global_active_power' correlated. But 'Voltage', 'Global_active_power' are less correlated. This is important observation for machine learning purpose.

### Comparison among features

```python
# Correlations among columns
plt.matshow(data.corr(method='spearman'),vmax=1,vmin=-1,cmap='RdBu')
plt.title('without resampling')
plt.colorbar()
plt.figure(figsize=(40,40))
plt.show()
```

### Comparison of mean of resampled features

```python
# Correlations of mean of features resampled over months
plt.matshow(data.resample('M').mean().corr(method='spearman'),vmax=1,vmin=-1,cmap='RdBu')
plt.title('resampled over month', size=15)
plt.colorbar()
plt.margins(0.02)
plt.matshow(data.resample('A').mean().corr(method='spearman'),vmax=1,vmin=-1,cmap='RdBu')
plt.title('resampled over year', size=15)
plt.colorbar()
plt.show()
```

It is seen from above that with resampling techniques one can change the correlations among features. This is important for feature engineering.

```python
display(data.corr())
pd.plotting.scatter_matrix(data, figsize=(12, 12))
plt.show()
```

      Global_active_power
      Global_reactive_power
      Voltage
      Global_intensity
      Sub_metering_1
      Sub_metering_2
      Sub_metering_3

      Global_active_power
      1.000000
      0.247017
      -0.399762
      0.998889
      0.484401
      0.434569
      0.638555

      Global_reactive_power
      0.247017
      1.000000
      -0.112246
      0.266120
      0.123111
      0.139231
      0.089617

      Voltage
      -0.399762
      -0.112246
      1.000000
      -0.411363
      -0.195976
      -0.167405
      -0.268172

      Global_intensity
      0.998889
      0.266120
      -0.411363
      1.000000
      0.489298
      0.440347
      0.626543

      Sub_metering_1
      0.484401
      0.123111
      -0.195976
      0.489298
      1.000000
      0.054721
      0.102571

      Sub_metering_2
      0.434569
      0.139231
      -0.167405
      0.440347
      0.054721
      1.000000
      0.080872

      Sub_metering_3
      0.638555
      0.089617
      -0.268172
      0.626543
      0.102571
      0.080872
      1.000000

Please check your scatter_matrix as below:

#### StatsModels

`statsmodels` is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. An extensive list of result statistics are available for each estimator. The results are tested against existing statistical packages to ensure that they are correct. The package is released under the open source Modified BSD (3-clause) license. The online documentation is hosted at [statsmodels.org](http://www.statsmodels.org/).

1. Statistics `stats`
    * statistical tests
    * kernel density estimation
    * generalized method of moments

1. Linear regression
    * Linear model
    * Generalized Linear Model (GLM)
    * Robust Linear Model
    * Linear Mixed Effects Model
    * ANOVA (Analysis of Variance)
    * Discrete Dependent Variable

1. Time-Series analysis
    * ARMA/ARIMA process
    * Vector ARMA process

```python
timeseries = data.Global_active_power.resample('D').mean()
# for i in range(num_timeseries):
#     timeseries.append(np.trim_zeros(data.iloc[:,i], trim='f'))
```

```python
timeseries.head()
```

    Date_Time
    2006-12-16    3.053475
    2006-12-17    2.354486
    2006-12-18    1.530435
    2006-12-19    1.157079
    2006-12-20    1.545658
    Freq: D, Name: Global_active_power, dtype: float64

```python
len(timeseries)
```

    1442

### Train and Test splits

Often times one is interested in evaluating the model or tuning its hyperparameters by looking at error metrics on a hold-out test set. Here we split the available data into train and test sets for evaluating the trained model. For standard machine learning tasks such as classification and regression, one typically obtains this split by randomly separating examples into train and test sets. However, in forecasting it is important to do this train/test split based on time rather than by time series.

In this example, we will reserve the last section of each of the time series for evalutation purpose and use only the first part as training data.

```python
# we use minute frequency for the time series
freq = 'D'

# we predict for 60 days
prediction_length = 60

# we also use 60 days as context length,
# this is the number of state updates accomplished before making predictions
context_length = 60
```

We specify here the portion of the data that is used for training: the model sees data from 2006-12-16 to 2008-12-31 for training.

```python
timeseries.head()
```

    Date_Time
    2006-12-16    3.053475
    2006-12-17    2.354486
    2006-12-18    1.530435
    2006-12-19    1.157079
    2006-12-20    1.545658
    Freq: D, Name: Global_active_power, dtype: float64

```python
start_dataset = pd.Timestamp("2006-12-16", freq=freq)
end_training = pd.Timestamp("2010-07-11", freq=freq)
```

The DeepAR JSON input format represents each time series as a JSON object. In the simplest case each time series just consists of a start time stamp (``start``) and a list of values (``target``). For more complex cases, DeepAR also supports the fields ``dynamic_feat`` for time-series features and ``cat`` for categorical features, which we will use  later.

```python
training_data = [
    {
        "start": str(start_dataset),
        "target": timeseries[start_dataset:end_training - 1].tolist()  # We use -1, because pandas indexing includes the upper bound
        # "target": ts[start_dataset:end_training - 1].tolist()  # We use -1, because pandas indexing includes the upper bound
    }
    #for ts in timeseries
]
# print(len(training_data))
```

```python
len(timeseries[start_dataset:end_training -1])
```

    1303

```python
test_data = [
    {
        "start": str(start_dataset),
        "target": timeseries[end_training:end_training + 10 * prediction_length].tolist()  # We use -1, because pandas indexing includes the upper bound
    }
    #for ts in timeseries
]
# print(len(test_data))
```

```python
len(timeseries[end_training:end_training + 10 * prediction_length])
```

    139

Let's now write the dictionary to the `jsonlines` file format that DeepAR understands (it also supports gzipped jsonlines and parquet).

```python
def write_dicts_to_file(path, data):
    with open(path, 'wb') as fp:
        for d in data:
            fp.write(json.dumps(d).encode("utf-8"))
            fp.write("\n".encode('utf-8'))
```

```python
%%time
write_dicts_to_file("train.json", training_data)
write_dicts_to_file("test.json", test_data)
```

    CPU times: user 4 ms, sys: 0 ns, total: 4 ms
    Wall time: 2.18 ms

Now that we have the data files locally, let us copy them to S3 where DeepAR can access them. Depending on your connection, this may take a couple of minutes.

```python
s3 = boto3.resource('s3')
def copy_to_s3(local_file, s3_path, override=False):
    assert s3_path.startswith('s3://')
    split = s3_path.split('/')
    bucket = split[2]
    path = '/'.join(split[3:])
    buk = s3.Bucket(bucket)

    if len(list(buk.objects.filter(Prefix=path))) > 0:
        if not override:
            print('File s3://{}/{} already exists.\nSet override to upload anyway.\n'.format(s3_bucket, s3_path))
            return
        else:
            print('Overwriting existing file')
    with open(local_file, 'rb') as data:
        print('Uploading file to {}'.format(s3_path))
        buk.put_object(Key=path, Body=data)
```

```python
%%time
copy_to_s3("train.json", s3_data_path + "/train/train.json")
copy_to_s3("test.json", s3_data_path + "/test/test.json")
```

    Uploading file to s3://sagemaker-us-west-2-742595989409/deepar-household-electricity-notebook/data/train/train.json
    Uploading file to s3://sagemaker-us-west-2-742595989409/deepar-household-electricity-notebook/data/test/test.json
    CPU times: user 28 ms, sys: 0 ns, total: 28 ms
    Wall time: 122 ms

Let's have a look to what we just wrote to S3.

```python
s3filesystem = s3fs.S3FileSystem()
with s3filesystem.open(s3_data_path + "/train/train.json", 'rb') as fp:
    print(fp.readline().decode("utf-8")[:100] + "...")
```

    {"start": "2006-12-16 00:00:00", "target": [3.0534747474747492, 2.354486111111111, 1.530434722222219...

We are all set with our dataset processing, we can now call DeepAR to train a model and generate predictions.

### Train a model

Here we define the estimator that will launch the training job.

```python
estimator = sagemaker.estimator.Estimator(
    sagemaker_session=sagemaker_session,
    image_name=image_name,
    role=role,
    train_instance_count=1,
    train_instance_type='ml.m4.xlarge',
    base_job_name='deepar-home-electricity-demo',
    output_path=s3_output_path
)
```

Next we need to set the hyperparameters for the training job. For example frequency of the time series used, number of data points the model will look at in the past, number of predicted data points. The other hyperparameters concern the model to train (number of layers, number of cells per layer, likelihood function) and the training options (number of epochs, batch size, learning rate...). We use default parameters for every optional parameter in this case (you can always use [Sagemaker Automated Model Tuning](https://aws.amazon.com/blogs/aws/sagemaker-automatic-model-tuning/) to tune them).

```python
# hyperparameters = {
#     "time_freq": freq,
#     "epochs": "5",
#     "early_stopping_patience": "10",
#     "mini_batch_size": "20",
#     "learning_rate": "0.001",
#     "context_length": str(context_length),
#     "prediction_length": str(prediction_length)
# }
hyperparameters = {
    "time_freq": freq,
    "context_length": str(context_length),
    "prediction_length": str(prediction_length),
    "num_cells": "40",
    "num_layers": "3",
    "likelihood": "gaussian",
    "epochs": "20",
    "mini_batch_size": "32",
    "learning_rate": "0.001",
    "dropout_rate": "0.05",
    "early_stopping_patience": "10"
}
```

```python
estimator.set_hyperparameters(**hyperparameters)
```

We are ready to launch the training job. SageMaker will start an EC2 instance, download the data from S3, start training the model and save the trained model.

If you provide the `test` data channel as we do in this example, DeepAR will also calculate accuracy metrics for the trained model on this test. This is done by predicting the last `prediction_length` points of each time-series in the test set and comparing this to the actual value of the time-series.

**Note:** the next cell may take a few minutes to complete, depending on data size, model complexity, training options.

```python
%%time
data_channels = {
    "train": "{}/train/".format(s3_data_path),
    "test": "{}/test/".format(s3_data_path)
}

estimator.fit(inputs=data_channels)
```

    INFO:sagemaker:Creating training-job with name: deepar-home-electricity-demo-2018-07-27-03-15-18-550

    ......................
    Arguments: train
    [07/27/2018 03:18:46 INFO 140074042410816] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'num_dynamic_feat': u'auto', u'dropout_rate': u'0.10', u'mini_batch_size': u'128', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'num_eval_samples': u'100', u'learning_rate': u'0.001', u'num_cells': u'40', u'num_layers': u'2', u'embedding_dimension': u'10', u'_kvstore': u'auto', u'_num_kv_servers': u'auto', u'cardinality': u'auto', u'likelihood': u'student-t', u'early_stopping_patience': u''}
    [07/27/2018 03:18:46 INFO 140074042410816] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'dropout_rate': u'0.05', u'learning_rate': u'0.001', u'num_cells': u'40', u'prediction_length': u'60', u'epochs': u'20', u'time_freq': u'D', u'context_length': u'60', u'num_layers': u'3', u'mini_batch_size': u'32', u'likelihood': u'gaussian', u'early_stopping_patience': u'10'}
    [07/27/2018 03:18:46 INFO 140074042410816] Final configuration: {u'dropout_rate': u'0.05', u'test_quantiles': u'[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]', u'_tuning_objective_metric': u'', u'num_eval_samples': u'100', u'learning_rate': u'0.001', u'num_layers': u'3', u'epochs': u'20', u'embedding_dimension': u'10', u'num_cells': u'40', u'_num_kv_servers': u'auto', u'mini_batch_size': u'32', u'likelihood': u'gaussian', u'num_dynamic_feat': u'auto', u'cardinality': u'auto', u'_num_gpus': u'auto', u'prediction_length': u'60', u'time_freq': u'D', u'context_length': u'60', u'_kvstore': u'auto', u'early_stopping_patience': u'10'}
    [07/27/2018 03:18:46 INFO 140074042410816] Detected entry point for worker worker
    [07/27/2018 03:18:46 INFO 140074042410816] Using early stopping with patience 10
    [07/27/2018 03:18:46 INFO 140074042410816] [cardinality=auto] `cat` field was NOT found in the file `/opt/ml/input/data/train/train.json` and will NOT be used for training.
    [07/27/2018 03:18:46 INFO 140074042410816] [num_dynamic_feat=auto] `dynamic_feat` field was NOT found in the file `/opt/ml/input/data/train/train.json` and will NOT be used for training.
    [07/27/2018 03:18:46 INFO 140074042410816] Training set statistics:
    [07/27/2018 03:18:46 INFO 140074042410816] Real time series
    [07/27/2018 03:18:46 INFO 140074042410816] number of time series: 1
    [07/27/2018 03:18:46 INFO 140074042410816] number of observations: 1303
    [07/27/2018 03:18:46 INFO 140074042410816] mean target length: 1303
    [07/27/2018 03:18:46 INFO 140074042410816] min/mean/max target: 0.173818051815/1.10822587149/3.31485128403
    [07/27/2018 03:18:46 INFO 140074042410816] mean abs(target): 1.10822587149
    [07/27/2018 03:18:46 INFO 140074042410816] contains missing values: no
    [07/27/2018 03:18:46 INFO 140074042410816] Small number of time series. Doing 10 number of passes over dataset per epoch.
    [07/27/2018 03:18:46 INFO 140074042410816] Test set statistics:
    [07/27/2018 03:18:46 INFO 140074042410816] Real time series
    [07/27/2018 03:18:46 INFO 140074042410816] number of time series: 1
    [07/27/2018 03:18:46 INFO 140074042410816] number of observations: 139
    [07/27/2018 03:18:46 INFO 140074042410816] mean target length: 139
    [07/27/2018 03:18:46 INFO 140074042410816] min/mean/max target: 0.364406943321/0.94621287833/1.88464164734
    [07/27/2018 03:18:46 INFO 140074042410816] mean abs(target): 0.94621287833
    [07/27/2018 03:18:46 INFO 140074042410816] contains missing values: no
    [07/27/2018 03:18:46 INFO 140074042410816] nvidia-smi took: 0.0251779556274 secs to identify 0 gpus
    [07/27/2018 03:18:46 INFO 140074042410816] Number of GPUs being used: 0
    [07/27/2018 03:18:46 INFO 140074042410816] Create Store: local
    #metrics {"Metrics": {"get_graph.time": {"count": 1, "max": 469.3880081176758, "sum": 469.3880081176758, "min": 469.3880081176758}}, "EndTime": 1532661526.713893, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661526.242651}

    [07/27/2018 03:18:46 INFO 140074042410816] Number of GPUs being used: 0
    #metrics {"Metrics": {"initialize.time": {"count": 1, "max": 971.4839458465576, "sum": 971.4839458465576, "min": 971.4839458465576}}, "EndTime": 1532661527.214226, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661526.71398}

    [07/27/2018 03:18:47 INFO 140074042410816] Epoch[0] Batch[0] avg_epoch_loss=1.272068
    [07/27/2018 03:18:48 INFO 140074042410816] Epoch[0] Batch[5] avg_epoch_loss=0.845399
    [07/27/2018 03:18:48 INFO 140074042410816] Epoch[0] Batch [5]#011Speed: 218.46 samples/sec#011loss=0.845399
    [07/27/2018 03:18:48 INFO 140074042410816] processed a total of 319 examples
    #metrics {"Metrics": {"epochs": {"count": 1, "max": 20, "sum": 20.0, "min": 20}, "update.time": {"count": 1, "max": 1658.9720249176025, "sum": 1658.9720249176025, "min": 1658.9720249176025}}, "EndTime": 1532661528.873359, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661527.214291}

    [07/27/2018 03:18:48 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=192.268629997 records/second
    [07/27/2018 03:18:48 INFO 140074042410816] #progress_metric: host=algo-1, completed 5 % of epochs
    [07/27/2018 03:18:48 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:48 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_c0f18e48-f8ab-4af3-abf5-86b6b3a4b369-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 47.096967697143555, "sum": 47.096967697143555, "min": 47.096967697143555}}, "EndTime": 1532661528.921088, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661528.873479}

    [07/27/2018 03:18:49 INFO 140074042410816] Epoch[1] Batch[0] avg_epoch_loss=0.450175
    [07/27/2018 03:18:49 INFO 140074042410816] Epoch[1] Batch[5] avg_epoch_loss=0.410062
    [07/27/2018 03:18:49 INFO 140074042410816] Epoch[1] Batch [5]#011Speed: 230.24 samples/sec#011loss=0.410062
    [07/27/2018 03:18:50 INFO 140074042410816] processed a total of 319 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1497.1671104431152, "sum": 1497.1671104431152, "min": 1497.1671104431152}}, "EndTime": 1532661530.41839, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661528.921164}

    [07/27/2018 03:18:50 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=213.052985582 records/second
    [07/27/2018 03:18:50 INFO 140074042410816] #progress_metric: host=algo-1, completed 10 % of epochs
    [07/27/2018 03:18:50 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:50 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_de513dee-4530-4948-b19a-39bb8395c93e-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 46.9210147857666, "sum": 46.9210147857666, "min": 46.9210147857666}}, "EndTime": 1532661530.465844, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661530.418465}

    [07/27/2018 03:18:50 INFO 140074042410816] Epoch[2] Batch[0] avg_epoch_loss=0.380706
    [07/27/2018 03:18:51 INFO 140074042410816] Epoch[2] Batch[5] avg_epoch_loss=0.315473
    [07/27/2018 03:18:51 INFO 140074042410816] Epoch[2] Batch [5]#011Speed: 221.60 samples/sec#011loss=0.315473
    [07/27/2018 03:18:52 INFO 140074042410816] Epoch[2] Batch[10] avg_epoch_loss=0.326351
    [07/27/2018 03:18:52 INFO 140074042410816] Epoch[2] Batch [10]#011Speed: 223.90 samples/sec#011loss=0.339405
    [07/27/2018 03:18:52 INFO 140074042410816] processed a total of 338 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1643.8491344451904, "sum": 1643.8491344451904, "min": 1643.8491344451904}}, "EndTime": 1532661532.109826, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661530.465917}

    [07/27/2018 03:18:52 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=205.592850456 records/second
    [07/27/2018 03:18:52 INFO 140074042410816] #progress_metric: host=algo-1, completed 15 % of epochs
    [07/27/2018 03:18:52 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:52 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_1858a189-e242-4797-867e-ab3feb7b666d-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 56.439876556396484, "sum": 56.439876556396484, "min": 56.439876556396484}}, "EndTime": 1532661532.166809, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661532.109963}

    [07/27/2018 03:18:52 INFO 140074042410816] Epoch[3] Batch[0] avg_epoch_loss=0.309939
    [07/27/2018 03:18:53 INFO 140074042410816] Epoch[3] Batch[5] avg_epoch_loss=0.259952
    [07/27/2018 03:18:53 INFO 140074042410816] Epoch[3] Batch [5]#011Speed: 219.63 samples/sec#011loss=0.259952
    [07/27/2018 03:18:53 INFO 140074042410816] Epoch[3] Batch[10] avg_epoch_loss=0.223093
    [07/27/2018 03:18:53 INFO 140074042410816] Epoch[3] Batch [10]#011Speed: 227.20 samples/sec#011loss=0.178861
    [07/27/2018 03:18:53 INFO 140074042410816] processed a total of 331 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1633.991003036499, "sum": 1633.991003036499, "min": 1633.991003036499}}, "EndTime": 1532661533.800934, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661532.166879}

    [07/27/2018 03:18:53 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=202.557603709 records/second
    [07/27/2018 03:18:53 INFO 140074042410816] #progress_metric: host=algo-1, completed 20 % of epochs
    [07/27/2018 03:18:53 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:53 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_0701de60-24fb-439e-945d-3dedfa47034d-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 48.83313179016113, "sum": 48.83313179016113, "min": 48.83313179016113}}, "EndTime": 1532661533.850233, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661533.801004}

    [07/27/2018 03:18:54 INFO 140074042410816] Epoch[4] Batch[0] avg_epoch_loss=0.124813
    [07/27/2018 03:18:54 INFO 140074042410816] Epoch[4] Batch[5] avg_epoch_loss=0.172714
    [07/27/2018 03:18:54 INFO 140074042410816] Epoch[4] Batch [5]#011Speed: 225.45 samples/sec#011loss=0.172714
    [07/27/2018 03:18:55 INFO 140074042410816] processed a total of 310 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1531.6309928894043, "sum": 1531.6309928894043, "min": 1531.6309928894043}}, "EndTime": 1532661535.382007, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661533.850316}

    [07/27/2018 03:18:55 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=202.384097219 records/second
    [07/27/2018 03:18:55 INFO 140074042410816] #progress_metric: host=algo-1, completed 25 % of epochs
    [07/27/2018 03:18:55 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:55 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_36d3075b-c282-486d-871b-d399ee9f1e44-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 45.822858810424805, "sum": 45.822858810424805, "min": 45.822858810424805}}, "EndTime": 1532661535.428317, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661535.382078}

    [07/27/2018 03:18:55 INFO 140074042410816] Epoch[5] Batch[0] avg_epoch_loss=0.178906
    [07/27/2018 03:18:56 INFO 140074042410816] Epoch[5] Batch[5] avg_epoch_loss=0.161604
    [07/27/2018 03:18:56 INFO 140074042410816] Epoch[5] Batch [5]#011Speed: 221.68 samples/sec#011loss=0.161604
    [07/27/2018 03:18:56 INFO 140074042410816] processed a total of 302 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1514.8308277130127, "sum": 1514.8308277130127, "min": 1514.8308277130127}}, "EndTime": 1532661536.943273, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661535.428385}

    [07/27/2018 03:18:56 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=199.347825875 records/second
    [07/27/2018 03:18:56 INFO 140074042410816] #progress_metric: host=algo-1, completed 30 % of epochs
    [07/27/2018 03:18:56 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:56 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_3f6d3df6-d554-4ddc-8d21-74d95b98ed84-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 45.50004005432129, "sum": 45.50004005432129, "min": 45.50004005432129}}, "EndTime": 1532661536.989284, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661536.943343}

    [07/27/2018 03:18:57 INFO 140074042410816] Epoch[6] Batch[0] avg_epoch_loss=0.135446
    [07/27/2018 03:18:57 INFO 140074042410816] Epoch[6] Batch[5] avg_epoch_loss=0.114501
    [07/27/2018 03:18:57 INFO 140074042410816] Epoch[6] Batch [5]#011Speed: 222.89 samples/sec#011loss=0.114501
    [07/27/2018 03:18:58 INFO 140074042410816] processed a total of 316 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1476.783037185669, "sum": 1476.783037185669, "min": 1476.783037185669}}, "EndTime": 1532661538.466206, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661536.989359}

    [07/27/2018 03:18:58 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=213.962974431 records/second
    [07/27/2018 03:18:58 INFO 140074042410816] #progress_metric: host=algo-1, completed 35 % of epochs
    [07/27/2018 03:18:58 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:18:58 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_d13c9e29-891f-454b-95ff-d389dd213fef-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 44.64602470397949, "sum": 44.64602470397949, "min": 44.64602470397949}}, "EndTime": 1532661538.511351, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661538.466273}

    [07/27/2018 03:18:58 INFO 140074042410816] Epoch[7] Batch[0] avg_epoch_loss=0.189694
    [07/27/2018 03:18:59 INFO 140074042410816] Epoch[7] Batch[5] avg_epoch_loss=0.141834
    [07/27/2018 03:18:59 INFO 140074042410816] Epoch[7] Batch [5]#011Speed: 223.07 samples/sec#011loss=0.141834
    [07/27/2018 03:19:00 INFO 140074042410816] processed a total of 312 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1551.4729022979736, "sum": 1551.4729022979736, "min": 1551.4729022979736}}, "EndTime": 1532661540.06295, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661538.51142}

    [07/27/2018 03:19:00 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=201.084053695 records/second
    [07/27/2018 03:19:00 INFO 140074042410816] #progress_metric: host=algo-1, completed 40 % of epochs
    [07/27/2018 03:19:00 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:00 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_dd0f6ada-f55e-4089-b999-2c57c062ebe1-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 74.94807243347168, "sum": 74.94807243347168, "min": 74.94807243347168}}, "EndTime": 1532661540.138372, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661540.063028}

    [07/27/2018 03:19:00 INFO 140074042410816] Epoch[8] Batch[0] avg_epoch_loss=0.041331
    [07/27/2018 03:19:01 INFO 140074042410816] Epoch[8] Batch[5] avg_epoch_loss=0.101388
    [07/27/2018 03:19:01 INFO 140074042410816] Epoch[8] Batch [5]#011Speed: 221.49 samples/sec#011loss=0.101388
    [07/27/2018 03:19:01 INFO 140074042410816] Epoch[8] Batch[10] avg_epoch_loss=0.103547
    [07/27/2018 03:19:01 INFO 140074042410816] Epoch[8] Batch [10]#011Speed: 222.41 samples/sec#011loss=0.106138
    [07/27/2018 03:19:01 INFO 140074042410816] processed a total of 333 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1634.108066558838, "sum": 1634.108066558838, "min": 1634.108066558838}}, "EndTime": 1532661541.772616, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661540.138446}

    [07/27/2018 03:19:01 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=203.765076203 records/second
    [07/27/2018 03:19:01 INFO 140074042410816] #progress_metric: host=algo-1, completed 45 % of epochs
    [07/27/2018 03:19:01 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:01 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_5186df37-c53f-4195-a4b6-d70c1b17804c-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 63.11511993408203, "sum": 63.11511993408203, "min": 63.11511993408203}}, "EndTime": 1532661541.836224, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661541.772702}

    [07/27/2018 03:19:02 INFO 140074042410816] Epoch[9] Batch[0] avg_epoch_loss=0.125514
    [07/27/2018 03:19:02 INFO 140074042410816] Epoch[9] Batch[5] avg_epoch_loss=0.074293
    [07/27/2018 03:19:02 INFO 140074042410816] Epoch[9] Batch [5]#011Speed: 226.06 samples/sec#011loss=0.074293
    [07/27/2018 03:19:03 INFO 140074042410816] Epoch[9] Batch[10] avg_epoch_loss=0.035307
    [07/27/2018 03:19:03 INFO 140074042410816] Epoch[9] Batch [10]#011Speed: 221.50 samples/sec#011loss=-0.011475
    [07/27/2018 03:19:03 INFO 140074042410816] processed a total of 324 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1648.5710144042969, "sum": 1648.5710144042969, "min": 1648.5710144042969}}, "EndTime": 1532661543.48493, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661541.836296}

    [07/27/2018 03:19:03 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=196.518469233 records/second
    [07/27/2018 03:19:03 INFO 140074042410816] #progress_metric: host=algo-1, completed 50 % of epochs
    [07/27/2018 03:19:03 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:03 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_40df461c-4580-46f5-8987-d1d28505c79a-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 49.41511154174805, "sum": 49.41511154174805, "min": 49.41511154174805}}, "EndTime": 1532661543.534826, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661543.485018}

    [07/27/2018 03:19:03 INFO 140074042410816] Epoch[10] Batch[0] avg_epoch_loss=0.083117
    [07/27/2018 03:19:04 INFO 140074042410816] Epoch[10] Batch[5] avg_epoch_loss=0.073549
    [07/27/2018 03:19:04 INFO 140074042410816] Epoch[10] Batch [5]#011Speed: 224.92 samples/sec#011loss=0.073549
    [07/27/2018 03:19:05 INFO 140074042410816] Epoch[10] Batch[10] avg_epoch_loss=0.068213
    [07/27/2018 03:19:05 INFO 140074042410816] Epoch[10] Batch [10]#011Speed: 214.43 samples/sec#011loss=0.061811
    [07/27/2018 03:19:05 INFO 140074042410816] processed a total of 349 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1659.013032913208, "sum": 1659.013032913208, "min": 1659.013032913208}}, "EndTime": 1532661545.193972, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661543.534898}

    [07/27/2018 03:19:05 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=210.350440248 records/second
    [07/27/2018 03:19:05 INFO 140074042410816] #progress_metric: host=algo-1, completed 55 % of epochs
    [07/27/2018 03:19:05 INFO 140074042410816] loss did not improve for 1 epochs
    [07/27/2018 03:19:05 INFO 140074042410816] Epoch[11] Batch[0] avg_epoch_loss=0.033785
    [07/27/2018 03:19:06 INFO 140074042410816] Epoch[11] Batch[5] avg_epoch_loss=0.054952
    [07/27/2018 03:19:06 INFO 140074042410816] Epoch[11] Batch [5]#011Speed: 215.06 samples/sec#011loss=0.054952
    [07/27/2018 03:19:06 INFO 140074042410816] processed a total of 309 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1523.3738422393799, "sum": 1523.3738422393799, "min": 1523.3738422393799}}, "EndTime": 1532661546.717783, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661545.194053}

    [07/27/2018 03:19:06 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=202.821246749 records/second
    [07/27/2018 03:19:06 INFO 140074042410816] #progress_metric: host=algo-1, completed 60 % of epochs
    [07/27/2018 03:19:06 INFO 140074042410816] loss did not improve for 2 epochs
    [07/27/2018 03:19:06 INFO 140074042410816] Epoch[12] Batch[0] avg_epoch_loss=0.125290
    [07/27/2018 03:19:07 INFO 140074042410816] Epoch[12] Batch[5] avg_epoch_loss=0.090269
    [07/27/2018 03:19:07 INFO 140074042410816] Epoch[12] Batch [5]#011Speed: 225.58 samples/sec#011loss=0.090269
    [07/27/2018 03:19:08 INFO 140074042410816] Epoch[12] Batch[10] avg_epoch_loss=0.041725
    [07/27/2018 03:19:08 INFO 140074042410816] Epoch[12] Batch [10]#011Speed: 222.48 samples/sec#011loss=-0.016528
    [07/27/2018 03:19:08 INFO 140074042410816] processed a total of 326 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1660.499095916748, "sum": 1660.499095916748, "min": 1660.499095916748}}, "EndTime": 1532661548.378843, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661546.717875}

    [07/27/2018 03:19:08 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=196.311152772 records/second
    [07/27/2018 03:19:08 INFO 140074042410816] #progress_metric: host=algo-1, completed 65 % of epochs
    [07/27/2018 03:19:08 INFO 140074042410816] loss did not improve for 3 epochs
    [07/27/2018 03:19:08 INFO 140074042410816] Epoch[13] Batch[0] avg_epoch_loss=0.038031
    [07/27/2018 03:19:09 INFO 140074042410816] Epoch[13] Batch[5] avg_epoch_loss=0.054937
    [07/27/2018 03:19:09 INFO 140074042410816] Epoch[13] Batch [5]#011Speed: 207.99 samples/sec#011loss=0.054937
    [07/27/2018 03:19:10 INFO 140074042410816] Epoch[13] Batch[10] avg_epoch_loss=0.036359
    [07/27/2018 03:19:10 INFO 140074042410816] Epoch[13] Batch [10]#011Speed: 221.30 samples/sec#011loss=0.014066
    [07/27/2018 03:19:10 INFO 140074042410816] processed a total of 342 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1699.0950107574463, "sum": 1699.0950107574463, "min": 1699.0950107574463}}, "EndTime": 1532661550.078365, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661548.378929}

    [07/27/2018 03:19:10 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=201.272015504 records/second
    [07/27/2018 03:19:10 INFO 140074042410816] #progress_metric: host=algo-1, completed 70 % of epochs
    [07/27/2018 03:19:10 INFO 140074042410816] loss did not improve for 4 epochs
    [07/27/2018 03:19:10 INFO 140074042410816] Epoch[14] Batch[0] avg_epoch_loss=0.075549
    [07/27/2018 03:19:11 INFO 140074042410816] Epoch[14] Batch[5] avg_epoch_loss=0.037262
    [07/27/2018 03:19:11 INFO 140074042410816] Epoch[14] Batch [5]#011Speed: 219.56 samples/sec#011loss=0.037262
    [07/27/2018 03:19:11 INFO 140074042410816] Epoch[14] Batch[10] avg_epoch_loss=0.039390
    [07/27/2018 03:19:11 INFO 140074042410816] Epoch[14] Batch [10]#011Speed: 224.12 samples/sec#011loss=0.041944
    [07/27/2018 03:19:11 INFO 140074042410816] processed a total of 330 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1684.4029426574707, "sum": 1684.4029426574707, "min": 1684.4029426574707}}, "EndTime": 1532661551.76323, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661550.078429}

    [07/27/2018 03:19:11 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=195.900116044 records/second
    [07/27/2018 03:19:11 INFO 140074042410816] #progress_metric: host=algo-1, completed 75 % of epochs
    [07/27/2018 03:19:11 INFO 140074042410816] loss did not improve for 5 epochs
    [07/27/2018 03:19:12 INFO 140074042410816] Epoch[15] Batch[0] avg_epoch_loss=0.021203
    [07/27/2018 03:19:12 INFO 140074042410816] Epoch[15] Batch[5] avg_epoch_loss=0.019402
    [07/27/2018 03:19:12 INFO 140074042410816] Epoch[15] Batch [5]#011Speed: 225.40 samples/sec#011loss=0.019402
    [07/27/2018 03:19:13 INFO 140074042410816] Epoch[15] Batch[10] avg_epoch_loss=0.040792
    [07/27/2018 03:19:13 INFO 140074042410816] Epoch[15] Batch [10]#011Speed: 217.36 samples/sec#011loss=0.066460
    [07/27/2018 03:19:13 INFO 140074042410816] processed a total of 325 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1688.2209777832031, "sum": 1688.2209777832031, "min": 1688.2209777832031}}, "EndTime": 1532661553.451881, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661551.763318}

    [07/27/2018 03:19:13 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=192.498137011 records/second
    [07/27/2018 03:19:13 INFO 140074042410816] #progress_metric: host=algo-1, completed 80 % of epochs
    [07/27/2018 03:19:13 INFO 140074042410816] loss did not improve for 6 epochs
    [07/27/2018 03:19:13 INFO 140074042410816] Epoch[16] Batch[0] avg_epoch_loss=-0.006675
    [07/27/2018 03:19:14 INFO 140074042410816] Epoch[16] Batch[5] avg_epoch_loss=0.010563
    [07/27/2018 03:19:14 INFO 140074042410816] Epoch[16] Batch [5]#011Speed: 219.53 samples/sec#011loss=0.010563
    [07/27/2018 03:19:14 INFO 140074042410816] processed a total of 299 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1507.7879428863525, "sum": 1507.7879428863525, "min": 1507.7879428863525}}, "EndTime": 1532661554.9601, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661553.451944}

    [07/27/2018 03:19:14 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=198.287036097 records/second
    [07/27/2018 03:19:14 INFO 140074042410816] #progress_metric: host=algo-1, completed 85 % of epochs
    [07/27/2018 03:19:14 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:15 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_e27fd8e1-f54e-4ab5-ab91-87fec4f72123-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 55.8319091796875, "sum": 55.8319091796875, "min": 55.8319091796875}}, "EndTime": 1532661555.016606, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661554.960187}

    [07/27/2018 03:19:15 INFO 140074042410816] Epoch[17] Batch[0] avg_epoch_loss=-0.001955
    [07/27/2018 03:19:15 INFO 140074042410816] Epoch[17] Batch[5] avg_epoch_loss=-0.004937
    [07/27/2018 03:19:15 INFO 140074042410816] Epoch[17] Batch [5]#011Speed: 216.02 samples/sec#011loss=-0.004937
    [07/27/2018 03:19:16 INFO 140074042410816] Epoch[17] Batch[10] avg_epoch_loss=0.002667
    [07/27/2018 03:19:16 INFO 140074042410816] Epoch[17] Batch [10]#011Speed: 225.72 samples/sec#011loss=0.011793
    [07/27/2018 03:19:16 INFO 140074042410816] processed a total of 329 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1667.4549579620361, "sum": 1667.4549579620361, "min": 1667.4549579620361}}, "EndTime": 1532661556.684201, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661555.016683}

    [07/27/2018 03:19:16 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=197.291643279 records/second
    [07/27/2018 03:19:16 INFO 140074042410816] #progress_metric: host=algo-1, completed 90 % of epochs
    [07/27/2018 03:19:16 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:16 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_cb82d8d0-d122-4842-80a4-8f69191b4d46-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 49.47805404663086, "sum": 49.47805404663086, "min": 49.47805404663086}}, "EndTime": 1532661556.734169, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661556.684287}

    [07/27/2018 03:19:16 INFO 140074042410816] Epoch[18] Batch[0] avg_epoch_loss=-0.043327
    [07/27/2018 03:19:17 INFO 140074042410816] Epoch[18] Batch[5] avg_epoch_loss=0.008369
    [07/27/2018 03:19:17 INFO 140074042410816] Epoch[18] Batch [5]#011Speed: 225.63 samples/sec#011loss=0.008369
    [07/27/2018 03:19:18 INFO 140074042410816] processed a total of 292 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1511.1079216003418, "sum": 1511.1079216003418, "min": 1511.1079216003418}}, "EndTime": 1532661558.245413, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661556.734244}

    [07/27/2018 03:19:18 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=193.22237955 records/second
    [07/27/2018 03:19:18 INFO 140074042410816] #progress_metric: host=algo-1, completed 95 % of epochs
    [07/27/2018 03:19:18 INFO 140074042410816] loss did not improve for 1 epochs
    [07/27/2018 03:19:18 INFO 140074042410816] Epoch[19] Batch[0] avg_epoch_loss=0.002925
    [07/27/2018 03:19:19 INFO 140074042410816] Epoch[19] Batch[5] avg_epoch_loss=-0.027517
    [07/27/2018 03:19:19 INFO 140074042410816] Epoch[19] Batch [5]#011Speed: 218.34 samples/sec#011loss=-0.027517
    [07/27/2018 03:19:19 INFO 140074042410816] processed a total of 320 examples
    #metrics {"Metrics": {"update.time": {"count": 1, "max": 1495.6309795379639, "sum": 1495.6309795379639, "min": 1495.6309795379639}}, "EndTime": 1532661559.741463, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661558.24548}

    [07/27/2018 03:19:19 INFO 140074042410816] #throughput_metric: host=algo-1, train throughput=213.939775141 records/second
    [07/27/2018 03:19:19 INFO 140074042410816] #progress_metric: host=algo-1, completed 100 % of epochs
    [07/27/2018 03:19:19 INFO 140074042410816] best epoch loss so far
    [07/27/2018 03:19:19 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/state_bdd9a716-e304-4987-a6ad-8ff21edbd8dc-0000.params"
    #metrics {"Metrics": {"state.serialize.time": {"count": 1, "max": 45.989990234375, "sum": 45.989990234375, "min": 45.989990234375}}, "EndTime": 1532661559.78796, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661559.741538}

    [07/27/2018 03:19:19 INFO 140074042410816] Loading parameters from best epoch (19)
    #metrics {"Metrics": {"state.deserialize.time": {"count": 1, "max": 18.826007843017578, "sum": 18.826007843017578, "min": 18.826007843017578}}, "EndTime": 1532661559.806988, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661559.788032}

    [07/27/2018 03:19:19 INFO 140074042410816] Final loss: -0.0369148883969 (occurred at epoch 19)
    [07/27/2018 03:19:19 INFO 140074042410816] #quality_metric: host=algo-1, train final_loss <loss>=-0.0369148883969
    [07/27/2018 03:19:19 INFO 140074042410816] Worker algo-1 finished training.
    [07/27/2018 03:19:19 WARNING 140074042410816] wait_for_all_workers will not sync workers since the kv store is not running distributed
    [07/27/2018 03:19:19 INFO 140074042410816] All workers finished. Serializing model for prediction.
    #metrics {"Metrics": {"get_graph.time": {"count": 1, "max": 2714.4081592559814, "sum": 2714.4081592559814, "min": 2714.4081592559814}}, "EndTime": 1532661562.522, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661559.807052}

    [07/27/2018 03:19:25 INFO 140074042410816] Number of GPUs being used: 0
    #metrics {"Metrics": {"finalize.time": {"count": 1, "max": 5285.989046096802, "sum": 5285.989046096802, "min": 5285.989046096802}}, "EndTime": 1532661565.093545, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661562.522086}

    [07/27/2018 03:19:25 INFO 140074042410816] Serializing to /opt/ml/model/model_algo-1
    [07/27/2018 03:19:25 INFO 140074042410816] Saved checkpoint to "/opt/ml/model/model_algo-1-0000.params"
    #metrics {"Metrics": {"model.serialize.time": {"count": 1, "max": 420.93801498413086, "sum": 420.93801498413086, "min": 420.93801498413086}}, "EndTime": 1532661565.514611, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661565.093624}

    [07/27/2018 03:19:25 INFO 140074042410816] Successfully serialized the model for prediction.
    [07/27/2018 03:19:25 INFO 140074042410816] Evaluating model accuracy on testset using 100 samples
    #metrics {"Metrics": {"model.bind.time": {"count": 1, "max": 0.03886222839355469, "sum": 0.03886222839355469, "min": 0.03886222839355469}}, "EndTime": 1532661565.515471, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661565.514672}

    #metrics {"Metrics": {"model.score.time": {"count": 1, "max": 3564.686059951782, "sum": 3564.686059951782, "min": 3564.686059951782}}, "EndTime": 1532661569.080128, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661565.515531}

    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, RMSE): 0.38677016793
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, mean_wQuantileLoss): 0.203314
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.1]): 0.119702
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.2]): 0.186436
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.3]): 0.232395
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.4]): 0.260534
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.5]): 0.270927
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.6]): 0.259778
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.7]): 0.228279
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.8]): 0.17313
    [07/27/2018 03:19:29 INFO 140074042410816] #test_score (algo-1, wQuantileLoss[0.9]): 0.0986483
    [07/27/2018 03:19:29 INFO 140074042410816] #quality_metric: host=algo-1, test RMSE <loss>=0.38677016793
    [07/27/2018 03:19:29 INFO 140074042410816] #quality_metric: host=algo-1, test mean_wQuantileLoss <loss>=0.203314334154
    #metrics {"Metrics": {"totaltime": {"count": 1, "max": 43657.426834106445, "sum": 43657.426834106445, "min": 43657.426834106445}, "setuptime": {"count": 1, "max": 10.553836822509766, "sum": 10.553836822509766, "min": 10.553836822509766}}, "EndTime": 1532661569.716042, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1532661569.080218}

    /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sagemaker/session.py:782: DeprecationWarning: generator 'multi_stream_iter' raised StopIteration
      for idx, event in sagemaker.logs.multi_stream_iter(client, log_group, stream_names, positions):

    ===== Job Complete =====
    Billable seconds: 168
    CPU times: user 468 ms, sys: 48 ms, total: 516 ms
    Wall time: 4min 42s

Since you pass a test set in this example, accuracy metrics for the forecast are computed and logged (see bottom of the log).
You can find the definition of these metrics from [our documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html). You can use these to optimize the parameters and tune your model or use SageMaker's [Automated Model Tuning service](https://aws.amazon.com/blogs/aws/sagemaker-automatic-model-tuning/) to tune the model for you.

### Create endpoint and predictor

Now that we have a trained model, we can use it to perform predictions by deploying it to an endpoint.

**Note: Remember to delete the endpoint after running this experiment. A cell at the very bottom of this notebook will do that: make sure you run it at the end.**

To query the endpoint and perform predictions, we can define the following utility class: this allows making requests using `pandas.Series` objects rather than raw JSON strings.

```python
class DeepARPredictor(sagemaker.predictor.RealTimePredictor):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, content_type=sagemaker.content_types.CONTENT_TYPE_JSON, **kwargs)

    def predict(self, ts, cat=None, dynamic_feat=None,
                num_samples=100, return_samples=False, quantiles=["0.1", "0.5", "0.9"]):
        """Requests the prediction of for the time series listed in `ts`, each with the (optional)
        corresponding category listed in `cat`.

        ts -- `pandas.Series` object, the time series to predict
        cat -- integer, the group associated to the time series (default: None)
        num_samples -- integer, number of samples to compute at prediction time (default: 100)
        return_samples -- boolean indicating whether to include samples in the response (default: False)
        quantiles -- list of strings specifying the quantiles to compute (default: ["0.1", "0.5", "0.9"])

        Return value: list of `pandas.DataFrame` objects, each containing the predictions
        """
        prediction_time = ts.index[-1] + 1
        quantiles = [str(q) for q in quantiles]
        req = self.__encode_request(ts, cat, dynamic_feat, num_samples, return_samples, quantiles)
        res = super(DeepARPredictor, self).predict(req)
        return self.__decode_response(res, ts.index.freq, prediction_time, return_samples)

    def __encode_request(self, ts, cat, dynamic_feat, num_samples, return_samples, quantiles):
        instance = series_to_dict(ts, cat if cat is not None else None, dynamic_feat if dynamic_feat else None)

        configuration = {
            "num_samples": num_samples,
            "output_types": ["quantiles", "samples"] if return_samples else ["quantiles"],
            "quantiles": quantiles
        }

        http_request_data = {
            "instances": [instance],
            "configuration": configuration
        }

        return json.dumps(http_request_data).encode('utf-8')

    def __decode_response(self, response, freq, prediction_time, return_samples):
        # we only sent one time series so we only receive one in return
        # however, if possible one will pass multiple time series as predictions will then be faster
        predictions = json.loads(response.decode('utf-8'))['predictions'][0]
        prediction_length = len(next(iter(predictions['quantiles'].values())))
        prediction_index = pd.DatetimeIndex(start=prediction_time, freq=freq, periods=prediction_length)
        if return_samples:
            dict_of_samples = {'sample_' + str(i): s for i, s in enumerate(predictions['samples'])}
        else:
            dict_of_samples = {}
        return pd.DataFrame(data={**predictions['quantiles'], **dict_of_samples}, index=prediction_index)

    def set_frequency(self, freq):
        self.freq = freq

def encode_target(ts):
    return [x if np.isfinite(x) else "NaN" for x in ts]

def series_to_dict(ts, cat=None, dynamic_feat=None):
    """Given a pandas.Series object, returns a dictionary encoding the time series.

    ts -- a pands.Series object with the target time series
    cat -- an integer indicating the time series category

    Return value: a dictionary
    """
    obj = {"start": str(ts.index[0]), "target": encode_target(ts)}
    if cat is not None:
        obj["cat"] = cat
    if dynamic_feat is not None:
        obj["dynamic_feat"] = dynamic_feat
    return obj
```

Now we can deploy the model and create and endpoint that can be queried using our custom DeepARPredictor class.

```python
predictor = estimator.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge',
    predictor_cls=DeepARPredictor)
```

    INFO:sagemaker:Creating model with name: forecasting-deepar-2018-07-27-03-22-49-329
    INFO:sagemaker:Creating endpoint with name deepar-home-electricity-demo-2018-07-27-03-15-18-550

    --------------------------------------------------------------!

### Make predictions and plot results

Now we can use the `predictor` object to generate predictions.

```python
predictor.predict(timeseries[:-prediction_length], quantiles=[0.10, 0.5, 0.90])
```

      0.1
      0.5
      0.9

      2010-09-28
      0.790016
      0.968579
      1.102380

      2010-09-29
      0.744394
      0.925553
      1.131745

      2010-09-30
      0.780791
      0.999924
      1.199642

      2010-10-01
      0.853651
      1.013829
      1.180801

      2010-10-02
      0.792879
      0.994975
      1.229360

      2010-10-03
      0.775380
      0.958632
      1.198079

      2010-10-04
      0.762203
      0.960841
      1.154595

      2010-10-05
      0.770622
      1.005522
      1.173487

      2010-10-06
      0.773807
      0.982053
      1.176765

      2010-10-07
      0.899876
      1.072179
      1.279244

      2010-10-08
      0.815179
      1.030676
      1.242005

      2010-10-09
      0.846799
      1.075428
      1.238595

      2010-10-10
      0.825716
      1.065997
      1.253097

      2010-10-11
      0.884659
      1.077818
      1.245890

      2010-10-12
      0.920730
      1.073006
      1.231640

      2010-10-13
      0.901258
      1.109938
      1.298712

      2010-10-14
      0.921772
      1.064525
      1.289613

      2010-10-15
      0.920833
      1.111202
      1.273061

      2010-10-16
      1.029230
      1.188139
      1.344286

      2010-10-17
      0.911581
      1.111804
      1.263128

      2010-10-18
      0.900885
      1.073449
      1.198113

      2010-10-19
      0.981259
      1.093404
      1.269331

      2010-10-20
      0.936951
      1.104434
      1.282887

      2010-10-21
      0.936540
      1.088976
      1.319242

      2010-10-22
      0.942080
      1.122903
      1.362140

      2010-10-23
      0.932080
      1.119267
      1.325692

      2010-10-24
      0.895720
      1.140803
      1.406739

      2010-10-25
      0.897375
      1.112538
      1.328852

      2010-10-26
      0.924637
      1.168070
      1.364797

      2010-10-27
      0.957276
      1.175334
      1.382472

      2010-10-28
      0.922212
      1.157789
      1.396143

      2010-10-29
      0.967077
      1.199347
      1.414492

      2010-10-30
      0.830030
      1.195158
      1.476995

      2010-10-31
      0.951630
      1.270762
      1.669064

      2010-11-01
      0.809570
      1.112877
      1.352301

      2010-11-02
      1.001853
      1.168526
      1.374803

      2010-11-03
      0.885834
      1.084763
      1.243589

      2010-11-04
      0.782202
      1.071941
      1.283330

      2010-11-05
      0.933213
      1.149291
      1.367097

      2010-11-06
      0.951722
      1.190081
      1.398906

      2010-11-07
      0.967498
      1.161788
      1.393864

      2010-11-08
      0.915501
      1.155778
      1.372192

      2010-11-09
      0.971213
      1.198470
      1.409445

      2010-11-10
      0.904380
      1.075270
      1.346587

      2010-11-11
      0.962840
      1.147342
      1.325590

      2010-11-12
      1.091086
      1.283538
      1.502536

      2010-11-13
      0.985451
      1.207216
      1.435151

      2010-11-14
      0.873823
      1.155347
      1.407026

      2010-11-15
      0.907594
      1.095161
      1.333333

      2010-11-16
      1.021516
      1.178446
      1.405428

      2010-11-17
      1.005996
      1.200050
      1.414176

      2010-11-18
      0.972625
      1.208396
      1.411030

      2010-11-19
      1.036822
      1.217191
      1.393905

      2010-11-20
      1.032539
      1.240849
      1.465980

      2010-11-21
      1.046555
      1.229090
      1.448083

      2010-11-22
      1.013634
      1.188815
      1.428163

      2010-11-23
      0.997702
      1.177898
      1.392498

      2010-11-24
      0.859340
      1.073107
      1.274164

      2010-11-25
      0.938809
      1.146115
      1.351901

      2010-11-26
      0.996817
      1.178803
      1.396186

Below we define a plotting function that queries the model and displays the forecast.

```python
def plot(
    predictor,
    target_ts,
    cat=None,
    dynamic_feat=None,
    forecast_date=end_training,
    show_samples=False,
    plot_history=7 * 12,
    confidence=80
):
    print("calling served model to generate predictions starting from {}".format(str(forecast_date)))
    assert(confidence > 50 and confidence < 100)
    low_quantile = 0.5 - confidence * 0.005
    up_quantile = confidence * 0.005 + 0.5

    # we first construct the argument to call our model
    args = {
        "ts": target_ts[:forecast_date],
        "return_samples": show_samples,
        "quantiles": [low_quantile, 0.5, up_quantile],
        "num_samples": 100
    }

    if dynamic_feat is not None:
        args["dynamic_feat"] = dynamic_feat
        fig = plt.figure(figsize=(20, 6))
        ax = plt.subplot(2, 1, 1)
    else:
        fig = plt.figure(figsize=(20, 3))
        ax = plt.subplot(1,1,1)

    if cat is not None:
        args["cat"] = cat
        ax.text(0.9, 0.9, 'cat = {}'.format(cat), transform=ax.transAxes)

    # call the end point to get the prediction
    prediction = predictor.predict(**args)

    # plot the samples
    if show_samples:
        for key in prediction.keys():
            if "sample" in key:
                prediction[key].plot(color='lightskyblue', alpha=0.2, label='_nolegend_')

    # plot the target
    target_section = target_ts[forecast_date-plot_history:forecast_date+prediction_length]
    target_section.plot(color="black", label='target')

    # plot the confidence interval and the median predicted
    ax.fill_between(
        prediction[str(low_quantile)].index,
        prediction[str(low_quantile)].values,
        prediction[str(up_quantile)].values,
        color="b", alpha=0.3, label='{}% confidence interval'.format(confidence)
    )
    prediction["0.5"].plot(color="b", label='P50')
    ax.legend(loc=2)

    # fix the scale as the samples may change it
    ax.set_ylim(target_section.min() * 0.5, target_section.max() * 1.5)

    if dynamic_feat is not None:
        for i, f in enumerate(dynamic_feat, start=1):
            ax = plt.subplot(len(dynamic_feat) * 2, 1, len(dynamic_feat) + i, sharex=ax)
            feat_ts = pd.Series(
                index=pd.DatetimeIndex(start=target_ts.index[0], freq=target_ts.index.freq, periods=len(f)),
                data=f
            )
            feat_ts[forecast_date-plot_history:forecast_date+prediction_length].plot(ax=ax, color='g')
```

We can interact with the function previously defined, to look at the forecast of any customer at any point in (future) time.

For each request, the predictions are obtained by calling our served model on the fly.

Here we forecast the consumption of an office after week-end (note the lower week-end consumption).
You can select any time series and any forecast date, just click on `Run Interact` to generate the predictions from our served endpoint and see the plot.

```python
style = {'description_width': 'initial'}
```

```python
list_of_df = predictor.predict(timeseries[:-prediction_length])
actual_data = timeseries[-prediction_length:]
```

```python
len(actual_data)
```

    60

```python
list_of_df
```

      0.1
      0.5
      0.9

      2010-09-28
      0.790016
      0.968579
      1.102380

      2010-09-29
      0.744394
      0.925553
      1.131745

      2010-09-30
      0.780791
      0.999924
      1.199642

      2010-10-01
      0.853651
      1.013829
      1.180801

      2010-10-02
      0.792879
      0.994975
      1.229360

      2010-10-03
      0.775380
      0.958632
      1.198079

      2010-10-04
      0.762203
      0.960841
      1.154595

      2010-10-05
      0.770622
      1.005522
      1.173487

      2010-10-06
      0.773807
      0.982053
      1.176765

      2010-10-07
      0.899876
      1.072179
      1.279244

      2010-10-08
      0.815179
      1.030676
      1.242005

      2010-10-09
      0.846799
      1.075428
      1.238595

      2010-10-10
      0.825716
      1.065997
      1.253097

      2010-10-11
      0.884659
      1.077818
      1.245890

      2010-10-12
      0.920730
      1.073006
      1.231640

      2010-10-13
      0.901258
      1.109938
      1.298712

      2010-10-14
      0.921772
      1.064525
      1.289613

      2010-10-15
      0.920833
      1.111202
      1.273061

      2010-10-16
      1.029230
      1.188139
      1.344286

      2010-10-17
      0.911581
      1.111804
      1.263128

      2010-10-18
      0.900885
      1.073449
      1.198113

      2010-10-19
      0.981259
      1.093404
      1.269331

      2010-10-20
      0.936951
      1.104434
      1.282887

      2010-10-21
      0.936540
      1.088976
      1.319242

      2010-10-22
      0.942080
      1.122903
      1.362140

      2010-10-23
      0.932080
      1.119267
      1.325692

      2010-10-24
      0.895720
      1.140803
      1.406739

      2010-10-25
      0.897375
      1.112538
      1.328852

      2010-10-26
      0.924637
      1.168070
      1.364797

      2010-10-27
      0.957276
      1.175334
      1.382472

      2010-10-28
      0.922212
      1.157789
      1.396143

      2010-10-29
      0.967077
      1.199347
      1.414492

      2010-10-30
      0.830030
      1.195158
      1.476995

      2010-10-31
      0.951630
      1.270762
      1.669064

      2010-11-01
      0.809570
      1.112877
      1.352301

      2010-11-02
      1.001853
      1.168526
      1.374803

      2010-11-03
      0.885834
      1.084763
      1.243589

      2010-11-04
      0.782202
      1.071941
      1.283330

      2010-11-05
      0.933213
      1.149291
      1.367097

      2010-11-06
      0.951722
      1.190081
      1.398906

      2010-11-07
      0.967498
      1.161788
      1.393864

      2010-11-08
      0.915501
      1.155778
      1.372192

      2010-11-09
      0.971213
      1.198470
      1.409445

      2010-11-10
      0.904380
      1.075270
      1.346587

      2010-11-11
      0.962840
      1.147342
      1.325590

      2010-11-12
      1.091086
      1.283538
      1.502536

      2010-11-13
      0.985451
      1.207216
      1.435151

      2010-11-14
      0.873823
      1.155347
      1.407026

      2010-11-15
      0.907594
      1.095161
      1.333333

      2010-11-16
      1.021516
      1.178446
      1.405428

      2010-11-17
      1.005996
      1.200050
      1.414176

      2010-11-18
      0.972625
      1.208396
      1.411030

      2010-11-19
      1.036822
      1.217191
      1.393905

      2010-11-20
      1.032539
      1.240849
      1.465980

      2010-11-21
      1.046555
      1.229090
      1.448083

      2010-11-22
      1.013634
      1.188815
      1.428163

      2010-11-23
      0.997702
      1.177898
      1.392498

      2010-11-24
      0.859340
      1.073107
      1.274164

      2010-11-25
      0.938809
      1.146115
      1.351901

      2010-11-26
      0.996817
      1.178803
      1.396186

```python
timeseries.tail()
```

    Date_Time
    2010-11-22    1.417733
    2010-11-23    1.095511
    2010-11-24    1.247394
    2010-11-25    0.993864
    2010-11-26    1.178230
    Freq: D, Name: Global_active_power, dtype: float64

```python
plt.figure(figsize=(20,6))
timeseries[800:-len(list_of_df)].plot(color='#C3C8C4', linewidth=1.0)
p10 = list_of_df['0.1']
p90 = list_of_df['0.9']
plt.fill_between(p10.index, p10, p90, color='#C5F7AB', alpha=0.5, label='80% confidence interval')
actual_data.plot(color='#FCE08F', label='target')
list_of_df['0.5'].plot(marker='^', linewidth=3.0, label='prediction median')
plt.legend()
plt.show()
```

# Additional features

DeepAR has additional features such as handling the missing values as below:

* missing values: DeepAR can handle missing values in the time series during training as well as for inference.
* Additional time features: DeepAR provides a set default time series features such as hour of day etc. However, you can provide additional feature time series via the `dynamic_feat` field.
* generalize frequencies: any integer multiple of the previously supported base frequencies (minutes `min`, hours `H`, days `D`, weeks `W`, month `M`) are now allowed; e.g., `15min`. We already demonstrated this above by using `2H` frequency.
* categories: If your time series belong to different groups (e.g. types of product, regions, etc), this information can be encoded as one or more categorical features using the `cat` field.

Based on the previous results, we will be able to impelement the advanced models to support the above features.

### Delete endpoints

```python
predictor.delete_endpoint()
```