Re:Invent 2019 | IoT-309

Combining IoT and Machine Learning for Predictive Maintenance

A multivariate LSTM neural network for prediction future pollution levels to efficiently operate air filters

# About

Predicting failure, remaining useful life or operating conditions is a classic request of IoT systems. By predicting which devices will fail, proactive maintenance can be scheduled to increase device uptime, optimize asset utilization, avoid costly catastrophic device failure and optimize field service efficiency. In this Notebook template, we will show how to implement a multivariate LSTM algorithm to predict pollution levels using the [Beijing PM2.5 data set](https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data).

This Jupyter notebook is part of a larger solution intended to be deployed to your AWS account. Run through this notebook after air pollution data has been published into your account and stored in IoT Analytics. After finishing all steps in this notebook, you will move on to the next stage of the solution which is deploying your LSTM model to an edge solution with IoT Greengrass.

Architecture diagram

# Set-up: import required notebook libraries

This notebook requires a few basic Python libraries including `pandas`, `numpy`, `keras`, `tensorflow`, `scikit-learn` and `matplotlib`.

```python
from pandas import read_csv
from datetime import datetime
from math import sqrt
from numpy import concatenate
from matplotlib import pyplot
from pandas import to_datetime
from pandas import DataFrame
from pandas import concat
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from joblib import dump, load
import numpy
import sagemaker
import pickle
import warnings
import tensorflow
from sagemaker.tensorflow import TensorFlow
import json
from sagemaker import get_execution_role
from urllib.parse import urlparse
import os
import boto3
import tarfile
from keras.models import model_from_json
from IPython.display import display, Markdown
import traceback

###Turn off warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
if type(tensorflow.contrib) != type(tensorflow): tensorflow.contrib._warning = None
```

    Using TensorFlow backend.

# Background: problem description and approach

In order to support business decision making, we must be able to take action on predictive analytics. For predictive operational analysis, we want to know what would be the operating condition of equipment to optimize the parameters for increasing the life of that equipment. To solve this for real-world equipment that has multiple operation modes and reports time-series measurements from multiple sensors, our primary approach is an Long Short-Term Memory neural network in TensorFlow. LSTM provides multivariate time series forecasting to predict the future pollution levels which could be used to automatically alter the parameters of the air filter to improve efficiency and increase remaining useful life (RUL) of air filters.

In the [CRISP-DM lifecycle](https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining), we have already completed the Business Understanding phase in knowing our objective is to increase RUL of our air filtration equipment and that we are going to try using pollution forecasting to drive improvements. The next phase, which starts in this notebook, is Data Understanding. We need to evaluate the available data, look for trends, and assess whether this data will be useful to our objective.

BEST PRACTICES NOTE Different modeling approaches provide different business trade-offs, and different teams may opt for different approaches. For example, although the field service team is interested in the most precise prediction for any given filter in order to eliminate false positives and unnecessary truck rolls, a supervisor is interested in all possible devices that aren't operating at full efficiency to increase operational equipment effectiveness (OEE).

Step 1  |  Loading data

# Data set description

The Beijing PM2.5 is an hourly data set that contains the PM2.5 data of US Embassy in Beijing along with meteorological data from Beijing Capital International Airport. The data is collected for time period starting from Jan 1st, 2010 to Dec 31st, 2014.

The data contain features like PM2.5 concentration (pollution level), dew point, temperature, pressure, wind direction, wind speed, cumulated hours of snow and cumulated hours of rain.

Data Set Attribution:
 Song Xi Chen, Guanghua School of Management, Center for Statistical Science, Peking University

The data can be downloaded from this page: https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data

The data has been split into training, validation and test data sets. The first 2 years of data is used for training data, followed by the next 2 year of data as validation to check the training accuracy. The last 1 year of data will later be used for device simulation testing.

To load data already ingested from your device to the cloud, we need to set up an IoT Analytics SDK client to access your data set. This code initializes the client, fetches content from your data set, sorts by time, and previews the first five rows.

```python
s3Bucket = "XYZ-S3Bucket-XYZ"
dataset = "XYZ-IotAnalyticsDataset-XYZ"
use_local_data = True

if use_local_data:
    dataset_url = "source_pollution.csv"
else:
    iotanalytics_client = boto3.client('iotanalytics')
    dataset_url = iotanalytics_client.get_dataset_content(datasetName = dataset)['entries'][0]['dataURI']

dataset = read_csv(dataset_url, header=0, index_col='date')
if dataset.empty:
    raise Exception('No data found')
dataset.sort_index(inplace=True)
dataset = dataset[['pollution', 'dew', 'temp', 'press', 'wnd_dir', 'wnd_spd', 'snow', 'rain']]
dataset.drop_duplicates(inplace=True)
dataset.to_csv('pollution.csv')

# displays the first few entries of the DataFrame
dataset.head()
```

      pollution
      dew
      temp
      press
      wnd_dir
      wnd_spd
      snow
      rain

      date

      2010-01-02 00:00:00
      129.0
      -16
      -4.0
      1020.0
      2
      1.79
      0
      0

      2010-01-02 01:00:00
      148.0
      -15
      -4.0
      1020.0
      2
      2.68
      0
      0

      2010-01-02 02:00:00
      159.0
      -11
      -5.0
      1021.0
      2
      3.57
      0
      0

      2010-01-02 03:00:00
      181.0
      -7
      -5.0
      1022.0
      2
      5.36
      1
      0

      2010-01-02 04:00:00
      138.0
      -7
      -5.0
      1022.0
      2
      6.25
      2
      0

Start a new session in SageMaker and make the training data available in S3.

```python
###Upload this queried data to s3
sagemaker_session = sagemaker.Session()
uploaded_data_path = sagemaker_session.upload_data(path='pollution.csv', bucket=s3Bucket, key_prefix='data')
```

Plot the columns to understand statistical properties of the data set.

```python
values = dataset.values
# specify columns to plot
columns_to_plot = [0, 1, 2, 3, 5, 6, 7]
for column_index in columns_to_plot:
	pyplot.figure()
	pyplot.subplot(1, 1, 1)
	pyplot.plot(values[:, column_index])
	pyplot.title(dataset.columns[column_index])

```

We can visually observe some seasonal values in dew, temp, pressure and snow attributes in the above plots. However, we don't observe any particular statistical features in the pollution readings. We need to further analyze the features of our pollution readings.

We are using a library `statsmodels` to see if there is any seasonality in the pollution readings. Seasonality or periodicity in data is often useful in picking the suitable ML algorithm.

Let's first observe the first month of pollution data.

```python
###Exploring short term seasonality in data
import statsmodels.api as sm
trend_dataset = dataset.reset_index()
trend_dataset = trend_dataset[['date', 'pollution']]
trend_dataset['date'] = to_datetime(trend_dataset['date'])
trend_dataset = trend_dataset.set_index('date')

pyplot.rcParams["figure.figsize"] = (20,10)
number_of_monthly_datapoints = 24*30
decomposition = sm.tsa.seasonal_decompose(trend_dataset[:number_of_monthly_datapoints], model='additive')
fig = decomposition.plot()
```

If we observe the seasonal plot above, we can find that our pollution dataset has a daily seasonality in it.

Let's try to look for long term seasonality by forcing the model to ignore short term seasonality we found above. By setting the `freq` to a larger number. We are setting it here to look for monthly seasonality in the first year of pollution data.

```python
###Exploring long seasonality in data
number_of_yearly_datapoints = 24*365
decomposition = sm.tsa.seasonal_decompose(trend_dataset[:number_of_yearly_datapoints], model='additive', freq=number_of_monthly_datapoints)
fig = decomposition.plot()
```

If we observe the seasonal plot above, we can find that our pollution dataset also has a monthly seasonality in it (observe the 12 similar segments in the 1 year of data).

We can observe that pollution data has both long (monthly) and short (daily) term periodicity.

Step 2  |  Processing data

# Sensor measurement normalization

The next step of the CRISP-DM lifecycle is Data Preparation. These next steps are about manipulating the desired data into a format best fit for machine learning.

Multivariate problems require normalization of sensor measurements and operational settings to remove the effects of different scales being used across the units of measure. For example, wind speed versus cumulated hours of snow/rain operate in different orders of magnitude. The 'MinMaxScaler' transforms each value into a given range, in our case between 0 and 1.

Below is the transformation we are performing on each sensor measurement. [Learn more](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) about preprocessing with **sklearn**.

\begin{align}
X_{std} & = (X - X_{min})/(X_{max} - X_{min}) \\
X_{scaled} & = X_{std} * (feature\_range_{max} - feature\_range_{min}) + feature\_range_{min}
\end{align}

```python
#Transform all columns as float
values = values.astype('float32')

#Normalize features
scaler = MinMaxScaler(feature_range=(0, 1))
normalized_values = scaler.fit_transform(values)
```

# Target parameter determination

In order to model a LSTM, we need a target parameter that puts all the data on the same basis for supervised training and the subsequent classification test. For time-series forecasting of pollution levels, we will assume attribute data from the previous timestamp being associated with pollution levels of the current timestamp.

To mark these observations into supervised learning, we first need to transform the data set where the all previous timestamp's attributes (including the pollution level) are the variables and current pollution level is the target value.

BEST PRACTICES NOTE In this example, only the last timestamp's attributes are used for training and prediction. A deeper lookback into the history may increase model accuracy while increasing the complexity and training time.

```python
#Convert series to supervised learning
def transform_to_supervised_series(data, columns, n_in=1, n_out=1, dropnan=True):
	n_vars = 1 if type(data) is list else data.shape[1]
	df = DataFrame(data)
	cols, names = list(), list()
	# input sequence (t-n, ... t-1)
	for i in range(n_in, 0, -1):
		cols.append(df.shift(i))
		names += [('%s(t-%d)' % (columns[j], i)) for j in range(n_vars)]
	# forecast sequence (t, t+1, ... t+n)
	for i in range(0, n_out):
		cols.append(df.shift(-i))
		if i == 0:
			names += [('%s(t)' % (columns[j])) for j in range(n_vars)]
		else:
			names += [('%s(t+%d)' % (columns[j], i)) for j in range(n_vars)]
	# put it all together
	agg = concat(cols, axis=1)
	agg.columns = names
	# drop rows with NaN values
	if dropnan:
		agg.dropna(inplace=True)
	return agg

#Frame as supervised learning
supervised_series_dataset = transform_to_supervised_series(normalized_values, dataset.columns, 1, 1)
#Drop the unnecessary columns
supervised_series_dataset.drop(['dew(t)', 'temp(t)', 'press(t)', 'wnd_dir(t)', 'wnd_spd(t)', 'snow(t)', 'rain(t)'], axis=1, inplace=True)
```

The data has been prepared for supervised learning as described in the previous cell. Let's take a look at the first five rows.

```python
supervised_series_dataset.head()
```

      pollution(t-1)
      dew(t-1)
      temp(t-1)
      press(t-1)
      wnd_dir(t-1)
      wnd_spd(t-1)
      snow(t-1)
      rain(t-1)
      pollution(t)

      1
      0.129779
      0.352941
      0.245902
      0.527273
      0.666667
      0.002290
      0.000000
      0.0
      0.148893

      2
      0.148893
      0.367647
      0.245902
      0.527273
      0.666667
      0.003811
      0.000000
      0.0
      0.159960

      3
      0.159960
      0.426471
      0.229508
      0.545454
      0.666667
      0.005332
      0.000000
      0.0
      0.182093

      4
      0.182093
      0.485294
      0.229508
      0.563637
      0.666667
      0.008391
      0.037037
      0.0
      0.138833

      5
      0.138833
      0.485294
      0.229508
      0.563637
      0.666667
      0.009912
      0.074074
      0.0
      0.109658

# Split the data into training, validation and test data sets

We will now split the data into training, validation and test data sets. The first 2 years of data is used for training data, followed by the next 2 years of data as validation to check the training accuracy. The last 1 year of data has been used for device simulation testing.

```python
######Splitting data into training and test data
values = supervised_series_dataset.values
n_train_hours = 24*365*2
n_validation_hours = 24*365*4
train = values[:n_train_hours, :]
validation = values[n_train_hours:n_validation_hours, :]
```

# Split the training and validation data sets into input and output data sets

LSTM model takes input in a 3D format where the 1st axis are the samples, 2nd axis is the timesteps that you want to take and the last axis is the features. We would need to transform out input data into this format.

```python
#Split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
validation_X, validation_y = validation[:, :-1], validation[:, -1]
#Reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
validation_X = validation_X.reshape((validation_X.shape[0], 1, validation_X.shape[1]))
```

Step 3  |  Train and test model

# Time-series forecasting via LSTM

LSTM or 'Long Short-Term Memory' is a recurrent neural network that is particularly forgiving of unknown time gaps between significant events in time-series data, such as the unknown periodicity in the pollution levels. LSTM is capable of handling large data sets. This makes it very useful for prediction with time-series IoT data.

Keras LSTM layers expect an input in the following shape: (batch_size, timesteps, input_dim)

batch_size - number of sequences

timesteps - the sequence length to look back, look back cycles in our example

input_dim - number of features of each sequence at each time step

HUMAN LEARNING NOTE Keras is a neural network (NN) framework that makes running NN fast and easy on TensorFlow. Alternatives to Keras include Theano and CNTK.

# LSTM model setup
Now that we have the data structured for training we can set the parameters in our LSTM model. Here we are using a sequential LSTM model for time-series forecasting. This model configuration choice makes our model capable of learning higher-level temporal representations. We could see in the earlier data visualization that our data has long-term temporal periodicity.

`Epoch` is a hyperparameter in ML model to control the number of complete passes through the training dataset. Larger epoch number usually allows the learning algorithm to run until the error from the model has been sufficiently minimized.

`Batch Size` is another hyperparameter in ML model that defines the number of samples to work through before updating the internal model parameters.

`Mean Absolute Error (MAE)` is one of the most commonly used regression loss function. MAE is the sum of absolute differences between our target and predicted variables. So, it measures the average magnitude of errors in a set of predictions, without considering their directions. The range is also 0 to ∞

`Adam optimizer` is an adaptive learning rate optimization algorithm that’s been designed specifically for training deep neural networks. The algorithms leverage the power of adaptive learning rates methods to find individual learning rates for each parameter. It also has advantages of Adagrad, which works really well in settings with sparse gradients, but struggles in non-convex optimization of neural networks, and RMSprop, which tackles to resolve some of the problems of Adagrad and works really well in on-line settings. Its name is derived from adaptive moment estimation and uses estimations of first and second moments of gradient to adapt the learning rate for each weight of the neural network.

HUMAN LEARNING NOTE You can read about different supported optimizers and losses on Keras `optimizers` and `losses`.

```python
#LSTM for time-series predictions
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

print(model.summary())
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    lstm_2 (LSTM)                (None, 50)                11800
    _________________________________________________________________
    dense_2 (Dense)              (None, 1)                 51
    =================================================================
    Total params: 11,851
    Trainable params: 11,851
    Non-trainable params: 0
    _________________________________________________________________
    None

We are using an LSTM with 50 neurons in the first hidden layer and 1 neuron in the output layer for predicting pollution. The input shape will be 1 time step with 8 features. We reached to this model after tuning these parameters based on some experimentation, you can try to play around with these to see if you can tune them better.

# Model training

The next step of the CRISP-DM lifecycle is Modeling, where we start the process of training and testing models.

Since LSTM is a recurrent neural network we can observe the incremental training at each iteration or epoch on the training data set.

`loss` is the value of cost function on our training data.
`val_loss` is the value of cost function for your cross-validation data.

These values are useful in determining if our model is overfitting to training data. If we have `loss` noticeably lower than `val_loss` then it is the sign of overfitting.

PRODUCTION NOTE we have chosen to train over 50 epochs. This allows us to test the model within a reasonable timeframe (~15 mins).

```python
### fit network
#history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(validation_X, validation_y), verbose=2, shuffle=False)
#history = history.history

###Instead of training here, we are calling it through a training job, since sagemaker will pack the model for us and
###upload it on to s3 to make it easier for us use

###We are performing a m5.xlarge instance for training , you can do local training if you want to.
###You can change the train_instance_type to a different p type sagemaker instance for faster training.
###Note: Local training doesn't result into a training job that can be used for green grass directly
use_local_training = False

if use_local_training:
    train_instance_type = "local"
else:
    train_instance_type = "ml.m5.xlarge"

tf_estimator = TensorFlow(entry_point='trainLSTM.py', role=get_execution_role(),
                          train_instance_count=1, train_instance_type=train_instance_type,
                          framework_version='1.12', py_version='py3', script_mode=True,
                          output_path = 's3://' + s3Bucket, base_job_name = "pollution-forecasting-lstm",
                          hyperparameters={'batch_size': 72,
                                           'epochs': 50,
                                           'n_train_hours': n_train_hours,
                                           'n_validation_hours': n_validation_hours})

tf_estimator.fit(uploaded_data_path)
```

    2019-11-26 19:49:34 Starting - Starting the training job...
    2019-11-26 19:49:35 Starting - Launching requested ML instances......
    2019-11-26 19:50:57 Starting - Preparing the instances for training......
    2019-11-26 19:51:58 Downloading - Downloading input data
    2019-11-26 19:51:58 Training - Downloading the training image..2019-11-26 19:52:12,958 sagemaker-containers INFO     Imported framework sagemaker_tensorflow_container.training
    2019-11-26 19:52:12,963 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)
    2019-11-26 19:52:13,229 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)
    2019-11-26 19:52:13,243 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)
    2019-11-26 19:52:13,253 sagemaker-containers INFO     Invoking user script

    Training Env:

    {
        "additional_framework_parameters": {},
        "channel_input_dirs": {
            "training": "/opt/ml/input/data/training"
        },
        "current_host": "algo-1",
        "framework_module": "sagemaker_tensorflow_container.training:main",
        "hosts": [
            "algo-1"
        ],
        "hyperparameters": {
            "batch_size": 72,
            "n_train_hours": 17520,
            "model_dir": "s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model",
            "epochs": 50,
            "n_validation_hours": 35040
        },
        "input_config_dir": "/opt/ml/input/config",
        "input_data_config": {
            "training": {
                "TrainingInputMode": "File",
                "S3DistributionType": "FullyReplicated",
                "RecordWrapperType": "None"
            }
        },
        "input_dir": "/opt/ml/input",
        "is_master": true,
        "job_name": "pollution-forecasting-lstm-2019-11-26-19-49-33-760",
        "log_level": 20,
        "master_hostname": "algo-1",
        "model_dir": "/opt/ml/model",
        "module_dir": "s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/source/sourcedir.tar.gz",
        "module_name": "trainLSTM",
        "network_interface_name": "eth0",
        "num_cpus": 4,
        "num_gpus": 0,
        "output_data_dir": "/opt/ml/output/data",
        "output_dir": "/opt/ml/output",
        "output_intermediate_dir": "/opt/ml/output/intermediate",
        "resource_config": {
            "current_host": "algo-1",
            "hosts": [
                "algo-1"
            ],
            "network_interface_name": "eth0"
        },
        "user_entry_point": "trainLSTM.py"
    }

    Environment variables:

    SM_HOSTS=["algo-1"]
    SM_NETWORK_INTERFACE_NAME=eth0
    SM_HPS={"batch_size":72,"epochs":50,"model_dir":"s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model","n_train_hours":17520,"n_validation_hours":35040}
    SM_USER_ENTRY_POINT=trainLSTM.py
    SM_FRAMEWORK_PARAMS={}
    SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}
    SM_INPUT_DATA_CONFIG={"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
    SM_OUTPUT_DATA_DIR=/opt/ml/output/data
    SM_CHANNELS=["training"]
    SM_CURRENT_HOST=algo-1
    SM_MODULE_NAME=trainLSTM
    SM_LOG_LEVEL=20
    SM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main
    SM_INPUT_DIR=/opt/ml/input
    SM_INPUT_CONFIG_DIR=/opt/ml/input/config
    SM_OUTPUT_DIR=/opt/ml/output
    SM_NUM_CPUS=4
    SM_NUM_GPUS=0
    SM_MODEL_DIR=/opt/ml/model
    SM_MODULE_DIR=s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/source/sourcedir.tar.gz
    SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"batch_size":72,"epochs":50,"model_dir":"s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model","n_train_hours":17520,"n_validation_hours":35040},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"pollution-forecasting-lstm-2019-11-26-19-49-33-760","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/source/sourcedir.tar.gz","module_name":"trainLSTM","network_interface_name":"eth0","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"trainLSTM.py"}
    SM_USER_ARGS=["--batch_size","72","--epochs","50","--model_dir","s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model","--n_train_hours","17520","--n_validation_hours","35040"]
    SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
    SM_CHANNEL_TRAINING=/opt/ml/input/data/training
    SM_HP_BATCH_SIZE=72
    SM_HP_N_TRAIN_HOURS=17520
    SM_HP_MODEL_DIR=s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model
    SM_HP_EPOCHS=50
    SM_HP_N_VALIDATION_HOURS=35040
    PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages

    Invoking script with the following command:

    /usr/bin/python trainLSTM.py --batch_size 72 --epochs 50 --model_dir s3://iot309demo-sagemaker-232skq6dqo4r/pollution-forecasting-lstm-2019-11-26-19-49-33-760/model --n_train_hours 17520 --n_validation_hours 35040

    Using TensorFlow backend.
    Train on 17520 samples, validate on 17520 samples
    Epoch 1/50
     - 3s - loss: 0.0480 - val_loss: 0.0474
    Epoch 2/50

    2019-11-26 19:52:10 Training - Training image download completed. Training in progress. - 1s - loss: 0.0188 - val_loss: 0.0272
    Epoch 3/50
     - 1s - loss: 0.0150 - val_loss: 0.0196
    Epoch 4/50
     - 1s - loss: 0.0146 - val_loss: 0.0173
    Epoch 5/50
     - 1s - loss: 0.0144 - val_loss: 0.0166
    Epoch 6/50
     - 1s - loss: 0.0144 - val_loss: 0.0157
    Epoch 7/50
     - 1s - loss: 0.0144 - val_loss: 0.0158
    Epoch 8/50
     - 1s - loss: 0.0143 - val_loss: 0.0162
    Epoch 9/50
     - 1s - loss: 0.0143 - val_loss: 0.0161
    Epoch 10/50
     - 1s - loss: 0.0143 - val_loss: 0.0163
    Epoch 11/50
     - 1s - loss: 0.0143 - val_loss: 0.0158
    Epoch 12/50
     - 1s - loss: 0.0142 - val_loss: 0.0156
    Epoch 13/50
     - 1s - loss: 0.0143 - val_loss: 0.0160
    Epoch 14/50
     - 1s - loss: 0.0142 - val_loss: 0.0160
    Epoch 15/50
     - 1s - loss: 0.0143 - val_loss: 0.0153
    Epoch 16/50
     - 1s - loss: 0.0143 - val_loss: 0.0151
    Epoch 17/50
     - 1s - loss: 0.0141 - val_loss: 0.0157
    Epoch 18/50
     - 1s - loss: 0.0142 - val_loss: 0.0157
    Epoch 19/50
     - 1s - loss: 0.0142 - val_loss: 0.0154
    Epoch 20/50
     - 1s - loss: 0.0142 - val_loss: 0.0154
    Epoch 21/50
     - 1s - loss: 0.0142 - val_loss: 0.0158
    Epoch 22/50
     - 1s - loss: 0.0142 - val_loss: 0.0154
    Epoch 23/50
     - 1s - loss: 0.0142 - val_loss: 0.0146
    Epoch 24/50
     - 1s - loss: 0.0142 - val_loss: 0.0153
    Epoch 25/50
     - 1s - loss: 0.0142 - val_loss: 0.0149
    Epoch 26/50
     - 1s - loss: 0.0142 - val_loss: 0.0148
    Epoch 27/50
     - 1s - loss: 0.0142 - val_loss: 0.0149
    Epoch 28/50
     - 1s - loss: 0.0142 - val_loss: 0.0151
    Epoch 29/50
     - 1s - loss: 0.0141 - val_loss: 0.0147
    Epoch 30/50
     - 1s - loss: 0.0141 - val_loss: 0.0148
    Epoch 31/50
     - 1s - loss: 0.0141 - val_loss: 0.0152
    Epoch 32/50
     - 1s - loss: 0.0141 - val_loss: 0.0150
    Epoch 33/50
     - 1s - loss: 0.0141 - val_loss: 0.0150
    Epoch 34/50
     - 1s - loss: 0.0141 - val_loss: 0.0152
    Epoch 35/50
     - 1s - loss: 0.0141 - val_loss: 0.0151
    Epoch 36/50
     - 1s - loss: 0.0142 - val_loss: 0.0147
    Epoch 37/50

#### Download the training model and history created using training job
This next code downloads the training model that we have created the through the SageMaker training job.
It copies the trained model in tar.gz-format from S3 to this notebook instance and unpacks it. The model files will be stored in the local file directory **[job\_name]/model**. [job\_name] is a placeholder that is different every time you run this notebook. When copying and unpacking are finished the names of your model files will be printed below the cell.

```python
print("job_name: {}".format(tf_estimator.latest_training_job.name))

s3_model_path = '{}'.format(tf_estimator.model_data)
print("s3_model_path: {}".format(s3_model_path))
s3_bucket = urlparse(s3_model_path).netloc
print(s3_bucket)
s3_key = urlparse(s3_model_path).path.lstrip('/')
print("s3_key: {}".format(s3_key))

local_model_dir = 'model'
if not os.path.exists(local_model_dir):
    try:
        os.makedirs(local_model_dir)
    except Exception as e:
        print("ERROR: failed to created directory {}: {}".format(local_model_dir, e))

if not os.path.exists(local_model_dir + '/model.tar.gz'):
    s3_client = boto3.client('s3')
    s3_client.download_file(s3_bucket, s3_key, local_model_dir + '/model.tar.gz')

tar = tarfile.open(local_model_dir + '/model.tar.gz')
tar.extractall(path=local_model_dir)
tar.close()

model_files = os.listdir(local_model_dir)
print("model_files: {}".format(model_files))
```

### Read the model file and history
The model files that we have downloaded above need to be loaded here in the notebook.

```python
scaler = load('%s/scaler.model' % local_model_dir)

with open('%s/model.json' % local_model_dir, 'r') as json_file:
    loaded_model_json = json_file.read()
loaded_model = model_from_json(loaded_model_json)
loaded_model.load_weights('%s/model.h5' % local_model_dir)

with open('%s/history.json' % local_model_dir) as f:
    history = json.load(f)
loaded_model.compile(loss='mae', optimizer='adam')
```

    /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.21.1 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
      UserWarning)

    WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

    WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:184: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

#### Model accuracy and overfit check
The next step in CRISP-DM is Evaluation. Now that we've trained a model, we must inspect its performance in forecasting pollution of time N against the actual time N. Let's compare the `loss` and `val_loss` of the model to see if we ended overfitting or not.

```python
# plot accuracy as we learn
pyplot.plot(history['loss'], label='loss (Train)')
pyplot.plot(history['val_loss'], label='val_loss (Validation)')
pyplot.legend()
pyplot.show()
```

As you may have noticed, the loss of the train dataset is similar to the val_loss of the validation data. This corresponds that we have not overfitted out model onto training data.

Step 4  |  Results summary

### Apply model to test data

Let's make inferences from the trained model onto our validation dataset

```python
yhat = loaded_model.predict(validation_X)
#If trained within the notebook use yhat = model.predict(validation_X)
```

### Calculate the root mean square error
We can calculate the root mean square error (RMSE) on the validation data set to evaluate the performance of our model. Unlike classification problems, in regressions problems like this one it is hard to calculate the accuracy of a model, so we use RMSE to benchmark the model's performance.

```python
validation_X = validation_X.reshape((validation_X.shape[0], validation_X.shape[2]))
# invert scaling for forecast
inv_yhat = concatenate((yhat, validation_X[:, 1:]), axis=1)
inv_yhat = scaler.inverse_transform(inv_yhat)
inv_yhat = inv_yhat[:,0]
# invert scaling for actual
validation_y = validation_y.reshape((len(validation_y), 1))
inv_y = concatenate((validation_y, validation_X[:, 1:]), axis=1)
inv_y = scaler.inverse_transform(inv_y)
inv_y = inv_y[:,0]

print("\n\nLSTM Metrics")
# calculate RMSE
rmse = sqrt(mean_squared_error(inv_y, inv_yhat))
print('Val RMSE: %.3f' % rmse)
```

    LSTM Metrics
    Val RMSE: 27.931

In this case, our RMSE is approximately 30 (your exact number may vary by a few percent).

### Benchmarking with a Persistence Model
The persistence forecast is where the observation from the prior time step (t-1) is used to predict the observation at the current time step (t).

```python
#Let's create a naive persistence model to benchmark the performance of our model

##Creating a dateframe with the previous timestep values and using that as prediction for next step
persistence_model_df = DataFrame()
persistence_model_df['pollution(t-1)'] = dataset['pollution']
persistence_model_df['predicted_pollution(t)'] = persistence_model_df['pollution(t-1)']
persistence_model_df.head(5)
```

      pollution(t-1)
      predicted_pollution(t)

      date

      2010-01-02 00:00:00
      129.0
      129.0

      2010-01-02 01:00:00
      148.0
      148.0

      2010-01-02 02:00:00
      159.0
      159.0

      2010-01-02 03:00:00
      181.0
      181.0

      2010-01-02 04:00:00
      138.0
      138.0

### Calculate the root mean square error for persistence model for validation data

```python
#Let's extract the actual values
persistence_model_y = persistence_model_df[['pollution(t-1)']].values[1:].astype('float32')
#Extract the predicted values
persistence_model_y_hat = persistence_model_df[['predicted_pollution(t)']].values[:-1].astype('float32')

#Filter out for validation dataset only
persistence_model_y = persistence_model_y[n_train_hours:n_validation_hours, :]
persistence_model_y_hat = persistence_model_y_hat[n_train_hours:n_validation_hours, :]

persistence_model_y = persistence_model_y.reshape((len(persistence_model_y)))
persistence_model_y_hat = persistence_model_y_hat.reshape((len(persistence_model_y_hat)))
persistence_model_rmse = sqrt(mean_squared_error(persistence_model_y, persistence_model_y_hat))
print('Val RMSE for Persistence Model: %.3f' % persistence_model_rmse)
```

    Val RMSE for Persistence Model: 27.436

In this case, our vanilla LSTM model is performing similar to a persistence model. We can experiment with hyperparameter tuning to further enhance the model and reduce its error.

Summary

At this stage you have successfully trained and tested a LSTM model to forecast air pollution! The next stage of the CRISP-DM lifecycle is Deployment. That happens outside the scope of this notebook. Resume the next step of this solution from the associated README file.

The code in this notebook is not tightly correlated to the air pollution data set. You could continue experimenting beyond the scope of this solution by bringing an alternate data set, changing the window of the Beijing 2.5PM used to train, or experimenting with the LSTM hyperparameters to see the effect they have on model accuracy (RMSE).


# ナーススケジューリング問題をD-Wave 2000Qで解く(検証編)

## 概要
記事「[ナーススケジューリング問題をD-Wave 2000Qで解く](https://qard.is.tohoku.ac.jp/T-Wave/?p=1756)」では、ナーススケジューリング問題(NSP)をD-Wave 2000Qで解くことにより、量子アニーリング(QA)の計算性能を評価する論文を紹介しました。結果として、NSPを解くことに関しては、QAよりもシミュレーテッドアニーリング(SA)の方が有用でした。しかし、この論文では制約の係数を人数や日数といった設定毎に変えていないため、係数を調整することで、より良い結果を得られる可能性があります。そこで本記事ではまず、最適だと考えられる係数を探索します。そして、その係数を用いることで、論文の結果よりも基底状態の解を見つける確率が向上するのか検証します。

## 問題

NSPの詳細については、先行研究の解説記事「[ナーススケジューリング問題をD-Wave 2000Qで解く](https://qard.is.tohoku.ac.jp/T-Wave/?p=1756)」を参照してください。

## 定式化

| 定数 | 説明 |
| ---- | ---- |
|  $n = 1 , \cdots, N$  |  看護師の人数  |
|  $d = 1 , \cdots, D$  |  日数  |
|   行列$J_{i(n, d), j\left(n, d^{\prime}\right)}=\left\{\begin{array}{cc} a>0 & d^{\prime}=d \pm 1 \\ 0 & \text { otherwise } \end{array}\right.$ |  看護師$n$が2日以上連続で勤務するとペナルティを課す  |
|  $W(d)$  |  $d$日に必要な労働力  |
|  $E(n)$  |  看護師$n$の持つ労働力  |
|  $F(n)$  |  看護師$n$が希望する勤務日数  |
|  $G(n, d) = h_{1}(n)h_{2}(d)$ |  看護師$n$が$d$日に勤務希望する  |

変数および目的関数は次のようになります。定式化の詳細は、記事「[ナーススケジューリング問題をD-Wave 2000Qで解く](nsp.md)」を参照してください。

**変数:**

$q_{n,d}$:  看護師$n$が$d$日に働く場合に1、働かない場合に0

**目的関数:**

$$
\begin{align}
H(\mathbf{q}) &= \alpha \sum_{n, n^{\prime}}^{N} \sum_{d, d^{\prime}}^{D} J_{i(n, d), j\left(n^{\prime}, d^{\prime}\right)} q_{i(n, d)} q_{j\left(n^{\prime}, d^{\prime}\right)}\\
&+ \lambda \sum_{d}^{D}\left(\sum_{n}^{N} E(n) q_{i(n, d)}-W(d)\right)^{2}\\
&+ \gamma \sum_{n}^{N}\left(\sum_{d}^{D} h_{1}(n) h_{2}(d) q_{i(n, d)}-F(n)\right)^{2}
\end{align}
$$

ただし、後述する実験のために、第一項の係数を$\alpha$としています。

論文の問題設定は次の通りでした。

| 定数 | 説明 |
| ---- | ---- |
| $N = 3, 4$ |  看護師の人数  |
| $D = 5, \cdots, 14$ |  日数  |
| $h_{1}(n) = 1$ |  全員暇  |
| $h_{2}(d) = 1$ |  常に平日  |
| $E(n) = 1$ |  全員が1人分の労働力を持つ  |
| $W(d) = 1$ |  各日に1人分の労働力が必要  |

| 目的関数・D-Waveマシンの設定 | 説明 |
| ---- | ---- |
| $a = 7/2$ |  2日以上勤務すると課されるペナルティ  |
| $\lambda = 0.3, \gamma = 1.3$ |  制約項の係数  |
| 200 $\mu s$ | アニーリング時間 |
| 1000 | サンプル数 |

本記事では、上記の設定で実験を行いました。
以下に、[PyQUBO](https://pyqubo.readthedocs.io/en/latest/)を用いたQUBO生成から最適化までのPythonコードを示します。

まず、必要なライブラリをインストールします。

```python
!pip install numpy
!pip install matplotlib
!pip install dwave-ocean-sdk  # D-Wave Ocean SDKにPyQUBOが含まれています
```

次に初期設定をします。

```python
N = 3  # 看護師の数
D = 14  # 日数
a = 7 / 2  # 2日以上連続で勤務した時のペナルティ
F = [4, 5, 5]  # 各看護師が希望する勤務日数
```

$J_{i(n, d), j\left(n, d^{\prime}\right)}=\left\{\begin{array}{cc} a>0 & d^{\prime}=d \pm 1 \\ 0 & \text { otherwise } \end{array}\right.$ は、次のように生成します。

```python
from itertools import product
import numpy as np

J = np.zeros((N, D, N, D))
for n1, d1, n2, d2 in product(range(N), range(D), range(N), range(D)):
    if n1 == n2 and d1 + 1 == d2:
        J[n1, d1, n2, d2] = a
```

定数の設定が終わったので、定式化していきます。

```python
from itertools import product
from pyqubo import Array, Constraint, Placeholder

# バイナリ変数
q = Array.create('q', shape=(N, D), vartype='BINARY')

# 2日以上連続で勤務するのを避けるための項
H1 = np.sum([J[n1, d1, n2, d2] * q[n1, d1] * q[n2, d2]
             for n1, n2, d1, d2 in product(range(N), range(N), range(D), range(D))])

# 各dに1人の看護師を確保するための項
H2 = np.sum([(np.sum([q[n,d] for n in range(N)]) - 1)**2 for d in range(D)])

# 全員の出勤回数を均等にするための項
H3 = np.sum([(np.sum([q[n,d] for d in range(D)]) - F[n])**2 for n in range(N)])

# 最小化したいQUBO
H = Placeholder('alpha') * Constraint(H1, 'H1') + Placeholder('lam') * Constraint(H2, 'H2') + Placeholder('gamma') * H3
model = H.compile()
```

```python
feed_dict = {'alpha': 1.0, 'lam': 1.3, 'gamma': 0.3} # 制約項の係数
bqm = model.to_bqm(feed_dict=feed_dict)
```

ここで、サンプラーの設定をします。

```python
# SAの場合
# from neal import SimulatedAnnealingSampler
# sampler = SimulatedAnnealingSampler()

# D-Waveマシンの場合
from dwave.system import DWaveSampler, EmbeddingComposite
sampler_config = {'solver': 'DW_2000Q_6', 'token': 'YOUR_TOKEN'}
sampler = EmbeddingComposite(DWaveSampler(**sampler_config))
```

全ての設定が終わりました。
いよいよアニーリングを実行します。

```python
num_reads = 1000
sampleset = sampler.sample(bqm, num_reads=num_reads)
```

得られた解は、次のように確認できます。

```python
sampleset.record[:10]
```

    rec.array([([0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0], 1.76214598e-12, 1),
               ([1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0], 1.76214598e-12, 1),
               ([1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1], 1.75504056e-12, 1),
               ([0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0], 1.75504056e-12, 1),
               ([0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0], 6.00000000e-01, 1),
               ([0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0], 1.75504056e-12, 1),
               ([1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0], 1.75504056e-12, 1),
               ([0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0], 1.75504056e-12, 1),
               ([0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0], 1.75504056e-12, 1),
               ([0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1], 1.75504056e-12, 1)],
              dtype=[('sample', 'i1', (42,)), ('energy', '<f8'), ('num_occurrences', '<i8')])

`sampleset.record`には、

- 解の状態
- エネルギー
- 解の個数
- チェーンブレイク

の割合の順番で格納されています。本実験では基底状態のとき、`energy == 0`となるように入力を設定しています。

次に、制約条件を満たしている解を取り出します。

```python
def extract_feasible_samples(samples, print_broken=False):
    feasible_samples = []
    for sample in decoded_samples:
        constraints = sample.constraints(only_broken=True)
        if len(constraints) == 0:
            feasible_samples.append(sample)
        elif print_broken:
            print(constraints)
    return feasible_samples
```

```python
decoded_samples = model.decode_sampleset(sampleset.aggregate(), feed_dict)
feasible_samples = extract_feasible_samples(decoded_samples)
print('実行可能解の個数:', len(feasible_samples))
```

    実行可能解の個数: 897

得られた解で最もエネルギーの低い解を可視化します。

```python
import matplotlib.pyplot as plt

lowest_sample = feasible_samples[0].sample

schedules = np.zeros((N, D))
for n in range(N):
    for d in range(D):
        if lowest_sample[f'q[{n}][{d}]'] == 1:
            schedules[n, d] = 1

plt.imshow(schedules, cmap="Greys")
plt.xlabel('D')
plt.ylabel('N')
plt.show()
```

各看護師が希望する勤務日数と実際の日数が、どの程度近いかを確認します。

```python
print('実際の勤務日数:', np.sum(schedules, axis=1))
print('希望する勤務日数:', F)
```

    実際の勤務日数: [4. 5. 5.]
    希望する勤務日数: [4, 5, 5]

## 実験
### 「基底状態の解を見つける確率」の求め方

本実験は、「基底状態の解を見つける確率(%)」を求めることが目的です。これは、次の式で求められます。本実験では基底状態のとき、`energy == 0`となるように入力を設定しています。

[`energy == 0`の個数] / [`num_reads`] × 100

```python
def calc_probability(samples, num_reads):
    return sum(1 for sample in samples if abs(sample.energy) < 1e-9) / num_reads * 100
```

### 係数の探索方法
最適と考えられる係数は次のように探索しました。

まず、ソフト制約である第3項の係数$\gamma$を1.0に固定します。そして、第1項の係数$\alpha$=第2項の係数$\lambda$=1.0からスタートし、両方とも10.0になるまで1.0ずつ動かします.
それぞれの係数で「基底状態を見つける確率」をD-Wave 2000Qで計算し、確率が最も高くなった係数を採用します。この探索を$d$(日数)毎に実行します。次に、係数探索を行うコードを示します。

```python
lam_list = np.arange(1, 11)
alpha_list = np.arange(1, 11)

prob_hists = {}
for lam in lam_list:
    for alpha in alpha_list:
        print(f'lam: {lam}, alpha: {alpha}')
        # 制約の係数を変更
        feed_dict = {'alpha': alpha, 'lam': lam, 'gamma': 1.0}
        bqm = model.to_bqm(feed_dict=feed_dict)

        # アニーリング
        sampleset = sampler.sample(bqm, num_reads=num_reads)
        decoded_samples = model.decode_sampleset(sampleset.aggregate(), feed_dict)
        feasible_samples = extract_feasible_samples(decoded_samples)

        # 確率を求める
        prob_hists[(alpha, lam)] = calc_probability(feasible_samples, num_reads)
```

以下の図は、$N=3,D=5$の時のヒートマップです。この場合、第1項の係数$\alpha=3.0$、第2項の係数$\lambda=2.0$にすると最適な結果が得られると考えられます。

### 結果

以下の図から、係数を調整することで精度は向上することが分かります。しかし、先行研究と同様、日数$D$が大きくなると確率はかなり低くなってしまいました。

## 結論

本記事では、制約項の係数調整を行うことで、論文よりもNSPを精度良く解くことが出来るのかを検証しました。係数調整により精度は向上したものの、日数が大きくなると、基底状態の解はほとんど得られませんでした。

## あと書き

最大で56変数の問題に対して、基底状態の解を上手く見つけられないのは残念でした。本記事よりも、より細かく係数探索を行うことで精度向上の余地はありますが、日数が大きくなるにつれてほとんど解けなくなるのは変わらないと考えられます。他に精度を向上させる方法として埋め込みを自分で設定する方法がありますが、現在の自分の知識ではどのような埋め込みを行えば良いのか分かりません。またこの結果では、せいぜい1週間のシフトしか組めないため、実用化は難しいと思います。最低でも1か月のスケジュールを作れるようになったら、もう1度実験を行ってみたいです。

### 本記事の担当者
鹿内怜央


+ This notebook is part of lecture 11 *Matrix spaces, rank 1, small world graphs* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
#import numpy as np
from sympy import init_printing, Matrix, symbols
#import matplotlib.pyplot as plt
#import seaborn as sns
#from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax') # Pretty Latex printing to the screen
#%matplotlib inline
filterwarnings('ignore')
```

# Matrix spaces

## New *vector* / matrix spaces

### Square matrices

* Consider *M* to be all 3×3 matrices (with real elements)

* Subspaces would be:
    * Upper or lower triangular matrices
    * Symmetric matrices

* Basis would be:
$$ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

* The dimension would be 9

* For upper and lower triangular matrices the dimensions would be 6 and the basis:
$$ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

* For symmetric matrices the dimension would also be six ( Knowing the diagonal and entries on one of the two sides)

* These are unique cases where the bases for the subspaces are contained in the basis of the 3×3 matrix *M*

### Other square matrices that are subspaces of *M*

* The intersection of symmetric and upper triangular matrices (that is symmetric AND upper triangular, *S*∩*U*)
    * This is a diagonal matrix
    * The dimension is 3
    * The basis is
$$ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

* The union of symmetric and upper triangular matrices (that is symmetric OR upper triangular, *S*∪*U*)
    * It is **NOT** a subspace

* The addition (sum) of symmetric and upper triangular matrices
    * It **IS** a subspace
    * It is actually all 3×3 matrices
    * The dimension is 9

* This gives the equation: dim(*S*) + dim(*U*) = dim(*S*∩*U*) + dim(*S*+*U*) = 12

## Example problems

### Example problem 1

* Show that the set of 2×3 matrices whose nullspace contains the column vector below is a vector subspace and find a basis for it
$$ \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix} $$

#### Solution

* In essence we have to show the following
$$ A\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix}\\ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & { a }_{ 13 } \\ { a }_{ 21 } & { a }_{ 22 } & { a }_{ 23 } \end{bmatrix}\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix} $$
* ... and ...
$$ B\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix} $$

* This can be shown by addition:
$$ \left( A+B \right) \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix}=A\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}+B\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix} $$
* Therefor (by virtue of the fact that addition remains in the nullspace) the set is vector subspace

* We also need to look at scalar multiplication (if we multiply a matrix in the set by a scalar, does it remain in the set)
$$ \left( cA \right) \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=c\left( A\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}= \right) c\begin{bmatrix} 0 \\ 0 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix} $$

#### Example problem 2

* Find a basis for the nullspace above

#### Solution

* Let's look at the first row:
$$ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & { a }_{ 13 } \end{bmatrix}\begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \end{bmatrix}\\ 2{ a }_{ 11 }+{ a }_{ 12 }+{ a }_{ 13 }=0\\ { a }_{ 13 }=-2{ a }_{ 11 }-{ a }_{ 12 } $$

* From this we can make the following row vectors
$$ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & { a }_{ 13 } \end{bmatrix}=\begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & \left( -2{ a }_{ 11 }-{ a }_{ 12 } \right)  \end{bmatrix}\\ =\quad \begin{bmatrix} { a }_{ 11 } & 0 & { -2a }_{ 11 } \end{bmatrix}+\begin{bmatrix} 0 & { a }_{ 12 } & { -a }_{ 12 } \end{bmatrix}\\ ={ a }_{ 11 }\begin{bmatrix} 1 & 0 & -2 \end{bmatrix}+{ a }_{ 12 }\begin{bmatrix} 0 & 1 & -1 \end{bmatrix} $$

* From this we can construct 4 basis:
$$ \begin{bmatrix} 1 & 0 & -2 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 1 & -1 \\ 0 & 0 & 0 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 1 & 0 & -2 \end{bmatrix},\begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & -1 \end{bmatrix} $$

#### Example problem 3

* What about the set of those whose column space contains the following column vector?
$$ \begin{bmatrix} 2 \\ 1 \end{bmatrix} $$

#### Solution

* Well, any subspace must contain the zero matrix
$$ \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix} $$
* It does not contain the above column vector, which is therefor not a subspace

```python

```


```python
%matplotlib inline
```

# Sequence-to-Sequence Modeling with nn.Transformer and TorchText

This is a tutorial on how to train a sequence-to-sequence model
that uses the
[`nn.Transformer`](https://pytorch.org/docs/master/nn.html?highlight=nn%20transformer#torch.nn.Transformer) module.

PyTorch 1.2 release includes a standard transformer module based on the paper [Attention is All You Need](https://arxiv.org/pdf/1706.03762.pdf).
The transformer model has been proved to be superior in quality for many sequence-to-sequence problems while being more parallelizable.
The `nn.Transformer` module relies entirely on an attention mechanism (another module recently implemented as [`nn.MultiheadAttention`](https://pytorch.org/docs/master/nn.html?highlight=multiheadattention#torch.nn.MultiheadAttention) to draw global dependencies between input and output.
The `nn.Transformer` module is now highly modularized such that a single component (like [`nn.TransformerEncoder`](https://pytorch.org/docs/master/nn.html?highlight=nn%20transformerencoder#torch.nn.TransformerEncoder) in this tutorial) can be easily adapted/composed.

## Define the model

In this tutorial, we train ``nn.TransformerEncoder`` model on a language modeling task.
The language modeling task is to assign a probability for the likelihood of a given word (or a sequence of words) to follow a sequence of words.
A sequence of tokens are passed to the embedding layer first, followed by a positional encoding layer to account for the order of the word (see the next paragraph for more details).
The ``nn.TransformerEncoder`` consists of multiple layers of [`nn.TransformerEncoderLayer`](https://pytorch.org/docs/master/nn.html?highlight=transformerencoderlayer#torch.nn.TransformerEncoderLayer).
Along with the input sequence, a square attention mask is required because the self-attention layers in ``nn.TransformerEncoder`` are only allowed to attend the earlier positions in the sequence.
For the language modeling task, any tokens on the future positions should be masked. To have the actual words, the output of ``nn.TransformerEncoder`` model is sent to the final Linear layer, which is followed by a log-Softmax function.

```python
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerModel(nn.Module):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):
        super(TransformerModel, self).__init__()
        from torch.nn import TransformerEncoder, TransformerEncoderLayer
        self.model_type = 'Transformer'
        self.src_mask = None
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = nn.Linear(ninp, ntoken)

        self.init_weights()

    def _generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src):
        if self.src_mask is None or self.src_mask.size(0) != len(src):
            device = src.device
            mask = self._generate_square_subsequent_mask(len(src)).to(device)
            self.src_mask = mask

        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src, self.src_mask)
        output = self.decoder(output)
        return output
```

``PositionalEncoding`` module injects some information about the relative or absolute position of the tokens in the sequence.
The positional encodings have the same dimension as the embeddings so that the two can be summed. Here, we use ``sine`` and ``cosine`` functions of different frequencies.

```python
class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
```

Load and batch data
-------------------

The training process uses Wikitext-2 dataset from ``torchtext``.
The vocab object is built based on the train dataset and is used to numericalize tokens into tensors.
Starting from sequential data, the ``batchify()`` function arranges the dataset into columns, trimming off any tokens remaining after the data has been divided into batches of size ``batch_size``.
For instance, with the alphabet as the sequence (total length of 26) and a batch size of 4, we would divide the alphabet into 4 sequences of length 6:

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

These columns are treated as independent by the model, which means that the dependence of ``G`` and ``F`` can not be learned, but allows more efficient batch processing.

```python
import torchtext
from torchtext.data.utils import get_tokenizer
#TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"),
TEXT = torchtext.data.Field(tokenize=get_tokenizer("spacy"),
#TEXT = torchtext.data.Field(tokenize=get_tokenizer("moses"),
                            init_token='<sos>',
                            eos_token='<eos>',
                            lower=True)
train_txt, val_txt, test_txt = torchtext.datasets.WikiText2.splits(TEXT)
TEXT.build_vocab(train_txt)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def batchify(data, bsz):
    data = TEXT.numericalize([data.examples[0].text])
    # Divide the dataset into bsz parts.
    nbatch = data.size(0) // bsz
    # Trim off any extra elements that wouldn't cleanly fit (remainders).
    data = data.narrow(0, 0, nbatch * bsz)
    # Evenly divide the data across the bsz batches.
    data = data.view(bsz, -1).t().contiguous()
    return data.to(device)

batch_size = 20
eval_batch_size = 10
train_data = batchify(train_txt, batch_size)
val_data = batchify(val_txt, eval_batch_size)
test_data = batchify(test_txt, eval_batch_size)
```

    downloading wikitext-2-v1.zip

    wikitext-2-v1.zip: 100%|██████████| 4.48M/4.48M [00:00<00:00, 8.66MB/s]

    extracting

### Functions to generate input and target sequence

``get_batch()`` function generates the input and target sequence for the transformer model.
It subdivides the source data into chunks of length ``bptt``.
For the language modeling task, the model needs the following words as ``Target``.
For example, with a ``bptt`` value of 2, we’d get the following two Variables for ``i`` = 0:

It should be noted that the chunks are along dimension 0, consistent with the ``S`` dimension in the Transformer model. The batch dimension ``N`` is along dimension 1.

```python
bptt = 35
def get_batch(source, i):
    seq_len = min(bptt, len(source) - 1 - i)
    data = source[i:i+seq_len]
    target = source[i+1:i+1+seq_len].view(-1)
    return data, target
```

Initiate an instance
--------------------

The model is set up with the hyperparameter below. The vocab size is
equal to the length of the vocab object.

```python
ntokens = len(TEXT.vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 2 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value
model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout).to(device)
```

Run the model
-------------

[`CrossEntropyLoss`](https://pytorch.org/docs/master/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss) is applied to track the loss and [`SGD`](https://pytorch.org/docs/master/optim.html?highlight=sgd#torch.optim.SGD) implements stochastic gradient descent method as the optimizer.
The initial learning rate is set to 5.0.
[`StepLR`](https://pytorch.org/docs/master/optim.html?highlight=steplr#torch.optim.lr_scheduler.StepLR) is applied to adjust the learn rate through epochs.
During the training, we use [`nn.utils.clip_grad_norm_`](https://pytorch.org/docs/master/nn.html?highlight=nn%20utils%20clip_grad_norm#torch.nn.utils.clip_grad_norm_) function to scale all the gradient together to prevent exploding.

```python
criterion = nn.CrossEntropyLoss()
lr = 5.0 # learning rate
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

import time
def train():
    model.train() # Turn on the train mode
    total_loss = 0.
    start_time = time.time()
    ntokens = len(TEXT.vocab.stoi)
    for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
        data, targets = get_batch(train_data, i)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        log_interval = 200
        if batch % log_interval == 0 and batch > 0:
            cur_loss = total_loss / log_interval
            elapsed = time.time() - start_time
            print('| epoch {:3d} | {:5d}/{:5d} batches | '
                  'lr {:02.2f} | ms/batch {:5.2f} | '
                  'loss {:5.2f} | ppl {:8.2f}'.format(
                    epoch, batch, len(train_data) // bptt, scheduler.get_lr()[0],
                    elapsed * 1000 / log_interval,
                    cur_loss, math.exp(cur_loss)))
            total_loss = 0
            start_time = time.time()

def evaluate(eval_model, data_source):
    eval_model.eval() # Turn on the evaluation mode
    total_loss = 0.
    ntokens = len(TEXT.vocab.stoi)
    with torch.no_grad():
        for i in range(0, data_source.size(0) - 1, bptt):
            data, targets = get_batch(data_source, i)
            output = eval_model(data)
            output_flat = output.view(-1, ntokens)
            total_loss += len(data) * criterion(output_flat, targets).item()
    return total_loss / (len(data_source) - 1)
```

Loop over epochs. Save the model if the validation loss is the best
we've seen so far. Adjust the learning rate after each epoch.

```python
best_val_loss = float("inf")
epochs = 10 # The number of epochs
best_model = None

for epoch in range(1, epochs + 1):
    epoch_start_time = time.time()
    train()
    val_loss = evaluate(model, val_data)
    print('-' * 89)
    print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '
          'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),
                                     val_loss, math.exp(val_loss)))
    print('-' * 89)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_model = model

    scheduler.step()
```

    /usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:351: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
      "please use `get_last_lr()`.", UserWarning)

    | epoch   1 |   200/ 3195 batches | lr 2.44 | ms/batch 11.64 | loss  4.30 | ppl    73.64
    | epoch   1 |   400/ 3195 batches | lr 2.44 | ms/batch 11.55 | loss  4.33 | ppl    75.62
    | epoch   1 |   600/ 3195 batches | lr 2.44 | ms/batch 11.55 | loss  4.22 | ppl    67.98
    | epoch   1 |   800/ 3195 batches | lr 2.44 | ms/batch 11.60 | loss  4.19 | ppl    66.25
    | epoch   1 |  1000/ 3195 batches | lr 2.44 | ms/batch 11.51 | loss  4.33 | ppl    76.24
    | epoch   1 |  1200/ 3195 batches | lr 2.44 | ms/batch 11.47 | loss  4.29 | ppl    72.90
    | epoch   1 |  1400/ 3195 batches | lr 2.44 | ms/batch 11.47 | loss  4.32 | ppl    75.56
    | epoch   1 |  1600/ 3195 batches | lr 2.44 | ms/batch 11.57 | loss  4.27 | ppl    71.31
    | epoch   1 |  1800/ 3195 batches | lr 2.44 | ms/batch 11.50 | loss  4.33 | ppl    75.59
    | epoch   1 |  2000/ 3195 batches | lr 2.44 | ms/batch 11.48 | loss  4.36 | ppl    78.14
    | epoch   1 |  2200/ 3195 batches | lr 2.44 | ms/batch 11.49 | loss  4.31 | ppl    74.23
    | epoch   1 |  2400/ 3195 batches | lr 2.44 | ms/batch 11.55 | loss  4.17 | ppl    64.78
    | epoch   1 |  2600/ 3195 batches | lr 2.44 | ms/batch 11.59 | loss  4.28 | ppl    72.48
    | epoch   1 |  2800/ 3195 batches | lr 2.44 | ms/batch 11.51 | loss  4.27 | ppl    71.81
    | epoch   1 |  3000/ 3195 batches | lr 2.44 | ms/batch 11.47 | loss  4.23 | ppl    68.58
    -----------------------------------------------------------------------------------------
    | end of epoch   1 | time: 38.80s | valid loss  4.92 | valid ppl   137.19
    -----------------------------------------------------------------------------------------
    | epoch   2 |   200/ 3195 batches | lr 2.32 | ms/batch 11.55 | loss  4.26 | ppl    70.52
    | epoch   2 |   400/ 3195 batches | lr 2.32 | ms/batch 11.51 | loss  4.28 | ppl    72.45
    | epoch   2 |   600/ 3195 batches | lr 2.32 | ms/batch 11.54 | loss  4.18 | ppl    65.23
    | epoch   2 |   800/ 3195 batches | lr 2.32 | ms/batch 11.48 | loss  4.15 | ppl    63.51
    | epoch   2 |  1000/ 3195 batches | lr 2.32 | ms/batch 11.47 | loss  4.29 | ppl    73.00
    | epoch   2 |  1200/ 3195 batches | lr 2.32 | ms/batch 11.46 | loss  4.25 | ppl    69.88
    | epoch   2 |  1400/ 3195 batches | lr 2.32 | ms/batch 11.51 | loss  4.28 | ppl    72.57
    | epoch   2 |  1600/ 3195 batches | lr 2.32 | ms/batch 11.49 | loss  4.22 | ppl    68.28
    | epoch   2 |  1800/ 3195 batches | lr 2.32 | ms/batch 11.50 | loss  4.28 | ppl    72.16
    | epoch   2 |  2000/ 3195 batches | lr 2.32 | ms/batch 11.47 | loss  4.31 | ppl    74.81
    | epoch   2 |  2200/ 3195 batches | lr 2.32 | ms/batch 11.56 | loss  4.26 | ppl    70.64
    | epoch   2 |  2400/ 3195 batches | lr 2.32 | ms/batch 11.54 | loss  4.14 | ppl    62.55
    | epoch   2 |  2600/ 3195 batches | lr 2.32 | ms/batch 11.53 | loss  4.24 | ppl    69.16
    | epoch   2 |  2800/ 3195 batches | lr 2.32 | ms/batch 11.53 | loss  4.23 | ppl    68.72
    | epoch   2 |  3000/ 3195 batches | lr 2.32 | ms/batch 11.51 | loss  4.19 | ppl    65.97
    -----------------------------------------------------------------------------------------
    | end of epoch   2 | time: 38.74s | valid loss  4.94 | valid ppl   139.39
    -----------------------------------------------------------------------------------------
    | epoch   3 |   200/ 3195 batches | lr 2.20 | ms/batch 11.69 | loss  4.21 | ppl    67.57
    | epoch   3 |   400/ 3195 batches | lr 2.20 | ms/batch 11.55 | loss  4.24 | ppl    69.67
    | epoch   3 |   600/ 3195 batches | lr 2.20 | ms/batch 11.53 | loss  4.14 | ppl    62.55
    | epoch   3 |   800/ 3195 batches | lr 2.20 | ms/batch 11.49 | loss  4.11 | ppl    61.23
    | epoch   3 |  1000/ 3195 batches | lr 2.20 | ms/batch 11.50 | loss  4.25 | ppl    70.45
    | epoch   3 |  1200/ 3195 batches | lr 2.20 | ms/batch 11.47 | loss  4.21 | ppl    67.59
    | epoch   3 |  1400/ 3195 batches | lr 2.20 | ms/batch 11.49 | loss  4.24 | ppl    69.69
    | epoch   3 |  1600/ 3195 batches | lr 2.20 | ms/batch 11.43 | loss  4.18 | ppl    65.42
    | epoch   3 |  1800/ 3195 batches | lr 2.20 | ms/batch 11.49 | loss  4.25 | ppl    69.82
    | epoch   3 |  2000/ 3195 batches | lr 2.20 | ms/batch 11.52 | loss  4.28 | ppl    72.38
    | epoch   3 |  2200/ 3195 batches | lr 2.20 | ms/batch 11.45 | loss  4.22 | ppl    68.36
    | epoch   3 |  2400/ 3195 batches | lr 2.20 | ms/batch 11.48 | loss  4.10 | ppl    60.41
    | epoch   3 |  2600/ 3195 batches | lr 2.20 | ms/batch 11.47 | loss  4.20 | ppl    66.84
    | epoch   3 |  2800/ 3195 batches | lr 2.20 | ms/batch 11.49 | loss  4.20 | ppl    66.59
    | epoch   3 |  3000/ 3195 batches | lr 2.20 | ms/batch 11.55 | loss  4.15 | ppl    63.73
    -----------------------------------------------------------------------------------------
    | end of epoch   3 | time: 38.69s | valid loss  4.95 | valid ppl   141.09
    -----------------------------------------------------------------------------------------
    | epoch   4 |   200/ 3195 batches | lr 2.09 | ms/batch 11.50 | loss  4.19 | ppl    65.77
    | epoch   4 |   400/ 3195 batches | lr 2.09 | ms/batch 11.45 | loss  4.21 | ppl    67.21
    | epoch   4 |   600/ 3195 batches | lr 2.09 | ms/batch 11.42 | loss  4.10 | ppl    60.55
    | epoch   4 |   800/ 3195 batches | lr 2.09 | ms/batch 11.46 | loss  4.09 | ppl    59.59
    | epoch   4 |  1000/ 3195 batches | lr 2.09 | ms/batch 11.39 | loss  4.22 | ppl    68.14
    | epoch   4 |  1200/ 3195 batches | lr 2.09 | ms/batch 11.46 | loss  4.18 | ppl    65.41
    | epoch   4 |  1400/ 3195 batches | lr 2.09 | ms/batch 11.50 | loss  4.21 | ppl    67.52
    | epoch   4 |  1600/ 3195 batches | lr 2.09 | ms/batch 11.45 | loss  4.15 | ppl    63.51
    | epoch   4 |  1800/ 3195 batches | lr 2.09 | ms/batch 11.46 | loss  4.21 | ppl    67.24
    | epoch   4 |  2000/ 3195 batches | lr 2.09 | ms/batch 11.47 | loss  4.25 | ppl    69.80
    | epoch   4 |  2200/ 3195 batches | lr 2.09 | ms/batch 11.47 | loss  4.19 | ppl    66.05
    | epoch   4 |  2400/ 3195 batches | lr 2.09 | ms/batch 11.50 | loss  4.07 | ppl    58.29
    | epoch   4 |  2600/ 3195 batches | lr 2.09 | ms/batch 11.53 | loss  4.17 | ppl    64.53
    | epoch   4 |  2800/ 3195 batches | lr 2.09 | ms/batch 11.55 | loss  4.16 | ppl    64.11
    | epoch   4 |  3000/ 3195 batches | lr 2.09 | ms/batch 11.43 | loss  4.12 | ppl    61.39
    -----------------------------------------------------------------------------------------
    | end of epoch   4 | time: 38.57s | valid loss  4.94 | valid ppl   140.24
    -----------------------------------------------------------------------------------------
    | epoch   5 |   200/ 3195 batches | lr 1.99 | ms/batch 11.55 | loss  4.15 | ppl    63.13
    | epoch   5 |   400/ 3195 batches | lr 1.99 | ms/batch 11.44 | loss  4.18 | ppl    65.10
    | epoch   5 |   600/ 3195 batches | lr 1.99 | ms/batch 11.51 | loss  4.07 | ppl    58.53
    | epoch   5 |   800/ 3195 batches | lr 1.99 | ms/batch 11.47 | loss  4.05 | ppl    57.53
    | epoch   5 |  1000/ 3195 batches | lr 1.99 | ms/batch 11.48 | loss  4.19 | ppl    66.02
    | epoch   5 |  1200/ 3195 batches | lr 1.99 | ms/batch 11.49 | loss  4.14 | ppl    62.98
    | epoch   5 |  1400/ 3195 batches | lr 1.99 | ms/batch 11.46 | loss  4.18 | ppl    65.26
    | epoch   5 |  1600/ 3195 batches | lr 1.99 | ms/batch 11.49 | loss  4.11 | ppl    61.23
    | epoch   5 |  1800/ 3195 batches | lr 1.99 | ms/batch 11.46 | loss  4.18 | ppl    65.20
    | epoch   5 |  2000/ 3195 batches | lr 1.99 | ms/batch 11.52 | loss  4.22 | ppl    67.92
    | epoch   5 |  2200/ 3195 batches | lr 1.99 | ms/batch 11.52 | loss  4.16 | ppl    64.12
    | epoch   5 |  2400/ 3195 batches | lr 1.99 | ms/batch 11.59 | loss  4.03 | ppl    56.27
    | epoch   5 |  2600/ 3195 batches | lr 1.99 | ms/batch 11.56 | loss  4.13 | ppl    62.20
    | epoch   5 |  2800/ 3195 batches | lr 1.99 | ms/batch 11.50 | loss  4.13 | ppl    62.10
    | epoch   5 |  3000/ 3195 batches | lr 1.99 | ms/batch 11.49 | loss  4.08 | ppl    59.44
    -----------------------------------------------------------------------------------------
    | end of epoch   5 | time: 38.68s | valid loss  4.95 | valid ppl   141.03
    -----------------------------------------------------------------------------------------
    | epoch   6 |   200/ 3195 batches | lr 1.89 | ms/batch 11.49 | loss  4.11 | ppl    60.95
    | epoch   6 |   400/ 3195 batches | lr 1.89 | ms/batch 11.47 | loss  4.14 | ppl    62.83
    | epoch   6 |   600/ 3195 batches | lr 1.89 | ms/batch 11.46 | loss  4.04 | ppl    56.84
    | epoch   6 |   800/ 3195 batches | lr 1.89 | ms/batch 11.47 | loss  4.03 | ppl    56.07
    | epoch   6 |  1000/ 3195 batches | lr 1.89 | ms/batch 11.55 | loss  4.16 | ppl    64.04
    | epoch   6 |  1200/ 3195 batches | lr 1.89 | ms/batch 11.54 | loss  4.12 | ppl    61.46
    | epoch   6 |  1400/ 3195 batches | lr 1.89 | ms/batch 11.44 | loss  4.15 | ppl    63.38
    | epoch   6 |  1600/ 3195 batches | lr 1.89 | ms/batch 11.41 | loss  4.08 | ppl    59.31
    | epoch   6 |  1800/ 3195 batches | lr 1.89 | ms/batch 11.48 | loss  4.15 | ppl    63.35
    | epoch   6 |  2000/ 3195 batches | lr 1.89 | ms/batch 11.41 | loss  4.18 | ppl    65.42
    | epoch   6 |  2200/ 3195 batches | lr 1.89 | ms/batch 11.51 | loss  4.13 | ppl    62.19
    | epoch   6 |  2400/ 3195 batches | lr 1.89 | ms/batch 11.46 | loss  4.00 | ppl    54.56
    | epoch   6 |  2600/ 3195 batches | lr 1.89 | ms/batch 11.47 | loss  4.10 | ppl    60.26
    | epoch   6 |  2800/ 3195 batches | lr 1.89 | ms/batch 11.46 | loss  4.09 | ppl    60.02
    | epoch   6 |  3000/ 3195 batches | lr 1.89 | ms/batch 11.45 | loss  4.06 | ppl    57.82
    -----------------------------------------------------------------------------------------
    | end of epoch   6 | time: 38.57s | valid loss  4.95 | valid ppl   140.76
    -----------------------------------------------------------------------------------------
    | epoch   7 |   200/ 3195 batches | lr 1.79 | ms/batch 11.53 | loss  4.09 | ppl    59.48
    | epoch   7 |   400/ 3195 batches | lr 1.79 | ms/batch 11.43 | loss  4.12 | ppl    61.42
    | epoch   7 |   600/ 3195 batches | lr 1.79 | ms/batch 11.45 | loss  4.01 | ppl    55.27
    | epoch   7 |   800/ 3195 batches | lr 1.79 | ms/batch 11.48 | loss  3.99 | ppl    54.23
    | epoch   7 |  1000/ 3195 batches | lr 1.79 | ms/batch 11.43 | loss  4.13 | ppl    62.08
    | epoch   7 |  1200/ 3195 batches | lr 1.79 | ms/batch 11.44 | loss  4.09 | ppl    59.74
    | epoch   7 |  1400/ 3195 batches | lr 1.79 | ms/batch 11.38 | loss  4.12 | ppl    61.53
    | epoch   7 |  1600/ 3195 batches | lr 1.79 | ms/batch 11.39 | loss  4.05 | ppl    57.58
    | epoch   7 |  1800/ 3195 batches | lr 1.79 | ms/batch 11.40 | loss  4.11 | ppl    61.25
    | epoch   7 |  2000/ 3195 batches | lr 1.79 | ms/batch 11.47 | loss  4.16 | ppl    64.01
    | epoch   7 |  2200/ 3195 batches | lr 1.79 | ms/batch 11.43 | loss  4.10 | ppl    60.42
    | epoch   7 |  2400/ 3195 batches | lr 1.79 | ms/batch 11.42 | loss  3.97 | ppl    53.14
    | epoch   7 |  2600/ 3195 batches | lr 1.79 | ms/batch 11.41 | loss  4.07 | ppl    58.66
    | epoch   7 |  2800/ 3195 batches | lr 1.79 | ms/batch 11.53 | loss  4.06 | ppl    58.17
    | epoch   7 |  3000/ 3195 batches | lr 1.79 | ms/batch 11.49 | loss  4.03 | ppl    56.04
    -----------------------------------------------------------------------------------------
    | end of epoch   7 | time: 38.48s | valid loss  4.96 | valid ppl   142.14
    -----------------------------------------------------------------------------------------
    | epoch   8 |   200/ 3195 batches | lr 1.70 | ms/batch 11.46 | loss  4.06 | ppl    57.71
    | epoch   8 |   400/ 3195 batches | lr 1.70 | ms/batch 11.48 | loss  4.09 | ppl    59.51
    | epoch   8 |   600/ 3195 batches | lr 1.70 | ms/batch 11.42 | loss  3.98 | ppl    53.61
    | epoch   8 |   800/ 3195 batches | lr 1.70 | ms/batch 11.39 | loss  3.97 | ppl    52.86
    | epoch   8 |  1000/ 3195 batches | lr 1.70 | ms/batch 11.44 | loss  4.10 | ppl    60.52
    | epoch   8 |  1200/ 3195 batches | lr 1.70 | ms/batch 11.46 | loss  4.06 | ppl    57.78
    | epoch   8 |  1400/ 3195 batches | lr 1.70 | ms/batch 11.45 | loss  4.09 | ppl    59.78
    | epoch   8 |  1600/ 3195 batches | lr 1.70 | ms/batch 11.42 | loss  4.03 | ppl    55.99
    | epoch   8 |  1800/ 3195 batches | lr 1.70 | ms/batch 11.39 | loss  4.09 | ppl    59.86
    | epoch   8 |  2000/ 3195 batches | lr 1.70 | ms/batch 11.45 | loss  4.13 | ppl    62.31
    | epoch   8 |  2200/ 3195 batches | lr 1.70 | ms/batch 11.43 | loss  4.07 | ppl    58.80
    | epoch   8 |  2400/ 3195 batches | lr 1.70 | ms/batch 11.47 | loss  3.94 | ppl    51.55
    | epoch   8 |  2600/ 3195 batches | lr 1.70 | ms/batch 11.54 | loss  4.04 | ppl    56.69
    | epoch   8 |  2800/ 3195 batches | lr 1.70 | ms/batch 11.42 | loss  4.04 | ppl    56.65
    | epoch   8 |  3000/ 3195 batches | lr 1.70 | ms/batch 11.43 | loss  4.00 | ppl    54.75
    -----------------------------------------------------------------------------------------
    | end of epoch   8 | time: 38.48s | valid loss  4.99 | valid ppl   146.84
    -----------------------------------------------------------------------------------------
    | epoch   9 |   200/ 3195 batches | lr 1.62 | ms/batch 11.42 | loss  4.03 | ppl    56.29
    | epoch   9 |   400/ 3195 batches | lr 1.62 | ms/batch 11.43 | loss  4.06 | ppl    58.20
    | epoch   9 |   600/ 3195 batches | lr 1.62 | ms/batch 11.42 | loss  3.96 | ppl    52.43
    | epoch   9 |   800/ 3195 batches | lr 1.62 | ms/batch 11.42 | loss  3.95 | ppl    51.71
    | epoch   9 |  1000/ 3195 batches | lr 1.62 | ms/batch 11.41 | loss  4.08 | ppl    59.05
    | epoch   9 |  1200/ 3195 batches | lr 1.62 | ms/batch 11.46 | loss  4.04 | ppl    56.62
    | epoch   9 |  1400/ 3195 batches | lr 1.62 | ms/batch 11.45 | loss  4.07 | ppl    58.45
    | epoch   9 |  1600/ 3195 batches | lr 1.62 | ms/batch 11.47 | loss  4.00 | ppl    54.56
    | epoch   9 |  1800/ 3195 batches | lr 1.62 | ms/batch 11.42 | loss  4.07 | ppl    58.37
    | epoch   9 |  2000/ 3195 batches | lr 1.62 | ms/batch 11.38 | loss  4.10 | ppl    60.50
    | epoch   9 |  2200/ 3195 batches | lr 1.62 | ms/batch 11.41 | loss  4.05 | ppl    57.59
    | epoch   9 |  2400/ 3195 batches | lr 1.62 | ms/batch 11.42 | loss  3.92 | ppl    50.42
    | epoch   9 |  2600/ 3195 batches | lr 1.62 | ms/batch 11.35 | loss  4.02 | ppl    55.44
    | epoch   9 |  2800/ 3195 batches | lr 1.62 | ms/batch 11.38 | loss  4.01 | ppl    55.34
    | epoch   9 |  3000/ 3195 batches | lr 1.62 | ms/batch 11.40 | loss  3.97 | ppl    53.18
    -----------------------------------------------------------------------------------------
    | end of epoch   9 | time: 38.39s | valid loss  4.99 | valid ppl   146.35
    -----------------------------------------------------------------------------------------
    | epoch  10 |   200/ 3195 batches | lr 1.54 | ms/batch 11.44 | loss  4.01 | ppl    55.16
    | epoch  10 |   400/ 3195 batches | lr 1.54 | ms/batch 11.47 | loss  4.03 | ppl    56.49
    | epoch  10 |   600/ 3195 batches | lr 1.54 | ms/batch 11.41 | loss  3.93 | ppl    51.11
    | epoch  10 |   800/ 3195 batches | lr 1.54 | ms/batch 11.45 | loss  3.92 | ppl    50.48
    | epoch  10 |  1000/ 3195 batches | lr 1.54 | ms/batch 11.40 | loss  4.06 | ppl    57.76
    | epoch  10 |  1200/ 3195 batches | lr 1.54 | ms/batch 11.38 | loss  4.01 | ppl    55.28
    | epoch  10 |  1400/ 3195 batches | lr 1.54 | ms/batch 11.36 | loss  4.04 | ppl    56.75
    | epoch  10 |  1600/ 3195 batches | lr 1.54 | ms/batch 11.37 | loss  3.97 | ppl    53.20
    | epoch  10 |  1800/ 3195 batches | lr 1.54 | ms/batch 11.45 | loss  4.04 | ppl    56.95
    | epoch  10 |  2000/ 3195 batches | lr 1.54 | ms/batch 11.39 | loss  4.08 | ppl    59.32
    | epoch  10 |  2200/ 3195 batches | lr 1.54 | ms/batch 11.38 | loss  4.02 | ppl    55.94
    | epoch  10 |  2400/ 3195 batches | lr 1.54 | ms/batch 11.43 | loss  3.89 | ppl    49.08
    | epoch  10 |  2600/ 3195 batches | lr 1.54 | ms/batch 11.41 | loss  3.99 | ppl    54.14
    | epoch  10 |  2800/ 3195 batches | lr 1.54 | ms/batch 11.41 | loss  3.99 | ppl    53.91
    | epoch  10 |  3000/ 3195 batches | lr 1.54 | ms/batch 11.41 | loss  3.95 | ppl    52.15
    -----------------------------------------------------------------------------------------
    | end of epoch  10 | time: 38.42s | valid loss  4.98 | valid ppl   145.14
    -----------------------------------------------------------------------------------------

Evaluate the model with the test dataset
-------------------------------------

Apply the best model to check the result with the test dataset.

```python
test_loss = evaluate(best_model, test_data)
print('=' * 89)
print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(
    test_loss, math.exp(test_loss)))
print('=' * 89)
```

    =========================================================================================
    | End of training | test loss  4.82 | test ppl   124.40
    =========================================================================================

```python

```


```python
%matplotlib nbagg
import warnings
import inspect
import matplotlib.pyplot as plt
import IPython.display
from cued_sf2_lab.familiarisation import load_mat_img, plot_image
import numpy as np
from typing import Tuple
```

Figure 5: An $L$ level binary discrete wavelet transform.

# 9 The Discrete Wavelet Transform (DWT)

This notebook is incomplete!

The final method of energy compaction that we shall investigate, is the
discrete wavelet transform. In some ways this attempts to combine the best features of
the Laplacian pyramid and the DCT:

* Like the pyramid, the DWT analyses the image at a range of different
  scales (levels) and employs symmetrical filters;

* Like the DCT, the DWT avoids any expansion in the number of coefficients.

Wavelet theory was evolved by mathematicians during the 1980's. As with the LBT, we shall not attempt to teach this theory here, just illustrate a relatively simple form of it.

Wavelets are short waveforms which are usually the impulse responses of
filters.  Wavelet transforms employ banks of bandpass filters, whose impulse
responses are scaled versions of each other, in
order to get pass-bands in different parts of the frequency spectrum.  If the
impulse response of a filter is scaled in time by a factor $a$, then the
filter frequency response is scaled by the factor $1/a$.  Typically $a = 2$
from one filter to the next, and each bandpass filter is designed to pass a
2:1 range of frequencies (one octave). We can split an image up using wavelets by a process known as a _binary wavelet tree_.

## 9.1 The binary wavelet tree

We start in 1-D with the
simplest possible pair of filters, operating on just two input samples, $x_n$
and $x_{n-1}$.  The two filter outputs, $u_n$ and $v_n$ at time $n$ are
given by:
$$
 u_n = \tfrac{1}{2} (x_n + x_{n-1}) \quad \mbox{and} \quad
 v_n = \tfrac{1}{2} (x_n - x_{n-1})
$$

The first filter averages adjacent samples, and so rejects the higher
frequency components of $x$, while the second filter differences these
samples, and so rejects the lower frequency components.  These filters are
known as the _analysis_ filter pair, $H_1(z) = \tfrac{1}{2} (1 + z^{-1})$
and $H_2(z) = \tfrac{1}{2} (1 - z^{-1})$.  It is clear that we can recover the
two input samples from the filter outputs using:
$$
 x_n = u_n + v_n \quad \mbox{and} \quad x_{n-1} = u_n - v_n
$$

Next it is important to note that we need only retain the samples of $u_n$
and $v_n$ at even values of $n$ in order to be able to recover all the
original samples of $x$.  Hence $u$ and $v$ may be decimated 2:1 and still
allow perfect reconstruction of $x$.  If $x$ is a finite length vector (e.g. a
row of image pixels), then $u$ and $v$ are each half as long as $x$, so the
total number of samples is preserved by the transformation.

A wavelet binary tree may be constructed using these filters, by using an
identical pair, $H_1$ and $H_2$, to filter the decimated lowpass signal
$u_{2n}$, to give a pair of outputs, $uu_{2n}$ and $uv_{2n}$, representing
the lower and upper halves of the first low band.  These may again be
decimated 2:1 and still permit perfect reconstruction of $u$.  This process
may be continued as often as desired: each time splitting the lowest band in
two, and decimating the sample rate of the filter outputs by 2:1.  At each
stage the bandwidth of the two lowest filters is halved, and their impulse
responses are doubled in length.  The total number of output samples remains
constant, however many stages are used.

For example, if $f_s$ is the input sample rate, a 3-stage binary tree will
split the input signal bandwidth of 0 to $f_s/2$ into the following four
bands:
$$
0 \rightarrow f_s/16; \ \ f_s/16 \rightarrow f_s/8; \ \ f_s/8 \rightarrow
f_s/4;  \ \ f_s/4 \rightarrow f_s/2.
$$

The very simple filters, given above, do not generate a filter tree with
good characteristics, since the wavelets turn out to be just a pair of
square pulses.  These generate _blocking_ artefacts when used for image
compression (in fact they are equivalent to the 2 point ($N=2$)
DCT). A better set of filters are the LeGall 5 and 3 tap pair,
given by:
$$
 u_n = \tfrac{1}{8} (-x_{n+2} + 2 x_{n+1} + 6 x_n + 2 x_{n-1} - x_{n-2})
  \quad \mbox{ and }  \quad
 v_{n+1} = \tfrac{1}{4} (-x_{n+2} + 2 x_{n+1} - x_n)
$$

If $u$ and $v$ are decimated by 2 by choosing even $n$ only, the lowband outputs
$u_n$ are centred on the even samples, and the highband outputs $v_{n+1}$ are
centred on the odd samples.  This is very important to allow perfect
reconstruction of $x$ from $u$ and $v$.

The equations for reconstruction may be obtained by solving the above to get:

\begin{align}
 x_n &= \tfrac{1}{2} (-v_{n+1} + 2 u_n - v_{n-1}) \quad \mbox{and}  \\
 x_{n+1} &= \tfrac{1}{2} (x_{n+2} + 4 v_{n+1} + x_n) =
 \tfrac{1}{4} (-v_{n+3} + 2 u_{n+2} + 6 v_{n+1} + 2 u_n - v_{n-1})
\end{align}

In general, most analysis filters will not yield such simple reconstruction
solutions, and the design of suitable filters is a non-trivial topic that we
shall not cover here.

# 9.2 Applying the DWT to images

As with the DCT, the 2-D DWT may be obtained by applying a 1-D transform to
first the rows and then the columns of an image.

Start by loading the Lighthouse image and defining the two LeGall
filters given above:

```python
X, _ = load_mat_img(img='lighthouse.mat', img_info='X', cmap_info={'map', 'map2'})
X = X - 128.0
h1 = np.array([-1, 2, 6, 2, -1])/8
h2 = np.array([-1, 2, -1])/4
```

We can use the function `rowdec` from the pyramid work, to
produce a decimated and lowpass filtered version of the rows of
`X` (remembering to subtract 128 as before) using:

```python
from cued_sf2_lab.laplacian_pyramid import rowdec
U = rowdec(X, h1)
```

To get the high-pass image `V`, it is important to align the decimated
samples with the odd columns of `X` (assuming the first column is $n = 0$)
whereas `U` is aligned with the even columns.  To do this we use a
slightly modified version of `rowdec`, called `rowdec2`.

```python
from cued_sf2_lab.laplacian_pyramid import rowdec2
V = rowdec2(X, h2)
```

Display `U` and `V` to see the outputs of the first filter pair
and comment on their relative energies (or standard deviations). Note that `U` and `V` are half the width of `X`, but that `U` is otherwise similar to `X`.

```python
# your code here
```

Now filter the columns of `U` and `V` using `rowdec
/ rowdec2` with the transpose operator:

```python
UU = rowdec(U.T, h1).T
UV = rowdec2(U.T, h2).T
VU = rowdec(V.T, h1).T
VV = rowdec2(V.T, h2).T
```

Display `np.block([[UU, VU], [UV, VV]])`, and comment
on what sort of edges or features are selected by each filter. You may need to multiply the high-pass images by a factor $k > 1$ to display them clearly. Why is this?

```python
# your code here
```

We must now check that it is possible to recover the image from
these sub-images, using reconstruction filters, `g1` and `g2`, and the functions, `rowint` and `rowint` (which
is modified in a similar way to `rowdec2` to allow correct
alignment of the high-pass samples). To reconstruct `Ur` and
`Vr` from `UU`, `UV`, `VU` and `VV` use:

```python
from cued_sf2_lab.laplacian_pyramid import rowint, rowint2

g1 = np.array([1, 2, 1])/2
g2 = np.array([-1, -2, 6, -2, -1])/4
Ur = rowint(UU.T, g1).T + rowint2(UV.T, g2).T
Vr = rowint(VU.T, g1).T + rowint2(VV.T, g2).T
```

Note the gain of 2 in the reconstruction filters, `g1` and
`g2` (to compensate for losing half the samples in the
decimation / interpolation processes).   These filters are also
not quite the same as those that might be inferred from the
equations for $x_n$ and $x_{n+1}$ on the previous page.  This is
because `g1` defines how {\it only} the $u$ samples contribute
both to the even and odd samples of $x$, while `g2` defines
how the $v$ samples contribute.

Check that `Ur` and `Vr` are the same as `U` and
`V`, and then reconstruct `Xr` from these:

```python
# your code here to check Ur and Vr
```

```python
# demonstrator answer here
np.testing.assert_equal(Ur, U)
np.testing.assert_equal(Vr, V)
```

```python
Xr = rowint(Ur,g1) + rowint2(Vr,g2)
```

Check that `Xr` is the same as `X`.

```python
# your code here
```

The above operations are a bit tedious to repeat if we want to
apply the DWT recursively to obtain several levels of filtering,
so we have written a pair of functions, `dwt` and `idwt`, to perform the 2-D analysis and reconstruction
operations. Examine these to see that they perform the same
operations as above, except that the transformed sub-images are
stored as parts of a single matrix, the same size as `X`,
rather than as separate matrices.

```python
from cued_sf2_lab.dwt import dwt
IPython.display.Code(inspect.getsource(dwt), language="python")
```

def dwt(X: np.ndarray, h1: np.ndarray = h1, h2: np.ndarray = h2) -> np.ndarray:
    """
    Return a 1-level 2-D discrete wavelet transform of X.

    Default h1 and h2 are the LeGall filter pair.

    Parameters:
        X: Image matrix (Usually 256x256)
        h1, h2: Filter coefficients
    Returns:
        Y: 1-level 2D DWT of X
    """
    m, n = X.shape
    if m % 2 or n % 2:
        raise ValueError("Image dimensions must be even")
    Y = np.concatenate([rowdec(X, h1), rowdec2(X, h2)], axis=1)
    Y = np.concatenate([rowdec(Y.T, h1).T, rowdec2(Y.T, h2).T], axis=0)
    return Y

```python
from cued_sf2_lab.dwt import idwt
IPython.display.Code(inspect.getsource(idwt), language="python")
```

def idwt(X: np.ndarray, g1: np.ndarray = g1, g2: np.ndarray = g2)-> np.ndarray:
    """
    Return a 1-level 2-D inverse discrete wavelet transform on X.

    If filters G1 and G2 are given, then they are used, otherwise the LeGall
    filter pair are used.
    """
    m, n = X.shape
    if m % 2 or n % 2:
        raise ValueError("Image dimensions must be even")
    m2 = m//2
    n2 = n//2
    Y = rowint(X[:m2, :].T, g1).T + rowint2(X[m2:, :].T,g2).T;
    Y = rowint(Y[:, :n2], g1) + rowint2(Y[:, n2:], g2)
    return Y

You can check their operation as below::

```python
Y = dwt(X)
Xr = idwt(Y)

fig, axs = plt.subplots(1, 2)
plot_image(Y, ax=axs[0])
axs[0].set(title="Y")
plot_image(Xr, ax=axs[1])
axs[1].set(title="Xr");
```

    <IPython.core.display.Javascript object>

`Y` should be the same as the composite `[UU VU; UV VV]` image that
you displayed earlier, and `Xr` should be the same as `X`.

Now implement a multilevel DWT by first applying `dwt` to
`X` using:

```python
m=256
Y=dwt(X)
plot_image(Y, ax=some_axis)
```

and then iteratively apply `dwt` to the top left sub-image
of `Y` by repeating:
```python
m = m//2
Y[:m,:m] = dwt(Y[:m,:m])
plot_image(Y, ax=some_axis)
```

```python
# your code here
```

We now have the image split using a binary wavelet tree (stricly a
quaternary tree in 2-D).  Write
similar iterative code to that given above, which can reconstruct
the image from the final set of `Y` sub-images after a 4-level
wavelet transform. Check that your reconstructed image is the
same as `X`.

```python
# your code here
```

## 9.3 Quantisation and coding efficiency

First rewrite the sequences of operations required to perform
$n$ levels of DWT and inverse DWT as two separate M-files, `nlevdwt` and `nlevidwt`. `nlevdwt` should transform
`X` into `Y`, and `nlevidwt` should inverse
transform a quantised set of sub-images `Yq` into the
reconstructed image `Z`.  Check your functions by ensuring
that `Z` is the same as `X` if `Yq = Y`.

```python
def nlevdwt(X, n):
    # your code here
    pass

def nlevidwt(Y, n):
    # your code here
    pass
```

```python
# your code here to test `nlevdwt` and `nlevidwt`
```

Now design a function, `quantdwt`, which will quantise the
sub-images of `Y` to give `Yq` and calculate their
entropy.  The sub-images at each level `i` of the DWT should
be quantised according to a $3 \times (n+1)$ matrix `dwtstep[k,i]` of
step-sizes, where $\mathtt{k}=\left\{0,1,2\right\}$ corresponds to each of the three high-pass images at level `i` (top right, bottom left, and bottom right, respectively), and the final low-pass image is quantised with `dwtstep[0,n]`. This matrix will be populated either with the same number in all elements (for equal-step-size quantisation) or a range of different numbers (for equal-MSE quantisation). The entropies for each sub-image should be stored in a similar $3 \times (n+1)$ matrix `dwtent[k,i]`.

```python
def quantdwt(Y: np.ndarray, dwtstep: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
    """
    Parameters:
        Y: the output of `dwt(X, n)`
        dwtstep: an array of shape `(3, n+1)`
    Returns:
        Yq: the quantized version of `Y`
        dwtenc: an array of shape `(3, n+1)` containing the entropies
    """
    # your code here
    Yq = None
    dwtent = None
    return Yq, dwtent
```

Using these functions, for a given number of levels $n$ (typically
between 3 and 5), you should generate `Y`, quantise it to
give `Yq` and reconstruct `Z` from `Yq`.

```python
# your code here
```

All of our experiments thus far have been performed on only one image. At this stage it is worth starting to experiment with the additional `Bridge` image, as well as Lighthouse. Bridge contains a lot more fine detail and may not lead to the same conclusions regarding performance.

```python
Xb, _ = load_mat_img(img='bridge.mat', img_info='X', cmap_info={'map'})
Xb = Xb - 128.0
```

```python
fig, ax = plt.subplots()
plot_image(Xb, ax=ax)
ax.set(title="bridge.mat");
```

    <IPython.core.display.Javascript object>

Investigate the performance of
both an equal-step-size and an equal-MSE scheme (follow a similar procedure as you used for the Laplacian Pyramid to find the appropriate step-size ratios). Hence determine how many levels of DWT are reasonably optimal for the Lighthouse and Bridge images. Also evaluate the subjective quality of your reconstructed images, and comment on how this depends on $n$ and on the way that step-sizes are assigned
to the different levels. Once again, for each image choose quantisation steps such that you match the rms error to that for direct quantisation with a step-size of 17.

```python
# your code here
```

## 9.4 Second Interim Report

This report should include the new results from the DCT, LBT and DWT energy
compaction methods in a format that will allow them to be compared with each other and contrasted to the
Laplacian pyramid work in your first report.  Again try to answer questions
raised in the text, and also include discussion of any topics that have led to
unexpected results or have proved particularly interesting.


```python
from sympy import init_printing, S, symbols, Eq, solveset
```

```python
init_printing()
```

```python
# length dimensions
m, cm, mm, Km, nm, um = symbols('m, cm, mm, Km, nm, um')

# Time
s, ms, us = symbols('s, ms, us')

# Energy
cm_1, eV, J, kJ = symbols('cm_1, eV, J, kJ')

# Frequency
hz, khz, mhz, ghz, thz = symbols('hz, khz, mhz, ghz, thz')

```

```python
def make_double_symbols(x):
    lower = x.split(',')
    upper = []
    for i in lower:
        temp = i.capitalize()
        print(temp)

        upper.append(temp)

    return tuple(lower), tuple(upper)

test = 'hz, khz, mhz, ghz, thz'
temp = [i.strip().capitalize() for i in test.split(',')]
temp
```

    ['Hz', 'Khz', 'Mhz', 'Ghz', 'Thz']

```python
test.split(',')
```

    ['hz', ' khz', ' mhz', ' ghz', ' thz']

```python

```

```python

```

```python

```