# Project 1
from Artem Opperman's course: Deep Learning for Predictive Analytics

## 0 Importing `tensorflow-gpu==1.14`

- Artem's course uses Tensorflow 1.13
- As of 19th August 2019, Tensorflow 2.0 has been released
- The methods and classes used in this notebook are from Tensorflow 1.14

```python
import tensorflow as tf
from IPython.display import clear_output

tf.__version__
```

    '1.14.0'

## 1 Problem statement discussion

The project is based on the dataset used by Moro et al., 2014 [[1]](http://dx.doi.org/10.1016/j.dss.2014.03.001) in their publication: _S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems_

---

The publication's abstract provides the problem statement for us: " ***...to predict the success of telemarketing calls for selling bank long-term deposits***."

---

The dataset has 20 variables (predictive features) for the target variable $y$ which is defined as the answer to the binary question: has the client subscribed a term deposit?
The dataset is based on "Bank Marketing" UCI dataset. [[2]](http://archive.ics.uci.edu/ml/datasets/Bank+Marketing)The data is enriched by the addition of five new social and economic features/attributes (national wide indicators from a ~10M population country, published by the Banco de Portugal and publicly available[[3]](https://www.bportugal.pt/estatisticasweb) online. Please note that the version of the dataset used for the project is not the same as the one used by Moro et.al. 2014. It does not include all attributes due to privacy concerns.

The original dataset actually had 150 features with bank client, product and social-economic attributes. A semi-automatic feature selection was explored in the modeling phase, performed with the data prior to July 2012 and that allowed to select a reduced set of 22 features. We will be working with this reduced version.

The zip file provided by Artem includes two fles:
 - `bank-additional-full.csv` with all examples, ordered by date (from May 2008 to November 2010).
 - `dataset_info.txt` which answers most of the questions relevant to descriptive analysis of the dataset.

 ---
***The binary classification goal, as specified by the instructor, is to predict if the client will subscribe a bank term deposit***

## 2 Extracting the dataset

```python
# In case it needs to be downloaded during notebook execution

import urllib.request

!rm -rf *.rar
download_URL = "https://www.filepicker.io/api/file/rAA6HuWARrOrIGOVzpiE"
urllib.request.urlretrieve(download_URL, "datasets.rar")
```

    ('datasets.rar', <http.client.HTTPMessage at 0x7fcee70f5190>)

```python
!ls
```

    datasets.rar  Predictive_Behavior_Modelling.ipynb  start_server.sh

```python
# `patool` can extract WINRAR archives
# you may need to install `unrar` executable for patool.extract_archive to work in this cell
# command for apt: `sudo apt-get install unrar`
#!pip install patool

import patoolib

# in case the archive has already been extracted from a previous iteration
!rm -rf dataset

patoolib.extract_archive("./datasets.rar", outdir = "./")
clear_output()
```

```python
!ls dataset
```

    bank_full.csv  dataset_info.txt

## 3 EDA

The sole objective for performing this step is to obtain a parsimonious set of explanatory variables

```python
# you may need to install swifter as `pip install swifter`

import pandas as pd
from swifter import swifter
import datetime
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

clear_output()
```

```python
df = pd.read_csv("./dataset/bank_full.csv", sep = ";")
df.head()
```

      age
      job
      marital
      education
      default
      housing
      loan
      contact
      month
      day_of_week
      ...
      campaign
      pdays
      previous
      poutcome
      emp.var.rate
      cons.price.idx
      cons.conf.idx
      euribor3m
      nr.employed
      y

      0
      56
      housemaid
      married
      basic.4y
      no
      no
      no
      telephone
      may
      mon
      ...
      1
      999
      0
      nonexistent
      1.1
      93.994
      -36.4
      4.857
      5191.0
      no

      1
      57
      services
      married
      high.school
      unknown
      no
      no
      telephone
      may
      mon
      ...
      1
      999
      0
      nonexistent
      1.1
      93.994
      -36.4
      4.857
      5191.0
      yes

      2
      37
      services
      married
      high.school
      no
      yes
      no
      telephone
      may
      mon
      ...
      1
      999
      0
      nonexistent
      1.1
      93.994
      -36.4
      4.857
      5191.0
      no

      3
      40
      admin.
      married
      basic.6y
      no
      no
      no
      telephone
      may
      mon
      ...
      1
      999
      0
      nonexistent
      1.1
      93.994
      -36.4
      4.857
      5191.0
      no

      4
      56
      services
      married
      high.school
      no
      no
      yes
      telephone
      may
      mon
      ...
      1
      999
      0
      nonexistent
      1.1
      93.994
      -36.4
      4.857
      5191.0
      no

5 rows × 21 columns

```python
df.columns
```

    Index(['age', 'job', 'marital', 'education', 'default', 'housing', 'loan',
           'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays',
           'previous', 'poutcome', 'emp.var.rate', 'cons.price.idx',
           'cons.conf.idx', 'euribor3m', 'nr.employed', 'y'],
          dtype='object')

### 3.1 Understanding variable dependencies

`dataset_info.txt` provides descriptions for the columns in `df`:

***Client information***

- age
- job : type of job
- marital : marital status
- education
- default: has credit in default?
- housing: has housing loan?
- loan: has personal loan?

***Related with the last contact of the current campaign***

- contact: contact communication type
- month: last contact month of year
- day_of_week
- duration: last contact duration, in seconds (numeric).

(Note:  the attribute `duration` highly affects the output target (e.g., if duration=0 then y="no"). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model)

***Other attributes***

- campaign: number of contacts performed during this campaign and for this client
- pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
- previous: number of contacts performed before this campaign and for this client (numeric)
- poutcome: outcome of the previous marketing campaign

***Social and economic context attributes***

- emp.var.rate: employment variation rate - quarterly indicator (numeric)
- cons.price.idx: consumer price index - monthly indicator (numeric)
- cons.conf.idx: consumer confidence index - monthly indicator (numeric)
- euribor3m: euribor 3 month rate - daily indicator (numeric)
- nr.employed: number of employees - quarterly indicator (numeric)

```python
df.dtypes
```

    age                 int64
    job                object
    marital            object
    education          object
    default            object
    housing            object
    loan               object
    contact            object
    month              object
    day_of_week        object
    duration            int64
    campaign            int64
    pdays               int64
    previous            int64
    poutcome           object
    emp.var.rate      float64
    cons.price.idx    float64
    cons.conf.idx     float64
    euribor3m         float64
    nr.employed       float64
    y                  object
    dtype: object

Might as well seperate the variables based on whether they are categorical or not. I intend to one-hot encode them using dummy variables

```python
categorical_variables = []
numeric_variables = []

for variable, dtype in zip(df.columns, df.dtypes):
    if dtype == "object":
        if variable != "y":
            # the target response is not a part of the analysis
            categorical_variables.append(variable)
    else:
        numeric_variables.append(variable)
```

```python
print(categorical_variables)
print(numeric_variables)
```

    ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'poutcome']
    ['age', 'duration', 'campaign', 'pdays', 'previous', 'emp.var.rate', 'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed']

### 3.2 Explanability heatmaps

- $R$ = coefficient of correlation between two variables $x_i$ and $x_j$
- $R^2$ = strength of correlation between $x_i$ and $x_j$
- $100.R^2$ = how much of target variable $x_i$ is explained by $x_j$ expressed as a percentage

#### 3.2.1 Amongst numeric variables

```python
correlation_matrix = df[numeric_variables].corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix, xticklabels = numeric_variables, yticklabels = numeric_variables)
plt.show()
```

```python
upper = explanability_matrix.where(np.triu(np.ones(explanability_matrix.shape), k = 1).astype(np.bool))
to_drop = [column for column in upper.columns if any(upper[column] > 75)]
to_drop
```

    ['euribor3m', 'nr.employed']

***Observations from the above heatmap between numeric variables***

- Age, Previous duration of campaign contact, and the number of campaigns when a client was contacted do not explain any variance of the other variables
- The number of employees a client has had quaterly strongly correlates with and explains a considerable amount of variability for quaterly employment variability rate, euribor 3 month rate and the consumer price index
- Quarterly Consumer Confidence doesn't correlate with any other variable.

***Aiming for the parsimonious model based on numeric variables only***

A parsimonious model is essentially a result of Occam's Razor. The principle is to obtain a model that has the lowest number of predictor variables with the maximum predictive power for the response variable. Having analyzed the heatmap, I decided to remove the following variables because they are explainable by more than `75%` by other variables.

- `euribor3m`
- `nr.employed`

#### 3.2.2 numeric variables ~ client job role

```python
temp = pd.get_dummies(data = df.drop("y", axis = 1, inplace = False), columns = ["job"])
correlation_matrix = temp.corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix)
plt.show()
```

***Observations from the above heatmap between numeric variables and the job kind the client has***

- There's none. No variable can be explained by the specific kind of job a client has.

#### 3.2.3 numeric variables ~ client's maritial status

```python
temp = pd.get_dummies(data = df.drop("y", axis = 1, inplace = False), columns = ["marital"])
correlation_matrix = temp.corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix)
plt.show()
```

***Observations from the above heatmap between numeric variables and the the client's martial status***

- There's none. No variable can be explained by the client's maritial status

#### 3.2.4 numeric variables ~ client's education qualification

```python
temp = pd.get_dummies(data = df.drop("y", axis = 1, inplace = False), columns = ["education"])
correlation_matrix = temp.corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix)
plt.show()
```

***Observations from the above heatmap between numeric variables and the the client's education qualifiation***

- There's none. No variable can be explained by the client's education qualification

#### 3.2.5 numeric variables ~ client's credit card's default status

```python
temp = pd.get_dummies(data = df.drop("y", axis = 1, inplace = False), columns = ["default"])
correlation_matrix = temp.corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix)
plt.show()
```

```python
upper = explanability_matrix.where(np.triu(np.ones(explanability_matrix.shape), k = 1).astype(np.bool))
to_drop = [column for column in upper.columns if any(upper[column] > 75)]
to_drop
```

    ['euribor3m', 'nr.employed', 'default_unknown']

***Observations from the above heatmap between numeric variables and the the client's credit card default status***

- There's none. No variable can be explained by the client's credit card default status
- `default_unknown` is strongly correlated with `default_no`.

***Aiming for the parsimonious model***

I decided to remove the following variable because they are explainable by more than 75% by other variables.

- `default_unknown`

#### 3.2.6 numeric variables ~ the default status of the client's loan statuses

```python
temp = pd.get_dummies(data = df.drop("y", axis = 1, inplace = False), columns = ["housing", "loan"])
correlation_matrix = temp.corr()
explanability_matrix = correlation_matrix.swifter.apply(lambda x: x ** 2) * 100

plt.rcParams['figure.figsize'] = 14, 6
plt.tight_layout()

sns.heatmap(explanability_matrix)
plt.show()
```

```python
upper = explanability_matrix.where(np.triu(np.ones(explanability_matrix.shape), k = 1).astype(np.bool))
to_drop = [column for column in upper.columns if any(upper[column] > 75)]
to_drop
```

    ['euribor3m', 'nr.employed', 'housing_yes', 'loan_unknown', 'loan_yes']

***Observations from the above heatmap between numeric variables and the the client's loan statuses***

- If we are uncertain about the client's housing loan status, so are we about his/her personal loan status and vice versa

***Aiming for the parsimonious model***

I decided to remove the following variable because they are explainable by more than 75% by other variables.

- `housing_yes`
- `loan_unknown`
- `loan_yes`

### 3.3 Removing dependencies

```python
df_encoded = pd.get_dummies(data = df, columns = categorical_variables)
df_encoded.drop(to_drop, axis = 1, inplace = True)
df_encoded.head()
```

      age
      duration
      campaign
      pdays
      previous
      emp.var.rate
      cons.price.idx
      cons.conf.idx
      y
      job_admin.
      ...
      month_oct
      month_sep
      day_of_week_fri
      day_of_week_mon
      day_of_week_thu
      day_of_week_tue
      day_of_week_wed
      poutcome_failure
      poutcome_nonexistent
      poutcome_success

      0
      56
      261
      1
      999
      0
      1.1
      93.994
      -36.4
      no
      0
      ...
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0

      1
      57
      149
      1
      999
      0
      1.1
      93.994
      -36.4
      yes
      0
      ...
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0

      2
      37
      226
      1
      999
      0
      1.1
      93.994
      -36.4
      no
      0
      ...
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0

      3
      40
      151
      1
      999
      0
      1.1
      93.994
      -36.4
      no
      1
      ...
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0

      4
      56
      307
      1
      999
      0
      1.1
      93.994
      -36.4
      no
      0
      ...
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0

5 rows × 59 columns

```python
df_encoded.columns
```

    Index(['age', 'duration', 'campaign', 'pdays', 'previous', 'emp.var.rate',
           'cons.price.idx', 'cons.conf.idx', 'y', 'job_admin.', 'job_blue-collar',
           'job_entrepreneur', 'job_housemaid', 'job_management', 'job_retired',
           'job_self-employed', 'job_services', 'job_student', 'job_technician',
           'job_unemployed', 'job_unknown', 'marital_divorced', 'marital_married',
           'marital_single', 'marital_unknown', 'education_basic.4y',
           'education_basic.6y', 'education_basic.9y', 'education_high.school',
           'education_illiterate', 'education_professional.course',
           'education_university.degree', 'education_unknown', 'default_no',
           'default_unknown', 'default_yes', 'housing_no', 'housing_unknown',
           'loan_no', 'contact_cellular', 'contact_telephone', 'month_apr',
           'month_aug', 'month_dec', 'month_jul', 'month_jun', 'month_mar',
           'month_may', 'month_nov', 'month_oct', 'month_sep', 'day_of_week_fri',
           'day_of_week_mon', 'day_of_week_thu', 'day_of_week_tue',
           'day_of_week_wed', 'poutcome_failure', 'poutcome_nonexistent',
           'poutcome_success'],
          dtype='object')

Ought to remove the target variable $y$ because I don't want to be a part of the dataset but an independent vector

```python
df_encoded.shape
```

    (41188, 59)

```python
from sklearn.preprocessing import LabelEncoder

y_label_encoder = LabelEncoder()
y_label_encoder.fit(df_encoded["y"])
df_encoded["y"] = y_label_encoder.transform(df_encoded["y"])

df_encoded["y"].value_counts()
```

    0    36547
    1     4641
    Name: y, dtype: int64

```python
y = df_encoded["y"].copy(deep = True)
df_encoded.drop("y", axis = 1, inplace = True)
```

```python
y.shape, df_encoded.shape
```

    ((41188,), (41188, 58))

## 4 Preprocessing the dataset for training, testing and validation

### 4.1 Addressing class imbalance

The dataset has an instristic fault of class imbalance.
This is understandable: People don't subscribe to deposit offers usually.

The idea is essentially to understand the hidden similarities of people who actually do, and then see if a new customer possess the same similarities.

---

(__Note__)

I highly recommend reading up an introductory tutorial on [Factor Analysis](https://www.datacamp.com/community/tutorials/introduction-factor-analysis) to find latent(hidden) variables also called "factors" which govern the effect of certain groups of features in `df_encoded` on the response variable `y`. Let me define the concept mathematically:

`df_encoded` has`58` explanatory variables for `y`. We can write this as a statistical model where the explanatory variables can be modelled to predict `y`:

$$\begin{aligned} y \sim M({x_1, x_2, x_3, ..., x_58}) \end{aligned}$$

where $M$ is the statistical model function. Now, there may exist another statistical model $\tilde{F}$ which can also model `y` as:

$$\begin{aligned} y \sim \tilde{F}({f_1, f_2, f_3, ..., f_n}) \end{aligned}$$

where $f_1, f_2, ..., f_n$ are the latent unobserved variables "working" behind certain "groups" of features from the 58 in `df_encoded`. Consider the following example image from the tutorial linked above:

The variables on the left are features we may have in our working dataset, while the 2 on the right are the actual albeit hidden "factors" underlying some groups of features. Of course, the number of factors that are available in a dataset should be less than or equal to the total number of features. It will be equal when all the features are perfectly non-collinear and have no correlation or relationship of any kind between them.

__This project doesn't have any implementation of factor analysis because this is essentially a practice homework to implement a classification model using `tensorflow`.__ Doing a factor analysis can (it will, for `df_encoded`) reduce the features we have. Then, it won't make any sense to use `tensorflow` for the transformed smaller dataset of factors: A simple ML algorithm like Random Forests or Support Vector Machines can do the job instead.

---

```python
import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = 8, 5
plt.tight_layout()

positive_examples = y.sum()
negative_examples = len(y) - positive_examples
plt.barh(["ACCEPTED DEPOSIT", "REJECTED DEPOSIT"], [positive_examples, negative_examples], color = "k")
plt.show()
```

#### 4.1.1 Attempt at resolving this: Downsampling the majority class

- let number of examples for minority class = `n`
- randomly choose `n` examples from the majority class
- create the new working dataset from these two set of examples

Note:  _An alternative to this approach is using a weighted approach. Negative responses will be assigned a lower weight while the positive responses will be assigned a higher weight to counter their effect during training and on performance metrics_

```python
from imblearn.under_sampling import RandomUnderSampler

sampler = RandomUnderSampler(random_state = 1)
X, Y = sampler.fit_resample(df_encoded, y)
```

```python
print(X.shape, Y.shape)
print(type(X), type(Y))
```

    (9282, 58) (9282,)
    <class 'numpy.ndarray'> <class 'numpy.ndarray'>

```python
positive_examples = np.sum(Y)
negative_examples = np.sum(~np.array(Y, dtype = bool))
plt.barh(["ACCEPTED DEPOSIT", "REJECTED DEPOSIT"], [positive_examples, negative_examples], color = "k")
plt.show()
```

### 4.2 Train-Validate-Test split

```python
from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size = 20/100, random_state = 1)
x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size = 8/20, random_state = 1)
```

```python
y_train.shape, y_val.shape, y_test.shape
```

    ((7425,), (1114,), (743,))

```python
x_train.shape, x_val.shape, x_test.shape
```

    ((7425, 58), (1114, 58), (743, 58))

### 4.3 Min-Max Feature Scaling

I will perform min-max feature scaling for every numeric variable in the three subsets

```python
from sklearn.preprocessing import MinMaxScaler

train_scaler = MinMaxScaler()
x_train = train_scaler.fit_transform(x_train)

val_scaler = MinMaxScaler()
x_val = val_scaler.fit_transform(x_val)

test_scaler = MinMaxScaler()
x_test = val_scaler.fit_transform(x_test)
```

## 5 Dumping records in `TFRecord` files

### 5.1 Initializing a dictionary of necessary hyperparameters

```python
FLAGS = dict(tensorboard_data_dump = "./tensorboard_data_dump/",
             number_of_features = x_train.shape[1],
             batch_size = 64,
             validation_batch_size = x_val.shape[0],
             test_batch_size = x_test.shape[0],
             num_parallel_calls = 3,
             shuffle_buffer = 64,
             prefetch_buffer = 3,
             epochs = 100,
             learning_rate = 1e-4)
```

```python
FLAGS
```

    {'tensorboard_data_dump': './tensorboard_data_dump/',
     'number_of_features': 58,
     'batch_size': 64,
     'validation_batch_size': 1114,
     'test_batch_size': 743,
     'num_parallel_calls': 3,
     'shuffle_buffer': 64,
     'prefetch_buffer': 3,
     'epochs': 100,
     'learning_rate': 0.0001}

### 5.2 Creating tfRecord dump directories

```python
!mkdir -p "train_tfrecords"
!mkdir -p "test_tfrecords"
!mkdir -p "val_tfrecords"
```

```python
train_dump = "./train_tfrecords/"
test_dump = "./test_tfrecords/"
val_dump = "./val_tfrecords/"
```

### Defining a function to dump a batched subset of a dataset

- `x_subset` = subset (sliced) `numpy.ndarray`
 - _`x_subset` is expected to be a matrix in the context of this project_

- `y_subset` = subset (sliced) `numpy.ndarray`
 - _`y_subset` is expected to be a vector with the same number of rows as that of `x_subset`_

- `writer` =  An instance of `tf.python_io.TFRecordWriter` which has been initialized with a dump directory path.

***How does the dumping work?***

- Initialize a writer object `writer`
- Convert all your dump data to lists (because the dumper only works with lists)
 - Since our numpy array is essentially a form of nested list, I need to extract each row(sub-list) from the nested list and then pass it to `tf.train.Feature`
- Create a `tf.train.Feature` object out of each wrapped-up data variable seperately. Use the appropiate argument name: `float_list`, `int32_list`, `int64_list`
- Create a native `dict` out of the `tf.train.Feature` instances by associating a key with each of them. This `dict` will now serve as a schema of storage and retrieval for each feature-label pair.
- Wrap the `dict` in a `tf.train.Example`. In essence, each row in the dataset is now a single `Example` object to be fed into a Tensorflow Data-Flow Graph.

***Why do we do this?***

- Need for speed
- Setting a highly-parallelized, optimized and efficient pipeline for flow of data

```python
def dump_into_tf_record_file(x, y, writer):
    data = x.tolist()
    label = y
    data = tf.train.Feature(float_list = tf.train.FloatList(value = data))
    label = tf.train.Feature(float_list = tf.train.FloatList(value = [label]))
    feature_dict = {"features":data, "label":label}
    feature_dict = tf.train.Features(feature = feature_dict)
    example = tf.train.Example(features = feature_dict)
    writer.write(example.SerializeToString())
```

### 5.3 Defining a function to dump a dataset as tfRecord files in a specified directory

- `dataset` = the `numpy.ndarray` of feature matrix to be dumped
- `labels` = the `numpy.ndarray` vector of output/target/response variable for `dataset`
- `dump_directory` = path to the local directory where the tfRecord files will be dumped
- `batch_size` = how many rows of `dataset` and `labels` will be there in each tfRecord file

The function will accept a dataset matrix and a label vector, divide it up by the batch size specified and then write each row of the dataset matrix and single label value associated with it in a `tfrecord` dump file.

```python
def dump_dataset_into_tf_records(dataset, labels, dump_directory, batch_size):
    m = dataset.shape[0]
    i = 0
    j = 0
    writer = None
    while(i < m):
        if i % batch_size == 0:
            if writer is not None:
                writer.close()
            j = j + 1
            record_file_name = dump_directory  + "Batch_" + str(j) + ".tfrecord"
            writer = tf.python_io.TFRecordWriter(record_file_name)
        x = dataset[i, :]
        y = labels[i]
        dump_into_tf_record_file(x, y, writer)
        i = i + 1
```

### 5.4 Dump `x_train` and `y_train` at `./train_tfrecords`

```python
!rm -rf $train_dump/*
dump_dataset_into_tf_records(x_train, y_train, train_dump, 500)
```

### 5.5 Dump `x_val` and `y_val` at `./val_tfrecords`

```python
!rm -rf $val_dump/*
dump_dataset_into_tf_records(x_val, y_val, val_dump, 500)
```

### 5.6 Dump `x_test` and `y_test` at `./test_tfrecords`

```python
!rm -rf $test_dump/*
dump_dataset_into_tf_records(x_test, y_test, test_dump, 500)
```

## 5.7 Creating Input Pipeline using tfRecords

### 5.7.1 Defining a function to retrive the tfRecord files from a directory in an efficient sequential manner

- Create an object of `tf.data.Dataset` by passing a list of the `tfrecord` files in the specified dump directory path
- Convert each `tf.train.Example` object as it is read from the `tfrecord` files back into the `feature_dict` format we specified before: `{"features":data, "label":label}`. This is done using the `tf.data.Dataset.map` method.
- To enhance the convertion from `Example` object to the `feature_dict` format, I specified an additional argument of `num_parallel_calls` = 3 (as specified previously in `FLAGS["num_parallel_calls"]`). Each conversion is independent of each other and hence this will allow to parallely convert 3 `Example` objects to `feature_dict` format.
- I also chose to shuffle the dataset during the retrieval pipeline and perform an indefinite random repeat of the data. This makes sure my model has no problem of "finishing" up the data flow.
- To further enhance the performance of the training process, I used `tf.train.Dataset.prefetch` method to set how many `tf.train.Example` objects will be fetched before it's turn has arrived to be fed into the Data-Flow graph

(__Note__)

This method (or more technically speaking, the pipeline construction on `tf.data.Dataset`) is most prone to bugs during implementation. This method defines how the `tensorflow` graph will receive data and how it will be trained (epochs and mini-batches per epoch). I spent 2 weeks trying to debug various errors like `OutOfRange`, `labels are []`, `NextIterator:0 End of sequence`. Here are some precautionary steps I took to debug them all, with the help of my instructor Artem:

1. `shuffle` must come before `batch`
2. Based on how many epochs and how many mini-batches you will have during training, please set your `batch_size` and `prefetch_buffer` accordingly
3. You would want your validation data to be the whole thing as 1 batch only. Then, don't perform `prefetch`.
4. The same (3) should follow for your testing data.
5. If you want to perform batched validation and testing on your model, set up your validation and testing pipeline accordingly.

```python
from os import listdir

def get_dataset(directory,
                type_of_dataset,
                batch_size,
                shuffle_size = FLAGS["shuffle_buffer"],
                buffer_size = FLAGS["prefetch_buffer"]):

    record_file_names = [directory + file_name for file_name in listdir(directory)]
    dataset = tf.data.TFRecordDataset(record_file_names)
    dataset = dataset.map(parse_record, num_parallel_calls = FLAGS["num_parallel_calls"])

    # error: data shape becomes [1, 1] instead of [FLAGS["batch_size"], 1]
    # solution offered by Artem on 26 September 2019 1427 hours IST: order of op: shuffle, batch
    if type_of_dataset.upper() == "TRAINING":
        dataset = dataset.shuffle(buffer_size = shuffle_size)

    dataset = dataset.repeat()
    dataset = dataset.batch(batch_size)

    # error: validation and test data are 1 batch only so no need to prefect for them
    # solution found by myself on 27 September 2019 Late Midnight IST
    if type_of_dataset.upper() == "TRAINING":
        dataset = dataset.prefetch(buffer_size = buffer_size)
    else:
        dataset = dataset.prefetch(buffer_size = 1)
    return(dataset)
```

### 5.7.2 Defining a function to parse each `tf.train.Example` from the tfRecord files

```python
def parse_record(serialized_record):
    features_dict = {"features":tf.FixedLenFeature([FLAGS["number_of_features"]], tf.float32),
                     "label":tf.FixedLenFeature([1], tf.float32)}
    parsed = tf.parse_single_example(serialized = serialized_record, features = features_dict)

    data = parsed["features"]
    label = parsed["label"]

    data = tf.cast(data, tf.float32)
    label = tf.cast(label, tf.int32)

    return((data, label))
```

## 6 Designing the DataFlow Graph and executing a session

### 6.1 Logical structure of the Data Flow Graph as an user-defined Python Class object

***Base Model Class***

- this class will define the weight matrices, the bias vectors and their corresponding initializers
- this class will most importantly define the feed-forward architecture of the fully-connected neural network
- I took inspiration from François Chollet's `keras` framework which serves as a high-level wrapper API for Tensorflow (as well as other backends) and decided to write this class as a wrapper for creating custom Data Flow Graphs in Tensorflow

- `FCNN` will be my base model class name
- the class will be initalizer with defining the weight and bias tensor initializers
- the class will have a method `add_weight_matrix` which will add a weight matrix to the FCNN. The order in which the method is called to add the weight matrix determines the hidden layer to which the weight matrix will be associated.
- similarly, the class have a method `add_bias_vector`
- the most important method of the class will be `forward_propagation` which creates the Data Flow graph using the defined weight and bias tensors and returns the graph generated till the last hidden layer and the classification output generated

***Justification for variable names used in `FCNN.forward_propagation`***

For $X =$ `x_subset` (a batch of features),
- linear model $z = X.W^T + B$
- actual hypothesis equation $h = g(z)$ where $g$ is a non-linear function. Some of the [desirable properties](https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions) of $g$ are:
    - Non-linearity (this is what makes Neural Networks _universal function approximators_)
    - Continuously differentiable
    - Monotonic
    - Smooth with monotonic derivative
    - Identity approximation near Origin
- For $g$ = ReLU,
$\begin{equation}
g(z) = \begin{cases}
            0, & \text{if}\ z \leq 0 \\
            z, & \text{if}\ z > 0
        \end{cases}
\end{equation}$
- $\hat{y}$ (pronounced `y-hat` or `y-cap`) $= g'(h)$ where $g'$ is a special activation function which gives the final desired output in a set domain.
    - For classification problems,
        - If it is a binary classification problem, $g'$ can be $\sigma$ where $\sigma(z) = \frac{1}{1 + e^{-z}}$ (most preferred)
        - If it is a multi-class classification problem where each of the classes are mutually exclusive, $g'$ can be the softmax function where $g'(z) = \frac{e^z_i}{\sum\limits_{j=1}^J {e^{z_j}}}$ where $J = $ number of output classes the solution requires
    - For regression problems, a linear function with the desirable output domain will do the job. ( _provided the input has been normalized_ )

`forward_propagation` returns $h$ and $\hat{y}$ which are also called `logits` and `prediction` in `tensorflow` vocabulary.

```python
class FCNN:

    def __init__(self,
                 weight_initializer = tf.random_normal_initializer(mean = 0.0, stddev = 0.25),
                 bias_initializer = tf.zeros_initializer()):
        self.weight_initializer = weight_initializer
        self.bias_initializer = bias_initializer
        self.weights = []
        self.biases = []

    def add_weight_matrix(self,
                          shape,
                          scope_name = "WEIGHTS",
                          dtype = tf.float32):

        with tf.name_scope(scope_name):
            i = len(self.weights)
            self.weights.append(tf.get_variable(name = "W" + str(i),
                                                shape = shape,
                                                dtype = dtype,
                                                initializer = self.weight_initializer))

    def add_bias_vector(self,
                        shape,
                        scope_name = "BIASES",
                        dtype = tf.float32):

        with tf.name_scope(scope_name):
            i = len(self.biases)
            self.biases.append(tf.get_variable(name = "B" + str(i),
                                               shape = shape,
                                               dtype = dtype,
                                               initializer = self.bias_initializer))

    def forward_propagation(self,
                            x_subset,
                            output_activation_function = tf.nn.softmax,
                            scope_name = "FEED_FORWARD_PROPAGATION"):

        with tf.name_scope(scope_name):
            x = x_subset
            for i in range(len(self.biases)):
                z = tf.matmul(x, self.weights[i]) + self.biases[i]
                h = tf.nn.relu(z)
                x = h
            h = tf.matmul(x, self.weights[-1], name = "LOGITS")
            yhat = output_activation_function(h, name = "SOFMAX_PREDICTION_FROM_LOGITS")
            return((h, yhat))
```

***Train Model Class***

- this class will be a child of `FCNN`
- this class will contain methods to define the loss function and  compute the loss incured by the Neural Network (FCNN instance)
- the most important method of this child class will be to define the back propagation of loss gradients

```python
class myModel(FCNN):

    def __init__(self, batch_size, learning_rate):
        self.batch_size = batch_size
        self.learning_rate = learning_rate
        super(myModel, self).__init__()

    def compute_loss_incured(self,
                             y_subset,
                             h,
                             loss_function = tf.nn.sparse_softmax_cross_entropy_with_logits,
                             scope_name = "LOSS",
                             add_penalty = True):

        with tf.name_scope(scope_name):

            # because dimensions of y_subset are (FLAGS["batch_size"], 1)
            labels = tf.squeeze(y_subset)

            cross_entropy = loss_function(labels = labels,
                                          logits = h,
                                          name = "CROSS_ENTROPY_LOSS")
            loss = tf.reduce_mean(cross_entropy, name = "AVERAGE_CROSS_ENTROPY_LOSS")

            if add_penalty:
                l2_loss = tf.add_n([tf.nn.l2_loss(T) for T in tf.trainable_variables()])
                loss = loss + self.learning_rate * l2_loss

        return(loss)

    def compute_naive_accuracy(self,
                               x_subset,
                               y_subset,
                               scope_name = "NAIVE_ACCURACY"):

        with tf.name_scope(scope_name):
            _, yhat = self.forward_propagation(x_subset)

            # because dimensions of y_subset are (FLAGS["batch_size"], 1)
            labels = tf.squeeze(y_subset)

            # because when tf.argmax(axis = 1) will return column IDs as integers
            labels = tf.cast(labels, tf.int64)

            '''
            yhat is a matrix of dimensions (FLAGS["batch_size"], number of classes = 2).
            labels will be a vector of length = FLAGS["batch_size]

            Using tf.argmax on yhat along the row axis(axis = 1), we find the column which has the
            maximum softmax probability, For the first column, ID will be 0. For the second, it will
            be 1. Incidentally, labels are also binary.

            Hence, class_ID_predicted should have dtype = an int type.

            Possible question: Are you sure the predictions mean what they should, after
            applying tf.argmax?
            '''
            class_ID_predicted = tf.argmax(yhat, axis = 1)

            # row-wise comparision of equality like xnor
            is_each_predicted_class_ID_correct = tf.equal(class_ID_predicted, labels)

            # to compute the accuracy using tf.reduce_mean, the values now need to be in a float type
            is_each_predicted_class_ID_correct = tf.cast(is_each_predicted_class_ID_correct, tf.float32)

            # add the values in is_each_predicted_class_ID_correct and divide by the length of the same
            accuracy = tf.reduce_mean(is_each_predicted_class_ID_correct)

        return(accuracy)

    def backward_propagation(self,
                             loss,
                             scope_name = "BACKWARD_PROPAGATION"):

        with tf.name_scope(scope_name):
            trainable_variables = tf.trainable_variables()
            gradients = tf.gradients(loss, trainable_variables, name = "dL")
            optimizer = tf.train.AdamOptimizer(learning_rate = self.learning_rate, name = "Min_dL")

            '''
            Exact Live Documentation on how apply_gradients can be found at this link:
            https://github.com/tensorflow/tensorflow/blob/r1.14/tensorflow/python/training/optimizer.py#L531-L637

            '''
            update_operation = optimizer.apply_gradients(grads_and_vars = zip(gradients, trainable_variables),
                                                         name = "dL_update")

        return(update_operation)
```

***Performance Evaluation Metrics***

- Recall
- Precision
- F1 score
- AUC

```python
from sklearn.metrics import confusion_matrix, roc_auc_score, classification_report

def evaluate_model(y_subset, yhat, pos_label_scores, interpretations_of_labels = None, complete = True):
    if complete:
        print(classification_report(y_true = y_subset,
                                    y_pred = yhat,
                                    target_names = interpretations_of_labels,
                                    digits = 2))
    else:
        tn, fp, fn, tp = confusion_matrix(y_subset, yhat).ravel()

        precision = tp / (tp + fp)
        recall = tp / (tp + fn)
        f1 = (2 * precision * recall) / (precision + recall)
        auc = roc_auc_score(y_subset, pos_label_scores)

        print('Precision: %0.02f%% '% (precision * 100))
        print('Recall: %0.02f%% '% (recall * 100))
        print('F1 score: %0.02f%% '% (f1 * 100))
        print('AUC: %0.02f%%' % (auc * 100))
```

### 6.2 Defining Session Protocols

```python
config = tf.ConfigProto(log_device_placement = True)
config.gpu_options.allow_growth = False
```

### 6.3 Create  the DataFlow Graph

```python
G = tf.Graph()

with G.as_default():

    with tf.name_scope("INPUT_PIPELINE"):

        train = get_dataset(train_dump,
                            batch_size = FLAGS["batch_size"],
                            type_of_dataset = "TRAINING")
        validation = get_dataset(val_dump,
                                 batch_size = FLAGS["validation_batch_size"],
                                 type_of_dataset = "VALIDATION")
        test = get_dataset(test_dump,
                           batch_size = FLAGS["test_batch_size"],
                           type_of_dataset = "TESTING")

        train_iterator = train.make_initializable_iterator()
        test_iterator = test.make_initializable_iterator()
        val_iterator = validation.make_initializable_iterator()

        x_train_subset, y_train_subset = train_iterator.get_next()
        x_val_subset, y_val_subset = val_iterator.get_next()
        x_test_subset, y_test_subset = test_iterator.get_next()

        '''
        We need to have a copy of x_train_subset and y_train_subset before moving to create the dataflow
        graph. Since they are iterators, having performed one forward pass, the variables will be empty
        by the time we need to compute the training accuracy and loss.
        '''
        x_train_subset_copy, y_train_subset_copy = tf.identity(x_train_subset), tf.identity(y_train_subset)

    model = myModel(FLAGS["batch_size"], FLAGS["learning_rate"])

    # H1
    model.add_weight_matrix(shape = [FLAGS["number_of_features"], 128])
    model.add_bias_vector(shape = [128])

    # H2
    model.add_weight_matrix(shape = [128, 256])
    model.add_bias_vector(shape = [256])

    # H3
    model.add_weight_matrix(shape = [256, 128])
    model.add_bias_vector(shape = [128])

    # H4
    model.add_weight_matrix(shape = [128, 32])
    model.add_bias_vector(shape = [32])

    # H5
    model.add_weight_matrix(shape = [32, 2])

    # perform one forward pass
    h, _ = model.forward_propagation(x_subset = x_train_subset)

    # compute loss after one forward pass
    loss = model.compute_loss_incured(y_subset = y_train_subset,
                                      h = h)
    loss_summary = tf.summary.scalar(name = "LOSS", tensor = loss)

    # based on loss incured, perform backprop
    update_operation = model.backward_propagation(loss = loss)

    # peri-training metrics
    train_accuracy = model.compute_naive_accuracy(x_subset = x_train_subset_copy, y_subset = y_train_subset_copy)
    train_accuracy_summary = tf.summary.scalar(name = "TRAIN_ACCURACY", tensor = train_accuracy)
    validation_accuracy = model.compute_naive_accuracy(x_subset = x_val_subset, y_subset = y_val_subset)
    validation_accuracy_summary = tf.summary.scalar(name = "VALIDATION_ACCURACY", tensor = validation_accuracy)

    # for testing
    _, predictions = model.forward_propagation(x_subset = x_test_subset)

    # collecting all the summary objects during training in one operation
    merged_summaries = tf.summary.merge([loss_summary, train_accuracy_summary])

    # node to dump the model
    saver = tf.train.Saver()

#clear up any deprecation warnings if there
clear_output()

number_of_batches = x_train.shape[0]//FLAGS["batch_size"]

print("Training Batch Size = %i, so number of mini-batches for each epoch = %i" % (FLAGS["batch_size"], number_of_batches))

print("After forward propagation, shape of h (logits) = ", h.shape)

print("Shape of INPUT FEATURES (train, validation, test) = ",
      x_train_subset.shape,
      x_val_subset.shape,
      x_test_subset.shape)

print("Shape of GROUND TRUTH LABEL (train, validation, test) = ",
      y_train_subset.shape,
      y_val_subset.shape,
      y_test_subset.shape)
```

    Training Batch Size = 64, so number of mini-batches for each epoch = 116
    After forward propagation, shape of h (logits) =  (?, 2)
    Shape of INPUT FEATURES (train, validation, test) =  (?, 58) (?, 58) (?, 58)
    Shape of GROUND TRUTH LABEL (train, validation, test) =  (?, 1) (?, 1) (?, 1)

### 6.4 Visualize the DataFlow Graph Inline

[Source code from Tensorflow Authors](https://nbviewer.jupyter.org/github/tensorflow/examples/blob/master/community/en/r1/deepdream.ipynb#Loading-and-displaying-the-model-graph)

```python
from IPython.display import Image, display, HTML

def strip_consts(graph_def, max_const_size = 32):
    """Strip large constant values from graph_def."""
    strip_def = tf.GraphDef()
    for n0 in graph_def.node:
        n = strip_def.node.add()
        n.MergeFrom(n0)
        if n.op == 'Const':
            tensor = n.attr['value'].tensor
            size = len(tensor.tensor_content)
            if size > max_const_size:
                tensor.tensor_content = "<stripped %d bytes>"%size
    return strip_def

def show_graph(graph_def, max_const_size = 32):
    """Visualize TensorFlow graph."""
    if hasattr(graph_def, 'as_graph_def'):
        graph_def = graph_def.as_graph_def()
    strip_def = strip_consts(graph_def, max_const_size=max_const_size)
    code = """

        <link rel="import" href="https://tensorboard.appspot.com/tf-graph-basic.build.html" onload=load()>
        <div style="height:600px">
          <tf-graph-basic id="{id}"></tf-graph-basic>
        </div>
    """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))

    iframe = """

    """.format(code.replace('"', '"'))
    display(HTML(iframe))
```

```python
show_graph(G)
```

### 6.5 Execute a session on `G`

```python
# if you require you may uncomment this line to load the tensorboard command for Juypyter

#%load_ext tensorboard
```

```python
tf.reset_default_graph()

# `writer` will dump the summary data during training/validation which
# will be read and visualized by tensorboard
temp = FLAGS["tensorboard_data_dump"]
writer = tf.summary.FileWriter(temp)

!rm -rf $temp/*
!mkdir -p "trained_model_dumps"
!rm -rf $trained_model_dumps/*.ckpt

# if you require you may uncomment this line to launch the tensorboard inline
#%tensorboard --logdir {temp}

loss_history = []
train_accuracy_history = []
validation_accuracy_history = []

with tf.Session(graph = G, config = config) as session:

    # initialize all `tf.Variables` declared in the graph
    session.run(tf.global_variables_initializer())

    print(" (Training)\n")
    print(" A - Training Accuracy\n L - Training Loss\n VA = Validation Accuracy\n\n")

    # I wanted to visualize the graph in tensorboard too
    writer.add_graph(G)

    for i in range(FLAGS["epochs"]):

        # you will need to start the iterator so that they are ready to
        # flow-transfer data into the graph
        session.run(train_iterator.initializer)
        session.run(val_iterator.initializer)

        temp_loss = 0
        temp_accuracy = 0

        for j in range(number_of_batches):

            # compute the loss by executing the loss sub-graph
            L = session.run(loss)

            # perform backprop on the whole graph G
            session.run(update_operation)

            # compute the training accuracy now the graph has been trained once
            A = session.run(train_accuracy)

            # obtain a live summary object containing data dumps for the summary objects declared
            # in conjunction with the graph G
            summary = session.run(merged_summaries)

            temp_loss = temp_loss + L
            temp_accuracy = temp_accuracy + A

            # dump the current summary object and mark it with iteration id = k
            # here, k = the iteration number/step -after- i epochs and j mini-batches of training
            k = (i + 1) * FLAGS["epochs"] + (j + 1)
            writer.add_summary(summary, k)

        # compute the validation accuracy by running the validation accuracy sub-graph after one epoch
        temp_validation_accuracy = session.run(validation_accuracy)

        # ought to add the summary collected for the validation accuracy data
        summary = session.run(validation_accuracy_summary)
        writer.add_summary(summary, k)

        # compute the average loss and average accuracy after each epoch
        temp_loss = temp_loss/number_of_batches
        temp_accuracy = temp_accuracy/number_of_batches

        # record loss, train accuracy and validation accuracy so that we plot some graphs
        loss_history.append(temp_loss)
        train_accuracy_history.append(temp_accuracy)
        validation_accuracy_history.append(temp_validation_accuracy)

        print(" Epoch %i:\tA = %0.02f%%\tL = %0.04f\tVA = %0.02f%%" % ((i + 1),
                                                                    temp_accuracy * 100,
                                                                    temp_loss,
                                                                    temp_validation_accuracy * 100))

        # make sure the any pending dumping work on the summary writer object is cleared
        writer.flush()

    # the training is complete, so now we can close the open write stream associated
    writer.close()

    # since we will be exiting our session after training, we ought to save all data associated with the session offline
    model_dump_path = "./trained_model_dumps/myModel_with_VA_" + str(int(temp_validation_accuracy * 100)) + "_percent.ckpt"
    saver.save(session, model_dump_path)
```

     (Training)

     A - Training Accuracy
     L - Training Loss
     VA = Validation Accuracy

     Epoch 1:	A = 58.27%	L = 2.2824	VA = 63.11%
     Epoch 2:	A = 64.57%	L = 1.3755	VA = 65.44%
     Epoch 3:	A = 66.69%	L = 1.1328	VA = 67.06%
     Epoch 4:	A = 69.17%	L = 0.9986	VA = 69.48%
     Epoch 5:	A = 71.19%	L = 0.9291	VA = 70.65%
     Epoch 6:	A = 73.37%	L = 0.8745	VA = 71.27%
     Epoch 7:	A = 74.14%	L = 0.8214	VA = 72.53%
     Epoch 8:	A = 75.51%	L = 0.8030	VA = 72.71%
     Epoch 9:	A = 77.44%	L = 0.7778	VA = 73.25%
     Epoch 10:	A = 77.87%	L = 0.7410	VA = 73.52%
     Epoch 11:	A = 79.51%	L = 0.7105	VA = 74.87%
     Epoch 12:	A = 79.96%	L = 0.7035	VA = 75.22%
     Epoch 13:	A = 81.36%	L = 0.6815	VA = 75.58%
     Epoch 14:	A = 81.26%	L = 0.6560	VA = 76.03%
     Epoch 15:	A = 82.61%	L = 0.6524	VA = 76.30%
     Epoch 16:	A = 83.31%	L = 0.6269	VA = 77.47%
     Epoch 17:	A = 83.28%	L = 0.6258	VA = 76.57%
     Epoch 18:	A = 84.38%	L = 0.5970	VA = 77.65%
     Epoch 19:	A = 84.63%	L = 0.5981	VA = 78.55%
     Epoch 20:	A = 85.64%	L = 0.5731	VA = 77.92%
     Epoch 21:	A = 86.06%	L = 0.5646	VA = 78.90%
     Epoch 22:	A = 86.49%	L = 0.5593	VA = 77.83%
     Epoch 23:	A = 86.68%	L = 0.5595	VA = 79.80%
     Epoch 24:	A = 86.95%	L = 0.5329	VA = 79.44%
     Epoch 25:	A = 87.14%	L = 0.5335	VA = 79.44%
     Epoch 26:	A = 88.15%	L = 0.5375	VA = 79.89%
     Epoch 27:	A = 88.16%	L = 0.5205	VA = 79.44%
     Epoch 28:	A = 88.87%	L = 0.5251	VA = 79.62%
     Epoch 29:	A = 88.13%	L = 0.4996	VA = 80.25%
     Epoch 30:	A = 89.13%	L = 0.5143	VA = 79.89%
     Epoch 31:	A = 88.91%	L = 0.4895	VA = 80.34%
     Epoch 32:	A = 89.82%	L = 0.4850	VA = 79.62%
     Epoch 33:	A = 89.71%	L = 0.4869	VA = 80.70%
     Epoch 34:	A = 89.80%	L = 0.4769	VA = 80.70%
     Epoch 35:	A = 89.48%	L = 0.4651	VA = 80.88%
     Epoch 36:	A = 90.06%	L = 0.4511	VA = 81.33%
     Epoch 37:	A = 90.37%	L = 0.4679	VA = 81.33%
     Epoch 38:	A = 90.50%	L = 0.4554	VA = 81.15%
     Epoch 39:	A = 90.40%	L = 0.4586	VA = 80.97%
     Epoch 40:	A = 90.52%	L = 0.4399	VA = 80.97%
     Epoch 41:	A = 90.91%	L = 0.4410	VA = 80.52%
     Epoch 42:	A = 90.95%	L = 0.4353	VA = 80.88%
     Epoch 43:	A = 91.10%	L = 0.4323	VA = 81.69%
     Epoch 44:	A = 91.84%	L = 0.4168	VA = 81.42%
     Epoch 45:	A = 91.77%	L = 0.4186	VA = 81.24%
     Epoch 46:	A = 91.74%	L = 0.4153	VA = 81.69%
     Epoch 47:	A = 91.70%	L = 0.4050	VA = 81.87%
     Epoch 48:	A = 92.08%	L = 0.4123	VA = 81.15%
     Epoch 49:	A = 92.43%	L = 0.4016	VA = 81.51%
     Epoch 50:	A = 92.44%	L = 0.4062	VA = 80.61%
     Epoch 51:	A = 93.00%	L = 0.4065	VA = 81.33%
     Epoch 52:	A = 92.66%	L = 0.3889	VA = 80.61%
     Epoch 53:	A = 92.93%	L = 0.3893	VA = 81.60%
     Epoch 54:	A = 93.22%	L = 0.3882	VA = 81.33%
     Epoch 55:	A = 92.61%	L = 0.3827	VA = 81.24%
     Epoch 56:	A = 93.52%	L = 0.3823	VA = 81.87%
     Epoch 57:	A = 93.40%	L = 0.3812	VA = 81.06%
     Epoch 58:	A = 93.01%	L = 0.3807	VA = 81.96%
     Epoch 59:	A = 93.45%	L = 0.3759	VA = 81.87%
     Epoch 60:	A = 92.90%	L = 0.3680	VA = 81.60%
     Epoch 61:	A = 93.71%	L = 0.3607	VA = 82.05%
     Epoch 62:	A = 93.48%	L = 0.3745	VA = 81.33%
     Epoch 63:	A = 93.52%	L = 0.3652	VA = 81.87%
     Epoch 64:	A = 93.95%	L = 0.3620	VA = 82.14%
     Epoch 65:	A = 93.67%	L = 0.3538	VA = 81.69%
     Epoch 66:	A = 93.43%	L = 0.3527	VA = 81.69%
     Epoch 67:	A = 94.05%	L = 0.3512	VA = 81.87%
     Epoch 68:	A = 93.76%	L = 0.3452	VA = 81.96%
     Epoch 69:	A = 94.13%	L = 0.3493	VA = 81.87%
     Epoch 70:	A = 93.79%	L = 0.3414	VA = 80.88%
     Epoch 71:	A = 94.10%	L = 0.3397	VA = 81.69%
     Epoch 72:	A = 94.26%	L = 0.3381	VA = 81.96%
     Epoch 73:	A = 93.91%	L = 0.3377	VA = 81.60%
     Epoch 74:	A = 94.48%	L = 0.3331	VA = 81.69%
     Epoch 75:	A = 94.64%	L = 0.3406	VA = 81.96%
     Epoch 76:	A = 94.49%	L = 0.3360	VA = 81.69%
     Epoch 77:	A = 94.57%	L = 0.3271	VA = 81.96%
     Epoch 78:	A = 94.44%	L = 0.3288	VA = 81.60%
     Epoch 79:	A = 94.99%	L = 0.3202	VA = 81.51%
     Epoch 80:	A = 94.64%	L = 0.3252	VA = 82.05%
     Epoch 81:	A = 94.75%	L = 0.3282	VA = 81.69%
     Epoch 82:	A = 94.84%	L = 0.3210	VA = 82.14%
     Epoch 83:	A = 95.06%	L = 0.3188	VA = 82.50%
     Epoch 84:	A = 95.19%	L = 0.3122	VA = 81.87%
     Epoch 85:	A = 95.47%	L = 0.3140	VA = 82.68%
     Epoch 86:	A = 94.89%	L = 0.3140	VA = 81.87%
     Epoch 87:	A = 95.30%	L = 0.3167	VA = 81.78%
     Epoch 88:	A = 94.84%	L = 0.3075	VA = 81.60%
     Epoch 89:	A = 95.34%	L = 0.2994	VA = 81.87%
     Epoch 90:	A = 95.30%	L = 0.3012	VA = 82.14%
     Epoch 91:	A = 95.19%	L = 0.3054	VA = 82.50%
     Epoch 92:	A = 95.33%	L = 0.2975	VA = 81.78%
     Epoch 93:	A = 95.30%	L = 0.3032	VA = 81.87%
     Epoch 94:	A = 95.61%	L = 0.2938	VA = 82.50%
     Epoch 95:	A = 95.45%	L = 0.3078	VA = 81.60%
     Epoch 96:	A = 95.54%	L = 0.2991	VA = 82.32%
     Epoch 97:	A = 95.47%	L = 0.3022	VA = 81.96%
     Epoch 98:	A = 95.78%	L = 0.3024	VA = 81.87%
     Epoch 99:	A = 95.84%	L = 0.3056	VA = 81.87%
     Epoch 100:	A = 95.41%	L = 0.3044	VA = 81.78%

### 6.5 Visualize loss, train accuracy and validation accuracy

#### 6.5.1 In `matplotlib`

```python
plt.rcParams['figure.figsize'] = 17, 12
plt.rcParams.update({'font.size': 20})
plt.subplots_adjust(left = 0.125, bottom = 0.1, right = 0.9, top = 0.9, wspace = 0.8, hspace = 0.8)

t = np.arange(len(train_accuracy_history))

plt.subplot(3, 1, 1)
plt.plot(t, loss_history, "r", label = "LOSS INCURED")
plt.xlabel("Epochs")
plt.ylabel("Score")
plt.legend()
plt.grid()

plt.subplot(3, 1, 2)
plt.plot(t, train_accuracy_history, "b", label = "TRAINING ACCURACY")
plt.xlabel("Epochs")
plt.ylabel("Score")
plt.ylim([0, 1])
plt.legend()
plt.grid()

t = np.arange(len(validation_accuracy_history))

plt.subplot(3, 1, 3)
plt.plot(t, validation_accuracy_history, "c", label = "VALIDATION_ACCURACY")
plt.xlabel("Epochs")
plt.ylabel("Score")
plt.ylim([0, 1])
plt.legend()
plt.grid()

plt.show()
```

#### 6.5.2 In `tensorboard`

##### LOSS

##### TRAINING ACCURACY

##### VALIDATION ACCURACY

### 6.7 Test the trained model `G` and compute evaluation metrics

```python
with tf.Session(graph = G) as session:

    # let's load our saved session data
    saver.restore(session, model_dump_path)
    clear_output()

    print("(Evaluation)\n")

    # ought to start the iterator for test data flow
    session.run(test_iterator.initializer)

    # get the predictions and the ground truth used to generate that
    # Note:    don't run this in two session calls.
    # Reason:  performing a second run with `y_test_subset` will get you the next batch of data
    yhat, y = session.run([predictions, y_test_subset])

    # obtain the column with the highest softmax probability
    # that will be our actual binary prediction
    y_predictions = np.argmax(yhat, axis = 1)

    # we need the positive class probability to compute the AUC score
    positive_class_softmax_probability = yhat[:, 1]

    evaluate_model(y_test,
                   y_predictions,
                   positive_class_softmax_probability,
                   interpretations_of_labels = ["WILL_NOT_SUBSCRIBE_TO_BANK_DEPOSIT",
                                                "WILL_SUBSCRIBE_TO_BANK_DEPOSIT"],
                  complete = True)
```

    (Evaluation)

                                        precision    recall  f1-score   support

    WILL_NOT_SUBSCRIBE_TO_BANK_DEPOSIT       0.83      0.83      0.83       374
        WILL_SUBSCRIBE_TO_BANK_DEPOSIT       0.83      0.83      0.83       369

                              accuracy                           0.83       743
                             macro avg       0.83      0.83      0.83       743
                          weighted avg       0.83      0.83      0.83       743

```python
# overall values
evaluate_model(y_test,
               y_predictions,
               positive_class_softmax_probability,
               interpretations_of_labels = ["WILL_NOT_SUBSCRIBE_TO_BANK_DEPOSIT",
                                            "WILL_SUBSCRIBE_TO_BANK_DEPOSIT"],
               complete = False)
```

    Precision: 82.66%
    Recall: 82.66%
    F1 score: 82.66%
    AUC: 88.91%

(__Note__)

In case you wish to perform inference on such a model trained using the `tf.data` API, look up this [answer]( https://stackoverflow.com/questions/50940667/inference-with-a-model-trained-with-tf-dataset).


## Histograms of Oriented Gradients (HOG)

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!