# Treatment Wetlands Aeration Efficiency: A Review

***Just a import of libraries***

```python
import pandas as pd
import datacompy
import plotly_express as px
import matplotlib.pyplot as plt
%matplotlib inline
```

## Introduction

This jupyter notebook serves as complementary material for the paper Treatment Wetlands Aeration Efficiency: A review

## Data search and preparation

### Raw data search

On **web of science** search engine three queries were done to find the input articles for review. The search was set as such:
1. **topic="treatment wetlands" AND topic="aeration"**
2. **topic="treatment wetland" AND topic="aeration"**
3. **topic="constructed wetlands" AND topic="aeration"**

At the first glance it seams obvious that queries for "wetland" and "wetlands" are the same, so the search query "constructed wetland" was omitted in this process. The similarity of the queries is tested further in this notebook.

The results were downloaded as text files for further analysis.

**Date of the search was 14. 11. 2018**

```python
# importing the text files as pandas dataframes for futher processing

query_1 = pd.read_csv("data/raw/treatment-wetlands_aeration.txt", sep="\t", header=0, index_col=False)
query_2 = pd.read_csv("data/raw/treatment-wetland_aeration.txt", sep="\t", header=0, index_col=False)
query_3 = pd.read_csv("data/raw/constructed-wetlands_aeration.txt", sep="\t", header=0, index_col=False)
```

```python
# checking the table structure

query_1.head()
```

      PT
      AU
      BA
      BE
      GP
      AF
      BF
      CA
      TI
      SO
      ...
      PG
      WC
      SC
      GA
      UT
      PM
      OA
      HC
      HP
      DA

      0
      J
      Liu, FF; Fan, JL; Du, JH; Shi, X; Zhang, J; Sh...
      NaN
      NaN
      NaN
      Liu, Fei-fei; Fan, Jinlin; Du, Jinhui; Shi, Xi...
      NaN
      NaN
      Intensified nitrogen transformation in intermi...
      SCIENCE OF THE TOTAL ENVIRONMENT
      ...
      8
      Environmental Sciences
      Environmental Sciences & Ecology
      GX6LB
      WOS:000447871400115
      30373064.0
      NaN
      NaN
      NaN
      2018-11-14

      1
      J
      Nivala, J; Kahl, S; Boog, J; van Afferden, M; ...
      NaN
      NaN
      NaN
      Nivala, Jaime; Kahl, Stefanie; Boog, Johannes;...
      NaN
      NaN
      Dynamics of emerging organic contaminant remov...
      SCIENCE OF THE TOTAL ENVIRONMENT
      ...
      13
      Environmental Sciences
      Environmental Sciences & Ecology
      GV4NL
      WOS:000446076500109
      30308886.0
      NaN
      NaN
      NaN
      2018-11-14

      2
      J
      Masi, F; Rizzo, A; Bresciani, R; Martinuzzi, N...
      NaN
      NaN
      NaN
      Masi, F.; Rizzo, A.; Bresciani, R.; Martinuzzi...
      NaN
      NaN
      Lessons learnt from a pilot study on residual ...
      SCIENCE OF THE TOTAL ENVIRONMENT
      ...
      9
      Environmental Sciences
      Environmental Sciences & Ecology
      GX5QQ
      WOS:000447805500015
      30114585.0
      NaN
      NaN
      NaN
      2018-11-14

      3
      J
      Turker, OC
      NaN
      NaN
      NaN
      Turker, Onur Can
      NaN
      NaN
      Simultaneous boron (B) removal and electricity...
      JOURNAL OF ENVIRONMENTAL MANAGEMENT
      ...
      12
      Environmental Sciences
      Environmental Sciences & Ecology
      GY0PX
      WOS:000448224900003
      30212671.0
      NaN
      NaN
      NaN
      2018-11-14

      4
      J
      Song, P; Huang, GH; An, CJ; Shen, J; Zhang, P;...
      NaN
      NaN
      NaN
      Song, Pei; Huang, Guohe; An, Chunjiang; Shen, ...
      NaN
      NaN
      Treatment of rural domestic wastewater using m...
      SCIENCE OF THE TOTAL ENVIRONMENT
      ...
      11
      Environmental Sciences
      Environmental Sciences & Ecology
      GU3GG
      WOS:000445164000056
      29990903.0
      NaN
      NaN
      NaN
      2018-11-14

5 rows × 68 columns

***Now we will test if query with term "constructed wetlands" is similar to the query with the term "constructed wetland"***

```python
# testing the similarity between query_1 and query_2 (search 1 and 2)

query_1.equals(query_2)
```

    True

**As we can see the two terms generate equal output so we can omitt the query_2 dataset in further processing**

### Comparing the search terms

We have two datasets from searching the terms "constructed wetlands" or "treatment wetlands" and "aeration" with similar number of datapoints (rows in the table).

```python
print("The number of rows for search term 'constructed wetlands' is: %d" % len(query_1))
print("The number of rows for search term 'treatment wetlands' is: %d" %len(query_3))
```

    The number of rows for search term 'constructed wetlands' is: 444
    The number of rows for search term 'treatment wetlands' is: 465

Are these dataset similar also in the content?

```python
compare = datacompy.Compare(query_1, query_3, join_columns='GA', df1_name='TW', df2_name='CW')
#print(compare.report())
```

**We can see from the report that both datasets has 379 common rows and query_1 (CW) has 65 unique rows whereas query_3 (TW) has 86 unique rows.**

### Joining and cleaning the dataset

We will now make a join with droping the duplicate rows and preserving the unique rows.

```python
cw_tw_join = pd.concat([query_1, query_3]).drop_duplicates().reset_index(drop=True)
print("The number of rows in joined dataset is: %d" %len(cw_tw_join))
```

    The number of rows in joined dataset is: 535

We will also drop unuseful columns from the dataset. (The whole list of Web of Science Core Collection Field Tags (what means the columns names abbreviations) can be found [here](https://images.webofknowledge.com/images/help/WOK/hs_wos_fieldtags.html))

```python
cw_tw = cw_tw_join.loc[:, ['au', 'ti', 'de', 'so', 'di']]
df_shape = cw_tw.shape

print("Final dataset with all unique datapoints"
      + "for search terms 'constructed wetlands'"
      + "or 'treatment wetlands' and 'aeration'"
      + "now contains %d rows and %d columns" % (df_shape[0], df_shape[1])
     )
```

    Final dataset with all unique datapointsfor search terms 'constructed wetlands'or 'treatment wetlands' and 'aeration'now contains 535 rows and 5 columns

```python
# First five rows of the dataset

cw_tw.head()
```

      au
      ti
      de
      so
      di

      0
      Liu, FF; Fan, JL; Du, JH; Shi, X; Zhang, J; Sh...
      Intensified nitrogen transformation in intermi...
      Constructed wetlands; Intermittent aeration; C...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.10.037

      1
      Nivala, J; Kahl, S; Boog, J; van Afferden, M; ...
      Dynamics of emerging organic contaminant remov...
      Aeration; Constructed wetland; Micropollutant;...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.08.339

      2
      Masi, F; Rizzo, A; Bresciani, R; Martinuzzi, N...
      Lessons learnt from a pilot study on residual ...
      Constructed wetland; Treatment wetland; Aerate...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.08.113

      3
      Turker, OC
      Simultaneous boron (B) removal and electricity...
      Water treatment; Duckweed wastewater treatment...
      JOURNAL OF ENVIRONMENTAL MANAGEMENT
      10.1016/j.jenvman.2018.08.112

      4
      Song, P; Huang, GH; An, CJ; Shen, J; Zhang, P;...
      Treatment of rural domestic wastewater using m...
      Rural wastewater; Multi-soil-layering system; ...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.06.331

We will rename the columns and export the raw dataset to csv file.

```python
cw_tw.columns = ["authors", "title", "keywords", "periodical", "id"]
```

and export this dataset as csv file

```python
# cw_tw.to_csv("data/intermediate/cleaned_raw_papers.csv", index=False)
```

### Selecting papers for further manual inspection

We will now try to narrow the final dataset even more. After inspecting several rows (or better say articles) we can see that they are from the field of constructed/treatment wetlands but they do not contain any information on topic of wetland aeration. The inspection of titles and keywords of all articles can help us to drop such articles from the dataset.

Because the topic of "aeration" can be expressed in several forms (like e.g aeration versus aerated), we will only search if the title or the keywords contains the string ***"aer"*** to include all possible forms being used.

```python
# importing the cleaned raw datafile
all_papers = pd.read_csv("data/intermediate/cleaned_raw_papers.csv")
all_papers.head()
```

      authors
      title
      keywords
      periodical
      id

      0
      Liu, FF; Fan, JL; Du, JH; Shi, X; Zhang, J; Sh...
      Intensified nitrogen transformation in intermi...
      Constructed wetlands; Intermittent aeration; C...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.10.037

      1
      Nivala, J; Kahl, S; Boog, J; van Afferden, M; ...
      Dynamics of emerging organic contaminant remov...
      Aeration; Constructed wetland; Micropollutant;...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.08.339

      2
      Masi, F; Rizzo, A; Bresciani, R; Martinuzzi, N...
      Lessons learnt from a pilot study on residual ...
      Constructed wetland; Treatment wetland; Aerate...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.08.113

      3
      Turker, OC
      Simultaneous boron (B) removal and electricity...
      Water treatment; Duckweed wastewater treatment...
      JOURNAL OF ENVIRONMENTAL MANAGEMENT
      10.1016/j.jenvman.2018.08.112

      4
      Song, P; Huang, GH; An, CJ; Shen, J; Zhang, P;...
      Treatment of rural domestic wastewater using m...
      Rural wastewater; Multi-soil-layering system; ...
      SCIENCE OF THE TOTAL ENVIRONMENT
      10.1016/j.scitotenv.2018.06.331

```python
# defining the search term
word = "aer"

# building the filters
# filtering titles and keywords which contains 'aer' and building new columns with True / False values
cw_tw['title_filter'] = cw_tw['title'].str.contains(word)
cw_tw['keywords_filter'] = cw_tw['keywords'].str.contains(word)
cw_tw['filter'] = cw_tw['title_filter'] + cw_tw['keywords_filter']
```

```python
# filtering the raw dataset
filtered_dataset = cw_tw[cw_tw['filter'] > 0]

print("The number of rows in final dataset is: %d" %len(filtered_dataset))
```

    The number of rows in final dataset is: 205

We will also export this dataset to the csv file

```python
filtered_dataset = filtered_dataset.drop(["title_filter", "keywords_filter", "filter"], axis=1)

# filtered_dataset.to_csv("data/intermediate/final_set_of_papers.csv", index=False)
```

**After this preparation there is no other way than to screen all the selected articles manually, if their topic is really about wetland active (or forced) aeration (because we can expect there will be also a lot of articles about passive aeration of wetlands).**

You can find the manually prescreened (which means they are somehow connected to our article topic) list of papers (counting 141 papers) in the intermediate data folder in [word file](files/data/intermediate/prescreened_papers.docx) or in [txt format](files/data/intermediate/prescreened_papers.txt).

## Final Dataset

The prescreened dataset of papers (references) was supplemented with the full versions of articles (in pdf available on request). For the paper to be included in the review three criteria had to be met. The paper had to include:

1. Sufficient information on the active aeration of the wetland (at least the daily aeration time and the air flow rate),
2. Inflow and outflow concentrations for COD, BOD or CBOD and TKN or NH4−N and
3. data for the experimental wetland such as dimensions, hydraulic retention time, hydraulic load and other data necessary for the calculations within the study.

**Based on these criteria only 21 research articles were processed further in the review.**

###  Basic statictical description of main parameters

```python
paper_list = pd.read_excel('data/raw/papers_final_list.xlsx', index_col=0)
paper_list
```

      Citation
      Type of WW
      Size
      Flow Type
      Feeding
      Aeration placement
      Aeration elements

      No

      1
      (Araya et al., 2016)
      pretreated municipal
      lab-scale
      vertical
      batch-filled
      bottom
      unknown

      2
      (Auvinen et al., 2017)
      mun. and hospital
      pilot-scale
      vertical
      continuous
      bottom
      perforated pipe in spiral

      3
      (Boog et al., 2014)
      pretreated domestic
      pilot-scale
      horizontal
      intermittent
      bottom
      perforated pipes (FBA)

      4
      (Butterworth et al., 2013)
      tertiary municipal
      full-scale
      horizontal
      continuous
      bottom
      perforated pipes (FBA)

      5
      (Fan et al., 2013a)
      synthetic
      lab-scale
      vertical
      batch-filled
      bottom
      porous air spargers

      6
      (Fan et al., 2013b)
      synthetic
      lab-scale
      vertical
      batch-filled
      bottom
      porous air spargers

      7
      (Fan et al., 2013c)
      synthetic
      pilot-scale
      horizontal
      continuous
      bottom-front
      quadrate tubing

      8
      (Foladori et al., 2013)
      pretreated domestic
      mesocosm
      vertical
      intermittent
      bottom
      perforated pipes

      9
      (Maltais-Landry et al., 2007)
      fish farm effluent
      mesocosm
      horizontal
      continuous
      bottom-front
      cicular tubing

      10
      (Nivala et al., 2007)
      landfill leachate
      pilot-scale
      horizontal
      continuous
      bottom
      perforated pipes (FBA)

      11
      (Ouellet-Plamondon et al., 2006)
      fish farm effluent
      mesocosm
      horizontal
      continuous
      bottom-front
      cicular tubing

      12
      (Pan et al., 2012)
      low strenght mucipial
      full-scale
      vertical
      continuous
      bottom
      perforated pipes

      13
      (Patil and Chakraborty, 2017)
      synthetic
      lab-scale
      horizontal
      continuous
      bottom-middle
      unknown

      14
      (Redmond et al., 2014)
      pretreated municipal
      pilot-scale
      horizontal
      intermittent
      bottom
      perforated pipes (FBA)

      15
      (Uggetti et al., 2016)
      pretreated municipal
      pilot-scale
      horizontal
      intermittent
      bottom
      perforated pipes

      16
      (Wu et al., 2015)
      synthetic
      lab-scale
      vertical
      batch-filled
      bottom
      porous air spargers

      17
      (Wu et al., 2016a)
      synthetic
      lab-scale
      vertical
      batch-filled
      bottom
      porous air spargers

      18
      (Wu et al., 2016b)
      synthetic
      lab-scale
      vertical
      batch-filled
      bottom
      porous air spargers

      19
      (Zapater-Pereyra et al., 2014)
      pretreated municipal
      lab-scale
      horizontal
      continuous
      bottom
      aeration tubings

      20
      (Zhong et al., 2015)
      pretreated municipal
      mesocosm
      horizontal
      continuous
      inflow chamber
      unknown

      21
      (Zhou et al., 2017)
      synthetic
      lab-scale
      HTS
      batch-filled
      bottom
      porous air spargers

Data from these articles were extracted as individual "review cases". We can find several different review cases in some articles (experiments with different parameters) so the total number of the review cases is 49.

```python
review_cases = pd.read_excel('data/raw/review_cases.xlsx')
# print(review_cases)
```

The review cases are number as such that number refers to the article number in the table (list) of papers and the 'a, b, c, etc.' distinguish the individual review cases identified in that article.

```python
review_cases[['water_depth_m', 'HLR_lit_m2d', 'OLR_gCOD_m2d']].describe()
```

      water_depth_m
      HLR_lit_m2d
      OLR_gCOD_m2d

      count
      49.000000
      49.000000
      49.000000

      mean
      0.532449
      121.140722
      29.973675

      std
      0.274949
      242.925088
      31.172179

      min
      0.150000
      4.000000
      2.510200

      25%
      0.300000
      44.600000
      9.000000

      50%
      0.600000
      64.000000
      26.885000

      75%
      0.650000
      95.000000
      31.072190

      max
      1.700000
      1600.000000
      150.960000

There seems to be some substantial difference in the data, especially for hydraulic loading rate. We can try to plot the data to see if these differences are only due to some specific review cases.

```python
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
review_cases[['HLR_lit_m2d']].plot(kind='hist',rwidth=0.8, bins=16, ax=ax1, figsize=(15, 5))
review_cases[['OLR_gCOD_m2d']].plot(kind='hist',rwidth=0.8, ax=ax2, bins=15, figsize=(15, 5))
plt.show()
```

According to the histograms we can clearly see that the majority of the review cases has hydraulic loading below 100 liters par day and ORL below 30 gCOD/day.

```python
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
review_cases[['HLR_lit_m2d']].plot(kind='box', ax=ax1, figsize=(15, 5), grid=True, ylim=(0, 200))
review_cases[['OLR_gCOD_m2d']].plot(kind='box', ax=ax2, figsize=(15, 5), grid=True, ylim=(0, 80))
plt.show()
```

### Data Analysis

#### Oxygen consumption rates

##### Methodology

Oxygen consumption rate is defined here similar to the definition of Kadlec and Wallace (2009) and is expressed as anticipated rate of oxygen consumption in a treatment wetland due to various microbial processes. For processes including the organic carbon the COD or CBOD parameters are being used and the respective quations differs accordingly (see below). For nitrification and other pathways of N-forms reduction the parameter TKN is used where possible. If not the NH4-N is used instead without changing the respective equations.

_All rates are calculated in a volumetric as well as areal aproach (the rate is not defined per per volume unit or areal)._

OCR calculation for review cases with COD parameter.

\begin{equation}\label{eq:1}
OCR_{max-COD-V} = \frac{[1(\Delta M_{COD} + 4.6(\Delta M_{TKN})]}{V}
\end{equation}

\begin{equation}\label{eq:2}
OCR_{int-COD-V} = \frac{[0.5(\Delta M_{COD} + 1.7(\Delta M_{TKN})]}{V}
\end{equation}

\begin{equation}\label{eq:3}
OCR_{min} = \frac{[1.7(\Delta M_{TKN})]}{V}
\end{equation}

OCR calculation for review cases with CBOD parameter.

\begin{equation}\label{eq:4}
OCR_{max-CBOD-V} = \frac{[1.5(\Delta M_{CBOD} + 4.6(\Delta M_{TKN})]}{V}
\end{equation}

\begin{equation}\label{eq:5}
OCR_{int-CBOD-V} = \frac{[1(\Delta M_{CBOD} + 1.7(\Delta M_{TKN})]}{V}
\end{equation}

\begin{equation}\label{eq:6}
OCR_{min} = \frac{[1.7(\Delta M_{TKN})]}{V}
\end{equation}

$\Delta M$ is always as:

\begin{equation}\label{eq:7}
\Delta M =  Q_i \times C_i - Q_0 \times C_0
\end{equation}

V - volume of wetland (m3), M - pollutant mass flow rate (g/d), Qi - inflow Rate (m3/d), *Qo* - outflow rate (m3/d); *Ci* - inlet concentration, (g/m3), *Co* - outlet concentration (g.m3), OCR - oxygen consumption rate (g/m2.d). *V* can be changed for *A* - wetalnd area (m2) for areal rate calculation.

In our table with all review cases we have these columns already defined:
- area_m2 - wetland area in square meters
- volume_m3 - wetted volume of the wetland (so not the total volume, but area x depth of water)
- daily_flow_lit - daily flow in liters
- CBOD_IN - inflow concentration for CBOD in mg per liter
- CBOD_OUT - outflow concentration for CBOD in mg per liter
- deltaCBOD - difference between inflow an outflow concetration in mg per liter
- columns for COD, TKN and NH4 are defined ni similar manner

```python
review_cases.columns
```

    Index(['Reference', 'number', 'water_depth_m', 'area_m2', 'volume_m3',
           'porosity', 'voids_m3', 'HLR_lit_m2d', 'daily_flow_lit', 'OLR_gCOD_m2d',
           'HRT_h', 'HRT_days', 'air_flow_m3h', 'aer_duration_h_day',
           'air_per_day_m3', 'air_per_HRT_m3', 'COD_IN', 'COD_OUT', 'deltaCOD',
           'CBOD_IN', 'CBOD_OUT', 'deltaCBOD', 'TKN_IN', 'TKN_OUT', 'deltaTKN',
           'NH4_IN', 'NH4_OUT', 'deltaNH4'],
          dtype='object')

As the inflow and outflow volume is still the same ((in liters)) and mg/l are same as g/m3 we can compute the $\Delta M$ form equation 7 directly as $Q_d \times 0.001 \times \Delta C  $

With the help of for loop we than calculate the oxygen consumption rates in both volumetric and areal form.

```python
# checking for missing COD and TKN values
COD_nan = review_cases['deltaCOD'].notnull()
TKN_nan = review_cases['deltaTKN'].notnull()

# adding blank columns for volumetric and areal OCR values before calculation
pd.concat([review_cases,
           pd.DataFrame(columns = ['OCR_max_v',
                                   'OCR_int_v',
                                   'OCR_min_v',
                                   'OCR_max_a',
                                   'OCR_int_a',
                                   'OCR_min_a'
                                  ])])

for i in range(0, len(review_cases)):

    # daily flow and volume is the same whatever conc parameters are used
    daily_flow = review_cases.loc[i, 'daily_flow_lit']
    volume = review_cases.loc[i, 'volume_m3']
    area = review_cases.loc[i, 'area_m2']

    # calculation for COD values
    if COD_nan[i] == True:
        deltaCOD = review_cases.loc[i, 'deltaCOD']

        M_COD = daily_flow * 0.001 * deltaCOD

        if TKN_nan[i] == True:
            deltaTKN = review_cases.loc[i, 'deltaTKN']
        else:
            deltaTKN = review_cases.loc[i, 'deltaNH4']

        M_TKN = daily_flow * 0.001 * deltaTKN

        # volumetric OCR values
        OCR_max_v = ((1 * M_COD) + (4.6 * M_TKN)) / volume
        OCR_int_v = ((0.5 * M_COD) + (1.7 * M_TKN)) / volume
        OCR_min_v = (1.7 * M_TKN) / volume

        # areal OCR values
        OCR_max_a = ((1 * M_COD) + (4.6 * M_TKN)) / area
        OCR_int_a = ((0.5 * M_COD) + (1.7 * M_TKN)) / area
        OCR_min_a = (1.7 * M_TKN) / area

    # calculation for CBOD values
    else:
        deltaCBOD = review_cases.loc[i, 'deltaCBOD']

        M_CBOD = daily_flow * 0.001 * deltaCBOD

        if TKN_nan[i] == True:
            deltaTKN = review_cases.loc[i, 'deltaTKN']
        else:
            deltaTKN = review_cases.loc[i, 'deltaNH4']

        M_TKN = daily_flow * 0.001 * deltaTKN

        # volumetric OCR values
        OCR_max_v = ((1.5 * M_CBOD) + (4.6 * M_TKN)) / volume
        OCR_int_v = ((1 * M_CBOD) + (1.7 * M_TKN)) / volume
        OCR_min_v = (1.7 * M_TKN) / volume

        # areal OCR values
        OCR_max_a = ((1.5 * M_CBOD) + (4.6 * M_TKN)) / area
        OCR_int_a = ((1 * M_CBOD) + (1.7 * M_TKN)) / area
        OCR_min_a = (1.7 * M_TKN) / area

    # writing the values to the dataframe
    review_cases.loc[i, 'OCR_max_v'] = OCR_max_v
    review_cases.loc[i, 'OCR_int_v'] = OCR_int_v
    review_cases.loc[i, 'OCR_min_v'] = OCR_min_v
    review_cases.loc[i, 'OCR_max_a'] = OCR_max_a
    review_cases.loc[i, 'OCR_int_a'] = OCR_int_a
    review_cases.loc[i, 'OCR_min_a'] = OCR_min_v
```

    c:\users\dell\.virtualenvs\aerationreview-wzvxscyu\lib\site-packages\ipykernel_launcher.py:12: FutureWarning:

    Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

```python
review_cases.head()
```

      Reference
      number
      water_depth_m
      area_m2
      volume_m3
      porosity
      voids_m3
      HLR_lit_m2d
      daily_flow_lit
      OLR_gCOD_m2d
      ...
      deltaTKN
      NH4_IN
      NH4_OUT
      deltaNH4
      OCR_max_v
      OCR_int_v
      OCR_min_v
      OCR_max_a
      OCR_int_a
      OCR_min_a

      0
      (Araya et al., 2016)
      1a
      0.15
      0.063
      0.00945
      NaN
      0.001417
      15.873016
      1.0
      4.444444
      ...
      NaN
      114.0
      27.36
      86.64
      61.433228
      25.215661
      15.586032
      9.214984
      3.782349
      15.586032

      1
      (Araya et al., 2016)
      1b
      0.15
      0.063
      0.00945
      NaN
      0.001417
      15.873016
      1.0
      6.317460
      ...
      NaN
      112.0
      72.80
      39.20
      44.351323
      19.686772
      7.051852
      6.652698
      2.953016
      7.051852

      2
      (Araya et al., 2016)
      1c
      0.15
      0.063
      0.00945
      NaN
      0.001417
      15.873016
      1.0
      5.396825
      ...
      NaN
      54.0
      11.34
      42.66
      47.390053
      20.986455
      7.674286
      7.108508
      3.147968
      7.674286

      3
      (Auvinen et al., 2017)
      2a
      0.75
      1.000
      0.75000
      0.4
      0.300000
      360.000000
      360.0
      60.480000
      ...
      NaN
      24.0
      0.00
      24.00
      121.152000
      53.664000
      19.584000
      90.864000
      40.248000
      19.584000

      4
      (Auvinen et al., 2017)
      2b
      0.75
      1.000
      0.75000
      0.4
      0.300000
      680.000000
      680.0
      150.960000
      ...
      NaN
      33.0
      5.60
      27.40
      289.262933
      129.725867
      42.232533
      216.947200
      97.294400
      42.232533

5 rows × 34 columns

##### Oxygen Consumption rate results

```python
r_col = ['OCR_max_v', 'OCR_int_v', 'OCR_min_v', 'OCR_max_a', 'OCR_int_a', 'OCR_min_a']

for col in r_col:
    avg = review_cases[col].mean()
    median = review_cases[col].median()
    min_v = review_cases[col].min()
    max_v = review_cases[col].max()

    if "max" in col:
        add = " maximum "
    elif "int" in col:
        add = " intermediate "
    else:
        add = " minimum "

    if "v" in col:
        print("For" + add + "volumetric oxygen consumption rate the:")
    else:
        print("For" + add + "areal oxygen consumption rate the:")

    print("- Average value is %0.2f and median value is %0.2f g/day." % (avg, median))
    print("- Minimum value is %0.2f and maximum is %0.2f g/day." % (min_v, max_v))
    print("")
```

    For maximum volumetric oxygen consumption rate the:
    - Average value is 78.50 and median value is 59.24 g/day.
    - Minimum value is 8.91 and maximum is 419.77 g/day.

    For intermediate volumetric oxygen consumption rate the:
    - Average value is 36.00 and median value is 27.29 g/day.
    - Minimum value is 4.10 and maximum is 188.28 g/day.

    For minimum volumetric oxygen consumption rate the:
    - Average value is 10.20 and median value is 6.84 g/day.
    - Minimum value is 1.01 and maximum is 61.22 g/day.

    For maximum areal oxygen consumption rate the:
    - Average value is 41.55 and median value is 33.76 g/day.
    - Minimum value is 3.38 and maximum is 216.95 g/day.

    For intermediate areal oxygen consumption rate the:
    - Average value is 19.13 and median value is 15.98 g/day.
    - Minimum value is 1.56 and maximum is 97.29 g/day.

    For minimum areal oxygen consumption rate the:
    - Average value is 10.20 and median value is 6.84 g/day.
    - Minimum value is 1.01 and maximum is 61.22 g/day.

**Oxygen consumption rate versus organic loading**

```python
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
fnt=14
labelpad=10
review_cases.plot(kind='scatter', x='OCR_int_v', y='OLR_gCOD_m2d', ax=ax1, figsize=(15, 5), grid=True)
review_cases.plot(kind='scatter', x='OLR_gCOD_m2d', y='OCR_int_a', ax=ax2, figsize=(15, 5), grid=True)
ax1.set_xlabel('Volumetric OCR (g/m3.d)', fontsize=fnt, labelpad=labelpad)
ax2.set_xlabel('Areal OCR (g/m3.d)', fontsize=fnt, labelpad=labelpad)
ax1.set_ylabel('OLR (gCOD/m2.d)', fontsize=fnt, labelpad=labelpad)
ax2.set_ylabel('', fontsize=fnt, labelpad=labelpad)
ax1.set_title('Organic loading rate versus volumetric OCR', fontsize=fnt, pad=labelpad)
ax2.set_title('Organic loading rate versus areal OCR', fontsize=fnt, pad=labelpad)
plt.show()
```

**Oxygen consumption rate versus hydraulic loading**

```python
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
fnt=16
labelpad=10
review_cases.plot(kind='scatter', x='HLR_lit_m2d', y='OCR_int_v', ax=ax1, figsize=(15, 10), grid=True,
                 title='Hydraulic loading rate versus volumetric OCR', fontsize=fnt)
review_cases.plot(kind='scatter', x='HLR_lit_m2d', y='OCR_int_a', ax=ax2, figsize=(15, 10), grid=True,
                 title='Hydraulic loading rate versus areal OCR', fontsize=fnt)
ax1.set_xlabel('Hydraulic loading rate (liters/m2.d)', fontsize=fnt, labelpad=labelpad)
ax2.set_xlabel('Hydraulic loading rate (liters/m2.d)', fontsize=fnt, labelpad=labelpad)
ax1.set_ylabel('Volumetric Oxygen consumption rate (g/d)', fontsize=fnt, labelpad=labelpad)
ax2.set_ylabel('Areal Oxygen consumption rate (g/d)', fontsize=fnt, labelpad=labelpad)
plt.show()
```

#### Oxygen consumption during HRT

##### Methodology

Oxygen consumption (OC) is calculated similar as the oxygen consumption rate. The main difference is that the oxygen consumption is not defined as rate (per unit of area / volume), but as a total oxygen consumption for a give hydraulic retention time. In other words - how much oxygen is theoretically consumed by microbial processes in the time when the wastewater enters the system till the time the water exits the system.

OC calculation for review cases with COD parameter.

\begin{equation}\label{eq:8}
OC_(max) = 1(\Delta m_{COD} + 4.6(\Delta m_{TKN})
\end{equation}

\begin{equation}\label{eq:9}
OC_(int) = 0.5(\Delta m_{COD} + 1.7(\Delta m_{TKN})
\end{equation}

\begin{equation}\label{eq:10}
OC_m = 1.7 \times m_TKN
\end{equation}

OC calculation for review cases with CBOD parameter.

\begin{equation}\label{eq:11}
OC_(max) = 1.5(\Delta m_{COD} + 4.6(\Delta m_{TKN})
\end{equation}

\begin{equation}\label{eq:12}
OC_{int} = 1(\Delta m_{COD} + 1.7(\Delta m_{TKN})
\end{equation}

\begin{equation}\label{eq:13}
OC_{min} = 1.7(\Delta m_{TKN})
\end{equation}

$ \Delta m $ is defined se the load difference after the hydraulic retention time:

\begin{equation}\label{eq:14}
\Delta m =  q_i \times C_i - q_0 \times C_0
\end{equation}

m - pollutant mass flow (g/HRT), qi - inflow (m3/HRT), *qo* - outflow (m3/HRT); *Ci* - inlet concentration, (g/m3), *Co* - outlet concentration (g.m3), OC - oxygen consumption during HRT (g).

We can use equivalent form of calculation for the oxygen consumption.

```python
# checking for missing COD and TKN values
COD_nan = review_cases['deltaCOD'].notnull()
TKN_nan = review_cases['deltaTKN'].notnull()

# adding blank columns for OC values before calculation
pd.concat([review_cases,
           pd.DataFrame(columns = ['OC_max',
                                   'OC_int',
                                   'OC_min',
                                  ])])

for i in range(0, len(review_cases)):

    # total flow during HRT
    hrt_flow = review_cases.loc[i, 'daily_flow_lit'] * review_cases.loc[i, 'HRT_days']

    # calculation for COD values
    if COD_nan[i] == True:
        deltaCOD = review_cases.loc[i, 'deltaCOD']

        m_COD = hrt_flow * 0.001 * deltaCOD

        if TKN_nan[i] == True:
            deltaTKN = review_cases.loc[i, 'deltaTKN']
        else:
            deltaTKN = review_cases.loc[i, 'deltaNH4']

        m_TKN = hrt_flow * 0.001 * deltaTKN

        # OC values - COD computation
        OC_max = ((1 * m_COD) + (4.6 * m_TKN))
        OC_int = ((0.5 * m_COD) + (1.7 * m_TKN))
        OC_min = (1.7 * m_TKN)

    # calculation for CBOD values
    else:
        deltaCBOD = review_cases.loc[i, 'deltaCBOD']

        m_CBOD = hrt_flow * 0.001 * deltaCBOD

        if TKN_nan[i] == True:
            deltaTKN = review_cases.loc[i, 'deltaTKN']
        else:
            deltaTKN = review_cases.loc[i, 'deltaNH4']

        m_TKN = hrt_flow * 0.001 * deltaTKN

        # OC values - CBOD computation
        OC_max = ((1.5 * m_CBOD) + (4.6 * m_TKN))
        OC_int = ((1 * m_CBOD) + (1.7 * m_TKN))
        OC_min = (1.7 * m_TKN)

    # writing the values to the dataframe
    review_cases.loc[i, 'OC_max'] = OC_max
    review_cases.loc[i, 'OC_int'] = OC_int
    review_cases.loc[i, 'OC_min'] = OC_min
```

    c:\users\dell\.virtualenvs\aerationreview-wzvxscyu\lib\site-packages\ipykernel_launcher.py:9: FutureWarning:

    Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

##### Oxygen Consumption results

```python
r_col = ['OC_max', 'OC_int', 'OC_min']

for col in r_col:
    avg = review_cases[col].mean()
    median = review_cases[col].median()
    min_v = review_cases[col].min()
    max_v = review_cases[col].max()

    if "max" in col:
        add = " maximum "
    elif "int" in col:
        add = " intermediate "
    else:
        add = " minimum "

    if "v" in col:
        print("For" + add + "oxygen consumption the:")
    else:
        print("For" + add + "oxygen consumption the:")

    print("- Average value is %0.2f and median value is %0.2f g during HRT." % (avg, median))
    print("- Minimum value is %0.2f and maximum is %0.2f g during HRT." % (min_v, max_v))
    print("")
```

    For maximum oxygen consumption the:
    - Average value is 2800.68 and median value is 10.68 g during HRT.
    - Minimum value is 1.19 and maximum is 80440.57 g during HRT.

    For intermediate oxygen consumption the:
    - Average value is 1203.79 and median value is 4.81 g during HRT.
    - Minimum value is 0.56 and maximum is 35293.19 g during HRT.

    For minimum oxygen consumption the:
    - Average value is 576.20 and median value is 1.48 g during HRT.
    - Minimum value is 0.11 and maximum is 13960.11 g during HRT.

#### Aeration efficiency

##### Methodology

In previous chapter we calculated the theoretical total amount of oxygen consumed by microbial processes during the hydraulic retention time of the wetland system. In the same manner we can now calculate the total oxygen amount delivered to the system:

\begin{equation}\label{eq:15}
OI = Q_(air) \times ATD \times HRT \times \rho_(air) \times \omega_(air)
\end{equation}

where

$OI$ - oxygen input (g/HRT), $Q_(air)$ - air flow (m3/h), $ATD$ - aeration time per day (h per day), $HRT$ - hydraulic retention time (days), $\rho_(air)$ - density of air (1215 g/m3) and  $\omega_(air)$ - a mass fraction of oxygen in air (21%)

Than we can calculate the aeration efficiency as the ratio between the total oxygen input to the system (with forced aeration) and the total amount of oxygen used in the system (oxygen input minus consumed oxygen).
The aeration efficiency will be computed for the three microbial activity scenarios (maximum, intermediate and minimum) and expressed as percents.

\begin{equation}\label{eq:16}
AE = \frac{OC}{OI} \times 100
\end{equation}

```python
Q_air = review_cases['air_flow_m3h']
ATD = review_cases['aer_duration_h_day']
HRT = review_cases['HRT_days']

review_cases['OI'] = Q_air * ATD * HRT * 1215 * 0.21
```

```python
OI = review_cases['OI']
OC_max = review_cases['OC_max']
OC_int = review_cases['OC_int']
OC_min = review_cases['OC_min']

review_cases['AE_max'] = OC_max / OI * 100
review_cases['AE_int'] = OC_int / OI * 100
review_cases['AE_min'] = OC_min / OI * 100
```

```python
# export of the calculated file

# review_cases.to_csv('data/final/review_cases_final.csv', index=False)
```

## Results

```python
# import of the final bresults

review_cases = pd.read_csv('data/final/review_cases_final.csv')
# review_cases.head()
```

### Overall results

```python
r_col = ['AE_max', 'AE_int', 'AE_min']

for col in r_col:
    avg = review_cases[col].mean()
    median = review_cases[col].median()
    min_v = review_cases[col].min()
    max_v = review_cases[col].max()
    std = review_cases[col].std()

    if "max" in col:
        add = " maximum "
    elif "int" in col:
        add = " intermediate "
    else:
        add = " minimum "

    print("For" + add + "aeration efficinecy the:")

    print("- Average value is %0.2f%% and median value is %0.2f%%." % (avg, median))
    print("- Minimum value is %0.2f%% and maximum is %0.2f%%." % (min_v, max_v))
    print("- Standard deviation is %0.2f%%." % (std))
    print("")

avg_depth = review_cases['water_depth_m'].mean()
print("Average depth is %0.2f m." % avg_depth)
```

    For maximum aeration efficinecy the:
    - Average value is 3.74% and median value is 1.08%.
    - Minimum value is 0.01% and maximum is 74.36%.
    - Standard deviation is 10.87%.

    For intermediate aeration efficinecy the:
    - Average value is 1.73% and median value is 0.48%.
    - Minimum value is 0.01% and maximum is 33.72%.
    - Standard deviation is 4.95%.

    For minimum aeration efficinecy the:
    - Average value is 0.45% and median value is 0.14%.
    - Minimum value is 0.00% and maximum is 9.80%.
    - Standard deviation is 1.40%.

    Average depth is 0.53 m.

#### Basic plots

**Histogram for intermediate aeration efficiency**

```python
fig = plt.figure(figsize=(8, 5))
ax1 = fig.add_subplot(1, 1, 1)
fnt=14
labelpad=10

review_cases[['AE_int']].plot(kind='hist',
                              rwidth=0.7,
                              bins=27,
                              legend=False,
                              xticks=range(0, 9, 1),
                              xlim=(0, 9),
                              ax=ax1
                             )

ax1.set_ylabel('Number of cases', fontsize=fnt, labelpad=labelpad)
ax1.set_xlabel('Intermediate aeration efficiency (%)', fontsize=fnt, labelpad=labelpad)

plt.show()
```

```python
review_cases[['AE_int']].plot(kind='box', figsize=(5, 5), ylim=(0,10))
plt.show()
```

**Boxplots**

```python
# without the one most efficient case
review_cases_filtered = review_cases[review_cases['AE_int'] < 10]
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
review_cases_filtered[['AE_max']].plot(kind='box', figsize=(15, 10), ax=ax1)
review_cases_filtered[['AE_int']].plot(kind='box', figsize=(15, 10), ax=ax2, label='title')
plt.show()
```

### Hydraulic and Organic loading rate versus aeration efficiency

```python
fig = plt.figure()
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)
fnt=16
labelpad=10
review_cases_filtered.plot(kind='scatter', x='HLR_lit_m2d', y='AE_int', ax=ax1, figsize=(15, 5), grid=True, fontsize=fnt)
review_cases_filtered.plot(kind='scatter', x='OLR_gCOD_m2d', y='AE_int', ax=ax2, figsize=(15, 5), grid=True, fontsize=fnt)
ax1.set_xlabel('Hydraulic loading rate (liters/m2.d)', fontsize=fnt, labelpad=labelpad)
ax2.set_xlabel('Organic loading rate (gCOD/m2.d)', fontsize=fnt, labelpad=labelpad)
ax1.set_ylabel('Intermediate aeration efficiency', fontsize=fnt, labelpad=labelpad)
ax2.set_ylabel('')
plt.show()
```

**Water depth versus aeration efficiency**

```python
review_cases_filtered.plot(kind='scatter', x='AE_int', y='water_depth_m', figsize=(10, 5), grid=True, fontsize=fnt)
plt.title('Water depth versus aeration efficiency', fontsize=16, pad=10)
plt.xlabel('Intermediate aeration efficiency', fontsize=fnt, labelpad=labelpad)
plt.ylabel('Water depth (m)', fontsize=fnt, labelpad=labelpad)
plt.show()
```

#### Separating the Forced Bed Aeration systems

```python
# building filter for separatin FBA studies
fba_list = ['3a', '3b', '4', '10a', '10b', '10c', '10d', '14a', '14b']

fba_cases = review_cases[review_cases['number'].isin(fba_list)]
fba_cases
```

      Reference
      number
      water_depth_m
      area_m2
      volume_m3
      porosity
      voids_m3
      HLR_lit_m2d
      daily_flow_lit
      OLR_gCOD_m2d
      ...
      OCR_max_a
      OCR_int_a
      OCR_min_a
      OC_max
      OC_int
      OC_min
      OI
      AE_max
      AE_int
      AE_min

      7
      (Boog et al., 2014)
      3a
      0.800
      6.2000
      4.960000
      0.40
      2.000000
      95.000000
      589.0
      26.88500
      ...
      72.834600
      38.779950
      15.201187
      1580.510820
      841.524915
      263.892615
      4.706599e+04
      3.358074
      1.787968
      0.560686

      8
      (Boog et al., 2014)
      3b
      0.800
      6.2000
      4.960000
      0.40
      2.000000
      95.000000
      589.0
      26.88500
      ...
      73.899550
      39.152350
      15.726062
      1603.620235
      849.605995
      273.004445
      3.137733e+04
      5.110761
      2.707707
      0.870069

      9
      (Butterworth et al., 2013)
      4
      0.550
      100.0000
      55.000000
      0.40
      30.250000
      272.000000
      27200.0
      19.40000
      ...
      16.586560
      6.857120
      9.500218
      1844.645000
      762.602500
      581.102500
      1.021538e+06
      0.180575
      0.074652
      0.056885

      21
      (Nivala et al., 2007)
      10a
      0.300
      93.0000
      27.900000
      0.35
      9.765000
      4.000000
      372.0
      2.93600
      ...
      4.610400
      1.870800
      4.102667
      11255.139000
      4567.090500
      3004.690500
      1.736041e+07
      0.064832
      0.026308
      0.017308

      22
      (Nivala et al., 2007)
      10b
      0.300
      93.0000
      27.900000
      0.35
      9.765000
      4.000000
      372.0
      3.09200
      ...
      4.562880
      1.877760
      3.812533
      11139.130800
      4584.081600
      2792.204100
      1.736041e+07
      0.064164
      0.026405
      0.016084

      23
      (Nivala et al., 2007)
      10c
      0.300
      93.0000
      27.900000
      0.35
      9.765000
      4.000000
      372.0
      3.49200
      ...
      6.668080
      2.737160
      5.637200
      16278.450300
      6682.091850
      4128.544350
      1.736041e+07
      0.093768
      0.038490
      0.023781

      24
      (Nivala et al., 2007)
      10d
      0.300
      93.0000
      27.900000
      0.35
      9.765000
      4.000000
      372.0
      2.51200
      ...
      4.132800
      1.641600
      4.012000
      10089.198000
      4007.556000
      2938.288500
      1.736041e+07
      0.058116
      0.023084
      0.016925

      31
      (Redmond et al., 2014)
      14a
      0.305
      5.4412
      1.659566
      NaN
      0.506168
      48.702492
      265.0
      31.07219
      ...
      33.759983
      16.164649
      6.645255
      606.192906
      290.251797
      36.393192
      3.940537e+03
      15.383512
      7.365794
      0.923559

      32
      (Redmond et al., 2014)
      14b
      0.305
      5.4412
      1.659566
      NaN
      0.506168
      48.702492
      265.0
      31.07219
      ...
      35.295475
      16.813172
      7.752798
      825.813882
      393.380469
      55.325004
      5.134639e+03
      16.083194
      7.661308
      1.077486

9 rows × 41 columns

```python
r_col = ['AE_max', 'AE_int', 'AE_min', 'OCR_max_v', 'OCR_int_v']

print("For FBA studies we have this values")

for col in r_col:
    avg = fba_cases[col].mean()
    median = fba_cases[col].median()
    min_v = fba_cases[col].min()
    max_v = fba_cases[col].max()
    std = fba_cases[col].std()

    if "max" in col:
        add = " maximum "
    elif "int" in col:
        add = " intermediate "
    else:
        add = " minimum "

    print("For" + add + "aeration efficinecy the:")
    print("- Average value is %0.2f%% and median value is %0.2f%%." % (avg, median))
    print("- Minimum value is %0.2f%% and maximum is %0.2f%%." % (min_v, max_v))
    print("- Standard deviation is %0.2f%%." % (std))
    print("")

avg_depth = fba_cases['water_depth_m'].mean()
print("Average depth is %0.2f m." % avg_depth)
print("")

avg_OCR = fba_cases['OCR_max_v'].mean()
median_OCR = fba_cases['OCR_max_v'].median()
min_OCR = fba_cases['OCR_max_v'].min()
max_OCR = fba_cases['OCR_max_v'].max()
print("Average OCR value is %0.2f and median value is %0.2f." % (avg_OCR, median_OCR))
print("Minimum OCR value is %0.2f and maximum is %0.2f." % (min_OCR, max_OCR))
```

    For FBA studies we have this values
    For maximum aeration efficinecy the:
    - Average value is 5.03% and median value is 1.73%.
    - Minimum value is 0.06% and maximum is 16.08%.
    - Standard deviation is 6.87%.

    For intermediate aeration efficinecy the:
    - Average value is 2.45% and median value is 0.91%.
    - Minimum value is 0.02% and maximum is 7.66%.
    - Standard deviation is 3.28%.

    For minimum aeration efficinecy the:
    - Average value is 0.44% and median value is 0.29%.
    - Minimum value is 0.02% and maximum is 1.08%.
    - Standard deviation is 0.47%.

    For maximum aeration efficinecy the:
    - Average value is 59.55% and median value is 56.64%.
    - Minimum value is 13.78% and maximum is 115.72%.
    - Standard deviation is 46.67%.

    For intermediate aeration efficinecy the:
    - Average value is 29.08% and median value is 28.80%.
    - Minimum value is 5.47% and maximum is 55.13%.
    - Standard deviation is 23.96%.

    Average depth is 0.43 m.

    Average OCR value is 59.55 and median value is 56.64.
    Minimum OCR value is 13.78 and maximum is 115.72.

```python
fig = plt.figure(figsize=(8, 5))
ax1 = fig.add_subplot(1, 1, 1)
fnt=14
labelpad=10

fba_cases[['AE_int']].plot(kind='hist',
                              rwidth=0.7,
                              bins=27,
                              legend=False,
                              xticks=range(0, 9, 1),
                              ax=ax1
                             )

ax1.set_ylabel('Number of cases', fontsize=fnt, labelpad=labelpad)
ax1.set_xlabel('Intermediate aeration efficiency (%) (FBA cases)', fontsize=fnt, labelpad=labelpad)

plt.show()
```

#### Average polutant reduction

```python
print("Average polutant efficiency:")
print("")

review_cases['eff_COD'] = (review_cases['COD_IN'] - review_cases['COD_OUT']) / review_cases['COD_IN'] * 100
review_cases['eff_TKN'] = (review_cases['TKN_IN'] - review_cases['TKN_OUT']) / review_cases['TKN_IN'] * 100
review_cases['eff_NH4'] = (review_cases['NH4_IN'] - review_cases['NH4_OUT']) / review_cases['NH4_IN'] * 100

r_col = ['eff_COD', 'eff_TKN', 'eff_NH4']

for col in r_col:
    avg = review_cases[col].mean()
    med = review_cases[col].median()

    if "COD" in col:
        print("Average COD reduction efficiency is %0.2f%%" % avg)
        print("Median COD reduction efficiency is %0.2f%%" % med)
    if "TKN" in col:
        print("Average TKN reduction efficiency is %0.2f%%" % avg)
        print("Median TKN reduction efficiency is %0.2f%%" % med)
    if "NH4" in col:
        print("Average NH4 reduction efficiency is %0.2f%%" % avg)
        print("Median NH4 reduction efficiency is %0.2f%%" % med)
    print("")
```

    Average polutant efficiency:

    Average COD reduction efficiency is 85.56%
    Median COD reduction efficiency is 93.46%

    Average TKN reduction efficiency is 85.39%
    Median TKN reduction efficiency is 95.64%

    Average NH4 reduction efficiency is 86.74%
    Median NH4 reduction efficiency is 92.99%

#### Correlation matrix

```python
review_cases.columns
```

    Index(['Reference', 'number', 'water_depth_m', 'area_m2', 'volume_m3',
           'porosity', 'voids_m3', 'HLR_lit_m2d', 'daily_flow_lit', 'OLR_gCOD_m2d',
           'HRT_h', 'HRT_days', 'air_flow_m3h', 'aer_duration_h_day',
           'air_per_day_m3', 'air_per_HRT_m3', 'COD_IN', 'COD_OUT', 'deltaCOD',
           'CBOD_IN', 'CBOD_OUT', 'deltaCBOD', 'TKN_IN', 'TKN_OUT', 'deltaTKN',
           'NH4_IN', 'NH4_OUT', 'deltaNH4', 'OCR_max_v', 'OCR_int_v', 'OCR_min_v',
           'OCR_max_a', 'OCR_int_a', 'OCR_min_a', 'OC_max', 'OC_int', 'OC_min',
           'OI', 'AE_max', 'AE_int', 'AE_min', 'eff_COD', 'eff_TKN', 'eff_NH4'],
          dtype='object')

```python
AE_corr = ['water_depth_m', 'area_m2', 'volume_m3', 'HLR_lit_m2d',
           'OLR_gCOD_m2d', 'HRT_days', 'air_flow_m3h', 'aer_duration_h_day',
           'AE_max']
```

```python
AE_review_cases = review_cases[[*AE_corr]]
AE_review_cases.columns
```

    Index(['water_depth_m', 'area_m2', 'volume_m3', 'HLR_lit_m2d', 'OLR_gCOD_m2d',
           'HRT_days', 'air_flow_m3h', 'aer_duration_h_day', 'AE_max'],
          dtype='object')

```python
AE_review_cases.corr()
```

      water_depth_m
      area_m2
      volume_m3
      HLR_lit_m2d
      OLR_gCOD_m2d
      HRT_days
      air_flow_m3h
      aer_duration_h_day
      AE_max

      water_depth_m
      1.000000
      0.602125
      0.617061
      0.699644
      0.468242
      -0.251322
      0.366829
      0.016899
      0.108795

      area_m2
      0.602125
      1.000000
      0.997962
      0.881686
      0.240732
      -0.023361
      0.812344
      0.193056
      0.063361

      volume_m3
      0.617061
      0.997962
      1.000000
      0.887795
      0.257333
      -0.079426
      0.774424
      0.168588
      0.069749

      HLR_lit_m2d
      0.699644
      0.881686
      0.887795
      1.000000
      0.581418
      -0.218217
      0.663513
      0.235477
      0.069916

      OLR_gCOD_m2d
      0.468242
      0.240732
      0.257333
      0.581418
      1.000000
      -0.312063
      0.049225
      0.051532
      0.068210

      HRT_days
      -0.251322
      -0.023361
      -0.079426
      -0.218217
      -0.312063
      1.000000
      0.425628
      0.263661
      -0.045951

      air_flow_m3h
      0.366829
      0.812344
      0.774424
      0.663513
      0.049225
      0.425628
      1.000000
      0.378768
      -0.019449

      aer_duration_h_day
      0.016899
      0.193056
      0.168588
      0.235477
      0.051532
      0.263661
      0.378768
      1.000000
      -0.211296

      AE_max
      0.108795
      0.063361
      0.069749
      0.069916
      0.068210
      -0.045951
      -0.019449
      -0.211296
      1.000000

```python
OCR_corr = ['water_depth_m', 'area_m2', 'volume_m3', 'HLR_lit_m2d',
           'OLR_gCOD_m2d', 'HRT_days', 'air_flow_m3h', 'aer_duration_h_day',
           'OCR_max_a']

OCR_review_cases = review_cases[[*OCR_corr]]
OCR_review_cases.columns
```

    Index(['water_depth_m', 'area_m2', 'volume_m3', 'HLR_lit_m2d', 'OLR_gCOD_m2d',
           'HRT_days', 'air_flow_m3h', 'aer_duration_h_day', 'OCR_max_a'],
          dtype='object')

```python
OCR_review_cases.corr()
```

      water_depth_m
      area_m2
      volume_m3
      HLR_lit_m2d
      OLR_gCOD_m2d
      HRT_days
      air_flow_m3h
      aer_duration_h_day
      OCR_max_a

      water_depth_m
      1.000000
      0.602125
      0.617061
      0.699644
      0.468242
      -0.251322
      0.366829
      0.016899
      0.416706

      area_m2
      0.602125
      1.000000
      0.997962
      0.881686
      0.240732
      -0.023361
      0.812344
      0.193056
      0.072514

      volume_m3
      0.617061
      0.997962
      1.000000
      0.887795
      0.257333
      -0.079426
      0.774424
      0.168588
      0.090193

      HLR_lit_m2d
      0.699644
      0.881686
      0.887795
      1.000000
      0.581418
      -0.218217
      0.663513
      0.235477
      0.457579

      OLR_gCOD_m2d
      0.468242
      0.240732
      0.257333
      0.581418
      1.000000
      -0.312063
      0.049225
      0.051532
      0.956408

      HRT_days
      -0.251322
      -0.023361
      -0.079426
      -0.218217
      -0.312063
      1.000000
      0.425628
      0.263661
      -0.306188

      air_flow_m3h
      0.366829
      0.812344
      0.774424
      0.663513
      0.049225
      0.425628
      1.000000
      0.378768
      -0.099401

      aer_duration_h_day
      0.016899
      0.193056
      0.168588
      0.235477
      0.051532
      0.263661
      0.378768
      1.000000
      0.031645

      OCR_max_a
      0.416706
      0.072514
      0.090193
      0.457579
      0.956408
      -0.306188
      -0.099401
      0.031645
      1.000000

```python
# from pandas.plotting import scatter_matrix
# scatter_matrix(OCR_review_cases, figsize=(15, 15), alpha=0.3)
```

#### Difference between continuos and intermitent aeration

```python
# filtering the continous cases
CA = review_cases[review_cases['aer_duration_h_day'] == 24]

# selecting only cases with AE_int less than 10 - more doesnt make sense
CA = CA[CA['AE_int'] < 10]
len(CA)
```

    21

```python
IA = review_cases[review_cases['aer_duration_h_day'] < 24]

# selecting only cases with AE_int less than 10 - more doesnt make sense
IA = IA[IA['AE_int'] < 10]

len(IA)
```

    27

So there are 21 review cases with continuous aeration and 27 (28 with one unrealistic case) with intermittent aeration.

```python
fig = plt.figure(figsize=(10, 5))
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)

fnt=16
labelpad=20

CA[['AE_int']].plot(kind='box',
                    ax=ax1,
                    fontsize=fnt,
                    ylim=(0, 8)
                   )
IA[['AE_int']].plot(kind='box',
                    ax=ax2,
                    fontsize=fnt,
                    ylim=(0, 8)
                   )

ax1.set_title('Continuous aeration', fontsize=fnt, pad=labelpad)
ax2.set_title('Intermittent aeration', fontsize=fnt, pad=labelpad)
fig.suptitle('Comparison of intermediate AE boxplos for CA and IA', fontsize=16, y=1.05)

plt.show()
```

```python
fig = plt.figure(figsize=(10, 5))
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)

fnt=16
labelpad=10
y_limits=(0, 200)

CA[['OCR_int_v']].plot(kind='box',
                    ax=ax1,
                    fontsize=fnt,
                    ylim=y_limits
                   )
IA[['OCR_int_v']].plot(kind='box',
                    ax=ax2,
                    fontsize=fnt,
                    ylim=y_limits
                   )

ax1.set_title('Continuous aeration', fontsize=fnt)
ax2.set_title('Intermittent aeration', fontsize=fnt)

plt.show()
```

```python
fig = plt.figure(figsize=(10, 5))
ax1 = fig.add_subplot(1, 2, 1)
ax2 = fig.add_subplot(1, 2, 2)

fnt=16
labelpad=10
y_limits=(0, 150)

CA[['OLR_gCOD_m2d']].plot(kind='box',
                    ax=ax1,
                    fontsize=fnt,
                    ylim=y_limits
                   )
IA[['OLR_gCOD_m2d']].plot(kind='box',
                    ax=ax2,
                    fontsize=fnt,
                    ylim=y_limits
                   )

ax1.set_title('Continuous aeration', fontsize=fnt)
ax2.set_title('Intermittent aeration', fontsize=fnt)

plt.show()
```

```python
r_col = ['AE_int', 'OCR_int_v', 'OLR_gCOD_m2d']

print("For CA studies we have this values")

for col in r_col:
    avg = CA[col].mean()
    median = CA[col].median()

    if "AE" in col:
        print("For intermediate aeration eficiency the values are:")
    elif "OCR" in col:
        print("For intermediate volumetric OCR the values are:")
    else:
        print("For OLR the values are:")

    print("- Average value is %0.2f and median value is %0.2f." % (avg, median))
    print("")

```

    For CA studies we have this values
    For intermediate aeration eficiency the values are:
    - Average value is 0.53 and median value is 0.07.

    For intermediate volumetric OCR the values are:
    - Average value is 31.84 and median value is 18.80.

    For OLR the values are:
    - Average value is 31.66 and median value is 13.80.

```python
r_col = ['AE_int', 'OCR_int_v', 'OLR_gCOD_m2d']

print("For IA studies we have this values")

for col in r_col:
    avg = IA[col].mean()
    median = IA[col].median()

    if "AE" in col:
        print("For intermediate aeration eficiency the values are:")
    elif "OCR" in col:
        print("For intermediate volumetric OCR the values are:")
    else:
        print("For OLR the values are:")

    print("- Average value is %0.2f and median value is %0.2f." % (avg, median))
    print("")
```

    For IA studies we have this values
    For intermediate aeration eficiency the values are:
    - Average value is 1.47 and median value is 0.61.

    For intermediate volumetric OCR the values are:
    - Average value is 39.48 and median value is 27.45.

    For OLR the values are:
    - Average value is 28.66 and median value is 27.28.


```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir( os.path.join('..', '..', 'notebook_format') )
from formats import load_style
load_style(css_style = 'custom2.css')
```

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2

from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor

%watermark -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    2017-03-14 15:58:58

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.12.0
    pandas 0.19.2
    matplotlib 2.0.0
    sklearn 0.18

# Gradient Boosting Machine (GBM)

Just like [Random Forest and Extra Trees](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/trees/random_forest.ipynb), Gradient Boosting Machine is also a type of Ensemble Tree method, the only difference is it is stemmed from the the boosting framework. The idea of boosting is to add a weak classifier to the ensemble at a time, and this newly added weak classifier is trained to improve on the already trained ensemble. Meaning it will pay higher attention on examples which are misclassiﬁed or have higher errors and focus on mitigating those errors. Boosting is a general framework can be applied to any sort of weak learner, although Decision Tree models is by far the commonly used due to the fact that they have the flexibility to be weak learners by simply restricting their depth and they are quite fast to train.

Suppose we are given some dataset $(x_1, y_1), (x_2, y_2), ...,(x_n, y_n)$, and the task is to fit a model $F(x)$ to minimize square loss. After training the model, we discovered the model is good but not perfect.
There are some mistakes: $F(x_1) = 0.8$, while $y_1 = 0.9$, and $F(x_2) = 1.4$ while $y_2 = 1.3$ .... Now the question is, how can we improve this model without changing anything from $F(x)$?

How about we simply add an additional model (e.g. regression tree) $h$ to the already existing $F$, so the new prediction becomes $F(x) + h(x)$. In other words, we wish to improve upon the existing model so that $F(x_1) + h(x_1) = y_1, F(x_2) + h(x_2) = y_2 ...$ or equivalent we wish to a new model $h$ such that $h(x_1) = y_1 - F(x_1), h(x_2) = y_2 - F(x_2) ...$. The idea is all well and good, but the bad news is probably no model $h$ (e.g. regression tree) will be able to do this perfectly. Fortunately, the good news is, some $h$ might be able to do this approximately.

The idea is, we fit the model $h$ to the data using $y_1 - F(x_1), y_2 - F(x_2)$ as the response variable. And the intuition for this is: the $y_i - F(x_i)$s are the residuals. These are the areas that the existing
model $F$ cannot do well, so now the role of $h$ is to compensate the shortcoming of existing model $F$. And if the model after adding the new model $h$, $F + h$ is still unsatisfactory, we will just add another new one.

To make sure we're actually learning the residuals, we'll employ the idea of gradient descent. Say our goal is to minimize $J$, an overall loss function additively calculated from all observations with regard to $F$, a classifier with some parameters. More formally, we're given the formula:

$$J(y, F) = \sum_i^n L\big(y_i, F(x_i)\big)$$

Where:

- $L$ is a cost/loss function comparing the response variable's value and the prediction of the model for each observation

Instead of trying to solve it directly, gradient descent is an iterative technique that allows us to approach the solution of an optimization problem. At each step of the algorithm, it will perform the following operations:

$$F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$$

Where:

- $F_b$ is the version of classifier at step/iteration $b$
- $\eta$ is the learning rate which controls the size of the learning process
- $\nabla$ is the gradient i.e. the first order partial derivative of the cost function with respect to the classifier
- The formula above actually refers to stochastic gradient descent only computing the function for a single observation, $x_i$

For example, say we're given, sum squares of errors, a well-known quality indicator for regression model as our loss function. So now our loss function $L\big(y_i, F(x_i)\big)$ is defined as: $\frac{1}{2} \big( y_i - F(x_i) \big)^2$ (the 1/2 is simply to make the notation cleaner later). Taking the gradient of this loss function we get:

$$\frac{ \partial L\big(y_i, F(x_i)\big) }{ \partial F(x_i) } = \frac{ \partial \frac{1}{2} \big( y_i - F(x_i) \big)^2 }{ \partial F(x_i) } = F(x_i) - y_i$$

Tying this back to our original problem, we wish to update our function $F$ at iteration $b$ with a new model $h$:

\begin{align}
F_b(x_i) &= F_{b-1}(x_i) + h(x_i) \nonumber \\
         &= F_{b-1}(x_i) + y_i - F_{b-1}(x_i) \nonumber \\
         &= F_{b-1}(x_i) - 1 \times \frac{ \partial L\big(y_i, F_{b-1}(x_i)\big) }{ \partial F_{b-1}(x_i) }
         \nonumber \\
\end{align}

As we can see, the formula above is 99% the same as as the gradient descent formula, $F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$. The only difference is that the learning rate $\eta$ is  1. Thus, we now have an iterative process constructing the additive model that minimizes our loss function (residuals).

In practice though, Gradient Boosting Machine is more prone to overfitting, since the week learner is tasked with optimally fitting the gradient. This means that boosting will select the optimal learner at each stage of the algorithm, although this strategy generates an optimal solution at the current stage, it has the drawbacks of not finding the optimal global model as well as overfitting the training data. A remedy for greediness is to constrain the learning process by setting the learning rate $\eta$ (also known as shrinkage). In the above algorithm, instead of directly adding the predicted value for a sample to next iteration's predicted value, so that only a fraction of the current predicted value is added to the previous iteration's predicted value. This parameter can take values between 0 and 1 and becomes another tuning parameter for the model. Small values of the learning parameter such as 0.1 tends to work better, but the value of the parameter is inversely proportional to the computation time required to find an optimal model, because more iterations are necessary.

To sum it all up in english the process of training a gradient boosting machine for regression is:

1. Initialize a predicted value for each observation (e.g. the original response or the average response or a value that minimizes the loss function). This will be our initial "residuals", $r$ (it can be called the residuals because we're dealing with a regression task, but often times this quantity is referred to as the negative gradient, so that the $- \nabla \times L\big(y_i, F(x_i) \big)$ part generalizes to any loss function we might want to use).
2. For step = 1 to $B$ (number of iterations that we specify) do:

    - Fit a regression tree $F_b$ to the training data $(X, r)$, where we use the residuals as the response variable
    - Update model $F$ by adding a shrunken version of the newly fitted regression tree. In code, this this basically mean we append the new tree to the array of trees we've already stored:
    $F_{b + 1}(X) = F_{b}(X) + \eta F_{b}(X)$
    - Update each observation's residual by adding the predicted value to it:
    $r_{b + 1} = r_b - \eta F_b(X)$
3. Our final output boosted model is simply $F(x) = \sum_{b = 1}^B \eta F_b(x)$, where we sum the values that each individual tree gives (times the learning rate).

## Implementation

Here, we will use the [Wine Quality Data Set](https://archive.ics.uci.edu/ml/datasets/Wine+Quality) to test our implementation. This [link](https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv) should download the .csv file. The task is to predict the quality of the wine (a scale of 1 ~ 10) given some of its features.

```python
# read in the data and shuffle the row order for model stability
np.random.seed(4321)
wine_path = os.path.join('..', 'winequality-white.csv')
wine = pd.read_csv(wine_path, sep = ';')
wine = wine.sample(frac = 1)

# train/test split the features and response column
y = wine['quality'].values
X = wine.drop('quality', axis = 1).values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1234)

print('dimension of the dataset: ', wine.shape)
wine.head()
```

    dimension of the dataset:  (4898, 12)

      fixed acidity
      volatile acidity
      citric acid
      residual sugar
      chlorides
      free sulfur dioxide
      total sulfur dioxide
      density
      pH
      sulphates
      alcohol
      quality

      857
      8.2
      0.40
      0.48
      13.70
      0.042
      59.0
      169.0
      0.99860
      3.10
      0.52
      9.4
      5

      2325
      7.0
      0.35
      0.17
      1.10
      0.049
      7.0
      119.0
      0.99297
      3.13
      0.36
      9.7
      6

      279
      7.0
      0.30
      0.49
      4.70
      0.036
      17.0
      105.0
      0.99160
      3.26
      0.68
      12.4
      7

      1687
      7.3
      0.26
      0.33
      17.85
      0.049
      41.5
      195.0
      1.00000
      3.06
      0.44
      9.1
      7

      531
      6.4
      0.18
      0.48
      4.00
      0.186
      64.0
      150.0
      0.99450
      3.06
      0.40
      9.3
      5

```python
class GBMReg:
    """
    regression gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """
    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        self.estimators = []

        # simply use the response as the original residuals
        # and covert it to float type to prevent error warning
        # that it's converting from int to float
        residual = y.astype(np.float)
        for i in range(self.n_estimators):
            tree = DecisionTreeRegressor(max_depth = self.max_depth)
            tree.fit(X, residual)
            y_pred = tree.predict(X)
            self.estimators.append(tree)
            residual -= self.learning_rate * y_pred

        return self

    def predict(self, X):
        y_pred = np.zeros(X.shape[0])
        for tree in self.estimators:
            y_pred += self.learning_rate * tree.predict(X)

        return y_pred
```

```python
# compare the results between a single decision tree,
# gradient boosting, the lower the mean square
# error, the better
tree = DecisionTreeRegressor(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print( 'tree: ', mean_squared_error(y_test, tree_y_pred) )

# library to confirm result
gbm_reg = GBMReg(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm_reg.fit(X_train, y_train)
gbm_reg_y_pred = gbm_reg.predict(X_test)
print( 'gbm: ', mean_squared_error(y_test, gbm_reg_y_pred) )

# gradient boosting for 100 trees and learning rate of 0.1
gbm = GradientBoostingRegressor(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print( 'gbm library: ', mean_squared_error(y_test, gbm_y_pred) )
```

    tree:  0.547700721197
    gbm:  0.420772942612
    gbm library:  0.422106982302

Clearly, Gradient Boosting has some similarities to Random Forests and Extra Trees: the final prediction is based on an ensemble of models, and trees are used as the base learner, so all the tuning parameters for the tree model also controls the variability of Gradient Boosting. And for interpretability we can also access the feature importance attribute.

```python
def viz_importance(model, feature_names, n_features):
    """Visualize the relative importance of predictors"""
    # sort the importance in decreasing order
    importances = model.feature_importances_
    idx = np.argsort(importances)[-n_features:]
    names = feature_names[idx]
    scores = importances[idx]

    y_pos = np.arange(1, n_features + 1)
    plt.barh(y_pos, scores, color = 'lightskyblue', align = 'center')
    plt.yticks(y_pos, names)
    plt.xlabel('Importance')
    plt.title('Feature Importance Plot')
```

```python
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12
viz_importance(gbm, wine.columns[:-1], X.shape[1])
```

But the way the ensembles are constructed differs substantially between each model. In Random Forests and Extra Trees, all trees are created independently and each tree contributes equally to the final model. The trees in Gradient Boosting, however, are dependent on past trees and contribute unequally to the final model. Despite these differences, Random Forests, Extra Trees and Gradient Boosting all offer competitive predictive performance (Gradient Boosting often wins when carefully tuned). As for computation time, Gradient Boosting is often greater than for Random Forests, Extra Trees, since their procedure can be easily parallel processed given that the trees are created independently.

## Classification

Gradient Boosting Machine can also be extended to handle classification tasks, as we'll soon see, even in the classification context, the underlying algorithm is still a regression tree. To adapt the algorithm to a classification process, we start by defining a new loss function, cross entropy (also known as multinomial deviance), denoted as:

$$L\big(y_i, F(x_i)\big) = -\sum_k ^ K y_k(x_i) \log p_k(x_i)$$

The notation above says:

- We have a total of $K$ output class (categorical response variable) that ranges from $1, ..., K$
- $y_k(x_i)$ is a dummy indicator of the response variable that takes the value of 1 if the $i_{th}$ observation belongs to class $k$ and 0 otherwise
- $p_k(x_i)$ is the predicted probability of the $i_{th}$ observation belonging to class $k$

So the next question is how do we get $p_k(x_i)$?

### Softmax

Softmax function takes an $N$-dimensional vector of arbitrary real values and produces another $N$-dimensional vector with real values in the range (0, 1) that add up to 1. The function's formula can be written as:

$$p_i = \frac{e^{o_i}}{\sum_k^K e^{o_k}}$$

For example, in the following code chunk, we see that how the softmax function transforms a 3-element vector 1.0, 2.0, 3.0 into probabilities that sums up to 1, while still preserving the relative size of the original elements.

```python
def compute_softmax(x):
    """compute the softmax of vector"""
    exp_x = np.exp(x)
    softmax = exp_x / np.sum(exp_x)
    return softmax

# this can be interpreted as the probability
# of belonging to the three classes
compute_softmax([1, 2, 3])
```

    array([ 0.09003057,  0.24472847,  0.66524096])

Next, we wish to compute the derivative of this function with respect to the input $o_i$ so we can use it later when computing the derivative of the loss function. To be explicit we wish to find:

$$\frac{\partial p_i}{\partial o_j} = \frac{\partial \frac{e^{o_i}}{\sum_{k=1}^{N}e^{o_k}}}{\partial o_j}$$

For any arbitrary output $i$ and input $j$. To do so, We'll be using the quotient rule of derivatives. The rule tells us that for a function $f(x) = \frac{g(x)}{h(x)}$:

$$f'(x) = \frac{g'(x)h(x) - h'(x)g(x)}{[h(x)]^2}$$

In our case, we have:

\begin{align*}
g &= e^{o_i} \nonumber \\
h &= \sum_{k=1}^{K}e^{o_k} \nonumber
\end{align*}

It's important to notice that no matter which $o_j$ we compute the derivative of $h$ for the output will always be $e^{o_j}$. Howewer, this is not the case for $g$. It's derivative will be $e^{o_j}$ only if $i = j$, because only then will it have the term $e^{o_j}$. Otherwise, the derivative is simply 0 (because it's simply taking the derivative of a constant).

So going back to using our quotient rule, we start with the $i = j$ case. In the following derivation we'll use the $\Sigma$ (Sigma) sign to represent $\sum_{k=1}^{K}e^{o_k}$ for simplicity and to prevent cluttering up the notation.

\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{e^{o_i}\Sigma-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= \frac{e^{o_i}}{\Sigma}\frac{\Sigma - e^{o_j}}{\Sigma} \nonumber \\
&= p_i(1 - p_j) \nonumber \\
&= p_i(1 - p_i) \nonumber
\end{align*}

The reason we can perform the operation in the last line is because we're considering the scenario where $i = j$. Similarly we can do the case where $i \neq j$.

\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{0-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= -\frac{e^{o_j}}{\Sigma}\frac{e^{o_i}}{\Sigma} \nonumber \\
&= -p_j p_i \nonumber \\
&= -p_i p_j \nonumber
\end{align*}

Just to sum it up, we now have:

$$\frac{\partial p_i}{\partial o_j} = p_i(1 - p_i),\quad i = j$$

$$\frac{\partial p_i}{\partial o_j} = -p_i p_j,\quad i \neq j$$

Given this, we can tie this back to the original loss function $-\sum_k^K y_k \log p_k \nonumber \\$ and compute it's negative gradient.

\begin{align}
\frac{\partial L}{\partial o_i}
&= -\sum_k y_k\frac{\partial \log p_k}{\partial o_i} \nonumber \\
&= -\sum_k y_k\frac{1}{p_k}\frac{\partial p_k}{\partial o_i} \nonumber \\
&= -y_i(1-p_i) - \sum_{k \neq i}y_k\frac{1}{p_k}(-p_kp_i) \nonumber \\
&= -y_i(1 - p_i) + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= -y_i + y_i p_i + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= p_i\left(\sum_ky_k\right) - y_i \nonumber \\
&= p_i - y_i \nonumber
\end{align}

Remember $\sum_ky_k=1$ (as $y$ is a vector with only one non-zero element, which is $1$ when the indicating the observation belongs to the $k_{th}$ class.

After a long journey, we now see, for every class $k$, the gradient is the difference between the associated dummy variable and the predicted probability of belonging to that class. This is essentially the "residuals" from the classification gradient boosting. Given this, we can now implement the algorithm, the overall process of training a regression tree has still not changed, only now we must deal with the dummy variables, $y_k$ and fit a regression tree on the negative gradient for each dummy variable.

## Implementation

For the dataset, we'll still use the Wine Quality Data Set that was used for the regression task, except we now treat the quality of the wine (a scale of 1 ~ 10) as categorical instead of numeric.

```python
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier
```

```python
class GBMClass:
    """
    classification gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """
    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        # encode labels with value between 0 and n_classes - 1,
        # so we can easily one-hot encode them
        self.le = LabelEncoder()
        labels = self.le.fit_transform(y)
        Y = self._to_categorical(labels)
        del labels

        # the predicted probability starts out with
        # a value that's uniform over all classes;
        # then we compute the residuals (negative gradient),
        # which is the difference between the predicted
        # probability and the class label
        y_proba = np.full( Y.shape, 1 / Y.shape[1] )
        residuals = Y - y_proba

        # train a base decision tree on the residuals
        # for every single class, hence we end up with
        # n_estimators * n_classes base tree models
        self.estimators = []
        for i in range(self.n_estimators):
            for j in range(self.n_classes):
                tree = DecisionTreeRegressor(max_depth = self.max_depth)
                tree.fit(X, residuals[:, j])
                y_pred = tree.predict(X)
                self.estimators.append(tree)
                residuals[:, j] -= self.learning_rate * y_pred

        return self

    def _to_categorical(self, y):
        """one hot encode class vector y"""
        self.n_classes = np.amax(y) + 1
        Y = np.zeros(( y.shape[0], self.n_classes ))
        for i in range(y.shape[0]):
            Y[i, y[i]] = 1.0

        return Y

    def predict(self, X):
        # after predicting the class remember to
        # transform it back to the actual class label
        y_prob = self.predict_proba(X)
        y_pred = np.argmax(y_prob, axis = 1)
        y_pred = self.le.inverse_transform(y_pred)
        return y_pred

    def predict_proba(self, X):
        # add up raw score for every class and convert
        # it to probability using softmax
        y_raw = np.zeros(( X.shape[0], self.n_classes ))

        # obtain the tree for each class and add up the prediction
        for c in range(self.n_classes):
            class_tree = self.estimators[c::self.n_classes]
            for tree in class_tree:
                y_raw[:, c] += self.learning_rate * tree.predict(X)

        y_proba = self._compute_softmax(y_raw)
        return y_proba

    def _compute_softmax(self, z):
        """
        compute the softmax of matrix z in a numerically stable way,
        by substracting each row with the max of each row
        """
        shift_z = z - np.amax(z, axis = 1, keepdims = 1)
        exp_z = np.exp(shift_z)
        softmax = exp_z / np.sum(exp_z, axis = 1, keepdims = 1)
        return softmax
```

```python
# compare the results between a single decision tree,
# gradient boosting, the higher the accuracy, the better
tree = DecisionTreeClassifier(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print( 'tree: ', accuracy_score(y_test, tree_y_pred) )

# gradient boosting for 150 trees and learning rate of 0.2
# unlike random forest, gradient boosting's base tree can be shallower
# meaning that there depth can be smaller
gbm_class = GBMClass(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm_class.fit(X_train, y_train)
gbm_class_y_pred = gbm_class.predict(X_test)
print( 'gbm: ', accuracy_score(y_test, gbm_class_y_pred) )

# library to confirm results are comparable
gbm = GradientBoostingClassifier(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print( 'gbm library: ', accuracy_score(y_test, gbm_y_pred) )
```

    tree:  0.54387755102
    gbm:  0.601020408163
    gbm library:  0.616326530612

## Understanding Model Complexity

In the following section, we generate a Sinoide function + random gaussian noise, with 80 training samples (blue points) and 20 test samples (red points).

```python
def ground_truth(x):
    """Ground truth -- function to approximate"""
    return x * np.sin(x) + np.sin(2 * x)

def gen_data(low, high, n_samples):
    """generate training and testing data from the ground truth function"""
    np.random.seed(15)
    X = np.random.uniform(low, high, size = n_samples)

    # generate the response from the ground truth function and add
    # some random noise to it
    y = ground_truth(X) + np.random.normal(scale = 2, size = n_samples)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 3)
    return X_train, X_test, y_train, y_test

def plot_data(x_plot, X_train, X_test, y_train, y_test):
    """plot training and testing data"""
    s = 20
    alpha = 0.4
    plt.plot(x_plot, ground_truth(x_plot), alpha = alpha, label = 'ground truth')
    plt.scatter(X_train, y_train, s = s, alpha = alpha)
    plt.scatter(X_test, y_test, s = s, alpha = alpha, color = 'red')
    plt.xlim(( 0, 10 ))
    plt.ylabel('y')
    plt.xlabel('x')
    plt.legend(loc = 'upper left')
    plt.show()
```

```python
low = 0
high = 10
x_plot = np.linspace(low, high, 500)
X_train, X_test, y_train, y_test = gen_data(low = low, high = high, n_samples = 100)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Recall that in a single regression tree, we can use the `max_depth` parameter to control how deep to grow the tree and the deeper the tree the more variance can be explained.

```python
# when using scikit-learn, the training data has to be
# a 2d-array even if it only has 1 features
tree1 = DecisionTreeRegressor(max_depth = 1)
tree1.fit(X_train[:, np.newaxis], y_train)
tree2 = DecisionTreeRegressor(max_depth = 3)
tree2.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, tree1.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=1', color = 'g', alpha = 0.9, linewidth = 2)
plt.plot(x_plot, tree2.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=3', color = 'g', alpha = 0.7, linewidth = 1)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

The plot above shows that the decision boundaries made by decision trees are always perpendicular to $x$ and $y$ axis (due to the fact that they consists of nested if-else statements). Let's see what happens when we use gradient boosting without tuning the parameters (by specifying a fix `max_depth`).

```python
gbm = GradientBoostingRegressor(n_estimators = 300, max_depth = 6, learning_rate = 0.1)
gbm.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM max_depth=6', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Hopefully, it should be clear that compared with decision trees, gradient boosting machine is far more susceptible to overfitting the training data, hence it is common to tune parameters including `max_depth`, `max_features`, `min_samples_leaf`, `subsample` (explained below) to reduce the overfitting phenomenon from  occurring.

The parameter `subsample` (technically called *stochastic gradient boosting*) borrows some idea from bagging techniques. What it does is: while iterating through each individual tree building process, it randomly select a fraction of the training data. Then the residuals and models in the remaining steps of the current iteration are based only on that sample of data. It turns out that this simple modification improved the predictive accuracy of boosting while also reducing the required computational resources (of course, this is based on the fact that you have enough observations to subsample).

The following section tunes the commonly tuned parameter and find the best one and draws the decision boundary. The resulting plot should be self-explanatory.

```python
param_grid = {
    'max_depth': [4, 6],
    'min_samples_leaf': [3, 5, 8],
    'subsample': [0.9, 1]
    # 'max_features': [1.0, 0.3, 0.1] # not possible in this example (there's only 1)
}
gs_gbm = GridSearchCV(gbm, param_grid, scoring = 'neg_mean_squared_error', n_jobs = 4)
gs_gbm.fit(X_train[:, np.newaxis], y_train)
print('Best hyperparameters: %r' % gs_gbm.best_params_)

plt.plot(x_plot, gs_gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM tuned', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

# Reference

- [Slide: Gradent boosting](http://eric.univ-lyon2.fr/~ricco/cours/slides/en/gradient_boosting.pdf)
- [Slide: A gentle introduction to gradient boosting](http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf)
- [Blog: The Softmax function and its derivative](http://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)
- [Notebook: Regression Trees and Rule-Based Models](http://nbviewer.jupyter.org/github/leig/Applied-Predictive-Modeling-with-Python/blob/master/notebooks/Chapter%208.ipynb)
- [Notebook: Gradient Boosted Regression Trees in scikit-learn](http://nbviewer.jupyter.org/github/pprett/pydata-gbrt-tutorial/blob/master/gbrt-tutorial.ipynb)
- [StackExchange: Derivative of Softmax loss function](http://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function)
- [StackExchange: How are individual trees added together in boosted regression tree?](http://stats.stackexchange.com/questions/135378/how-are-individual-trees-added-together-in-boosted-regression-tree)