# CBMPy Tutorial 01 Getting Started

## Checking dependencies and loading CBMPy

PySCeS CBMPy is a flexible framework for contraint based modelling. With this tutorial we setup a CBMPy session and load a model. To begin with let's see if we have all the basic dependencies installed. For more information please see the CBMPy reference guide (available from http://cbmpy.sourceforge.net).

No additional files are required for this tutorial.

```python
# Required for basic CBMPy functionality
import numpy

# extended functionaliy
import libsbml
print('\nYou are using libSBML version {}'.format(libsbml.LIBSBML_VERSION_STRING))
if libsbml.LIBSBML_VERSION > 51100 and libsbml.LIBSBML_VERSION < 51106:
    print('Your version of libSBML supports SBML 3 FBC version 1, please consider upgrading to the latest version for FBC version 2 support.\n')
elif libsbml.LIBSBML_VERSION >= 51106:
    print('Your version of libSBML supports SBML 3 FBC versions 1 and 2.\n')
else:
    print('You are using an extrememly old version of libSBML please upgrade now!\n')

# optional, but highly recommended for rerading/writing excel spreadsheets
import xlrd
import xlwt
# sympy is only required for methods utilizing exact arithmetic
try:
    import sympy
except ImportError:
    print('\nSympy is only required for selected methods utilizing exact arithmetic and is not required for normal functionality.')

```

    You are using libSBML version 51401
    Your version of libSBML supports SBML 3 FBC versions 1 and 2.

If any of of the above imports failed consider installing them. Please see the online documentation for comprehensive installation [instructions](http://cbmpy.sourceforge.net/reference/cbmpy.html). Next we need to import CBMPy, at this point it is convenient to alias the imported module. In all tutorials this will be `cbm`.

```python
import cbmpy
```

    *****
    Using CPLEX
    *****

    WX GUI tools available.

    CBMPy environment
    ******************
    Revision: r562

    ***********************************************************************
    * Welcome to CBMPy (0.7.11) - PySCeS Constraint Based Modelling        *
    *                http://cbmpy.sourceforge.net                         *
    * Copyright(C) Brett G. Olivier 2014 - 2016                           *
    * Dept. of Systems Bioinformatics                                     *
    * Vrije Universiteit Amsterdam, Amsterdam, The Netherlands            *
    * CBMPy is developed as part of the BeBasic MetaToolKit Project       *
    * Distributed under the GNU GPL v 3.0 licence, see                    *
    * LICENCE (supplied with this release) for details                    *
    ***********************************************************************

You should some status text and a version of the welcome screen:
```
***********************************************************************
* Welcome to CBMPy (0.7.4) - PySCeS Constraint Based Modelling        *
*                http://cbmpy.sourceforge.net                         *
* Copyright(C) Brett G. Olivier 2014 - 2016                           *
* Dept. of Systems Bioinformatics                                     *
* Vrije Universiteit Amsterdam, Amsterdam, The Netherlands            *
* CBMPy is developed as part of the BeBasic MetaToolKit Project       *
* Distributed under the GNU GPL v 3.0 licence, see                    *
* LICENCE (supplied with this release) for details                    *
***********************************************************************
```

## Testing the installation, loading the default models

CBMPy now includes a growing set of Nose based unittests, try run them to test your installation (this may take a few minutes) unfortunately the test runner does not seem to like the Notebook environment but try the following in a console:

```import cbmpy
cbmpy.test()```

In addition CBMPy is also shipped with two models that can be used to test your installation and get going quickly. In order to use these models at least SBML FBC version 1 support is required and both models can be loaded using the *special* filenames: `cbmpy_test_core` and  `cbmpy_test_ecoli`.

In CBMPy models are *read* from files and *instantiated* as Python objects. Here I will instantiated two model objectes from the two test files: `cmod` and `ecoli` but you can use any Python name. Note it is possible to instantiate multiple *independent* instances from the same input file.

```python
cmod = cbmpy.readSBML3FBC('cbmpy_test_core')
ecoli = cbmpy.readSBML3FBC('cbmpy_test_ecoli')
```

    core_memesa_model.l3.xml
    FBC version: 1
    M.getNumReactions: 26
    M.getNumSpecies: 22
    FBC.getNumObjectives: 1
    FBC.getNumGeneAssociations: 0
    FBC.getNumFluxBounds: 52
    Zero dimension compartment detected: Cell
    INFO: Active objective: objMaxJ25
    Adding objective: objMaxJ25

    SBML3 load time: 0.023

    INFO: no standard gene encoding detected, attempting to load from annotations.
    INFO: used key(s) '[]'
    INFO: Added 0 new genes and 0 associations to model
    Ecoli_iJR904.glc.l3.xml
    FBC version: 1
    M.getNumReactions: 1066
    M.getNumSpecies: 904
    FBC.getNumObjectives: 1
    FBC.getNumGeneAssociations: 1066
    FBC.getNumFluxBounds: 2132
    INFO: Active objective: obj1
    Adding objective: obj1

    SBML3 load time: 1.264

Once model objects have been instantiated they can be modified and analysed, here we will perform FBA on each model and test that CBMPy is calculating the correct result:

```python
print('\n*****\nCalculating FBA for model: core\n*****')
optcm = cbmpy.analyzeModel(cmod)

print('\n*****\nCalculating FBA for model: ecoli\n*****')
optec = cbmpy.analyzeModel(ecoli)

if 1.0 - abs(round(optcm, 2)) < 1.2e-10:
    print('\nCore model test FBA: PASSED.\n Value expected: 1.0, calculated: {}'.format(round(optcm, 2)))
else:
    print('\nCore model test FBA: FAILED.\n Value expected: 1.0, calculated: {}'.format(round(optcm, 2)))
if 1.0 - abs(round(optec, 2)) < 1.2e-10:
    print('\nE. Coli model test FBA: PASSED.\n Value expected: 1.0, calculated: {}'.format(round(optec, 2)))
else:
    print('\nE. Coli model test FBA: FAILED.\n Value expected: 1.0, calculated: {}'.format(round(optec, 2)))

```

    *****
    Calculating FBA for model: core
    *****

    cplx_constructLPfromFBA time: 0.0090000629425

    cplx_analyzeModel FBA --> LP time: 0.0090000629425

    Tried aggregator 1 time.
    LP Presolve eliminated 0 rows and 4 columns.
    Aggregator did 11 substitutions.
    Reduced LP has 8 rows, 11 columns, and 20 nonzeros.
    Presolve time = 0.00 sec. (0.02 ticks)
    Initializing dual steep norms . . .

    Iteration log . . .
    Iteration:     1   Dual objective     =          1000.000000
    INFO: Model is optimal: 1
    Solution status =  1 : optimal
    Solution method =  2 : dual
    Objective value =  1.0
    Model is optimal
    Status: LPS_OPT
    Model is optimal
    Model is optimal

    analyzeModel objective value: 1.0

    *****
    Calculating FBA for model: ecoli
    *****

    cplx_constructLPfromFBA time: 0.0590000152588

    cplx_analyzeModel FBA --> LP time: 0.0599999427795

    Tried aggregator 1 time.
    LP Presolve eliminated 273 rows and 354 columns.
    Aggregator did 230 substitutions.
    Reduced LP has 258 rows, 482 columns, and 2449 nonzeros.
    Presolve time = 0.02 sec. (0.98 ticks)
    Initializing dual steep norms . . .

    Iteration log . . .
    Iteration:     1   Dual objective     =       1073881.855205
    Iteration:    62   Dual objective     =        207254.367443
    Iteration:   124   Dual objective     =        158540.756315
    Iteration:   186   Dual objective     =         49426.164215
    Iteration:   248   Dual objective     =             1.274029
    INFO: Model is optimal: 1
    Solution status =  1 : optimal
    Solution method =  2 : dual
    Objective value =  1.0
    Model is optimal
    Status: LPS_OPT
    Model is optimal
    Model is optimal

    analyzeModel objective value: 1.0

    Core model test FBA: PASSED.
     Value expected: 1.0, calculated: 1.0

    E. Coli model test FBA: PASSED.
     Value expected: 1.0, calculated: 1.0

If both of the above tests passed, congratulations you are ready to model with CBMPy. In tutorial 2 we will look in more detail at model ecoding formats that can be used with CBMPy.


```python
print("Hello, World!")
```

    Hello, World!

```python
import time
time.sleep(3)
```

```python
def say_hello(recipient):
    return 'Hello, {}!'.format(recipient)

say_hello('Bugx')
```

    'Hello, Bugx!'

# Title
## Section
### Title

\begin{equation}
y = ax^2 + bx + c + \frac{e^x}{\tan x}
\end{equation}

```python

```


```
# Data: US regional unemployment data from Bureau of Labor Statistics
! pip install qeds
```

    Collecting qeds
      Downloading https://files.pythonhosted.org/packages/41/50/509c79a019156862898acb9e23b28e872818c20492da187090167f6702ad/qeds-0.6.2.tar.gz
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from qeds) (1.0.5)
    Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from qeds) (2.23.0)
    Collecting quandl
      Downloading https://files.pythonhosted.org/packages/1b/29/185269dbd2e2698c8098b35c52ce73a2c52cf76163e709f9f7789d03ebbb/Quandl-3.5.2-py2.py3-none-any.whl
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from qeds) (1.4.1)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from qeds) (1.18.5)
    Collecting quantecon
      Downloading https://files.pythonhosted.org/packages/5d/e3/4fd5f948de917036b5696347b28fa25da7bd7df995e4f9f42db1c3070eb8/quantecon-0.4.8-py3-none-any.whl (230kB)
         |████████████████████████████████| 235kB 6.0MB/s
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from qeds) (3.2.2)
    Requirement already satisfied: pyarrow in /usr/local/lib/python3.6/dist-packages (from qeds) (0.14.1)
    Requirement already satisfied: openpyxl in /usr/local/lib/python3.6/dist-packages (from qeds) (2.5.9)
    Requirement already satisfied: plotly in /usr/local/lib/python3.6/dist-packages (from qeds) (4.4.1)
    Requirement already satisfied: pandas_datareader in /usr/local/lib/python3.6/dist-packages (from qeds) (0.8.1)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from qeds) (0.22.2.post1)
    Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from qeds) (0.10.1)
    Requirement already satisfied: statsmodels in /usr/local/lib/python3.6/dist-packages (from qeds) (0.10.2)
    Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->qeds) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->qeds) (2018.9)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->qeds) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->qeds) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->qeds) (2020.6.20)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->qeds) (3.0.4)
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from quandl->qeds) (8.5.0)
    Collecting inflection>=0.3.1
      Downloading https://files.pythonhosted.org/packages/59/91/aa6bde563e0085a02a435aa99b49ef75b0a4b062635e606dab23ce18d720/inflection-0.5.1-py2.py3-none-any.whl
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from quandl->qeds) (1.15.0)
    Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from quantecon->qeds) (1.1.1)
    Requirement already satisfied: numba>=0.38 in /usr/local/lib/python3.6/dist-packages (from quantecon->qeds) (0.48.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->qeds) (2.4.7)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->qeds) (1.2.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->qeds) (0.10.0)
    Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.6/dist-packages (from openpyxl->qeds) (1.0.1)
    Requirement already satisfied: jdcal in /usr/local/lib/python3.6/dist-packages (from openpyxl->qeds) (1.4.1)
    Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.6/dist-packages (from plotly->qeds) (1.3.3)
    Requirement already satisfied: lxml in /usr/local/lib/python3.6/dist-packages (from pandas_datareader->qeds) (4.2.6)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->qeds) (0.16.0)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from statsmodels->qeds) (0.5.1)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy->quantecon->qeds) (1.1.0)
    Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.38->quantecon->qeds) (0.31.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba>=0.38->quantecon->qeds) (50.3.0)
    Building wheels for collected packages: qeds
      Building wheel for qeds (setup.py) ... done
      Created wheel for qeds: filename=qeds-0.6.2-cp36-none-any.whl size=27821 sha256=7e13f5972afd7d6676eb9587e4cf0b29bcfdaf9b215dc4e26962f5e8ae8709df
      Stored in directory: /root/.cache/pip/wheels/b7/0b/74/c09109813c2b6116a2d4f2833c354b24163672f846a50fc7b4
    Successfully built qeds
    Installing collected packages: inflection, quandl, quantecon, qeds
    Successfully installed inflection-0.5.1 qeds-0.6.2 quandl-3.5.2 quantecon-0.4.8

```
import pandas as pd
%matplotlib inline
# activating plot theme
import qeds
qeds.themes.mpl_style
```

    <function qeds.themes.mpl_style>

```
pd.__version__
```

    '1.0.5'

```
values = [5.6,5.3,4.3,4.2,5.8,5.3,4.6,7.8,9.1,8.,5.7]
years = list(range(1995,2017,2))
unemp = pd.Series(data=values,index=years,name='Unemployment')
unemp
```

    1995    5.6
    1997    5.3
    1999    4.3
    2001    4.2
    2003    5.8
    2005    5.3
    2007    4.6
    2009    7.8
    2011    9.1
    2013    8.0
    2015    5.7
    Name: Unemployment, dtype: float64

```
unemp.index
```

    Int64Index([1995, 1997, 1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015], dtype='int64')

```
unemp.values
```

    array([5.6, 5.3, 4.3, 4.2, 5.8, 5.3, 4.6, 7.8, 9.1, 8. , 5.7])

```
unemp.name
```

    'Unemployment'

```
unemp.plot() #python matplotlib plot command
```

```
# unique values
unemp.unique()
```

    array([5.6, 5.3, 4.3, 4.2, 5.8, 4.6, 7.8, 9.1, 8. , 5.7])

```
unemp.plot.bar()
```

```
# DataFrame
data = {
    "NorthEast": [5.9,  5.6,  4.4,  3.8,  5.8,  4.9,  4.3,  7.1,  8.3,  7.9,  5.7],
    "MidWest": [4.5,  4.3,  3.6,  4. ,  5.7,  5.7,  4.9,  8.1,  8.7,  7.4,  5.1],
    "South": [5.3,  5.2,  4.2,  4. ,  5.7,  5.2,  4.3,  7.6,  9.1,  7.4,  5.5],
    "West": [6.6, 6., 5.2, 4.6, 6.5, 5.5, 4.5, 8.6, 10.7, 8.5, 6.1],
    "National": [5.6, 5.3, 4.3, 4.2, 5.8, 5.3, 4.6, 7.8, 9.1, 8., 5.7]
}
unemp_region = pd.DataFrame(data,index=years)
unemp_region
```

      NorthEast
      MidWest
      South
      West
      National

      1995
      5.9
      4.5
      5.3
      6.6
      5.6

      1997
      5.6
      4.3
      5.2
      6.0
      5.3

      1999
      4.4
      3.6
      4.2
      5.2
      4.3

      2001
      3.8
      4.0
      4.0
      4.6
      4.2

      2003
      5.8
      5.7
      5.7
      6.5
      5.8

      2005
      4.9
      5.7
      5.2
      5.5
      5.3

      2007
      4.3
      4.9
      4.3
      4.5
      4.6

      2009
      7.1
      8.1
      7.6
      8.6
      7.8

      2011
      8.3
      8.7
      9.1
      10.7
      9.1

      2013
      7.9
      7.4
      7.4
      8.5
      8.0

      2015
      5.7
      5.1
      5.5
      6.1
      5.7

```
# Computations with Columns
unemp_region['West']/100
```

    1995    0.066
    1997    0.060
    1999    0.052
    2001    0.046
    2003    0.065
    2005    0.055
    2007    0.045
    2009    0.086
    2011    0.107
    2013    0.085
    2015    0.061
    Name: West, dtype: float64

```
# finding correlation
unemp_region.corr()
```

      NorthEast
      MidWest
      South
      West
      National

      NorthEast
      1.000000
      0.875654
      0.964415
      0.967875
      0.976016

      MidWest
      0.875654
      1.000000
      0.951379
      0.900638
      0.952389

      South
      0.964415
      0.951379
      1.000000
      0.987259
      0.995030

      West
      0.967875
      0.900638
      0.987259
      1.000000
      0.981308

      National
      0.976016
      0.952389
      0.995030
      0.981308
      1.000000

```
unemp_region.dtypes
```

    NorthEast    float64
    MidWest      float64
    South        float64
    West         float64
    National     float64
    dtype: object

```
str_unemp = unemp_region.copy()
str_unemp['South'] = str_unemp['South'].astype(str)
str_unemp.dtypes
```

    NorthEast    float64
    MidWest      float64
    South         object
    West         float64
    National     float64
    dtype: object

```
str_unemp.head()
```

      NorthEast
      MidWest
      South
      West
      National

      1995
      5.9
      4.5
      5.3
      6.6
      5.6

      1997
      5.6
      4.3
      5.2
      6.0
      5.3

      1999
      4.4
      3.6
      4.2
      5.2
      4.3

      2001
      3.8
      4.0
      4.0
      4.6
      4.2

      2003
      5.8
      5.7
      5.7
      6.5
      5.8

```
str_unemp.sum()
```

    NorthEast                                 63.7
    MidWest                                     62
    South        5.35.24.24.05.75.24.37.69.17.45.5
    West                                      72.8
    National                                  65.7
    dtype: object

```
# creating new columns
unemp_region["UnweightedMean"] = (unemp_region["NorthEast"] +
                                  unemp_region["MidWest"] +
                                  unemp_region["South"] +
                                  unemp_region["West"])/4
```

```
unemp_region.tail()
```

      NorthEast
      MidWest
      South
      West
      National
      UnweightedMean

      2007
      4.3
      4.9
      4.3
      4.5
      4.6
      4.50

      2009
      7.1
      8.1
      7.6
      8.6
      7.8
      7.85

      2011
      8.3
      8.7
      9.1
      10.7
      9.1
      9.20

      2013
      7.9
      7.4
      7.4
      8.5
      8.0
      7.80

      2015
      5.7
      5.1
      5.5
      6.1
      5.7
      5.60

```
# renaming columns by passing a dictionary to the 'rename' method
names = {'NorthEast':'NE','MidWest':'MW','South':'S','West':'W'}
unemp_region.rename(columns=names,inplace=True)
unemp_region.head()
```

      NE
      MW
      S
      W
      National
      UnweightedMean

      1995
      5.9
      4.5
      5.3
      6.6
      5.6
      5.575

      1997
      5.6
      4.3
      5.2
      6.0
      5.3
      5.275

      1999
      4.4
      3.6
      4.2
      5.2
      4.3
      4.350

      2001
      3.8
      4.0
      4.0
      4.6
      4.2
      4.100

      2003
      5.8
      5.7
      5.7
      6.5
      5.8
      5.925

```
###############################project##################################################
# loading up data
url = "https://datascience.quantecon.org/assets/data/state_unemployment.csv"
unemp_raw = pd.read_csv(url,parse_dates=['Date'])
# load up the Date column as a Python datetime type
unemp_raw.head()
```

      Date
      state
      LaborForce
      UnemploymentRate

      0
      2000-01-01
      Alabama
      2142945.0
      4.7

      1
      2000-01-01
      Alaska
      319059.0
      6.3

      2
      2000-01-01
      Arizona
      2499980.0
      4.1

      3
      2000-01-01
      Arkansas
      1264619.0
      4.4

      4
      2000-01-01
      California
      16680246.0
      5.0

```
unemp_raw.shape
```

    (10800, 4)

```
# looking at unemployment rate across different states over time
# data transformation
unemp_all = (unemp_raw
             .reset_index()
             .pivot_table(index='Date',columns='state',values='UnemploymentRate')
             )
unemp_all.head()
```

      state
      Alabama
      Alaska
      Arizona
      Arkansas
      California
      Colorado
      Connecticut
      Delaware
      Florida
      Georgia
      Hawaii
      Idaho
      Illinois
      Indiana
      Iowa
      Kansas
      Kentucky
      Louisiana
      Maine
      Maryland
      Massachusetts
      Michigan
      Minnesota
      Mississippi
      Missouri
      Montana
      Nebraska
      Nevada
      New Hampshire
      New Mexico
      New York
      New jersey
      North Carolina
      North Dakota
      Ohio
      Oklahoma
      Oregon
      Pennsylvania
      Rhode island
      South Carolina
      South Dakota
      Tennessee
      Texas
      Utah
      Vermont
      Virginia
      Washington
      West Virginia
      Wisconsin
      Wyoming

      Date

      2000-01-01
      4.7
      6.3
      4.1
      4.4
      5.0
      2.8
      2.8
      3.5
      3.7
      3.7
      4.7
      4.6
      4.2
      3.2
      2.4
      3.4
      4.2
      5.1
      3.6
      3.4
      3.0
      3.3
      3.0
      5.4
      3.1
      5.1
      2.8
      3.9
      2.7
      5.1
      4.7
      3.8
      3.3
      3.1
      4.1
      3.2
      5.0
      4.1
      4.0
      4.0
      2.4
      3.7
      4.6
      3.1
      2.7
      2.6
      4.9
      5.8
      3.2
      4.1

      2000-02-01
      4.7
      6.3
      4.1
      4.3
      5.0
      2.8
      2.7
      3.6
      3.7
      3.6
      4.6
      4.6
      4.2
      3.3
      2.4
      3.4
      4.1
      5.1
      3.5
      3.4
      2.9
      3.2
      3.0
      5.5
      3.2
      5.0
      2.8
      3.9
      2.7
      5.0
      4.7
      3.7
      3.3
      3.0
      4.1
      3.1
      5.0
      4.0
      4.0
      3.9
      2.4
      3.7
      4.6
      3.1
      2.6
      2.5
      4.9
      5.6
      3.2
      3.9

      2000-03-01
      4.6
      6.3
      4.0
      4.3
      5.0
      2.7
      2.6
      3.6
      3.7
      3.6
      4.5
      4.6
      4.3
      3.3
      2.4
      3.5
      4.1
      5.1
      3.5
      3.5
      2.8
      3.2
      3.0
      5.5
      3.3
      5.0
      2.8
      4.0
      2.7
      4.9
      4.6
      3.6
      3.4
      3.0
      4.0
      3.1
      5.0
      4.0
      4.0
      3.8
      2.4
      3.8
      4.5
      3.1
      2.6
      2.4
      5.0
      5.5
      3.3
      3.9

      2000-04-01
      4.6
      6.3
      4.0
      4.3
      5.1
      2.7
      2.5
      3.7
      3.7
      3.7
      4.4
      4.6
      4.3
      3.3
      2.5
      3.5
      4.1
      5.1
      3.4
      3.5
      2.7
      3.3
      3.0
      5.6
      3.3
      5.0
      2.8
      4.0
      2.7
      4.9
      4.6
      3.6
      3.4
      2.9
      4.0
      3.1
      5.0
      4.0
      4.0
      3.7
      2.4
      3.8
      4.4
      3.1
      2.7
      2.4
      5.0
      5.4
      3.4
      3.8

      2000-05-01
      4.5
      6.3
      4.0
      4.2
      5.1
      2.7
      2.4
      3.7
      3.7
      3.7
      4.3
      4.6
      4.3
      3.3
      2.5
      3.5
      4.1
      5.1
      3.3
      3.6
      2.6
      3.5
      3.0
      5.6
      3.3
      5.0
      2.8
      4.0
      2.7
      4.9
      4.6
      3.6
      3.5
      2.9
      4.1
      3.1
      5.1
      4.0
      4.1
      3.8
      2.4
      3.9
      4.3
      3.2
      2.7
      2.3
      5.1
      5.4
      3.5
      3.8

```
states = [
          'Arizona','California','Florida','Illinois',
          'Michigan','New York','Texas'
]
unemp = unemp_all[states]
unemp.head()
# a subset
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

```
unemp.plot(figsize=(8,6))
```

```
# Dates in pandas
unemp.index
```

    DatetimeIndex(['2000-01-01', '2000-02-01', '2000-03-01', '2000-04-01',
                   '2000-05-01', '2000-06-01', '2000-07-01', '2000-08-01',
                   '2000-09-01', '2000-10-01',
                   ...
                   '2017-03-01', '2017-04-01', '2017-05-01', '2017-06-01',
                   '2017-07-01', '2017-08-01', '2017-09-01', '2017-10-01',
                   '2017-11-01', '2017-12-01'],
                  dtype='datetime64[ns]', name='Date', length=216, freq=None)

```
# Data corresponding to a single date
unemp.loc['01/01/2000']
```

    state
    Arizona       4.1
    California    5.0
    Florida       3.7
    Illinois      4.2
    Michigan      3.3
    New York      4.7
    Texas         4.6
    Name: 2000-01-01 00:00:00, dtype: float64

```
# Data for all days between New Years Day and June first in 2000
unemp.loc['01/01/2000':'06/01/2000']
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

      2000-06-01
      4.0
      5.1
      3.8
      4.3
      3.7
      4.6
      4.3

```
# Data aggregations: combining multiple values into a single value
# Data aggregations are used extensively to analyze and manipulate data
unemp.mean()
```

    state
    Arizona       6.301389
    California    7.299074
    Florida       6.048611
    Illinois      6.822685
    Michigan      7.492593
    New York      6.102315
    Texas         5.695370
    dtype: float64

```
# aggregations on rows
unemp.var(axis=1)
```

    Date
    2000-01-01    0.352381
    2000-02-01    0.384762
    2000-03-01    0.364762
    2000-04-01    0.353333
    2000-05-01    0.294762
                    ...
    2017-08-01    0.141429
    2017-09-01    0.163333
    2017-10-01    0.165714
    2017-11-01    0.165714
    2017-12-01    0.148095
    Length: 216, dtype: float64

```
# custom-built aggregations
#classify states as “low unemployment” or “high unemployment” based on whether their mean unemployment level is above or below 6.5
def high_or_low(s):
    """
    high
    if the mean is above 6.5 and low if the mean is below 6.5
    """
    if s.mean() < 6.5:
        out ='Low'
    else:
        out = 'High'

    return out
```

```
unemp.agg(high_or_low)
```

    state
    Arizona        Low
    California    High
    Florida        Low
    Illinois      High
    Michigan      High
    New York       Low
    Texas          Low
    dtype: object

```
unemp.agg(high_or_low,axis=1)
```

    Date
    2000-01-01    Low
    2000-02-01    Low
    2000-03-01    Low
    2000-04-01    Low
    2000-05-01    Low
                 ...
    2017-08-01    Low
    2017-09-01    Low
    2017-10-01    Low
    2017-11-01    Low
    2017-12-01    Low
    Length: 216, dtype: object

```
unemp.agg([min,max,high_or_low])
```

      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      min
      3.6
      4.5
      3.1
      4.2
      3.2
      4.2
      3.9

      max
      10.9
      12.3
      11.3
      11.3
      14.6
      8.9
      8.3

      high_or_low
      Low
      High
      Low
      High
      High
      Low
      Low

```
unemp.agg([min,max,high_or_low]).columns
```

    Index(['Arizona', 'California', 'Florida', 'Illinois', 'Michigan', 'New York',
           'Texas'],
          dtype='object')

```
# Transforms

```


## Initial setup

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

```python
import tensorflow as tf
print(tf.__version__)
# tensorflow version used is 2.8.0
import torch
print(torch.__version__)
# torch version used is 1.10+cu111
```

    2.8.0
    1.10.0+cu111

```python
!nvidia-smi
```

    Sun Apr 17 21:06:30 2022
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   43C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

```python
# Other imports
! pip install tensorflow_addons
! pip install tensorflow_io
! pip install lightgbm
! pip install xgboost
! pip install catboost

import os
from tensorflow.keras.layers import *
from tensorflow.keras.models import *
from tensorflow import keras as tfkeras
import matplotlib.pyplot as plt
from imutils import paths
from tqdm import tqdm
import tensorflow as tf
import tensorflow_addons as tfa
import tensorflow_datasets as tfds
import tensorflow_io as tfio
import tensorflow_hub as hub
import numpy as np
import cv2
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
from sklearn.metrics.pairwise import cosine_similarity as cos
from sympy.utilities.iterables import multiset_permutations
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix, mean_absolute_error, r2_score
from sklearn.model_selection import *
from IPython.display import Image, display
from scipy import ndimage

from xgboost import XGBRegressor as xgb
from sklearn.ensemble import ExtraTreesRegressor as extratree
from sklearn.ensemble import RandomForestRegressor as randomforest
from sklearn.svm import NuSVR as svr
from sklearn.gaussian_process import GaussianProcessRegressor as gpr
from sklearn.ensemble import AdaBoostRegressor as ada
from sklearn.tree import DecisionTreeRegressor as tree
from sklearn.ensemble import GradientBoostingRegressor as gdb
from sklearn.neural_network import MLPRegressor as mlp
from sklearn.experimental import enable_hist_gradient_boosting
from sklearn.ensemble import HistGradientBoostingRegressor as hgb
from lightgbm import LGBMRegressor as lgb
from catboost import CatBoostRegressor as cgb
from sklearn.ensemble import StackingRegressor as stk
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import cross_validate, cross_val_predict
from sklearn.ensemble import VotingRegressor as vot

import zipfile
import concurrent.futures

# Random seed fix
random_seed = 42
tf.random.set_seed(random_seed)
np.random.seed(random_seed)
```

    Collecting tensorflow_addons
      Downloading tensorflow_addons-0.16.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 28.5 MB/s
    Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow_addons) (2.7.1)
    Installing collected packages: tensorflow-addons
    Successfully installed tensorflow-addons-0.16.1
    Collecting tensorflow_io
      Downloading tensorflow_io-0.24.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (23.4 MB)
         |████████████████████████████████| 23.4 MB 60.3 MB/s
    Requirement already satisfied: tensorflow-io-gcs-filesystem==0.24.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_io) (0.24.0)
    Installing collected packages: tensorflow-io
    Successfully installed tensorflow-io-0.24.0
    Requirement already satisfied: lightgbm in /usr/local/lib/python3.7/dist-packages (2.2.3)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from lightgbm) (1.4.1)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from lightgbm) (1.0.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from lightgbm) (1.21.5)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->lightgbm) (3.1.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->lightgbm) (1.1.0)
    Requirement already satisfied: xgboost in /usr/local/lib/python3.7/dist-packages (0.90)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.4.1)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.21.5)
    Collecting catboost
      Downloading catboost-1.0.5-cp37-none-manylinux1_x86_64.whl (76.6 MB)
         |████████████████████████████████| 76.6 MB 1.2 MB/s
    Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from catboost) (1.21.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from catboost) (1.4.1)
    Requirement already satisfied: graphviz in /usr/local/lib/python3.7/dist-packages (from catboost) (0.10.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from catboost) (3.2.2)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from catboost) (1.15.0)
    Requirement already satisfied: plotly in /usr/local/lib/python3.7/dist-packages (from catboost) (5.5.0)
    Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.7/dist-packages (from catboost) (1.3.5)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->catboost) (2.8.2)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->catboost) (2018.9)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (3.0.8)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (1.4.2)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (0.11.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->catboost) (4.1.1)
    Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly->catboost) (8.0.1)
    Installing collected packages: catboost
    Successfully installed catboost-1.0.5

    /usr/local/lib/python3.7/dist-packages/sklearn/experimental/enable_hist_gradient_boosting.py:17: UserWarning: Since version 1.0, it is not needed to import enable_hist_gradient_boosting anymore. HistGradientBoostingClassifier and HistGradientBoostingRegressor are now stable and can be normally imported from sklearn.ensemble.
      "Since version 1.0, "

## Data utility, only need to use once to preprocess tiff images to png format

```python
%%time
zf = zipfile.ZipFile('/content/drive/MyDrive/ML course 2022/rawImages.zip')
def unzip(file):
    zf.extract(file)
with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(unzip, zf.infolist())
```

    CPU times: user 14 s, sys: 1.32 s, total: 15.3 s
    Wall time: 5min 4s

```python
drive.flush_and_unmount()
```

```python
for i in range(len(level1_images_train)):
  img = cv2.imread(level1_images_train[i])
  if img is None:
    continue
  img = cv2.imread(level1_images_train[i])
  cv2.imwrite('/content/drive/MyDrive/ML course 2022_Team BNL/level1/%s.jpeg' %level1_images_train[i].split('/')[-1][:-5], img)
```

```python
import shutil
for i in range(len(level1_images_train)):
  shutil.copy(level1_images_train[i], "/content/drive/MyDrive/ML course 2022_Team BNL/level1_png/%s" %level1_images_train[i].split("/")[-1])
```

```python
# the snippet that I use to create the jpegs of the high pass filtered images
pixel_std_values = np.zeros((len(train_images_directory_select)))
for i in range(train_images_directory_select.shape[0]):
    im = cv2.imread(train_images_directory_select[i])                                                       # high pass filter then grayscale then standard deviation of all pixel values
    high_pass_image = cv2.cvtColor(ndimage.gaussian_filter(im, 63) - im, cv2.COLOR_BGR2GRAY)
    # pixel_std_values[i] = np.std(high_pass_image)
    cv2.imwrite('/content/drive/MyDrive/ML course 2022_Team BNL/level2_kernel63/%s.jpeg' %train_images_directory_select[i].split('/')[-1][:-5], high_pass_image)
# plt.scatter(labels, pixel_std_values)
# plt.xlabel('true_focal_distance')
# plt.ylabel('std value')
```

## Dataset gathering and preparation

```python
%cd /content/drive/MyDrive/ML\ course\ 2022_Team\ BNL
```

    /content/drive/MyDrive/ML course 2022_Team BNL

```python
training_batch_size = 1
test_batch_size = 1

imageSizeheight=1200
imageSizewidth=1920

# imageSizeheight=300
# imageSizewidth=480
```

```python
# Image preprocessing utils
@tf.function
def parse_images(image_path):

    image_string = tf.io.read_file(image_path)
    image = tf.io.decode_jpeg(image_string, channels=3)
    # image = tfio.experimental.image.decode_tiff(image_string, index=0, name=None)[:,:,:-1]   # in the doc, it transforms tiff to 4 channels, with additional channel of opacity which is not needed.
    image = tf.image.convert_image_dtype(image, tf.float32)
    image = tf.image.resize(image, size=[imageSizeheight, imageSizewidth])

    return image
```

```python
level2_images = np.array(list(paths.list_files("/content/drive/MyDrive/ML course 2022_Team BNL/level2_kernel63", validExts="jpeg")))
# train_images_directory_select = np.random.choice(level2_images_train, 6000, replace=False)

labels = []                                                    # initialize labels
positions = []

for i in range(level2_images.shape[0]):
  label = np.abs(float(level2_images[i].split("_")[-1][4:-7].split("(")[0]))               # individual label, can be put into absolute or not
  labels.append(label)
  position = level2_images[i].split("_")[-3] + '_' + level2_images[i].split("_")[-2]      # individual position
  positions.append(position)
labels = np.array(labels)
positions = np.array(positions)

# these test images are selected by our national lab mentor.
test_positions = np.array(['ix000_iy004', 'ix000_iy009', 'ix001_iy007', 'ix003_iy013', 'ix006_iy003',
                           'ix013_iy008', 'ix013_iy012', 'ix013_iy013', 'ix015_iy006', 'ix017_iy012',
                           'ix017_iy013', 'ix019_iy000', 'ix019_iy011', 'ix020_iy007', 'ix020_iy011',
                           'ix020_iy015', 'ix021_iy000', 'ix021_iy006', 'ix022_iy004', 'ix023_iy005',
                           'ix023_iy014', 'ix023_iy015', 'ix025_iy015', 'ix028_iy005', 'ix031_iy007',
                           'ix032_iy007', 'ix032_iy008', 'ix033_iy006', 'ix034_iy004', 'ix034_iy006',
                           'ix034_iy012', 'ix034_iy013', 'ix034_iy014', 'ix035_iy009', 'ix035_iy015',
                           'ix036_iy006', 'ix036_iy012', 'ix036_iy013', 'ix037_iy013', 'ix038_iy014',
                           'ix039_iy004', 'ix039_iy009', 'ix039_iy013', 'ix040_iy006', 'ix040_iy013',
                           'ix041_iy011', 'ix045_iy008', 'ix047_iy009', 'ix048_iy013', 'ix049_iy006',
                           'ix049_iy012', 'ix051_iy009', 'ix051_iy010', 'ix053_iy002', 'ix053_iy010',
                           'ix053_iy013', 'ix054_iy001', 'ix054_iy012'])

test_images_directory_select = level2_images[np.where(np.isin(positions, test_positions) == True)[0]]
train_images_directory_select = level2_images[np.where(np.isin(positions, test_positions) == False)[0]]
test_label = labels[np.where(np.isin(positions, test_positions) == True)[0]]

print(len(level2_images))
print(len(train_images_directory_select))
print(len(test_images_directory_select))
```

    9462
    8829
    633

```python
labels = []                                                    # initialize labels
positions = []

for i in range(train_images_directory_select.shape[0]):
  label = np.abs(float(train_images_directory_select[i].split("_")[-1][4:-7].split("(")[0]))               # individual label, can be put into absolute or not
  labels.append(label)
  position = train_images_directory_select[i].split("_")[-3] + '_' + train_images_directory_select[i].split("_")[-2]      # individual position
  positions.append(position)
labels = np.array(labels)
positions = np.array(positions)                              # initialize the positions of the images, needed for record keeping
```

```python
edge_pixel_intensity_high_threshold = 0.50
edge_pixel_intensity_low_threshold = 0.16
edge_percentage_threshold = 0.02
cap = 50

infocus_images_coordinate = np.where(labels==0)[0]                                       # locations where the infocus images are
edge_percentages = []
valid = []
for i in range(train_images_directory_select[infocus_images_coordinate].shape[0]):
  edge_percentage = len(np.where((parse_images(train_images_directory_select[infocus_images_coordinate][i])[:,:,0]>=edge_pixel_intensity_high_threshold)
                                  | (parse_images(train_images_directory_select[infocus_images_coordinate][i])[:,:,0]<=edge_pixel_intensity_low_threshold))[0]) / imageSizeheight / imageSizewidth
  edge_percentages.append(edge_percentage)
  if edge_percentage >= edge_percentage_threshold:                                                             # picking out infocus images where the area containing edges is smaller than a threshold value
    valid.append(positions[infocus_images_coordinate][i])

for i in range(len(valid)):
  valid_position = np.where(positions==valid[i])[0]
  if i == 0:
    valid_positions = valid_position
  else:
    valid_positions = np.append(valid_positions, valid_position)

valid_directories = train_images_directory_select[valid_positions]
valid_labels = labels[valid_positions]
# features = all_features_45_smooth3[valid_positions]

capped_valid_labels = valid_labels[np.where((np.abs(valid_labels)<=cap) & (np.abs(valid_labels)>=0))[0]]                                # not include the focal distance larger than cap
capped_valid_directories = valid_directories[np.where((np.abs(valid_labels)<=cap) & (np.abs(valid_labels)>=0))[0]]
# features = features[np.where((np.abs(valid_labels)<=cap) & (np.abs(valid_labels)>=0))[0]]

print(len(capped_valid_labels))
```

    WARNING:tensorflow:5 out of the last 9 calls to <function parse_images at 0x7fad29805c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
    WARNING:tensorflow:6 out of the last 11 calls to <function parse_images at 0x7fad29805c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

## Initiate our cnn model

```python
Resnet50_transfer = tf.keras.applications.resnet50.ResNet50(
    include_top=False,
    weights="imagenet",
    # weights=None,
    input_tensor=None,
    input_shape=(imageSizeheight, imageSizewidth, 3),
    pooling=None,
)

Resnet50_transfer.trainable = True
```

    Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5
    94773248/94765736 [==============================] - 1s 0us/step
    94781440/94765736 [==============================] - 1s 0us/step

```python
Resnet50_transfer.trainable = False

supervise_model = tf.keras.models.Sequential([

    Resnet50_transfer,
    GlobalAveragePooling2D(),
    # BatchNormalization(epsilon=0.1),
    # Dense(64, activation='relu'),
    # Dropout(0.5),
    # BatchNormalization(epsilon=0.1),
    # Dense(32, activation='relu'),
    # Dropout(0.5),
    # BatchNormalization(epsilon=0.1),
    Dense(16, activation='relu'),
    Dropout(0.5),
    Dense(1)
])
```

```python
# adam = tf.keras.optimizers.Adam(learning_rate=0.001)
# sgd = tf.keras.optimizers.SGD(learning_rate=0.001)
# metrics = ['mae'
#                 ]
# supervise_model.compile(loss = 'mse', optimizer = adam, metrics = metrics)
# supervise_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     resnet50 (Functional)       (None, 38, 60, 2048)      23587712

     global_average_pooling2d (G  (None, 2048)             0
     lobalAveragePooling2D)

     dense (Dense)               (None, 16)                32784

     dropout (Dropout)           (None, 16)                0

     dense_1 (Dense)             (None, 1)                 17

    =================================================================
    Total params: 23,620,513
    Trainable params: 32,801
    Non-trainable params: 23,587,712
    _________________________________________________________________

```python
# cnn feature extraction in segments so as to avoid OOM problem, that we encountered with image set size larger than 1000.
feature_extractor = Model(supervise_model.input, supervise_model.layers[-4].output)
n_segments = len(test_images_directory_select) // 500 + 1
for i in range(n_segments):
  if i == 0:
    test_ds = tf.data.Dataset.from_tensor_slices(test_images_directory_select[:(i + 1) * 500])
    test_ds = (
        test_ds
        .map(parse_images, num_parallel_calls=tf.data.experimental.AUTOTUNE)
        .batch(training_batch_size
              #  , drop_remainder=True
              )
        .prefetch(tf.data.experimental.AUTOTUNE)
    )

    features_test = feature_extractor.predict(test_ds)

  else:
    test_ds = tf.data.Dataset.from_tensor_slices(test_images_directory_select[i * 500 :(i + 1) * 500])
    test_ds = (
        test_ds
        .map(parse_images, num_parallel_calls=tf.data.experimental.AUTOTUNE)
        .batch(training_batch_size
              #  , drop_remainder=True
              )
        .prefetch(tf.data.experimental.AUTOTUNE)
    )

    features_test = np.append(features_test, feature_extractor.predict(test_ds), axis=0)
# np.savez_compressed('performance_log/level2/feature_map.npz', feature_map=features)
```

```python
n_segments = len(labels) // 500 + 1
for i in range(n_segments):
  if i == 0:
    train_ds = tf.data.Dataset.from_tensor_slices(train_images_directory_select[:(i + 1) * 500])
    train_ds = (
        train_ds
        .map(parse_images, num_parallel_calls=tf.data.experimental.AUTOTUNE)
        .batch(training_batch_size
              #  , drop_remainder=True
              )
        .prefetch(tf.data.experimental.AUTOTUNE)
    )

    features_train = feature_extractor.predict(train_ds)

  else:
    train_ds = tf.data.Dataset.from_tensor_slices(train_images_directory_select[i * 500 :(i + 1) * 500])
    train_ds = (
        train_ds
        .map(parse_images, num_parallel_calls=tf.data.experimental.AUTOTUNE)
        .batch(training_batch_size
              #  , drop_remainder=True
              )
        .prefetch(tf.data.experimental.AUTOTUNE)
    )

    features_train = np.append(features_train, feature_extractor.predict(train_ds), axis=0)
```

```python
rf_regressor = randomforest(max_depth=8)
# features_train, features_test, train_label, test_label = train_test_split(features, capped_valid_labels, test_size=0.2, random_state=42)
rf_regressor.fit(features_train, labels)
y_pred_rf = rf_regressor.predict(features_test)
```

```python
print('rf prediction r2_score = %f' % r2_score(y_pred_rf, test_label))
print('rf prediction mean_absolute_error = %f' % mean_absolute_error(y_pred_rf, test_label))
plt.scatter(test_label, y_pred_rf)
plt.plot(np.linspace(0, 50, 150), np.linspace(0, 50, 150), c='r')
plt.xlabel('true_distance [$\mu m$]')
plt.ylabel('predicted_distance [$\mu m$]')
```

```python
depth = np.array([8, 16])
lr = np.linspace(0.01, 0.20, 21)
performance = np.zeros((len(depth), len(lr), 2))

# features_train, features_test, train_label, test_label = train_test_split(features, capped_valid_labels, test_size=0.2, random_state=42)

for i in range(len(depth)):
  for j in range(len(lr)):
    xg_boost_regressor = xgb(max_depth=depth[i], learning_rate=lr[j], tree_method='gpu_hist', objective='reg:squarederror')         # don't go beyond max_depth of 16, session will crash due to OOM
    xg_boost_regressor.fit(features_train, labels)
    y_pred_xg = xg_boost_regressor.predict(features_test)
    performance[i, j, 0] = r2_score(y_pred_xg, test_label)
    performance[i, j, 1] = mean_absolute_error(y_pred_xg, test_label)
```

```python
xg_boost_regressor = xgb(max_depth=depth[np.where(performance == np.min(performance[:, :, 1]))[0][0]], learning_rate=lr[np.where(performance == np.min(performance[:, :, 1]))[1][0]], tree_method='gpu_hist', objective='reg:squarederror')         # don't go beyond max_depth of 16, session will crash due to OOM
xg_boost_regressor.fit(features_train, labels)
y_pred_xg = xg_boost_regressor.predict(features_test)
print('rf prediction r2_score = %f' % r2_score(y_pred_xg, test_label))
print('rf prediction mean_absolute_error = %f' % mean_absolute_error(y_pred_xg, test_label))
plt.scatter(test_label, y_pred_xg)
plt.plot(np.linspace(0, 50, 150), np.linspace(0, 50, 150), c='r')
plt.xlabel('true_distance [$\mu m$]')
plt.ylabel('predicted_distance [$\mu m$]')
```

```python
plt.scatter(lr[2:], performance[0, 2:, 1])
plt.scatter(lr[2:], performance[1, 2:, 1])
```

kernel 63
r2 scores    (mae are between 5 and 8)

0.10: rf: 0.60, xgb: 0.85    ~ 132 images

0.08: rf: 0.48, xgb: 0.70    ~ 230 images

0.05: rf: 0.16, xgb: 0.60    ~ 532 images

0.04: rf: 0.19, xgb: 0.55    ~ 884 images

kernel 45
r2 scores    (mae are between 5 and 8)

0.10: rf: 0.69, xgb: 0.73    ~ 132 images

0.08: rf: 0.67, xgb: 0.73    ~ 230 images

0.05: rf: 0.44, xgb: 0.64    ~ 422 images

0.04: rf: 0.44, xgb: 0.67    ~ 763 images

kernel 33
r2 scores    (mae are between 5 and 8)

0.10: rf: 0.69, xgb: 0.73    ~ 132 images

0.08: rf: 0.67, xgb: 0.73    ~ 230 images

0.05: rf: 0.44, xgb: 0.64    ~ 422 images

0.04: rf: 0.44, xgb: 0.67    ~ 763 images

```python
print('xgboost prediction r2_score = %f' % r2_score(y_pred_xg, test_label))
print('xgboost prediction mean_absolute_error = %f' % mean_absolute_error(y_pred_xg, test_label))
plt.scatter(test_label, y_pred_xg)
# plt.plot(np.linspace(0, cap, 150), np.linspace(0, cap, 150), c='r')
plt.xlabel('true_distance [$\mu m$]')
plt.ylabel('predicted_distance [$\mu m$]')
```


```python
import numpy as np
import sympy
T = symbols('T')
A = np.array([])
```

```python

```