## Set up the environment

```
!nvidia-smi
```

    NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

```
#!pip install pymatgen==2020.12.31
!pip install pymatgen==2019.11.11
!pip install --pre graphdot
!pip install gdown
```

    Collecting pymatgen==2019.11.11
      Downloading https://files.pythonhosted.org/packages/57/f8/1d5f9bacde237107f917a5ca31bdf73e55a774ffb8e7b9e5f9cd7d7cf114/pymatgen-2019.11.11.tar.gz (2.5MB)
         |████████████████████████████████| 2.5MB 6.7MB/s
    Requirement already satisfied: numpy>=1.14.3 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.19.5)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2.23.0)
    Collecting ruamel.yaml>=0.15.6
      Downloading https://files.pythonhosted.org/packages/74/c6/0aa1aaaa56c9d320f19d9ca3f8304bc6ace8fab95932b1602cbe091ae4a3/ruamel.yaml-0.17.9-py3-none-any.whl (108kB)
         |████████████████████████████████| 112kB 32.3MB/s
    Collecting monty>=3.0.2
      Downloading https://files.pythonhosted.org/packages/31/26/e14afc57b4dd6c9b8c6d4b3f70dd0b4f1550ba263711e345845354bc84d9/monty-2021.6.10-py3-none-any.whl (65kB)
         |████████████████████████████████| 71kB 7.1MB/s
    Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.4.1)
    Collecting pydispatcher>=2.0.5
      Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (0.8.9)
    Collecting spglib>=1.9.9.44
      Downloading https://files.pythonhosted.org/packages/4b/98/fa4760b9c71e2eace5a60b85dbf36dece49c379f8291cf1203056f287766/spglib-1.16.1-cp37-cp37m-manylinux2010_x86_64.whl (296kB)
         |████████████████████████████████| 296kB 29.7MB/s
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (2.5.1)
    Requirement already satisfied: matplotlib>=1.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (3.2.2)
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (3.3.0)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.7.1)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11) (1.1.5)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11) (2020.12.5)
    Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10"
      Downloading https://files.pythonhosted.org/packages/5e/6e/f652c56bbb2c3d3fca252ffc7c0358597f57a1bbdf484dac683054950c63/ruamel.yaml.clib-0.2.2-cp37-cp37m-manylinux1_x86_64.whl (547kB)
         |████████████████████████████████| 552kB 33.3MB/s
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.2->pymatgen==2019.11.11) (4.4.2)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen==2019.11.11) (2.4.7)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pymatgen==2019.11.11) (1.2.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->pymatgen==2019.11.11) (2018.9)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib>=1.5->pymatgen==2019.11.11) (1.15.0)
    Building wheels for collected packages: pymatgen, pydispatcher
      Building wheel for pymatgen (setup.py) ... done
      Created wheel for pymatgen: filename=pymatgen-2019.11.11-cp37-cp37m-linux_x86_64.whl size=3324761 sha256=6cf93e611e6c47a15bab07470b9e6718adc4a580ac5569075626887f5b463b02
      Stored in directory: /root/.cache/pip/wheels/98/23/3c/3b929997e5f8976361933411e8a3b9bdf1b2702e7830e7a7ad
      Building wheel for pydispatcher (setup.py) ... done
      Created wheel for pydispatcher: filename=PyDispatcher-2.0.5-cp37-none-any.whl size=11517 sha256=9b0107e8759a73beda58fbdf3018d4c4562209d5d65c3b4ccdda07dedb62904e
      Stored in directory: /root/.cache/pip/wheels/88/99/96/cfef6665f9cb1522ee6757ae5955feedf2fe25f1737f91fa7f
    Successfully built pymatgen pydispatcher
    Installing collected packages: ruamel.yaml.clib, ruamel.yaml, monty, pydispatcher, spglib, pymatgen
    Successfully installed monty-2021.6.10 pydispatcher-2.0.5 pymatgen-2019.11.11 ruamel.yaml-0.17.9 ruamel.yaml.clib-0.2.2 spglib-1.16.1
    Collecting graphdot
      Downloading https://files.pythonhosted.org/packages/71/e6/af9d1784a6757a9e70a8ef2160f006d0fc46a913c82354b2825cd79a4e40/graphdot-0.8a17.tar.gz (103kB)
         |████████████████████████████████| 112kB 6.9MB/s
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.19.5)
    Requirement already satisfied: scipy>=1.3.0 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.4.1)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.7.1)
    Requirement already satisfied: pandas>=0.24 in /usr/local/lib/python3.7/dist-packages (from graphdot) (1.1.5)
    Requirement already satisfied: networkx>=2.4 in /usr/local/lib/python3.7/dist-packages (from graphdot) (2.5.1)
    Collecting pycuda>=2019
      Downloading https://files.pythonhosted.org/packages/5a/56/4682a5118a234d15aa1c8768a528aac4858c7b04d2674e18d586d3dfda04/pycuda-2021.1.tar.gz (1.7MB)
         |████████████████████████████████| 1.7MB 10.0MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
        Preparing wheel metadata ... done
    Collecting treelib>=1.6.1
      Downloading https://files.pythonhosted.org/packages/04/b0/2269c328abffbb63979f7143351a24a066776b87526d79956aea5018b80a/treelib-1.6.1.tar.gz
    Collecting kahypar>=1.1.4
      Downloading https://files.pythonhosted.org/packages/d2/c6/4b6ff807f2136079e9bc7289a793544541de4a04b402c678524647d6b0cb/kahypar-1.1.6-cp37-cp37m-manylinux2014_x86_64.whl (1.1MB)
         |████████████████████████████████| 1.1MB 24.3MB/s
    Requirement already satisfied: numba>=0.51.0 in /usr/local/lib/python3.7/dist-packages (from graphdot) (0.51.2)
    Collecting tqdm>=4.55
      Downloading https://files.pythonhosted.org/packages/42/d7/f357d98e9b50346bcb6095fe3ad205d8db3174eb5edb03edfe7c4099576d/tqdm-4.61.0-py2.py3-none-any.whl (75kB)
         |████████████████████████████████| 81kB 7.5MB/s
    Collecting ase>=3.17
      Downloading https://files.pythonhosted.org/packages/a5/36/de17e79f29e06d9a92746d0dd9ec4636487ab03f6af10e78586aae533f7a/ase-3.21.1-py3-none-any.whl (2.2MB)
         |████████████████████████████████| 2.2MB 39.4MB/s
    Requirement already satisfied: pymatgen==2019.11.11 in /usr/local/lib/python3.7/dist-packages (from graphdot) (2019.11.11)
    Collecting mendeleev>=0.7
      Downloading https://files.pythonhosted.org/packages/1d/8c/364f691a79162bed173069e3882c39baa315daf1336434c6c9b322078e61/mendeleev-0.7.0-py3-none-any.whl (175kB)
         |████████████████████████████████| 184kB 40.5MB/s
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->graphdot) (1.2.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->graphdot) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->graphdot) (2.8.1)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.4->graphdot) (4.4.2)
    Collecting pytools>=2011.2
      Downloading https://files.pythonhosted.org/packages/52/26/c7ab098ceb4e4e3f0e66e21257a286bb455ea22af7afefbd704d9ccf324c/pytools-2021.2.7.tar.gz (63kB)
         |████████████████████████████████| 71kB 7.8MB/s
    Requirement already satisfied: appdirs>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from pycuda>=2019->graphdot) (1.4.4)
    Collecting mako
      Downloading https://files.pythonhosted.org/packages/f3/54/dbc07fbb20865d3b78fdb7cf7fa713e2cba4f87f71100074ef2dc9f9d1f7/Mako-1.1.4-py2.py3-none-any.whl (75kB)
         |████████████████████████████████| 81kB 8.5MB/s
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from treelib>=1.6.1->graphdot) (0.16.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.51.0->graphdot) (57.0.0)
    Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.51.0->graphdot) (0.34.0)
    Requirement already satisfied: matplotlib>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from ase>=3.17->graphdot) (3.2.2)
    Requirement already satisfied: pydispatcher>=2.0.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2.0.5)
    Requirement already satisfied: monty>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2021.6.10)
    Requirement already satisfied: spglib>=1.9.9.44 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (1.16.1)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (0.17.9)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (0.8.9)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (2.23.0)
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2019.11.11->graphdot) (3.3.0)
    Collecting Pygments<3.0.0,>=2.8.0
      Downloading https://files.pythonhosted.org/packages/a6/c9/be11fce9810793676017f79ffab3c6cb18575844a6c7b8d4ed92f95de604/Pygments-2.9.0-py3-none-any.whl (1.0MB)
         |████████████████████████████████| 1.0MB 39.8MB/s
    Requirement already satisfied: six<2.0.0,>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (1.15.0)
    Requirement already satisfied: SQLAlchemy<2.0.0,>=1.3.23 in /usr/local/lib/python3.7/dist-packages (from mendeleev>=0.7->graphdot) (1.4.15)
    Collecting pyfiglet<0.9,>=0.8.post1
      Downloading https://files.pythonhosted.org/packages/33/07/fcfdd7a2872f5b348953de35acce1544dab0c1e8368dca54279b1cde5c15/pyfiglet-0.8.post1-py2.py3-none-any.whl (865kB)
         |████████████████████████████████| 870kB 38.4MB/s
    Collecting colorama<0.5.0,>=0.4.4
      Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
    Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from mako->pycuda>=2019->graphdot) (2.0.1)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.0->ase>=3.17->graphdot) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.0->ase>=3.17->graphdot) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.0->ase>=3.17->graphdot) (2.4.7)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml>=0.15.6->pymatgen==2019.11.11->graphdot) (0.2.2)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen==2019.11.11->graphdot) (2.10)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (4.0.1)
    Requirement already satisfied: greenlet!=0.4.17; python_version >= "3" in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (1.1.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (3.4.1)
    Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->SQLAlchemy<2.0.0,>=1.3.23->mendeleev>=0.7->graphdot) (3.7.4.3)
    Building wheels for collected packages: pycuda
      Building wheel for pycuda (PEP 517) ... canceled
    ERROR: Operation cancelled by user
    Requirement already satisfied: gdown in /usr/local/lib/python3.7/dist-packages (3.6.4)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from gdown) (2.23.0)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from gdown) (4.41.1)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from gdown) (1.15.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->gdown) (2020.12.5)

```
%matplotlib inline
import io
import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')
import os
import urllib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import graphdot
from graphdot import Graph
from graphdot.graph.adjacency import AtomicAdjacency
from graphdot.graph.reorder import rcm
from graphdot.kernel.marginalized import MarginalizedGraphKernel # https://graphdot.readthedocs.io/en/latest/apidoc/graphdot.kernel.marginalized.html
from graphdot.kernel.marginalized.starting_probability import Uniform
from graphdot.model.gaussian_process import (
    GaussianProcessRegressor,
    LowRankApproximateGPR
)
from graphdot.kernel.fix import Normalization
import graphdot.microkernel as uX
import ase.io

# for getting all file names into a list under a directory
from os import listdir
# for getting file names that match certain pattern
import glob
import time
```

```
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
```

```
#cd gdrive/MyDrive/Google\ Colab/Covid-Data
%cd gdrive/MyDrive/Covid-Data/
```

    /content/gdrive/.shortcut-targets-by-id/1wtzMcocuK8kPsz8K0ktjCZPkv567W6M2/Covid-Data

```
!pwd
```

    /content/gdrive/.shortcut-targets-by-id/1wtzMcocuK8kPsz8K0ktjCZPkv567W6M2/Covid-Data

```
!mkdir /content/pkls
```

## load the data

```
files  = ['uncharged_NSP15_6W01_A_3_H.Orderable_zinc_db_enaHLL.2col.csv.1.xz']
dataset = pd.read_pickle(files[0]) # length of each csv file is 100000
```

```
target = 'energy'
batch_size = 1000
batch_num_train = 0.8*len(dataset)//batch_size # batch number of training
num_test = 0.2*len(dataset)
train_data = dataset.iloc[:int(batch_num_train)*batch_size]
test_data = dataset.iloc[int(batch_num_train)*batch_size:len(dataset)]
#print(len(train_data))
#print(len(test_data))
```

```
gpr = GaussianProcessRegressor(
    # kernel is the covariance function of the gaussian process (GP)
    kernel=Normalization( # kernel equals to normalization -> normalizes a kernel using the cosine of angle formula, k_normalized(x,y) = k(x,y)/sqrt(k(x,x)*k(y,y))
        # graphdot.kernel.fix.Normalization(kernel), set kernel as marginalized graph kernel, which is used to calculate the similarity between 2 graphs
        # implement the random walk-based graph similarity kernel as Kashima, H., Tsuda, K., & Inokuchi, A. (2003). Marginalized kernels between labeled graphs. ICML
        MarginalizedGraphKernel(
            # node_kernel - A kernelet that computes the similarity between individual nodes
            # uX - graphdot.microkernel - microkernels are positive-semidefinite functions between individual nodes and edges of graphs
            node_kernel=uX.Additive( # addition of kernal matrices: sum of k_a(X_a, Y_a) cross for a in features
                # uX.Constant - a kernel that returns a constant value, always mutlipled with other microkernels as an adjustable weight
                #   c, the first input arg. as 0.5, (0.01, 10) the lower and upper bounds of c that is allowed to vary during hyperpara. optimizartion
                # uX.KroneckerDelta - a kronecker delta returns 1 when two features are equal and return h (the first input arg here, which is 0.5 in this case) otherwise
                #   (0.1, 0.9) the lower and upper bounds that h is allowed to vary during hyperpara. optimization
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 2nd element of graphdot.graph.Graph.nodes
                atomic_number=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 3rd element of graphdot.graph.Graph.nodes
                # uX.SquareExponential - Equ. 26 in the paper
                # input arg. length_sacle is a float32, set as 1 in this case, which correspond to approx. 1 of the kernal value.
                # This is used to determins how quicklys should the kernel decay to zero.
                charge=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0), # the 4th element of graphdot.graph.Graph.nodes
                chiral=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 5th element of graphdot.graph.Graph.nodes
                hcount=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0), # the 6th element of graphdot.graph.Graph.nodes
                hybridization=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 7th element of graphdot.graph.Graph.nodes
                # uX.Convolution - a convolutional microkernel which averages evaluations of a base microkernel between pairs pf elememts of two variable-length feature sequences
                # uX.KroneckerDelta as the base kernel
                ring_list=uX.Constant(0.5, (0.01, 100.0)) * uX.Convolution(uX.KroneckerDelta(0.5,(0.1, 0.9))) # the 8th element of graphdot.graph.Graph.nodes
            ).normalized,
            # edge_kernel - A kernelet that computes the similarity between individual edge
            edge_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 3rd element of graphdot.graph.Graph.nodes
                conjugated=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)), # the 4th element of graphdot.graph.Graph.nodes
                order=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 5th element of graphdot.graph.Graph.nodes
                ring_stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)), # the 6th element of graphdot.graph.Graph.nodes
                stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)) # the 7th element of graphdot.graph.Graph.nodes
            ).normalized,
            p=Uniform(1.0, p_bounds='fixed'), # the strating probability of the random walk on each node
            q=0.05 # the probability for the random walk to stop during each step
        )
    ),
    alpha=1e-4, # value added to the diagonal of the kernel matrix during fitting
    optimizer=True, # default optimizer of L-BFGS-B based on scipy.optimize.minimize
    normalize_y=True, # normalize the y values so taht the means and variance is 0 and 1, repsectively. Will be reversed when predicions are returned
    regularization='+', # alpha (1e-4 in this case) is added to the diagonals of the kernal matrix
)
```

```
def train_pipeline_batch(model, train_dataset, test_dataset, target, repeats, batch_num_train, batch_size, verbose = True, print_batch = True, print_repeat = True):
  start_time = time.time()
  for repeat in range(0, repeats):
    for batch in range(0, int(batch_num_train)-1):
      batch_dataset = dataset.iloc[batch*batch_size:(batch+1)*batch_size] # divide the training data into different batches
      np.random.seed(0)
      if batch == 0 and repeat == 0:
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename='batch_0_repeat_0.pkl', overwrite=True)
      elif batch == 0 and repeat !=0:
        filename_load = 'batch_0_repeat_'+str(repeat-1)+'.pkl'
        filename_save = 'batch_0_repeat_'+str(repeat)+'.pkl'
        model.load(path="/content/pkls", filename=filename_load)
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename=filename_save, overwrite=True)
      else:
        filename_load = 'batch_'+str(batch-1)+'_repeat_'+str(repeat)+'.pkl'
        filename_save = 'batch_'+str(batch)+'_repeat_'+str(repeat)+'.pkl'
        model.load(path="/content/pkls", filename=filename_load)
        model.fit(batch_dataset.graphs, batch_dataset[target], repeat=1, verbose=verbose)
        model.save(path="/content/pkls", filename=filename_save, overwrite=True)

      if print_batch:
        mu = gpr.predict(train_dataset.graphs)
        print('Training set of repeat '+str(repeat)+' and batch '+str(batch))
        print('MAE '+str(repeat)+' and batch '+str(batch)+' '+str(np.mean(np.abs(train_dataset[target] - mu))))
        print('MAE '+str(repeat)+' and batch '+str(batch)+' '+str(np.std(train_dataset[target] - mu)))

    if print_repeat:
      mu = gpr.predict(train_dataset.graphs)
      print('Training set of repeat '+str(repeat))
      print('MAE '+str(repeat)+' '+str(np.mean(np.abs(train_dataset[target] - mu))))
      print('RMSE '+str(repeat)+' '+str(np.std(train_dataset[target] - mu)))

      mu = gpr.predict(test_dataset.graphs)
      print('Test set of repeat '+str(repeat))
      print('MAE '+str(repeat)+' '+str(np.mean(np.abs(test_dataset[target] - mu))))
      print('RMSE '+str(repeat)+' '+str(np.std(test_dataset[target] - mu)))

  end_time = time.time()
  print("the total time consumption is " + str(end_time - start_time) + ".")
```

```
train_pipeline_batch(gpr, train_data, test_data, target, repeats=2, batch_num_train=batch_num_train, batch_size=batch_size, print_batch = False, print_repeat = True)
```

    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  2.7019e+06|  2.2685e+07|  2.7092e+06|       -7280|  8.8043e+06|        24|      0.25|
    |  1.8512e+06|  1.8823e+07|  1.8579e+06|     -6735.2|  8.2346e+06|        19|      0.27|
    |  1.2939e+06|  1.4683e+07|  1.3001e+06|     -6265.2|  7.4903e+06|        19|      0.24|
    |   1.021e+06|  1.1806e+07|   1.027e+06|     -5976.5|  6.9101e+06|        18|      0.24|
    |  8.4287e+05|  9.4751e+06|  8.4862e+05|     -5749.5|  6.3466e+06|        18|      0.23|
    |  7.2809e+05|  7.8466e+06|  7.3366e+05|     -5576.7|  5.8859e+06|        17|      0.25|
    |  6.6283e+05|  6.9824e+06|   6.683e+05|     -5466.4|  5.6067e+06|        17|      0.24|
    |   6.453e+05|  6.6156e+06|  6.5075e+05|     -5449.8|  5.3891e+06|        16|      0.26|
    |  6.2547e+05|  6.4794e+06|  6.3087e+05|     -5403.8|  5.3829e+06|        17|      0.24|
    |  6.2055e+05|  6.4456e+06|  6.2594e+05|     -5391.4|   5.384e+06|        17|      0.24|
    |  6.1894e+05|  6.4304e+06|  6.2433e+05|     -5388.3|  5.3926e+06|        17|      0.25|
    |   6.178e+05|  6.4214e+06|  6.2318e+05|     -5386.2|  5.3914e+06|        17|      0.24|
    |  6.1814e+05|  6.3747e+06|  6.2354e+05|     -5397.9|  5.3791e+06|        17|      0.25|
    |  6.1618e+05|  6.3821e+06|  6.2157e+05|     -5388.5|  5.3778e+06|        17|      0.25|
    |  6.1335e+05|  6.3652e+06|  6.1873e+05|     -5381.7|  5.3525e+06|        17|      0.26|
    |  6.1233e+05|  6.3563e+06|  6.1771e+05|     -5379.5|  5.3384e+06|        17|      0.25|
    |  6.1205e+05|  6.3576e+06|  6.1743e+05|     -5378.5|  5.3368e+06|        17|      0.26|
    |  6.1146e+05|  6.3598e+06|  6.1684e+05|       -5377|  5.3372e+06|        17|      0.24|
    |  6.1112e+05|  6.3567e+06|  6.1649e+05|     -5377.5|  5.3386e+06|        17|      0.26|
    |  6.1095e+05|  6.3619e+06|  6.1633e+05|     -5379.9|  5.3312e+06|        17|      0.25|
    |  6.1097e+05|  6.3587e+06|  6.1635e+05|     -5379.1|  5.3324e+06|        17|      0.24|
    |  6.1098e+05|  6.3615e+06|  6.1636e+05|     -5379.6|   5.331e+06|        17|      0.25|
    |  6.1095e+05|  6.3619e+06|  6.1633e+05|     -5379.8|  5.3323e+06|        17|      0.27|
    |  6.1109e+05|   6.364e+06|  6.1647e+05|     -5379.7|  5.3318e+06|        17|      0.25|
    |  6.1094e+05|  6.3613e+06|  6.1632e+05|     -5379.8|  5.3306e+06|        17|      0.25|
    |  6.1084e+05|  6.3595e+06|  6.1622e+05|     -5379.8|  5.3289e+06|        17|      0.26|
    |  6.1099e+05|  6.3624e+06|  6.1637e+05|     -5379.8|  5.3313e+06|        17|      0.25|
    |  6.1114e+05|   6.366e+06|  6.1652e+05|     -5379.8|  5.3311e+06|        17|      0.24|
    |  6.1091e+05|  6.3612e+06|  6.1629e+05|     -5379.8|  5.3308e+06|        17|      0.25|
    |  6.1081e+05|  6.3455e+06|  6.1619e+05|     -5378.9|  5.3347e+06|        17|      0.26|
    |  6.1077e+05|  6.3501e+06|  6.1615e+05|     -5378.8|  5.3316e+06|        17|      0.25|
    |  6.1084e+05|  6.3545e+06|  6.1622e+05|     -5378.8|  5.3302e+06|        17|      0.27|
    |  6.1078e+05|  6.3509e+06|  6.1616e+05|     -5378.8|   5.336e+06|        17|      0.25|
    |  6.1085e+05|  6.3511e+06|  6.1622e+05|     -5378.8|  5.3315e+06|        17|      0.24|
    |  6.1093e+05|  6.3539e+06|  6.1631e+05|     -5378.8|  5.3313e+06|        17|      0.25|
    |  6.1077e+05|  6.3502e+06|  6.1614e+05|     -5378.8|  5.3306e+06|        17|      0.26|
    |  6.1078e+05|  6.3505e+06|  6.1616e+05|     -5378.8|   5.332e+06|        17|      0.25|
    |  6.1086e+05|  6.3518e+06|  6.1624e+05|     -5378.8|  5.3318e+06|        17|      0.25|
    Optimization result:
          fun: 610859.9288216699
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 6.34797657e+02,  1.24118936e+02,  2.81935737e+01, -3.04708834e+01,
            1.44803317e+04,  2.72483890e+02,  0.00000000e+00,  1.70986591e+01,
            3.73893389e+01, -3.08340510e+02,  6.74914910e-19,  9.64312176e+01,
            3.50275984e+01, -1.88838360e+02,  1.59526427e+04,  1.86513089e+02,
            2.60343189e+03,  2.09813558e+02,  2.99666451e+03, -4.67071932e+02,
            1.27795063e+02,  1.00516230e+02,  0.00000000e+00, -3.03309420e+01,
            3.52006775e+02])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 38
          nit: 22
       status: 0
      success: True
            x: array([-9.21034037, -4.60517019, -1.83482135,  1.56469333, -2.30258509,
           -4.60517019,  0.        , -4.34462918, -2.30258509,  2.30258509,
           -2.38498025, -4.60517019, -1.79605729,  1.66070648, -2.30258509,
            0.85422024, -2.30258509,  0.995156  , -2.30258509, -2.68851944,
           -2.30258509, -4.60517019, -0.22314355, -1.16410687, -2.30258509])
    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  5.9481e+05|  6.2118e+06|  6.0024e+05|     -5429.6|  5.7437e+06|        18|      0.24|
    |  6.0133e+05|  6.1615e+06|  6.0679e+05|     -5458.5|  5.8272e+06|        17|      0.25|
    |   5.944e+05|  6.1849e+06|  5.9983e+05|     -5431.9|  5.7542e+06|        17|      0.28|
    |  5.9439e+05|  6.1855e+06|  5.9982e+05|     -5431.7|  5.7515e+06|        17|      0.25|
    |  5.9431e+05|  6.1854e+06|  5.9974e+05|     -5430.8|  5.7528e+06|        17|      0.25|
    |  5.9384e+05|  6.1725e+06|  5.9927e+05|       -5430|  5.7482e+06|        17|      0.24|
    |   5.938e+05|  6.1683e+06|  5.9923e+05|     -5429.8|  5.7531e+06|        17|      0.26|
    |   5.937e+05|  6.1624e+06|  5.9913e+05|     -5429.9|  5.7516e+06|        17|      0.25|
    |   5.937e+05|  6.1583e+06|  5.9913e+05|       -5430|   5.752e+06|        17|      0.24|
    |  5.9345e+05|  6.1559e+06|  5.9888e+05|     -5429.9|  5.7505e+06|        17|      0.24|
    |  5.9359e+05|    6.15e+06|  5.9902e+05|       -5430|  5.7568e+06|        17|      0.24|
    |  5.9367e+05|  6.1598e+06|   5.991e+05|     -5429.9|  5.7535e+06|        17|      0.24|
    |  5.9365e+05|  6.1605e+06|  5.9908e+05|     -5429.9|  5.7486e+06|        17|      0.24|
    |   5.935e+05|  6.1566e+06|  5.9893e+05|     -5429.9|  5.7468e+06|        17|      0.25|
    |   5.936e+05|  6.1592e+06|  5.9903e+05|     -5429.9|  5.7476e+06|        17|      0.25|
    |  5.9361e+05|  6.1594e+06|  5.9904e+05|     -5429.9|   5.754e+06|        17|      0.24|
    |  5.9352e+05|  6.1573e+06|  5.9895e+05|     -5429.9|  5.7496e+06|        17|      0.25|
    Optimization result:
          fun: 593521.9933959157
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 6.15385325e+02,  1.01738091e+02,  2.82061742e+01, -1.82698311e+02,
            1.40009664e+04,  2.61063767e+02,  0.00000000e+00,  9.53384764e+01,
            2.31015442e+01, -1.30701897e+02,  6.56896566e-19, -5.34736159e+01,
            8.94133316e+01, -1.06596521e+02,  1.42422016e+04,  3.80569253e+02,
            2.36758463e+03, -3.88042888e+01,  2.54268625e+03, -3.44421752e+02,
            1.49739713e+02,  7.35780675e+01,  0.00000000e+00, -6.95844521e+01,
            8.65877797e+01])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 17
          nit: 8
       status: 0
      success: True
            x: array([-9.21034037, -4.60517019, -1.89424069,  1.5728804 , -2.30258509,
           -4.60517019,  0.        , -4.45529998, -2.30258509,  2.30258509,
           -2.38498025, -4.05997156, -1.8923729 ,  1.59072464, -2.30258509,
            1.08135167, -2.30258509,  1.13352734, -2.30258509, -1.99203168,
           -2.30258509, -4.60517019, -0.22314355, -2.33867933, -2.30258509])
    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  6.7918e+05|  6.7795e+06|  6.8463e+05|     -5443.6|  5.6734e+06|        19|      0.25|
    |  6.8865e+05|  7.0808e+06|   6.941e+05|     -5452.2|  5.6674e+06|        18|      0.27|
    |  6.7806e+05|   6.816e+06|   6.835e+05|       -5439|  5.6573e+06|        18|      0.25|
    |  6.7756e+05|  6.8056e+06|    6.83e+05|     -5439.1|  5.6628e+06|        18|      0.24|
    |  6.7769e+05|  6.8098e+06|  6.8312e+05|     -5439.1|  5.6645e+06|        18|      0.23|
    |  6.7751e+05|  6.8055e+06|  6.8295e+05|     -5439.1|  5.6599e+06|        18|      0.25|
    |  6.7756e+05|  6.8066e+06|    6.83e+05|     -5439.1|  5.6626e+06|        18|      0.25|
    |   6.775e+05|  6.8048e+06|  6.8294e+05|     -5439.1|    5.66e+06|        18|      0.26|
    |  6.7761e+05|  6.8072e+06|  6.8305e+05|     -5439.1|  5.6612e+06|        18|      0.24|
    |  6.7753e+05|  6.8053e+06|  6.8297e+05|     -5439.1|  5.6621e+06|        18|      0.26|
    |  6.7773e+05|    6.81e+06|  6.8317e+05|     -5439.1|  5.6581e+06|        18|      0.24|
    |  6.7767e+05|  6.8085e+06|  6.8311e+05|     -5439.1|   5.661e+06|        18|      0.24|
    |  6.7769e+05|  6.8095e+06|  6.8313e+05|     -5439.1|  5.6576e+06|        18|      0.23|
    |  6.7766e+05|  6.8076e+06|   6.831e+05|     -5439.1|  5.6636e+06|        18|      0.24|
    |  6.7761e+05|  6.8066e+06|  6.8305e+05|     -5439.1|  5.6631e+06|        18|      0.24|
    |  6.7769e+05|  6.8089e+06|  6.8313e+05|     -5439.1|  5.6632e+06|        18|      0.25|
    Optimization result:
          fun: 677692.6401785224
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 6.80512584e+02,  1.28840642e+02,  2.96629868e+01, -7.29874759e+02,
            1.36275207e+04,  2.98268612e+02,  0.00000000e+00, -1.77306432e+02,
            6.43238550e+01, -1.79567708e+02,  6.78413983e-19,  5.27798047e+01,
            7.98979646e+01,  6.02840236e+02,  1.72504386e+04,  4.53894124e+02,
            2.48814034e+03, -1.14971440e+03,  3.28390348e+03,  5.53570210e+01,
            1.40288671e+02,  8.81082261e+01,  0.00000000e+00,  5.53603835e+02,
            2.79202758e+01])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 16
          nit: 4
       status: 0
      success: True
            x: array([-9.21034037, -4.60517019, -1.9033415 ,  1.40832592, -2.30258509,
           -4.60517019,  0.        , -4.30759609, -2.30258509,  2.1892741 ,
           -2.38498025, -4.07223178, -1.90198197,  1.65353414, -2.30258509,
            0.95306768, -2.30258509,  1.17073124, -2.30258509, -1.89984995,
           -2.30258509, -4.60517019, -0.22314355, -2.39282005, -2.30258509])
    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  5.8564e+05|   6.384e+06|  5.9109e+05|     -5447.9|  5.8972e+06|        19|      0.25|
    |  5.9558e+05|  6.7613e+06|  6.0105e+05|     -5478.2|  5.9725e+06|        18|      0.26|
    |  5.8358e+05|  6.4363e+06|  5.8903e+05|     -5448.4|  5.9087e+06|        17|      0.24|
    |  5.8333e+05|  6.4272e+06|  5.8878e+05|     -5448.5|  5.9124e+06|        18|      0.25|
    |  5.8349e+05|  6.4306e+06|  5.8894e+05|     -5448.6|  5.9182e+06|        18|      0.24|
    |   5.834e+05|  6.4289e+06|  5.8885e+05|     -5448.5|  5.9177e+06|        18|      0.24|
    |  5.8332e+05|  6.4267e+06|  5.8877e+05|     -5448.5|  5.9153e+06|        18|      0.24|
    |  5.8345e+05|  6.4296e+06|  5.8889e+05|     -5448.5|  5.9147e+06|        18|      0.26|
    |  5.8339e+05|  6.4282e+06|  5.8884e+05|     -5448.5|  5.9176e+06|        17|      0.24|
    |   5.835e+05|  6.4307e+06|  5.8895e+05|     -5448.5|  5.9164e+06|        17|      0.24|
    Optimization result:
          fun: 583498.1821292299
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 6.42636679e+02,  1.49586433e+02,  2.61368055e+01, -7.23074350e+02,
            1.43314276e+04,  3.00500436e+02,  0.00000000e+00, -2.05942373e+02,
            7.55571246e+01, -7.56525792e+02,  5.58467338e-19,  1.68639173e+02,
            5.83197677e+01,  1.03984575e+03,  1.81859964e+04,  5.85507772e+02,
            1.73017504e+03, -4.20613902e+02,  2.69164210e+03,  2.03198468e+02,
            8.87212800e+01,  7.15099972e+01,  0.00000000e+00, -4.38153824e+02,
            1.30113827e+02])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 10
          nit: 3
       status: 0
      success: True
            x: array([-9.21034037, -4.60517019, -1.91300979,  1.451537  , -2.30258509,
           -4.60517019,  0.        , -4.14911734, -2.30258509,  2.03112438,
           -2.38498025, -4.0860788 , -1.91191534,  1.70143255, -2.30258509,
            0.81516434, -2.30258509,  1.20228816, -2.30258509, -1.96563782,
           -2.30258509, -4.60517019, -0.22314355, -2.27803721, -2.30258509])
    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  5.6983e+05|  6.1842e+06|   5.752e+05|     -5376.5|  5.3707e+06|        19|      0.24|
    |  5.7184e+05|  6.0505e+06|  5.7724e+05|     -5396.4|  5.4669e+06|        17|      0.24|
    |  5.6769e+05|  6.0911e+06|  5.7307e+05|       -5379|  5.4036e+06|        17|      0.23|
    |  5.6717e+05|  6.0754e+06|  5.7255e+05|     -5378.1|  5.3927e+06|        17|      0.24|
    |   5.672e+05|  6.0787e+06|  5.7258e+05|     -5378.2|  5.3932e+06|        17|      0.23|
    |  5.6724e+05|  6.0771e+06|  5.7261e+05|     -5378.1|   5.396e+06|        17|      0.24|
    |  5.6712e+05|  6.0745e+06|   5.725e+05|     -5378.1|  5.3921e+06|        17|      0.25|
    |  5.6718e+05|  6.0754e+06|  5.7256e+05|     -5378.1|  5.3904e+06|        17|      0.24|
    |  5.6719e+05|  6.0766e+06|  5.7257e+05|     -5378.1|  5.3949e+06|        17|      0.25|
    |  5.6715e+05|  6.0753e+06|  5.7253e+05|     -5378.1|  5.3903e+06|        17|      0.23|
    |  5.6725e+05|    6.08e+06|  5.7262e+05|     -5378.2|  5.3918e+06|        17|      0.23|
    |  5.6717e+05|  6.0761e+06|  5.7255e+05|     -5378.1|  5.3925e+06|        17|      0.25|
    |  5.6715e+05|  6.0754e+06|  5.7253e+05|     -5378.1|  5.3897e+06|        17|      0.24|
    |  5.6708e+05|  6.0739e+06|  5.7245e+05|     -5378.1|  5.3918e+06|        17|      0.24|
    |  5.6736e+05|  6.0803e+06|  5.7274e+05|     -5378.1|  5.3887e+06|        17|      0.25|
    |  5.6724e+05|  6.0783e+06|  5.7261e+05|     -5378.1|  5.3911e+06|        17|      0.24|
    |  5.6717e+05|  6.0757e+06|  5.7254e+05|     -5378.1|  5.3936e+06|        17|      0.24|
    Optimization result:
          fun: 567166.579508662
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 6.07208753e+02,  1.16400806e+02,  2.94031809e+01, -1.72776793e+03,
            1.35443977e+04,  2.88635663e+02,  0.00000000e+00, -1.08433778e+03,
            1.83096192e+02,  2.13385801e+03,  6.02314568e-19,  1.24870900e+02,
            6.00556807e+01,  1.45101987e+02,  1.41791665e+04,  1.50959272e+03,
            2.28162930e+03, -1.61859228e+03,  3.18130856e+03, -3.51209513e+02,
            2.08717946e+02,  9.51302212e+01,  0.00000000e+00,  3.65194505e+02,
            5.97260799e+01])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 17
          nit: 4
       status: 0
      success: True
            x: array([-9.21034037, -4.60517019, -1.92550128,  1.42696149, -2.30258509,
           -4.60517019,  0.        , -3.93589447, -2.30258509,  2.10117436,
           -2.38498025, -4.10359282, -1.9247698 ,  1.4879669 , -2.30258509,
            0.84022785, -2.30258509,  1.04438252, -2.30258509, -1.82947763,
           -2.30258509, -4.60517019, -0.22314355, -2.35477534, -2.30258509])
    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |  6.0983e+05|  6.5503e+06|  6.1526e+05|       -5422|  5.6888e+06|        19|      0.23|
    |  6.2633e+05|  7.0232e+06|  6.3178e+05|     -5450.2|  5.7844e+06|        18|      0.24|
    |  6.0916e+05|  6.5794e+06|  6.1458e+05|       -5421|  5.6922e+06|        17|      0.23|
    |  6.0896e+05|   6.576e+06|  6.1438e+05|     -5420.9|  5.6982e+06|        17|      0.25|
    |  6.0878e+05|  6.5684e+06|   6.142e+05|     -5422.1|  5.7015e+06|        17|      0.24|
    |  6.0878e+05|    6.57e+06|   6.142e+05|     -5422.5|  5.7048e+06|        17|      0.23|
    |  6.0875e+05|  6.5676e+06|  6.1417e+05|     -5422.2|  5.7044e+06|        17|      0.25|
    |  6.0865e+05|  6.5657e+06|  6.1407e+05|     -5422.2|  5.7012e+06|        17|      0.24|
    |  6.0856e+05|  6.5647e+06|  6.1399e+05|     -5422.4|  5.7036e+06|        17|      0.24|
    |   6.087e+05|  6.5675e+06|  6.1412e+05|     -5422.4|  5.7052e+06|        17|      0.25|
    |  6.0858e+05|  6.5645e+06|    6.14e+05|     -5422.4|  5.7034e+06|        17|      0.25|
    |  6.0817e+05|  6.5775e+06|   6.136e+05|     -5426.9|  5.7249e+06|        17|      0.25|
    |   6.079e+05|   6.572e+06|  6.1333e+05|     -5424.7|  5.7167e+06|        17|      0.25|
    |  6.0784e+05|  6.5813e+06|  6.1326e+05|       -5424|   5.709e+06|        18|      0.25|
    |  6.0782e+05|  6.5832e+06|  6.1325e+05|     -5424.1|  5.7066e+06|        18|      0.26|
    |  6.0774e+05|  6.5826e+06|  6.1316e+05|       -5424|  5.7022e+06|        17|      0.24|
    |  6.0778e+05|  6.5896e+06|  6.1321e+05|     -5423.9|  5.7019e+06|        17|      0.24|
    |   6.078e+05|   6.586e+06|  6.1322e+05|       -5424|  5.7052e+06|        17|      0.23|
    |  6.0772e+05|  6.5833e+06|  6.1314e+05|       -5424|  5.7057e+06|        17|      0.25|
    |  6.0775e+05|  6.5827e+06|  6.1317e+05|       -5424|  5.7018e+06|        17|      0.24|
    |  6.0771e+05|  6.5828e+06|  6.1313e+05|       -5424|  5.7036e+06|        17|      0.23|
    |   6.078e+05|  6.5908e+06|  6.1322e+05|     -5424.2|  5.7032e+06|        17|      0.23|
    |  6.0783e+05|  6.5851e+06|  6.1325e+05|     -5424.1|  5.7023e+06|        17|      0.22|
    |  6.0773e+05|  6.5828e+06|  6.1315e+05|       -5424|  5.7061e+06|        17|      0.23|
    |  6.0772e+05|   6.583e+06|  6.1314e+05|       -5424|  5.7041e+06|        17|      0.23|

```
gpr.kernel.hyperparameters
```

    starting_probability : Uniform
    	p : 1.0
    stopping_probability : 0.00010000000000000009
    node_kernel : Composite
    	aromatic : Multiply
    		lhs : Constant
    			c : 0.02343420535068002
    		rhs : KroneckerDelta
    			h : 0.18860200789175988
    	atomic_number : Multiply
    		lhs : Constant
    			c : 5.400776905055444
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	charge : Multiply
    		lhs : Constant
    			c : 0.0145590563621646
    		rhs : SquareExponential
    			length_scale : 1.0
    	chiral : Multiply
    		lhs : Constant
    			c : 0.05149063984443944
    		rhs : KroneckerDelta
    			h : 0.12853164901299938
    	hcount : Multiply
    		lhs : Constant
    			c : 10.000000000000002
    		rhs : SquareExponential
    			length_scale : 0.07795674396128768
    	hybridization : Multiply
    		lhs : Constant
    			c : 0.02783411902444354
    		rhs : KroneckerDelta
    			h : 0.17414808329497075
    	ring_list : Multiply
    		lhs : Constant
    			c : 6.550981640628747
    		rhs : Convolution
    			base : KroneckerDelta
    				h : 0.10000000000000002
    edge_kernel : Composite
    	aromatic : Multiply
    		lhs : Constant
    			c : 2.331300736214766
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	conjugated : Multiply
    		lhs : Constant
    			c : 3.166641607522053
    		rhs : KroneckerDelta
    			h : 0.10000000000000002
    	order : Multiply
    		lhs : Constant
    			c : 0.05549109210250382
    		rhs : KroneckerDelta
    			h : 0.25478481015091287
    	ring_stereo : Multiply
    		lhs : Constant
    			c : 0.033323606027544916
    		rhs : KroneckerDelta
    			h : 0.8
    	stereo : Multiply
    		lhs : Constant
    			c : 0.1324351226762118
    		rhs : KroneckerDelta
    			h : 0.14521546104851712

```
mu = gpr.predict(train.graphs)
```

```
plt.scatter(train[target], mu)
plt.show()
```

```
print('Training set')
print('MAE:', np.mean(np.abs(train[target] - mu)))
print('RMSE:', np.std(train[target] - mu))
```

    Training set
    MAE: 0.2074369392067814
    RMSE: 0.3195501910711287

```
mu_test = gpr.predict(test.graphs)
```

```
plt.scatter(test[target], mu_test)
plt.show()
```

```
print('Test set')
print('MAE:', np.mean(np.abs(test[target] - mu_test)))
print('RMSE:', np.std(test[target] - mu_test))
```

    Test set
    MAE: 1.2461703788060545
    RMSE: 1.6040210737240486

Workon the kernel. Find a kernel that trains and predicts well.

```
gpr2 = GaussianProcessRegressor(
    kernel=Normalization(
        MarginalizedGraphKernel(
            node_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                atomic_number=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                charge=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0),
                chiral=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                hcount=uX.Constant(0.5, (0.01, 10.0)) * uX.SquareExponential(1.0),
                hybridization=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                ring_list=uX.Constant(0.5, (0.01, 100.0)) * uX.Convolution(uX.KroneckerDelta(0.5,(0.1, 0.9)))
            ).normalized,
            edge_kernel=uX.Additive(
                aromatic=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                conjugated=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.5,(0.1, 0.9)),
                order=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                ring_stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9)),
                stereo=uX.Constant(0.5, (0.01, 10.0)) * uX.KroneckerDelta(0.8,(0.1, 0.9))
            ).normalized,
            p=Uniform(1.0, p_bounds='fixed'),
            q=0.05
        )
    ),
    alpha=1e-2, #different from gpr in alpha where gpr's alpha is 1e-4
    optimizer=True,
    normalize_y=True,
    regularization='+',
)
```

```
#gpr2.fit(train.graphs, train[target], repeat=3, verbose=True)
gpr2.fit(train.graphs, train[target], repeat=1, verbose=True)
```

    |        logP|       dlogP|     y^T.K.y|     log|K| |     Cond(K)|  GPU time|  CPU time|
    |------------|------------|------------|------------|------------|----------|----------|
    |   1.667e+05|   1.928e+05|  1.7773e+05|      -11034|  2.2742e+05|   1.3e+02|       1.7|
    |  1.5585e+05|  2.2171e+05|  1.6671e+05|      -10857|  2.2218e+05|   1.3e+02|       1.7|
    |  1.3711e+05|  2.4873e+05|  1.4756e+05|      -10447|  2.0803e+05|   1.2e+02|       1.8|
    |  1.3286e+05|  2.3627e+05|   1.432e+05|      -10338|  2.0336e+05|   1.1e+02|       1.7|
    |  1.2977e+05|  2.2744e+05|  1.4002e+05|      -10249|  1.9714e+05|   1.1e+02|       1.7|
    |  1.3006e+05|  2.1429e+05|  1.4032e+05|      -10258|  2.0119e+05|     1e+02|       1.7|
    |  1.2923e+05|  2.2103e+05|  1.3946e+05|      -10238|  1.9906e+05|   1.1e+02|       1.7|
    |  1.2888e+05|  2.2097e+05|  1.3911e+05|      -10228|  1.9865e+05|   1.1e+02|       1.7|
    |  1.2843e+05|  2.2338e+05|  1.3864e+05|      -10211|  1.9582e+05|   1.1e+02|       1.7|
    |  1.2837e+05|  2.2244e+05|  1.3858e+05|      -10212|   1.969e+05|   1.1e+02|       1.7|
    |  1.2836e+05|  2.2225e+05|  1.3857e+05|      -10212|  1.9696e+05|   1.1e+02|       1.7|
    |  1.2831e+05|  2.2155e+05|  1.3852e+05|      -10211|  1.9715e+05|   1.1e+02|       1.7|
    |  1.2826e+05|  2.2184e+05|  1.3847e+05|      -10209|  1.9704e+05|   1.1e+02|       1.7|
    |  1.2821e+05|  2.2181e+05|  1.3841e+05|      -10207|  1.9671e+05|   1.1e+02|       1.7|
    |  1.2819e+05|  2.2165e+05|   1.384e+05|      -10207|  1.9664e+05|   1.1e+02|       1.8|
    |  1.2818e+05|  2.2125e+05|  1.3838e+05|      -10207|  1.9671e+05|   1.1e+02|       1.8|
    |  1.2817e+05|  2.2114e+05|  1.3838e+05|      -10207|  1.9673e+05|   1.1e+02|       1.7|
    |  1.2815e+05|  2.2077e+05|  1.3836e+05|      -10207|  1.9677e+05|   1.1e+02|       1.7|
    |  1.2812e+05|   2.185e+05|  1.3833e+05|      -10209|  1.9695e+05|   1.1e+02|       1.7|
    |  1.2809e+05|   2.178e+05|   1.383e+05|      -10209|    1.97e+05|   1.1e+02|       1.7|
    |  1.2807e+05|  2.1764e+05|  1.3828e+05|      -10208|  1.9695e+05|   1.1e+02|       1.7|
    |  1.2805e+05|  2.1597e+05|  1.3826e+05|      -10209|  1.9693e+05|   1.1e+02|       1.7|
    |  1.2805e+05|   2.172e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1693e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.8|
    |  1.2804e+05|  2.1674e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1691e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2805e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    |  1.2804e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.8|
    |  1.2805e+05|  2.1694e+05|  1.3825e+05|      -10208|  1.9694e+05|   1.1e+02|       1.7|
    Optimization result:
          fun: 128045.20193517115
     hess_inv: <25x25 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 21.67035135,   5.42797631,   0.91584794,   6.62684845,
           636.88297304,  10.22203408,   0.        ,   6.13155297,
             0.80318432, -45.79780911,   0.        ,   9.32446636,
             2.06137402,   7.94609186, 767.483899  ,  -8.19180354,
            92.93654292,   7.92503084, 118.62629465,   3.36719643,
            38.71995733,   7.96834618,   0.        , -11.08475724,
             7.2860016 ])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 31
          nit: 21
       status: 0
      success: True
            x: array([-9.21034037, -4.51710969, -1.98267626,  1.72521606, -2.30258509,
           -4.60517019,  0.        , -4.60517019, -1.8070945 ,  2.30258509,
           -2.88031384, -3.6019383 , -2.30258509,  1.91177723, -2.30258509,
            0.28746812, -2.30258509,  0.54884256, -2.30258509, -0.5686784 ,
           -2.30258509, -4.35637808, -0.22314355, -2.55717239, -2.21179129])

    <graphdot.model.gaussian_process.gpr.GaussianProcessRegressor at 0x7f6bf00630d0>

```
mu = gpr2.predict(train.graphs)
plt.scatter(train[target], mu)
plt.show()
```

```
print('Training set')
print('MAE:', np.mean(np.abs(train[target] - mu)))
print('RMSE:', np.std(train[target] - mu))
```

    Training set
    MAE: 0.7238389142657052
    RMSE: 0.9634943206325994

```
mu_test = gpr2.predict(test.graphs)
plt.scatter(test[target], mu_test)
plt.show()
```

```
print('Test set')
print('MAE:', np.mean(np.abs(test[target] - mu_test)))
print('RMSE:', np.std(test[target] - mu_test))
```

    Test set
    MAE: 0.9561539409612109
    RMSE: 1.2284268143181998


```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir( os.path.join('..', 'notebook_format') )
from formats import load_style
load_style()
```

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2

from sklearn import datasets
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    Ethen 2017-06-22 14:02:06

    CPython 3.5.2
    IPython 5.4.1

    numpy 1.12.1
    pandas 0.20.2
    matplotlib 2.0.2
    sklearn 0.18.1

# Softmax Regression

**Softmax Regression** is a generalization of logistic regression that we can use for multi-class classification. If we want to assign probabilities to an object being one of several different things, softmax is the thing to do. Even later on, when we start training neural network models, the final step will be a layer of softmax.

A softmax regression has two steps: first we add up the evidence of our input being in certain classes, and then we convert that evidence into probabilities.

In **Softmax Regression**, we replace the sigmoid logistic function by the so-called *softmax* function $\phi(\cdot)$.

$$P(y=j \mid z^{(i)}) = \phi(z^{(i)}) = \frac{e^{z^{(i)}}}{\sum_{j=1}^{k} e^{z_{j}^{(i)}}}$$

where we define the net input *z* as

$$z = w_1x_1 + ... + w_mx_m  + b= \sum_{l=1}^{m} w_l x_l + b= \mathbf{w}^T\mathbf{x} + b$$

(**w** is the weight vector, $\mathbf{x}$ is the feature vector of 1 training sample. Each $w$ corresponds to a feature $x$ and there're $m$ of them in total. $b$ is the bias unit. $k$ denotes the total number of classes.)

Now, this softmax function computes the probability that the $i_{th}$ training sample $\mathbf{x}^{(i)}$ belongs to class $l$ given the weight and net input $z^{(i)}$. So given the obtained weight $w$, we're basically compute the probability, $p(y = j \mid \mathbf{x^{(i)}; w}_j)$, the probability of the training sample belonging to class $j$ for each class label in $j = 1, \ldots, k$. Note the normalization term in the denominator which causes these class probabilities to sum up to one.

We can picture our softmax regression as looking something like the following, although with a lot more $x_s$. For each output, we compute a weighted sum of the $x_s$, add a bias, and then apply softmax.

If we write that out as equations, we get:

We can "vectorize" this procedure, turning it into a matrix multiplication and vector addition. This is helpful for computational efficiency. (It's also a useful way to think.)

To illustrate the concept of softmax, let us walk through a concrete example. Suppose we have a training set consisting of 4 samples from 3 different classes (0, 1, and 2)

- $x_0 \rightarrow \text{class }0$
- $x_1 \rightarrow \text{class }1$
- $x_2 \rightarrow \text{class }2$
- $x_3 \rightarrow \text{class }2$

First, we apply one-hot encoding to encode the class labels into a format that we can more easily work with.

```python
y = np.array([0, 1, 2, 2])

def one_hot_encode(y):
    n_class = np.unique(y).shape[0]
    y_encode = np.zeros((y.shape[0], n_class))
    for idx, val in enumerate(y):
        y_encode[idx, val] = 1.0

    return y_encode

y_encode = one_hot_encode(y)
y_encode
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.],
           [ 0.,  0.,  1.]])

A sample that belongs to class 0 (the first row) has a 1 in the first cell, a sample that belongs to class 1 has a 1 in the second cell of its row, and so forth.

Next, let us define the feature matrix of our 4 training samples. Here, we assume that our dataset consists of 2 features; thus, we create a 4x2 dimensional matrix of our samples and features.
Similarly, we create a 2x3 dimensional weight matrix (one row per feature and one column for each class).

```python
X = np.array([[0.1, 0.5],
              [1.1, 2.3],
              [-1.1, -2.3],
              [-1.5, -2.5]])

W = np.array([[0.1, 0.2, 0.3],
              [0.1, 0.2, 0.3]])

bias = np.array([0.01, 0.1, 0.1])

print('Inputs X:\n', X)
print('\nWeights W:\n', W)
print('\nbias:\n', bias)
```

    Inputs X:
     [[ 0.1  0.5]
     [ 1.1  2.3]
     [-1.1 -2.3]
     [-1.5 -2.5]]

    Weights W:
     [[ 0.1  0.2  0.3]
     [ 0.1  0.2  0.3]]

    bias:
     [ 0.01  0.1   0.1 ]

To compute the net input, we multiply the 4x2 matrix feature matrix `X` with the 2x3 (n_features x n_classes) weight matrix `W`, which yields a 4x3 output matrix (n_samples x n_classes) to which we then add the bias unit:

$$\mathbf{Z} = \mathbf{X}\mathbf{W} + \mathbf{b}$$

```python
def net_input(X, W, b):
    return X.dot(W) + b

net_in = net_input(X, W, bias)
print('net input:\n', net_in)
```

    net input:
     [[ 0.07  0.22  0.28]
     [ 0.35  0.78  1.12]
     [-0.33 -0.58 -0.92]
     [-0.39 -0.7  -1.1 ]]

Now, it's time to compute the softmax activation that we discussed earlier:

$$P(y=j \mid z^{(i)}) = \phi_{softmax}(z^{(i)}) = \frac{e^{z^{(i)}}}{\sum_{j=1}^{k} e^{z_{j}^{(i)}}}$$

```python
def softmax(z):
    return np.exp(z) / np.sum(np.exp(z), axis = 1, keepdims = True)

smax = softmax(net_in)
print('softmax:\n', smax)
```

    softmax:
     [[ 0.29450637  0.34216758  0.36332605]
     [ 0.21290077  0.32728332  0.45981591]
     [ 0.42860913  0.33380113  0.23758974]
     [ 0.44941979  0.32962558  0.22095463]]

As we can see, the values for each sample (row) nicely sum up to 1 now. E.g., we can say that the first sample  `[ 0.29450637  0.34216758  0.36332605]` has a 29.45% probability to belong to class 0. Now, in order to turn these probabilities back into class labels, we could simply take the argmax-index position of each row:

[[ 0.29450637  0.34216758  **0.36332605**] -> 2
[ 0.21290077  0.32728332  **0.45981591**]  -> 2
[ **0.42860913**  0.33380113  0.23758974]  -> 0
[ **0.44941979**  0.32962558  0.22095463]] -> 0

```python
def to_classlabel(z):
    return z.argmax(axis = 1)

print('predicted class labels: ', to_classlabel(smax))
```

    predicted class labels:  [2 2 0 0]

As we can see, our predictions are terribly wrong, since the correct class labels are `[0, 1, 2, 2]`. Now, in order to train our model we need to measuring how inefficient our predictions are for describing the truth and then optimize on it. To do so we first need to define a loss/cost function $J(\cdot)$ that we want to minimize. One very common function is "cross-entropy":

$$J(\mathbf{W}; \mathbf{b}) = \frac{1}{n} \sum_{i=1}^{n} H( T^{(i)}, O^{(i)} )$$

which is the average of all cross-entropies $H$ over our $n$ training samples. The cross-entropy  function is defined as:

$$H( T^{(i)}, O^{(i)} ) = -\sum_k T^{(i)} \cdot log(O^{(i)})$$

Where:

- $T$ stands for "target" (i.e., the *true* class labels)
- $O$ stands for output -- the computed *probability* via softmax; **not** the predicted class label.
- $\sum_k$ denotes adding up the difference between the target and the output for all classes.

```python
def cross_entropy_cost(y_target, output):
    return np.mean(-np.sum(y_target * np.log(output), axis = 1))

cost = cross_entropy_cost(y_target = y_encode, output = smax)
print('Cross Entropy Cost:', cost)
```

    Cross Entropy Cost: 1.32159787159

## Gradient Descent

Our objective in training a neural network is to find a set of weights that gives us the lowest error when we run it against our training data. There're many ways to find these weights and simplest one is so called **gradient descent**. It does this by giving us directions (using derivatives) on how to "shift" our weights to an optimum. It tells us whether we should increase or decrease the value of a specific weight in order to lower the error function.

Let's imagine we have a function $f(x) = x^4 - 3x^3 + 2$ and we want to find the minimum of this function using gradient descent. Here's a graph of that function:

```python
from sympy.plotting import plot
from sympy import symbols, init_printing

# change default figure and font size
plt.rcParams['figure.figsize'] = 6, 4
plt.rcParams['font.size'] = 12

# plotting f(x) = x^4 - 3x^3 + 2, showing -2 < x <4
init_printing()
x = symbols('x')
fx = x ** 4 - 3 * x ** 3 + 2
p1 = plot(fx, (x, -2, 4), ylim = (-10, 50))
```

As you can see, there appears to be a minimum around ~2.3 or so. Gradient descent answers this question: If we were to start with a random value of x, which direction should we go if we want to get to the lowest point on this function? Let's imagine we pick a random x value, say x = 4, which would be somewhere way up on the right side of the graph. We obviously need to start going to the left if we want to get to the bottom. This is obvious when the function is an easily visualizable 2d plot, but when dealing with functions of multiple variables, we need to rely on the raw mathematics.

Calculus tells us that the derivative of a function at a particular point is the rate of change/slope of the tangent to that part of the function. So let's use derivatives to help us get to the bottom of this function. The derivative of $f(x) = x^4 - 3x^3 + 2$ is $f'(x) = 4x^3 - 9x^2$. So if we plug in our random point from above (x=4) into the first derivative of $f(x)$ we get $f'(4) = 4(4)^3 - 9(4)^2 = 112$. So how does 112 tell us where to go? Well, first of all, it's positive. If we were to compute $f'(-1)$ we get a negative number (-13). So it looks like we can say that whenever the $f'(x)$ for a particular $x$ is positive, we should move to the left (decrease x) and whenever it's negative, we should move to the right (increase x).

We'll now formalize this: When we start with a random x and compute it's deriative $f'(x)$, our new x should then be proportional to $x - f'(x)$. The word proportional is there because we wish to control to what degree we move at each step, for example when we compute $f'(4)=112$, do we really want our new $x$ to be $x - 112 = -108$? No, if we jump all the way to -108, we're even farther from the minimum than we were before. Instead, we want to take relatively small steps toward the minimum.

Let's say that for any random $x$, we want to take a step (change $x$ a little bit) such that our new $x$ $ = x - \alpha*f'(x)$. We'll call $\alpha$ (alpha) our learning rate or step size because it determines how big of a step we take. $\alpha$ is something we will just have to play around with to find a good value. Some functions might require bigger steps, others smaller steps.

Suppose we've set our $\alpha$ to be 0.001. This means, if we randomly started at $f'(4)=112$ then our new $x$ will be $ = 4 - (0.001 * 112) = 3.888$. So we moved to the left a little bit, toward the optimum. Let's do it again. $x_{new} = x - \alpha*f'(3.888) = 3.888 - (0.001 * 99.0436) = 3.79$. Nice, we're indeed moving to the left, closer to the minimum of $f(x)$, little by little. And we'll keep on doing this until we've reached convergence. By convergence, we mean that if the absolute value of the difference between the updated $x$ and the old $x$ is smaller than some randomly small number that we set, denoted as $\epsilon$ (epsilon).

```python
x_old = 0
x_new = 4 # The algorithm starts at x = 4
alpha = 0.01 # step size
epsilon = 0.00001

def f_derivative(x):
    return 4 * x ** 3 - 9 * x ** 2

while abs(x_new - x_old) > epsilon:
    x_old = x_new
    x_new = x_old - alpha * f_derivative(x_old)

print("Local minimum occurs at", x_new)
```

    Local minimum occurs at 2.2500325268933734

The script above says that if the absolute difference of $x$ between the two iterations is not changing by more than 0.00001, then we're probably at the bottom of the "bowl" because our slope is approaching 0, and therefore we should stop and call it a day. Now, if you remember some calculus and algebra, you could have solved for this minimum analytically, and you should get 2.25. Very close to what our gradient descent algorithm above found.

## More Gradient Descent...

As you might imagine, when we use gradient descent for a neural network, things get a lot more complicated. Not because gradient descent gets more complicated, it still ends up just being a matter of taking small steps downhill, it's that we need that pesky derivative in order to use gradient descent, and the derivative of a neural network cost function (with respect to its weights) is pretty intense. It's not a matter of just analytically solving $f(x)=x^2, f'(x)=2x$ , because the output of a neural net has many nested or "inner" functions.

Also unlike our toy math problem above, a neural network may have many weights. We need to find the optimal value for each individual weight to lower the cost for our entire neural net output. This requires taking the partial derivative of the cost/error function with respect to a single weight, and then running gradient descent for each individual weight. Thus, for any individual weight $W_j$, we'll compute the following:

$$ W_j^{(t + 1)} = W_j^{(t)} - \alpha * \frac{\partial L}{\partial W_j}$$

Where:

- $L$ denotes the loss function that we've defined.
- $W_j^{(t)}$ denotes the weight of the $j_{th}$ feature at iteration $t$.

And as before, we do this iteratively for each weight, many times, until the whole network's cost function is minimized.

In order to learn the weight for our softmax model via gradient descent, we then need to compute the gradient of our cost function for each class $j \in \{0, 1, ..., k\}$.

$$\nabla \mathbf{w}_j \, J(\mathbf{W}; \mathbf{b})$$

We won't be going through the tedious details here, but this cost's gradient turns out to be simply:

$$\nabla \mathbf{w}_j \, J(\mathbf{W}; \mathbf{b}) = \frac{1}{n} \sum^{n}_{i=0} \big[\mathbf{x}^{(i)}_j\ \big( O^{(i)} - T^{(i)} \big) \big]$$

We can then use the cost derivate to update the weights in opposite direction of the cost gradient with learning rate $\eta$:

$$\mathbf{w}_j := \mathbf{w}_j - \eta \nabla \mathbf{w}_j \, J(\mathbf{W}; \mathbf{b})$$

(note that $\mathbf{w}_j$ is the weight vector for the class $y=j$), and we update the bias units using:

$$
\mathbf{b}_j := \mathbf{b}_j   - \eta \bigg[ \frac{1}{n} \sum^{n}_{i=0} \big( O^{(i)} - T^{(i)} \big) \bigg]
$$

As a penalty against complexity, an approach to reduce the variance of our model and decrease the degree of overfitting by adding additional bias, we can further add a regularization term such as the L2 term with the regularization parameter $\lambda$:

$$\frac{\lambda}{2} ||\mathbf{w}||_{2}^{2}$$

where $||\mathbf{w}||_{2}^{2}$ simply means adding up the squared weights across all the features and classes.

$$||\mathbf{w}||_{2}^{2} = \sum^{m}_{l=0} \sum^{k}_{j=0} w_{l, j}^2$$

so that our cost function becomes

$$
J(\mathbf{W}; \mathbf{b}) = \frac{1}{n} \sum_{i=1}^{n} H( T^{(i)}, O^{(i)} ) + \frac{\lambda}{2} ||\mathbf{w}||_{2}^{2}
$$

and we define the "regularized" weight update as

$$
\mathbf{w}_j := \mathbf{w}_j -  \eta \big[\nabla \mathbf{w}_j \, J(\mathbf{W}) + \lambda \mathbf{w}_j \big]
$$

Note that we don't regularize the bias term, thus the update function for it stays the same.

## Softmax Regression Code

Bringing the concepts together, we could come up with an implementation as follows: Note that for the weight and bias parameter, we'll have initialize a value for it. Here we'll simply draw the weights from a normal distribution and set the bias as zero. The code can be obtained [here](https://github.com/ethen8181/machine-learning/blob/master/deep_learning/softmax.py).

```python
# import some data to play with
iris = datasets.load_iris()
X = iris.data
y = iris.target

# standardize the input features
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
```

```python
from softmax import SoftmaxRegression

# train the softmax using batch gradient descent,
# eta: learning rate, epochs : number of iterations, minibatches, number of
# training data to use for training at each iteration
softmax_reg = SoftmaxRegression(eta = 0.1, epochs = 10, minibatches = y.shape[0])
softmax_reg.fit(X_std, y)

# print the training accuracy
y_pred = softmax_reg.predict(X_std)
accuracy = np.sum(y_pred == y) / y.shape[0]
print('accuracy: ', accuracy)

# use a library to ensure comparable results
log_reg = LogisticRegression()
log_reg.fit(X_std, y)
y_pred = log_reg.predict(X_std)
print('accuracy library: ', accuracy_score(y_true = y, y_pred = y_pred))
```

    accuracy:  0.953333333333
    accuracy library:  0.926666666667

# Reference

- [Blog: Softmax Regression](http://nbviewer.jupyter.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/softmax-regression.ipynb)
- [Blog: Gradient Descent with Backpropagation](http://outlace.com/Beginner-Tutorial-Backpropagation/)
- [TensorFlow Documentation: MNIST For ML Beginners](https://www.tensorflow.org/get_started/mnist/beginners)