# Ikeda $B_e$ assumtion.

```python
from rolldecayestimators import equations
```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 461 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.dpi       : 100        ## figure dots per inch')

# Purpose
The quadratic or cubic model can be expressed using the linearized equivalent damping ($B_e$) according to .:

```python
equations.B_e_equation
```

$\displaystyle B_{e} = B_{1} + \frac{8 B_{2} \omega_{0} \phi_{a}}{3 \pi}$

```python
equations.B_e_equation_cubic
```

$\displaystyle B_{e} = B_{1} + \frac{8 B_{2} \omega_{0} \phi_{a}}{3 \pi} + 0.75 B_{3} \omega_{0}^{2} \phi_{a}^{2}$

But I have some doubt about the validity of this, which will be investigated in this notebook.

# Methodology
A quadratic and cubic model from Simplified Ikeda will be used to calculate $B_e$. $B_e$ will also be obtained from Roll-decay simulations with these models, will the value be the same?

# WIP - improvements
(WORK IN PROGRESS)
Use this section only if the notebook is not final.

Notable TODOs:
* todo 1
* todo 2
* todo 3

## Results
Describe and comment the most important results.

# Suggested next steps
State suggested next steps, based on results obtained in this notebook.

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables

```

```python
from rolldecayestimators.simplified_ikeda_class import SimplifiedIkeda
import rolldecayestimators
```

```python
df_ikeda = database.load(rolldecay_table_name='rolldecay_simplified_ikeda_unlimited', limit_score=0.95,
                             exclude_table_name='rolldecay_exclude')
```

```python
df_ikeda.head()
```

      model_number
      loading_condition_id
      ship_speed
      B_44_1
      B_F_1
      B_W_1
      B_E_1
      B_BK_1
      B_L_1
      B_44_2
      B_F_2
      B_W_2
      B_E_2
      B_BK_2
      B_L_2
      B_3
      omega0
      omega0_fft
      zeta
      d
      score
      phi_start
      phi_stop
      mean_damping
      B_1
      B_2
      B_e
      id
      project_number
      series_number
      run_number
      test_number
      ship_name
      ascii_name
      comment
      file_path_ascii
      file_path_ascii_temp
      file_path_log
      file_path_hdf5
      date
      test_type
      facility
      angle1
      angle2
      Körfallstyp
      name
      project_path
      lcg
      kg
      gm
      CW
      TF
      TA
      BWL
      KXX
      KZZ
      BTT1
      CP
      Volume
      A0
      RH
      scale_factor
      lpp
      beam
      ABULB
      BKX
      TWIN
      DCLR
      VDES
      RHBL
      ASKEG
      PD
      ARH
      CFP
      AIX
      PDTDES
      RTYPE
      SFP
      BKL
      BKB
      PROT
      D
      LSKEG
      RR
      XSKEG
      NDES
      AR
      BR
      BRA
      IRUD
      PTYPE
      XRUD
      AI
      HSKEG
      RSKEG
      LOA
      ship_type_id

      run_id

      14848
      3541-A
      107
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.691760
      6.336792
      None
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      None
      10.150949
      38.090976
      21.286281
      14848
      20136522
      1
      32
      1
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      None
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      None
      2014-12-29
      roll decay
      MDL
      None
      None
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.190
      18.38
      3.8
      0.847
      11.50
      11.50
      46.97
      15.5
      65.32
      3.36
      0.7669
      116472.0
      0.9899
      8.5
      55.000
      284.00
      46.97
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      None
      None
      20000.0
      1.0
      None
      92.13
      0.43
      3.0
      8.3
      28.0
      2.320
      -123.0
      67.8
      49.40
      0.0
      0.0
      2.0
      1.0
      -142.000
      None
      7.7
      58.0
      290.0
      4.0

      5509
      3338-B
      44
      0.0
      5.062226
      1.081253
      1.638319
      2.342654
      0.000000
      0.000000
      7.404880
      1.081253
      1.638319
      4.685307
      0.000000
      0.000000
      None
      2.450442
      2.450442
      0.004466
      0.118153
      0.985986
      0.153420
      -0.016432
      None
      2.719572
      14.682270
      7.404880
      5509
      20095421
      1
      18
      1
      3338-B
      20095421-ser001-k018-100hz.asc
      Kursstyrning Rolldecay 0.0 kn
      N:\Gamla_Projekt\ascii_files\20095421-ser001-k...
      None
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\gamla_p...
      None
      2010-09-02
      roll decay
      MDL
      None
      None
      Kursstyrning
      20095421_STX_Polar_Supply
      \\sspa.local\gbg\ProjektArkiv\2009\20095421_ST...
      -2.327
      9.00
      2.2
      0.830
      6.74
      6.74
      22.00
      8.3
      30.35
      NaN
      0.6325
      11260.0
      0.9890
      5.1
      23.889
      121.40
      22.00
      NaN
      0.000
      0.0
      NaN
      14.0
      NaN
      NaN
      0.990
      NaN
      None
      None
      NaN
      1.0
      None
      0.00
      0.00
      NaN
      4.3
      NaN
      NaN
      NaN
      NaN
      12.45
      NaN
      NaN
      2.0
      1.0
      NaN
      None
      NaN
      NaN
      133.4
      8.0

      5510
      3338-B
      44
      3.0
      8.241247
      1.081253
      1.720727
      1.645810
      0.000000
      3.793457
      9.887057
      1.081253
      1.720727
      3.291620
      0.000000
      3.793457
      None
      2.450442
      2.450442
      0.010830
      0.084283
      0.984364
      0.151097
      -0.002482
      None
      6.595437
      10.473446
      9.887057
      5510
      20095421
      1
      19
      1
      3338-B
      20095421-ser001-k019-100hz.asc
      Kursstyrning Rolldecay 3.0 kn
      N:\Gamla_Projekt\ascii_files\20095421-ser001-k...
      None
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\gamla_p...
      None
      2010-09-02
      roll decay
      MDL
      None
      None
      Kursstyrning
      20095421_STX_Polar_Supply
      \\sspa.local\gbg\ProjektArkiv\2009\20095421_ST...
      -2.327
      9.00
      2.2
      0.830
      6.74
      6.74
      22.00
      8.3
      30.35
      NaN
      0.6325
      11260.0
      0.9890
      5.1
      23.889
      121.40
      22.00
      NaN
      0.000
      0.0
      NaN
      14.0
      NaN
      NaN
      0.990
      NaN
      None
      None
      NaN
      1.0
      None
      0.00
      0.00
      NaN
      4.3
      NaN
      NaN
      NaN
      NaN
      12.45
      NaN
      NaN
      2.0
      1.0
      NaN
      None
      NaN
      NaN
      133.4
      8.0

      3462
      3250
      22
      0.0
      9.687631
      1.070726
      5.466792
      0.649940
      2.500173
      0.000000
      14.004694
      1.070726
      5.466792
      1.299880
      6.167296
      0.000000
      None
      2.689203
      2.689203
      0.009063
      0.220252
      0.996212
      -0.155870
      0.003981
      None
      5.370568
      24.266868
      14.004694
      3462
      20074604
      1
      15
      1
      3250
      20074604-ser001-k015-100hz.asc
      Kursstyrning Rolldecay
      N:\Gamla_Projekt\ascii_files\20074604-ser001-k...
      None
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\gamla_p...
      None
      2008-01-17
      roll decay
      MDL
      None
      None
      Kursstyrning
      20074604-1
      \\sspa.local\gbg\ProjektArkiv\2007\20074604
      6.401
      15.10
      5.1
      NaN
      15.25
      15.25
      48.00
      17.8
      63.30
      1.95
      NaN
      161867.0
      0.9985
      12.0
      56.463
      263.75
      48.00
      67.0
      19.780
      0.0
      0.0
      NaN
      12.1
      130.0
      0.870
      16.9
      None
      None
      16730.0
      1.0
      None
      65.90
      0.45
      1.0
      8.3
      26.0
      2.708
      -114.0
      NaN
      88.40
      0.0
      0.0
      1.0
      1.0
      -131.875
      None
      8.4
      2.0
      285.0
      1.0

      5511
      3338-B
      44
      14.0
      27.818388
      1.100489
      8.753260
      0.261840
      0.000000
      17.702798
      28.080227
      1.100489
      8.753260
      0.523680
      0.000000
      17.702798
      None
      2.538407
      2.538407
      0.046872
      0.013848
      0.964556
      -0.151557
      0.002947
      None
      27.556548
      1.603654
      28.080227
      5511
      20095421
      1
      20
      1
      3338-B
      20095421-ser001-k020-100hz.asc
      Kursstyrning Rolldecay 14.0 kn
      N:\Gamla_Projekt\ascii_files\20095421-ser001-k...
      None
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\gamla_p...
      None
      2010-09-02
      roll decay
      MDL
      None
      None
      Kursstyrning
      20095421_STX_Polar_Supply
      \\sspa.local\gbg\ProjektArkiv\2009\20095421_ST...
      -2.327
      9.00
      2.2
      0.830
      6.74
      6.74
      22.00
      8.3
      30.35
      NaN
      0.6325
      11260.0
      0.9890
      5.1
      23.889
      121.40
      22.00
      NaN
      0.000
      0.0
      NaN
      14.0
      NaN
      NaN
      0.990
      NaN
      None
      None
      NaN
      1.0
      None
      0.00
      0.00
      NaN
      4.3
      NaN
      NaN
      NaN
      NaN
      12.45
      NaN
      NaN
      2.0
      1.0
      NaN
      None
      NaN
      NaN
      133.4
      8.0

```python
row = df_ikeda.iloc[0]

db = database.get_db()
run = db.session.query(Run).get(int(row.id))
run = database.load_run(run, save_as_example=False, prefer_hdf5=True)
```

    c:\dev\evaluation\signal_lab\mdl_to_evaluation.py:106: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
      df_.units = units

```python
lowpass_filter = LowpassFilterDerivatorTransformer(cutoff=2, minimum_score=0.99)
scaler = ScaleFactorTransformer(scale_factor=None)  # dummy value None for now
cutter = CutTransformer(phi_max=np.deg2rad(9), phi_min=np.deg2rad(0.25), phi1d_start_tolerance=0.015)
offset_transformer = OffsetTransformer()
```

```python
steps = [
    ('filter',lowpass_filter),
    ('cutter', cutter),
#    ('offset_transformer',offset_transformer),
]

df = run.df.copy()

preprosessor = Pipeline(steps=steps)
preprosessor.fit(X=df[['phi']])
X = preprosessor.transform(df[['phi']])
```

```python
data = row.copy()
scale_factor=run.model.scale_factor
rho=1000
g=9.81
data['rho']=rho
data['g']=g

data['lpp']/=scale_factor
data['TA']/=scale_factor
data['TF']/=scale_factor
data['beam']/=scale_factor
data['BKL']/=scale_factor
data['BKB']/=scale_factor
data['kg']/=scale_factor
data['Volume']/=scale_factor**3
data['gm']/=scale_factor
data['V']=data['ship_speed']*1.852/3.6/np.sqrt(scale_factor)  #[m/s]
data['KXX']/=scale_factor

estimator_ikeda_quadratic_db = IkedaQuadraticEstimator.load(data=data, X=X)
```

```python
estimator_ikeda_quadratic = IkedaQuadraticEstimator(**data,
                                        verify_input=False, limit_inputs=False)
estimator_ikeda_quadratic.fit(X=X)
```

```python
fig,ax=plt.subplots()
estimator_ikeda_quadratic_db.plot_fit(ax=ax)
estimator_ikeda_quadratic.plot_fit(ax=ax, model_test=False)

fig,ax=plt.subplots()
estimator_ikeda_quadratic_db.plot_damping(ax=ax)
estimator_ikeda_quadratic.plot_damping(ax=ax, include_model_test=False)
```

## Calculate for each $\phi_a$

```python
estimator_ikeda_quadratic_db.plot_omega0()
```

```python
estimator_ikeda_quadratic.calculate_amplitudes_and_damping()
X_pred = estimator_ikeda_quadratic.predict(X=X)
X_amplitudes = rolldecayestimators.measure.calculate_amplitudes_and_damping(X=X_pred)
```

```python
X_amplitudes.head()
```

      phi
      phi1d
      phi2d
      phi_a
      zeta_n
      B_n
      omega0

      10.292912
      0.205437
      5.045877e-16
      1.271821
      0.102718
      0.056811
      0.113622
      2.483525

      11.557885
      0.170887
      7.274997e-16
      1.057928
      0.085443
      0.053522
      0.107044
      2.483964

      12.822412
      0.143766
      1.916327e-16
      0.890036
      0.071883
      0.050915
      0.101829
      2.484656

      14.086680
      0.122084
      2.853078e-16
      0.755810
      0.061042
      0.048909
      0.097818
      2.485092

      15.350763
      0.104405
      2.222614e-16
      0.646357
      0.052203
      0.047206
      0.094412
      2.485439

```python
data
```

    model_number                                                       3541-A
    loading_condition_id                                                  107
    ship_speed                                                             13
    B_44_1                                                            15.7186
    B_F_1                                                             1.10144
    B_W_1                                                             4.21872
    B_E_1                                                            0.468786
    B_BK_1                                                            3.59288
    B_L_1                                                             6.33679
    B_44_2                                                            21.2863
    B_F_2                                                             1.10144
    B_W_2                                                             4.21872
    B_E_2                                                            0.937572
    B_BK_2                                                            8.69176
    B_L_2                                                             6.33679
    B_3                                                                  None
    omega0                                                            2.48814
    omega0_fft                                                        2.48814
    zeta                                                            0.0266151
    d                                                                0.496991
    score                                                            0.968976
    phi_start                                                       -0.138416
    phi_stop                                                       0.00640469
    mean_damping                                                         None
    B_1                                                               10.1509
    B_2                                                                38.091
    B_e                                                               21.2863
    id                                                                  14848
    project_number                                                   20136522
    series_number                                                           1
    run_number                                                             32
    test_number                                                             1
    ship_name                                                          3541-A
    ascii_name                                 20136522-ser001-k032-100hz.asc
    comment                                           Kursstyrning Roll decay
    file_path_ascii         N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
    file_path_ascii_temp                                                 None
    file_path_log           \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
    file_path_hdf5                                                       None
    date                                                           2014-12-29
    test_type                                                      roll decay
    facility                                                              MDL
    angle1                                                               None
    angle2                                                               None
    Körfallstyp                                                  Kursstyrning
    name                             20136522_Hudong-Zhonghua 170k class LNGC
    project_path              S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
    lcg                                                                 -4.19
    kg                                                               0.334182
    gm                                                              0.0690909
    CW                                                                  0.847
    TF                                                               0.209091
    TA                                                               0.209091
    BWL                                                                 46.97
    KXX                                                              0.281818
    KZZ                                                                 65.32
    BTT1                                                                 3.36
    CP                                                                 0.7669
    Volume                                                           0.700057
    A0                                                                 0.9899
    RH                                                                    8.5
    scale_factor                                                           55
    lpp                                                               5.16364
    beam                                                                0.854
    ABULB                                                                 5.1
    BKX                                                                -7.665
    TWIN                                                                    1
    DCLR                                                                  8.4
    VDES                                                                 19.8
    RHBL                                                                  9.2
    ASKEG                                                                 130
    PD                                                                  0.991
    ARH                                                                     0
    CFP                                                                  None
    AIX                                                                  None
    PDTDES                                                              20000
    RTYPE                                                                   1
    SFP                                                                  None
    BKL                                                               1.67509
    BKB                                                            0.00781818
    PROT                                                                    3
    D                                                                     8.3
    LSKEG                                                                  28
    RR                                                                   2.32
    XSKEG                                                                -123
    NDES                                                                 67.8
    AR                                                                   49.4
    BR                                                                      0
    BRA                                                                     0
    IRUD                                                                    2
    PTYPE                                                                   1
    XRUD                                                                 -142
    AI                                                                   None
    HSKEG                                                                 7.7
    RSKEG                                                                  58
    LOA                                                                   290
    ship_type_id                                                            4
    rho                                                                  1000
    g                                                                    9.81
    V                                                                 0.90178
    Name: 14848, dtype: object

```python
data['draught']=(data['TA']+data['TF'])/2
data['volume']=data['Volume']
```

```python
data2 = data.copy()
```

```python
N = len(X_amplitudes)
data_ = np.tile(data2.values,(N,1))
inputs_raw = pd.DataFrame(data=data_, columns=data2.index)
```

```python
for key,value in inputs_raw.items():
    try:
        inputs_raw[key] =inputs_raw[key].astype(float)
    except:
        continue

```

```python
inputs_raw.head()
```

      model_number
      loading_condition_id
      ship_speed
      B_44_1
      B_F_1
      B_W_1
      B_E_1
      B_BK_1
      B_L_1
      B_44_2
      B_F_2
      B_W_2
      B_E_2
      B_BK_2
      B_L_2
      B_3
      omega0
      omega0_fft
      zeta
      d
      score
      phi_start
      phi_stop
      mean_damping
      B_1
      B_2
      B_e
      id
      project_number
      series_number
      run_number
      test_number
      ship_name
      ascii_name
      comment
      file_path_ascii
      file_path_ascii_temp
      file_path_log
      file_path_hdf5
      date
      test_type
      facility
      angle1
      angle2
      Körfallstyp
      name
      project_path
      lcg
      kg
      gm
      CW
      TF
      TA
      BWL
      KXX
      KZZ
      BTT1
      CP
      Volume
      A0
      RH
      scale_factor
      lpp
      beam
      ABULB
      BKX
      TWIN
      DCLR
      VDES
      RHBL
      ASKEG
      PD
      ARH
      CFP
      AIX
      PDTDES
      RTYPE
      SFP
      BKL
      BKB
      PROT
      D
      LSKEG
      RR
      XSKEG
      NDES
      AR
      BR
      BRA
      IRUD
      PTYPE
      XRUD
      AI
      HSKEG
      RSKEG
      LOA
      ship_type_id
      rho
      g
      V
      draught
      volume

      0
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057

      1
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057

      2
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057

      3
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057

      4
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057

```python
inputs = inputs_raw.copy()
inputs['w']=inputs['omega0'].astype(float)
inputs['fi_a']=np.array(X_amplitudes['phi_a'])
inputs['g']=9.81

```

```python
inputs.head()
```

      model_number
      loading_condition_id
      ship_speed
      B_44_1
      B_F_1
      B_W_1
      B_E_1
      B_BK_1
      B_L_1
      B_44_2
      B_F_2
      B_W_2
      B_E_2
      B_BK_2
      B_L_2
      B_3
      omega0
      omega0_fft
      zeta
      d
      score
      phi_start
      phi_stop
      mean_damping
      B_1
      B_2
      B_e
      id
      project_number
      series_number
      run_number
      test_number
      ship_name
      ascii_name
      comment
      file_path_ascii
      file_path_ascii_temp
      file_path_log
      file_path_hdf5
      date
      test_type
      facility
      angle1
      angle2
      Körfallstyp
      name
      project_path
      lcg
      kg
      gm
      CW
      TF
      TA
      BWL
      KXX
      KZZ
      BTT1
      CP
      Volume
      A0
      RH
      scale_factor
      lpp
      beam
      ABULB
      BKX
      TWIN
      DCLR
      VDES
      RHBL
      ASKEG
      PD
      ARH
      CFP
      AIX
      PDTDES
      RTYPE
      SFP
      BKL
      BKB
      PROT
      D
      LSKEG
      RR
      XSKEG
      NDES
      AR
      BR
      BRA
      IRUD
      PTYPE
      XRUD
      AI
      HSKEG
      RSKEG
      LOA
      ship_type_id
      rho
      g
      V
      draught
      volume
      w
      fi_a

      0
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057
      2.488141
      0.102718

      1
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057
      2.488141
      0.085443

      2
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057
      2.488141
      0.071883

      3
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057
      2.488141
      0.061042

      4
      3541-A
      107.0
      13.0
      15.718615
      1.101438
      4.218719
      0.468786
      3.592881
      6.336792
      21.286281
      1.101438
      4.218719
      0.937572
      8.69176
      6.336792
      NaN
      2.488141
      2.488141
      0.026615
      0.496991
      0.968976
      -0.138416
      0.006405
      NaN
      10.150949
      38.090976
      21.286281
      14848.0
      20136522.0
      1.0
      32.0
      1.0
      3541-A
      20136522-ser001-k032-100hz.asc
      Kursstyrning Roll decay
      N:\Gamla_Projekt\ascii_files\20136522-ser001-k...
      NaN
      \\sspa.local\gbg\MDLdata\Gamla_Projekt\Gamla_P...
      NaN
      2014-12-29
      roll decay
      MDL
      NaN
      NaN
      Kursstyrning
      20136522_Hudong-Zhonghua 170k class LNGC
      S:\2013\20136522-Hudong-Zhonghua-170k-class-LNG
      -4.19
      0.334182
      0.069091
      0.847
      0.209091
      0.209091
      46.97
      0.281818
      65.32
      3.36
      0.7669
      0.700057
      0.9899
      8.5
      55.0
      5.163636
      0.854
      5.1
      -7.665
      1.0
      8.4
      19.8
      9.2
      130.0
      0.991
      0.0
      NaN
      NaN
      20000.0
      1.0
      NaN
      1.675091
      0.007818
      3.0
      8.3
      28.0
      2.32
      -123.0
      67.8
      49.4
      0.0
      0.0
      2.0
      1.0
      -142.0
      NaN
      7.7
      58.0
      290.0
      4.0
      1000.0
      9.81
      0.90178
      0.209091
      0.700057
      2.488141
      0.052203

```python
def calculate(inputs, IkedaClass=SimplifiedIkeda):
    si = IkedaClass(**inputs)
    output = pd.DataFrame(index=inputs.index)
    output['B_44_hat'] = si.calculate_B44()
    output['B_W0'] =si.calculate_B_W0()
    output['B_W'] =si.calculate_B_W()
    output['B_F'] =si.calculate_B_F()
    output['B_E'] =si.calculate_B_E()
    output['B_BK'] =si.calculate_B_BK()
    output['B_L'] =si.calculate_B_L()
    output['Bw_div_Bw0'] =si.calculate_Bw_div_Bw0()

    return output
```

```python
output = calculate(inputs=inputs, IkedaClass=SimplifiedIkeda)
```

```python
output.head()
```

      B_44_hat
      B_W0
      B_W
      B_F
      B_E
      B_BK
      B_L
      Bw_div_Bw0

      0
      0.010222
      0.000248
      0.001724
      0.000452
      0.001160
      0.004297
      0.002589
      6.959384

      1
      0.009467
      0.000248
      0.001724
      0.000452
      0.000965
      0.003737
      0.002589
      6.959384

      2
      0.008896
      0.000248
      0.001724
      0.000452
      0.000812
      0.003319
      0.002589
      6.959384

      3
      0.008452
      0.000248
      0.001724
      0.000452
      0.000689
      0.002997
      0.002589
      6.959384

      4
      0.008099
      0.000248
      0.001724
      0.000452
      0.000589
      0.002744
      0.002589
      6.959384

```python
output['B_44'] = lambdas.B_from_hat_lambda(B_44_hat=output['B_44_hat'], Disp=inputs['volume'],
                          beam=inputs['beam'],
                          g=inputs['g'], rho=inputs['rho'])
```

```python
fig,ax=plt.subplots()
x = inputs['fi_a']
y = output['B_44']
ax.plot(x,y)
ax.set_xlabel('$\phi_a$ [rad]')
ax.set_ylabel('$B_{44}$')

```

```python
np.array(output['B_44'])/np.array(X_amplitudes['B_n'])
```

    array([220.15619203, 216.43961076, 213.78553395, 211.45020088,
           209.92964181, 209.14145408, 208.29162558, 207.59734788,
           207.36961231, 206.97347589, 206.39749995, 206.38231801,
           206.68514401, 206.46781746, 205.99165004, 206.03015756,
           206.28829518, 206.13001126, 205.87092656, 206.10920137,
           206.4537088 , 206.27099344, 205.86343667, 205.67919139,
           205.86164755, 206.26391714, 206.52329792,          nan])

```python
equations.C_equation_linear
```

$\displaystyle C = GM g m$

```python
equations.A44
```

$\displaystyle \frac{GM g m}{\omega_{0}^{2}}$

```python
mass = inputs.Volume*inputs.rho
A44_tot = lambdas.A44_lambda(GM=inputs.gm, g=inputs.g, m=mass, omega0=inputs.w)
A44_mass = mass*data.KXX**2
KXX_tot = np.sqrt(A44_tot/mass)
(KXX_tot - data.KXX)/data.KXX
```

    0     0.174089
    1     0.174089
    2     0.174089
    3     0.174089
    4     0.174089
    5     0.174089
    6     0.174089
    7     0.174089
    8     0.174089
    9     0.174089
    10    0.174089
    11    0.174089
    12    0.174089
    13    0.174089
    14    0.174089
    15    0.174089
    16    0.174089
    17    0.174089
    18    0.174089
    19    0.174089
    20    0.174089
    21    0.174089
    22    0.174089
    23    0.174089
    24    0.174089
    25    0.174089
    26    0.174089
    27    0.174089
    dtype: float64

```python
results = estimator_ikeda_quadratic.result_for_database()
results
```

    {'zeta': 0.036588024189946715,
     'd': 0.5442643525426031,
     'omega0': 2.48814138164312,
     'score': 0.9042684962009301,
     'phi_start': -0.1381428102953512,
     'phi_stop': 0.0079412480965742,
     'omega0_fft': 2.48814138164312,
     'B_44_1': 20.039803465353693,
     'B_44_2': 26.125010601697632,
     'B_F_1': 1.1014377904648913,
     'B_F_2': 1.1014377904648913,
     'B_W_1': 4.218718560323839,
     'B_W_2': 4.218718560323839,
     'B_E_1': 0.4678599368133092,
     'B_E_2': 0.9357198736266183,
     'B_BK_1': 7.914995241710201,
     'B_BK_2': 13.532342441240832,
     'B_L_1': 6.336791936041452,
     'B_L_2': 6.336791936041452,
     'B_1': 13.954596329009753,
     'B_2': 41.71412718172688,
     'B_e': 26.125010601697632}

```python
equations.extinction_equation
```

$\displaystyle \phi_{a} = \phi_{0}{\left(t \right)} e^{- \omega_{0} t \zeta}$

```python
fig,ax=plt.subplots()
X_amplitudes.plot(y='phi_a', ax=ax)
```

```python
t = X_amplitudes.index
y = np.log(X_amplitudes['phi_a'])

fig,ax=plt.subplots()
ax.plot(t,y)
```

```python
sp.Eq(symbols.zeta,sp.solve(equations.extinction_equation,symbols.zeta)[0])
```

$\displaystyle \zeta = \frac{\log{\left(\frac{\phi_{0}{\left(t \right)}}{\phi_{a}} \right)}}{\omega_{0} t}$

```python
zeta_lambda = lambdify(sp.solve(equations.extinction_equation,symbols.zeta)[0])
phi_0 = X_amplitudes['phi_a'].iloc[0]
t = X_amplitudes.index - X_amplitudes.index[0]
X_amplitudes['zeta2'] = zeta_lambda(omega0=X_amplitudes['omega0'],phi_0=phi_0, phi_a=X_amplitudes['phi_a'],
            t=t)
```

```python
omega0 = inputs.iloc[0]['omega0']

for i in range(len(X_amplitudes)-1):

    row1 = X_amplitudes.iloc[i]
    row2 = X_amplitudes.iloc[i+1]
    t_ = row2.name - row1.name
    B_n = zeta_lambda(omega0=omega0,phi_0=row1['phi_a'], phi_a=row2['phi_a'],
            t=t_)

    X_amplitudes.loc[row2.name,'B_n2'] = B_n
```

```python
X_amplitudes['B_n3'] = np.array(output['B_44']/(A44_tot))
```

```python
fig,ax=plt.subplots()
X_amplitudes.plot(x='phi_a',y='zeta2',ax=ax)
X_amplitudes.plot(x='phi_a',y='B_n', ax=ax)

X_amplitudes['B_n/2'] = X_amplitudes['B_n']/2
X_amplitudes.plot(x='phi_a',y='B_n/2', ax=ax)

X_amplitudes.plot(x='phi_a',y='B_n2', ax=ax)
#X_amplitudes.plot(x='phi_a',y='B_n3', ax=ax)
```


```python
%matplotlib inline
```

Write your own GNN module
=========================

Sometimes, your model goes beyond simply stacking existing GNN modules.
For example, you would like to invent a new way of aggregating neighbor
information by considering node importance or edge weights.

By the end of this tutorial you will be able to

-  Understand DGL’s message passing APIs.
-  Implement GraphSAGE convolution module by your own.

This tutorial assumes that you already know :doc:`the basics of training a
GNN for node classification <1_introduction>`.

(Time estimate: 10 minutes)

```python
import dgl
import torch
import torch.nn as nn
import torch.nn.functional as F
```

    Using backend: pytorch

Message passing and GNNs
------------------------

DGL follows the *message passing paradigm* inspired by the Message
Passing Neural Network proposed by `Gilmer et
al. <https://arxiv.org/abs/1704.01212>`__ Essentially, they found many
GNN models can fit into the following framework:

\begin{align}m_{u\to v}^{(l)} = M^{(l)}\left(h_v^{(l-1)}, h_u^{(l-1)}, e_{u\to v}^{(l-1)}\right)\end{align}

\begin{align}m_{v}^{(l)} = \sum_{u\in\mathcal{N}(v)}m_{u\to v}^{(l)}\end{align}

\begin{align}h_v^{(l)} = U^{(l)}\left(h_v^{(l-1)}, m_v^{(l)}\right)\end{align}

where DGL calls $M^{(l)}$ the *message function*, $\sum$ the
*reduce function* and $U^{(l)}$ the *update function*. Note that
$\sum$ here can represent any function and is not necessarily a
summation.

For example, the `GraphSAGE convolution (Hamilton et al.,
2017) <https://cs.stanford.edu/people/jure/pubs/graphsage-nips17.pdf>`__
takes the following mathematical form:

\begin{align}h_{\mathcal{N}(v)}^k\leftarrow \text{Average}\{h_u^{k-1},\forall u\in\mathcal{N}(v)\}\end{align}

\begin{align}h_v^k\leftarrow \text{ReLU}\left(W^k\cdot \text{CONCAT}(h_v^{k-1}, h_{\mathcal{N}(v)}^k) \right)\end{align}

You can see that message passing is directional: the message sent from
one node $u$ to other node $v$ is not necessarily the same
as the other message sent from node $v$ to node $u$ in the
opposite direction.

Although DGL has builtin support of GraphSAGE via
:class:`dgl.nn.SAGEConv <dgl.nn.pytorch.SAGEConv>`,
here is how you can implement GraphSAGE convolution in DGL by your own.

```python
import dgl.function as fn

class SAGEConv(nn.Module):
    """Graph convolution module used by the GraphSAGE model.

    Parameters
    ----------
    in_feat : int
        Input feature size.
    out_feat : int
        Output feature size.
    """
    def __init__(self, in_feat, out_feat):
        super(SAGEConv, self).__init__()
        # A linear submodule for projecting the input and neighbor feature to the output.
        self.linear = nn.Linear(in_feat * 2, out_feat)

    def forward(self, g, h):
        """Forward computation

        Parameters
        ----------
        g : Graph
            The input graph.
        h : Tensor
            The input node feature.
        """
        with g.local_scope():
            g.ndata['h'] = h
            # update_all is a message passing API.
            g.update_all(message_func=fn.copy_u('h', 'm'), reduce_func=fn.mean('m', 'h_N'))
            h_N = g.ndata['h_N']
            h_total = torch.cat([h, h_N], dim=1)
            return self.linear(h_total)
```

The central piece in this code is the
:func:`g.update_all <dgl.DGLGraph.update_all>`
function, which gathers and averages the neighbor features. There are
three concepts here:

* Message function ``fn.copy_u('h', 'm')`` that
  copies the node feature under name ``'h'`` as *messages* sent to
  neighbors.

* Reduce function ``fn.mean('m', 'h_N')`` that averages
  all the received messages under name ``'m'`` and saves the result as a
  new node feature ``'h_N'``.

* ``update_all`` tells DGL to trigger the
  message and reduce functions for all the nodes and edges.

Afterwards, you can stack your own GraphSAGE convolution layers to form
a multi-layer GraphSAGE network.

```python
class Model(nn.Module):
    def __init__(self, in_feats, h_feats, num_classes):
        super(Model, self).__init__()
        self.conv1 = SAGEConv(in_feats, h_feats)
        self.conv2 = SAGEConv(h_feats, num_classes)

    def forward(self, g, in_feat):
        h = self.conv1(g, in_feat)
        h = F.relu(h)
        h = self.conv2(g, h)
        return h
```

Training loop
~~~~~~~~~~~~~
The following code for data loading and training loop is directly copied
from the introduction tutorial.

```python
import dgl.data

dataset = dgl.data.CoraGraphDataset()
g = dataset[0]

def train(g, model):
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    all_logits = []
    best_val_acc = 0
    best_test_acc = 0

    features = g.ndata['feat']
    labels = g.ndata['label']
    train_mask = g.ndata['train_mask']
    val_mask = g.ndata['val_mask']
    test_mask = g.ndata['test_mask']
    for e in range(200):
        # Forward
        logits = model(g, features)

        # Compute prediction
        pred = logits.argmax(1)

        # Compute loss
        # Note that we should only compute the losses of the nodes in the training set,
        # i.e. with train_mask 1.
        loss = F.cross_entropy(logits[train_mask], labels[train_mask])

        # Compute accuracy on training/validation/test
        train_acc = (pred[train_mask] == labels[train_mask]).float().mean()
        val_acc = (pred[val_mask] == labels[val_mask]).float().mean()
        test_acc = (pred[test_mask] == labels[test_mask]).float().mean()

        # Save the best validation accuracy and the corresponding test accuracy.
        if best_val_acc < val_acc:
            best_val_acc = val_acc
            best_test_acc = test_acc

        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        all_logits.append(logits.detach())

        if e % 5 == 0:
            print('In epoch {}, loss: {:.3f}, val acc: {:.3f} (best {:.3f}), test acc: {:.3f} (best {:.3f})'.format(
                e, loss, val_acc, best_val_acc, test_acc, best_test_acc))

model = Model(g.ndata['feat'].shape[1], 16, dataset.num_classes)
train(g, model)
```

      NumNodes: 2708
      NumEdges: 10556
      NumFeats: 1433
      NumClasses: 7
      NumTrainingSamples: 140
      NumValidationSamples: 500
      NumTestSamples: 1000
    Done loading data from cached files.
    In epoch 0, loss: 1.953, val acc: 0.156 (best 0.156), test acc: 0.144 (best 0.144)
    In epoch 5, loss: 1.868, val acc: 0.304 (best 0.304), test acc: 0.298 (best 0.298)
    In epoch 10, loss: 1.706, val acc: 0.576 (best 0.576), test acc: 0.594 (best 0.594)
    In epoch 15, loss: 1.463, val acc: 0.640 (best 0.640), test acc: 0.644 (best 0.644)
    In epoch 20, loss: 1.151, val acc: 0.664 (best 0.664), test acc: 0.682 (best 0.682)
    In epoch 25, loss: 0.813, val acc: 0.700 (best 0.700), test acc: 0.718 (best 0.718)
    In epoch 30, loss: 0.514, val acc: 0.730 (best 0.730), test acc: 0.738 (best 0.738)
    In epoch 35, loss: 0.297, val acc: 0.754 (best 0.754), test acc: 0.759 (best 0.759)
    In epoch 40, loss: 0.164, val acc: 0.760 (best 0.762), test acc: 0.758 (best 0.755)
    In epoch 45, loss: 0.092, val acc: 0.758 (best 0.762), test acc: 0.754 (best 0.755)
    In epoch 50, loss: 0.055, val acc: 0.762 (best 0.762), test acc: 0.753 (best 0.755)
    In epoch 55, loss: 0.035, val acc: 0.760 (best 0.762), test acc: 0.756 (best 0.755)
    In epoch 60, loss: 0.024, val acc: 0.760 (best 0.762), test acc: 0.755 (best 0.755)
    In epoch 65, loss: 0.018, val acc: 0.758 (best 0.762), test acc: 0.754 (best 0.755)
    In epoch 70, loss: 0.014, val acc: 0.758 (best 0.762), test acc: 0.752 (best 0.755)
    In epoch 75, loss: 0.012, val acc: 0.756 (best 0.762), test acc: 0.750 (best 0.755)
    In epoch 80, loss: 0.010, val acc: 0.760 (best 0.762), test acc: 0.754 (best 0.755)
    In epoch 85, loss: 0.009, val acc: 0.760 (best 0.762), test acc: 0.753 (best 0.755)
    In epoch 90, loss: 0.008, val acc: 0.760 (best 0.762), test acc: 0.754 (best 0.755)
    In epoch 95, loss: 0.007, val acc: 0.760 (best 0.762), test acc: 0.754 (best 0.755)
    In epoch 100, loss: 0.007, val acc: 0.764 (best 0.764), test acc: 0.755 (best 0.755)
    In epoch 105, loss: 0.006, val acc: 0.764 (best 0.764), test acc: 0.754 (best 0.755)
    In epoch 110, loss: 0.006, val acc: 0.764 (best 0.764), test acc: 0.755 (best 0.755)
    In epoch 115, loss: 0.005, val acc: 0.764 (best 0.764), test acc: 0.755 (best 0.755)
    In epoch 120, loss: 0.005, val acc: 0.764 (best 0.766), test acc: 0.755 (best 0.755)
    In epoch 125, loss: 0.005, val acc: 0.764 (best 0.766), test acc: 0.757 (best 0.755)
    In epoch 130, loss: 0.004, val acc: 0.764 (best 0.766), test acc: 0.757 (best 0.755)
    In epoch 135, loss: 0.004, val acc: 0.764 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 140, loss: 0.004, val acc: 0.764 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 145, loss: 0.004, val acc: 0.764 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 150, loss: 0.004, val acc: 0.762 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 155, loss: 0.003, val acc: 0.762 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 160, loss: 0.003, val acc: 0.764 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 165, loss: 0.003, val acc: 0.764 (best 0.766), test acc: 0.758 (best 0.755)
    In epoch 170, loss: 0.003, val acc: 0.766 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 175, loss: 0.003, val acc: 0.766 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 180, loss: 0.003, val acc: 0.766 (best 0.766), test acc: 0.759 (best 0.755)
    In epoch 185, loss: 0.003, val acc: 0.766 (best 0.766), test acc: 0.758 (best 0.755)
    In epoch 190, loss: 0.002, val acc: 0.766 (best 0.766), test acc: 0.758 (best 0.755)
    In epoch 195, loss: 0.002, val acc: 0.766 (best 0.766), test acc: 0.758 (best 0.755)

More customization
------------------

In DGL, we provide many built-in message and reduce functions under the
``dgl.function`` package. You can find more details in `the API
doc <apifunction>`.

These APIs allow one to quickly implement new graph convolution modules.
For example, the following implements a new ``SAGEConv`` that aggregates
neighbor representations using a weighted average. Note that ``edata``
member can hold edge features which can also take part in message
passing.

```python
class WeightedSAGEConv(nn.Module):
    """Graph convolution module used by the GraphSAGE model with edge weights.

    Parameters
    ----------
    in_feat : int
        Input feature size.
    out_feat : int
        Output feature size.
    """
    def __init__(self, in_feat, out_feat):
        super(WeightedSAGEConv, self).__init__()
        # A linear submodule for projecting the input and neighbor feature to the output.
        self.linear = nn.Linear(in_feat * 2, out_feat)

    def forward(self, g, h, w):
        """Forward computation

        Parameters
        ----------
        g : Graph
            The input graph.
        h : Tensor
            The input node feature.
        w : Tensor
            The edge weight.
        """
        with g.local_scope():
            g.ndata['h'] = h
            g.edata['w'] = w
            g.update_all(message_func=fn.u_mul_e('h', 'w', 'm'), reduce_func=fn.mean('m', 'h_N'))
            h_N = g.ndata['h_N']
            h_total = torch.cat([h, h_N], dim=1)
            return self.linear(h_total)
```

Because the graph in this dataset does not have edge weights, we
manually assign all edge weights to one in the ``forward()`` function of
the model. You can replace it with your own edge weights.

```python
class Model(nn.Module):
    def __init__(self, in_feats, h_feats, num_classes):
        super(Model, self).__init__()
        self.conv1 = WeightedSAGEConv(in_feats, h_feats)
        self.conv2 = WeightedSAGEConv(h_feats, num_classes)

    def forward(self, g, in_feat):
        h = self.conv1(g, in_feat, torch.ones(g.num_edges()).to(g.device))
        h = F.relu(h)
        h = self.conv2(g, h, torch.ones(g.num_edges()).to(g.device))
        return h

model = Model(g.ndata['feat'].shape[1], 16, dataset.num_classes)
train(g, model)
```

Even more customization by user-defined function
------------------------------------------------

DGL allows user-defined message and reduce function for the maximal
expressiveness. Here is a user-defined message function that is
equivalent to ``fn.u_mul_e('h', 'w', 'm')``.

```python
def u_mul_e_udf(edges):
    return {'m' : edges.src['h'] * edges.data['w']}
```

``edges`` has three members: ``src``, ``data`` and ``dst``, representing
the source node feature, edge feature, and destination node feature for
all edges.

You can also write your own reduce function. For example, the following
is equivalent to the builtin ``fn.sum('m', 'h')`` function that sums up
the incoming messages:

```python
def sum_udf(nodes):
    return {'h': nodes.mailbox['m'].sum(1)}
```

In short, DGL will group the nodes by their in-degrees, and for each
group DGL stacks the incoming messages along the second dimension. You
can then perform a reduction along the second dimension to aggregate
messages.

For more details on customizing message and reduce function with
user-defined function, please refer to the `API
reference <apiudf>`.

Best practice of writing custom GNN modules
-------------------------------------------

DGL recommends the following practice ranked by preference:

-  Use ``dgl.nn`` modules.
-  Use ``dgl.nn.functional`` functions which contain lower-level complex
   operations such as computing a softmax for each node over incoming
   edges.
-  Use ``update_all`` with builtin message and reduce functions.
-  Use user-defined message or reduce functions.

What’s next?
------------

-  `Writing Efficient Message Passing
   Code <guide-message-passing-efficient>`.

```python
# Thumbnail Courtesy: Representation Learning on Networks, Jure Leskovec, WWW 2018
# sphinx_gallery_thumbnail_path = '_static/blitz_3_message_passing.png'
```


# training an unsupervised VAE with APOGEE DR14 spectra
this notebooke takes you through the building and training of a fairly deep VAE. I have not actually done too much work with DR14, so it may pose some potential difficulties, but this should be a good start. The training is suited to be put into a python script and run from the command line.

If you're inclined, you may want to experiment with the model architecture, but I'm pretty sure this will work.

```python
import numpy as np
import time
import h5py
import keras
import matplotlib.pyplot as plt

import sys

from keras.layers import (Input, Dense, Lambda, Flatten, Reshape, BatchNormalization, Activation,
                          Dropout, Conv1D, UpSampling1D, MaxPooling1D, ZeroPadding1D, LeakyReLU)
from keras.engine.topology import Layer
from keras.optimizers import Adam
from keras.models import Model
from keras import backend as K

plt.switch_backend('agg')
```

    Using TensorFlow backend.

### load data
this is file dependent

this particular function expects the aspcap dr14 h5 file that can be downloaded from vos:starnet/public

```python
# Define edges of detectors (for APOGEE)
blue_chip_begin = 322
blue_chip_end = 3242
green_chip_begin = 3648
green_chip_end = 6048
red_chip_begin = 6412
red_chip_end = 8306

# function for loading data
def load_train_data_weighted(data_file,indices=None):

    # grab all
    if indices is None:
        with h5py.File(data_file,"r") as F:
            ap_spectra = F['spectrum'][:]
            ap_err_spectra = F['error_spectrum'][:]
    # grab a batch
    else:
        with h5py.File(data_file, "r") as F:
            indices_bool = np.ones((len(F['spectrum']),),dtype=bool)
            indices_bool[:] = False
            indices_bool[indices] = True

            ap_spectra = F['spectrum'][indices_bool,:]
            ap_err_spectra = F['error_spectrum'][indices_bool,:]

    # combine chips
    ap_spectra = np.hstack((ap_spectra[:,blue_chip_begin:blue_chip_end],
                                     ap_spectra[:,green_chip_begin:green_chip_end],
                                     ap_spectra[:,red_chip_begin:red_chip_end]))
    # set nan values to zero
    ap_spectra[np.isnan(ap_spectra)]=0.

    ap_err_spectra = np.hstack((ap_err_spectra[:,blue_chip_begin:blue_chip_end],
                                     ap_err_spectra[:,green_chip_begin:green_chip_end],
                                     ap_err_spectra[:,red_chip_begin:red_chip_end]))

    return ap_spectra,ap_err_spectra
```

```python
# function for reshaping spectra into appropriate format for CNN
def cnn_reshape(spectra):
    return spectra.reshape(spectra.shape[0],spectra.shape[1],1)
```

set some model hyper-parameters

```python
img_cols, img_chns = 7214, 1
num_fluxes=7214
input_shape=(num_fluxes,1)

# z_dims is the dimension of the latent space
z_dims = 64
batch_size = 64
epsilon_std = 1.0
learning_rate = 0.001
decay = 0.0
padding=u'same'
kernel_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.01)
bias_init = keras.initializers.Zeros()
```

## Zero-Augmentation

In an effort to evenly distribute the weighting of the VAE, throughout training, a *zero-augmentation* technique was applied to the training spectra samples - both synthetic and observed. The zero-augmentation is implemented as the first layer in the encoder where a zero-augmentation mask is sent as an input along with the input spectrum and the two are multiplied together. The zero-augmentation mask is the same size as the input spectrum vector and is composed of ones and zeros. For the APOGEE wave-grid, the spectral region is divided into seven *chunks* and for each input spectrum a random 0-3 of these chunks are assigned to be zeros while the remainder of the zero-augmentation mask is made up of ones. This means for a given spectrum, the input for training may be 4/7ths, 5/7ths, 6/7ths, or the entire spectrum. This augmentation is done randomly throughout training, meaning that each spectrum will be randomly assigned a different zero-augmentation mask at every iteration.

```python
# zero-augmentation layer (a trick I use to input chunks of zeros into the input spectra)
class ZeroAugmentLayer(Layer):
    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(ZeroAugmentLayer, self).__init__(**kwargs)

    def zero_agument(self, x_real, zero_mask):
        return x_real*zero_mask

    def call(self, inputs):
        x_real = inputs[0]
        zero_mask = inputs[1]
        x_augmented = self.zero_agument(x_real, zero_mask)

        return x_augmented

# a function for creating the zero-masks used during training
def create_zero_mask(spectra,min_chunks,max_chunks,chunk_size,dataset=None,ones_padded=False):
    if dataset is None:
        zero_mask = np.ones_like(spectra)
    elif dataset=='apogee':
        zero_mask = np.ones((spectra.shape[0],7214))
    elif dataset=='segue':
        zero_mask = np.ones((spectra.shape[0],3688))

    num_spec = zero_mask.shape[0]
    len_spec = zero_mask.shape[1]
    num_bins = len_spec/chunk_size
    remainder = len_spec%chunk_size
    spec_sizes = np.array([chunk_size for i in range(num_bins)])
    spec_sizes[-1]=spec_sizes[-1]+remainder

    num_bins_removed = np.random.randint(min_chunks,max_chunks+1,size=(num_spec,))
    for i, mask in enumerate(zero_mask):
        bin_indx_removed = np.random.choice(num_bins, num_bins_removed[i], replace=False)
        for indx in bin_indx_removed:
            if indx==0:
                mask[indx*spec_sizes[indx]:(indx+1)*spec_sizes[indx]]=0.
            else:
                mask[indx*spec_sizes[indx-1]:indx*spec_sizes[indx-1]+spec_sizes[indx]]=0.

    return zero_mask
```

# build encoder
takes spectra (x) and zero-augmentation mask as inputs and outputs latent distribution (z_mean and z_log_var)

```python
def build_encoder(input_1,input_2):

    # zero-augment input spectrum
    x = ZeroAugmentLayer()([input_1,input_2])

    # first conv block
    x = Conv1D(filters=16, kernel_size=8, strides=1, kernel_initializer=kernel_init,
                   bias_initializer=bias_init, padding=padding)(x)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)
    x = Dropout(0.2)(x)

    # second conv bloack
    x = Conv1D(filters=16, kernel_size=8, strides=1, kernel_initializer=kernel_init,
                   bias_initializer=bias_init, padding=padding)(x)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)

    # maxpooling layer and flatten
    x = MaxPooling1D(pool_size=4, strides=4, padding='valid')(x)
    x = Flatten()(x)
    x = Dropout(0.2)(x)

    # intermediate dense block
    x = Dense(256)(x)
    x = LeakyReLU(0.3)(x)
    x = BatchNormalization()(x)
    x = Dropout(0.3)(x)

    # latent distribution output
    z_mean = Dense(z_dims)(x)
    z_log_var = Dense(z_dims)(x)

    return Model([input_1,input_2],[z_mean,z_log_var])

# function for obtaining a latent sample given a distribution
def sampling(args, latent_dim=z_dims, epsilon_std=epsilon_std):
    z_mean, z_log_var = args

    epsilon = K.random_normal(shape=(z_dims,),
                              mean=0., stddev=epsilon_std)

    return z_mean + K.exp(z_log_var) * epsilon
```

# build decoder
takes  z (latent variables) as an input and outputs a stellar spectrum

```python
def build_decoder(inputs):

    # input fully-connected block
    x = Dense(256)(inputs)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)
    x = Dropout(0.2)(x)

    # intermediate fully-connected block
    w = input_shape[0] // (2 ** 3)
    x = Dense(w * 16)(x)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)
    x = Dropout(0.2)(x)

    # reshape for convolutional blocks
    x = Reshape((w, 16))(x)

    # first deconv block
    x = UpSampling1D(size=4)(x)
    x = Conv1D(kernel_initializer=kernel_init,bias_initializer=bias_init,padding="same",
                   filters=16,kernel_size=8)(x)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)
    x = Dropout(0.1)(x)

    # zero-padding to get x in the right dimension to create the spectra
    x = ZeroPadding1D(padding=(2,1))(x)

    # second deconv block
    x = UpSampling1D(size=2)(x)
    x = Conv1D(kernel_initializer=kernel_init,bias_initializer=bias_init,padding="same",
                   filters=16,kernel_size=8)(x)
    x = LeakyReLU(0.1)(x)
    x = BatchNormalization()(x)

    # output conv layer
    x = Conv1D(kernel_initializer=kernel_init,bias_initializer=bias_init,padding="same",
                   filters=1,kernel_size=8,activation='linear')(x)

    return Model(inputs,x)
```

## build models

```python
# encoder and predictor input placeholders
input_spec = Input(shape=input_shape)
input_mask = Input(shape=input_shape)

# error spectra placeholder
input_err_spec = Input(shape=input_shape)

# decoder input placeholder
input_z = Input(shape=(z_dims,))

model_name='vae_test'
start_e = 0

# if you want to continue training from a certain epoch, you can uncomment the load models lines
# and comment out the build_encoder, build_decoder lines
'''
encoder  = keras.models.load_model('models/encoder_'+model_name+'_epoch_'+str(start_e)+'.h5',
                                    custom_objects={'ZeroAugmentLayer':ZeroAugmentLayer})
decoder = keras.models.load_model('models/decoder_'+model_name+'_epoch_'+str(start_e)+'.h5',
                                 custom_objects={'ZeroAugmentLayer':ZeroAugmentLayer})

'''
# encoder model
encoder = build_encoder(input_spec, input_mask)

# decoder layers
decoder = build_decoder(input_z)
#'''

encoder.summary()
decoder.summary()

# outputs for encoder
z_mean, z_log_var = encoder([input_spec, input_mask])

# sample from latent distribution given z_mean and z_log_var
z = Lambda(sampling, output_shape=(z_dims,))([z_mean, z_log_var])

# outputs for decoder
output_spec = decoder(z)

```

    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            (None, 7214, 1)      0
    __________________________________________________________________________________________________
    input_2 (InputLayer)            (None, 7214, 1)      0
    __________________________________________________________________________________________________
    zero_augment_layer_1 (ZeroAugme [(None, 7214, 1), (N 0           input_1[0][0]
                                                                     input_2[0][0]
    __________________________________________________________________________________________________
    conv1d_1 (Conv1D)               (None, 7214, 16)     144         zero_augment_layer_1[0][0]
    __________________________________________________________________________________________________
    leaky_re_lu_1 (LeakyReLU)       (None, 7214, 16)     0           conv1d_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_1 (BatchNor (None, 7214, 16)     64          leaky_re_lu_1[0][0]
    __________________________________________________________________________________________________
    dropout_1 (Dropout)             (None, 7214, 16)     0           batch_normalization_1[0][0]
    __________________________________________________________________________________________________
    conv1d_2 (Conv1D)               (None, 7214, 16)     2064        dropout_1[0][0]
    __________________________________________________________________________________________________
    leaky_re_lu_2 (LeakyReLU)       (None, 7214, 16)     0           conv1d_2[0][0]
    __________________________________________________________________________________________________
    batch_normalization_2 (BatchNor (None, 7214, 16)     64          leaky_re_lu_2[0][0]
    __________________________________________________________________________________________________
    max_pooling1d_1 (MaxPooling1D)  (None, 1803, 16)     0           batch_normalization_2[0][0]
    __________________________________________________________________________________________________
    flatten_1 (Flatten)             (None, 28848)        0           max_pooling1d_1[0][0]
    __________________________________________________________________________________________________
    dropout_2 (Dropout)             (None, 28848)        0           flatten_1[0][0]
    __________________________________________________________________________________________________
    dense_1 (Dense)                 (None, 256)          7385344     dropout_2[0][0]
    __________________________________________________________________________________________________
    leaky_re_lu_3 (LeakyReLU)       (None, 256)          0           dense_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_3 (BatchNor (None, 256)          1024        leaky_re_lu_3[0][0]
    __________________________________________________________________________________________________
    dropout_3 (Dropout)             (None, 256)          0           batch_normalization_3[0][0]
    __________________________________________________________________________________________________
    dense_2 (Dense)                 (None, 64)           16448       dropout_3[0][0]
    __________________________________________________________________________________________________
    dense_3 (Dense)                 (None, 64)           16448       dropout_3[0][0]
    ==================================================================================================
    Total params: 7,421,600
    Trainable params: 7,421,024
    Non-trainable params: 576
    __________________________________________________________________________________________________
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_4 (InputLayer)         (None, 64)                0
    _________________________________________________________________
    dense_4 (Dense)              (None, 256)               16640
    _________________________________________________________________
    leaky_re_lu_4 (LeakyReLU)    (None, 256)               0
    _________________________________________________________________
    batch_normalization_4 (Batch (None, 256)               1024
    _________________________________________________________________
    dropout_4 (Dropout)          (None, 256)               0
    _________________________________________________________________
    dense_5 (Dense)              (None, 14416)             3704912
    _________________________________________________________________
    leaky_re_lu_5 (LeakyReLU)    (None, 14416)             0
    _________________________________________________________________
    batch_normalization_5 (Batch (None, 14416)             57664
    _________________________________________________________________
    dropout_5 (Dropout)          (None, 14416)             0
    _________________________________________________________________
    reshape_1 (Reshape)          (None, 901, 16)           0
    _________________________________________________________________
    up_sampling1d_1 (UpSampling1 (None, 3604, 16)          0
    _________________________________________________________________
    conv1d_3 (Conv1D)            (None, 3604, 16)          2064
    _________________________________________________________________
    leaky_re_lu_6 (LeakyReLU)    (None, 3604, 16)          0
    _________________________________________________________________
    batch_normalization_6 (Batch (None, 3604, 16)          64
    _________________________________________________________________
    dropout_6 (Dropout)          (None, 3604, 16)          0
    _________________________________________________________________
    zero_padding1d_1 (ZeroPaddin (None, 3607, 16)          0
    _________________________________________________________________
    up_sampling1d_2 (UpSampling1 (None, 7214, 16)          0
    _________________________________________________________________
    conv1d_4 (Conv1D)            (None, 7214, 16)          2064
    _________________________________________________________________
    leaky_re_lu_7 (LeakyReLU)    (None, 7214, 16)          0
    _________________________________________________________________
    batch_normalization_7 (Batch (None, 7214, 16)          64
    _________________________________________________________________
    conv1d_5 (Conv1D)            (None, 7214, 1)           129
    =================================================================
    Total params: 3,784,625
    Trainable params: 3,755,217
    Non-trainable params: 29,408
    _________________________________________________________________

## create loss function

This VAE has two loss functions that are minimized simultaneously:

1. a weighted mean-squared-error to analyze the predicted spectra:
\begin{equation}
mse = \frac{1}{N}\sum{\frac{(x_{true}-x_{pred})^2}{(x_{err})^2}}
\end{equation}

2. a relative entropy, KL (Kullback–Leibler divergence) loss to keep the latent variables within a similar distribuition:
\begin{equation}
KL = \frac{1}{N}\sum{-\frac{1}{2}(1.0+z_{log\_var} - z_{avg}^2 - e^{z_{log\_var}})}
\end{equation}

```python
# loss for evaluating the regenerated spectra and the latent distribution
class VAE_LossLayer_weighted(Layer):
    __name__ = u'vae_labeled_loss_layer'

    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(VAE_LossLayer_weighted, self).__init__(**kwargs)

    def lossfun(self, x_true, x_pred, z_avg, z_log_var, x_err):
        mse = K.mean(K.square((x_true - x_pred)/x_err))
        kl_loss_x = K.mean(-0.5 * K.sum(1.0 + z_log_var - K.square(z_avg) - K.exp(z_log_var), axis=-1))

        return mse + kl_loss_x

    def call(self, inputs):
        # inputs for the layer:
        x_true = inputs[0]
        x_pred = inputs[1]
        z_avg = inputs[2]
        z_log_var = inputs[3]
        x_err = inputs[4]

        # calculate loss
        loss = self.lossfun(x_true, x_pred, z_avg, z_log_var, x_err)

        # add loss to model
        self.add_loss(loss, inputs=inputs)

        # returned value not really used for anything
        return x_true

# dummy loss to give zeros, hence no gradients to train
# the real loss is computed as the layer shown above and therefore this dummy loss is just
# used to satisfy keras notation when compiling the model
def zero_loss(y_true, y_pred):
    return K.zeros_like(y_true)
```

## build and compile model trainer

```python
# create loss layer
vae_loss = VAE_LossLayer_weighted()([input_spec, output_spec, z_mean, z_log_var, input_err_spec])

# build trainer with spectra, zero-masks, and error spectra as inputs. output is the final loss layer
vae = Model(inputs=[input_spec, input_mask, input_err_spec], outputs=[vae_loss])

# compile trainer
vae.compile(loss=[zero_loss],
                         optimizer=Adam(lr=1.0e-4, beta_1=0.5))

vae.summary()
```

    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            (None, 7214, 1)      0
    __________________________________________________________________________________________________
    input_2 (InputLayer)            (None, 7214, 1)      0
    __________________________________________________________________________________________________
    model_1 (Model)                 [(None, 64), (None,  7421600     input_1[0][0]
                                                                     input_2[0][0]
    __________________________________________________________________________________________________
    lambda_1 (Lambda)               (None, 64)           0           model_1[1][0]
                                                                     model_1[1][1]
    __________________________________________________________________________________________________
    model_2 (Model)                 (None, 7214, 1)      3784625     lambda_1[0][0]
    __________________________________________________________________________________________________
    input_3 (InputLayer)            (None, 7214, 1)      0
    __________________________________________________________________________________________________
    vae__loss_layer_weighted_1 (VAE [(None, 7214, 1), (N 0           input_1[0][0]
                                                                     model_2[1][0]
                                                                     model_1[1][0]
                                                                     model_1[1][1]
                                                                     input_3[0][0]
    ==================================================================================================
    Total params: 11,206,225
    Trainable params: 11,176,241
    Non-trainable params: 29,984
    __________________________________________________________________________________________________

```python
# a model that encodes and then decodes a spectrum (this is used to plot the intermediate results during training)
gen_x_to_x = Model([input_spec,input_mask], output_spec)
gen_x_to_x.compile(loss='mse',
                         optimizer=Adam(lr=1.0e-4, beta_1=0.5))
```

```python
# a function to display the time remaining or elapsed
def time_format(t):
    m, s = divmod(t, 60)
    m = int(m)
    s = int(s)
    if m == 0:
        return u'%d sec' % s
    else:
        return u'%d min' % (m)

# function for training on a batch
def train_on_batch(x_batch,x_err_batch):

    # create zero-augmentation mask for batch
    zero_mask = create_zero_mask(x_batch,0,3,1030,dataset=None,ones_padded=False)

    # train on batch
    loss = [vae.train_on_batch([cnn_reshape(x_batch),
                                           cnn_reshape(zero_mask),cnn_reshape(x_err_batch)],
                                          cnn_reshape(x_batch))]

    losses = {'vae_loss': loss[0]}
    return losses
```

```python
def fit_model(model_name, data_file, epochs, reporter):

    # get the number of spectra in the data_file
    with h5py.File(data_file, "r") as F:
            num_data_ap = len(F['spectrum'])

    # lets use 90% of the samples for training
    num_data_train_ap = int(num_data_ap*0.9)

    # the remainder will be grabbed for testing the model throughout training
    test_indices_range_ap = [num_data_train_ap,num_data_ap]

    # loop through the number of epochs
    for e in xrange(start_e,epochs):

        # create a randomized array of indices to grab batches of the spectra
        perm_ap = np.random.permutation(num_data_train_ap)

        start_time = time.time()

        # loop through the batches
        losses_=[]
        for b in xrange(0, num_data_train_ap, batchsize):

            # determine current batch size
            bsize = min(batchsize, num_data_train_ap - b)

            # grab a batch of indices
            indx_batch = perm_ap[b:b+bsize]

            # load a batch of data
            x_batch, x_err_batch= load_train_data_weighted(data_file,indices=indx_batch)

            # train on batch
            losses = train_on_batch(x_batch,x_err_batch)

            losses_.append(losses)

            # Print current status
            ratio = 100.0 * (b + bsize) / num_data_train_ap
            print unichr(27) + u"[2K",; sys.stdout.write(u'')
            print u'\rEpoch #%d | %d / %d (%6.2f %%) ' % \
                  (e + 1, b + bsize, num_data_train_ap, ratio),; sys.stdout.write(u'')

            for k in reporter:
                if k in losses:
                    print u'| %s = %5.3f ' % (k, losses[k]),; sys.stdout.write(u'')

            # Compute ETA
            elapsed_time = time.time() - start_time
            eta = elapsed_time / (b + bsize) * (num_data_train_ap - (b + bsize))
            print u'| ETA: %s ' % time_format(eta),; sys.stdout.write(u'')

            sys.stdout.flush()

        print u''

        # Print epoch status
        ratio = 100.0
        print unichr(27) + u"[2K",; sys.stdout.write(u'')
        print u'\rEpoch #%d | %d / %d (%6.2f %%) ' % \
              (e + 1, num_data_train_ap, num_data_train_ap, ratio),; sys.stdout.write(u'')

        losses_all = {}
        for k in losses_[0].iterkeys():
            losses_all[k] = tuple(d[k] for d in losses_)
        for k in reporter:
            if k in losses_all:
                losses_all[k]=np.sum(losses_all[k])/len(losses_)
        for k in reporter:
            if k in losses_all:
                print u'| %s = %5.3f ' % (k, losses_all[k]),; sys.stdout.write(u'')

        # save loss to evaluate progress
        myfile = open(model_name+'.txt', 'a')
        for k in reporter:
            if k in losses:
                myfile.write("%s," % losses[k])
        myfile.write("\n")
        myfile.close()

        # Compute Time Elapsed
        elapsed_time = time.time() - start_time
        eta = elapsed_time
        print u'| TE: %s ' % time_format(eta),; sys.stdout.write(u'')

        #sys.stdout.flush()
        print('\n')

        # save models
        encoder.save('models/encoder_'+model_name+'_epoch_'+str(e)+'.h5')
        decoder.save('models/decoder_'+model_name+'_epoch_'+str(e)+'.h5')

        # plot results for a test set to evaluate how the vae is able to reproduce a spectrum
        test_sample_indices = np.random.choice(range(test_indices_range_ap[0],test_indices_range_ap[1]), 5, replace=False)
        sample_orig,_, = load_train_data_weighted(data_file,indices=test_sample_indices)
        zero_mask_test = create_zero_mask(sample_orig,0,3,1030)
        test_x = gen_x_to_x.predict([cnn_reshape(sample_orig),cnn_reshape(zero_mask_test)])
        sample_orig_aug = sample_orig*zero_mask_test

        sample_diff = sample_orig-test_x.reshape(test_x.shape[0],test_x.shape[1])

        # save test results
        fig, axes = plt.subplots(20,1,figsize=(70, 20))
        for i in range(len(test_sample_indices)):
            # original spectrum
            axes[i*4].plot(sample_orig[i],c='r')
            axes[i*4].set_ylim((0.4,1.2))
            # input zero-augmented spectrum
            axes[1+4*i].plot(sample_orig_aug[i],c='g')
            axes[1+4*i].set_ylim((0.4,1.2))
            # regenerated spectrum
            axes[2+4*i].plot(test_x[i],c='b')
            axes[2+4*i].set_ylim((0.4,1.2))
            # residual between original and regenerated spectra
            axes[3+4*i].plot(sample_diff[i],c='m')
            axes[3+4*i].set_ylim((-0.3,0.3))
        # save results
        plt.savefig('results/test_sample_ap_'+model_name+'_epoch_'+str(e)+'.jpg')
        plt.close('all')
```

# train model

you can experiment with the number of epochs. I suggest starting with fewer and seeing if the results are adequate. if not, continue training. The models and results are saved in models/ and results/ after each epoch, so you can run analyses throughout training.

```python
reporter=['vae_loss']
epochs=30
batchsize=64
if start_e>0:
    start_e=start_e+1

data_file = '/data/stars/aspcapStar_combined_main_dr14.h5'

fit_model(model_name,data_file, epochs,reporter)
```

    Epoch #1 | 24896 / 233481 ( 10.66 %)  | vae_loss = 5541.463  | ETA: 201 min  [2K

## analyze results

Note, this is a dummy result. I haven't trained the models for a proper epoch yet

```python
import numpy as np
import h5py
import keras
import matplotlib.pyplot as plt

import sys

from keras.layers import (Input, Lambda)
from keras.engine.topology import Layer
from keras import backend as K

%matplotlib inline
```

    Using TensorFlow backend.

```python
# Define edges of detectors (for APOGEE)
blue_chip_begin = 322
blue_chip_end = 3242
green_chip_begin = 3648
green_chip_end = 6048
red_chip_begin = 6412
red_chip_end = 8306

# function for loading data
def load_train_data_weighted(data_file,indices=None):

    # grab all
    if indices is None:
        with h5py.File(data_file,"r") as F:
            ap_spectra = F['spectrum'][:]
            ap_err_spectra = F['error_spectrum'][:]
    # grab a batch
    else:
        with h5py.File(data_file, "r") as F:
            indices_bool = np.ones((len(F['spectrum']),),dtype=bool)
            indices_bool[:] = False
            indices_bool[indices] = True

            ap_spectra = F['spectrum'][indices_bool,:]
            ap_err_spectra = F['error_spectrum'][indices_bool,:]

    # combine chips
    ap_spectra = np.hstack((ap_spectra[:,blue_chip_begin:blue_chip_end],
                                     ap_spectra[:,green_chip_begin:green_chip_end],
                                     ap_spectra[:,red_chip_begin:red_chip_end]))
    # set nan values to zero
    ap_spectra[np.isnan(ap_spectra)]=0.

    ap_err_spectra = np.hstack((ap_err_spectra[:,blue_chip_begin:blue_chip_end],
                                     ap_err_spectra[:,green_chip_begin:green_chip_end],
                                     ap_err_spectra[:,red_chip_begin:red_chip_end]))

    return ap_spectra,ap_err_spectra

# zero-augmentation layer (a trick I use to input chunks of zeros into the input spectra)
class ZeroAugmentLayer(Layer):
    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(ZeroAugmentLayer, self).__init__(**kwargs)

    def zero_agument(self, x_real, zero_mask):
        return x_real*zero_mask

    def call(self, inputs):
        x_real = inputs[0]
        zero_mask = inputs[1]
        x_augmented = self.zero_agument(x_real, zero_mask)

        return x_augmented

# a function for creating the zero-masks used during training
def create_zero_mask(spectra,min_chunks,max_chunks,chunk_size,dataset=None,ones_padded=False):
    if dataset is None:
        zero_mask = np.ones_like(spectra)
    elif dataset=='apogee':
        zero_mask = np.ones((spectra.shape[0],7214))
    elif dataset=='segue':
        zero_mask = np.ones((spectra.shape[0],3688))

    num_spec = zero_mask.shape[0]
    len_spec = zero_mask.shape[1]
    num_bins = len_spec/chunk_size
    remainder = len_spec%chunk_size
    spec_sizes = np.array([chunk_size for i in range(num_bins)])
    spec_sizes[-1]=spec_sizes[-1]+remainder

    num_bins_removed = np.random.randint(min_chunks,max_chunks+1,size=(num_spec,))
    for i, mask in enumerate(zero_mask):
        bin_indx_removed = np.random.choice(num_bins, num_bins_removed[i], replace=False)
        for indx in bin_indx_removed:
            if indx==0:
                mask[indx*spec_sizes[indx]:(indx+1)*spec_sizes[indx]]=0.
            else:
                mask[indx*spec_sizes[indx-1]:indx*spec_sizes[indx-1]+spec_sizes[indx]]=0.

    return zero_mask

# function for reshaping spectra into appropriate format for CNN
def cnn_reshape(spectra):
    return spectra.reshape(spectra.shape[0],spectra.shape[1],1)
```

```python
losses = np.zeros((1,))
with open("vae_test.txt", "r") as f:
    for i,line in enumerate(f):
        currentline = np.array(line.split(",")[0],dtype=float)
        if i ==0:
            losses[0]=currentline.reshape((1,))
        else:
            losses = np.hstack((losses,currentline.reshape((1,))))
```

```python
plt.plot(losses[0:16],label='vae_loss')
plt.legend()
plt.show()
```

```python
# function for encoding a spectrum into the latent space
def encode_spectrum(model_name,epoch,spectra):

    encoder  = keras.models.load_model('models/encoder_'+model_name+'_epoch_'+str(epoch)+'.h5',
                                    custom_objects={'ZeroAugmentLayer':ZeroAugmentLayer})
    z_avg,z_log_var = encoder.predict([cnn_reshape(spectra),cnn_reshape(np.ones_like(spectra))])

    return z_avg, z_log_var
```

```python
data_file = '/data/stars/aspcapStar_combined_main_dr14.h5'

test_range = [0,30000]

test_sample_indices = np.random.choice(range(0,30000), 5000, replace=False)
sample_x,_, = load_train_data_weighted(data_file,indices=test_sample_indices)
```

```python
model_name = 'vae_test'
epoch=16

z_avg, z_log_var = encode_spectrum(model_name,epoch,sample_x)
```

    /opt/conda/lib/python2.7/site-packages/keras/models.py:252: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
      warnings.warn('No training configuration found in save file: '

## t-sne

an example of an unsupervised clustering method on the latent space.

```python
from tsne import bh_sne

perplex=80

t_data = z_avg

# convert data to float64 matrix. float64 is need for bh_sne
t_data = np.asarray(t_data).astype('float64')
t_data = t_data.reshape((t_data.shape[0], -1))
```

```python
# perform t-SNE embedding
vis_data = bh_sne(t_data, perplexity=perplex)
```

```python
# separate 2D into x and y axes information
vis_x = vis_data[:, 0]
vis_y = vis_data[:, 1]
```

```python
fig = plt.figure(figsize=(10, 10))
synth_ap = plt.scatter(vis_x, vis_y, marker='o', c='r',label='APOGEE', alpha=0.4)

plt.tick_params(
    axis='x',
    which='both',
    bottom='off',
    top='off',
    labelbottom='off')

plt.tick_params(
    axis='y',
    which='both',
    right='off',
    left='off',
    labelleft='off')

plt.legend(fontsize=30)
plt.tight_layout()
plt.show()
```

you could also use a colour gradient based on, say, the star's Teff to see how hot and cool stars clump together

```python

```


# Surfinpy

#### Tutorial 3 - Pressure

Chemical potential can be converted to pressure values using

\begin{align}
P & = \frac{\mu_O}{k_B T} ,
\end{align}

where P is the pressure, $\mu$ is the chemical potential of oxygen, $k_B$ is the Boltzmnann constant and T is the temperature.

```python
import matplotlib.pyplot as plt
from surfinpy import mu_vs_mu
from surfinpy import utils as ut
from surfinpy import data
```

```python
Oxygen_exp = ut.fit_nist("O2.txt")[298]
Water_exp = ut.fit_nist("H2O.txt")[298]
Oxygen_corrected = (-9.08 + -0.86 + Oxygen_exp) / 2
Water_corrected = -14.84 + 0.55 + Water_exp
```

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 2, energy = -780.0, funits = 4)

pure = data.DataSet(cation = 24, x = 48, y = 0, area = 60.0, energy = -575.0, label = "0.00 $TiO_2$", nspecies = 1)
H2O = data.DataSet(cation = 24, x = 48, y = 2, area = 60.0, energy = -612.0, label = "0.16 $TiO_2$", nspecies = 1)
H2O_2 = data.DataSet(cation = 24, x = 48, y = 4, area = 60.0, energy = -640.0, label = "0.32 $TiO_2$", nspecies = 1)
H2O_3 = data.DataSet(cation = 24, x = 48, y = 8, area = 60.0, energy = -676.0, label = "0.64 $TiO_2$", nspecies = 1)
Vo = data.DataSet(cation = 24, x = 46, y = 0, area = 60.0, energy = -558.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_1 = data.DataSet(cation = 24, x = 46, y = 2, area = 60.0, energy = -594.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_2 = data.DataSet(cation = 24, x = 46, y = 4, area = 60.0, energy = -624.0, label = "0.16 $TiO_1.9$", nspecies = 1)
H2O_Vo_3 = data.DataSet(cation = 24, x = 46, y = 6, area = 60.0, energy = -640.0, label = "0.32 $TiO_1.9$", nspecies = 1)
H2O_Vo_4 = data.DataSet(cation = 24, x = 46, y = 8, area = 60.0, energy = -670.0, label = "0.64 $TiO_1.9$", nspecies = 1)

data = [pure, Vo, H2O, H2O_Vo_1, H2O_2, H2O_Vo_2, H2O_3, H2O_Vo_3,  H2O_Vo_4]

```

```python
deltaX = {'Range': [ -12, -6],  'Label': 'O'}
deltaY = {'Range': [ -19, -12], 'Label': 'H_2O'}
```

```python
system, SE =  mu_vs_mu.calculate(data, bulk, deltaX, deltaY, x_energy=Oxygen_corrected, y_energy=Water_corrected)
```

As before we can generate a basic plot of oxygen chemical potential vs water chemical potential at 298 K

```python
ax = system.plot_phase(temperature=298, cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_4.png", dpi=600)
plt.show()
```

We can also generate the same plot but with the $\mu$ values converted to pressure.

```python
system.plot_pressure(temperature=298, cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_5.png", dpi=600)
plt.show()
```

Finally, we can also combine these two plots into one

```python
system.plot_mu_p(temperature=298, cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_6.png", dpi=600)
plt.show()
```