# Tutorial-IllinoisGRMHD: `convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C`

## Authors: Leo Werneck & Zach Etienne

**This module is currently under development**

## In this tutorial module we explain the algorithm used to get the BSSN variables from the ADM variables. We will also explain how to impose the constraint that the conformal metric has unit determinant.

### Required and recommended citations:

* **(Required)** Etienne, Z. B., Paschalidis, V., Haas R., Mösta P., and Shapiro, S. L. IllinoisGRMHD: an open-source, user-friendly GRMHD code for dynamical spacetimes. Class. Quantum Grav. 32 (2015) 175009. ([arxiv:1501.07276](http://arxiv.org/abs/1501.07276)).
* **(Required)** Noble, S. C., Gammie, C. F., McKinney, J. C., Del Zanna, L. Primitive Variable Solvers for Conservative General Relativistic Magnetohydrodynamics. Astrophysical Journal, 641, 626 (2006) ([astro-ph/0512420](https://arxiv.org/abs/astro-ph/0512420)).
* **(Recommended)** Del Zanna, L., Bucciantini N., Londrillo, P. An efficient shock-capturing central-type scheme for multidimensional relativistic flows - II. Magnetohydrodynamics. A&A 400 (2) 397-413 (2003). DOI: 10.1051/0004-6361:20021641 ([astro-ph/0210618](https://arxiv.org/abs/astro-ph/0210618)).

# Table of Contents
$$\label{toc}$$

This module is organized as follows

0. [Step 0](#src_dir): **Source directory creation**
1. [Step 1](#introduction): **Introduction**
1. [Step 2](#convert_adm_to_bssn__det_gammabar_eq_1): **`convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C`**
    1. [Step 2.a](#physical_metric_and_its_determinant): *Loading the physical metric $\gamma_{ij}$ and computing $\gamma = \det\left(\gamma_{ij}\right)$*
    1. [Step 2.b](#phi_and_psi): *Computing $\phi$ and $\psi$*
    1. [Step 2.c](#enforce_det_gammabar_eq_1): *Enforcing the constraint $\bar\gamma = 1$*
    1. [Step 2.d](#update_gfs): *Updating the gridfunctions*
1. [Step 3](#code_validation): **Code validation**
1. [Step 4](#latex_pdf_output): **Output this notebook to $\LaTeX$-formatted PDF file**

# Step 0: Source directory creation \[Back to [top](#toc)\]
$$\label{src_dir}$$

We will now use the [cmdline_helper.py NRPy+ module](Tutorial-Tutorial-cmdline_helper.ipynb) to create the source directory within the `IllinoisGRMHD` NRPy+ directory, if it does not exist yet.

```python
# Step 0: Creation of the IllinoisGRMHD source directory
# Step 0a: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..","..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0b: Load up cmdline_helper and create the directory
import cmdline_helper as cmd
IGM_src_dir_path = os.path.join("..","src")
cmd.mkdir(IGM_src_dir_path)

# Step 0c: Create the output file path
outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C  = os.path.join(IGM_src_dir_path,"convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C")
```

# Step 1: Introduction \[Back to [top](#toc)\]
$$\label{introduction}$$

In this module we will explain the procedure used within `IllinoisGRMHD` to compute the conformal metric, $\bar\gamma_{ij}$, and its inverse, $\bar\gamma^{ij}$, from the physical metric $\gamma_{ij}$. We will also describe how to compute $\phi$, the conformal factor, and $\psi\equiv e^{\phi}$. Finally, we also explain the procedure used to enforce the constraint $\bar\gamma = \det\left(\bar\gamma_{ij}\right) = 1$.

**A note on notation**: The notation used throuhgout the NRPy tutorial notebooks for the conformal metric is $\bar\gamma_{ij}$. However, in the literature the notation $\tilde\gamma_{ij}$ is also used to represent the conformal metric. It is important to note that in `IllinoisGRMHD` we refer to the conformal metric using the *latter* notation, i.e. ${\rm gtij} := \tilde\gamma_{ij}$ and ${\rm gtupij} := \tilde\gamma^{ij}$. To keep the discussion consistent with the other notebooks, however, we will still present the discussion using the notation $\tilde\gamma_{ij} \to \bar\gamma_{ij}$. Bottom line, here $\tilde\gamma_{ij}$ and $\bar\gamma_{ij}$ represent exactly the same quantity, the conformal metric.

# Step 2: `convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C` \[Back to [top](#toc)\]
$$\label{convert_adm_to_bssn__det_gammabar_eq_1}$$

## Step 2.a: Loading the physical metric $\gamma_{ij}$ and computing $\gamma = \det\left(\gamma_{ij}\right)$ \[Back to [top](#toc)\]
$$\label{physical_metric_and_its_determinant}$$

We start by reading in the physical metric ${\rm gij_physL} := \gamma_{ij}$ and computing its determinant

$$
\boxed{
\begin{align}
\gamma = \det\left(\gamma_{ij}\right)
&= \gamma_{xx}\gamma_{yy}\gamma_{zz} + \gamma_{xy}\gamma_{yz}\gamma_{xz} + \gamma_{xz}\gamma_{xy}\gamma_{yz}\\
&- \gamma_{xz}\gamma_{yy}\gamma_{xz} - \gamma_{xy}\gamma_{xy}\gamma_{zz} - \gamma_{xx}\gamma_{yz}\gamma_{yz}
\end{align}
}\ .
$$

Notice that we have used the fact that $\gamma_{ij}$ is symmetric above, i.e. $\gamma_{ij}=\gamma_{ji}$.

```python
%%writefile $outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C
#include <cmath>
#include <cstdio>
#include <cstdlib>
#ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
#include "cctk.h"
#include "cctk_Parameters.h"
#endif

void IllinoisGRMHD_convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij
(const cGH *cctkGH,const int *cctk_lsh,
 CCTK_REAL *gxx,CCTK_REAL *gxy,CCTK_REAL *gxz,CCTK_REAL *gyy,CCTK_REAL *gyz,CCTK_REAL *gzz,CCTK_REAL *alp,
 CCTK_REAL *gtxx,CCTK_REAL *gtxy,CCTK_REAL *gtxz,CCTK_REAL *gtyy,CCTK_REAL *gtyz,CCTK_REAL *gtzz,
 CCTK_REAL *gtupxx,CCTK_REAL *gtupxy,CCTK_REAL *gtupxz,CCTK_REAL *gtupyy,CCTK_REAL *gtupyz,CCTK_REAL *gtupzz,
 CCTK_REAL *phi,CCTK_REAL *psi,CCTK_REAL *lapm1) {

#ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
 DECLARE_CCTK_PARAMETERS;
#endif

#pragma omp parallel for
  for(int k=0;k<cctk_lsh[2];k++) for(int j=0;j<cctk_lsh[1];j++) for(int i=0;i<cctk_lsh[0];i++) {
        int index=CCTK_GFINDEX3D(cctkGH,i,j,k);
        CCTK_REAL gxx_physL=gxx[index];
        CCTK_REAL gxy_physL=gxy[index];
        CCTK_REAL gxz_physL=gxz[index];
        CCTK_REAL gyy_physL=gyy[index];
        CCTK_REAL gyz_physL=gyz[index];
        CCTK_REAL gzz_physL=gzz[index];

        /**********************************************************************
         * Compute \tilde{\gamma_{ij}}, phi, and psi (BSSN) from g_{ij} (ADM) *
         **********************************************************************/
        CCTK_REAL gijdet = gxx_physL * gyy_physL * gzz_physL + gxy_physL * gyz_physL * gxz_physL + gxz_physL * gxy_physL * gyz_physL
          - gxz_physL * gyy_physL * gxz_physL - gxy_physL * gxy_physL * gzz_physL - gxx_physL * gyz_physL * gyz_physL;

        gijdet = fabs(gijdet);
```

    Writing ../src/convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C

## Step 2.b: Computing $\phi$ and $\psi$ \[Back to [top](#toc)\]
$$\label{phi_and_psi}$$

The BSSN formalism relates the physical metric, $\gamma_{ij}$, to a conformal metric, $\bar\gamma{ij}$, via the relation

$$
\boxed{\bar\gamma_{ij} = e^{-4\phi}\gamma_{ij}}\ ,
$$

with the constraint that $\bar\gamma \equiv \det\left(\bar\gamma_{ij}\right) = 1$. This immediately implies that

$$
e^{-12\phi} \gamma = \bar\gamma = 1 \implies -12\phi = \log\left(\frac{1}{\gamma}\right) = - \log\gamma
\implies \boxed{\phi = \frac{1}{12}\log\gamma}\ .
$$

Useful quantities to compute are

$$
\boxed{
\begin{align}
\psi      &\equiv e^{\phi}\\
\psi^{-4} &\equiv e^{-4\phi}
\end{align}
}\ .
$$

All the boxed quantities above are evaluated below.

```python
%%writefile -a $outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C

        CCTK_REAL phiL = (1.0/12.0) * log(gijdet);
        CCTK_REAL psiL = exp(phiL);

        CCTK_REAL Psim4 = 1.0/(psiL*psiL*psiL*psiL);
        CCTK_REAL gtxxL = gxx_physL*Psim4;
        CCTK_REAL gtxyL = gxy_physL*Psim4;
        CCTK_REAL gtxzL = gxz_physL*Psim4;
        CCTK_REAL gtyyL = gyy_physL*Psim4;
        CCTK_REAL gtyzL = gyz_physL*Psim4;
        CCTK_REAL gtzzL = gzz_physL*Psim4;
```

    Appending to ../src/convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C

## Step 2.c: Enforcing the constraint $\bar\gamma = 1$ \[Back to [top](#toc)\]
$$\label{enforce_det_gammabar_eq_1}$$

The BSSN formalism evolves the determinant of the conformal metric through $\partial_{t}\bar\gamma = 0$. Since initially $\bar\gamma=1$, one would expect that $\bar\gamma=1$ at all times. However, numerical errors can cause the value of the determinant of the conformal metric to drift away from unity. To remedy this, we enforce the constraint $\bar\gamma=1$ by transforming the conformal metric according to the algebraic condition (cf. eq. 53 of [Ruchlin, Etienne, and Baumgarte (2018)](https://arxiv.org/pdf/1712.07658.pdf) with $\hat\gamma = 1$).

$$
\boxed{\bar{\gamma}_{ij} \to \left(\frac{1}{\bar{\gamma}}\right)^{1/3} \bar{\gamma}_{ij}}\ .
$$

We then start by evaluating

$$
\boxed{
\begin{align}
\bar\gamma = \det\left(\bar\gamma_{ij}\right)
&= \bar\gamma_{xx}\bar\gamma_{yy}\bar\gamma_{zz} + \bar\gamma_{xy}\bar\gamma_{yz}\bar\gamma_{xz} + \bar\gamma_{xz}\bar\gamma_{xy}\bar\gamma_{yz}\\
&- \bar\gamma_{xz}\bar\gamma_{yy}\bar\gamma_{xz} - \bar\gamma_{xy}\bar\gamma_{xy}\bar\gamma_{zz} - \bar\gamma_{xx}\bar\gamma_{yz}\bar\gamma_{yz}
\end{align}
}\ .
$$

In the code below we also define

$$
\boxed{{\rm gtijdet\_Fm1o3} = \left(\frac{1}{\bar{\gamma}}\right)^{1/3}}\ .
$$

```python
%%writefile -a $outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C

        /*********************************
         * Apply det gtij = 1 constraint *
         *********************************/
        CCTK_REAL gtijdet = gtxxL * gtyyL * gtzzL + gtxyL * gtyzL * gtxzL + gtxzL * gtxyL * gtyzL -
          gtxzL * gtyyL * gtxzL - gtxyL * gtxyL * gtzzL - gtxxL * gtyzL * gtyzL;

        CCTK_REAL gtijdet_Fm1o3 = fabs(1.0/cbrt(gtijdet));

        gtxxL = gtxxL * gtijdet_Fm1o3;
        gtxyL = gtxyL * gtijdet_Fm1o3;
        gtxzL = gtxzL * gtijdet_Fm1o3;
        gtyyL = gtyyL * gtijdet_Fm1o3;
        gtyzL = gtyzL * gtijdet_Fm1o3;
        gtzzL = gtzzL * gtijdet_Fm1o3;

        if(gtijdet<0.0) {
#ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
          CCTK_VWarn(CCTK_WARN_ALERT,__LINE__, __FILE__, CCTK_THORNSTRING,
#else
          fprintf(stderr,
#endif
"WARNING: det[3-metric]<0.0 at point  %d %d %d | cctk_lsh: %d %d %d. Hopefully this is occurring in gz's! gtij_phys = %.2e %.2e %.2e %.2e %.2e %.2e gtij_new = %.2e %.2e %.2e %.2e %.2e %.2e | gijdet = %.2e | gtijdet = %.2e\n",
i,j,k,cctk_lsh[0],cctk_lsh[1],cctk_lsh[2],gxx_physL,gxy_physL,gxz_physL,gyy_physL,gyz_physL,gzz_physL,gtxxL,gtxyL,gtxzL,gtyyL,gtyzL,gtzzL,-gijdet,gtijdet);
}

```

    Appending to ../src/convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C

## Step 2.d: Updating the gridfunctions \[Back to [top](#toc)\]
$$\label{update_gfs}$$

Finally, we update the gridfunctions for $\phi$, $\psi$, $\bar\gamma_{ij}$, $\gamma_{ij}$, and $\bar\gamma^{ij}$, respectively, in the code below.

If $\mathcal{M}$ is a $3\times3$ symmetric matrix and $\det(\mathcal{M})\neq0$, [recall that we have](https://en.wikipedia.org/wiki/Invertible_matrix#Inversion_of_3_%C3%97_3_matrices)

$$
\mathcal{M}^{-1} =
\begin{pmatrix}
a & b & c\\
b & d & e\\
c & e & f
\end{pmatrix}^{-1}
=
\frac{1}{\det(\mathcal{M})}
\begin{pmatrix}
A & B & C\\
B & D & E\\
C & E & F
\end{pmatrix}\ ,
$$

where

$$
\boxed{
\begin{align}
A &= \ \ \left(df - ee\right)\\
B &= -\left(bf - ce\right)\\
C &= \ \ \left(be - cd\right)\\
D &= \ \ \left(af - cc\right)\\
E &= -\left(ae - bc\right)\\
F &= \ \ \left(ad - bb\right)
\end{align}
}\ .
$$

Notice that if we replace $\mathcal{M}\to\bar\gamma_{ij}$, then we have also $\det(\mathcal{M})\to1$.

```python
%%writefile -a $outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C

        CCTK_REAL Psi4 = psiL*psiL*psiL*psiL;
        /*****************************************
         * Set all the needed BSSN gridfunctions *
         *****************************************/
        phi[index] = phiL;
        psi[index] = psiL;

        lapm1[index] = alp[index] - 1.0;

        gtxx[index] = gtxxL;
        gtxy[index] = gtxyL;
        gtxz[index] = gtxzL;
        gtyy[index] = gtyyL;
        gtyz[index] = gtyzL;
        gtzz[index] = gtzzL;

        gxx[index] = gtxxL*Psi4;
        gxy[index] = gtxyL*Psi4;
        gxz[index] = gtxzL*Psi4;
        gyy[index] = gtyyL*Psi4;
        gyz[index] = gtyzL*Psi4;
        gzz[index] = gtzzL*Psi4;

        gtupxx[index] =   ( gtyyL * gtzzL - gtyzL * gtyzL );
        gtupxy[index] = - ( gtxyL * gtzzL - gtyzL * gtxzL );
        gtupxz[index] =   ( gtxyL * gtyzL - gtyyL * gtxzL );
        gtupyy[index] =   ( gtxxL * gtzzL - gtxzL * gtxzL );
        gtupyz[index] = - ( gtxxL * gtyzL - gtxyL * gtxzL );
        gtupzz[index] =   ( gtxxL * gtyyL - gtxyL * gtxyL );
      }
}

```

    Appending to ../src/convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C

# Step 3: Code validation \[Back to [top](#toc)\]
$$\label{code_validation}$$

First we download the original `IllinoisGRMHD` source code and then compare it to the source code generated by this tutorial notebook.

```python
# Verify if the code generated by this tutorial module
# matches the original IllinoisGRMHD source code

# First download the original IllinoisGRMHD source code
import urllib
from os import path

original_IGM_file_url  = "https://bitbucket.org/zach_etienne/wvuthorns/raw/5611b2f0b17135538c9d9d17c7da062abe0401b6/IllinoisGRMHD/src/convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C"
original_IGM_file_name = "convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij-original.C"
original_IGM_file_path = os.path.join(IGM_src_dir_path,original_IGM_file_name)

# Then download the original IllinoisGRMHD source code
# We try it here in a couple of ways in an attempt to keep
# the code more portable
try:
    original_IGM_file_code = urllib.request.urlopen(original_IGM_file_url).read().decode("utf-8")
    # Write down the file the original IllinoisGRMHD source code
    with open(original_IGM_file_path,"w") as file:
        file.write(original_IGM_file_code)
except:
    try:
        original_IGM_file_code = urllib.urlopen(original_IGM_file_url).read().decode("utf-8")
        # Write down the file the original IllinoisGRMHD source code
        with open(original_IGM_file_path,"w") as file:
            file.write(original_IGM_file_code)
    except:
        # If all else fails, hope wget does the job
        !wget -O $original_IGM_file_path $original_IGM_file_url

# Perform validation
Validation__ADM_to_BSSN__det_gammabar_eq_1__C  = !diff $original_IGM_file_path $outfile_path__ADM_to_BSSN__det_gammabar_eq_1__C

if Validation__ADM_to_BSSN__det_gammabar_eq_1__C == []:
    # If the validation passes, we do not need to store the original IGM source code file
    !rm $original_IGM_file_path
    print("Validation test for convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C: PASSED!")
else:
    # If the validation fails, we keep the original IGM source code file
    print("Validation test for convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C: FAILED!")
    # We also print out the difference between the code generated
    # in this tutorial module and the original IGM source code
    print("Diff:")
    for diff_line in Validation__ADM_to_BSSN__det_gammabar_eq_1__C:
        print(diff_line)
```

    Validation test for convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.C: FAILED!
    Diff:
    1d0
    < #include "cctk.h"
    4a4,5
    > #ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
    > #include "cctk.h"
    5a7
    > #endif
    13c15,20
    <   DECLARE_CCTK_PARAMETERS;
    ---
    >
    > #ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
    >  DECLARE_CCTK_PARAMETERS;
    > #endif
    >
    >
    31a39
    >
    41a50
    >
    58,60c67,76
    <         if(gtijdet<0.0) { CCTK_VWarn(CCTK_WARN_ALERT,__LINE__, __FILE__, CCTK_THORNSTRING,
    <                                      "WARNING: det[3-metric]<0.0 at point  %d %d %d | cctk_lsh: %d %d %d. Hopefully this is occurring in gz's! gtij_phys = %.2e %.2e %.2e %.2e %.2e %.2e gtij_new = %.2e %.2e %.2e %.2e %.2e %.2e | gijdet = %.2e | gtijdet = %.2e",
    < 				     i,j,k,cctk_lsh[0],cctk_lsh[1],cctk_lsh[2],gxx_physL,gxy_physL,gxz_physL,gyy_physL,gyz_physL,gzz_physL,gtxxL,gtxyL,gtxzL,gtyyL,gtyzL,gtzzL,-gijdet,gtijdet); }
    ---
    >         if(gtijdet<0.0) {
    > #ifndef ENABLE_STANDALONE_IGM_C2P_SOLVER
    >           CCTK_VWarn(CCTK_WARN_ALERT,__LINE__, __FILE__, CCTK_THORNSTRING,
    > #else
    >           fprintf(stderr,
    > #endif
    > "WARNING: det[3-metric]<0.0 at point  %d %d %d | cctk_lsh: %d %d %d. Hopefully this is occurring in gz's! gtij_phys = %.2e %.2e %.2e %.2e %.2e %.2e gtij_new = %.2e %.2e %.2e %.2e %.2e %.2e | gijdet = %.2e | gtijdet = %.2e\n",
    > i,j,k,cctk_lsh[0],cctk_lsh[1],cctk_lsh[2],gxx_physL,gxy_physL,gxz_physL,gyy_physL,gyz_physL,gzz_physL,gtxxL,gtxyL,gtxzL,gtyyL,gtyzL,gtzzL,-gijdet,gtijdet);
    > }
    >
    92a109
    >

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.pdf](Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means).

```python
latex_nrpy_style_path = os.path.join(nrpy_dir_path,"latex_nrpy_style.tplx")
#!jupyter nbconvert --to latex --template $latex_nrpy_style_path --log-level='WARN' Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.ipynb
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__convert_ADM_to_BSSN__enforce_detgtij_eq_1__and_compute_gtupij.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```


```python
%load_ext rpy2.ipython
```

```r
%%R
install.packages('psych')
```

```python
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 1000)
plt.plot(x, np.sin(x));

plt.show()
```

```python
!pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl
!pip3 install torchvision
```

    ERROR: torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl is not a supported wheel on this platform.
    Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (0.11.1+cu111)
    Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision) (7.1.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchvision) (1.19.5)
    Requirement already satisfied: torch==1.10.0 in /usr/local/lib/python3.7/dist-packages (from torchvision) (1.10.0+cu111)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.10.0->torchvision) (3.10.0.2)

```python
from sympy import Integral, sqrt, symbols, init_printing

init_printing()

x = symbols('x')

Integral(sqrt(1 / x), x)
```

```r
%%R
library(tidyverse)
library(psych)
dat_loan <- read.table('loan.txt', header=T)
```

```r
%%R
des <- psych::describe(dat_loan)
```

```r
%%R
c(1:10) |> sum()
```

    [1] 55

```r
%%R
sessionInfo()
```

    R version 4.1.2 (2021-11-01)
    Platform: x86_64-pc-linux-gnu (64-bit)
    Running under: Ubuntu 18.04.5 LTS

    Matrix products: default
    BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
    LAPACK: /usr/lib/x86_64-linux-gnu/openblas/liblapack.so.3

    locale:
     [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
     [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
     [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
     [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
     [9] LC_ADDRESS=C               LC_TELEPHONE=C
    [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

    attached base packages:
    [1] tools     stats     graphics  grDevices utils     datasets  methods
    [8] base

    other attached packages:
    [1] forcats_0.5.1   stringr_1.4.0   dplyr_1.0.7     purrr_0.3.4
    [5] readr_2.1.1     tidyr_1.1.4     tibble_3.1.6    ggplot2_3.3.5
    [9] tidyverse_1.3.1

    loaded via a namespace (and not attached):
     [1] Rcpp_1.0.7       cellranger_1.1.0 pillar_1.6.4     compiler_4.1.2
     [5] dbplyr_2.1.1     jsonlite_1.7.2   lubridate_1.8.0  lifecycle_1.0.1
     [9] gtable_0.3.0     pkgconfig_2.0.3  rlang_0.4.12     reprex_2.0.1
    [13] cli_3.1.0        rstudioapi_0.13  DBI_1.1.2        haven_2.4.3
    [17] xml2_1.3.3       withr_2.4.3      httr_1.4.2       fs_1.5.2
    [21] generics_0.1.1   vctrs_0.3.8      hms_1.1.1        grid_4.1.2
    [25] tidyselect_1.1.1 glue_1.6.0       R6_2.5.1         fansi_0.5.0
    [29] readxl_1.3.1     tzdb_0.2.0       modelr_0.1.8     magrittr_2.0.1
    [33] backports_1.4.1  scales_1.1.1     ellipsis_0.3.2   rvest_1.0.2
    [37] assertthat_0.2.1 colorspace_2.0-2 utf8_1.2.2       stringi_1.7.6
    [41] munsell_0.5.0    broom_0.7.10     crayon_1.4.2

```python
%%shell
jupyter nbconvert --to html /content/Untitled9.ipynb
```

    [NbConvertApp] Converting notebook /content/Untitled9.ipynb to html
    [NbConvertApp] Writing 302251 bytes to /content/Untitled9.html


# Use Python in Advanced Statistics

*這個學期剛剛學完了概率論與數理統計，這好趁這個機會複習一下並複習一下 `python`*

## Chapter One Probability

### 隨機試驗與樣本空間（Random Experiment and Sample Space）

#### 隨機試驗
隨機試驗是概率論中一個基本的概念。概括的講，在概率論中把符合下面三個特點的試驗叫做隨機試驗：
* 可以在向空的條件下重複進行；
* 每次試驗的可能結果不只一個，並且事先明確試驗的所有可能結果；
* 進行一次試驗之前不能確定哪一個結果會出現。

隨機試驗有很多種，例如常出現的擲骰子，摸球，射擊等。所有的隨機試驗的結果可以分為兩類來表示：

* 數量化表示：射擊命中的次數，商場每個小時的客流量，每天經過某個收費站的車輛等，這個結果本事就是數字；
* 非數量化表示：拋硬幣的結果（正面/反面），化驗的結果（陽性/陰性）等，這些結果是定型的，非數量化的。但是可以用示性函數來表示，例如可以規定正面（陽性）為$1$，反面為$0$，這樣就可以實現了非數量化結果的數量化。

#### 樣本空間（Sample Space）：

* 隨機試驗的所有可能結果構成的集合。一般即為$S$（capital S）；
* $S$ 中的元素$e$稱為樣本點（也可以叫基本事件）；
* 事件是樣本空間的子集，同樣是一個集合。

#### 事件的關係
* 事件的包含：$A \subseteq B$;
* 事件的相等：$A = B$;
* 互斥事件（互不相容事件）：不能同時出現；
* 事件的和（並）：$A cup B$
* 事件的差： $A - B$,$A$發生，$B$不發生；
* 對立事件（逆事件）：互斥，必須出現其中一個。

事件的運算性質就是集合的性質

### 頻率和概率

#### 頻率：

頻率是指$0～1$之間的一個實數，在大量重複試驗的基礎上給出了隨機事件發生可能性的估計。

概率的穩定性：在充分多次試驗中，事件的頻率總在一個定值附近擺動，而且，試驗次數越多擺動越小。這個性質叫做頻率的穩定性。

#### 概率：

概率的統計性定義：當試驗次數增加時，隨機時間$A$發生的頻率的穩定值為$p$就稱為概率。記為$P(A) = P$

概率的公理化定義：設隨機試驗對於的樣本空間為$S$。對每一個事件$A$，定義為$P(A)$，滿足：

1. 非負性：$P(A) \geq 0$;
2. 規範性：$P(S) = 1;
3. 可列可加性：$A_1, A_2, \dots 兩兩互斥，及$A_iA_j = \oslash, i \neq j$則 $P(\cup A_i) = \sum P(A_i)$

#### 條件概率（Conditional Probability）:
$P(A|B)$表示在事件$B$發生的條件下，事件$A$發生的概率，相當於$A$在$B$所佔的比例。此時，樣本空間從原來的完整樣本空間$S$縮小到了$B$，由於有了條件的約束（事件$B$），使得原本的樣本空間減少了。

下面我們可以通過韋恩圖做示例：

plot one：條件概率的樣本空間；

plot two：條件概率應如何計算

```python
from matplotlib import pyplot as plt
import numpy as np
import sympy

from matplotlib_venn import venn3, venn3_circles
plt.figure(figsize=(4,4))
v = venn2(subsets=(2,2,1), set_labels = ('A', 'B'))

plt.title("Sample Venn diagram - plot one")
plt.annotate('P(AB)', xy=v.get_label_by_id('11').get_position() - np.array([0, 0.05]), xytext=(-70,-70),
             ha='center', textcoords='offset points', bbox=dict(boxstyle='round,pad=0.5', fc='gray', alpha=0.1),
             arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.5',color='gray'))
plt.show()

```

```python
from matplotlib import pyplot as plt
import numpy as np
import sympy

from matplotlib_venn import venn3, venn3_circles
plt.figure(figsize=(4,4))
v = venn2(subsets=(2,2,1), set_labels = ('A', 'B'))

c = venn2_circles(subsets=(2, 2, 1), linestyle='dashed')
c[0].set_lw(1.0)
c[0].set_ls('dotted')
plt.title("Sample Venn diagram")
plt.annotate('P(AB)', xy=v.get_label_by_id('11').get_position() - np.array([0, 0.05]), xytext=(-70,-70),
             ha='center', textcoords='offset points', bbox=dict(boxstyle='round,pad=0.5', fc='gray', alpha=0.1),
             arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.5',color='gray'))
plt.show()
```

$P(B|A) = \frac{P(AB)}{P(A)}$

$P(A|B) = \frac{P(AB)}{P(B)}$

例題：一個家庭中有兩個小孩，已知至少一個是女孩，問兩個都是女孩的概率是多少？（假設生男生女是等可能的）

**解**：由題意可得：樣本空間為

    S = {(兄,弟), (兄,妹),(姐,弟),(姐,妹)}

    B = {(兄,妹), (姐,弟),(姐,妹)}

    A = {(姐,妹)}

由於，事件 $B$ 已經發生了，所以這時試驗的所有可能只有三種，而事件 $A$ 包含的基本事件只占其中的一種，所以有：$P(A|B) = \frac{1}{3}$

即，在已知至少一個是女孩的請卡滾下，兩個都是女孩的概率為$\frac{1}{3}$。在這個例子中，如果不知道事件 $B$ 發生，則事件 $A$ 發生的概率為 $P(A) = \frac{1}{4}$ 這裡的 $P(A) \neq P(A|B)$，其中的原因在於事件 $B$ 的發生改變了樣本空間，使它由原來的 $S$ 縮減為新的樣本空間 $S_B = B$

#### 隨機變量（Random Variable）

在幾乎所有教材裡，介紹概率論都是從事件和樣本空間說起的，但是後面的概率論都是圍繞著隨機變量展開的。可以說前面的事件和樣本空間都是引子，引出了隨機變量這個概率論的核心概念。後面的統計學是建立在概率論的理論基礎之上的，因此可以說理解隨機變量這個概念是學習和運用概率論與數理統計的關鍵。

**隨機變量**：

* 首先這是一個變量，變量與常數相對，也就是說其取值是不明確的，其實隨機變量的整個取值範圍就是前面說的樣本空間；
* 其次這個量是隨機的，也就是說它的去職代有不確定性，讓然是在樣本空間這個範圍內的。

**定義：**
> 設隨機試驗的樣本空間是 $S$ ，若對 $S$ 中的每一個樣本點 $e$ ，都有唯一的實數值 $X(e)$ 為隨機變量，間記為 $X$

隨機變量的定義並不複雜，但是理解起來去不是這麼直觀。

* 首先，隨即變量與之前定義的事件是有關係的，因為每個樣本點本身就是一個基本事件；
* 在前面隨機試驗結果的表示中提到，無論是數量化的結果還是非數量化的結果，即不管試驗結果是否與數值有關，都可以引入變量，使試驗結果與數建立對應關係；
* 隨機變量本質上是一種函數，其目的就是建立試驗結果（樣本中的點，同基本事件$e$）與實數之間的對應關係（例如將“正面”影射為$1$，“反面”影射為$0$）；
* 自變量為基本事件$e$，定義域為樣本空間$S$，值域為某個實數集合，多個自變量可以對應同一個函數值，但不允許一個自變量對應多個函數值；
* 隨機變量$X$取某個值或某些值就表示某種事件，且具有一定的概率；
* 隨機變量中的隨機來源於隨機試驗結果的不確定性。

我們可以通過引入隨機變量，我們簡化了隨機試驗結果（事件）的表示，從而可以更加方便的對隨機試驗進行研究。

**隨機變量的分類**:

* 離散隨機變量；
* 連續隨機變量；
* 每類隨機變量都有其獨特的概率密度函數和概率分佈函數。

**隨機變量的數字特徵**：

* 期望（均值），眾數，分位數，中位數；
* 方差；
* 協方差；
* 相關係數。

### 隨機變量（Random Variable）

*對隨機變量以及其取值規律的研究是概率的核心內容。在上一個小結中，總結了隨機變量的概念以及隨機變量與事件的聯繫。這個小結會更加深入的討論隨機變量。*

#### 隨機變量與事件

隨機變量的本質是一種函數（映射關係），在古典概率模型中，“事件和事件的概率”是核心概念；但是在現代概率論中，“隨機變量及其取值規律”是核心概念。

**隨機變量與事件的聯繫與區別**

小結 1 中對著練歌概念的聯繫進行了非常詳細的描述。隨機變量實際上只是事件的另一種表達方式，這種表達方式更加的形式化和符號化，也更佳便於理解以及進行邏輯運算。不同的事件，其實就是隨機變量不同取值的組合。在陈希孺先生的書中，有一個很好的例子來說明這兩者的區別：

> 對於隨機試驗，我們所關心的往往是與所研究的特定問題有關的某個或某些變量，而這些量就是隨機變量。當然，有事我們所關心的是某個或某些特定的隨機時間。例如，在特定一群人中，年收入在萬元以上的高收入者，以及年收入在$3000$元以下的低收入者，各自比率如何？者看上去是兩個孤立的事件。可是當我們引入一個隨機變量$X$：
>
>

>  $X = $ 隨機抽出一個人其年收入
>

>
>則$X$是我們關心的隨機變量。上述兩個事件可以分表表示為$\{X > 10000\}$或$\{X < 3000\}$。這就看出：**隨機事件**這個概念實際上包容在**隨機變量**這個更廣的概念之中。也就是說，隨機事件是靜態的觀點來研究隨機現象，而隨機變量則是一種動態的觀點。「概率論能從計算一些孤立事件的概率發展成為一個更高的理論體系，其根本概念就是隨機變量

*這段話，非常清楚的解釋了隨機變量與事件的區別：就跟變量和常量之間的區別那樣*

#### 隨機變量的分類（The Classification of the Random Variable）

**隨機變量從其可能的取值的性質分為兩大類：離散型隨機變量（Discrete Random Variable）和連續性隨機變量（Continuous Random Variable）**

##### 離散型隨機變量（Discrete Random Variable）

離散型隨機變量的取值在整個實數軸上是有間隔的，要麼只有有限個取值，要麼就是無限可數。

如下圖：

```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

def poisson_pmf(mu=3):

    poisson_dis = stats.poisson(mu)
    x = np.arange(poisson_dis.ppf(0.001), poisson_dis.ppf(0.999))
    print(x)

    fig, ax = plt.subplots(1, 1)
    ax.plot(x, poisson_dis.pmf(x), 'bo', ms=8, label='poisson pmf')
    ax.vlines(x, 0, poisson_dis.pmf(x), colors='b', lw=5, alpha=0.5)
    ax.legend(loc='best', frameon=False)
    plt.ylabel('Probability')
    plt.title('PMF of poisson distribution(mu={}) - plot three'.format(mu))
    plt.show()

poisson_pmf(mu=8)
```

```python
def binom_pmf(n=1, p=0.1):
    binom_dis = stats.binom(n, p)
    x = np.arange(binom_dis.ppf(0.0001), binom_dis.ppf(0.9999))
    print(x)

    fig, ax = plt.subplots(1, 1)
    ax.plot(x, binom_dis.pmf(x), 'bo', label='binom pmf')
    ax.vlines(x, 0, binom_dis.pmf(x), colors='b', lw=5, alpha=0.5)
    ax.legend(loc='best', frameon=False)
    plt.ylabel('Probability')
    plt.title('PMF of binomial distribution(n={}, p={}) - plot four'.format(n, p))

    plt.show()

binom_pmf(n=20, p=0.6)
```

* Plot three 是 Poisson Distribution
* Plot four 是 Binomal Distribution

常見的**離散型隨機變量**包括以下這幾種：

* 0-1分佈（Bernoulli Distribution）
* 二項分布（Binomial Distribution）
* 幾何分佈（Geometric Distribution）
* 泊松分佈（Poisson Distribution）
* 超幾何分佈（Hyper-geometric Distribution）

##### 連續型隨機變量（Continuous Random Variable）

連續型隨機變量的取值要麼包括了整個實數集$(-\infty, \infty)$，要麼在一個區間內連續，總之，這一類的隨機變量的可能取值要比離散型隨機變量的取值多得多，個數是無窮不可數的。

**常見的連續型隨機變量包括以下幾種**：

* 均勻分布
* 指數分佈
* 正太分佈（$\gamma$分佈， $\beta$分佈，$\chi^2$分佈等）

##### 概率密度函數的性質

所有的概率密度函數$f(x)$都滿足一下的兩條性質；所有滿足下面兩條性質的一元函數也都可以作為概率密度函數。

$f(x) \geq 0$，以及$\int_{-\infty}^{+\infty}f(x)dx = 1$.

#### 隨機變量的基本性質

隨機變量最主要的性質是其所有可能取到的這些值的取值規律，即取到的概率大小。如果我們把一個隨機變量的所有可能的取值的規律都研究透徹了，那麼這個隨機變量也就研究透徹了。隨機案變量的性質只要有兩類：一類是大而全的性質，這類性質可以詳細描述所有可能取值的概率，律如**累積分佈函數（Cumulative Distribution Funtion）**和**概率密度函數（Probability Density Function）**；另一類是找到該隨機變量的一些特徵或者代表值，例如隨機變量的方差或者期望等數字特徵。常見的隨機變量的性質如下表：

|    name    | 解釋 |
| :---: | :---: |
| CDF: Cumulative Distribution Function |  連續型和離散型隨機變量都有，一般以$F(X)$表示 |
| PDF: Probability Density Function    |  連續型隨機變量在各點的取值規律，用$f(x)$表示 |
| PDF: Probability Density Function    |  連續型隨機變量在各點的取值規律，用$f(x)$表示 |
| PMF: Probability Mass Function    |  離散型型隨機變量在各特定取值上的概率 |
| RVS: Random Variate Sample    |  從一個給定分佈取樣 |
| PPF: Percentile Point Function  |  CDF 的反函數 |
| IQR: Inter Quartile Range   |  $25%$分位數與$75%$ 分位數之差 |

*PDF 只有**連續型隨機變量**才有， PMF 只有**離散型隨機變量**才有；一個分佈的 CDF 求導等於 PDF， 一個分佈的 PDF 積分後就是 CDF*

### 一位離散型隨機變量及其 Python 實現

上一小節，對隨機變量做了一個概述，這一節主要紀錄以為離散變量以及關於他們的一些性質。對於概率論與數理統計方面的計算以及可視化，主要的`pyhton`包有`scipy`，`numpy`和`matplotlib`等。

```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
```

`scipy` 是 `python` 中使用最為廣泛的科學計算工具包，在加上`numpy`和`matplotlib`，基本可以處理大多數的計算和作圖任務。下面是`wikipedia`對`scipy`的介紹：

> SciPy是一个开源的Python算法库和数学工具包。SciPy包含的模块有最优化、线性代数、积分、插值、特殊函数、快速傅里叶变换、信号处理和图像处理、常微分方程求解和其他科学与工程中常用的计算。与其功能相类似的软件还有MATLAB、GNU Octave和Scilab。SciPy目前在BSD许可证下发布。它的开发由Enthought资助。

我們使用的是 `scipy`中的 `stats`模塊，這個模塊包歡樂概率論以及統計相關的函數。
相關函數可以查詢： scipy stats

####  伯努利分佈（Bernoulli Distribution）

又名兩點分佈或者$0-1$分佈，是一個離散型概率分佈。若伯努利試驗成功，則伯努利隨機變量取值為$1$，如果失敗則取值為$0$。記其成功概率為$p(0\leq p \leq 1)$，失敗概率為：$q = 1-p$。其概率質量函數（PMF） 為：

    $\begin{equation}     \nonumber P_X(x) = \left\{     \begin{array}{l l}     p& \quad \text{for  } x=1\\     1-p & \quad \text{ for } x=0\\     0  & \quad \text{ otherwise }     \end{array} \right. \end{equation}$

```python
from scipy.stats import bernoulli
import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(1, 1)

p = 0.8
x = np.linspace(0,1)

plt.plot(x,bernoulli.pmf(x,p),'o-',label='bernoulli pmf')

plt.title('bernoulli pmf')
plt.xlim(-0.1,1.1)
plt.ylim(0,1)

plt.show()
```

伯努利分佈只有一個參數$p$，記做$X ~ Bernuolli(p)$，或$X ~ B(1,p)$，讀做$X$服從參數為$p$的伯努利分佈。伯努利分佈適合於試驗結果只有兩種可能的單次試驗。例如拋一次硬幣，其結果只有正面或者反面兩種可能；一次產品質量檢測，結果只有合格還是不合格這兩種可能。

```python
import scipy.stats as stats
from scipy.stats import bernoulli
import matplotlib.pyplot as plt

def bernoulli_pmf(p=0.0):

    ber_dist = stats.bernoulli(p)
    x = [0, 1]
    x_name = ['0', '1']
    pmf = [ber_dist.pmf(x[0]), ber_dist.pmf(x[1])]
    plt.bar(x, pmf, width=0.15)
    plt.xticks(x, x_name)
    plt.ylabel('Probability')
    plt.title('PMF of bernoulli distribution')
    plt.show()

bernoulli_pmf(p=0.8)
```

上面兩幅圖都是表示伯努利分佈的 PMF；我們為了得到比較準確的某個服從伯努利分佈的隨機變量的期望，需要大量重複伯努利試驗，例如重複$n$次，然後利用$\frac{正面朝上的次數}{n}$來估計$p$值，當我們重複$n$次以後，這就變成了二項分布，就是下面會提到的**二項分布**。

#### 二项分布（Binomial Distribution）

二項分布是指$n$個獨立的是/非試驗中成功的次數的離散概率分佈，其中每次試驗的成功概率為$p$。這樣的單詞成功/失敗試驗又稱為伯努利試驗。實際上，當$n = 1$時，二項分布就是伯努利分佈。二項分布時顯著性差異的二項試驗的基礎。

```python
from scipy.stats import binom
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 1)

n, p = 5, 0.4
mean, var, skew, kurt = binom.stats(n, p, moments='mvsk')

x = np.arange(binom.ppf(0.01, n, p),
              binom.ppf(0.99, n, p))
ax.plot(x, binom.pmf(x, n, p), 'bo', ms=8, label='binom pmf')
ax.vlines(x, 0, binom.pmf(x, n, p), colors='b', lw=5, alpha=0.5)

rv = binom(n, p)
ax.vlines(x, 0, rv.pmf(x), colors='k', linestyles='-', lw=1,
        label='frozen pmf')
ax.legend(loc='best', frameon=False)
plt.show()

```

```python

```

Notebooks/.ipynb_checkpoints/Use PY in Advanced Statistics -checkpoint.ipynb


# Galvanostatic Intermitten Titration Technique (GITT)

```python
%load_ext autoreload
%autoreload 2
```

```python
import os
from pathlib import Path
from pprint import pprint

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import cellpy
from cellpy import prms
from cellpy import prmreader
from cellpy import cellreader
from cellpy.utils import ocv_rlx
import holoviews as hv

%matplotlib inline
hv.extension('bokeh')
```

```python
######################################################################
##                                                                  ##
##                       development                                ##
##                                                                  ##
######################################################################

if os.name == 'nt':
    # Use these when working on my work PC:
    raw_data_path = r"C:\Scripting\MyFiles\development_cellpy\dev_data\gitt"
    out_data_path = r"C:\Scripting\MyFiles\development_cellpy\out"
else:
    # Use these when working on my MacBook:
    raw_data_path = "/Users/jepe/scripting/cellpy/dev_data/gitt"
    out_data_path = "/Users/jepe/scripting/cellpy/dev_data/out"

raw_data_path = Path(raw_data_path)
out_data_path = Path(out_data_path)

print(" SETTING SOME PRMS ".center(80, "="))
prms.Paths["db_filename"] = "cellpy_db.xlsx"
prms.Paths["cellpydatadir"] = out_data_path
prms.Paths["outdatadir"] = out_data_path
prms.Paths["rawdatadir"] = raw_data_path
prms.Paths["db_path"] = out_data_path
prms.Paths["filelogdir"] = out_data_path
pprint(prms.Paths)

pd.set_option('display.max_rows', 30)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
```

```python
fn = "20190403_cen59_04_rateGITT_01"
mass = 0.3 # mg
resn = fn + ".res"
cellpyn = fn + ".h5"
filename = prms.Paths["rawdatadir"] / resn
cellpyname = prms.Paths["cellpydatadir"] / cellpyn
```

```python
cell = cellreader.get(filename, mass=mass, logging_mode="INFO")
cell.save(cellpyname)
cell = cellreader.get(cellpyname)
```

```python
# print(cell)
```

## Some functions

### Custom step-table for GITT
Need to allow for several ocv steps with same step number in the same cycle. This is not calculated by default and we therefore have to (re-) run the `make_step_table` method selecting
`all_steps=True`.

```python
steptable = cell.make_step_table(all_steps=True).cell.steps
```

#### Checking

```python
steptable.loc[steptable.cycle==5, ["index", "step", "ustep", "type"]].head(4)
```

## Initial exploration

### Overview of the full test

#### Curve with labels

```python
# retrieving the voltage vs time curve
t, v = cell.get_timestamp(), cell.get_voltage()
all_cycs = hv.Curve((t,v), ('t', 'time (sec)'), ('v', 'voltage (v)'), label="voltage-time").opts()
```

```python
# creating labels
cycle_label_df = steptable.drop_duplicates("cycle")
cycle_label_df["label"] = cycle_label_df.cycle.apply(str)
cycle_labels = hv.Labels((cycle_label_df.test_time_first, cycle_label_df.voltage_first, cycle_label_df.label)).opts(
    text_align = "left",
)
```

#### Holomap (with selector)

```python
# creating a dictionary of curves (for each cycle)
cycs_dict = dict()
for c in cell.get_cycle_numbers():
    t = cell.get_timestamp(cycle=c)
    t = t - np.amin(t.values)  # setting first point to t=0
    curve = hv.Curve((t, cell.get_voltage(cycle=c)), ("time", "time (seconds)"), ("voltage", "voltage (v vs. Li/Li+)"))
    cycs_dict[c] = curve
```

```python
# creating a holomap object
hmap = hv.HoloMap(cycs_dict,"cycle")
```

### Curve and HoloMap

```python
%%opts Curve [width=800, xformatter="%6.0f" tools=["hover"]]
(all_cycs * cycle_labels + hmap).cols(1)
```

## Processing a cycle

As we have seen, only some of the cycles contains the GITT curves. We need to select one of those. Lets start with cycle 5

```python
dfdata = cell.cell.raw
```

```python
cyc5 = dfdata.loc[dfdata.Cycle_Index == 5, :]
gt5 = steptable.loc[steptable.cycle==5, :]
```

```python
cyc5curve = hv.Curve(cyc5,  ("Test_Time", "time"), ("Voltage", "voltage")).opts(color="grey", alpha=0.5)
cyc5points = hv.Scatter(cyc5,  ("Test_Time", "time"), ("Voltage", "voltage")).opts(size=5, fill_alpha=0.2)
```

```python
slabels5 = hv.Labels((gt5.test_time_first, gt5.voltage_first, gt5.ustep.apply(str))).opts(text_align="left")
spikes5 = hv.Spikes(gt5.test_time_first).opts(
    line_alpha=0.3,
    spike_length=1,
)
```

```python
%%opts Curve [width=1000, height=600, tools=['hover']]
print("the cycle starts where the label is")
cyc5curve * cyc5points * slabels5 * spikes5
```

### Some calculations are needed...

```python
gt5.head()
```

```python
# approximating electrode-electrolyte contact area

diameter_si = 200.0 * 10e-7  # cm
rho_si = 2.32  # g/cm3
_mass = mass/1000
area = (2*3/diameter_si) * (_mass/rho_si) # CHECK THIS

print(f"Diameter: {diameter_si:8.6f} cm")
print(f"Mass:     {mass:8.6f} mg")
print(f"Calculated contact area (ideal case): {area:8.6f} cm2")

# For reference (v, a, m is pr Si particle)
v = (4/3)*np.pi*((diameter_si/2)**3)
m = rho_si * v
n = mass / m
a = 4 * np.pi * (diameter_si/2)**2
```

```python
# estimate some values
# electrolyte-electrode area (cm2)
radius = 0.75  # cm
roughness = 1.0 # a dimensionless parameter ranging from 1 to a lot.
print("Should probably use BET to find out this?")
area = (1*roughness) * np.pi * (radius**2)
print(f"S: {area:6.2f} cm2")
print("You should compare this number to what was calculated in the cell above (the theoretical surface area for the set of same sized nanoparticles)")
# number of moles
# 1 mol Si weighs 28.0855 g
_mass = mass / 1000 # convert mass from mg to g
number_of_moles = _mass / 28.0855
print(f"n_m: {number_of_moles:8.6f} mol")

# molar volume
# The 2006 CODATA recommended value for the molar volume of silicon is 12.0588349(11)×10−6 m3/mol, with a relative standard uncertainty of 9.1×10−8
molar_volume = 12.06 # cm3/mol
print(f"V_m: {molar_volume:6.2f} cm3/mol")

```

```python

```

```python

```

The following equation to estimate the Diffusion Coefficients appears in a paper:

\begin{equation}
D = \frac{4}{\pi\cdot\tau} \cdot ( \frac {m \cdot V_m}{M \cdot S})^2 \cdot ( \frac {\Delta E_s}{ \Delta E_t})^2 \: for \: \tau << L^2 / D
\end{equation}

$ \Delta E_s $  = change of steady state voltage at the end of two sequential open-circuit relaxation periods

$ \Delta E_t $ = total change in the cell voltage $ E $ during the current pulse, neglecting the IR drop

$ V_s $ = mole volume of active material (cm^3/mol) (molar mass divided by the materials density)

$ M $ = ?

$ S $ = Surface area of the electrode (cm^2)

$ \tau $ = time of the current pulse

$ L $ = diffustion length

Andrzej P. Nowak *et al.*, Procedia Engineering **98** (2014) 8

```python
# Simple calculations by reading values from the graph
m = mass/1000
molar_mass = 28.0855 # g/mol
density = 2.32  # g/cm3
Vm = molar_mass / density # cm3/mol
S = 1.7 # cm2
t = 2880 # s
M = 1
Es1 = 0.378
Es2 = 0.344
DEs = Es2-Es1

Et11 = 0.378  # starting the current pulse
Et12 = 0.309 # after (possible) IR drop
Et2 = 0.28
DEt1 = Et2 - Et11
DEt2 = Et2 - Et12

p1 = (4 / (np.pi * t)) * (m * Vm / (M * S))**2
p21 = (DEs / DEt1)**2
p22 = (DEs / DEt2)**2

D1 = p1*p21
D2 = p1*p22
print(f"Assuming no IR drop: {D1}\nWith IR drop: {D2}")
print(f"A: {p1}")
print(f"DEt: {DEt1}")
print(f"DEs: {DEs}")

```

Comparing another reference (below) I assume that M is 1/number_of_moles ?

```python

```

### A function to automatically extract diffusion constant(s) from the steptable

This function is based on the Metrohm Autolab document. It will need some adjustments.

```python
def calc_A(n_m=0.0011, V_m=12.06, S=1.77):
    """
    D = 4 /((pi)(tau)) * (n_m * V_m / S)^2 * (DEs / DEt)^2 = A /tau * (DEs / DEt)^2
    A = 4 /(pi) * (n_m * V_m / S)^2

    tau: duration of the current pulse (s)
    n_m: number of moles (mol)
    V_m: molar volume of electrode (cm3/mol)
    S: electrode-electrolyte contact area (cm2)
    DEs: steady state voltage change due to the current pulse (V)
    DEt: voltage change during the constant current pulse (eliminating the iR drop) (V)

    Ref.: application note from Metrohm Autolab b.v. pdf (BAT03)
    """
    A = (4 / np.pi) * (n_m * V_m / S)**2
    return A
```

```python
def auto_calc_D(steptable, cycle_number, A=1.0, tau=None, ustep_first=None, ustep_last=None):
    """Function for extracting diffusion constant(s) and inserting into steptable.

    D = 4 /((pi)(tau)) * (n_m * V_m / S)^2 * (DEs / DEt)^2 = A /tau * (DEs / DEt)^2
    A = 4 /(pi) * (n_m * V_m / S)^2

    OBS! not corrected for IR drop yet.
    """
    st = None
    if cycle_number is None:
        # This function is intended to only work on a pr. cycle basis
        # to prevent users from "poluting" the steptable for "non_GITT" experiments.
        print("no cycle number given")
        return

    st = steptable[steptable.cycle==cycle_number]
    st = st[st.type.isin(["charge", "discharge", "ocvrlx_up", "ocvrlx_down"])]
    if st.empty:
        print("the given cycle is not found")
        return

    if ustep_first is not None:
        st = st[st.ustep>=ustep_first]
    if ustep_last is not None:
        st = st[st.ustep<=ustep_last]

    # used for finding DE
    n3 = st["voltage_last"].shift(periods=-3)
    n2 = st["voltage_last"].shift(periods=-2)
    n1 = st["voltage_last"].shift(periods=-1)
    n0 = st["voltage_last"]
  #  st["n3"] = n3
  #  st["n2"] = st["voltage_last"].shift(periods=-2)
  #  st["n1"] = n1
  #  st["n0"] = st["voltage_last"].shift(periods=0)

    # used for finding tau
    if tau is None:
        tau = st["step_time_last"] - st["step_time_first"]

    # used for validating if proper GITT step
    t0 = st["type"]
    t1 = st["type"].shift(periods=-1)
    t2 = st["type"].shift(periods=-2)
    t3 = st["type"].shift(periods=-3)

    st["valid_D"] = (t0==t2) & (t1.str.contains("ocv"))

    # calculating
    st["DEt"] = st["voltage_last"] - st["voltage_first"] # should remove the IR drop here (maybe calculate it based on the measured IR time the current?)
    st["DEs"] = n3 - n1
    st["D"] = A / tau * (st["DEs"] / st["DEt"])**2

    return st
```

### Using the function to extract Diffusion constants vs capacity

```python
# Lets just run it and see what happens
A = calc_A(n_m=0.0011, V_m=12.06, S=1.77)
print(f"A: {A}")
x = auto_calc_D(steptable, 5, A=A, ustep_first=27)
discharge = x.loc[(x.type=="discharge") & (x.valid_D)]
charge = x.loc[(x.type=="charge") & (x.valid_D)]
# discharge.head(25)
```

```python
%%opts Curve [width=600 xformatter="%6.0f" tools=["hover"]]
X = discharge["discharge_avr"]*1_000_000/mass
Y = discharge["D"]
discharge_diffcurve = hv.Scatter((X,Y), ("capacity"), ("diffusion coef. (cm2 s-1)"), label="discharge").opts(size=12) * hv.Curve((X,Y)).opts(alpha=0.4)

X = charge["charge_avr"]*1_000_000/mass
Y = charge["D"]
charge_diffcurve = hv.Scatter((X,Y), ("capacity"), ("diffusion coef. (cm2 s-1)"), label="charge").opts(size=12) * hv.Curve((X,Y)).opts(alpha=0.4)

discharge_diffcurve * charge_diffcurve
```

### TODO
- make a plot where we show each point that is used in the calculations
- tweak the method so that we get realistic results
- find a paper to compare with

```python

```

## Picking out titration curves

```python
cycle = 5
titcurve_ustep = 28

# filtering wrt cycle number to get the Data_Points for the step

datapoints = x.loc[(x.type=="discharge") & (x.ustep==titcurve_ustep),  ["point_first", "point_last"]]
datapoints
```

```python
# Using the data points to select the data from dfdata
first = datapoints.iloc[0, 0]
last = datapoints.iloc[0, 1]
dftit = dfdata.loc[first:last, :]
dftit.head()
```

```python
dfcurve = hv.Curve(dftit,  ("Step_Time", "time"), ("Voltage", "voltage")).opts(color="grey", alpha=0.5).opts(width=1000, xformatter="%6.0f")
min_time = hv.VLine(100.0)
max_time = hv.VLine(1000.0)
a = hv.Arrow(x=500, y = 0.31, text="fit", direction= "v")
dfcurve * min_time * max_time * a
```

```python
# plotting dE vs sqrt(t) and doing a linear regression to find the slope
dE = dftit["Voltage"]
t = np.sqrt(dftit["Step_Time"])
```

```python
decurve = hv.Scatter((t, dE), "sqrt-time", "voltage").opts(width=1000, xformatter="%6.0f")
decurve
```

# Temporary stuff (will probably be deleted)

```python

```

### Methodology for finding diff coeffs
#### Equation
```
Eq. 1.1.
D = (4 / pi) (i*V_m / (Z_A*F*S)^2 * ((dE/dd)/(dE/d(sqrt(t)))^2

i: current (A)
Z_A: charge number
F: Faraday´s constant (96_458 C/mol)
V_m: molar volume of electrode (cm3/mol)
S: electrode-electrolyte contact area (cm2)

(dE/dd): the slope of the coulometric titration curve, found by plotting the
steady state voltages E (V) measured after each titration step δ

steady state voltage change due to the current pulse (V)
(dE/d(sqrt(t)):  the slope of the linearized plot of the potential E (V) during the current pulse of duration t (s).
```

#### step 1
Need to find the slope of E vs δ. This is the same as finding dV/dQ. One way to do this is to fit the "relaxed" voltage curve (the OCV_RLX points) to a polynomial and diffing it.
#### step 2
Then we also need to find the slope of E vs time during the pulse. For this we can e.g. use the polytfit function from scipy. The first 100 seconds should not be used as they
are typically from charge-transfere resistance and the concentration overpotential.
#### step 3
Now it is time to get all the other prms (where the most difficult one is the electrode-electrolyte contact area). It should ideally be found using BET. But I guess the best is to calculate it using a sensible model, or from some scattering experiments (SANS?).

```python

```

## Example
Based on application note from Metrohm Autolab b.v. pdf (BAT03)

```
Eq. 1.1.
D = (4 / pi) (i*V_m / (Z_A*F*S)^2 * ((dE/dd)/(dE/d(sqrt(t)))^2

i: current (A)
Z_A: charge number
F: Faraday´s constant (96_458 C/mol)
V_m: molar volume of electrode (cm3/mol)
S: electrode-electrolyte contact area (cm2)

(dE/dd): the slope of the coulometric titration curve, found by plotting the
steady state voltages E (V) measured after each titration step δ

steady state voltage change due to the current pulse (V)
(dE/d(sqrt(t)):  the slope of the linearized plot of the potential E (V) during the current pulse of duration t (s).
```

```
Eq. 1.2.
D = 4 /((pi)(tau)) * (n_m * V_m / S)^2 * (DEs / DEt)^2

tau: duration of the current pulse (s)
n_m: number of moles (mol)
V_m: molar volume of electrode (cm3/mol)
S: electrode-electrolyte contact area (cm2)
DEs: steady state voltage change due to the current pulse (V)
DEt: voltage change during the constant current pulse (eliminating the iR drop) (V)
```

```python

```

### Example code:

```python

D = (A / tau) * (DEs / DEt)**2
# during charge
DEt = ustep_52.voltage[-1] - ustep_52.voltage[0]  # charge step
DEs = ustep_55.voltage[-1] - ustep_53.voltage[0]  # ocv steps
```

```python
# during discharge
DEt = ustep_27.voltage[-1] - ustep_27.voltage[0]  # discharge step
DEs = ustep_30.voltage[-1] - ustep_28.voltage[0]  # ocv steps
```

Using step-table

```python
DEt = ustep[n].voltage_last - ustep[n].voltage_first  # charge step
DEs = ustep[n+3].voltage_last - ustep[n+1].voltage_first # ocv steps
```


# Ikeda eddy regression

# Purpose
It has proven to be very hard to reproduce the results from . This notebook will develop a regression to correct the implementation.

# Methodology
Regression will be made to the $C_r$ coefficient.

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

#from jupyterthemes import jtplot
#jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify, run
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables
import shipflowmotionshelpers.shipflowmotionshelpers as helpers
import src.visualization.visualize as visualize

```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
import rolldecayestimators.equations_ikeda_naked as equations_ikeda_naked
import rolldecayestimators.ikeda_naked as ikeda_naked
from src.helpers import calculate_lewis,calculate_B_star_hat,estimate_bilge_radius, estimate_sigma
import seaborn as sns
import joblib
```

```python
df_ikeda_sections = pd.read_csv('../data/interim/ikeda_sections.csv', index_col=0)

df_ikeda_sections.rename(columns= {
    'B_e_star_hat':'B_E_star_hat',
    'd':'T',
    'w_hat':'omega_hat',
    'L':'L_pp',
    'B':'beam',
    'volume':'Disp',
}, inplace=True)

df_ikeda_sections.head()
```

      phi_a
      B_star_hat
      omega_hat
      model
      B_W+B_F
      B_E_star_hat
      L_pp
      beam
      T
      Disp
      H0
      sigma
      OG/d
      R

      0
      0.23
      0.041169
      0.751
      A
      0.003636
      0.037534
      0.8
      0.28
      0.112
      0.02509
      1.25
      1.000
      0.0
      0.00

      1
      0.24
      0.029803
      0.507
      A
      0.001391
      0.028413
      0.8
      0.28
      0.112
      0.02509
      1.25
      1.000
      0.0
      0.00

      2
      0.30
      0.023966
      0.751
      B
      0.003265
      0.020700
      0.8
      0.28
      0.112
      0.02501
      1.25
      0.997
      0.0
      0.01

      3
      0.30
      0.014438
      0.536
      B
      0.001111
      0.013328
      0.8
      0.28
      0.112
      0.02501
      1.25
      0.997
      0.0
      0.01

      4
      0.27
      0.008265
      0.751
      C
      0.002115
      0.006151
      0.8
      0.28
      0.112
      0.02495
      1.25
      0.995
      0.0
      0.02

```python
#import functools
#def create_wrapped(expr):
#    func = lambdify(expr)
#    @functools.wraps(func)
#    def wrapped(*_args, **kwargs):
#        # Write the logic here to parse _args and kwargs for the arguments as you want them
#
#
#        return func(*actual_args)
#    return wrapped
```

```python
solution = sp.solve(equations_ikeda_naked.eq_B_E_star_hat, symbols.B_E0_hat)[0]
lambda_B0_hat = lambdify(solution)

equations_ikeda_naked.eq_B_E_star_hat
```

$\displaystyle B_{E0 HAT} = \frac{8 B_{E star hat}}{3 \pi}$

```python
df_ikeda_sections['B_E0_HAT'] = run(lambda_B0_hat, df_ikeda_sections)
```

```python
solution = sp.solve(equations_ikeda_naked.eq_B_E0_hat, symbols.C_r)[0]
lambda_C_r = lambdify(solution)
eq_C_r = sp.Eq(symbols.C_r,solution)
eq_C_r
```

$\displaystyle C_{r} = \frac{3 \pi B_{E0 HAT} Disp beam^{2}}{4 L_{pp} T^{4} \omega_{hat} \phi_{a}}$

```python
df_input = df_ikeda_sections.copy()
df_ikeda_sections['C_r'] = run(lambda_C_r,df_ikeda_sections)
```

## Calculate Lewis coefficients

```python
df_ = df_ikeda_sections.copy()
df_['B'] = df_['beam']
df_['d'] = df_['T']
a, a_1, a_3, sigma_s, H = calculate_lewis(df_)
df_ikeda_sections['a_1'] = a_1
df_ikeda_sections['a_3'] = a_3
df_ikeda_sections['H0'] = H
```

```python
df_ikeda_sections.to_csv('../data/processed/ikeda_sections.csv')
```

## Estimate bilge radius

```python
df_ = df_ikeda_sections.copy()
df_['B'] = df_['beam']
df_['d'] = df_['T']
mask = df_ikeda_sections['R'].isnull()
df_['B'] = df_['beam']
df_ikeda_sections.loc[mask,'R'] = estimate_bilge_radius(**df_.loc[mask])
```

```python
df_ikeda_sections.to_csv('../data/processed/ikeda_sections_R.csv')
```

```python
df_ikeda_sections
```

      phi_a
      B_star_hat
      omega_hat
      model
      B_W+B_F
      B_E_star_hat
      L_pp
      beam
      T
      Disp
      H0
      sigma
      OG/d
      R
      B_E0_HAT
      C_r
      a_1
      a_3

      0
      0.23
      0.041169
      0.751
      A
      0.003636
      0.037534
      0.8
      0.280
      0.112
      0.02509
      1.250000
      1.000
      0.0
      0.000000
      0.031860
      6.791066
      0.095721
      -0.138514

      1
      0.24
      0.029803
      0.507
      A
      0.001391
      0.028413
      0.8
      0.280
      0.112
      0.02509
      1.250000
      1.000
      0.0
      0.000000
      0.024117
      7.297515
      0.095721
      -0.138514

      2
      0.30
      0.023966
      0.751
      B
      0.003265
      0.020700
      0.8
      0.280
      0.112
      0.02501
      1.250000
      0.997
      0.0
      0.010000
      0.017571
      2.862276
      0.095950
      -0.136454

      3
      0.30
      0.014438
      0.536
      B
      0.001111
      0.013328
      0.8
      0.280
      0.112
      0.02501
      1.250000
      0.997
      0.0
      0.010000
      0.011313
      2.582057
      0.095950
      -0.136454

      4
      0.27
      0.008265
      0.751
      C
      0.002115
      0.006151
      0.8
      0.280
      0.112
      0.02495
      1.250000
      0.995
      0.0
      0.020000
      0.005221
      0.942686
      0.096102
      -0.135084

      5
      0.21
      0.005487
      0.625
      C
      0.001500
      0.003987
      0.8
      0.280
      0.112
      0.02495
      1.250000
      0.995
      0.0
      0.020000
      0.003384
      0.943989
      0.096102
      -0.135084

      6
      0.31
      0.009504
      0.939
      D
      0.002039
      0.007466
      0.8
      0.280
      0.112
      0.02475
      1.250000
      0.988
      0.0
      0.030000
      0.006337
      0.790684
      0.096632
      -0.130314

      7
      0.31
      0.007645
      0.751
      D
      0.001500
      0.006145
      0.8
      0.280
      0.112
      0.02475
      1.250000
      0.988
      0.0
      0.030000
      0.005216
      0.813690
      0.096632
      -0.130314

      8
      0.24
      0.099861
      0.814
      G
      0.088356
      0.011505
      0.8
      0.185
      0.192
      0.02321
      0.481771
      0.799
      0.0
      0.182397
      0.009766
      0.086061
      -0.347078
      -0.007600

      9
      0.30
      0.002040
      0.560
      H
      0.001400
      0.000640
      0.8
      0.398
      0.193
      0.05490
      1.031088
      0.893
      0.0
      0.195702
      0.000543
      0.059677
      0.014252
      -0.068862

      10
      0.30
      0.003414
      0.889
      H
      0.002290
      0.001124
      0.8
      0.398
      0.193
      0.05490
      1.031088
      0.893
      0.0
      0.195702
      0.000954
      0.066051
      0.014252
      -0.068862

      11
      0.25
      0.002769
      0.580
      I
      0.001000
      0.001769
      0.8
      0.237
      0.096
      0.01775
      1.234375
      0.977
      0.0
      0.049381
      0.001502
      0.358037
      0.091987
      -0.123059

      12
      0.25
      0.004455
      0.861
      I
      0.002505
      0.001950
      0.8
      0.237
      0.096
      0.01775
      1.234375
      0.977
      0.0
      0.049381
      0.001655
      0.265878
      0.091987
      -0.123059

      13
      0.23
      0.007458
      1.030
      J
      0.002762
      0.004697
      0.8
      0.343
      0.192
      0.03163
      0.893229
      0.593
      0.0
      0.353409
      0.003987
      0.135721
      -0.063366
      0.123590

      14
      0.29
      0.018963
      0.780
      K
      0.001304
      0.017660
      1.0
      0.193
      0.125
      0.01228
      0.772000
      0.541
      0.0
      0.227156
      0.014990
      0.292548
      -0.148721
      0.155846

      15
      0.26
      0.163100
      0.551
      L
      0.028802
      0.134298
      1.0
      0.151
      0.144
      0.00933
      0.524306
      0.430
      0.0
      0.240320
      0.113995
      0.927596
      -0.377793
      0.210592

```python
data = df_ikeda_sections.copy()
data.drop(columns=['B_star_hat','B_E_star_hat','B_E0_HAT','B_W+B_F','Disp','L_pp','beam','T'],
          inplace=True)
data.head()
```

      phi_a
      omega_hat
      model
      H0
      sigma
      OG/d
      R
      C_r
      a_1
      a_3

      0
      0.23
      0.751
      A
      1.25
      1.000
      0.0
      0.00
      6.791066
      0.095721
      -0.138514

      1
      0.24
      0.507
      A
      1.25
      1.000
      0.0
      0.00
      7.297515
      0.095721
      -0.138514

      2
      0.30
      0.751
      B
      1.25
      0.997
      0.0
      0.01
      2.862276
      0.095950
      -0.136454

      3
      0.30
      0.536
      B
      1.25
      0.997
      0.0
      0.01
      2.582057
      0.095950
      -0.136454

      4
      0.27
      0.751
      C
      1.25
      0.995
      0.0
      0.02
      0.942686
      0.096102
      -0.135084

```python
data.drop(columns=['OG/d'], inplace=True)  # always 0
```

```python
sns.pairplot(data = data)
```

```python
x_vars = list(data.columns)
x_vars.remove('C_r')

grid = sns.pairplot(data = data, x_vars=x_vars, y_vars=['C_r'], height=3)
grid.fig.set_size_inches(12,3)
```

```python
data.plot(x='sigma', y='a_3', style='.')
```

```python
data.plot(x='sigma', y='C_r', style='.')
```

```python
data.plot(x='a_3', y='C_r', style='.')
```

```python
features = list(data.columns)
features
```

    ['phi_a', 'omega_hat', 'model', 'H0', 'sigma', 'R', 'C_r', 'a_1', 'a_3']

```python
data_ = data.select_dtypes(exclude='object')
y = data_.pop('C_r')
X = data_.copy()
feature_names = X.columns
```

```python
from sklearn.feature_selection import RFE
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeRegressor
from sklearn import tree

# create pipeline
rfe = RFE(estimator=DecisionTreeRegressor(), n_features_to_select=3)
descision_tree = DecisionTreeRegressor(max_depth=4, random_state=42)
pipeline = Pipeline(steps=[('s',rfe),('m',descision_tree)])
model = pipeline.fit(X,y)
```

```python

# Fit regression model
descision_tree = DecisionTreeRegressor(max_depth=3)
descision_tree.fit(X, y)

df_trees = pd.DataFrame()
df_trees['max_depth'] = np.arange(1,len(X))
df_trees.set_index('max_depth', inplace=True)

for max_depth, row in df_trees.iterrows():
    descision_tree = DecisionTreeRegressor(max_depth=max_depth,max_features=3, random_state=42)
    descision_tree.fit(X, y)
    df_trees.loc[max_depth,'score'] = descision_tree.score(X, y)

fig,ax=plt.subplots()
df_trees.loc[2:].plot(y='score', ax=ax)
```

```python
from sklearn.tree import DecisionTreeRegressor
from sklearn import tree

# Fit regression model
descision_tree = DecisionTreeRegressor(max_depth=3)
descision_tree.fit(X, y)

df_trees = pd.DataFrame()
df_trees['n_features'] = np.arange(1,len(X))
df_trees.set_index('n_features', inplace=True)

for n_features, row in df_trees.iterrows():

    descision_tree = DecisionTreeRegressor(max_depth=n_features, random_state=42)
    rfe = RFE(estimator=descision_tree, n_features_to_select=2)

    pipeline = Pipeline(steps=[('s',rfe),('m',descision_tree)])
    model = pipeline.fit(X,y)

    df_trees.loc[n_features,'score'] = model.score(X, y)

fig,ax=plt.subplots()
df_trees.loc[2:].plot(y='score', ax=ax)
```

```python
descision_tree = DecisionTreeRegressor(max_depth=4, random_state=42)
rfe = RFE(estimator=descision_tree, n_features_to_select=2)
rfe.fit(X,y)
good_feature_names = feature_names[rfe.get_support()]
X_good = X[good_feature_names].copy()
descision_tree.fit(X_good,y)
descision_tree.score(X_good,y)
```

    0.9967264383051466

```python
good_feature_names
```

    Index(['sigma', 'a_3'], dtype='object')

```python
y_pred = descision_tree.predict(X_good)
data_pred = df_ikeda_sections.copy()
data_pred['C_r'] = y_pred
```

```python
fig,ax=plt.subplots()
data_pred.plot(x='a_3', y='C_r', style='o', ax=ax)
data.plot(x='a_3', y='C_r', style='.', ax=ax)

```

```python
fig,ax = plt.subplots()
fig.set_size_inches(15,8)
tree.plot_tree(descision_tree, ax=ax, feature_names=good_feature_names, rounded=True);
```

## Go back to $\hat{B_E*}$

```python
solution = sp.solve(equations_ikeda_naked.eq_B_E0_hat, symbols.B_E0_hat)[0]
lambda_B_E0_hat = lambdify(solution)

equations_ikeda_naked.eq_B_E0_hat
```

$\displaystyle B_{E0 HAT} = \frac{4 C_{r} L_{pp} T^{4} \omega_{hat} \phi_{a}}{3 \pi Disp beam^{2}}$

```python
solution = sp.solve(equations_ikeda_naked.eq_B_E_star_hat, symbols.B_E_star_hat)[0]
eq_B_E_star_hat = sp.Eq(symbols.B_E_star_hat, solution)
lambda_B_E_star_hat = lambdify(solution)
eq_B_E_star_hat
```

$\displaystyle B_{E star hat} = \frac{3 \pi B_{E0 HAT}}{8}$

```python
data_pred['B_E0_HAT'] = run(lambda_B_E0_hat, data_pred)
```

```python
data_pred['B_E_star_hat'] = run(lambda_B_E_star_hat, data_pred)
```

```python
data_pred.head()
```

      phi_a
      B_star_hat
      omega_hat
      model
      B_W+B_F
      B_E_star_hat
      L_pp
      beam
      T
      Disp
      H0
      sigma
      OG/d
      R
      B_E0_HAT
      C_r
      a_1
      a_3

      0
      0.23
      0.041169
      0.751
      A
      0.003636
      0.038933
      0.8
      0.28
      0.112
      0.02509
      1.25
      1.000
      0.0
      0.00
      0.033048
      7.044290
      0.095721
      -0.138514

      1
      0.24
      0.029803
      0.507
      A
      0.001391
      0.027427
      0.8
      0.28
      0.112
      0.02509
      1.25
      1.000
      0.0
      0.00
      0.023280
      7.044290
      0.095721
      -0.138514

      2
      0.30
      0.023966
      0.751
      B
      0.003265
      0.019687
      0.8
      0.28
      0.112
      0.02501
      1.25
      0.997
      0.0
      0.01
      0.016711
      2.722167
      0.095950
      -0.136454

      3
      0.30
      0.014438
      0.536
      B
      0.001111
      0.014051
      0.8
      0.28
      0.112
      0.02501
      1.25
      0.997
      0.0
      0.01
      0.011927
      2.722167
      0.095950
      -0.136454

      4
      0.27
      0.008265
      0.751
      C
      0.002115
      0.006155
      0.8
      0.28
      0.112
      0.02495
      1.25
      0.995
      0.0
      0.02
      0.005224
      0.943338
      0.096102
      -0.135084

```python
len(df_ikeda_sections.groupby(by='model'))
```

    10

```python
def plot():

    data_pred_groups = data_pred.groupby(by=['model','omega_hat'])

    y_key = 'B_E_star_hat'
    y_max = np.max([df_ikeda_sections[y_key].max(), data_pred[y_key].max()])

    model_groups = df_ikeda_sections.groupby(by='model')
    fig,axes=plt.subplots(ncols=3, nrows=4)
    axes=axes.flatten()[0:len(model_groups)]

    for (model_name, ikeda_data),ax in zip(model_groups,axes):

        ax.set_title(model_name)

        for w_hat, df_ in ikeda_data.groupby(by='omega_hat'):

            label = '$\hat{\omega}=%0.3f$' % w_hat
            x = df_['phi_a']
            y = df_[y_key]
            lines = ax.plot([0,x], [0,y], 'o', label=label)
            line = lines[-1]
            color = line.get_color()

            df_pred = data_pred_groups.get_group((model_name,w_hat))

            label = 'prediction'
            x = df_pred['phi_a']
            y = df_pred[y_key]
            lines = ax.plot([0,x], [0,y], '--', color=color, label=label)

        ax.set_ylim((0,y_max))
        ax.legend()

```

```python
plot()
```

## KVLCC2

```python
df_kvlcc2 = pd.read_csv('../data/interim/kvlcc_areas.csv', sep=';', index_col=0)
df_kvlcc2.head()
```

      area
      x
      t
      b
      r_b

      no

      0
      13.826612
      -5.495000
      2.00
      11.638577
      6.636080

      1
      123.851306
      10.159932
      18.25
      27.893522
      42.367175

      2
      428.211409
      28.051284
      20.80
      41.824284
      45.369454

      3
      683.709165
      43.706216
      20.80
      50.282514
      41.080696

      4
      917.895066
      61.597568
      20.80
      56.159232
      34.146143

```python
df_kvlcc2.rename(columns={
    'b':'beam',
    't':'T',
    'r_b':'R',
}, inplace=True)
df_kvlcc2['sigma']=df_kvlcc2.area/(df_kvlcc2.beam*df_kvlcc2['T'])
df_kvlcc2['OG/d']=0  # For now...
```

```python
df_ = df_kvlcc2.copy()
df_.rename(columns={
    'beam':'B',
    'T':'d',
}, inplace=True)

a, a_1, a_3, sigma_s, H = calculate_lewis(df_)
df_kvlcc2['a_1'] = a_1
df_kvlcc2['a_3'] = a_3
df_kvlcc2['H0'] = H
```

```python
df_kvlcc2_ = df_kvlcc2.copy()
OG = df_kvlcc2_['OG/d']*df_kvlcc2_['T']
ra = 1000
df_kvlcc2_['C_r'] = ikeda_naked.calculate_C_r(bwl=df_kvlcc2_.beam,
                          a_1=df_kvlcc2_.a_1, a_3=df_kvlcc2_.a_3, sigma=df_kvlcc2_.sigma,
                                              H0=df_kvlcc2_.H0, d=df_kvlcc2_['T'], OG=OG,
                          R=df_kvlcc2_.R, ra=ra)
```

```python
df_kvlcc2['C_r'] = descision_tree.predict(X=df_kvlcc2[good_feature_names])

fig,ax=plt.subplots()
df_kvlcc2.plot(y='C_r', label='model', ax=ax)
df_kvlcc2_.plot(y='C_r', label='ikeda', ax=ax)
```

```python
good_feature_names
```

    Index(['sigma', 'a_3'], dtype='object')

```python
joblib.dump(descision_tree, '../models/C_r_tree.pkl')
```

    ['../models/C_r_tree.pkl']

```python

```