##### Copyright 2018 The TensorFlow Authors.

Licensed under the Apache License, Version 2.0 (the "License");

```
#@title Licensed under the Apache License, Version 2.0 (the "License"); { display-mode: "form" }
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Fitting Generalized Linear Mixed-effects Models Using Variational Inference

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

```
#@title Install { display-mode: "form" }
TF_Installation = 'System' #@param ['TF Nightly', 'TF Stable', 'System']

if TF_Installation == 'TF Nightly':
  !pip install -q --upgrade tf-nightly
  print('Installation of `tf-nightly` complete.')
elif TF_Installation == 'TF Stable':
  !pip install -q --upgrade tensorflow
  print('Installation of `tensorflow` complete.')
elif TF_Installation == 'System':
  pass
else:
  raise ValueError('Selection Error: Please select a valid '
                   'installation option.')
```

```
#@title Install { display-mode: "form" }
TFP_Installation = "System" #@param ["Nightly", "Stable", "System"]

if TFP_Installation == "Nightly":
  !pip install -q tfp-nightly
  print("Installation of `tfp-nightly` complete.")
elif TFP_Installation == "Stable":
  !pip install -q --upgrade tensorflow-probability
  print("Installation of `tensorflow-probability` complete.")
elif TFP_Installation == "System":
  pass
else:
  raise ValueError("Selection Error: Please select a valid "
                   "installation option.")
```

## Abstract

In this colab we demonstrate how to fit a generalized linear mixed-effects model using variational inference in TensorFlow Probability.

## Model Family

[Generalized linear mixed-effect models](https://en.wikipedia.org/wiki/Generalized_linear_mixed_model) (GLMM) are similar to [generalized linear models](https://en.wikipedia.org/wiki/Generalized_linear_model) (GLM) except that they incorporate a sample specific noise into the predicted linear response.  This is useful in part because it allows rarely seen features to share information with more commonly seen features.

As a generative process, a Generalized Linear Mixed-effects Model (GLMM) is characterized by:

$$
\begin{align}
\text{for } & r = 1\ldots R:  \hspace{2.45cm}\text{# for each random-effect group}\\
 &\begin{aligned}
  \text{for } &c = 1\ldots |C_r|:  \hspace{1.3cm}\text{# for each category ("level") of group $r$}\\
  &\begin{aligned}
    \beta_{rc}
    &\sim \text{MultivariateNormal}(\text{loc}=0_{D_r}, \text{scale}=\Sigma_r^{1/2})
  \end{aligned}
\end{aligned}\\\\
\text{for } & i = 1 \ldots N:  \hspace{2.45cm}\text{# for each sample}\\
&\begin{aligned}
  &\eta_i = \underbrace{\vphantom{\sum_{r=1}^R}x_i^\top\omega}_\text{fixed-effects} + \underbrace{\sum_{r=1}^R z_{r,i}^\top \beta_{r,C_r(i) }}_\text{random-effects} \\
  &Y_i|x_i,\omega,\{z_{r,i} , \beta_r\}_{r=1}^R \sim \text{Distribution}(\text{mean}= g^{-1}(\eta_i))
\end{aligned}
\end{align}
$$

where:

$$
\begin{align}
R &= \text{number of random-effect groups}\\
|C_r| &= \text{number of categories for group $r$}\\
N &= \text{number of training samples}\\
x_i,\omega &\in \mathbb{R}^{D_0}\\
D_0 &= \text{number of fixed-effects}\\
C_r(i) &= \text{category (under group $r$) of the $i$th sample}\\
z_{r,i} &\in \mathbb{R}^{D_r}\\
D_r &= \text{number of random-effects associated with group $r$}\\
\Sigma_{r} &\in \{S\in\mathbb{R}^{D_r \times D_r} : S \succ 0 \}\\
\eta_i\mapsto g^{-1}(\eta_i) &= \mu_i, \text{inverse link function}\\
\text{Distribution} &=\text{some distribution parameterizable solely by its mean}
\end{align}
$$

In other words, this says that every category of each group is associated with a sample, $\beta_{rc}$,  from a multivariate normal. Although the $\beta_{rc}$ draws are always independent, they are only indentically distributed for a group $r$: notice there is exactly one $\Sigma_r$ for each $r\in\{1,\ldots,R\}$.

When affinely combined with a sample's group's features ($z_{r,i}$), the result is sample-specific noise on the $i$-th predicted linear response (which is otherwise $x_i^\top\omega$).

When we estimate $\{\Sigma_r:r\in\{1,\ldots,R\}\}$ we're essentially estimating the amount of noise a random-effect group carries which would otherwise drown out the signal present in $x_i^\top\omega$.

There are a variety of options for the $\text{Distribution}$ and [inverse link function](https://en.wikipedia.org/wiki/Generalized_linear_model#Link_function), $g^{-1}$. Common choices are:
- $Y_i\sim\text{Normal}(\text{mean}=\eta_i, \text{scale}=\sigma)$,
- $Y_i\sim\text{Binomial}(\text{mean}=n_i \cdot \text{sigmoid}(\eta_i), \text{total_count}=n_i)$, and,
- $Y_i\sim\text{Poisson}(\text{mean}=\exp(\eta_i))$.

For more possibilities, see the [`tfp.glm`](https://github.com/tensorflow/probability/tree/master/tensorflow_probability/python/glm) module.

## Variational Inference

Unfortunately, finding the maximum likelihood estimates of the parameters $\beta,\{\Sigma_r\}_r^R$ entails a non-analytical integral. To circumvent this problem, we instead
1. Define a parameterized family of distributions (the "surrogate density"), denoted $q_{\lambda}$ in the appendix.
2. Find parameters $\lambda$ so that $q_{\lambda}$ is close to our true target denstiy.

The family of distributions will be independent Gaussians of the proper dimensions, and by "close to our target density", we will mean "minimizing the [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)". See, for example [Section 2.2 of "Variational Inference: A Review for Statisticians"](https://arxiv.org/abs/1601.00670) for a well-written derivation and motivation. In particular, it shows that minimizing the K-L divergence is equivalent to minimizing the negative evidence lower bound (ELBO).

## Toy Problem

[Gelman et al.'s (2007) "radon dataset"](http://www.stat.columbia.edu/~gelman/arm/) is a dataset sometimes used to demonstrate approaches for regression. (E.g., this closely related [PyMC3 blog post](http://twiecki.github.io/blog/2014/03/17/bayesian-glms-3/).) The radon dataset contains indoor measurements of Radon taken throughout the United States. [Radon](https://en.wikipedia.org/wiki/Radon) is naturally ocurring radioactive gas which is [toxic](http://www.radon.com/radon_facts/) in high concentrations.

For our demonstration, let's suppose we're interested in validating the hypothesis that Radon levels are higher in households containing a basement. We also suspect Radon concentration is related to soil-type, i.e., geography matters.

To frame this as an ML problem, we'll try to predict log-radon levels based on a linear function of the floor on which the reading was taken.  We'll also use the county as a random-effect and in so doing account for variances due to geography. In other words, we'll use a [generalized linear mixed-effect model](https://en.wikipedia.org/wiki/Generalized_linear_mixed_model).

```
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os
from six.moves import urllib

import matplotlib.pyplot as plt; plt.style.use('ggplot')
import numpy as np
import pandas as pd
import seaborn as sns; sns.set_context('notebook')
import tensorflow_datasets as tfds

import tensorflow.compat.v2 as tf
tf.enable_v2_behavior()

import tensorflow_probability as tfp
tfd = tfp.distributions
tfb = tfp.bijectors
```

We will also do a quick check for availablility of a GPU:

```
if tf.test.gpu_device_name() != '/device:GPU:0':
  print("We'll just use the CPU for this run.")
else:
  print('Huzzah! Found GPU: {}'.format(tf.test.gpu_device_name()))
```

    We'll just use the CPU for this run.

### Obtain Dataset:

We load the dataset from TensorFlow datasets and do some light preprocessing.

```
def load_and_preprocess_radon_dataset(state='MN'):
  """Load the Radon dataset from TensorFlow Datasets and preprocess it.

  Following the examples in "Bayesian Data Analysis" (Gelman, 2007), we filter
  to Minnesota data and preprocess to obtain the following features:
  - `county`: Name of county in which the measurement was taken.
  - `floor`: Floor of house (0 for basement, 1 for first floor) on which the
    measurement was taken.

  The target variable is `log_radon`, the log of the Radon measurement in the
  house.
  """
  ds = tfds.load('radon', split='train')
  radon_data = tfds.as_dataframe(ds)
  radon_data.rename(lambda s: s[9:] if s.startswith('feat') else s, axis=1, inplace=True)
  df = radon_data[radon_data.state==state.encode()].copy()

  df['radon'] = df.activity.apply(lambda x: x if x > 0. else 0.1)
  # Make county names look nice.
  df['county'] = df.county.apply(lambda s: s.decode()).str.strip().str.title()
  # Remap categories to start from 0 and end at max(category).
  df['county'] = df.county.astype(pd.api.types.CategoricalDtype())
  df['county_code'] = df.county.cat.codes
  # Radon levels are all positive, but log levels are unconstrained
  df['log_radon'] = df['radon'].apply(np.log)

  # Drop columns we won't use and tidy the index
  columns_to_keep = ['log_radon', 'floor', 'county', 'county_code']
  df = df[columns_to_keep].reset_index(drop=True)

  return df

df = load_and_preprocess_radon_dataset()
df.head()
```

      log_radon
      floor
      county
      county_code

      0
      0.788457
      1
      Aitkin
      0

      1
      0.788457
      0
      Aitkin
      0

      2
      1.064711
      0
      Aitkin
      0

      3
      0.000000
      0
      Aitkin
      0

      4
      1.131402
      0
      Anoka
      1

### Specializing the GLMM Family

In this section, we specialize the GLMM family to the task of predicting radon levels. To do this, we first consider the fixed-effect special case of a GLMM:
$$
\mathbb{E}[\log(\text{radon}_j)] = c + \text{floor_effect}_j
$$

This model posits that the log radon in observation $j$ is (in expectation) governed by the floor the $j$th reading is taken on, plus some constant intercept. In pseudocode, we might write

    def estimate_log_radon(floor):
        return intercept + floor_effect[floor]

there's a weight learned for every floor and a universal `intercept` term. Looking at the radon measurements from floor 0 and 1, it looks like this might be a good start:

```
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 4))
df.groupby('floor')['log_radon'].plot(kind='density', ax=ax1);
ax1.set_xlabel('Measured log(radon)')
ax1.legend(title='Floor')

df['floor'].value_counts().plot(kind='bar', ax=ax2)
ax2.set_xlabel('Floor where radon was measured')
ax2.set_ylabel('Count')
fig.suptitle("Distribution of log radon and floors in the dataset");
```

To make the model a little more sophisticated, including something about geography is probably even better: radon is part of the decay chain of uranium, which may be present in the ground, so geography seems key to account for.

$$
\mathbb{E}[\log(\text{radon}_j)] = c + \text{floor_effect}_j + \text{county_effect}_j
$$

Again, in pseudocode, we have

    def estimate_log_radon(floor, county):
        return intercept + floor_effect[floor] + county_effect[county]

the same as before except with a county-specific weight.

Given a sufficiently large training set, this is a reasonable model.  However, given our data from Minnesota, we see that there's there's a large number of counties with a small number of measurements. For example, 39 out of 85 counties have fewer than five observations.

This motivates sharing statistical strength between all our observations, in a way that converges to the above model as the number of observations per county increases.

```
fig, ax = plt.subplots(figsize=(22, 5));
county_freq = df['county'].value_counts()
county_freq.plot(kind='bar', ax=ax)
ax.set_xlabel('County')
ax.set_ylabel('Number of readings');
```

If we fit this model, the `county_effect` vector would likely end up memorizing the results for counties which had only a few training samples, perhaps overfitting and leading to poor generalization.

GLMM's offer a happy middle to the above two GLMs.  We might consider fitting

$$
\log(\text{radon}_j) \sim c + \text{floor_effect}_j + \mathcal{N}(\text{county_effect}_j, \text{county_scale})
$$

This model is the same as the first, but we have fixed our likelihood to be a normal distribution, and will share the variance across all counties through the (single) variable `county_scale`.  In pseudocode,

    def estimate_log_radon(floor, county):
        county_mean = county_effect[county]
        random_effect = np.random.normal() * county_scale + county_mean
        return intercept + floor_effect[floor] + random_effect

We will infer the joint distribution over `county_scale`, `county_mean`, and the `random_effect` using our observed data. The global `county_scale` allows us to share statistical strength across counties: those with many observations provide a hit at the variance of counties with few observations. Furthermore, as we gather more data, this model will converge to the model without a pooled scale variable - even with this dataset, we will come to similar conclusions about the most observed counties with either model.

## Experiment

We'll now try to fit the above GLMM using variational inference in TensorFlow. First we'll split the data into features and labels.

```
features = df[['county_code', 'floor']].astype(int)
labels = df[['log_radon']].astype(np.float32).values.flatten()
```

### Specify Model

```
def make_joint_distribution_coroutine(floor, county, n_counties, n_floors):

  def model():
    county_scale = yield tfd.HalfNormal(scale=1., name='scale_prior')
    intercept = yield tfd.Normal(loc=0., scale=1., name='intercept')
    floor_weight = yield tfd.Normal(loc=0., scale=1., name='floor_weight')
    county_prior = yield tfd.Normal(loc=tf.zeros(n_counties),
                                    scale=county_scale,
                                    name='county_prior')
    random_effect = tf.gather(county_prior, county, axis=-1)

    fixed_effect = intercept + floor_weight * floor
    linear_response = fixed_effect + random_effect
    yield tfd.Normal(loc=linear_response, scale=1., name='likelihood')
  return tfd.JointDistributionCoroutineAutoBatched(model)

joint = make_joint_distribution_coroutine(
    features.floor.values, features.county_code.values, df.county.nunique(),
    df.floor.nunique())

# Define a closure over the joint distribution
# to condition on the observed labels.
def target_log_prob_fn(*args):
  return joint.log_prob(*args, likelihood=labels)
```

### Specify surrogate posterior

We now put together a surrogate family $q_{\lambda}$, where the parameters $\lambda$ are trainable. In this case, our family is independent multivariate normal distributions, one for each parameter, and $\lambda = \{(\mu_j, \sigma_j)\}$, where $j$ indexes the four parameters.

The method we use to fit the surrogate family uses `tf.Variables`. We also use `tfp.util.TransformedVariable` along with `Softplus` to constrain the (trainable) scale parameters to be positive. Additionally, we apply `Softplus` to the entire `scale_prior`, which is a positive parameter.

We initialize these trainable variables with a bit of jitter to aid in optimization.

```
# Initialize locations and scales randomly with `tf.Variable`s and
# `tfp.util.TransformedVariable`s.
_init_loc = lambda shape=(): tf.Variable(
    tf.random.uniform(shape, minval=-2., maxval=2.))
_init_scale = lambda shape=(): tfp.util.TransformedVariable(
    initial_value=tf.random.uniform(shape, minval=0.01, maxval=1.),
    bijector=tfb.Softplus())
n_counties = df.county.nunique()

surrogate_posterior = tfd.JointDistributionSequentialAutoBatched([
  tfb.Softplus()(tfd.Normal(_init_loc(), _init_scale())),           # scale_prior
  tfd.Normal(_init_loc(), _init_scale()),                           # intercept
  tfd.Normal(_init_loc(), _init_scale()),                           # floor_weight
  tfd.Normal(_init_loc([n_counties]), _init_scale([n_counties]))])  # county_prior
```

Note that this cell can be replaced with [`tfp.experimental.vi.build_factored_surrogate_posterior`](https://www.tensorflow.org/probability/api_docs/python/tfp/experimental/vi/build_factored_surrogate_posterior?version=nightly), as in:

```python
surrogate_posterior = tfp.experimental.vi.build_factored_surrogate_posterior(
  event_shape=joint.event_shape_tensor()[:-1],
  constraining_bijectors=[tfb.Softplus(), None, None, None])
```

### Results

Recall that our goal is to define a tractable parameterized family of distributions, and then select parameters so that we have a tractable distribution that is close to our target distribution.

We have built the surrogate distribution above, and can use [`tfp.vi.fit_surrogate_posterior`](https://www.tensorflow.org/probability/api_docs/python/tfp/vi/fit_surrogate_posterior), which accepts an optimizer and a given number of steps to find the parameters for the surrogate model minimizing the negative ELBO (which corresonds to minimizing the Kullback-Liebler divergence between the surrogate and the target distribution).

The return value is the negative ELBO at each step, and the distributions in `surrogate_posterior` will have been updated with the parameters found by the optimizer.

```
optimizer = tf.optimizers.Adam(learning_rate=1e-2)

losses = tfp.vi.fit_surrogate_posterior(
    target_log_prob_fn,
    surrogate_posterior,
    optimizer=optimizer,
    num_steps=3000,
    seed=42,
    sample_size=2)

(scale_prior_,
 intercept_,
 floor_weight_,
 county_weights_), _ = surrogate_posterior.sample_distributions()
```

```
print('        intercept (mean): ', intercept_.mean())
print('     floor_weight (mean): ', floor_weight_.mean())
print(' scale_prior (approx. mean): ', tf.reduce_mean(scale_prior_.sample(10000)))
```

            intercept (mean):  tf.Tensor(1.4352839, shape=(), dtype=float32)
         floor_weight (mean):  tf.Tensor(-0.6701997, shape=(), dtype=float32)
     scale_prior (approx. mean):  tf.Tensor(0.28682157, shape=(), dtype=float32)

```
fig, ax = plt.subplots(figsize=(10, 3))
ax.plot(losses, 'k-')
ax.set(xlabel="Iteration",
       ylabel="Loss (ELBO)",
       title="Loss during training",
       ylim=0);
```

We can plot the estimated mean county effects, along with the uncertainty of that mean. We have ordered this by number of observations, with the largest on the left. Notice that the uncertainty is small for the counties with many observations, but is larger for the counties that have only one or two observations.

```
county_counts = (df.groupby(by=['county', 'county_code'], observed=True)
                   .agg('size')
                   .sort_values(ascending=False)
                   .reset_index(name='count'))

means = county_weights_.mean()
stds = county_weights_.stddev()

fig, ax = plt.subplots(figsize=(20, 5))

for idx, row in county_counts.iterrows():
  mid = means[row.county_code]
  std = stds[row.county_code]
  ax.vlines(idx, mid - std, mid + std, linewidth=3)
  ax.plot(idx, means[row.county_code], 'ko', mfc='w', mew=2, ms=7)

ax.set(
    xticks=np.arange(len(county_counts)),
    xlim=(-1, len(county_counts)),
    ylabel="County effect",
    title=r"Estimates of county effects on log radon levels. (mean $\pm$ 1 std. dev.)",
)
ax.set_xticklabels(county_counts.county, rotation=90);
```

Indeed, we can see this more directly by plotting the log-number of observations against the estimated standard deviation, and see the relationship is approximately linear.

```
fig, ax = plt.subplots(figsize=(10, 7))
ax.plot(np.log1p(county_counts['count']), stds.numpy()[county_counts.county_code], 'o')
ax.set(
    ylabel='Posterior std. deviation',
    xlabel='County log-count',
    title='Having more observations generally\nlowers estimation uncertainty'
);
```

## Comparing to `lme4` in R

```
%%shell
exit  # Trick to make this block not execute.

radon = read.csv('srrs2.dat', header = TRUE)
radon = radon[radon$state=='MN',]
radon$radon = ifelse(radon$activity==0., 0.1, radon$activity)
radon$log_radon = log(radon$radon)

# install.packages('lme4')
library(lme4)
fit <- lmer(log_radon ~ 1 + floor + (1 | county), data=radon)
fit

# Linear mixed model fit by REML ['lmerMod']
# Formula: log_radon ~ 1 + floor + (1 | county)
#    Data: radon
# REML criterion at convergence: 2171.305
# Random effects:
#  Groups   Name        Std.Dev.
#  county   (Intercept) 0.3282
#  Residual             0.7556
# Number of obs: 919, groups:  county, 85
# Fixed Effects:
# (Intercept)        floor
#       1.462       -0.693
```

    <IPython.core.display.Javascript at 0x7f90b888e9b0>

    <IPython.core.display.Javascript at 0x7f90b888e780>

    <IPython.core.display.Javascript at 0x7f90b888e780>

    <IPython.core.display.Javascript at 0x7f90bce1dfd0>

    <IPython.core.display.Javascript at 0x7f90b888e780>

    <IPython.core.display.Javascript at 0x7f90b888e780>

    <IPython.core.display.Javascript at 0x7f90b888e780>

    <IPython.core.display.Javascript at 0x7f90b888e780>

The following table summarizes the results.

```
print(pd.DataFrame(data=dict(intercept=[1.462, tf.reduce_mean(intercept_.mean()).numpy()],
                             floor=[-0.693, tf.reduce_mean(floor_weight_.mean()).numpy()],
                             scale=[0.3282, tf.reduce_mean(scale_prior_.sample(10000)).numpy()]),
                   index=['lme4', 'vi']))
```

          intercept   floor     scale
    lme4   1.462000 -0.6930  0.328200
    vi     1.435284 -0.6702  0.287251

This table indicates the VI results are within ~10% of `lme4`'s.  This is somewhat surprising since:
- `lme4` is based on [Laplace's method](https://www.jstatsoft.org/article/view/v067i01/) (not VI),
- no effort was made in this colab to actually converge,
- minimal effort was made to tune hyperparameters,
- no effort was taken regularize or preprocess the data (eg, center features, etc.).

## Conclusion

In this colab we described Generalized Linear Mixed-effects Models and showed how to use variational inference to fit them using TensorFlow Probability. Although the toy problem only had a few hundred training samples, the techniques used here are identical to what is needed at scale.


# Abstract:

The abstract should be 1-3 sentences at most. Describe a) what you measured, b) how you measured it and c) the result you found. Not a great deal of detail, but enough to answer these basic questions. Also include the result from your analysis. So for example in this report we could say: The measured time constant was 3.676 +/- 0.031 msec.

# Description

In the description you would describe the circuit you
built/designed/tested this week and any important features that were
investigated. For example, in the first week'd lab you might say:

This project was meant to familiarize us with the Arduino IDE
and to remind us of the behavior of "RC" circuits. When the charging voltage is suddenly removed from a capacitor the charge "decays"
through some resistance with an exponential time dependence:

\begin{equation}
V(t) = V_0 e^{-t/RC}
\end{equation}

We used an Arduino microcontroller to control the charging and discharging
of a capacitor as well as to record the voltage across the capacitor
as the voltage dropped over time.

# Schematic, Background Discussion

You should always include a schematic describing the circuit you actually built.

If the reader will need any background information or further discussion to explain the experiment, this is a good place to put that.

The circuit was constructed according to the following schematic:

The resitance in the sample circuit was $1\,k\Omega$ and the capacitance was $4\,\mu F$. You may have different values for resistance and capacitance in your circuit.

# Circuit Function, including code + data

We programmed the Arduino with the following "C" program:

    // Pin 13 has an LED connected on most Arduino boards.
    // give it a name:

    #define LED 13
    #define OUTPWM 3          // send PWM output to pin 3
    #define IN_ANALOG 0       // which analog pin are we reading?
    #define NUMLOOPS 40       // how much data to collect?
    #define STARTUP_DELAY 500 // how long to hold voltage high
    #define DECAY_DELAY   10  // how long to measure
    #define LOOP_DELAY 500    // how many millis to delay each time

    int data[NUMLOOPS];  // data from analog input

    short flag=0;        // flag to check for first time

    // the setup routine runs once when you press reset:

    void setup()
    {
        Serial.begin(9600);       // start serial port at 9600 bps:
        pinMode(LED, OUTPUT);     // digital sensor is on digital pin 2
        pinMode(OUTPWM, OUTPUT);  // set PWM output pin for output
        flag=0;
    }

    // the loop routine runs over and over again forever:
    void loop() {
        if (flag==0) {
            analogWrite(OUTPWM, 255);  // set PWM pin high 100% of the time for a while
            delay(STARTUP_DELAY);      // wait for cap to charge
            analogWrite(OUTPWM, 0);    // now set PWM pin low 100% of the time, and measure discharge
            collectData((int)((1000.0*DECAY_DELAY)/NUMLOOPS)); // grab the discharge V vs. t
            flag=1;                    // we're finished collecting data, set flag to 1
        }
        else if (flag==1) {            // we have data
            printData();               // send it to the computer
            flag=2;                    // finished sending, set flag to 2
        }
        else {
            digitalWrite(LED, HIGH);   // Toggle LED to indicate that the program has finished.
            delay(LOOP_DELAY);         // wait for LOOP_DELAY seconds
            digitalWrite(LED, LOW);    // turn the LED off by making the voltage LOW
            delay(LOOP_DELAY);         // wait for LOOP_DELAY seconds
        }
    }

    void collectData(int delayMicros) {         // collectData read the analog port (analog pin 0)
        for (int i=0; i<NUMLOOPS; i++) {        // and stores the results in the "data" array.
            data[i]=analogRead(IN_ANALOG);      // grab one voltage and store it
            delayMicroseconds(delayMicros);     // wait for a bit between samples
        }
    }

    void printData() {
        Serial.println("time,voltage");
        for (int i=0; i<NUMLOOPS; i++) {
            Serial.print(((1.0*i*DECAY_DELAY)/(NUMLOOPS)),4);  // convert times to milliseconds
            Serial.print(',');                                 // make a comma separated value file
            Serial.println((5.0*data[i])/1023.0,4);            // convert A/D values to "volts".
        }
    }

Data
=====

The `printData` function sent the "CSV" strings to the computer over the serial monitor. The code below reads in the CSV data and loads it into a 'pandas' (Python Data Analysis) table. You can use Excel, or whatever if you like. If you want to use python, I'd be happy to help you get started.

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
```

```python
data = pd.read_excel('data.xlsx')
data
```

      time
      voltage

      0
      0.00
      4.8680

      1
      0.25
      4.5161

      2
      0.50
      4.1935

      3
      0.75
      3.8954

      4
      1.00
      3.6266

      5
      1.25
      3.3675

      6
      1.50
      3.1329

      7
      1.75
      2.9130

      8
      2.00
      2.7126

      9
      2.25
      2.5269

      10
      2.50
      2.3509

      11
      2.75
      2.1896

      12
      3.00
      2.0381

      13
      3.25
      1.8964

      14
      3.50
      1.7693

      15
      3.75
      1.6520

      16
      4.00
      1.5396

      17
      4.25
      1.4370

      18
      4.50
      1.3441

      19
      4.75
      1.2561

      20
      5.00
      1.1681

      21
      5.25
      1.0948

      22
      5.50
      1.0215

      23
      5.75
      0.9580

      24
      6.00
      0.8944

      25
      6.25
      0.8407

      26
      6.50
      0.7820

      27
      6.75
      0.7331

      28
      7.00
      0.6843

      29
      7.25
      0.6452

      30
      7.50
      0.6012

      31
      7.75
      0.5670

      32
      8.00
      0.5327

      33
      8.25
      0.4985

      34
      8.50
      0.4692

      35
      8.75
      0.4350

      36
      9.00
      0.4154

      37
      9.25
      0.3861

      38
      9.50
      0.3666

      39
      9.75
      0.3421

# Results (including analysis and stats exercises)

The idea of this section is to tell the story of what you learned from the data you collected. Use whatever tools you need to make it clear how you analyzed the data to make whatever claims you can make. For example from the first week's experiment you might say the following:

Looking at the raw data we can see an exponential decay:

```python
#
# let's plot the raw data to see how it looks
#

plt.title("Capacitor voltage over time")
plt.xlabel("time (ms)")
plt.ylabel("Vc (volts)")
plt.plot(data.time.values, data.voltage.values, 'b.')
```

We can also take the natural log (just `log` in python) and estimate the time constant since the voltage is exponential:

\begin{equation}
V(t) = V_0 e^{-t/RC}
\end{equation}

we can take the "ln" of both sides to produce a linear relationship:

\begin{equation}
ln(V(t)) = ln(V_0) - \frac{t}{RC}
\end{equation}

therefore the slope should be ${-1/RC}$.

```python
#
# This code uses scipy.optimize to fit the collected data, you can use excel or whatever
#

from scipy.optimize import curve_fit

def fLinear(x, m, b):
    return m*x + b

popt, pcov = curve_fit(fLinear, data.time.values, np.log(data.voltage.values+0.001))

m=popt[0]          # slope
b=popt[1]          # int

dm, db = np.sqrt(np.diag(pcov))

ystar=fLinear(data.time.values, m, b)

print("Slope: %4.4f +/- %4.4f (per msec)" % (m, 2*dm))
print("Time constant: %4.3f +/- %4.3f msec  (4 msec theoretical)" % (abs(1/m), abs(2*dm/m**2)))

plt.title("Decay of charge on a capacitor")
plt.ylabel("ln(V)")
plt.xlabel("time (ms)")
plt.plot(data.time.values, np.log(data.voltage.values+0.001), 'b.')
plt.text(2, 1.5, "$ln(V) = %6.3f\  + (%6.3f t)$" % (b, m))
plt.plot(data.time.values, ystar,'r-')
```

Then go on to explain the significance of the slope/intercept or whatever else you can from the analysis of the data you collected. For example you can compare the measured to the expected slope:

So, the measured slope was $m=-0.272$ per millisecond giving a time constant of, but we expected a slope of:

\begin{equation}
m =\frac{-1}{RC} = \frac{-1}{1k\Omega\cdot 4\mu F} = -0.250\,{\rm ms^{-1}}
\end{equation}

Conclusion
===========

Be sure to answer any questions posed in the project description (handout) and give some kind of final assessment of the project overall. (Did it meet expectations, goals, etc.?) For example you might say:

Since the expected slope (-0.25 per millisecond) is withing 10% of the measured value (-0.272 per millisecond) and the capacitance is not a high-precision component this seems an acceptable variation.

```python

```


Install py-pde library

```python
%pip install py-pde
```

    Requirement already satisfied: py-pde in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (0.17.1)
    Requirement already satisfied: matplotlib>=3.1.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (3.5.1)
    Requirement already satisfied: sympy>=1.5.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.9)
    Requirement already satisfied: numpy>=1.18.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.21.5)
    Requirement already satisfied: numba>=0.50.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (0.55.1)
    Requirement already satisfied: scipy>=1.4.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from py-pde) (1.7.3)
    Requirement already satisfied: cycler>=0.10 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (0.11.0)
    Requirement already satisfied: pillow>=6.2.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (9.0.1)
    Requirement already satisfied: pyparsing>=2.2.1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (3.0.7)
    Requirement already satisfied: fonttools>=4.22.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (4.29.1)
    Requirement already satisfied: packaging>=20.0 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (21.3)
    Requirement already satisfied: python-dateutil>=2.7 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (2.8.2)
    Requirement already satisfied: kiwisolver>=1.0.1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from matplotlib>=3.1.0->py-pde) (1.3.2)
    Requirement already satisfied: setuptools in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from numba>=0.50.0->py-pde) (58.0.4)
    Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from numba>=0.50.0->py-pde) (0.38.0)
    Requirement already satisfied: six>=1.5 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from python-dateutil>=2.7->matplotlib>=3.1.0->py-pde) (1.16.0)
    Requirement already satisfied: mpmath>=0.19 in /Users/sebastian/opt/anaconda3/envs/neuralOperator37/lib/python3.7/site-packages (from sympy>=1.5.0->py-pde) (1.2.1)
    Note: you may need to restart the kernel to use updated packages.

Generate dataset

```python
from pde import CartesianGrid, ScalarField, solve_poisson_equation
import torch
import numpy as np

num_param_steps = 64
field_values = np.linspace(0,10,num_param_steps)
grid_size = 64

training_input = torch.zeros(num_param_steps, grid_size,2)
training_output = torch.zeros(num_param_steps, grid_size,1)

grid = CartesianGrid([[0, 1]], grid_size, periodic=False)

for index, val in enumerate(field_values):
  field = ScalarField(grid, val)
  result = solve_poisson_equation(field, bc=[{"value": 0}, {"derivative": 1}])
  training_input[index,:,0] = torch.tensor(field.data)
  training_input[index,:,1] = torch.linspace(0,1, grid_size)
  training_output[index,:,0] = torch.tensor(result.data)

```

```python
import math as m

radius = 3.0
area = m.pi * m.pow(radius, 2)
print(area)
```

    28.274333882308138

```python
import torch
!python --version

!python -m site --user-site
#!pip list -v
torch.__version__

```

    Python 3.7.11
    /Users/sebastian/.local/lib/python3.7/site-packages

    '1.10.2'

Model definitions copied from https://github.com/zongyi-li/fourier_neural_operator

```python
import torch.nn as nn
import torch.nn.functional as F

################################################################
#  1d fourier layer
################################################################
class SpectralConv1d(nn.Module):
    def __init__(self, in_channels, out_channels, modes1):
        super(SpectralConv1d, self).__init__()

        """
        1D Fourier layer. It does FFT, linear transform, and Inverse FFT.
        """

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.modes1 = modes1  #Number of Fourier modes to multiply, at most floor(N/2) + 1

        self.scale = (1 / (in_channels*out_channels))
        self.weights1 = nn.Parameter(self.scale * torch.rand(in_channels, out_channels, self.modes1, dtype=torch.cfloat))

    # Complex multiplication
    def compl_mul1d(self, input, weights):
        # (batch, in_channel, x ), (in_channel, out_channel, x) -> (batch, out_channel, x)
        return torch.einsum("bix,iox->box", input, weights)

    def forward(self, x):
        batchsize = x.shape[0]
        #Compute Fourier coeffcients up to factor of e^(- something constant)
        x_ft = torch.fft.rfft(x)

        # Multiply relevant Fourier modes
        out_ft = torch.zeros(batchsize, self.out_channels, x.size(-1)//2 + 1,  device=x.device, dtype=torch.cfloat)
        out_ft[:, :, :self.modes1] = self.compl_mul1d(x_ft[:, :, :self.modes1], self.weights1)

        #Return to physical space
        x = torch.fft.irfft(out_ft, n=x.size(-1))
        return x

class FNO1d(nn.Module):
    def __init__(self, modes, width):
        super(FNO1d, self).__init__()

        """
        The overall network. It contains 4 layers of the Fourier layer.
        1. Lift the input to the desire channel dimension by self.fc0 .
        2. 4 layers of the integral operators u' = (W + K)(u).
            W defined by self.w; K defined by self.conv .
        3. Project from the channel space to the output space by self.fc1 and self.fc2 .

        input: the solution of the initial condition and location (a(x), x)
        input shape: (batchsize, x=s, c=2)
        output: the solution of a later timestep
        output shape: (batchsize, x=s, c=1)
        """

        self.modes1 = modes
        self.width = width
        self.padding = 2 # pad the domain if input is non-periodic
        self.fc0 = nn.Linear(2, self.width) # input channel is 2: (a(x), x)

        self.conv0 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv1 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv2 = SpectralConv1d(self.width, self.width, self.modes1)
        self.conv3 = SpectralConv1d(self.width, self.width, self.modes1)
        self.w0 = nn.Conv1d(self.width, self.width, 1)
        self.w1 = nn.Conv1d(self.width, self.width, 1)
        self.w2 = nn.Conv1d(self.width, self.width, 1)
        self.w3 = nn.Conv1d(self.width, self.width, 1)

        self.fc1 = nn.Linear(self.width, 128)
        self.fc2 = nn.Linear(128, 1)

    def forward(self, x):
        x = self.fc0(x)
        x = x.permute(0, 2, 1)
        # x = F.pad(x, [0,self.padding]) # pad the domain if input is non-periodic

        x1 = self.conv0(x)
        x2 = self.w0(x)
        x = F.gelu(x1) + x2

        x1 = self.conv1(x)
        x2 = self.w1(x)
        x = F.gelu(x1) + x2

        x1 = self.conv2(x)
        x2 = self.w2(x)
        x = F.gelu(x1) + x2

        x1 = self.conv3(x)
        x2 = self.w3(x)
        x = F.gelu(x1) + x2

        # x = x[..., :-self.padding] # pad the domain if input is non-periodic
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        x = F.gelu(x)
        x = self.fc2(x)
        return x

```

```python
modes = 32
width = 16

epochs = 5000
learning_rate = 1e-4
batch_size = 64

model = FNO1d(modes, width) #.to('cuda')

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr = 1e-3, epochs=epochs, steps_per_epoch= num_param_steps // batch_size)

dataloader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(training_input, training_output), batch_size=batch_size, shuffle=True)

for ep in range(epochs):
  for input, output in dataloader:
    #input, output = input.cuda(), output.cuda()
    optimizer.zero_grad()
    pred_output = model(input)
    loss = torch.nn.functional.mse_loss(pred_output, output)
    loss.backward()
    optimizer.step()
    scheduler.step()
  print("\r",'loss:' + str(loss.detach().cpu().numpy()), end = "")

```

     loss:nan51985

Check output

```python
import matplotlib.pyplot as plt

grid_start = 0
grid_end = 1
test_grid_size = 64
field_val = 2

test_field = ScalarField(CartesianGrid([[grid_start, grid_end]], test_grid_size, periodic=False), field_val)
test_result = solve_poisson_equation(test_field, bc=[{"value": 0}, {"derivative": 1}])

model_input = torch.tensor(test_field.data, dtype= torch.float).reshape(1,test_grid_size,1)
model_input = torch.cat((model_input.reshape(1,test_grid_size,1), torch.linspace(grid_start,grid_end, test_grid_size).reshape(1,test_grid_size,1)), dim = -1)
#model_input = model_input.to('cuda')

model_result = model(model_input)

plt.figure()
plt.plot(test_result.data)
plt.plot(model_result.detach().cpu().flatten().numpy())
```

```python

```


# Models from motions

# Purpose
Create roll decay models from the simulation results from Motions (with/without viscous damping). save these models into pkl files.

# Methodology
* Load
* Cut
* Fit
* save

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')
from reports.paper_writing import save_fig

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import src.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables
import shipflowmotionshelpers.shipflowmotionshelpers as helpers
import src.visualization.visualize as visualize

```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 463 ('figure.dpi       : 100        ## figure dots per inch')

```python
from copy import deepcopy
import joblib
from rolldecayestimators import measure
from src.helpers import get_ikeda, calculate_ikeda, get_estimator_variation, get_data_variation , get_variation, hatify
import scipy
from sklearn.pipeline import Pipeline
#from reports import mdl_results
```

## Load from MDL

```python
def load():
    db = database.get_db()

    sql = """
    SELECT * from run
    INNER JOIN loading_conditions
    ON (run.loading_condition_id = loading_conditions.id)
    INNER JOIN models
    ON (run.model_number = models.model_number)
    INNER JOIN ships
    ON (run.ship_name = ships.name)
    WHERE run.model_number='M5057-01-A' and run.test_type='roll decay' and run.project_number=40178362;
    """
    df_rolldecays = pd.read_sql(sql=sql, con=db.engine)
    df_rolldecays['rho']=1000
    df_rolldecays['g']=9.81
    df_rolldecays=df_rolldecays.loc[:,~df_rolldecays.columns.duplicated()]
    df_rolldecays.set_index('id', inplace=True)

    df_rolldecays['ship_speed'].fillna(0, inplace=True)

    df_rolldecays=df_rolldecays.loc[[21337,21338,21340,]].copy()
    df_rolldecays['paper_name'] = np.arange(len(df_rolldecays)) + 1

    return df_rolldecays

df_rolldecays = load()
```

## Load data from Motions:

```python
file_paths = [
    '../data/external/kvlcc2_rolldecay_0kn',
    '../data/external/kvlcc2_rolldecay_15-5kn_const_large',
    '../data/external/kvlcc2_rolldecay_15-5kn_const_large2',
    '../data/external/kvlcc2_rolldecay_15-5kn_const_large_5deg',
    '../data/external/kvlcc2_rolldecay_15-5kn_const_large_ikeda',
    '../data/external/kvlcc2_rolldecay_15-5kn_ikeda_dev',
]
df_parameters = pd.DataFrame()
df_parameters =  helpers.load_parameters(file_path=file_paths)

mask_0 = (df_parameters['vm_s'].round(5) == 0)
df_parameters.loc[mask_0,'id'] = 21338  # MDL DB run id
df_parameters.loc[~mask_0,'id'] = 21340
```

```python
mask_visc = (df_parameters[['b4l','b4q']] > 0).any(axis=1)
df_parameters.loc[mask_visc]
```

      B
      IXY
      IXZ
      IYY
      IYZ
      IZZ
      KXX
      KXY
      KYY
      KZZ
      LPP
      S
      V
      XCG
      YCG
      ZCG
      accx
      acto
      b1c
      b1cr
      b1l
      b1q
      b2c
      b2cr
      b2l
      b2q
      b3c
      b3cr
      b3l
      b3q
      b4c
      b4cr
      b4l
      b4q
      b5c
      b5cr
      b5l
      b5q
      b6c
      b6cr
      b6l
      b6q
      bdens
      body
      clev
      clevel
      curv
      dens
      densi
      dofa
      dofactor
      dopa
      dopadding
      dopo
      dopower
      down
      downs
      downst
      dt
      file
      file_path_ts
      fn
      form
      free
      gravi
      heave
      heel
      hull
      inter
      k1nd
      k2nd
      k3nd
      k4nd
      k5nd
      k6nd
      kxx
      kyy
      leve
      level
      lpp
      m
      maxti
      maxtime
      mesh
      name
      npxp
      npyp
      nstep
      nthr
      pitch
      powe
      power
      refle
      reflen
      rn
      roll
      rud1
      rud2
      rud3
      rud4
      rud5
      rud6
      rudt
      side
      slim
      stre
      strength
      surge
      sway
      ta
      tf
      tfnhi
      tfnhigh
      tfnlo
      tfnlow
      titl
      title
      trim
      upst
      upstr
      vm_s
      wlin
      wlme
      yaw
      ymax
      ymin
      zcg
      conv
      encounters
      id

      kvlcc2_rolldecay_15-5kn_const_large_ikeda
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.025
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      6.07217
      2.74371
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00002
      1000.0
      1000.0
      10.0
      10.0
      2.0
      2.0
      2.0
      2.0
      1.00
      -5.0
      1.00
      0.02
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      0.142341
      0.23
      0.3
      9.80665
      0.0
      0.0
      0.002
      0.000000e+00
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      200.0
      200.0
      0.000000e+00
      TRAN
      40.0
      40.0
      30.0
      32.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3826600.0
      10.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.00
      30.0
      0.1
      0.1
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      1.00
      5.0
      0.966976
      0.000001
      0.000001
      0.0
      5.0
      -5.0
      0.2735
      NaN
      NaN
      21340.0

      kvlcc2_rolldecay_15-5kn_ikeda_dev
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.050
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      6.07217
      2.74371
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00004
      1000.0
      1000.0
      0.0
      0.0
      2.0
      2.0
      1.0
      1.0
      0.25
      -2.0
      0.25
      NaN
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      0.142341
      0.20
      0.3
      9.80665
      0.0
      0.0
      0.002
      1.000000e-07
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      600.0
      600.0
      1.000000e-07
      TRAN
      24.0
      24.0
      30.0
      6.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3826600.0
      10.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.25
      30.0
      1.0
      1.0
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      0.25
      2.0
      0.966976
      0.000001
      0.000001
      0.0
      2.0
      -2.0
      0.2735
      0.0001
      0.0
      21340.0

```python
df_parameters.loc[~mask_visc]
```

      B
      IXY
      IXZ
      IYY
      IYZ
      IZZ
      KXX
      KXY
      KYY
      KZZ
      LPP
      S
      V
      XCG
      YCG
      ZCG
      accx
      acto
      b1c
      b1cr
      b1l
      b1q
      b2c
      b2cr
      b2l
      b2q
      b3c
      b3cr
      b3l
      b3q
      b4c
      b4cr
      b4l
      b4q
      b5c
      b5cr
      b5l
      b5q
      b6c
      b6cr
      b6l
      b6q
      bdens
      body
      clev
      clevel
      curv
      dens
      densi
      dofa
      dofactor
      dopa
      dopadding
      dopo
      dopower
      down
      downs
      downst
      dt
      file
      file_path_ts
      fn
      form
      free
      gravi
      heave
      heel
      hull
      inter
      k1nd
      k2nd
      k3nd
      k4nd
      k5nd
      k6nd
      kxx
      kyy
      leve
      level
      lpp
      m
      maxti
      maxtime
      mesh
      name
      npxp
      npyp
      nstep
      nthr
      pitch
      powe
      power
      refle
      reflen
      rn
      roll
      rud1
      rud2
      rud3
      rud4
      rud5
      rud6
      rudt
      side
      slim
      stre
      strength
      surge
      sway
      ta
      tf
      tfnhi
      tfnhigh
      tfnlo
      tfnlow
      titl
      title
      trim
      upst
      upstr
      vm_s
      wlin
      wlme
      yaw
      ymax
      ymin
      zcg
      conv
      encounters
      id

      kvlcc2_rolldecay_0kn
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.050
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00002
      1000.0
      1000.0
      50.0
      50.0
      2.0
      2.0
      2.0
      2.0
      1.0
      -5.0
      1.0
      0.02
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      1.472020e-07
      0.23
      0.3
      9.80665
      0.0
      0.0
      0.002
      0.0
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      180.0
      180.0
      0.0
      TRAN
      40.0
      40.0
      30.0
      32.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3.957280e+00
      10.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      30.0
      0.50
      0.50
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      1.0
      5.0
      0.000001
      0.000001
      0.000001
      0.0
      5.0
      -5.0
      0.2735
      NaN
      NaN
      21338.0

      kvlcc2_rolldecay_15-5kn_const_large
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.025
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00002
      1000.0
      1000.0
      10.0
      10.0
      2.0
      2.0
      2.0
      2.0
      1.0
      -5.0
      1.0
      0.02
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      1.423410e-01
      0.23
      0.3
      9.80665
      0.0
      0.0
      0.002
      0.0
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      200.0
      200.0
      0.0
      TRAN
      40.0
      40.0
      30.0
      32.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3.826600e+06
      10.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      30.0
      0.10
      0.10
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      1.0
      5.0
      0.966976
      0.000001
      0.000001
      0.0
      5.0
      -5.0
      0.2735
      NaN
      NaN
      21340.0

      kvlcc2_rolldecay_15-5kn_const_large2
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.025
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00002
      1000.0
      1000.0
      1.0
      1.0
      2.0
      2.0
      1.0
      1.0
      0.7
      -5.0
      0.7
      0.02
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      1.423410e-01
      0.23
      0.3
      9.80665
      0.0
      0.0
      0.002
      0.0
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      200.0
      200.0
      0.0
      TRAN
      40.0
      40.0
      30.0
      32.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3.826600e+06
      10.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.7
      30.0
      0.05
      0.05
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      0.7
      5.0
      0.966976
      0.000001
      0.000001
      0.0
      5.0
      -5.0
      0.2735
      NaN
      NaN
      21340.0

      kvlcc2_rolldecay_15-5kn_const_large_5deg
      0.853
      0.0
      0.0
      26.055
      0.0
      26.055
      0.341
      7.556
      1.177
      1.177
      4.706
      5.981
      0.993
      2.519
      0.0
      0.274
      0.025
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.6
      0.3
      6.0
      6.0
      0.00002
      1000.0
      1000.0
      10.0
      10.0
      2.0
      2.0
      2.0
      2.0
      1.0
      -5.0
      1.0
      0.02
      ..
      C:\Dev\Prediction-of-roll-damping-using-fully-...
      1.423410e-01
      0.23
      0.3
      9.80665
      0.0
      0.0
      0.002
      0.0
      0.1
      0.1
      0.0
      0.0
      0.0
      0.1
      0.341185
      1.1765
      6.0
      6.0
      4.706
      993.42
      200.0
      200.0
      0.0
      TRAN
      40.0
      40.0
      30.0
      32.0
      0.0
      2.0
      2.0
      4.706
      4.706
      3.826600e+06
      5.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      30.0
      0.10
      0.10
      0.0
      0.0
      0.3059
      0.3059
      4.0
      4.0
      0.5
      0.5
      KVLCC2
      KVLCC2
      0.0
      1.0
      5.0
      0.966976
      0.000001
      0.000001
      0.0
      5.0
      -5.0
      0.2735
      NaN
      NaN
      21340.0

## Load time series

```python
def find_t_max(X, phi_max):

    Xs = X.copy()
    Xs.index = pd.TimedeltaIndex(X.index, unit='s')
    X_ = Xs.abs().resample('5S').max()
    X_.index = X_.index.total_seconds()
    mask = X_['phi'] < phi_max

    X_small = X_.loc[mask]
    if len(X_small) > 0:
        t_max = X_small.iloc[0].name
    else:
        t_max = X.index[-1]

    return t_max
```

```python
time_series = helpers.load_time_series(df_parameters=df_parameters)
time_series_raw = deepcopy(time_series)

phi_max = np.deg2rad(3.5)

for key,df in time_series.items():

    if df.mean().abs()['V1'] > 0.01:

        phi1d_limit = 10**-2
        index0 = (df['phi1d'].abs() > phi1d_limit).argmax()
        X = df.iloc[index0:].copy()
        t_max = find_t_max(X=X, phi_max=phi_max)
        mask = X.index <= t_max
        X = X.loc[mask]

        time_series[key] = X
```

## Save raw datasets

```python
interesting = ['phi','phi1d','phi2d','V1']

save_runs = [
'kvlcc2_rolldecay_0kn',
'kvlcc2_rolldecay_15-5kn_const_large2',
'kvlcc2_rolldecay_15-5kn_ikeda_dev',
]

for key in save_runs:

    df_save = time_series[key][interesting].copy()
    save_file_name = '%s.csv' % key
    save_path = os.path.join('../data/processed/', save_file_name)
    df_save.to_csv(save_path)

df_parameters_save = df_parameters.loc[save_runs].copy()
save_path = os.path.join('../data/processed/', 'fnpf_parameters.csv')
df_parameters_save.to_csv(save_path)

df_rolldecays = df_rolldecays.copy()
save_path = os.path.join('../data/processed/', 'model_test_parameters.csv')
df_rolldecays.to_csv(save_path)
```

## Cutting

```python
#X = time_series['kvlcc2_rolldecay_15-5kn_const_large_ikeda']
#time_series['kvlcc2_rolldecay_15-5kn_const_large_ikeda'] = X.loc[0:110]
```

```python
for key, df in time_series.items():

    fig,ax = plt.subplots()
    fig.set_size_inches(20,4)
    time_series_raw[key].plot(y='phi', ax=ax)
    df.plot(y='phi', ax=ax)
    ax.set_title(key)
```

## Fitting

```python
models = OrderedDict()

for key,X in time_series.items():

    pre_model= EstimatorQuadraticB(fit_method='derivation')
    pre_model.fit(X)

    model_motions = EstimatorQuadraticB(p0=pre_model.parameters)

    try:
        model_motions.fit(X=X)
    except scipy.linalg.LinAlgError:
        model_motions.fit(X=X)  # Retry

    if pre_model.score() > model_motions.score():
        model_motions = pre_model

    models[key] = model_motions
```

## Genereate results

```python
for key, model in models.items():

    parameters = df_parameters.loc[key]

    row = df_rolldecays.loc[parameters.id]

    scale_factor = row.scale_factor
    meta_data = {
        'Volume' : row.Volume/(scale_factor**3),
        'rho' : row.rho,
        'g' : row.g,
        'GM' : row.gm/scale_factor,
        }

    model.result_for_database(meta_data=meta_data)  # Results are stored in object

    if not 'B_3' in model.results:
        model.results['B_3'] = 0

```

## Saving models

```python
for key,model in models.items():

    # Saving the model as a pipline:
    pipeline = Pipeline(steps=[('estimator',model)])

    joblib.dump(pipeline, '../models/%s.pkl' % key)

```

```python
for key,model in models.items():

    model.plot_damping()

```


```python
import numpy as np
import pandas as pd
import sympy

import weldx
import weldx.measurement as msm
import weldx.transformations as tf
from weldx import Q_
from weldx import util
```

```python
import ipywidgets as widgets
from IPython.display import display
import xarray as xr
import io
```

```python
class CsvMappingFormGroup:

    def __init__(self, form_title="Unbekannte Messreihe", unit_value=""):

        title = widgets.HTML(value="<h3>" + form_title + "</h3>")
        file = widgets.FileUpload(multiple=False)
        out = widgets.Output()

        def on_file_change(e):
            out.clear_output()

            keys = list(e["new"].keys())
            content = list(e["new"].values())[0]["content"]
            with out:
                print(keys)
                df = pd.read_csv(io.BytesIO(content), nrows=5)
            with out:
                display(df)
                cols = list(df.dtypes)

            with out:
                cols = list(df.columns) + ["keine Zuweisung"]
                target_cols = ["time", "value"]
                col_mapping_dropdowns = []
                form_group = []

                for i in range(0, len(target_cols)):
                    drop_down = widgets.Dropdown(
                        options=cols,
                        value=cols[i if i < len(cols) else 0],
                        disabled=False,
                     )
                    col_mapping_dropdowns.append(drop_down)
                    form_group = form_group + [widgets.Label(target_cols[i] + "-Spalte"), drop_down]

                unit = widgets.Text(value=unit_value)
                form_group.append(widgets.Label("Einheit"))
                form_group.append(unit)
                convert_button = widgets.Button(icon="file-export", description="Konvertieren")
                form_group.append(convert_button)
                validation_out = widgets.Output()
                form_group.append(validation_out)
                def on_click(e):
                    validation_out.clear_output()
                    try:
                        time_column = col_mapping_dropdowns[0].value
                        value_column = col_mapping_dropdowns[1].value

                        values = Q_(np.array(df[value_column]), unit.value)
                        da = xr.DataArray(values, coords=[pd.to_datetime(df[time_column])], dims=["time"])
                        self.value = da
                        with validation_out:
                            display(self.value)
                    except Exception as e:
                        with validation_out:
                            print(e)

                convert_button.on_click(on_click)

                form_group_box = widgets.VBox(form_group)
                display(form_group_box)
        file.observe(on_file_change, names='value')

        self.title = title
        self.file = file
        self.out = out
        self.value = []
    def build_widget_box(self):
        return widgets.VBox([self.title, self.file, self.out])

    def weldx_measurement_data(self, name="Unknown"):
        data = msm.Data(name=name, data=self.value)
        return data
```

```python
form_title = widgets.HTML(value="<h2>Messreihen erfassen</h2>")
fg_I = CsvMappingFormGroup("1.) Stromstärke", "ampere")
fg_U = CsvMappingFormGroup("2.) Spannung", "volt")
```

```python
form = widgets.VBox([form_title ,fg_I.build_widget_box(), fg_U.build_widget_box()])
form
```

    VBox(children=(HTML(value='
Messreihen erfassen
'), VBox(children=(HTML(value='
1.) Stromstärke
…

```python
import asdf

save_out = widgets.Output()
def on_save(e):
    measurement_data = [fg_I.weldx_measurement_data("Welding current"), fg_U.weldx_measurement_data("Welding voltage")]
    tree = {
        "data": measurement_data
    }
    af = asdf.AsdfFile(tree)
    af.write_to('measuremen_data.asdf')
    buffer = weldx.asdf.util._write_buffer(tree)

    save_out.clear_output()
    with save_out:
        display(weldx.asdf.util.notebook_fileprinter(buffer))

save_button = widgets.Button(icon="save", description="Speichern")
save_button.on_click(on_save)
display(save_button, save_out)
```

    Button(description='Speichern', icon='save', style=ButtonStyle())

    Output()

```python

```