# Survival Regression with `estimators.SurvivalModel`

Author: ***Willa Potosnak***  <wpotosna@andrew.cmu.edu>

# Contents

### 1. [Introduction](#introduction)
####                  1.1 [The SUPPORT Dataset](#support)
####                  1.2 [Preprocessing the Data](#preprocess)

### 2. [Cox Proportional Hazards (CPH)](#cph)
####                   2.1 [Fit CPH Model](#fitcph)
####                   2.2 [Evaluate CPH Model](#evalcph)

### 3. [Deep Cox Proportional Hazards (DCPH)](#fsn)
####                   3.1 [Fit DCPH Model](#fitfsn)
####                   3.2 [Evaluate DCPH Model](#evalfsn)

### 4. [Deep Survival Machines (DSM)](#dsm)
####                   4.1 [Fit DSM Model](#fitdsm)
####                   4.2 [Evaluate DSM Model](#evaldsm)

### 5. [Deep Cox Mixtures (DCM)](#dcm)
####                   5.1 [Fit DCM Model](#fitdcm)
####                   5.2 [Evaluate DCM Model](#evaldcm)

### 6. [Random Survival Forests (RSF)](#rsf)
####                   6.1 [Fit RSF Model](#fitrsf)
####                   6.2 [Evaluate RSF Model](#evalrsf)

## 1. Introduction

The `SurvivalModels` class offers a steamlined approach to train two `auton-survival` models and three baseline survival models for right-censored time-to-event data. The fit method requires the same inputs across all five models, however, model parameter types vary and must be defined and tuned for the specified model.

### Native `auton-survival` Models

* **Faraggi-Simon Net (FSN)/DeepSurv**
* **Deep Survival Machines (DSM)**
* **Deep Cox Mixtures (DCM)**

### External Models

* **Random survival Forests (RSF)**
* **Cox Proportional Hazards (CPH)**

$\textbf{Hyperparameter tuning}$ and $\textbf{model evaluation}$ can be performed using the following metrics, among others.

* $\textbf{Brier Score (BS)}$: the Mean Squared Error (MSE) around the probabilistic prediction at a certain time horizon. The Brier Score can be decomposed into components that measure both discriminative performance and calibration.

\begin{align}
\text{BS}(t) = \mathop{\mathbf{E}}_{x\sim\mathcal{D}}\big[ ||\mathbf{1}\{ T > t \} - \widehat{\mathbf{P}}(T>t|X)\big)||_{_\textbf{2}}^\textbf{2}  \big]
\end{align}

* $\textbf{Integrated Brier Score (IBS)}$: the integral of the time-dependent $\textbf{BS}$ over the interval $[t_1; t_{max}]$ where the weighting function is $w(t)= \frac{t}{t_{max}}$.

\begin{align}
\text{IBS} = \int_{t_1}^{t_{max}} \mathrm{BS}^{c}(t)dw(t)
\end{align}

* $\textbf{Area under ROC Curve (ROC-AUC)}$: survival model evaluation can be treated as binary classification to compute the **True Positive Rate (TPR)** and **False Positive Rate (FPR)** dependent on time, $t$. ROC-AUC is used to assess how well the model can distinguish samples that fail by a given time, $t$ from those that fail after this time.

\begin{align}
\widehat{AUC}(t) = \frac{\sum_{i=1}^{n} \sum_{j=1}^{n}I(y_j>t)I(y_i \leq t)w_iI(\hat{f}(x_j) \leq \hat{f}(x_i))}{(\sum_{i=1}^{n} I(y_i > t))(\sum_{i=1}^{n}I(y_i \leq t)w_i)}
\end{align}

* $\textbf{Time Dependent Concordance Index (C$^{td}$)}$: estimates ranking ability by exhaustively comparing relative risks across all pairs of individuals in the test set. We employ the ‘Time Dependent’ variant of Concordance Index that truncates the pairwise comparisons to the events occurring within a fixed time horizon.

\begin{align}
C^{td}(t) = P(\hat{F}(t|x_i) > \hat{F} (t|x_j)|\delta_i = 1, T_i < T_j, T_i \leq t)
\end{align}

### 1.1. The SUPPORT Dataset

*For the original datasource, please refer to the following [website](https://biostat.app.vumc.org/wiki/Main/SupportDesc).*

Data features $x$ are stored in a pandas dataframe with rows corresponding to individual samples and columns as covariates. Data outcome consists of 'time', $t$, and 'event', $e$, that correspond to the time to event and the censoring indicator, respectively.

```python
import pandas as pd
import sys
sys.path.append('../')

from auton_survival.datasets import load_dataset
```

```python
# Load the SUPPORT dataset
outcomes, features = load_dataset(dataset='SUPPORT')

# Identify categorical (cat_feats) and continuous (num_feats) features
cat_feats = ['sex', 'dzgroup', 'dzclass', 'income', 'race', 'ca']
num_feats = ['age', 'num.co', 'meanbp', 'wblc', 'hrt', 'resp',
             'temp', 'pafi', 'alb', 'bili', 'crea', 'sod', 'ph',
             'glucose', 'bun', 'urine', 'adlp', 'adls']

# Let's take a look at the features
display(features.head(5))

# Let's take a look at the outcomes
display(outcomes.head(5))
```

      sex
      dzgroup
      dzclass
      income
      race
      ca
      age
      num.co
      meanbp
      wblc
      ...
      alb
      bili
      crea
      sod
      ph
      glucose
      bun
      urine
      adlp
      adls

      0
      male
      Lung Cancer
      Cancer
      $11-$25k
      other
      metastatic
      62.84998
      0
      97.0
      6.000000
      ...
      1.799805
      0.199982
      1.199951
      141.0
      7.459961
      NaN
      NaN
      NaN
      7.0
      7.0

      1
      female
      Cirrhosis
      COPD/CHF/Cirrhosis
      $11-$25k
      white
      no
      60.33899
      2
      43.0
      17.097656
      ...
      NaN
      NaN
      5.500000
      132.0
      7.250000
      NaN
      NaN
      NaN
      NaN
      1.0

      2
      female
      Cirrhosis
      COPD/CHF/Cirrhosis
      under $11k
      white
      no
      52.74698
      2
      70.0
      8.500000
      ...
      NaN
      2.199707
      2.000000
      134.0
      7.459961
      NaN
      NaN
      NaN
      1.0
      0.0

      3
      female
      Lung Cancer
      Cancer
      under $11k
      white
      metastatic
      42.38498
      2
      75.0
      9.099609
      ...
      NaN
      NaN
      0.799927
      139.0
      NaN
      NaN
      NaN
      NaN
      0.0
      0.0

      4
      female
      ARF/MOSF w/Sepsis
      ARF/MOSF
      NaN
      white
      no
      79.88495
      1
      59.0
      13.500000
      ...
      NaN
      NaN
      0.799927
      143.0
      7.509766
      NaN
      NaN
      NaN
      NaN
      2.0

5 rows × 24 columns

      event
      time

      0
      0
      2029

      1
      1
      4

      2
      1
      47

      3
      1
      133

      4
      0
      2029

### 1.2. Preprocess the Data

```python
import numpy as np
from sklearn.model_selection import train_test_split

# Split the SUPPORT data into training, validation, and test data
x_tr, x_te, y_tr, y_te = train_test_split(features, outcomes, test_size=0.2, random_state=1)
x_tr, x_val, y_tr, y_val = train_test_split(x_tr, y_tr, test_size=0.25, random_state=1)

print(f'Number of training data points: {len(x_tr)}')
print(f'Number of validation data points: {len(x_val)}')
print(f'Number of test data points: {len(x_te)}')
```

    Number of training data points: 5463
    Number of validation data points: 1821
    Number of test data points: 1821

```python
from auton_survival.preprocessing import Preprocessor

# Fit the imputer and scaler to the training data and transform the training, validation and test data
preprocessor = Preprocessor(cat_feat_strat='ignore', num_feat_strat= 'mean')
transformer = preprocessor.fit(features, cat_feats=cat_feats, num_feats=num_feats,
                                one_hot=True, fill_value=-1)
x_tr = transformer.transform(x_tr)
x_val = transformer.transform(x_val)
x_te = transformer.transform(x_te)
```

## 2. Cox Proportional Hazards (CPH)

CPH [2]  model assumes that individuals across the population have constant proportional hazards overtime. In this model, the estimator of the survival function conditional on $X, S(·|X) , P(T > t|X)$, is assumed to have constant proportional hazard. Thus, the relative proportional hazard between individuals is constant across time.

*For full details on CPH, please refer to the following paper*:

[2] [Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological).](https://www.jstor.org/stable/2985181)

### 2.1. Fit CPH Model

```python
from auton_survival.estimators import SurvivalModel
from auton_survival.metrics import survival_regression_metric
from sklearn.model_selection import ParameterGrid

# Define parameters for tuning the model
param_grid = {'l2' : [1e-3, 1e-4]}
params = ParameterGrid(param_grid)

# Define the times for tuning the model hyperparameters and for evaluating the model
times = np.quantile(y_tr['time'][y_tr['event']==1], np.linspace(0.1, 1, 10)).tolist()

# Perform hyperparameter tuning
models = []
for param in params:
    model = SurvivalModel('cph', random_seed=2, l2=param['l2'])

    # The fit method is called to train the model
    model.fit(x_tr, y_tr)

    # Obtain survival probabilities for validation set and compute the Integrated Brier Score
    predictions_val = model.predict_survival(x_val, times)
    metric_val = survival_regression_metric('ibs', y_tr, y_val, predictions_val, times)
    models.append([metric_val, model])

# Select the best model based on the mean metric value computed for the validation set
metric_vals = [i[0] for i in models]
first_min_idx = metric_vals.index(min(metric_vals))
model = models[first_min_idx][1]
```

### 2.2. Evaluate CPH Model

```python
from estimators_demo_utils import plot_performance_metrics

# Obtain survival probabilities for test set
predictions_te = model.predict_survival(x_te, times)

# Compute the Brier Score and time-dependent concordance index for the test set to assess model performance
results = dict()
results['Brier Score'] = survival_regression_metric('brs', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
results['Concordance Index'] = survival_regression_metric('ctd', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
plot_performance_metrics(results, times)
```

## 3. Deep Cox Proportional Hazards (DCPH)

DCPH [2], [3] is an extension to the CPH model. DCPH involves modeling the proportional hazard ratios over the individuals with Deep Neural Networks allowing the ability to learn non linear hazard ratios.

*For full details on DCPH models, Faraggi-Simon Net (FSN) and DeepSurv, please refer to the following papers*:

[2] [Faraggi, David, and Richard Simon. "A neural network model for survival data." Statistics in medicine 14.1 (1995): 73-82.](https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.4780140108)

[3] [Katzman, Jared L., et al. "DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network." BMC medical research methodology 18.1 (2018): 1-12.](https://arxiv.org/abs/1606.00931v3)

### 3.1. Fit DCPH Model

```python
from auton_survival.estimators import SurvivalModel
from auton_survival.metrics import survival_regression_metric
from sklearn.model_selection import ParameterGrid

# Define parameters for tuning the model
param_grid = {'bs' : [100, 200],
              'learning_rate' : [ 1e-4, 1e-3],
              'layers' : [ [100], [100, 100] ]
             }

params = ParameterGrid(param_grid)

# Define the times for tuning the model hyperparameters and for evaluating the model
times = np.quantile(y_tr['time'][y_tr['event']==1], np.linspace(0.1, 1, 10)).tolist()

# Perform hyperparameter tuning
models = []
for param in params:
    model = SurvivalModel('dcph', random_seed=0, bs=param['bs'], learning_rate=param['learning_rate'], layers=param['layers'])

    # The fit method is called to train the model
    model.fit(x_tr, y_tr)

    # Obtain survival probabilities for validation set and compute the Integrated Brier Score
    predictions_val = model.predict_survival(x_val, times)
    metric_val = survival_regression_metric('ibs', y_tr, y_val, predictions_val, times)
    models.append([metric_val, model])

# Select the best model based on the mean metric value computed for the validation set
metric_vals = [i[0] for i in models]
first_min_idx = metric_vals.index(min(metric_vals))
model = models[first_min_idx][1]
```

    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:03<00:00, 15.29it/s]
     32%|███████████████████████████████████▌                                                                           | 16/50 [00:01<00:02, 12.67it/s]
     64%|███████████████████████████████████████████████████████████████████████                                        | 32/50 [00:03<00:02,  8.56it/s]
     20%|██████████████████████▏                                                                                | 10/50 [00:01<00:04,  8.21it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:02<00:00, 24.79it/s]
     50%|███████████████████████████████████████████████████████▌                                                       | 25/50 [00:01<00:01, 19.35it/s]
     82%|███████████████████████████████████████████████████████████████████████████████████████████                    | 41/50 [00:02<00:00, 16.10it/s]
     16%|█████████████████▉                                                                                | 8/50 [00:00<00:03, 13.96it/s]

### 3.2. Evaluate DCPH Model

Compute the Brier Score and time-dependent concordance index for the test set. See notebook introduction for more details.

```python
from estimators_demo_utils import plot_performance_metrics

# Obtain survival probabilities for test set
predictions_te = model.predict_survival(x_te, times)

# Compute the Brier Score and time-dependent concordance index for the test set to assess model performance
results = dict()
results['Brier Score'] = survival_regression_metric('brs', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
results['Concordance Index'] = survival_regression_metric('ctd', outcomes_train=y_tr, outcomes_test=y_te,
                                                            predictions=predictions_te, times=times)
plot_performance_metrics(results, times)
```

## 4. Deep Survival Machines (DSM)

DSM [5] is a fully parametric approach to modeling the event time distribution as a fixed size mixture over Weibull or Log-Normal distributions. The individual mixture distributions are parametrized with neural networks to learn complex non-linear representations of the data.

Figure A: DSM works by modeling the conditional distribution $P(T |X = x)$ as a mixture over $k$ well-defined, parametric distributions. DSM generates representation of the individual covariates, $x$, using a deep multilayer perceptron followed by a softmax over mixture size, $k$. This representation then interacts with the additional set of parameters,  to determine the mixture weights $w$ and the parameters of each of $k$ underlying survival distributions $\{\eta_k, \beta_k\}^K_{k=1}$. The final individual survival distribution for the event time, $T$, is a weighted average over these $K$ distributions.

*For full details on Deep Survival Machines (DSM), please refer to the following paper*:

[5] [Chirag Nagpal, Xinyu Li, and Artur Dubrawski. Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks. 2020.](https://arxiv.org/abs/2003.01176)

### 4.1. Fit DSM Model

```python
from auton_survival.estimators import SurvivalModel
from auton_survival.metrics import survival_regression_metric
from sklearn.model_selection import ParameterGrid

# Define parameters for tuning the model
param_grid = {'layers' : [[100], [100, 100], [200]],
              'distribution' : ['Weibull', 'LogNormal'],
              'max_features' : ['sqrt', 'log2']
             }

params = ParameterGrid(param_grid)

# Define the times for tuning the model hyperparameters and for evaluating the model
times = np.quantile(y_tr['time'][y_tr['event']==1], np.linspace(0.1, 1, 10)).tolist()

# Perform hyperparameter tuning
models = []
for param in params:
    model = SurvivalModel('dsm', random_seed=0, layers=param['layers'], distribution=param['distribution'], max_features=param['max_features'])

    # The fit method is called to train the model
    model.fit(x_tr, y_tr)

    # Obtain survival probabilities for validation set and compute the Integrated Brier Score
    predictions_val = model.predict_survival(x_val, times)
    metric_val = survival_regression_metric('ibs', y_tr, y_val, predictions_val, times)
    models.append([metric_val, model])

# Select the best model based on the mean metric value computed for the validation set
metric_vals = [i[0] for i in models]
first_min_idx = metric_vals.index(min(metric_vals))
model = models[first_min_idx][1]
```

     18%|██████████████████▉                                                                                | 1799/10000 [00:04<00:20, 400.33it/s]
     90%|████████████████████████████████████████████████████████████████████████████████████████████████████▊           | 9/10 [00:01<00:00,  5.31it/s]
     18%|██████████████████▉                                                                                | 1799/10000 [00:04<00:22, 369.96it/s]
     90%|████████████████████████████████████████████████████████████████████████████████████████████████████▊           | 9/10 [00:01<00:00,  4.96it/s]
     18%|██████████████████▉                                                                                | 1799/10000 [00:05<00:23, 350.53it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.25it/s]
     18%|██████████████████▉                                                                                | 1799/10000 [00:04<00:21, 376.50it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.96it/s]
     18%|██████████████████▉                                                                                | 1799/10000 [00:04<00:22, 364.78it/s]
     90%|████████████████████████████████████████████████████████████████████████████████████████████████████▊           | 9/10 [00:02<00:00,  4.01it/s]
     18%|██████████████████▉                                                                                | 1799/10000 [00:05<00:24, 340.31it/s]
     90%|████████████████████████████████████████████████████████████████████████████████████████████████████▊           | 9/10 [00:01<00:00,  4.63it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:03<00:21, 399.50it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.22it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:03<00:22, 393.39it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.22it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:03<00:23, 368.06it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.61it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:03<00:23, 372.02it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  4.72it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:02<00:16, 516.68it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.32it/s]
     12%|████████████▊                                                                                | 1224/10000 [00:03<00:22, 391.68it/s]
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  5.24it/s]

### 4.2. Evaluate DSM Model

Compute the Brier Score and time-dependent concordance index for the test set. See notebook introduction for more details.

```python
from estimators_demo_utils import plot_performance_metrics

# Obtain survival probabilities for test set
predictions_te = model.predict_survival(x_te, times)

# Compute the Brier Score and time-dependent concordance index for the test set to assess model performance
results = dict()
results['Brier Score'] = survival_regression_metric('brs', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
results['Concordance Index'] = survival_regression_metric('ctd', outcomes_train=y_tr, outcomes_test=y_te,
                                                            predictions=predictions_te, times=times)
plot_performance_metrics(results, times)
```

## 5. Deep Cox Mixtures (DCM)

DCM [2] generalizes the proportional hazards assumption via a mixture model, by assuming that there are latent groups and within each, the proportional hazards assumption holds. DCM allows the hazard ratio in each latent group, as well as the latent group membership, to be flexibly modeled by a deep neural network.

Figure B: DCM works by generating representation of the individual covariates, $x$, using an encoding neural network. The output representation, $xe$, then interacts with linear functions, $f$ and $g$, that determine the proportional hazards within each cluster $Z ∈ {1, 2, ...K}$ and the mixing weights $P(Z|X)$ respectively. For each cluster, baseline survival rates $Sk(t)$ are estimated non-parametrically. The final individual survival curve $S(t|x)$ is an average over the cluster specific individual survival curves weighted by the mixing probabilities $P(Z|X = x)$.

*For full details on Deep Cox Mixtures (DCM), please refer to the following paper*:

[2] [Nagpal, C., Yadlowsky, S., Rostamzadeh, N., and Heller, K. (2021c). Deep cox mixtures for survival regression. In
Machine Learning for Healthcare Conference, pages 674–708. PMLR.](https://arxiv.org/abs/2101.06536)

### 5.1. Fit DCM Model

```python
from auton_survival.estimators import SurvivalModel
from auton_survival.metrics import survival_regression_metric
from sklearn.model_selection import ParameterGrid

# Define parameters for tuning the model
param_grid = {'k' : [2, 3],
              'learning_rate' : [1e-3, 1e-4],
              'layers' : [[100], [100, 100]]
             }

params = ParameterGrid(param_grid)

# Define the times for tuning the model hyperparameters and for evaluating the model
times = np.quantile(y_tr['time'][y_tr['event']==1], np.linspace(0.1, 1, 10)).tolist()

# Perform hyperparameter tuning
models = []
for param in params:
    model = SurvivalModel('dcm', random_seed=7, k=param['k'], learning_rate=param['learning_rate'], layers=param['layers'])

    # The fit method is called to train the model
    model.fit(x_tr, y_tr)

    # Obtain survival probabilities for validation set and compute the Integrated Brier Score
    predictions_val = model.predict_survival(x_val, times)
    metric_val = survival_regression_metric('ibs', y_tr, y_val, predictions_val, times)
    models.append([metric_val, model])

# Select the best model based on the mean metric value computed for the validation set
metric_vals = [i[0] for i in models]
first_min_idx = metric_vals.index(min(metric_vals))
model = models[first_min_idx][1]
```

      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
     14%|███████████████▋                                                                                | 7/50 [00:02<00:14,  3.04it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     32%|███████████████████████████████████▌                                                                           | 16/50 [00:05<00:12,  2.75it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:58: RuntimeWarning: invalid value encountered in power
      return spl(ts)**risks
    C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:53: RuntimeWarning: invalid value encountered in power
      s0ts = (-risks)*(spl(ts)**(risks-1))
     34%|█████████████████████████████████████▋                                                                         | 17/50 [00:06<00:12,  2.64it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
     14%|███████████████▋                                                                                | 7/50 [00:02<00:14,  3.02it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     72%|███████████████████████████████████████████████████████████████████████████████▉                               | 36/50 [00:12<00:04,  2.90it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
      6%|██████▋                                                                                | 3/50 [00:01<00:19,  2.41it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     54%|███████████████████████████████████████████████████████████▉                                                   | 27/50 [00:12<00:10,  2.18it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
     30%|█████████████████████████████████▎                                                                             | 15/50 [00:06<00:15,  2.22it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:22<00:00,  2.27it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
      6%|██████▋                                                                                | 3/50 [00:01<00:21,  2.20it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     52%|█████████████████████████████████████████████████████████▋                                                     | 26/50 [00:11<00:10,  2.26it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:58: RuntimeWarning: invalid value encountered in power
      return spl(ts)**risks
    C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:53: RuntimeWarning: invalid value encountered in power
      s0ts = (-risks)*(spl(ts)**(risks-1))
     70%|█████████████████████████████████████████████████████████████████████████████▋                                 | 35/50 [00:15<00:06,  2.21it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
      4%|████▍                                                                                | 2/50 [00:00<00:15,  3.03it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     72%|███████████████████████████████████████████████████████████████████████████████▉                               | 36/50 [00:15<00:07,  1.84it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:58: RuntimeWarning: invalid value encountered in power
      return spl(ts)**risks
    C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:53: RuntimeWarning: invalid value encountered in power
      s0ts = (-risks)*(spl(ts)**(risks-1))
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:23<00:00,  2.17it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
      2%|██▏                                                                                | 1/50 [00:00<00:27,  1.78it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
     78%|██████████████████████████████████████████████████████████████████████████████████████▌                        | 39/50 [00:20<00:05,  1.91it/s]
      0%|                                                                                | 0/50 [00:00<?, ?it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: invalid value encountered in log
      probs = gates+np.log(event_probs)
     20%|██████████████████████▏                                                                                | 10/50 [00:04<00:21,  1.83it/s]C:\Users\Willa Potosnak\OneDrive\Documents\CMU Research\CMU_Projects\auton-survival\examples\..\auton_survival\models\dcm\dcm_utilities.py:105: RuntimeWarning: divide by zero encountered in log
      probs = gates+np.log(event_probs)
    100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:22<00:00,  2.27it/s]

### 5.2. Evaluate DCM Model

Compute the Brier Score and time-dependent concordance index for the test set. See notebook introduction for more details.

```python
from estimators_demo_utils import plot_performance_metrics

# Obtain survival probabilities for test set
predictions_te = model.predict_survival(x_te, times)

# Compute the Brier Score and time-dependent concordance index for the test set to assess model performance
results = dict()
results['Brier Score'] = survival_regression_metric('brs', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
results['Concordance Index'] = survival_regression_metric('ctd', outcomes_train=y_tr, outcomes_test=y_te,
                                                            predictions=predictions_te, times=times)
plot_performance_metrics(results, times)
```

## 6. Random Survival Forests (RSF)

RSF [4] is an extension of Random Forests to the survival settings where risk scores are computed by creating Nelson-Aalen estimators in the splits induced by the Random Forest.

We observe that performance of the Random Survival Forest model, especially in terms of calibration is strongly influenced by the choice for the hyperparameters for the number of features considered at each split and the minimum number of data samples to continue growing a tree. We thus advise carefully tuning these hyper-parameters while benchmarking RSF.

*For full details on Random Survival Forests (RSF), please refer to the following paper*:

[4] [Hemant Ishwaran et al. Random survival forests. The annals of applied statistics, 2(3):841–860, 2008.](https://arxiv.org/abs/0811.1645)

### 6.1. Fit RSF Model

```python
from auton_survival.estimators import SurvivalModel
from auton_survival.metrics import survival_regression_metric
from sklearn.model_selection import ParameterGrid

# Define parameters for tuning the model
param_grid = {'n_estimators' : [100, 300],
              'max_depth' : [3, 5],
              'max_features' : ['sqrt', 'log2']
             }

params = ParameterGrid(param_grid)

# Define the times for tuning the model hyperparameters and for evaluating the model
times = np.quantile(y_tr['time'][y_tr['event']==1], np.linspace(0.1, 1, 10)).tolist()

# Perform hyperparameter tuning
models = []
for param in params:
    model = SurvivalModel('rsf', random_seed=8, n_estimators=param['n_estimators'], max_depth=param['max_depth'], max_features=param['max_features'])

    # The fit method is called to train the model
    model.fit(x_tr, y_tr)

    # Obtain survival probabilities for validation set and compute the Integrated Brier Score
    predictions_val = model.predict_survival(x_val, times)
    metric_val = survival_regression_metric('ibs', y_tr, y_val, predictions_val, times)
    models.append([metric_val, model])

# Select the best model based on the mean metric value computed for the validation set
metric_vals = [i[0] for i in models]
first_min_idx = metric_vals.index(min(metric_vals))
model = models[first_min_idx][1]
```

### 6.2. Evaluate RSF Model

Compute the Brier Score and time-dependent concordance index for the test set. See notebook introduction for more details.

```python
from estimators_demo_utils import plot_performance_metrics

# Obtain survival probabilities for test set
predictions_te = model.predict_survival(x_te, times)

# Compute the Brier Score and time-dependent concordance index for the test set to assess model performance
results = dict()
results['Brier Score'] = survival_regression_metric('brs', outcomes_train=y_tr, outcomes_test=y_te,
                                                    predictions=predictions_te, times=times)
results['Concordance Index'] = survival_regression_metric('ctd', outcomes_train=y_tr, outcomes_test=y_te,
                                                            predictions=predictions_te, times=times)
plot_performance_metrics(results, times)
```

```python

```

examples/Survival Regression with Auton-Survival.ipynb


Blankenbach Benchmark Case 1
======

Steady isoviscous thermal convection
----

Two-dimensional, incompressible, bottom heated, steady isoviscous thermal convection in a 1 x 1 box, see case 1 of Blankenbach *et al.* 1989 for details.

**This example introduces:**
1. Loading/Saving variables to disk.
2. Defining analysis tools.
3. Finding a steady state.

**Keywords:** Stokes system, advective diffusive systems, analysis tools

**References**

B. Blankenbach, F. Busse, U. Christensen, L. Cserepes, D. Gunkel, U. Hansen, H. Harder, G. Jarvis, M. Koch, G. Marquart, D. Moore, P. Olson, H. Schmeling and T. Schnaubelt. A benchmark comparison for mantle convection codes. Geophysical Journal International, 98, 1, 23–38, 1989
http://onlinelibrary.wiley.com/doi/10.1111/j.1365-246X.1989.tb05511.x/abstract

```python
import underworld as uw
from underworld import function as fn
import glucifer
import math
import numpy as np
```

Setup parameters
-----

```python
boxHeight = 1.0
boxLength = 1.0
# Set grid resolution.
res = 128
# Set max & min temperautres
tempMin = 0.0
tempMax = 1.0
```

Choose which Rayleigh number, see case 1 of Blankenbach *et al.* 1989 for details.

```python
case = "a"
if(case=="a"):
    Ra=1.e4
    eta0=1.e23
elif(case=="b"):
    Ra=1.e5
    eta0=1.e22
else:
    Ra=1.e6
    eta0=1.e21
```

Set input and output file directory

```python
inputPath  = 'input/1_03_BlankenbachBenchmark/'
outputPath = 'output/'
# Make output directory if necessary.
if uw.mpi.rank==0:
    import os
    if not os.path.exists(outputPath):
        os.makedirs(outputPath)
```

Create mesh and variables
------

```python
mesh = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                 elementRes  = (res, res),
                                 minCoord    = (0., 0.),
                                 maxCoord    = (boxLength, boxHeight))

velocityField       = mesh.add_variable(         nodeDofCount=2 )
pressureField       = mesh.subMesh.add_variable( nodeDofCount=1 )
temperatureField    = mesh.add_variable(         nodeDofCount=1 )
temperatureDotField = mesh.add_variable(         nodeDofCount=1 )

# initialise velocity, pressure and temperatureDot field
velocityField.data[:]       = [0.,0.]
pressureField.data[:]       = 0.
temperatureField.data[:]    = 0.
temperatureDotField.data[:] = 0.
```

Set up material parameters and functions
-----

Set values and functions for viscosity, density and buoyancy force.

```python
# Set a constant viscosity.
viscosity = 1.

# Create our density function.
densityFn = Ra * temperatureField

# Define our vertical unit vector using a python tuple (this will be automatically converted to a function).
z_hat = ( 0.0, 1.0 )

# A buoyancy function.
buoyancyFn = densityFn * z_hat
```

Set initial temperature field
-----
The initial temperature field can be loaded from a pre-run steady state data set ( ``LoadFromFile = True`` ) or set to a sinusodial perterbation ( ``LoadFromFile = False`` ).

```python
# Steady state temperature field to be loaded from data file.
LoadFromFile = True
```

**If loading steady state data set**

Data is stored in h5 format from a 64\*64 grid resolution model.  Data has been saved for 3 different Rayleigh numbers, $Ra = 10^4$, $10^5$ or $10^6$.

Once loaded the data will need to be re-meshed onto a new grid, unless the new resolution is also 64\*64.

For more information on using meshes see the user guide.

```python
if(LoadFromFile == True):
    # Setup mesh and temperature field for 64*64 data file.
    mesh64 = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                       elementRes  = (64, 64),
                                       minCoord    = (0., 0.),
                                       maxCoord    = (boxLength, boxHeight),
                                       partitioned = False )
    temperatureField64  = mesh64.add_variable( nodeDofCount=1 )

    # read in saved steady state temperature field data
    if( case == "a" ):
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e4.h5')
        print('Loading 64*64 for Ra = 1e4')
    elif( case == "b" ):
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e5.h5')
        print('Loading 64*64 for Ra = 1e5')
    else:
        temperatureField64.load(inputPath+'tempfield_inp_64_Ra1e6.h5')
        print('Loading 64*64 for Ra = 1e6')

    if( res==64 ): # no remeshing needed, copy directly
        temperatureField.data[:] = temperatureField64.data[:]
    else: # remeshing needed
        temperatureField.data[:] = temperatureField64.evaluate(mesh)
```

    Loading 64*64 for Ra = 1e4

**If using sinusodial perturbation**

```python
if(LoadFromFile == False):
    temperatureField.data[:] = 0.
    pertStrength = 0.1
    deltaTemp = tempMax - tempMin
    for index, coord in enumerate(mesh.data):
        pertCoeff = math.cos( math.pi * coord[0]/boxLength ) * math.sin( math.pi * coord[1]/boxLength )
        temperatureField.data[index] = tempMin + deltaTemp*(boxHeight - coord[1]) + pertStrength * pertCoeff
        temperatureField.data[index] = max(tempMin, min(tempMax, temperatureField.data[index]))
```

**Show initial temperature field**

```python
fig = glucifer.Figure()
fig.append( glucifer.objects.Surface(mesh, temperatureField) )
fig.show()
```

Create boundary conditions
----------

Set temperature boundary conditions on the bottom ( ``MinJ`` ) and top ( ``MaxJ`` ).

```python
for index in mesh.specialSets["MinJ_VertexSet"]:
    temperatureField.data[index] = tempMax
for index in mesh.specialSets["MaxJ_VertexSet"]:
    temperatureField.data[index] = tempMin
```

Construct sets for the both horizontal and vertical walls. Combine the sets of vertices to make the ``I`` (left and right side walls) and ``J`` (top and bottom walls) sets.

```python
iWalls = mesh.specialSets["MinI_VertexSet"] + mesh.specialSets["MaxI_VertexSet"]
jWalls = mesh.specialSets["MinJ_VertexSet"] + mesh.specialSets["MaxJ_VertexSet"]

freeslipBC = uw.conditions.DirichletCondition( variable        = velocityField,
                                               indexSetsPerDof = (iWalls, jWalls) )
tempBC     = uw.conditions.DirichletCondition( variable        = temperatureField,
                                               indexSetsPerDof = (jWalls,) )
```

System setup
-----

**Setup a Stokes system**

```python
stokes = uw.systems.Stokes(    velocityField = velocityField,
                               pressureField = pressureField,
                               conditions    = [freeslipBC,],
                               fn_viscosity  = viscosity,
                               fn_bodyforce  = buoyancyFn )
# get the default stokes equation solver
solver = uw.systems.Solver( stokes )
```

**Create an advection diffusion system**

```python
advDiff = uw.systems.AdvectionDiffusion( phiField       = temperatureField,
                                         phiDotField    = temperatureDotField,
                                         velocityField  = velocityField,
                                         fn_diffusivity = 1.0,
                                         conditions     = [tempBC,] )
```

Analysis tools
-----

**Nusselt number**

The Nusselt number is the ratio between convective and conductive heat transfer

\\[
Nu = -h \frac{ \int_0^l \partial_z T (x, z=h) dx}{ \int_0^l T (x, z=0) dx}
\\]

```python
nuTop    = uw.utils.Integral( fn=temperatureField.fn_gradient[1],
                              mesh=mesh, integrationType='Surface',
                              surfaceIndexSet=mesh.specialSets["MaxJ_VertexSet"])

nuBottom = uw.utils.Integral( fn=temperatureField,
                              mesh=mesh, integrationType='Surface',
                              surfaceIndexSet=mesh.specialSets["MinJ_VertexSet"])
```

```python
nu = - nuTop.evaluate()[0]/nuBottom.evaluate()[0]
print('Nusselt number = {0:.6f}'.format(nu))
```

    Nusselt number = 4.856902

**RMS velocity**

The root mean squared velocity is defined by intergrating over the entire simulation domain via

\\[
\begin{aligned}
v_{rms}  =  \sqrt{ \frac{ \int_V (\mathbf{v}.\mathbf{v}) dV } {\int_V dV} }
\end{aligned}
\\]

where $V$ denotes the volume of the box.

```python
intVdotV = uw.utils.Integral( fn.math.dot( velocityField, velocityField ), mesh )

vrms = math.sqrt( intVdotV.evaluate()[0] )
print('Initial vrms = {0:.3f}'.format(vrms))
```

    Initial vrms = 0.000

Main simulation loop
-----

If the initial conditions are loaded from file then this loop will only take a single step. If you would like to run the entire simulation from a small perturbation then change the ``LoadFromFile`` variable above to equal ``False``. Warning: the simulation will take a long time to get to steady state.

```python
#initialise time, step, output arrays
time = 0.
step = 0
timeVal = []
vrmsVal = []

# starting from steady state == True
if(LoadFromFile == True):
    step_end = 1
else:
    step_end = 5000

# output frequency
step_output = max(1,min(100, step_end/10))
epsilon    = 1.e-8

velplotmax = 0.0
nuLast     = -1.0
```

```python
# define an update function
def update():
    # Determining the maximum timestep for advancing the a-d system.
    dt = advDiff.get_max_dt()
    # Advect using this timestep size.
    advDiff.integrate(dt)
    return time+dt, step+1
```

```python
# Perform steps.
while step<=step_end:
    # Solving the Stokes system.
    solver.solve()

    # Calculate & store the RMS velocity and Nusselt number.
    vrms = math.sqrt( intVdotV.evaluate()[0] )
    nu   = - nuTop.evaluate()[0]/nuBottom.evaluate()[0]
    vrmsVal.append(vrms)
    timeVal.append(time)
    velplotmax = max(vrms, velplotmax)

    # print output statistics
    if step%(step_end/step_output) == 0:
        if(uw.mpi.rank==0):
            print('steps = {0:6d}; time = {1:.3e}; v_rms = {2:.3f}; Nu = {3:.3f}; Rel change = {4:.3e}'
              .format(step, time, vrms, nu, abs((nu - nuLast)/nu)))
    # Check loop break conditions.
    if(abs((nu - nuLast)/nu) < epsilon):
        if(uw.mpi.rank==0):
            print('steps = {0:6d}; time = {1:.3e}; v_rms = {2:.3f}; Nu = {3:.3f}; Rel change = {4:.3e}'
              .format(step, time, vrms, nu, abs((nu - nuLast)/nu)))
        break
    nuLast = nu

    # update
    time, step = update()
```

    steps =      0; time = 0.000e+00; v_rms = 42.869; Nu = 4.857; Rel change = 1.206e+00
    steps =      1; time = 3.052e-05; v_rms = 42.869; Nu = 4.864; Rel change = 1.416e-03

Post analysis
-----

**Benchmark values**

The time loop above outputs $v_{rms}$ and $Nu$ as general statistics for the system. For comparison, the benchmark values for the RMS velocity and Nusselt number are shown below for different Rayleigh numbers. All benchmark values shown below were determined in Blankenbach *et al.* 1989 by extroplation of numerical results.

| $Ra$          | $v_{rms}$   | $Nu$  |   $q_1$  | $q_2$  |
| ------------- |:-------------:|:-----:|:-----:|:-----:|
| 10$^4$ | 42.865 |  4.884 | 8.059 | 0.589 |
| 10$^5$ | 193.215 | 10.535 | 19.079 | 0.723 |
| 10$^6$ | 833.990 | 21.972 | 45.964 | 0.877 |

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(nu,4.884,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected Nusselt number.")
    if not np.isclose(vrms,42.865,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected Nusselt number.")
```

**Resulting pressure field**

Use the same method as above to plot the new temperature field. This can also be used to plot the pressure field, or any other data structures of interest.

```python
figtemp = glucifer.Figure()
figtemp.append( glucifer.objects.Surface( mesh, pressureField ) )
figtemp.show()
```

**Plot the velocity vector field**

For this example the velocity field is interesting to see. This is visualised in two ways, firstly plotting a surface colour map of the velocity magnitude, and secondly the velocity vectors at points on the mesh. For aesthetics the vector arrows are scaled by a little more than the maximum $v_{rms}$ value found in the time loop above.

```python
fig2 = glucifer.Figure()
velmagfield = uw.function.math.sqrt( uw.function.math.dot( velocityField, velocityField ) )
fig2.append( glucifer.objects.VectorArrows(mesh, velocityField/(2.5*velplotmax), arrowHead=0.2, scaling=0.1) )
fig2.append( glucifer.objects.Surface(mesh, temperatureField) )
fig2.show()
```

Parallel friendly post analysis
----

When running underworld in parallel the data of each mesh variable is spread across all the processors. However often we will want to calculate a quantity based on data at specific points that may not all be on the same processor.

A solution is presented here which consists of saving the data from all processors to file, then reloading the mesh variable data using a new non-partitioned mesh. This enables all the data to be available to each processor. We will the carry out the post analysis using the first processor.

**Save temperature, pressure and velocity data**

Save the basic mesh variable data to files using the HDF5 format. This is the same file type as is loaded above.

```python
mesh.save(outputPath+"mesh.h5")
temperatureField.save(outputPath+'tempfield.h5')
pressureField.save(outputPath+'presfield.h5')
velocityField.save(outputPath+'velfield.h5')
```

    <underworld.utils._utils.SavedFileData at 0x7f0f5bb66e90>

**Construct new mesh and variable on non-partitioned mesh**

Read saved mesh variable data into a new mesh variable where the information is not partitioned across multiple processors. This means that we can use a single processor to access all the data and calculate some quantities of interest.

```python
# build a non-partitioned mesh with same box size
mesh0 = uw.mesh.FeMesh_Cartesian( elementType = ("Q1/dQ0"),
                                  elementRes  = (res, res),
                                  minCoord    = (0., 0.),
                                  maxCoord    = (boxLength, boxHeight),
                                  partitioned = False )
# load previous mesh coordinate data onto new non-partitioned mesh
mesh0.load(outputPath+'mesh.h5')

# load T, P and V data onto the new mesh
# note that pressure is always on the submesh
temperatureField0  = mesh0.add_variable( nodeDofCount=1 )
pressureField0     = mesh0.subMesh.add_variable( nodeDofCount=1 )
velocityField0     = mesh0.add_variable( nodeDofCount=2 )
temperatureField0.load(outputPath+"tempfield.h5")
pressureField0.load(outputPath+"presfield.h5")
velocityField0.load(outputPath+"velfield.h5")
```

**Temperature gradient**

The final benchmarks in the Blankenbach paper involve the temperature gradient in the vertical direction ($\frac{\partial T}{\partial z}$). This is easy to find using the underworld functions, as shown below.

```python
if(uw.mpi.rank==0):
    tempgradField = temperatureField0.fn_gradient
    vertTGradField = - boxHeight * tempgradField[1] / tempMax # scaled for direct benchmarking below
```

**More benchmark values**

The vertical temperature gradient (above) is set up to be non-dimensional as per Blankenbach et al 1989. To compare to the benchmark values in their work the gradient is compared at the corners of the simulation box: $q_1$ at $x = 0$, $z = h$; $q_2$ at $x = l$, $z = h$; $q_3$ at $x = l$, $z = 0$; $q_4$ at $x = 0$, $z = 0$. Where $h$ = Box_Height and $l$ = Box_Length and the non-dimensional gradient field is given by
\\[
q = \frac{-h}{\Delta T} \left( \frac{\partial T}{\partial z} \right)
\\]

Provided the simulation is run to steady-state with sufficent resolution then the $q$ values should be close to the benchmark values given again below for different Rayleigh numbers.

| $Ra$          |    $q_1$  | $q_2$  |
| ------------- |:-----:|:-----:|
| 10$^4$ | 8.059 | 0.589 |
| 10$^5$ | 19.079 | 0.723 |
| 10$^6$ | 45.964 | 0.877 |

```python
if(uw.mpi.rank==0):
    q1 = vertTGradField.evaluate( (0., boxHeight))[0][0]
    q2 = vertTGradField.evaluate( (boxLength, boxHeight))[0][0]
    q3 = vertTGradField.evaluate( (boxLength, 0.))[0][0]
    q4 = vertTGradField.evaluate( (0., 0.))[0][0]

    print('Rayleigh number = {0:.1e}'.format(Ra))
    print('q1 = {0:.3f}; q2 = {1:.3f}'.format(q1, q2))
    print('q3 = {0:.3f}; q4 = {1:.3f}'.format(q3, q4))
```

    Rayleigh number = 1.0e+04
    q1 = 8.021; q2 = 0.589
    q3 = 8.021; q4 = 0.589

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(q1,8.020,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected q1.")
    if not np.isclose(q2,0.589,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected q2.")
```

**Save time and rms values**

The following command uses the ``numpy`` package save to text file function to output all $v_{RMS}$ values as a function of time. This is particularly useful if you have run the simulation from the perturbed initial condition rather than the saved data file, as you can see the system coming to steady state.

The format for this text file is:

    timeVal[0], vrmsVal[0]
    timeVal[1], vrmsVal[1]
    ...
    timeVal[N], vrmsVal[N]

```python
if(uw.mpi.rank==0):
    np.savetxt(outputPath+'vrms.txt', np.c_[timeVal, vrmsVal], header="Time, VRMS" )
```

**Calculate stress values for benchmark comparison**

Determine stress field for whole box in dimensionless units (King 2009)
\begin{equation}
\tau_{ij} = \eta \frac{1}{2} \left[ \frac{\partial v_j}{\partial x_i} + \frac{\partial v_i}{\partial x_j}\right]
\end{equation}
which for vertical normal stress becomes
\begin{equation}
\tau_{zz} = \eta \frac{1}{2} \left[ \frac{\partial v_z}{\partial z} + \frac{\partial v_z}{\partial z}\right] = \eta \frac{\partial v_z}{\partial z}
\end{equation}
which is implemented for the whole box in the functions defined below.

```python
# get topography from non-partitioned stress tensor
if(uw.mpi.rank==0):
    stresstensorFn = 2.* stokes.fn_viscosity*fn.tensor.symmetric( velocityField0.fn_gradient ) - (1.,1.,0.)*pressureField0
    verticalStressFn  = stresstensorFn[1]
    stress_zz_top     = -verticalStressFn.evaluate(mesh0.specialSets["MaxJ_VertexSet"])

# subtract the average value for benchmark.
    mean_sigma_zz_top = np.mean(stress_zz_top)
    sigma_zz_top      = stress_zz_top - mean_sigma_zz_top
```

Dimensionalise the stress from the vertical normal stress at the top of the box (King 2009)

$$
    \sigma_{t} = \frac{\eta_0 \kappa}{\rho g h^2}\tau _{zz} \left( x, z=h\right)
$$

where all constants have been defined above. Finally calculate the topography, defined using $h = \sigma_{top} / (\rho g)$.

```python
# Set parameters in SI units
if(uw.mpi.rank==0):
    grav = 10        # m.s^-2
    height = 1.e6    # m
    rho  = 4.0e3     # g.m^-3
    kappa  = 1.0e-6  # m^2.s^-1

# dimensionalise
    dim_sigma_zz_top  = (eta0 * kappa / (height*height)) * sigma_zz_top

# find topography in [m]
    topography        = dim_sigma_zz_top / (rho * grav)
```

**Calculate x-coordinate at zero stress**

Calculate the zero point for the stress along the x-axis at the top of the box using the **interpolation function** from ``numpy``. Note that ``numpy`` requires that the first array input for ``np.interp`` must be increasing, so the negative of the topography is used.

```python
if(uw.mpi.rank==0):
    xCoordFn = fn.input()[0]
    x = xCoordFn.evaluate(mesh0.specialSets["MinJ_VertexSet"])
    xIntercept = np.interp(0.0,-1.0*topography[:, 0],x[:, 0])
```

**Topography comparison**

Topography of the top boundary calculated in the left and right corners as given in Table 9 of Blankenbach et al 1989.

| $Ra$          |    $\xi_1$  | $\xi_2$  |  $x$ ($\xi = 0$) |
| ------------- |:-----------:|:--------:|:--------------:|
| 10$^4$  | 2254.02   | -2903.23  | 0.539372          |
| 10$^5$  | 1460.99   | -2004.20  | 0.529330          |
| 10$^6$  | 931.96   | -1283.80  | 0.506490          |

```python
if(uw.mpi.rank==0):
    e1 = float(topography[0])
    e2 = float(topography[len(topography)-1])
    print('Rayleigh number = {0:.1e}'.format(Ra))
    print('Topography[x=0],[x=max] = {0:.2f}, {1:.2f}'.format(e1, e2))
    print('x(topo=0) = {0:.6f}'.format(xIntercept))
    # output a summary file with benchmark values (useful for parallel runs)
    np.savetxt(outputPath+'summary.txt', [Ra, e1, e2, xIntercept, q1, q2, q3, q4])
```

    Rayleigh number = 1.0e+04
    Topography[x=0],[x=max] = 2243.93, -2909.09
    x(topo=0) = 0.540372

```python
# Let's add a test to ensure things are working as expected
if case == "a":
    if not np.isclose(e1,2254.02,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected xi1.")
    if not np.isclose(e2,-2903.23,rtol=1.e-2):
        raise RuntimeError("Model did not produce the expected xi2.")
```


# 06_Business_Insights

In this section, we will expend upon the features used by the model and attempt to explain its significance as well as contributions to the pricing model.

Accordingly, in Section Four, we identified the following key features that that are strong predictors of housing price based upon a combination of feature engineering coupled with recursive feature elimination.

 $$
 \hat{y} = \beta_0 + \begin{align} \beta_1\textit{(age_since_built)} + \beta_2\textit{(Gr_Liv_Area)} + \beta_3\textit{(Total_Bsmt_SF)} + \beta_4\textit{(house_exter_score)} + \beta_j\textit{(Land_Contour)}_j
 \end{align}$$

 Where:

 $\textit{house_exter_score}$ = ['Overall Qual'] + ['Overall Cond'] +['Exter Qual'] + ['Exter Cond']

| Score  |
| --- |
|age_since_built |
|Total Bsmt SF |
|Land Contour_Lvl |
|house_exter_score |
|Gr Liv Area |
|Land Contour_Low |
|Land Contour_HLS |

```python
# model coefficients
prod_model_rfe
```

      feature
      coefficient

      0
      Total Bsmt SF
      15864.667337

      1
      Gr Liv Area
      30824.380459

      2
      age_since_built
      -20019.703415

      3
      house_exter_score
      25762.168495

      4
      Land Contour_HLS
      48572.704690

      5
      Land Contour_Low
      36453.726086

      6
      Land Contour_Lvl
      12958.893938

      7
      y_intercept
      167047.745662

## Import Libraries

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.ticker as ticker

%matplotlib inline
```

```python
rfe_columns = ['Total Bsmt SF', 'Gr Liv Area', 'age_since_built', 'house_exter_score',
       'Land Contour_HLS', 'Land Contour_Low', 'Land Contour_Lvl']
```

## Load in Training Set for Exploration

```python
train = pd.read_csv('./datasets/imputed_train.csv')
```

```python
new_train = train[(train['Total Bsmt SF']<4000) & (train['Total Bsmt SF']>0)]
```

```python
plt.figure(figsize=(15,10))
ax = sns.scatterplot(x='Total Bsmt SF',y='SalePrice',data=new_train,hue='Total Bsmt SF')
ax.set_title("Total Basement Area against Sale Prices", fontname='Helvetica', fontsize=18,loc='left')
ax.set_xlabel('Total Basement Area / SqFt',fontname='Helvetica',fontsize=12)
ax.set_ylabel('Sale Price / $',fontname='Helvetica',fontsize=12)
plt.savefig("./img/bsmt_area.png",dpi=300)
```

```python
plt.figure(figsize=(15,10))
ax = sns.scatterplot(x='age_since_built',y='SalePrice',data=new_train,hue='age_since_built')
ax.set_title("Building Age against Sale Prices", fontname='Helvetica', fontsize=18,loc='left')
ax.set_xlabel('Building Age / yr',fontname='Helvetica',fontsize=12)
ax.set_ylabel('Sale Price / $',fontname='Helvetica',fontsize=12)
plt.savefig("./img/building_age.png",dpi=300)
```

```python
train.columns
```

    Index(['Unnamed: 0', 'has_remod', 'Land Contour_HLS', 'Land Contour_Low',
           'Land Contour_Lvl', 'House Style_2Story', 'House Style_Split',
           'Central Air_Y', 'Bedroom AbvGr_1', 'Bedroom AbvGr_2',
           'Bedroom AbvGr_3', 'Bedroom AbvGr_>=4', 'TotRms AbvGrd_6',
           'TotRms AbvGrd_7', 'TotRms AbvGrd_<=4', 'TotRms AbvGrd_>=8',
           'Mo Sold_spring', 'Mo Sold_summer', 'Mo Sold_winter', 'Yr Sold_2007',
           'Yr Sold_2008', 'Yr Sold_2009', 'Yr Sold_2010', 'total_baths_1',
           'total_baths_2', 'total_baths_>=3', 'lot_area_frontage', 'Mas Vnr Area',
           'Total Bsmt SF', 'Gr Liv Area', 'Garage Area', 'outdoor_SF',
           'age_since_remodel', 'age_since_built', 'Heating QC',
           'house_exter_score', 'fireplace_score', 'garage_basement_score',
           'kitchen_score', 'SalePrice'],
          dtype='object')

```python
plt.figure(figsize=(15,10))
ax = sns.pointplot(x='house_exter_score',y='SalePrice',data=new_train)
ax.set_title("Housing Scores against Sale Prices", fontname='Helvetica', fontsize=18,loc='left')
ax.set_xlabel('Score',fontname='Helvetica',fontsize=12)
ax.set_ylabel('Sale Price / $',fontname='Helvetica',fontsize=12)
for ind, label in enumerate(ax.get_xticklabels()):
    if ind % 5 == 0:  # every 10th label is kept
        label.set_visible(True)
    else:
        label.set_visible(False)

plt.savefig("./img/housing_score.png",dpi=300)
```

```python
plt.figure(figsize=(15,10))
ax = sns.scatterplot(x='Gr Liv Area',y='SalePrice',data=new_train,hue='Gr Liv Area')
ax.set_title("Living Area against Sale Prices", fontname='Helvetica', fontsize=18,loc='left')
ax.set_xlabel('Living Area / Sqft',fontname='Helvetica',fontsize=12)
ax.set_ylabel('Sale Price / $',fontname='Helvetica',fontsize=12)
plt.savefig("./img/living_area.png",dpi=300)
```

```python
new_train_melt = new_train[['Land Contour_Low', 'Land Contour_HLS','Land Contour_Lvl','SalePrice']].melt(id_vars='SalePrice' ,value_vars=['Land Contour_Low', 'Land Contour_HLS','Land Contour_Lvl'])

```

```python
plt.figure(figsize=(15,10))
ax = sns.boxplot(x='variable',y='SalePrice',order=['Land Contour_Lvl','Land Contour_Low','Land Contour_HLS'],data=new_train_melt[new_train_melt['value']!=0])
ax.set_title("Land Contours and relationship to Sale Prices", fontname='Helvetica', fontsize=18,loc='left')
ax.set_xlabel('Score',fontname='Helvetica',fontsize=12)
ax.set_ylabel('Sale Price / $',fontname='Helvetica',fontsize=12)
plt.savefig("./img/contour_plot.png",dpi=300)

```

## Key Takeaways

## Conclusion And Recommendations

House age, Land Contours, Housing Scores as well as Gross floor areas are strong predictors of housing prices. Using these few variables, a prospective home seller can look into improving areas such as home quality, condition as well as look at expanding gross floor areas via careful remodelling of their homes.

To make this model location agnostic, we may impute features such as accessibility to the city (via distances) and crime rates which can affect buyer's judgement.


# Lead-Acid Models

We compare a standard porous-electrode model for lead-acid batteries with two asymptotic reductions. For a more in-depth introduction to PyBaMM models, see the [SPM notebook](./SPM.ipynb). Further details on the models can be found in [[1]](#ref)

[1] Sulzer, Valentin, S. Jon Chapman, Colin P. Please, David A. Howey, and Charles W. Monroe. "Faster Lead-Acid Battery Simulations from Porous-Electrode Theory: II. Asymptotic Analysis." arXiv preprint arXiv:1902.01774 (2019).

```python
import pybamm
import numpy as np
import os
import matplotlib.pyplot as plt
os.chdir(pybamm.__path__[0]+'/..')
```

### "Full" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Electrolyte Current

$$
    \frac{\partial i_{\text{e}}}{\partial x} = j, \\
    \mathcal{C}_\text{e} i_{\text{e}} = \epsilon_k^b \kappa(c)
        \left( \chi \frac{\partial}{\partial x}\log(c)
        - \frac{\partial\phi_{\text{e}}}{\partial x}\right)\\
    i_{\text{e}}\big|_{x=0}= i_{\text{e}}\big|_{x=1}=0,
$$

#### Electrode Current

$$
    \frac{\partial i_{\text{s}}}{\partial x} = -j,\\
    i_{\text{s}} = -\sigma\frac{\partial\phi_{\text{s}}}{\partial x},\\
    \phi_{\text{s}}\big|_{x=0}
        = i_{\text{s}}\big|_{x=l_\text{n}}
        = i_{\text{s}}\big|_{x=1-l_\text{p}} = 0, \\
    i_{\text{s}}\big|_{x=1}=\mathcal{I},
$$

#### interfacial current density

$$
j = \begin{cases}
    2j_0(c) \sinh\left(\eta\right), \quad &0 < x < l_\text{n} \\
    0, \quad &l_\text{n} < x < 1-l_\text{p} \\
    2j_0(c) \sinh\left(\eta\right), \quad &1-l_\text{p} < x < 1
\end{cases}
\\
\eta = \phi_{\text{s}} - \phi_\text{e} - U(c),
$$

This model is implemented in PyBaMM as the `Full` model

```python
full = pybamm.lead_acid.Full()
```

### "Leading-order" model

$$
    \frac{\mathrm{d} }{\mathrm{d} t}\left(\epsilon c\right) = (s_\text{n} - s_\text{p})\mathrm{I}, \\
    \frac{\mathrm{d} \epsilon}{\mathrm{d} t} = -\beta^\text{surf}j, \\
    j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
    \phi_\text{e} = -U_\text{n}(c) + \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{n}j_{0\text{n}}(c)}\right) \\
    V = -\phi_\text{e} + U_\text{p}(c) - \sinh^{-1}\left(\frac{\mathrm{I}}{2l_\text{p}j_{0\text{p}}(c)}\right) \\
$$

This model is implemented in PyBaMM as `LOQS` (leading-order quasi-static)

```python
loqs = pybamm.lead_acid.LOQS()
```

### "Composite" model

#### Electrolyte Concentration

$$
    \frac{\partial }{\partial t}\left(\epsilon c\right) = -\frac{\partial N}{\partial x} + sj, \\
    N = -\frac{\epsilon^b  D(c)}{\mathcal{C}_\text{e}}  \frac{\partial c}{\partial x}\\
    N\big|_{x=0}= N\big|_{x=1}=0, \\
    c\big|_{t=0} = 1
$$

#### Porosity

$$
    \frac{\partial \epsilon}{\partial t} = -\beta^\text{surf}j, \\
    \epsilon\big|_{t=0} = \epsilon^0
$$

#### Interfacial current density

$$
j = \begin{cases}
        \mathrm{I}/l_\text{n}, \quad &0 < x < l_\text{n} \\
        0, \quad &l_\text{n} < x < 1-l_\text{p} \\
        -\mathrm{I}/l_\text{p}, \quad &1-l_\text{p} < x < 1
    \end{cases} \\
$$

#### Voltage

$$
V = U_{\text{eq}} + \eta_r + \eta_c + \Delta\Phi_{\text{Elec}} + \Delta\Phi_{\text{Solid}}
$$
where
\begin{align}
    & U_{\text{eq}} = U_p(\bar{c}_p) - U_n(\bar{c}_n), \\
     &\eta_{r} = -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0n} l_n}\right)
     -\sinh^{-1}\left(\frac{\mathcal{I}}{2\bar{j}_{0p} l_p}\right), \\
     &\eta_c =  \mathcal{C}_e\chi\left(\log(\bar{c}_{p}) - \log(\bar{c}_{n})\right), \\
    &\Delta \Phi_{\text{Elec}}= -\frac{\mathcal{C}_e \mathcal{I}}{\kappa(\bar{c})}\left(\frac{l_n}{3\epsilon_n^b} + \frac{l_s}{\epsilon_{s}^b} + \frac{l_p}{3\epsilon_p^b} \right),
     \label{eqn:SPMe:electrolyte_ohmic_losses} \\
     &\Delta \Phi_{\text{Solid}} =  -\frac{\mathcal{I}}{3}\left(\frac{l_p}{\sigma_p} + \frac{l_n}{\sigma_n} \right),
\end{align}
with
\begin{equation}
    \bar{\cdot}_{\text{n}} =  \frac{1}{l_n}\int_0^{l_n} \cdot_{\text{n}} \, \text{d}x, \quad
    \bar{\cdot}_{\text{p}} =  \frac{1}{l_p}\int_{1-l_p}^{1} \cdot_{\text{p}} \, \text{d}x,
\end{equation}

This model is implemented in PyBaMM as `Composite`

```python
composite = pybamm.lead_acid.Composite()
```

## Solving the models

We load process parameters for each model, using the same set of (default) parameters for all

```python
# load models
models = [loqs, composite, full]

# process parameters
param = models[0].default_parameter_values
for model in models:
    param.process_model(model)
```

Then, we discretise the models, using the default settings

```python
discs = {}
for model in models:
    # load and process default geometry
    geometry = model.default_geometry
    param.process_geometry(geometry)

    # discretise using default settings
    mesh = pybamm.Mesh(geometry, model.default_submesh_types, model.default_var_pts)
    disc = pybamm.Discretisation(mesh, model.default_spatial_methods)
    disc.process_model(model)
    discs[model] = disc
```

Finally, we solve each model using its own individual solver

```python
timer = pybamm.Timer()
solutions = {}
t_eval = np.linspace(0, 0.5, 100)
for model in models:
    start = timer.time()
    solution = model.default_solver.solve(model, t_eval)
    end = timer.time()
    print("Solved the {} in {:.3f} seconds".format(model.name, end-start))
    solutions[model] = solution
```

    Solved the LOQS model in 0.092 seconds
    Solved the Composite model in 10.736 seconds
    Solved the Full model in 9.238 seconds

## Results

To plot the results, we create `ProcessedVariable` objects, which can be evaluated at any time and position. For example, we can compare the voltages:

```python
for model in models:
    t, y = solutions[model].t, solutions[model].y
    time = pybamm.ProcessedVariable(model.variables["Time [h]"], t, y, discs[model].mesh)(t)
    voltage = pybamm.ProcessedVariable(model.variables["Terminal voltage [V]"], t, y, discs[model].mesh)
    plt.plot(time, voltage(t), lw=2, label=model.name)
plt.xlabel("Time [h]", fontsize=15)
plt.ylabel("Terminal voltage [V]", fontsize=15)
plt.legend(fontsize=15)
plt.show()
```

Alternatively, using `QuickPlot`, we can compare the values of some variables

```python
mesh = list(discs.values())[0].mesh
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(models, mesh, solution_values)
import ipywidgets as widgets
widgets.interact(quick_plot.plot, t=widgets.FloatSlider(min=0,max=quick_plot.max_t,step=0.05,value=0));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=0.5, step=0.05), Output()), _dom_classes=('w…

If we update the current, setting it to be 20 A, we observe a greater discrepancy between the full model and the reduced-order models.

```python
# update parameter values and solve again
param.update({"Typical current [A]": 20})
for model in models:
    param.update_model(model, discs[model])
    solutions[model] = model.default_solver.solve(model, t_eval)

# Plot
solution_values = [solutions[model] for model in models]
quick_plot = pybamm.QuickPlot(models, mesh, solution_values)
widgets.interact(quick_plot.plot, t=widgets.FloatSlider(min=0,max=quick_plot.max_t,step=0.05,value=0));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=0.5, step=0.05), Output()), _dom_classes=('w…

```python

```


## Surfinpy

#### Tutorial 2 - Introducing temperature dependence

In tutorial 1 we generated a phase diagram at 0 K.  However, this is not representative of normal conditions.  Temperature is an important consideration for materials chemistry and we may wish to evaluate the phase thermodynamic stability at various synthesis conditions.  In order to overcome this and intorrduce temperature we need to modife the equations from tutorial 1.  This example will again be using the MgO-CO2-H2O system.

##### Methodology

As before the free energy can be calculated using;

\begin{align}
\Delta G^{0}_{f} = \sum\Delta G_{f}^{0,\text{products}} - \sum\Delta G_{f}^{0,\text{reactants}}
\end{align}

Where for this tutorial the free energy (G) for solid phases  is equal to is equal to the calculated DFT energy (U0). For gaseous species, the standard free energy varies significantly with temperature, and as DFT simulations are designed for condensed phase systems, we use experimental data to determine the temperature dependent free energy term for gaseous species, where $S_{\text{expt}}(T)$ is specific entropy value for a given T and  $H-H^0(T)$ is the , both can be obtained from the NIST database and can be calculated as;

\begin{align}
G =  U_0 + (H-H^0(T) - T S_{\text{expt}}(T))
\end{align}

##### Code

```python
import matplotlib.pyplot as plt
from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data
```

Same as tutorial 1 we first need to enter the DFT data in to the various data classes and create a list of the data.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

```python
MgO = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, label = "Periclase")
Art = data.DataSet(cation = 10, x = 5, y = 20, energy = -468.0, label = "Artinite")
Bru = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, label = "Brucite")
Nes = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, label = "Nesquehonite")
Mag = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, label = "Magnesite")
Lan = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, label = "Lansfordite")
Hyd = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, label = "Hydromagnesite")
```

```python
data = [MgO, Art, Bru, Nes,  Mag, Lan, Hyd]
```

```python
x_energy=-20.53412969
y_energy=-12.83725889
```

In order to calculate $S_{\text{expt}}(T)$ for H2O and CO2 we need to use experimental data from the NSIT JANAF database.  As a user you will need to download the tables for the species you are interested in (in this example water and carbon dioxide).  SurfinPy has a function that can read this data, assuming it is in the correct format and calculate the temperature correction for you.  Provide the math to the file and the temperature you want.

```python
CO2_exp = ut.fit_nist("CO2.txt")[298]
Water_exp = ut.fit_nist("H2O.txt")[298]

CO2_corrected = x_energy + CO2_exp
Water_corrected = y_energy + Water_exp

deltaX = {'Range': [ -3, 2],  'Label': 'CO_2'}
deltaY = {'Range': [ -3, 2], 'Label': 'H_2O'}
```

CO2_corrected and H2O_corrected are now temperature depenent terms correcsponding to a temperature of 298 K.  The resulting phase diagram will now be at a temperature of 298 K.

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy=CO2_corrected, y_energy=Water_corrected)

ax = system.plot_phase(temperature=298, figsize=(6, 4))

plt.show()
```

```python

```

```python

```


# Qiskit Aer: Pulse simulation of two qubits using a Duffing oscillator model

This notebook shows how to use the Qiskit Aer pulse simulator, which simulates experiments specified as pulse `Schedule` objects at the Hamiltonian level. The simulator solves the Schrodinger equation for a specified Hamiltonian model and pulse `Schedule` in the frame of the drift Hamiltonian.

In particular, in this tutorial we will:

- Construct a model of a two qubit superconducting system.
- Calibrate $\pi$ pulses on each qubit in the simulated system.
- Observe cross-resonance oscillations when driving qubit 1 with target qubit 0.

The Introduction outlines the concepts and flow of this notebook.

## 1. Introduction

The main sections proceed as follows.

### Section 3: Duffing oscillator model

To simulate a physical system, it is necessary to specify a model. In this notebook, we will model superconducting qubits as a collection of *Duffing oscillators*. The model is specified in terms of the following parameters:

-  Each Duffing oscillator is specified by a frequency $\nu$, anharmonicity $\alpha$, and drive strength $r$, which result in the Hamiltonian terms:

$$\begin{equation}
    2\pi\nu a^\dagger a + \pi \alpha a^\dagger a(a^\dagger a - 1) + 2 \pi r (a + a^\dagger) \times D(t),
\end{equation}$$

where $D(t)$ is the signal on the drive channel for the qubit, and $a^\dagger$ and $a$ are, respectively, the creation and annihilation operators for the qubit. Note that the drive strength $r$ sets the scaling of the control term, with $D(t)$ assumed to be a complex and unitless number satisfying $|D(t)| \leq 1$.

-  A coupling between a pair of oscillators $(l,k)$ is specified by the coupling strength $J$, resulting in an exchange coupling term:

$$\begin{equation}
    2 \pi J (a_l a_k^\dagger + a_l^\dagger a_k),
\end{equation}$$

where the subscript denotes which qubit the operators act on.

- Additionally, for numerical simulation, it is necessary to specify a cutoff dimension; the Duffing oscillator model is *infinite dimensional*, and computer simulation requires restriction of the operators to a finite dimensional subspace.

**In the code:** We will define a model of the above form for two coupled qubits using the helper function `duffing_system_model`.

### Section 4: $\pi$-pulse calibration using Ignis

Once the model is defined, we will calibrate $\pi$-pulses on each qubit. A $\pi$-pulse is defined as a pulse on the drive channel of a qubit that "flips" the qubit; i.e. that takes the ground state to the first excited state, and the first excited state to the ground state.

We will experimentally find a $\pi$-pulse for each qubit using the following procedure:

- A fixed pulse shape is set - in this case it will be a Gaussian pulse.
- A sequence of experiments is run, each consisting of a Gaussian pulse on the qubit, followed by a measurement, with each experiment in the sequence having a subsequently larger amplitude for the Gaussian pulse.
- The measurement data is fit, and the pulse amplitude that completely flips the qubit is found (i.e. the $\pi$-pulse amplitude).

**In the code:** Using Ignis we will construct `Schedule` objects for the above experiments, then fit the data to find the $\pi$-pulse amplitudes.

### Section 5: Cross-resonance oscillations

Once the $\pi$-pulses are calibrated, we will simulate the effects of cross-resonance driving on qubit $1$ with target qubit $0$. This means that we will drive qubit $1$ at the frequency of qubit $0$, with the goal of observing that the trajectory and oscillations of qubit $0$ *depends* on the state of qubit $1$. This phenomenon provides a basis for creating two-qubit *controlled* gates. Note: This section requires the calibration of the $\pi$-pulse in Section 4.

To observe cross-resonance driving, we will use experiments very similar to the $\pi$-pulse calibration case:

- Initially, qubit $1$ is either left in the ground state, or is driven to its first excited state using the $\pi$-pulse found in Section 4.
- A sequence of experiments is run, each consisting of a Gaussian pulse on qubit $1$ driven at the frequency of qubit $0$, followed by a measurement of both qubits, with each experiment of the sequence having a subsequently larger amplitude for the Gaussian pulse.

**In the code:** Functions for defining the experiments and visualizing the data are constructed, including a visualization of the trajectory of the target qubit on the Bloch sphere.

## 2. Imports

This notebook makes use of the following imports.

```python
import numpy as np
from scipy.optimize import curve_fit, root

# visualization tools
import matplotlib.pyplot as plt
from qiskit.visualization.bloch import Bloch
```

Import qiskit libraries for working with `pulse` and calibration:

```python
import qiskit.pulse as pulse
from qiskit.pulse.library import Gaussian, GaussianSquare
from qiskit.compiler import assemble
import qiskit.ignis

from qiskit.ignis.characterization.calibrations import rabi_schedules, RabiFitter
```

Imports for qiskit pulse simulator:

```python
# The pulse simulator
from qiskit.providers.aer import PulseSimulator

# function for constructing duffing models
from qiskit.providers.aer.pulse import duffing_system_model
```

## 3. Duffing oscillator system model

An object representing a model for a collection of Duffing oscillators can be constructed using the `duffing_system_model` function. Here we construct a $2$ Duffing oscillator model with cutoff dimension $3$.

```python
# cutoff dimension
dim_oscillators = 3

# frequencies for transmon drift terms, harmonic term and anharmonic term
# Number of oscillators in the model is determined from len(oscillator_freqs)
oscillator_freqs = [5.0e9, 5.2e9]
anharm_freqs = [-0.33e9, -0.33e9]

# drive strengths
drive_strengths = [0.02e9, 0.02e9]

# specify coupling as a dictionary (qubits 0 and 1 are coupled with a coefficient 0.002e9)
coupling_dict = {(0,1): 0.002e9}

# sample duration for pulse instructions
dt = 1e-9

# create the model
two_qubit_model = duffing_system_model(dim_oscillators=dim_oscillators,
                                       oscillator_freqs=oscillator_freqs,
                                       anharm_freqs=anharm_freqs,
                                       drive_strengths=drive_strengths,
                                       coupling_dict=coupling_dict,
                                       dt=dt)
```

    /tmp/qiskit_release/lib/python3.9/site-packages/qiskit/providers/aer/pulse/system_models/string_model_parser/string_model_parser.py:280: DeprecationWarning: Using the `__mul__` operator `A * B` as shorthand for `A.dot(B)` is deprecated as of version 0.17.0 and will be  removed no earlier than 3 months after the release date. As an alternative, use the compose operator `B & A` in place of `A * B` as a replacement.
      stack.append(op1 * op2)

The function `duffing_system_model` returns a `PulseSystemModel` object, which is a general object for storing model information required for simulation with the `PulseSimulator`.

## 4 Calibrating $\pi$ pulses on each qubit using Ignis

As described in the introduction, we now calibrate $\pi$ pulses on each qubit in `two_qubit_model`. The experiments in this calibration procedure are known as *Rabi experiments*, and the data we will observe are known as *Rabi oscillations*.

### 4.1 Constructing the schedules

We construct the schedules using the `rabi_schedules` function in Ignis. To do this, we need to supply an `InstructionScheduleMap` containing a measurement schedule.

```python
# list of qubits to be used throughout the notebook
qubits = [0, 1]

# Construct a measurement schedule and add it to an InstructionScheduleMap
meas_amp = 0.025
meas_samples = 1200
meas_sigma = 4
meas_width = 1150
meas_pulse = GaussianSquare(duration=meas_samples, amp=meas_amp,
                            sigma=meas_sigma, width=meas_width)

acq_sched = pulse.Acquire(meas_samples, pulse.AcquireChannel(0), pulse.MemorySlot(0))
acq_sched += pulse.Acquire(meas_samples, pulse.AcquireChannel(1), pulse.MemorySlot(1))

measure_sched = pulse.Play(meas_pulse, pulse.MeasureChannel(0)) | pulse.Play(meas_pulse, pulse.MeasureChannel(1)) | acq_sched

inst_map = pulse.InstructionScheduleMap()
inst_map.add('measure', qubits, measure_sched)
```

Next, construct the Rabi schedules.

```python
# construct Rabi experiments
drive_amps = np.linspace(0, 0.9, 48)
drive_sigma = 16
drive_duration = 128
drive_channels = [pulse.DriveChannel(0), pulse.DriveChannel(1)]

rabi_experiments, rabi_amps = rabi_schedules(amp_list=drive_amps,
                                             qubits=qubits,
                                             pulse_width=drive_duration,
                                             pulse_sigma=drive_sigma,
                                             drives=drive_channels,
                                             inst_map=inst_map,
                                             meas_map=[[0, 1]])
```

The `Schedule`s in `rabi_schedules` correspond to experiments to generate Rabi oscillations on both qubits in parallel. Each experiment consists of a Gaussian pulse on the qubits of a given magnitude, followed by measurement.

For example:

```python
rabi_experiments[10].draw()
```

### 4.2 Simulate the Rabi experiments

To simulate the Rabi experiments, assemble the `Schedule` list into a qobj. When assembling, pass the `PulseSimulator` as the backend.

Here, we want to use local oscillators with frequencies automatically computed from Duffing model Hamiltonian.

```python
# instantiate the pulse simulator
backend_sim = PulseSimulator(system_model=two_qubit_model)

# compute frequencies from the Hamiltonian
qubit_lo_freq = two_qubit_model.hamiltonian.get_qubit_lo_from_drift()

rabi_qobj = assemble(rabi_experiments,
                     backend=backend_sim,
                     qubit_lo_freq=qubit_lo_freq,
                     meas_level=1,
                     meas_return='avg',
                     shots=512)
```

Run the simulation using the simulator backend.

```python
# run the simulation
rabi_result = backend_sim.run(rabi_qobj).result()
```

    /tmp/qiskit_release/lib/python3.9/site-packages/qiskit/providers/aer/pulse/system_models/string_model_parser/operator_generators.py:154: DeprecationWarning: Using the `__matmul__` operator `A @ B` as shorthand for `A.compose(B)` is deprecated as of version 0.17.0 and will be  removed no earlier than 3 months after the release date. Use the `A & B` instead.
      proj_op += estate @ estate.adjoint()

### 4.3 Fit and plot the data

Next, we use `RabiFitter` in Ignis to fit the data, extract the $\pi$-pulse amplitude, and then plot the data.

```python
rabifit = RabiFitter(rabi_result, rabi_amps, qubits, fit_p0 = [0.5,0.5,0.6,1.5])

plt.figure(figsize=(15, 10))
q_offset = 0
for qubit in qubits:
    ax = plt.subplot(2, 2, qubit + 1)
    rabifit.plot(qubit, ax=ax)
    print('Pi Amp: %f'%rabifit.pi_amplitude(qubit))
plt.show()
```

Plotted is the averaged IQ data for observing each qubit. Observe that here, each qubit oscillates between the 0 and 1 state. The amplitude at which a given qubit reaches the peak of the oscillation is the desired $\pi$-pulse amplitude.

## 5. Oscillations from cross-resonance drive

Next, we simulate the effects of a cross-resonance drive on qubit $1$ with target qubit $0$, observing that the trajectory and oscillations of qubit $0$ *depends* on the state of qubit $1$.

**Note:** This section depends on the $\pi$-pulse calibrations of Section 2.

### 5.1 Cross-resonance `ControlChannel` indices

Driving qubit $1$ at the frequency of qubit $0$ requires use of a pulse `ControlChannel`. The model generating function `duffing_system_model`, automatically sets up `ControlChannels` for performing cross-resonance drives between pairs of coupled qubits. The index of the `ControlChannel` for performing a particular cross-resonance drive is retrievable using the class method `control_channel_index` on the returned `PulseSystemModel`. For example, to get the `ControlChannel` index corresponding to a CR drive on qubit 1 with target 0, call the function `control_channel_index` with the tuple `(1,0)`:

```python
two_qubit_model.control_channel_index((1,0))
```

    1

Hence, to perform a cross-resonance drive on qubit $1$ with target qubit $0$, use `ControlChannel(1)`. This will be made use of when constructing `Schedule` objects in this section.

### 5.2 Functions to generate the experiment list, and analyze the output

First, we define a function `cr_drive_experiments`, which, given the drive and target indices, and the option to either start with the drive qubit in the ground or excited state, returns a list of experiments for observing the oscillations.

```python
# store the pi amplitudes from Section 2 in a list
pi_amps = [rabifit.pi_amplitude(0), rabifit.pi_amplitude(1)]

def cr_drive_experiments(drive_idx,
                         target_idx,
                         flip_drive_qubit = False,
                         cr_drive_amps=np.linspace(0, 0.9, 16),
                         cr_drive_samples=800,
                         cr_drive_sigma=4,
                         pi_drive_samples=128,
                         pi_drive_sigma=16):
    """Generate schedules corresponding to CR drive experiments.

    Args:
        drive_idx (int): label of driven qubit
        target_idx (int): label of target qubit
        flip_drive_qubit (bool): whether or not to start the driven qubit in the ground or excited state
        cr_drive_amps (array): list of drive amplitudes to use
        cr_drive_samples (int): number samples for each CR drive signal
        cr_drive_sigma (float): standard deviation of CR Gaussian pulse
        pi_drive_samples (int): number samples for pi pulse on drive
        pi_drive_sigma (float): standard deviation of Gaussian pi pulse on drive

    Returns:
        list[Schedule]: A list of Schedule objects for each experiment
    """

    # Construct measurement commands to be used for all schedules
    meas_amp = 0.025
    meas_samples = 1200
    meas_sigma = 4
    meas_width = 1150
    meas_pulse = GaussianSquare(duration=meas_samples, amp=meas_amp,
                               sigma=meas_sigma, width=meas_width)

    acq_sched = pulse.Acquire(meas_samples, pulse.AcquireChannel(0), pulse.MemorySlot(0))
    acq_sched += pulse.Acquire(meas_samples, pulse.AcquireChannel(1), pulse.MemorySlot(1))

    # create measurement schedule
    measure_sched = (pulse.Play(meas_pulse, pulse.MeasureChannel(0)) |
                     pulse.Play(meas_pulse, pulse.MeasureChannel(1))|
                     acq_sched)

    # Create schedule
    schedules = []
    for ii, cr_drive_amp in enumerate(cr_drive_amps):

        # pulse for flipping drive qubit if desired
        pi_pulse = Gaussian(duration=pi_drive_samples, amp=pi_amps[drive_idx], sigma=pi_drive_sigma)

        # cr drive pulse
        cr_width = cr_drive_samples - 2*cr_drive_sigma*4
        cr_rabi_pulse = GaussianSquare(duration=cr_drive_samples,
                                       amp=cr_drive_amp,
                                       sigma=cr_drive_sigma,
                                       width=cr_width)

        # add commands to schedule
        schedule = pulse.Schedule(name='cr_rabi_exp_amp_%s' % cr_drive_amp)

        # flip drive qubit if desired
        if flip_drive_qubit:
            schedule += pulse.Play(pi_pulse, pulse.DriveChannel(drive_idx))

        # do cr drive
        # First, get the ControlChannel index for CR drive from drive to target
        cr_idx = two_qubit_model.control_channel_index((drive_idx, target_idx))
        schedule += pulse.Play(cr_rabi_pulse, pulse.ControlChannel(cr_idx))  << schedule.duration

        schedule += measure_sched << schedule.duration

        schedules.append(schedule)
    return schedules
```

Next we create two functions for observing the data:

- `plot_cr_pop_data` - for plotting the oscillations between the ground state and the first excited state
- `plot_bloch_sphere` - for viewing the trajectory of the target qubit on the bloch sphere

```python
def plot_cr_pop_data(drive_idx,
                     target_idx,
                     sim_result,
                     cr_drive_amps=np.linspace(0, 0.9, 16)):
    """Plot the population of each qubit.

    Args:
        drive_idx (int): label of driven qubit
        target_idx (int): label of target qubit
        sim_result (Result): results of simulation
        cr_drive_amps (array): list of drive amplitudes to use for axis labels
    """

    amp_data_Q0 = []
    amp_data_Q1 = []

    for exp_idx in range(len(cr_drive_amps)):
        exp_mem = sim_result.get_memory(exp_idx)
        amp_data_Q0.append(np.abs(exp_mem[0]))
        amp_data_Q1.append(np.abs(exp_mem[1]))

    plt.plot(cr_drive_amps, amp_data_Q0, label='Q0')
    plt.plot(cr_drive_amps, amp_data_Q1, label='Q1')
    plt.legend()
    plt.xlabel('Pulse amplitude, a.u.', fontsize=20)
    plt.ylabel('Signal, a.u.', fontsize=20)
    plt.title('CR (Target Q{0}, driving on Q{1})'.format(target_idx, drive_idx), fontsize=20)
    plt.grid(True)

def bloch_vectors(drive_idx, drive_energy_level, sim_result):
    """Plot the population of each qubit.

    Args:
        drive_idx (int): label of driven qubit
        drive_energy_level (int): energy level of drive qubit at start of CR drive
        sim_result (Result): results of simulation

    Returns:
        list: list of Bloch vectors corresponding to the final state of the target qubit
              for each experiment
    """

    # get the dimension used for simulation
    dim = int(np.sqrt(len(sim_result.get_statevector(0))))

    # get the relevant dressed state indices
    idx0 = 0
    idx1 = 0
    if drive_idx == 0:
        if drive_energy_level == 0:
            idx0, idx1 = 0, dim
        elif drive_energy_level == 1:
            idx0, idx1 = 1, dim + 1
    if drive_idx == 1:
        if drive_energy_level == 0:
            idx0, idx1 = 0, 1
        elif drive_energy_level == 1:
            idx0, idx1 = dim, dim + 1

    # construct Pauli operators for correct dressed manifold
    state0 = np.array([two_qubit_model.hamiltonian._estates[idx0]])
    state1 = np.array([two_qubit_model.hamiltonian._estates[idx1]])

    outer01 = np.transpose(state0)@state1
    outer10 = np.transpose(state1)@state0
    outer00 = np.transpose(state0)@state0
    outer11 = np.transpose(state1)@state1

    X = outer01 + outer10
    Y = -1j*outer01 + 1j*outer10
    Z = outer00 - outer11

    # function for computing a single bloch vector
    bloch_vec = lambda vec: np.real(np.array([np.conj(vec)@X@vec, np.conj(vec)@Y@vec, np.conj(vec)@Z@vec]))

    return [bloch_vec(sim_result.get_statevector(idx)) for idx in range(len(sim_result.results))]

def plot_bloch_sphere(bloch_vectors):
    """Given a list of Bloch vectors, plot them on the Bloch sphere

    Args:
        bloch_vectors (list): list of bloch vectors
    """
    sphere = Bloch()
    sphere.add_points(np.transpose(bloch_vectors))
    sphere.show()
```

### 5.3 Drive qubit 1 to observe CR oscillations on qubit 0

#### Qubit 1 in the ground state

First, we drive with both qubit 0 and qubit 1 in the ground state.

```python
# construct experiments
drive_idx = 1
target_idx = 0
flip_drive = False
experiments = cr_drive_experiments(drive_idx, target_idx, flip_drive)

# compute frequencies from the Hamiltonian
qubit_lo_freq = two_qubit_model.hamiltonian.get_qubit_lo_from_drift()

# assemble the qobj
cr_rabi_qobj = assemble(experiments,
                        backend=backend_sim,
                        qubit_lo_freq=qubit_lo_freq,
                        meas_level=1,
                        meas_return='avg',
                        shots=512)
```

Run the simulation:

```python
sim_result = backend_sim.run(cr_rabi_qobj).result()

plot_cr_pop_data(drive_idx, target_idx, sim_result)
```

Observe that qubit 1 remains in the ground state, while excitations are driven in qubit 0.

We may also observe the trajectory of qubit 0 on the Bloch sphere:

```python
bloch_vecs = bloch_vectors(drive_idx, int(flip_drive), sim_result)
plot_bloch_sphere(bloch_vecs)
```

#### Qubit 1 in the first excited state

Next, we again perform a CR drive qubit 1 with qubit 0 as the target, but now we start each experiment by flipping qubit 1 into the first excited state.

```python
# construct experiments, now with flip_drive == True
drive_idx = 1
target_idx = 0
flip_drive = True
experiments = cr_drive_experiments(drive_idx, target_idx, flip_drive)

# compute frequencies from the Hamiltonian
qubit_lo_freq = two_qubit_model.hamiltonian.get_qubit_lo_from_drift()

# assemble the qobj
cr_rabi_qobj = assemble(experiments,
                        backend=backend_sim,
                        qubit_lo_freq=qubit_lo_freq,
                        meas_level=1,
                        meas_return='avg',
                        shots=512)
```

```python
sim_result = backend_sim.run(cr_rabi_qobj).result()

plot_cr_pop_data(drive_idx, target_idx, sim_result)
```

Observe that now qubit 1 is in the excited state, while oscillations are again being driven on qubit 0, now at a different rate as before.

Again, observe the trajectory of qubit 0 on the Bloch sphere:

```python
bloch_vecs = bloch_vectors(drive_idx, int(flip_drive), sim_result)
plot_bloch_sphere(bloch_vecs)
```

Here we see that qubit 0 takes a *different* trajectory on the Bloch sphere when qubit 1 is in the excited state. This is what enables controlled operations between two qubits.

```python
import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright
```

    /tmp/qiskit_release/lib/python3.9/site-packages/qiskit/aqua/__init__.py:86: DeprecationWarning: The package qiskit.aqua is deprecated. It was moved/refactored to qiskit-terra For more information see <https://github.com/Qiskit/qiskit-aqua/blob/main/README.md#migration-guide>
      warn_package('aqua', 'qiskit-terra')

Version Information

Qiskit Software	Version

qiskit-terra	0.18.2

qiskit-aer	0.8.2

qiskit-ignis	0.6.0

qiskit-ibmq-provider	0.16.0

qiskit-aqua	0.9.5

qiskit	0.29.1

qiskit-nature	0.2.1

qiskit-finance	0.2.1

qiskit-optimization	0.2.2

qiskit-machine-learning	0.2.1

System information

Python	3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0]

OS	Linux

CPUs	32

Memory (Gb)	125.65557479858398

Thu Sep 16 07:39:30 2021 EDT

This code is a part of Qiskit

© Copyright IBM 2017, 2021.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
 of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals.