# Hantush response functions

This notebook compares the two Hantush response function implementations in Pastas.

*Developed by D.A. Brakenhoff (Artesia, 2021)*

## Contents

- [`Hantush` versus `HantushWellModel`](#`Hantush`-versus-`HantushWellModel`)
- [Which Hantush should I use?](#Which-Hantush-should-I-use?)
- [Synthetic example](#Synthetic-example)

```python
import numpy as np
import pandas as pd
import pastas as ps

ps.show_versions()
```

    Python version: 3.7.6 (default, Jan  8 2020, 19:59:22)
    [GCC 7.3.0]
    Numpy version: 1.20.1
    Scipy version: 1.5.2
    Pandas version: 1.1.5
    Pastas version: 0.17.0b
    Matplotlib version: 3.3.3

## `Hantush` versus `HantushWellModel`

The reason there are two implementations in Pastas is that each implementation currently has advantages and disadvantages. We will discuss those soon, but first let's introduce the two implementations. The two Hantush response functions are very similar, but differ in the definition of the parameters. The table below shows the formulas for both implementations.

| Name             | Parameters  | Formula                                                                 | Description                                                                    |
|------------------|-------------|:------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| Hantush          | 3 - A, a, b | $$ \theta(t) = At^{-1} e^{-t/a - ab/t} $$                               | Response function commonly used for groundwater abstraction wells.             |
| HantushWellModel | 3 - A, a, b | $$ \theta(t) = A K_0 \left( \sqrt{4b} \right) t^{-1} e^{-t/a - ab/t} $$ | Implementation of the Hantush well function that allows scaling with distance. |

In the first implementation the parameters $A$, $a$, and $b$ can be written as:

$$
\begin{align}
A &= \frac{1}{2 \pi T} K_0 \left( \sqrt{4b} \right) \\
a &= cS \\
b &= \frac{r^2}{4 \lambda^2}
\end{align}
$$

In this case parameter $A$ is also known as the "gain", which is equal to the steady-state contribution of a stress with unit 1. For example, the drawdown caused by a well with a continuous extraction rate of 1.0 (the units don't really matter here and are determined by what units the user puts in).

In the second implementation, the definition of the parameters $A$ is different, which allows the distance $r$ between an extraction well and an observation well to be passed as a variable. This allows multiple wells to have the same response function, which can be useful to e.g. reduce the number of parameters in a model with multiple extraction wells. When $r$ is passed as a parameter, the formula for $b$ below is simplified by substituting in $1$ for $r$. Note that $r$ is never optimized, but has to be provided by the user.

$$
\begin{align}
A &= \frac{1}{2 \pi T} \\
a &= cS \\
b &= \frac{r^2}{4 \lambda^2}
\end{align}
$$

## Which Hantush should I use?

So why two implementations? Well, there are advantages and disadvantages to both implementations, which are listed below.

#### Hantush

**Pro**:
- Parameter A is the gain, which makes it easier to interpret the results.
- Estimates the uncertainty of the gain directly.

**Con**:
- Cannot be used to simulate multiple wells.
- More challenging to relate to aquifer characteristics.

#### HantushWellModel

**Pro**:
- Can be used with WellModel to simulate multiple wells with one response function.
- Easier to relate parameters to aquifer characteristics.

**Con**:
- Does not directly estimate the uncertainty of the gain but this can be calculated using special methods.
- More sensitive to the initial value of parameters, in rare cases the initial parameter values have to be tweaked to get a good fit result.

So which one should you use? It depends on your use-case:

- Use `Hantush` if you are considering a single extraction well and you're interested in calculating the gain and the uncertainty of the gain.
- Use `HantushWellModel` if you are simulating multiple extraction wells or want to pass the distance between extraction and observation well as a known parameter.

Of course these aren't strict rules and it is encouraged to explore different model structures when building your timeseries models. But as a first general guiding principle this should help in selecting which approach is appropriate to your specific problem.

## Synthetic example

A synthetic example is used to show both Hantush implementations. First, we create a synthetic timeseries generated with the Hantush response function to which we add autocorrelated residuals. We set the parameter values for the Hantush response function:

```python
# A defined so that 100 m3/day results in 5 m drawdown
A = -5 / 100.0
a = 200
b = 0.5

d = 0.0  # reference level
```

```python
# auto-correlated residuals AR(1)
sigma_n = 0.05
alpha = 50
sigma_r = sigma_n / np.sqrt(1 - np.exp(-2 * 14 / alpha))
print(f'sigma_r = {sigma_r:.2f} m')
```

    sigma_r = 0.08 m

Create a head observations timeseries and a timeseries with the well extraction rate.

```python
# head observations between 2000 and 2010
idx = pd.date_range("2000", "2010", freq="D")
ho = pd.Series(index=idx, data=0)

# extraction of 100 m3/day between 2002 and 2006
well = pd.Series(index=idx, data=0.0)
well.loc["2002":"2006"] = 100.0
```

Create the synthetic head timeseries based on the extraction rate and the parameters we defined above.

```python
ml0 = ps.Model(ho) # alleen de tijdstippen waarop gemeten is worden gebruikt
rm = ps.StressModel(well, ps.Hantush, name='well', up=False)
ml0.add_stressmodel(rm)
ml0.set_parameter('well_A', initial=A)
ml0.set_parameter('well_a', initial=a)
ml0.set_parameter('well_b', initial=b)
ml0.set_parameter('constant_d', initial=d)
hsynthetic_no_error = ml0.simulate()[ho.index]
```

    INFO: Time series None updated to dtype float.
    INFO: Inferred frequency for time series None: freq=D
    INFO: Inferred frequency for time series None: freq=D
    WARNING: Model is not optimized yet, initial parameters are used.

Add the auto-correlated residuals.

```python
delt = (ho.index[1:] - ho.index[:-1]).values / pd.Timedelta("1d")
np.random.seed(1)
noise = sigma_n * np.random.randn(len(ho))
residuals = np.zeros_like(noise)
residuals[0] = noise[0]
for i in range(1, len(ho)):
    residuals[i] = np.exp(-delt[i - 1] / alpha) * residuals[i - 1] + noise[i]
hsynthetic = hsynthetic_no_error + residuals
```

Plot the timeseries.

```python
ax = hsynthetic_no_error.plot(label='synthetic heads (no error)', figsize=(10, 5))
hsynthetic.plot(ax=ax, color="C1", label="synthetic heads (with error)")
ax.legend(loc='best')
ax.set_ylabel("head (m+ref)")
ax.grid(b=True)
```

Create three models:

1. Model with `Hantush` response function.
2. Model with `HantushWellModel` response function, but $r$ is not passed as a known parameter.
3. Model with `WellModel`, which uses `HantushWellModel` and where $r$ is set to 1.0 m.

All three models should yield the similar results and be able to estimate the true values of the parameters reasonably well.

```python
# Hantush
ml_h1 = ps.Model(hsynthetic, name="gain")
wm_h1 = ps.StressModel(well, ps.Hantush, name='well', up=False)
ml_h1.add_stressmodel(wm_h1)
ml_h1.solve(report=False, noise=True)
```

    INFO: Inferred frequency for time series Simulation: freq=D
    INFO: Inferred frequency for time series None: freq=D

Solve with noise model and Hantush_scaled

```python
# HantushWellModel
ml_h2 = ps.Model(hsynthetic, name="scaled")
wm_h2 = ps.StressModel(well, ps.HantushWellModel, name='well', up=False)
ml_h2.add_stressmodel(wm_h2)
ml_h2.solve(report=False, noise=True)
```

    INFO: Inferred frequency for time series Simulation: freq=D
    INFO: Inferred frequency for time series None: freq=D

```python
# WellModel
r = np.array([1.0])  # parameter r
well.name = "well"

ml_h3 = ps.Model(hsynthetic, name="wellmodel")
wm_h3 = ps.WellModel([well], ps.HantushWellModel, "well", r, up=False)
ml_h3.add_stressmodel(wm_h3)
ml_h3.solve(report=False, noise=True, solver=ps.LmfitSolve)
```

    INFO: Inferred frequency for time series Simulation: freq=D
    WARNING: It is recommended to use LmfitSolve as the solver when implementing WellModel. See https://github.com/pastas/pastas/issues/177.
    INFO: Inferred frequency for time series well: freq=D
    INFO: Time Series well was extended to 1990-01-03 00:00:00 by adding 0.0 values.

Plot a comparison of all three models. The three models all yield similar results (all the lines overlap).

```python
axes = ps.plots.compare([ml_h1, ml_h2, ml_h3], adjust_height=True,
                        figsize=(10, 8));
```

Compare the optimized parameters for each model with the true values we defined at the beginning of this example. Note that we're comparing the value of the gain (not parameter $A$) and that each model has its own method for calculating the gain. As expected, the parameter estimates are reasonably close to the true values defined above.

```python
df = pd.DataFrame(index=["well_gain", "well_a", "well_b"],
                  columns=["True value", "Hantush",
                           "HantushWellModel", "WellModel"])

df["True value"] = A, a, b

df["Hantush"] = (
    # gain (same as A in this case)
    wm_h1.rfunc.gain(ml_h1.get_parameters("well")),
    # a
    ml_h1.parameters.loc["well_a", "optimal"],
    # b
    ml_h1.parameters.loc["well_b", "optimal"]
)

df["HantushWellModel"] = (
    # gain (not same as A)
    wm_h2.rfunc.gain(ml_h2.get_parameters("well")),
    # a
    ml_h2.parameters.loc["well_a", "optimal"],
    # b
    ml_h2.parameters.loc["well_b", "optimal"]
)

df["WellModel"] = (
    # gain, use WellModel.get_parameters() to get params: A, a, b and r
    wm_h3.rfunc.gain(wm_h3.get_parameters(model=ml_h3, istress=0)),
    # a
    ml_h3.parameters.loc["well_a", "optimal"],
    # b (multiply parameter value by r^2 for comparison)
    ml_h3.parameters.loc["well_b", "optimal"] * r[0]**2
)

df
```

      True value
      Hantush
      HantushWellModel
      WellModel

      well_gain
      -0.05
      -0.050272
      -0.050272
      -0.050272

      well_a
      200.00
      231.499643
      231.494464
      231.498489

      well_b
      0.50
      0.376504
      0.376514
      0.376509

Recall from earlier that when using `ps.Hantush` the gain and uncertainty of the gain are calculated directly. This is not the case for `ps.HantushWellModel`, so to obtain the uncertainty of the gain when using that response function there is a method called `ps.HantushWellModel.variance_gain()` that computes the variance based on the optimal values and (co)variance of parameters $A$ and $b$.

The code below shows the calculated gain for each model, and how to calculate the variance and standard deviation of the gain for each model. The results show that the calculated values are all very close, as was expected.

```python
def variance_gain(ml, wm_name, istress=None):
    """Calculate variance of the gain for WellModel.

    Variance of the gain is calculated based on propagation of
    uncertainty using optimal values and the variances of A and b
    and the covariance between A and b.

    Parameters
    ----------
    ml : pastas.Model
        optimized model
    wm_name : str
        name of the WellModel
    istress : int or list of int, optional
        index of stress to calculate variance of gain for

    Returns
    -------
    var_gain : float
        variance of the gain calculated from model results
        for parameters A and b

    See Also
    --------
    pastas.HantushWellModel.variance_gain

    """
    wm = ml.stressmodels[wm_name]

    if ml.fit is None:
        raise AttributeError("Model not optimized! Run solve() first!")
    if wm.rfunc._name != "HantushWellModel":
        raise ValueError("Response function must be HantushWellModel!")

    # get parameters and (co)variances
    A = ml.parameters.loc[wm_name + "_A", "optimal"]
    b = ml.parameters.loc[wm_name + "_b", "optimal"]
    var_A = ml.fit.pcov.loc[wm_name + "_A", wm_name + "_A"]
    var_b = ml.fit.pcov.loc[wm_name + "_b", wm_name + "_b"]
    cov_Ab = ml.fit.pcov.loc[wm_name + "_A", wm_name + "_b"]

    if istress is None:
        r = np.asarray(wm.distances)
    elif isinstance(istress, int) or isinstance(istress, list):
        r = wm.distances[istress]
    else:
        raise ValueError("Parameter 'istress' must be None, list or int!")

    return wm.rfunc.variance_gain(A, b, var_A, var_b, cov_Ab, r=r)
```

```python
# create dataframe
var_gain = pd.DataFrame(index=df.columns[1:])

# add calculated gain
var_gain["gain"] = df.iloc[0, 1:].values

# Hantush: variance gain is computed directly
var_gain.loc["Hantush", "var gain"] = ml_h1.fit.pcov.loc["well_A", "well_A"]

# HantushWellModel: calculate variance gain
var_gain.loc["HantushWellModel", "var gain"] = wm_h2.rfunc.variance_gain(
    ml_h2.parameters.loc["well_A", "optimal"],  # A
    ml_h2.parameters.loc["well_b", "optimal"],  # b
    ml_h2.fit.pcov.loc["well_A", "well_A"],  # var_A
    ml_h2.fit.pcov.loc["well_b", "well_b"],  # var_b
    ml_h2.fit.pcov.loc["well_A", "well_b"]  # cov_Ab
)

# WellModel: calculate variance gain using helper function
var_gain.loc["WellModel", "var gain"] = variance_gain(ml_h3, "well", istress=0)

# calculate std dev gain
var_gain["std gain"] = np.sqrt(var_gain["var gain"])

# show table
var_gain.style.format("{:.5e}")
```

        	        gain	        var gain	        std gain

                        Hantush
                        -5.02721e-02
                        1.24684e-06
                        1.11662e-03

                        HantushWellModel
                        -5.02720e-02
                        1.24685e-06
                        1.11662e-03

                        WellModel
                        -5.02721e-02
                        1.24629e-06
                        1.11637e-03

```python

```


# Molecular Determinants
Instead of focusing on the crude raw data use bootstrapping to emphasize the real differences between increasing/decreasing and emergent. Given that drug perturbation range from a broad range of perturbation e.g. when looking at the feature chemical similarity almost the whole spectrum of similarities is covered but by using bootstrap one can focus on the mean differences and the variation of the mean.

1.) Load all features
2.) perform bootstrap analysis
3.) save results (+ plots)

```python
# Load all important python modules
import numpy as np
from matplotlib import pylab as plt
import scipy.stats as stats
from scipy.stats import mannwhitneyu as mu
import seaborn as sns
import os
from math import pi
import math
from sympy import Symbol, solve, sqrt
import networkx as nx
```

### 1. Load features

```python
#Load features per drug pair e.g. if two drugs share a common transporter, mean PPI distance between their drug targets etc.
fp = open('../data/Molecular_Determinants/DrugPair_Feature_Overview.csv','r')
features = fp.readline().strip().split(',')[4:]
print ('Number of features: %d' %len(features))

#Define interaction types as well as colors for the final plots (uniform with previous color coding)
interactionTypes = ['NoInteraction','Interaction','Increasing','Decreasing','Emergent']
interaction_colors = {'Increasing':'#ACD900','Decreasing':'#F70020','Emergent':'#0096FF','Interaction':'#F8B301','NoInteraction':'grey'}

#Create a dictionary where the individual results per feature can be split into the 5 types of interactions e.g. ChemicalSimilarity: {Increasing:[], Decreasing: [] ... }
dic_feature_results = {}
for f in features:
    dic_feature_results[f] = {}
    for iT in interactionTypes:
        dic_feature_results[f][iT] = []

#Go through the results of the drug pair feature file
for line in fp:
    tmp = line.strip().split(',')
    interactionType = tmp[3]

    #add the results to the corresponding list
    for f,i in zip(features, range(4,len(tmp))):
        val = tmp[i]
        #if the val is 'nan' ignore this row
        if val != 'nan':
            val = float(val)
            #if interaction type == None, then NoInteraction
            if interactionType == 'None':
                dic_feature_results[f]['NoInteraction'].append(val)
            #Else split into one of the posible other interaction types, keep only pure row e.g. only increasing/decreasing
            else:
                if  interactionType ==  'Increasing' or interactionType ==  'Increasing;Increasing':
                    dic_feature_results[f]['Increasing'].append(val)
                    dic_feature_results[f]['Interaction'].append(val)
                if  interactionType == 'Decreasing' or interactionType ==  'Decreasing;Decreasing':
                    dic_feature_results[f]['Decreasing'].append(val)
                    dic_feature_results[f]['Interaction'].append(val)
                if  interactionType == 'Emergent':
                    dic_feature_results[f]['Emergent'].append(val)
                    dic_feature_results[f]['Interaction'].append(val)
print ('Done loading data')
```

### 2. Perform Statistical Analysis

#### 2.1 Define functions for calculating bootstrapping and effect size

```python
def bootstrapping(data, number_iterations=10000, bootstrap_sample_size = None):
    '''
    Function for bootstrapping
    data = Data that needs to be bootsraped
    number_iteration = how often should bootstrapping be perfomred
    bootstrap_sample_size = sample size to draw for, if None then sample size = len(data) which is the typical procedure for bootstrapping
    '''

    #Define the bootstrap sample size
    if bootstrap_sample_size == None:
        bootstrap_sample_size = len(data)

    #draw randomly from data to get an estimation of it's variation. Save both the mean per bootstrap run as well as the calculated std
    bootstrap_samples_means = []
    bootstrap_samples_stds = []
    for i in range(0,number_iterations):
        bootstrap_sample = np.random.choice(data,bootstrap_sample_size,replace=True)
        bootstrap_samples_means.append(np.mean(bootstrap_sample))
        bootstrap_samples_stds.append(np.std(bootstrap_sample))

    #return the results
    return bootstrap_samples_means, bootstrap_samples_stds
```

```python
def cohen_d(x, y):
    '''
    Cohen's D is a typical meassure of effect size including the standard deviation of both samples (compared to ZScore which only uses one)
    '''

    #Get length of the two samples
    nx = len(x)
    ny = len(y)

    #Define degrees of freedom
    dof = nx + ny - 2

    #Calculate Cohen's D and return
    return (np.mean(x) - np.mean(y)) / np.sqrt(
        ((nx - 1) * np.std(x, ddof=1) ** 2 + (ny - 1) * np.std(y, ddof=1) ** 2) / dof)

```

#### 2.2 Calculate Statistics for each feature
1. Binary features e.g. have overlap/no overlap use Fisher Exact test
2. Continues features e.g. PPI distance, use Mann Whitney U test

```python
#Define a significance threshold
alpha = 0.05

#define the output file (shows the results for all features)
fp_out = open('../results/Molecular_Determinants/ResultsOverview.csv','w')
fp_out.write('Feature,InteractionType1,InteractionType2,Mean1,Mean2,FisherTest,PVal,PercentChange/OddsRatio,CohenD,BootstrapSign\n')

#Go thorugh each feature
for f in features:
    print f

    #check if all values of the given features are either 1 or 0 => then use Fisher Exact test to determine significance
    make_Fisher = False
    if all(v == 0 or v ==1 for v in dic_feature_results[f].values()[0]):
        make_Fisher = True

    #Define and create the output folder for the Bootstrapping plots (if doesnt exist)
    directory = os.path.dirname('../results/Molecular_Determinants/Bootstrapping/' + f + '/')
    if not os.path.exists(directory):
        os.makedirs(directory)

    ####
    # CREATE a standard boxplot for the features (only rly makes sense for continues values - make for binary features still though)
    bplot = sns.boxplot(data=[dic_feature_results[f]['NoInteraction'],dic_feature_results[f]['Interaction'],dic_feature_results[f]['Increasing'],dic_feature_results[f]['Decreasing'],dic_feature_results[f]['Emergent']],orient='h',showmeans = True, showfliers = False)

    #Define labels and colors
    interaction_types_2 = ['NoInteraction','Interaction','Increasing','Decreasing','Emergent']
    interaction_colors_2 = ['grey','#F8B301','#ACD900','#F70020','#0096FF']
    color_dict = dict(zip(interaction_types_2, interaction_colors_2))
    for i in range(0,5):
        mybox = bplot.artists[i]
        mybox.set_facecolor(color_dict[interaction_types_2[i]])

    #Add title and proper ticks
    plt.title(f)
    plt.yticks(range(0,5),['NoInteraction','NoInteraction','Increasing','Decreasing','Emergent'])
    plt.ylabel('Interaction Type')
    plt.tick_params(axis = 'y', which = 'major', labelsize = 5)
    plt.xlabel('Amount')
    plt.savefig(directory+'/Boxplot.pdf')
    plt.close()
    # END creating standard boxplot
    #########

    #####
    # CREATE Bootstrap plot (histogram showing the results for the 5 interaction types)
    #save the temporary bootstrap results for each interaction thype separately
    bootstrap_results = {}
    #Calculate bootstrap results for the 5 interaction types
    for iT in interactionTypes:
        #save mean and std as results
        bootstrap_results[iT] = {'b_mean':[],'b_std':[]}
        #get the actual data
        data = dic_feature_results[f][iT]
        #perform bootstrapping with standard bootstrapping rules
        b_means, b_stds = bootstrapping(data,number_iterations=10000, bootstrap_sample_size=None)

        #Save results
        bootstrap_results[iT]['b_mean'] = b_means
        bootstrap_results[iT]['b_std'] = b_stds

        #Xreate a histogram
        plt.hist(b_means,bins='auto', color = interaction_colors[iT], alpha=0.4)
        plt.savefig(directory+'/BootstrapOVerview.pdf')

    plt.close()
    # END creating Bootsrap plot
    #########

    #####
    # Comparison of mean results per interaction types (Interacting, Increasing, Decrasing, Emergent) compared to NO_INTERACTION
    # Create a Histogram for NoInteraction and compare to 4 individual points (represented as lines)
    plt.hist(bootstrap_results['NoInteraction']['b_mean'],bins='auto')
    plt.axvline(np.mean(dic_feature_results[f]['Interaction']),color=interaction_colors['Interaction'])
    plt.axvline(np.mean(dic_feature_results[f]['Increasing']),color=interaction_colors['Increasing'])
    plt.axvline(np.mean(dic_feature_results[f]['Decreasing']),color=interaction_colors['Decreasing'])
    plt.axvline(np.mean(dic_feature_results[f]['Emergent']),color=interaction_colors['Emergent'])
    plt.savefig(directory+'/OldBootstrapPlot.pdf')
    plt.close()
    # END creating NoInteraction comparison plot
    ######

    ###
    # COMPARE the bootstrap results between two interaction types to see if they are significantly different
    # Go through all different pairs
    for iT1 in interactionTypes:
        for iT2 in interactionTypes:
            if iT1 > iT2:

                #Extract data
                data1 = np.array(bootstrap_results[iT1]['b_mean'])
                data2 = np.array(bootstrap_results[iT2]['b_mean'])

                # Create a new distribution by substracting all the bootstrap results from each other
                # If 0 is completely outside this distribution (outside 95 percentile) then significant difference
                bootstrap_mean_diff =  list(data1 - data2)
                CI = (np.percentile(bootstrap_mean_diff,2.5), np.percentile(bootstrap_mean_diff,97.5))
                bootstrapSign = (0 > CI[0] and 0 > CI[1]) or (0 < CI[0] and 0 < CI[1])

                # Calculate corresponding Cohen's D
                c_d = cohen_d(data1,data2)

                # Calculate if two groups are significant different according to Fisher test (if binary data)
                if make_Fisher:
                    group1_Overlap = sum(dic_feature_results[f][iT1])
                    group1_NonOverlap = len(dic_feature_results[f][iT1]) - group1_Overlap

                    group2_Overlap = sum(dic_feature_results[f][iT2])
                    group2_NonOverlap = len(dic_feature_results[f][iT2]) - group2_Overlap

                    effect, pval = stats.fisher_exact([[group1_Overlap, group1_NonOverlap], [group2_Overlap, group2_NonOverlap]])
                # Else calulate according to Mann Whitney U
                else:
                    pval = mu(dic_feature_results[f][iT1],dic_feature_results[f][iT2])[1]
                    effect = (np.mean(data1) - np.mean(data2))/np.mean(data2) * 100

                # Create the difference bootstrap plot, with percentile and zero as markers; Add significance calculation to the title
                plt.hist(bootstrap_mean_diff,bins='auto', color='grey')
                plt.title(iT1 +'_' +iT2+': %.2f' %pval)
                plt.axvline(CI[0])
                plt.axvline(CI[1])
                plt.axvline(0,c='red',ls='--')
                #plt.show()
                plt.savefig(directory+'/Bootstrap_'+iT1 +'_' +iT2+'.pdf')
                plt.close()

                # Save the results to the overview file
                fp_out.write(f+','+iT1+','+iT2+','+str(np.mean(dic_feature_results[f][iT1]))+','+str(np.mean(dic_feature_results[f][iT2]))+','+str(make_Fisher)+','+str(pval)+','+str(effect)+','+str(c_d)+','+str(bootstrapSign)+'\n')
fp_out.close()
```

### 3. Radar plots
Additionally also create per feature radar plots, that are capable showing the disticnt moleular properties per interaction type

#### 3.1 Define functions for the creation of radar plots

```python
def radiusAngle_ToCoordinates(r, phi):
    '''
    Transform the radius and angle into x and y coordinates. Depending on the quadrant in, the sin gives either the x
    or y coordinate (and cos the other). As the angle is given between zero and 2pi, depending on the quadrant
    adjusted so we can create triangles

    :param r: radius of the point
    :param phi: angle between 0 and 2pi
    :return: x and y coordinate
    '''

    if phi <= pi / 2:
        #print 'Upper Right'
        x = math.sin(phi) * r
        y = math.cos(phi) * r
        quadr = 'UR'
    elif phi <= pi:
        #print 'Below Right'
        phi = phi - (pi / 2)
        x = math.cos(phi) * r
        y = math.sin(phi) * r * (-1)
        quadr = 'BR'
    elif phi <= (3 * pi) / 2:
        #print 'Below Left'
        phi = phi - (pi)
        x = math.sin(phi) * r * (-1)
        y = math.cos(phi) * r * (-1)
        quadr = 'BL'
    else:
        #print 'Upper Left'
        phi = phi - (3 * pi / 2)
        x = math.cos(phi) * r * (-1)
        y = math.sin(phi) * r
        quadr = 'UL'

    return x, y, quadr

def Find_Intersection(rc, phi1, r1, phi2, r2):
    '''
    Find the intersection of a line drawn between two points (given by their radius and angle) with a circle centered
    around zero and a given radius

    :param rc: radius of the circle
    :param phi1: first angle
    :param r1:   first radius
    :param phi2: second angle
    :param r2:   second radius
    :return: angle of the intersection (as radius has to be rc)
    '''

    # transform radius and angle into x and y coordinates (using sin/cos)
    x1, y1, quadr1 = radiusAngle_ToCoordinates(r1, phi1)
    x2, y2, quadr2 = radiusAngle_ToCoordinates(r2, phi2)

    # Create Function to plot
    # factor = (y2-y1)/(x2-x1)
    # print 'y = %.3fx + %.3f' %(factor,-(factor*x1) + y1)

    # Define the symbol to solve for
    x = Symbol("x")
    # Intersect the line with the circle
    x_intersect = solve(((y2 - y1) * (x - x1)) / (x2 - x1) + y1 - sqrt(
        rc * rc - x * x))  # take positive values of circle results (minus times plus = minus) // gives you all result for the positive circle (> 0)

    # Go thre all POSITIVE VALUES (check if one of the angles is between the two original angles; intersection net to be between)
    for x in x_intersect:

        # Get the corresponding y coordinate
        y_intersect = ((y2 - y1) * (x - x1)) / (x2 - x1) + y1

        # calculate Phi
        result_phi = math.acos(abs(x) / rc)

        # Again adjust to quadrant
        if x >= 0 and y_intersect >= 0:
            #print 'Upper Right'
            result = (pi / 2 - result_phi)
        elif x >= 0 and y_intersect <= 0:
            #print 'Lower Right'
            result = (pi / 2 + result_phi)
        elif x <= 0 and y_intersect <= 0:
            #print 'Lower Left'
            result = (((3 * pi) / 2) - result_phi)
        else:
            #print 'Upper Left'
            result = (((3 * pi) / 2) + result_phi)

        # if proper angle found return
        if result > phi1 and result < phi2:
            return result

    # Define the symbol to solve for
    x = Symbol("x")
    # Intersect the line with the circle
    x_intersect = solve(((y2 - y1) * (x - x1)) / (x2 - x1) + y1 + sqrt(
        rc * rc - x * x))  # take negative values of circle results (minus times plus = minus)// gives you all result for the negative circle (< 0)

    # Go thre all NEGATIVE VALUES (check if one of the angles is between the two original angles; intersection net to be between)
    for x in x_intersect:

        # Get the corresponding y coordinate
        y_intersect = ((y2 - y1) * (x - x1)) / (x2 - x1) + y1

        # calculate Phi
        result_phi = math.acos(abs(x) / rc)

        # Again adjust to quadrant
        if x >= 0 and y_intersect >= 0:
            #print 'Upper Right'
            result = (pi / 2 - result_phi)
        elif x >= 0 and y_intersect <= 0:
            #print 'Lower Right'
            result = (pi / 2 + result_phi)
        elif x <= 0 and y_intersect <= 0:
            #print 'Lower Left'
            result = (((3 * pi) / 2) - result_phi)
        else:
            #print 'Upper Left'
            result = (((3 * pi) / 2) + result_phi)

        # if proper angle found return
        if result > phi1 and result < phi2:
            return result

    return 'Nothing Found'

def my_SpiderPlot(categories, values, color, title,num='None', toNormalizeSmallest='None', toNormalizeBiggest="None"):
    '''
    Create a Spider Plot

    :param categories: categories of the spiderplots (the individual factors)
    :param values: actual values
    :param color:  the colorscheme (e.g. deactivating = red)
    :param title:  name of the spiederplot
    :param num:    in case of overlay (else just None for individual Spiderplots)
    :return:
    '''

    if toNormalizeSmallest !="None":
        #Normalize all values to a pre given value
        nullValue = int(toNormalizeSmallest) - 3

        newValues = [x + abs(toNormalizeSmallest) + 3 for x in values]

        max_yticks = int(toNormalizeBiggest) + 1
    else:
        #Get the lowest value (e.g . -10), for ploting this will be zero; add three so the lowest value is NOT in the middle but a bit away
        nullValue =  int(min(values)) - 3

        #Normalize all values, e.g. the -10 to zero, whereas the zero will be 10 in the plot
        newValues = [x+abs(min(values))+3 for x in values]

        #Define the max tick as max value plus one (for aesthetics)
        max_yticks = int(max(values))+3

    #get the negative ticks and positive ticks
    negative_ticks = [str(x) for x in range(nullValue,0,1)]
    positive_ticks = [str(x) for x in range(0,max_yticks+1,1)]
    negative_ticks.extend(positive_ticks)

    #print negative_ticks
    #exit()
    #Take only 8 tick marks
    to_take = len(negative_ticks)/8
    chosen_ticks = [negative_ticks[x] for x in range(0,len(negative_ticks),to_take)]

    #take the normalized values to plot (i.e. the values where the -10 became the zero
    values = newValues

    #Find number of categories
    N = len(categories)

    # What will be the angle of each axis in the plot? (we divide the plot / number of variable)
    # The total of 2pi (around 6.2) is divided into the amount of categories; In final plot it will be just from 0 till 2 in pi (factor 3.1415 is missing)
    angles = [n / float(N) * 2 * pi for n in range(N)]
    angles += angles[:1]

    # Initialise the spider plot
    if num != 'None':
        ax = plt.subplot(1, 3, num+ 1, polar=True, )
    else:
        ax = plt.subplot(1, 1, 1, polar=True, )

    # If you want the first axis to be on top:
    ax.set_theta_offset(pi / 2)
    ax.set_theta_direction(-1)

    # Draw one axe per variable + add labels labels yet
    #categories = [x.split('AllRandom')[0] for x in categories]
    plt.xticks(angles[:-1], categories, color='grey', size=8)

    # Draw ylabels
    ax.set_rlabel_position(0)

    #add last value, to close the circle
    values.append(values[0])

    #plot the line
    ax.plot(angles, values, color=color, linewidth=2, linestyle='solid')

    #ax.fill(angles, values, color=color, alpha=0.4)

    #Go threw all the points, whenever there is a switch between a positive and a negative ZScore, the line
    #intersects with the zero line, hence new color; use the find_intersection function to find probper intersection
    i_was = 'Nowhere'
    tmp = []
    tmp_angles = []
    to_save = []
    prev_val = 0
    prev_ang = 0

    angles_to_save_cut = []
    normal_angles = []

    #Go thre all values and angles
    for val,ang in zip(newValues,angles):

        #Check if value is positive or negative
        if val > abs(nullValue):
            i_am = 'Positive'
        else:
            i_am = 'Negative'

        #Check if there is a switch between positive and negative
        if i_was != i_am and i_was != 'Nowhere':

            #Define the radius of the circle (=y)
            y = abs(nullValue)

            #if the last line is between 3 quadrant and the origin (change 0.0 to 6.2831 = 2pi = full circle)
            if prev_ang > 3.15 and ang == 0.0:
                ang = 6.2831

            #Find the actual intersection
            result = Find_Intersection(y,prev_ang,prev_val,ang,val)
            angles_to_save_cut.append(result)

            #if more than one angle belongs to one section, before creating new tmp, add current to save
            if len(tmp) >0:
                to_save.append(tmp)
                normal_angles.append(tmp_angles)

            #start new tmp (= section of color)
            tmp = [val]
            tmp_angles = [ang]
        #if still in same section just add angle and value
        else:
            tmp.append(val)
            tmp_angles.append(ang)

        #Remember previous location
        i_was = i_am
        prev_val = val
        prev_ang = ang

    #Final results of intersection parts (angles and values)
    to_save.append(tmp)
    normal_angles.append(tmp_angles)

    #make a fine grained amount of angles (361 individual degrees), and close circle again
    angles2 = [n / float(360) * 2 * pi for n in range(360)]
    angles2 += angles2[:1]

    #Define color scheme
    '''
    colorscheme = {'green':{0:'#acd900',1:'#a6c143',2:'#648a58',3:'#5c5e4c',4:'#acd900',5:'#a6c143',6:'#648a58',7:'#5c5e4c'},
                   'red': {0: '#f70020', 1: '#e66a22', 2: '#e79935', 3: '#dcb471', 4: '#f70020',5:'#e66a22',6:'#e79935',7:'#dcb471'},
                   'blue':{0: '#0096ff', 1: '#2bbfb8', 2: '#29a2ac', 3: '#4c7584', 4: '#0096ff', 5: '#2bbfb8',6:'#29a2ac',7:'#4c7584'},
                   'grey':{0:'#252525',1:'#636363',2:'#969696',3:'#cccccc',4:'#f7f7f7'}

                   }
    '''
    '''
    colorscheme = {'green':{0:'#acd900',1:'#acd900',2:'#acd900',3:'#acd900',4:'#acd900',5:'#acd900',6:'#acd900',7:'#acd900'},
                   'red': {0: '#f70020', 1: '#f70020', 2: '#f70020', 3: '#f70020', 4: '#f70020',5:'#f70020',6:'#f70020',7:'#f70020'},
                   'blue':{0: '#0096ff', 1: '#0096ff', 2: '#0096ff', 3: '#0096ff', 4: '#0096ff', 5: '#0096ff',6:'#0096ff',7:'#0096ff'},
                   'grey':{0:'#252525',1:'#252525',2:'#252525',3:'#252525',4:'#252525'}

    }
    '''
    colorscheme = {'green':{0:'#acd900',1:'#a6c143',2:'#acd900',3:'#a6c143',4:'#acd900',5:'#a6c143',6:'#acd900',7:'#a6c143'},
                   'red': {0: '#f70020', 1: '#e66a22', 2: '#f70020', 3: '#e66a22', 4: '#f70020',5:'#e66a22',6:'#f70020',7:'#e66a22'},
                   'blue':{0: '#0096ff', 1: '#2bbfb8', 2: '#0096ff', 3: '#2bbfb8', 4: '#0096ff', 5: '#2bbfb8',6:'#0096ff',7:'#2bbfb8'},
                   'grey':{0:'#252525',1:'#636363',2:'#252525',3:'#636363',4:'#252525'}

    }

    #If the first section is bigger than one immedieatly
    nofirstcut = False
    if len(to_save[0]) > 0:
        angles_to_save_cut.insert(0,0)
        nofirstcut = True
    angles_to_save_cut += angles_to_save_cut[:1]

    #fill the individual parts
    for i in range(0,len(to_save)):

        #save_cut[i] to savecut[i+1] define the whole area, + add all the angles between these two
        to_fill_angles = [angles_to_save_cut[i]]
        to_fill_Values = [abs(nullValue)]

        to_fill_Values.extend(to_save[i])
        to_fill_angles.extend(normal_angles[i])

        to_fill_angles.append(angles_to_save_cut[i+1])
        to_fill_Values.append(abs(nullValue))

        #This part followes the zero line back to define where things should be filled
        if angles_to_save_cut[i+1] > angles_to_save_cut[i]:
            go_back = [x for x in angles2 if x < angles_to_save_cut[i+1] and x > angles_to_save_cut[i]]
            go_back = go_back[::-1]
            go_back.pop(0)

        else:
            go_back = [x for x in angles2 if  x < angles_to_save_cut[i+1]]
            go_back2 = [x for x in angles2 if x > angles_to_save_cut[i]]

            go_back = go_back[::-1]
            if 0 in go_back:
                go_back.pop(0)

            go_back2 = go_back2[::-1]

            go_back.extend(go_back2)

        #add here the previously go back angles and values (values is always the radius of the zero line)
        to_fill_angles.extend(go_back)
        to_fill_Values.extend([abs(nullValue)] * len(go_back))

        #in case there is a not directly a first cut adjust color
        if nofirstcut == True and i == len(to_save)-1:

            ax.fill(to_fill_angles, to_fill_Values, color=colorscheme[color][0])

        else:
            ax.fill(to_fill_angles, to_fill_Values, color=colorscheme[color][i])

    #Plot the zero zScore line plus and minus 2 (significance
    plt.plot(angles2,[abs(nullValue)]*361, color = 'black')
    plt.yticks(range(0,len(negative_ticks),to_take),chosen_ticks)
    # Add a title
    plt.title(title, size=11, color=color, y=1.1)
    plt.setp( ax.get_yticklabels(), visible=False)
    plt.setp( ax.get_xticklabels(), visible=False)
```

#### 3.2 Create actual radar plots
Go through the results of the indivdiual interactions types and create radar plots. The radar plots show Cohen's D (effect size) difference between the individual interaction results e.g. Increasing, Decreasing, Emergent compared to the overall interaction results. In case all 3 interaction types have very similar results, then also all 3 interactions types result in Cohen's D close to zero. High Cohen's D indicate big variability between interaction types

```python
# Interaction types
selected_results = {'Increasing':{},'Decreasing':{},'Emergent':{},'Interaction':{}}

# Define what to compare the results to, here choosen overall interaction results
compare_to = 'Interaction'

# The 12 representative features chosen to cover all feature classes
selected_features = ['ChemicalSimilarity','Enzymes_Overlap','Transporters_Overlap','PPI_Mean_AB_All_Filtered',
                     'KeGG_Indirect_Overlap','GO_Component','GO_Function','GO_Process','Msig_ChemGen_Perturbation_Overlap',
                     'SideEffects_CLOUD_to_Offsides_Overlap','SideEffects_TwoSide_CLOUDs','Disease']

# Read the result file and save the corresponding results
fp = open('../results/Molecular_Determinants/ResultsOverview.csv','r')
fp.next()
for line in fp:
    tmp = line.strip().split(',')

    if tmp[0] in selected_features:
        if tmp[1] == compare_to:
            selected_results[tmp[2]][tmp[0]] = float(tmp[8]) * -1
```

```python
#Define colors
interaction_colors = {'Increasing':'green','Decreasing':'red','Emergent':'blue','Interaction':'grey'}

# Create a spider plot for each interaction type separately
for key in ['Increasing','Decreasing','Emergent']:
    part =  key
    categories = []
    values = []
    for f in selected_features:
        categories.append(f)
        values.append(selected_results[key][f])

    my_SpiderPlot(categories, values, interaction_colors[part], part,'None',-1,2) #-1 and 2 for compare to Interaction, or -11 and 6
    plt.savefig('../results/Molecular_Determinants/SpiderPlots/'+part+'.pdf',format='pdf')
    plt.close()

# Create one spider plot for all 3 interaction types together
for key in ['Increasing','Decreasing','Emergent']:
    part =  key
    categories = []
    values = []
    for f in selected_features:
        categories.append(f)
        values.append(selected_results[key][f])

    my_SpiderPlot(categories, values, interaction_colors[part], part,'None',-1,2) #-1 and 2 for compare to Interaction, or -11 and 6
#plt.show()
plt.savefig('../results/Molecular_Determinants/SpiderPlots/Combined.pdf',format='pdf')
plt.close()
```

### 4 Create final overview plot
Create a final overview plot that visualizes which feature, in which network compartment (e.g. Core, Periphery ...) for which interaction type (increasing, decreasing ...) significant is. Therefor calculate depending on the type of interaction the signficance as well as a foldchange/oddsRation to get an idea wheter the feature is rather depleeted or enriched.

#### 4.1 Create binned results (per network layer)
Similar as in previous parts here, split the results accordingly into the various parts. Add here the network layer so that each result is properly sorted for each network layer as well as interaction type

```python
# The 12 representative features chosen to cover all feature classes
selected_features = ['ChemicalSimilarity','Enzymes_Overlap','Transporters_Overlap','PPI_Mean_AB_All_Filtered',
                     'KeGG_Indirect_Overlap','GO_Component','GO_Function','GO_Process','Msig_ChemGen_Perturbation_Overlap',
                     'SideEffects_CLOUD_to_Offsides_Overlap','SideEffects_TwoSide_CLOUDs','Disease']

# Define size and colors for the final plot
interaction_colors = {'Increasing':'#ACD900','Decreasing':'#F70020','Emergent':'#0096FF','All':'black'}
interaction_sizes = {'Increasing':200,'Decreasing':200,'Emergent':200,'All':2000}
network_parts = ['Complete','Core','CoreToPeriphery','Periphery']

# Get the result file
fp = open('../data/Molecular_Determinants/DrugPair_Feature_Overview.csv','r')

# Dictionary that will contain the information which interactions belong to which network layer
network_part_interactions = {}

# add the individual network parts to the network_part_interactions (result dictionary)
for part in network_parts:
    network_part_interactions[part] = []
    network_part = nx.read_gml('../data/Molecular_Determinants/Networks/DPI_Network_'+part+'.gml')
    network_part_interactions[part] = network_part

# List containing all features (i.e. features in DrugPair_Feature_Overview.csv = all investigated features )
features = fp.readline().strip().split(',')[4:]

# Dictionary that will contain the individual results, split for network layers as well as interaction types
network_part_values = {}

# go through all features
for f in range(0,len(features)):
    # always start at row one (first row containing results, zero row = header)
    fp.seek(0)
    fp.next()

    # only continue if the feature is one of the representative features
    if features[f] not in selected_features:
        continue

    print features[f]
    # add section for this feature to the result dictionary:  network_part_values
    network_part_values[features[f]] = {}

    # as next level add the individual network parts as well as interaction types
    for part in network_parts:
        network_part_values[features[f]][part] = {'Increasing':[],'Decreasing':[],'Emergent':[]}
    network_part_values[features[f]]['AllCLOUDS'] = []
    network_part_values[features[f]]['NonInteracting'] = []

    # now go through all results and add every feature result into the correct bin
    for line in fp:
        tmp = line.strip().split(',')

        # do not include 'nan' values (e.g. if one drug has no targets then PPI mean distance = nan)
        if tmp[f+4] == 'nan':
            continue

        interaction_found = False

        #only include pure single edges e.g. do not include increasing/decreasing interactions
        if tmp[3] == 'Increasing' or  tmp[3] == 'Decreasing' or  tmp[3] == 'Emergent' or tmp[3] == 'None':

            # AllCLOUDs = all pairs (is always added)
            network_part_values[features[f]]['AllCLOUDS'].append(float(tmp[f+4]))

            ######
            # Add the result accordingly (which interaction type or network layer it belongs)
            # This creates the actual final network_part_values dictionary that will be used in the next step to create the overview plot
            for part in network_parts:
                if  network_part_interactions[part].has_edge(tmp[0],tmp[1]):
                    interaction_found = True
                    for key in network_part_interactions[part][tmp[0]][tmp[1]]:
                        network_part_values[features[f]][part][network_part_interactions[part][tmp[0]][tmp[1]][key]['Type']].append(float(tmp[f+4]))

                if  network_part_interactions[part].has_edge(tmp[1],tmp[0]):
                    interaction_found = True
                    for key in network_part_interactions[part][tmp[1]][tmp[0]]:
                        network_part_values[features[f]][part][network_part_interactions[part][tmp[1]][tmp[0]][key]['Type']].append(float(tmp[f+4]))

            if interaction_found == False:
                network_part_values[features[f]]['NonInteracting'].append(float(tmp[f+4]))
fp.close()
```

#### 4.2 Create actual overview plot
Use the results stored in network_part_values to create an easy overview plot. Split the result into the individual network layers (= rows) and features (=columns). Each cell (row X column) will have 4 triangles: one black big triangle = All interaction types, and 3 smaller ones indicating the individual interaction type results (emergent, increasing and decreasing)

```python
# Create overview plot for SELECTED features
all_Features = selected_features

# Create overview plot for ALL features
#all_Features = features

# Get number of features
number_features = len(all_Features)

# Change size of the final plot accordingly to the number of features (more features = larger plot)
plt.figure(figsize=(number_features,4))
#plt.tight_layout()
feature_names = []

# position to start adding entries, with each features will be increased by 0.2
current_x = 0.8

# Go through all features
for f in  all_Features:
    print f

    #add the feature name
    feature_names.append(f)

    #save NoInteraction values
    no_interaction_values = network_part_values[f]['NonInteracting']

    # similar as current_x, defines where to put the results on the y axis
    y = 4.2

    #Go through all network parts
    for part in network_parts:

        # AllInteractions is simply the union of the 3 different interaction types
        AllInteractions = network_part_values[f][part]['Increasing']  + network_part_values[f][part]['Decreasing'] + network_part_values[f][part]['Emergent']

        # Things to test include AllInteraction, Increasing, Decreasing and Emergent
        things_to_test = {'All':AllInteractions,'Increasing':network_part_values[f][part]['Increasing'],'Decreasing':network_part_values[f][part]['Decreasing'],'Emergent':network_part_values[f][part]['Emergent']}

        # Check wheter the feature is a continues feature (Mann Whitney U) or a binary feature (Fisher Exact test)
        continues_features = True
        if all(v == 0 or v ==1 for v in no_interaction_values):
            continues_features = False

        x = current_x
        # Calculate the signficance accordingly
        for subset in ['All','Increasing','Decreasing','Emergent']:

            # If continues feature calculate significance according to Mann Whitney U
            if  continues_features:
                direction = np.mean(things_to_test[subset]) > np.mean(no_interaction_values)
                sign =  mu(things_to_test[subset],no_interaction_values)[1] < 0.05
            # If binary feature calculate significance according to Fisher Exact test
            else:
                real_Overlap = sum(things_to_test[subset])
                real_NonOverlap = len(things_to_test[subset]) - real_Overlap

                non_Interactions_Overlap = sum(no_interaction_values)
                non_Interactions_NonOverlap = len(no_interaction_values) - sum(no_interaction_values)

                oddsratio, pvalue = stats.fisher_exact([[real_Overlap, real_NonOverlap], [non_Interactions_Overlap, non_Interactions_NonOverlap]])

                sign = pvalue < 0.05
                direction = oddsratio > 1

            # Depending on the fold change/ oddsRation define if the feature is rather depleeted or enriched (arrow down or arrow up)
            if direction:
                symbol = '^'
            else:
                symbol = 'v'

            if sign:
                color = interaction_colors[subset]
            else:
                color = 'grey'

            # add the cell entry accordingly (color if significant, arrow according to depleetion or emergence)
            x = x + 0.2
            plt.scatter([x],[y],marker=symbol, s=interaction_sizes[subset], color=color)
        y = y - 1
    current_x = current_x + 1

# Create the output folder if it doesn't exist
directory = os.path.dirname('../results/Molecular_Determinants/')
if not os.path.exists(directory):
    os.makedirs(directory)

# Create the final output (overview plot)
plt.ylim([0.6,4.8])
plt.xlim([0.3,number_features+1])
plt.yticks([1.2,2.2,3.2,4.2],['Periphery','CoreToPeriphery','Core','Complete'])
plt.xticks(range(1,number_features),feature_names, rotation='vertical')
plt.savefig('../results/Molecular_Determinants/Overviewplot.pdf', bbox_inches = "tight")
plt.close()
```


# Particle-Field interaction example

This notebook illustrates a simple way to make particles interact with a ``` Field```  object and modify it. The ``` Field``` will thus change at each step of the simulation, and will be written using the same time resolution as the particle outputs, in as many ```netCDF``` files.

The concept is similar to that of [Field sampling](https://nbviewer.jupyter.org/github/OceanParcels/parcels/blob/particle_interaction_prototype/parcels/examples/tutorial_sampling.ipynb): here instead, on top of reading the field value at their location, particles are able to alter it as defined in the ```Kernel```. To do this, it is important to keep in mind that:

- Particles have to be defined as ```ScipyParticles```
- ```Field``` writing at each ```outputdt``` is not default and has to be enabled
- The time of the ```Field``` to be saved has to be updated within a ```Kernel```

In this example, particles will carry a tracer and release it into a clean ```Field``` during their advection by surface currents. To show how can particles interact with a ```Field``` and alter it, the exchange of such tracer is modelled here with a discretized version of the mass transfer equation, defined as follows:

\begin{equation}
\Delta c_{particle}(t) = aC_{field}(t-1) - bc_{particle}(t-1)
\tag{1}
\label{1}
\end{equation}

In Eq.1, $c_{particle}$ is the tracer concentration associated with the particle, $C_{field}$ is the tracer concentration in seawater at particle location, and $a$ and $b$ are weights that modulate the sorption of tracer from seawater and its desorption, respectively.

Additionally to a relevant ```Kernel```, we will define a suitable particle class to store $c_{particle}$, as it is needed to solve Eq.1.

### Particle altering a Field during advection

```python
%matplotlib inline
from parcels import Variable, Field, FieldSet, ParticleSet, ScipyParticle, AdvectionRK4, plotTrajectoriesFile

import numpy as np
from datetime import timedelta as delta
import netCDF4
import matplotlib.pyplot as plt
```

In this specific example, particles will be advected by surface ocean velocities stored in netCDF files in the folder ```GlobCurrent_example_data```. We will store these in a ```FieldSet``` object, and then add a ```Field``` to it to represent the tracer field. This latter field will be initialized with zeroes, as we assume that this tracer is absent on the ocean surface and released by particles only. Note that, in order to conserve mass, it is important to set `interp_method='nearest'` for the tracer Field.

As we are interested in storing this new field during the simulation, we will set its ```.to_write``` method to ```True```. Note that this only works for Fields that consist of only _one_ snapshot in time.

```python
# Velocity fields
fname = r'GlobCurrent_example_data/*.nc'
filenames = {'U': fname, 'V': fname}
variables = {'U': 'eastward_eulerian_current_velocity', 'V': 'northward_eulerian_current_velocity'}
dimensions = {'U': {'lat': 'lat', 'lon': 'lon', 'time': 'time'},
              'V': {'lat': 'lat', 'lon': 'lon', 'time': 'time'},
             }
fieldset = FieldSet.from_netcdf(filenames, variables, dimensions)

# In order to assign the same grid to the tracer field, it is convenient to load a single velocity file
fname1 = r'GlobCurrent_example_data/20030101000000-GLOBCURRENT-L4-CUReul_hs-ALT_SUM-v02.0-fv01.0.nc'
filenames1 = {'U': fname1, 'V': fname1}

field_for_size = FieldSet.from_netcdf(filenames1, variables, dimensions) # this field has the same variables and dimensions as the other velocity fields

# Adding the tracer field to the FieldSet
dimsC = [len(field_for_size.U.lat),len(field_for_size.U.lon)] # it has to have the same dimentions as the velocity fields
dataC = np.zeros([dimsC[0],dimsC[1]])

fieldC = Field('C', dataC, grid=field_for_size.U.grid, interp_method='nearest') # the new Field will be called C, for tracer Concentration. For mass conservation, interp_method='nearest'

fieldset.add_field(fieldC) # C field added to the velocity FieldSet

fieldset.C.to_write = True # enabling the writing of Field C during execution
```

```python
fieldset.C.show() # our new C field has been added to the FieldSet
```

Some global parameters have to be defined, such as $a$ and $b$ of Eq.1, and a weight that works as a conversion factor from $\Delta c_{particle}$ to $C_{field}$.
We will add these parameters to the ```FieldSet```.

```python
fieldset.add_constant('a', 10)
fieldset.add_constant('b', .2)
fieldset.add_constant('weight', .01)
```

We will now define a new particle class. A ```VectorParticle``` is a ```ScipyParticle``` having a ```Variable``` to store the current tracer concentration ```c``` associated with it. As in this case we want our particles to release a tracer into a clean field, we will initialize ```c``` with an arbitrary value of `100`.

We also need to define the ```Kernel``` that performs the particle-field interaction. In this Kernel, we will implement Eq.1, so that $\Delta c_{particle}$ can be used to update $c_{particle}$ and $C_{field}$ at the particle location, and thus get their values at the current time $t$.

Additionally, the time of ```fieldset.C``` is updated within the ```Interaction``` Kernel, which is an important step to properly write its value.

```python
class VectorParticle(ScipyParticle):
    c = Variable('c', dtype=np.float32, initial=100.) # particle concentration c is initialized with a non-zero value

def Interaction(particle, fieldset, time):
    deltaC = (fieldset.a*fieldset.C[particle]-fieldset.b*particle.c) # the exchange is obtained as a discretized mass transfer equation

    xi, yi = particle.xi[fieldset.C.igrid], particle.yi[fieldset.C.igrid],
    if abs(particle.lon - fieldset.C.grid.lon[xi+1]) < abs(particle.lon - fieldset.C.grid.lon[xi]):
        xi += 1
    if abs(particle.lat - fieldset.C.grid.lat[yi+1]) < abs(particle.lat - fieldset.C.grid.lat[yi]):
        yi += 1

    particle.c += deltaC
    fieldset.C.data[0, yi, xi] += -deltaC*fieldset.weight # weight, defined as a constant for the FieldSet, acts here as a conversion factor between c_particle and C_field
    fieldset.C.grid.time[0] = time # updating Field C time

def WriteInitial(particle, fieldset, time): # will be used to store the initial conditions of fieldset.C
    fieldset.C.grid.time[0] = time

pset = ParticleSet(fieldset=fieldset, pclass=VectorParticle, lon=[24.5], lat=[-34.8]) # for simplicity, we'll track a single particle here
```

Three things are worth noticing in the code above:
- The use of ```fieldset.C[particle]```
- The computation of the relevant grid cell (`xi`, `yi`)
- Writing $C_{field}$ through ```fieldset.C.data[0, yi, xi]```

Because `fieldset.C[particle]` interpolates the $C_{field}$ value from the nearest grid cell to the particle, it is important to also write to that same grid cell. That is not completely trivial to do in Parcels, which is why lines 7-11 in the cell above calculate which `xi` and `yi` are closest to the particle longitude and latitude (this extends trivially to depth too). Our first guess is `particle.xi[fieldset.C.igrid]`, the location in the particular grid, but it could be that `xi+1` is closer, which is what the `if`-statements are for.

The new indices are then used in ```fieldset.C.data[0, yi, xi]``` to access the field value in the cell where the particle is found, that can be different from the result of the interpolation at the particle's coordinates. Note that here we need to use ```fieldset.C.data[0, yi, xi]``` both for calculating ```deltaC``` and for the consequent update of the cell value for consistency between the forcing (the seawater-particle gradient) and its effect (the exchange, and consequent alteration of the field).

Remember that reading and writing ```Fields``` at particle location through particle indices is only possible for ```ScipyParticles``` (and returns an error if ```JITParticles``` are used).

```python
pset.show(field=fieldset.C) # Initial particle location and the tracer field C
```

Now we are going to execute the advection of the particle and the simultaneous release of the tracer it carries. We will thus add the ```interactionKernel``` defined above to the built-in Kernel ```AdvectionRK4```.

Before running the advection, we will execute the ```pset``` with the ```WriteInitial``` for ```dt=0```: this will write the initial condition of fieldset.C to a ```netCDF``` file.

While particle outputs will be written in a file named ```interaction.nc``` at every ```outputdt```, the field will be automatically written in ```netCDF``` files named ```interaction_wxyzC.nc```, with ```wxyz``` being the number of the output and ```C``` the ```FieldSet``` variable of our interest. Note that you can use tools like [ncrcat](https://linux.die.net/man/1/ncrcat) (on linux/macOS) to combine these separate files into one large ```netCDF``` file after the simualtion.

```python
output_file = pset.ParticleFile(name=r'interaction.nc', outputdt=delta(days=1))

pset.execute(WriteInitial, dt=0., output_file=output_file)

pset.execute(AdvectionRK4 + pset.Kernel(Interaction), # the particle will FIRST be transported by currents and THEN interact with the field
             dt=delta(days=1),
             runtime=delta(days=24), # we are going to track the particle and save its trajectory and tracer concentration for 24 days
             output_file=output_file)

output_file.close()
```

    WARNING: dt or runtime are zero, or endtime is equal to Particle.time. The kernels will be executed once, without incrementing time

We can see that $c_{particle}$ has been saved along with particle trajectory, as expected.

```python
pset_traj = netCDF4.Dataset(r'interaction.nc')

print(pset_traj['c'][:])

plotTrajectoriesFile('interaction.nc');
```

But what about ```fieldset.C```? We can see that it has been accordingly modified during particle motion. Using ```fieldset.C``` we can access the field as resulting at the end of the run, with no information about the previous time steps.

```python
c_results = fieldset.C.data[0,:,:].copy() # Copying the final field data in a new array
c_results[[field_for_size.U.data==0][0][0]]= np.nan # using a mask for fieldset.C.data on land
c_results[c_results==0] = np.nan # masking the field where its value is zero -- areas that have not been modified by the particle, for clearer plotting

try: # Works if Cartopy is installed
    import cartopy
    import cartopy.crs as ccrs
    extent = [10, 33, -37, -29]

    X = fieldset.U.lon
    Y = fieldset.U.lat

    plt.figure(figsize=(12, 6))
    ax = plt.axes(projection=ccrs.Mercator())
    ax.set_extent(extent)

    ax.add_feature(cartopy.feature.OCEAN, facecolor='lightgrey')
    ax.add_feature(cartopy.feature.LAND, edgecolor='black', facecolor='floralwhite')
    gl=ax.gridlines(xlocs = np.linspace(10,34,13) , ylocs=np.linspace(-29,-37,9),draw_labels=True)
    gl.right_labels = False
    gl.bottom_labels = False

    xx, yy = np.meshgrid(X,Y)

    results = ax.pcolormesh(xx,yy,(c_results),transform=ccrs.PlateCarree(),vmin=0,)
    cbar=plt.colorbar(mappable = results, ax=ax)
    cbar.ax.text(.8,.070,'$C_{field}$ concentration', rotation=270, fontsize=12)

except:
    print('Please install the Cartopy package.')
```

When looking at tracer concentrations, we see that $c_{particle}$ decreases along its trajectory (right to left), as it is releasing the tracer it carries. Accordingly, values of $C_{field}$ provided by particle interaction progressively reduce along the particle's route.

Notice that the first particle-field interaction occurs at time $t = 1$ day, and namely after the execution of the first step of ```AdvectionRK4```, as shown by the unaltered field value at the particle's starting location.
In order to let the particle interact before being advected, we would have to change the order in which the two Kernels are added together in ```pset.execute```, i.e. ```pset.execute(interactionKernel + AdvectionRK4, ...)```. In this latter case, the interaction would not occur at the particle's final position instead.

```python
x_centers, y_centers = np.meshgrid(fieldset.U.lon-np.diff(fieldset.U.lon[:2])/2, fieldset.U.lat-np.diff(fieldset.U.lat[:2])/2)

fig,ax = plt.subplots(1,1,figsize=(10,7),constrained_layout=True)
ax.set_facecolor('lightgrey') # For visual coherence with the plot above

fieldplot=ax.pcolormesh(x_centers[-28:-17,22:41],y_centers[-28:-17,22:41],c_results[-28:-18,22:40], vmin=0, vmax=0.2,cmap='viridis')
# Zoom on the area of interest
field_cbar = plt.colorbar(fieldplot,ax=ax)
field_cbar.ax.text(.6,.070,'$C_{field}$ concentration', rotation=270, fontsize=12)

particle = plt.scatter(pset_traj['lon'][:].data[0,:],pset_traj['lat'][:].data[0,:], c=pset_traj['c'][:].data[0,:],vmin=0, s=100, edgecolor='white')
particle_cbar = plt.colorbar(particle,ax=ax, location = 'top')
particle_cbar.ax.text(40,300,'$c_{particle}$ concentration', fontsize=12);
```

Finally, to see the ```C``` field in time we have to load the ```.nc``` files produced during the run. In the following plots, particle location and field values are shown at each time step.

```python
fig, ax = plt.subplots(5,5, figsize=(30,20))

daycounter = 1

for i in range(len(ax)):
    for j in range(len(ax)):
        data = netCDF4.Dataset(r'interaction_00'+ '%02d' % daycounter+'C.nc')

        c_results = data['C'][0,0,:,:].data.copy() # copying the final field data in a new array
        c_results[[field_for_size.U.data==0][0][0]]= np.nan # using a mask for fieldset.C.data on land
        c_results[c_results==0] = np.nan # masking the field where its value is zero -- areas that have not been modified by the particle, for clearer plotting

        ax[i,j].set_facecolor('lightgrey') # For visual coherence with the plots above

        fieldplot=ax[i,j].pcolormesh(x_centers[-28:-17,22:41],y_centers[-28:-17,22:41],c_results[-28:-18,22:40], vmin=0, vmax=0.2,cmap='viridis')

        particle = ax[i,j].scatter(pset_traj['lon'][:].data[0,daycounter-1],pset_traj['lat'][:].data[0,daycounter-1], c=pset_traj['c'][:].data[0,daycounter-1],vmin=0, vmax=100, s=100, edgecolor='white')
        # plotting particle location at current time step -- daycounter-1 due to different indexing

        ax[i,j].set_title('Day '+ str(daycounter-1))

        daycounter +=1 # next day

fig.subplots_adjust(right=0.8)
fig.subplots_adjust(top=0.8)

cbar_ax = fig.add_axes([0.82, 0.12, 0.03, 0.7])
fig.colorbar(fieldplot, cax=cbar_ax)
cbar_ax.tick_params(labelsize=18)
cbar_ax.text(.4,.08,'$C_{field}$ concentration', fontsize=25, rotation=270)

cbar_ax1 = fig.add_axes([0.1, .85, .7, 0.04])
fig.colorbar(particle, cax=cbar_ax1, orientation = 'horizontal')
cbar_ax1.tick_params(labelsize=18)
cbar_ax1.text(42,170,'$c_{particle}$ concentration', fontsize=25);
```


# Shear/Strain Parameterization

This section is available as jupyter notebook in the `docs/` directory or at https://github.com/modscripps/mixsea/tree/master/docs.

Import `mixsea` and a few other modules:

```python
import mixsea as mx
import numpy as np
import matplotlib.pyplot as plt
import gsw
```

**Basic idea**

(Unfortunately shear-strain parameterizations for internal wave-driven mixing are not basic at all; this is as clear and concise a summary as possible.)

At the root of the shear-strain parameterization is the assumption of a scale separation between the production of turbulent kinetic energy $\mathcal{P}$ (extracted from a mean shear) at low wave numbers and its destruction by both viscous dissipation $\epsilon$ and vertical buoyancy flux $\overline{w'b'}$ at high wave numbers. In stationary turbulence, non-linear wave-wave interactions at some intermediate ("fine"-)scale $m_{c}$ must be responsible for this downscale energy flux $\mathcal{T}_{\text{IW}}$, i.e.
\begin{equation}
-\mathcal{T}_{\text{IW}}(m_{c}) =  \mathcal{P} = - \epsilon + \overline{w'b'}
\end{equation}

Shear-strain parameterizations get their popular name based on the functional form $\mathcal{G}$ of the downscale transfer function that arises from either heuristic (Polzin 2004) or ray-tracing (Henyey et al. 1986) methods:
\begin{equation}
\mathcal{T}_{\text{IW}} = \mathcal{G}(E_{\text{KE}}, E_{\text{PE}}; \overline{N}, f),
\end{equation}
where:

- $E_{\text{KE}}$ is the *kinetic energy spectral density*, often expressed as the shear variance $\langle S^{2} \rangle = \int_{m_{\text{low}}}^{\min\{m_{\text{high}}, m_{c}\}} m^{2}E_{\text{KE}} \; \text{d} m$ integrated over a low wavenumber band [$m_{\text{low}}$, $m_{\text{high}}$], which is possibly cutoff early at $m_{c}$ as soon the cumulative shear variance increases above $1/6$ the amount required for wave breaking by shear instability $\big(\text{i.e. }R_{f} = \langle N^{2} \rangle / \langle S^{2} \rangle = 1/4 \big)$;

- $E_{\text{PE}}$ is the *potential energy spectral density*, often expressed in terms of a strain variance $\langle \xi_{z}^{2} \rangle$, where $\xi$ is the displacement of a density surface, or in terms of the ratio  of shear to strain variance, $R_{\omega} = E_{\text{KE}}/E_{\text{PE}}$, which also doubles as a characteristic internal wave aspect ratio (strain variance also has a high cutoff wavenumber);

- $\overline{N}$ is the mean stratification; and

- $f$ is the Coriolis frequency.

Strain-only parameterizations assume a characteristic constant shear to strain ratio and thus replace the shear variance with more readily available strain variance measurements, $\langle S^{2} \rangle \simeq \langle \xi_{z}^{2} \rangle R_{\omega}$.

In practice, such parameterizations are conventionally formulated as scalings of the Garrett-Munk (1975) empirical internal wave energy spectrum. The exact forms of the parameterization used are described in the underlying functions' docstrings– see the Methods section of Gregg (2003) for a concise example or Kunze (2006) for a detailed exposition of the default method implemented in `mixsea`.

#### Read example data

```python
ctd = mx.helpers.read_ctd_testfile()
ladcp = mx.helpers.read_ladcp_testfile()
```

Our example data contain some NaN values, as is common with observational datasets. Below we isolate the good data by removing NaNs. See the section [Missing Data](#Missing-Data) for using `nan_shearstrain` with input data that contains NaNs.

```python
def nonan(data):
    notnan = [np.isfinite(v) for k, v in data.items()]
    notnan = np.vstack(notnan)
    notnan = np.all(notnan, axis=0)
    return notnan

notnan = nonan(ctd)
depth = ctd["depth"][notnan]
t = ctd["t"][notnan]
SP = ctd["SP"][notnan]
lon = ctd["lon"][0]
lat = ctd["lat"][0]

notnan = nonan(ladcp)
u = ladcp["u"][notnan]
v = ladcp["v"][notnan]
uz = ladcp["uz"][notnan]
vz = ladcp["vz"][notnan]
depth_sh = ladcp["depth"][notnan]
```

A quick overview plot of the data:

```python
fig, ax = plt.subplots(
    nrows=1, ncols=3, figsize=(9, 4), constrained_layout=True, sharey=True
)
ax[0].plot(t, depth)
ax[0].set(ylabel="depth [m]", xlabel="temperature [°C]")
ax[1].plot(u, depth_sh, label="u")
ax[1].plot(v, depth_sh, label="v")
ax[1].set(xlabel="velocity [m/s]")
ax[1].legend()
ax[2].plot(uz, depth_sh, label=r"u$_{z}$")
ax[2].plot(vz, depth_sh, label=r"v$_{z}$")
ax[2].set(xlabel="shear [1/s]")
ax[2].legend()
ax[0].invert_yaxis()
```

## Example of the shear/strain internal wave parameterization

We begin by choosing our parameters. The key parameter choices are:

- `window_size`: the vertical extent of the spectral windows used to estimate shear/strain spectra;
- `min_size`: the shortest vertical wavelength resolved by the data (should be a factor of `window_size`); doesn't matter much in practice because we separately constrain the spectral windows with the two following parameters.
- `mi_sh`: index limits for integrating shear variance (excluding upper limit, following Python convention). Lower bound is generally just set to 0 so that the gravest mode included is the `window_size`; upper bound is generally set to 100-150 m.
- `mi_st`: index limits for integrating strain variance (excluding upper limit, following Python convention). Lower bound is generally set to 100-150 m and upper bound set to 10-20 m. In practice, the integration is often cut-off well before 10 m.

```python
shst_params = dict()

# Center points of depth windows. Windows are half overlapping, i.e.
# their size (300m) is double the spacing here (150m).
window_size = 300.0
min_size = 10.0
dz = window_size / 2
shst_params["depth_bin"] = np.arange(dz, 10000.0, dz)
shst_params["window_size"] = window_size

# Set up wavenumber vector.
shst_params["m"] = np.arange(
    2 * np.pi / window_size, 2 * np.pi / min_size, 2 * np.pi / window_size
)

# Set up limits for shear and strain variance integrations
mi_sh = np.array([0, 3])
mii_sh = np.array(range(*mi_sh))
mi_st = np.array([2, 20])
mii_st = np.array(range(*mi_st))

shst_params["m_include_sh"] = mii_sh
shst_params["m_include_st"] = mii_st

# Convert indices to more intuitive length scales
m_sh = 2 * np.pi / shst_params["m"][[mi_sh[0], mi_sh[1] - 1]]
m_st = 2 * np.pi / shst_params["m"][[mi_st[0], mi_st[1] - 1]]
print(
    f"Wavenumber indices for integration:\n"
    f"- Shear is integrated from {round(m_sh[0])}m to {round(m_sh[1])}m scales.\n"
    f"- Strain is integrated from {round(m_st[0])}m to {round(m_st[1])}m."
)

shst_params["ladcp_is_shear"] = True
shst_params["return_diagnostics"] = True
```

We now pass our input data and parameter dictionary to the `shearstrain` function and retrieve the dissipation rate of turbulent kinetic energy $\epsilon$, the diapycnal diffusivity $\kappa_{\rho}$, and an optional dictionary of detailed diagnostics.

```python
eps, krho, diag = mx.shearstrain.shearstrain(
    depth, t, SP, lon, lat, uz, vz, depth_sh, **shst_params
)
```

Let's visualize the vertical profiles of the estimated mixing rates due to internal waves:

```python
depth_bin = diag["depth_bin"]
depth_bin_edges = np.concatenate(
    (
        [np.min(depth_bin) - dz / 2],
        0.5 * (depth_bin[1:] + depth_bin[:-1]),
        [np.max(depth_bin) - dz / 2],
    )
)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(6, 4), sharey=True, constrained_layout=True)
ax[0].stairs(eps, depth_bin_edges, orientation="horizontal", baseline=None, lw=1.5)
ax[0].set(xscale="log", xlabel=r"$\epsilon$ [W/kg]", ylabel="depth [m]")
ax[0].invert_yaxis()

ax[1].stairs(krho, depth_bin_edges, orientation="horizontal", baseline=None, lw=1.5)
ax[1].set(xscale="log", xlabel=r"$\kappa$ [m$^{2}$/s]")

k = -2
ax[0].fill_betweenx(
    np.array([-dz, dz]) + depth_bin[k],
    [0, 0],
    [eps[k], eps[k]],
    facecolor="C3",
    alpha=0.2,
)
ax[1].fill_betweenx(
    np.array([-dz, dz]) + depth_bin[k],
    [0, 0],
    [krho[k], krho[k]],
    facecolor="C3",
    alpha=0.2,
);
```

#### Deep dive for a single high-dissipation window near the bottom

Let's zoom in on the second to last window (shaded in red above), which exhibits abnormally strong strain variance and thus corresponds to a large dissipation rate, to get a better understanding of what the shear/strain variance and strain variance algorithms are doing.

```python
ax = plt.subplot()
plt.loglog(diag["m"][mii_sh], diag["P_shear"][k][mii_sh])
plt.loglog(diag["m"], diag["P_shear"][k], "C0--", alpha=0.25)
S_shear_gm, P_shear_gm = mx.shearstrain.gm_shear_variance(
    diag["m"], mii_sh, diag["Nmseg"][k]
)
plt.loglog(diag["m"], P_shear_gm, label="GM76 spectrum (minus rolloff)")
ylims = plt.gca().get_ylim()
plt.fill_between(
    diag["m"][[mi_sh[0], mi_sh[1] - 1]],
    [ylims[0], ylims[0]],
    [ylims[1], ylims[1]],
    alpha=0.1,
)
plt.ylim(ylims)
plt.xlim(diag["m"][0], diag["m"][-1])
plt.ylabel(r"$\langle S^{2} \rangle$", fontsize=13)
plt.xlabel(r"wavenumber $m$ [rad/m]", fontsize=13)
plt.legend()

def same(x):
    return x

secax = ax.secondary_xaxis("top", functions=(same, same))
secax.set_xticks([diag["m"][mii_sh[0]], diag["m"][mii_sh[-1]]], minor=False)
secax.set_xticklabels([r"$m_{low}$", r"$m_{high}$"])

var_ratio = diag["Int_sh"][k] / diag["Int_sh_gm"][k]
print(
    f"Shear variance levels are {round(var_ratio,1)} times higher than\n"
    f"the GM spectrum over the low wavenumber band."
)
```

```python
ax = plt.subplot()
plt.loglog(diag["m"][mii_st], diag["P_strain"][k][mii_st])
plt.loglog(diag["m"], diag["P_strain"][k], "C0--", alpha=0.25)
S_strain_gm, P_strain_gm = mx.shearstrain.gm_strain_variance(
    diag["m"], mii_st, diag["Nmseg"][k]
)
plt.loglog(diag["m"], P_strain_gm, label="GM76 spectrum")
mii_st_cutoff = mii_st[diag["m"][mii_st] <= diag["Mmax_st"][k]]
S_strain_gm, P_strain_gm = mx.shearstrain.gm_strain_variance(
    diag["m"], mii_st_cutoff, diag["Nmseg"][k]
)
ylims = plt.gca().get_ylim()
plt.fill_between(
    [diag["m"][mii_st_cutoff[0]], diag["Mmax_st"][k]],
    [ylims[0], ylims[0]],
    [ylims[1], ylims[1]],
    alpha=0.1,
)
plt.ylim(ylims)
plt.xlim(diag["m"][0], diag["m"][-1])
plt.ylabel(r"$\langle \xi_{z}^{2} \rangle$", fontsize=13)
plt.xlabel(r"wavenumber $m$ [rad/m]", fontsize=13)
plt.legend()

secax = ax.secondary_xaxis("top", functions=(same, same))
secax.set_xticks(
    [diag["m"][mii_st[0]], diag["m"][mii_st[-1]], diag["Mmax_st"][k]], minor=False
)
secax.set_xticklabels([r"$m_{low}$", r"$m_{high}$", r"$m_{c}$"])

print(
    rf"Strain variance levels are {round(diag['Int_st'][k]/S_strain_gm,1)} times "
    f"higher than\nthe GM spectrum over the low wavenumber band."
)
```

In this example, since the strain variance is much larger than GM levels while the shear variance is only slightly large than GM levels, the shear/strain ratio drops well below the GM level of $R_{\omega}$=3. Since shear/strain ratios of less than $1$ are unphysical for internal waves (for example the aspect ratio correction terms contain a factor $\sqrt{R_{\omega} - 1}$), we set any $R_{\omega} \leq 1$ to $R_{\omega}=1.01$.

```python
print(
    f"Raw shear/strain ratio: {round(diag['Rwtot'][k], 2)}, "
    f"corrected ratio: {diag['Rwcor'][k]}"
)
```

#### Decomposing the terms of the parameterization

The dissipation rate for the shear/strain parameterization is given as:
\begin{equation}
\epsilon_{\text{fine}} = \epsilon_{0} \; \frac{\overline{N}^{2}}{N_{0}^{2}} \; \frac{\langle S^{2} \rangle}{\langle S^{2}_{\text{GM}} \rangle} \; h_{1}(R_{\omega}) \; L(f, \overline{N})
\end{equation}

It is useful to plot their vertical profiles to get a sense of how their various dependencies combine to produce a typical dissipation rate profile, where the grey dashed lines denote the reference values for the GM spectrum.

```python
N0 = 5.24e-3
f = np.absolute(gsw.f(ctd["lat"][0]))

fig, ax = plt.subplots(
    nrows=1, ncols=5, figsize=(9, 4), sharey=True, constrained_layout=True
)
ax[0].stairs(
    eps,
    depth_bin_edges,
    label=r"$\epsilon$",
    orientation="horizontal",
    baseline=None,
    lw=1.5,
)
ax[0].set(xscale="log", xlabel=r"$\epsilon$ [W/kg]", ylabel="depth [m]")
ax[0].invert_yaxis()

ax[1].axvline(1.0, color="k", alpha=0.3, ls="--")
ax[1].stairs(
    diag["Nmseg"] ** 2 / N0 ** 2,
    depth_bin_edges,
    orientation="horizontal",
    baseline=None,
    lw=1.5,
)
ax[1].set(xscale="log", xlabel=r"$\overline{N}^{2}/N_{0}^{2}$")

ax[2].axvline(1.0, color="k", alpha=0.3, ls="--")
ax[2].stairs(
    diag["Int_sh"] ** 2 / diag["Int_sh_gm"] ** 2,
    depth_bin_edges,
    orientation="horizontal",
    baseline=None,
    lw=1.5,
)
ax[2].set(
    xscale="log",
    xlabel=r"$\langle S^{2} \rangle^{2} / \langle S^{2}_{GM} \rangle^{2}$ [W/kg]",
)

ax[3].axvline(
    mx.shearstrain.aspect_ratio_correction_shst(3.0), color="k", alpha=0.3, ls="--"
)
ax[3].stairs(
    mx.shearstrain.aspect_ratio_correction_shst(diag["Rwcor"]),
    depth_bin_edges,
    orientation="horizontal",
    baseline=None,
    lw=1.5,
)
ax[3].set(xscale="log", xlabel=r"$h_{1}(R_{\omega})$")

ax[4].axvline(1.0, color="k", alpha=0.3, ls="--")
ax[4].stairs(
    mx.shearstrain.latitude_correction(f, diag["Nmseg"]),
    depth_bin_edges,
    orientation="horizontal",
    baseline=None,
    lw=1.5,
)
ax[4].set(xlabel=r"$L(f, N)$")

eps0 = 7.8e-10
eps_verify = (
    eps0
    * diag["Nmseg"] ** 2
    / N0 ** 2
    * diag["Int_sh"] ** 2
    / diag["Int_sh_gm"] ** 2
    * mx.shearstrain.aspect_ratio_correction_shst(diag["Rwcor"])
    * mx.shearstrain.latitude_correction(f, diag["Nmseg"])
)
ax[0].stairs(
    eps_verify,
    depth_bin_edges,
    label=r"$\epsilon_{verify}$",
    orientation="horizontal",
    baseline=None,
    lw=1.2,
    ls="--",
    alpha=1.0,
)
ax[0].legend()

print(f"Verification successful: {np.isclose(eps_verify, eps).all()}")
```

## Stratification Smoothing Techniques

`shearstrain` allows to select between two different techniques for obtaining a smooth profile of $N^2$. Adiabatic leveling is the default (used in the example above).

**Adiabatic Leveling** finds locally stable density gradients to calculate a smooth $N^2$ profile following Bray & Fofonoff (1981), see e.g. Polzin et al. (2014), Waterman et al. (2013, 2014).

**Polynomial Fitting** applies a second order polynomial fit to $N^2$ for each segment to obtain a smooth profile, compare e.g. Kunze et al. (2006), Whalen et al. (2012, 2015).

Run the shear/strain parameterization for both the techniques:

```python
eps_PF, krho_PF, diag_PF = mx.shearstrain.shearstrain(
    depth, t, SP, lon, lat, uz, vz, depth_sh, smooth="PF", **shst_params
)
```

Compare results from the two techniques:

```python
fig, ax = plt.subplots(
    nrows=1, ncols=2, figsize=(6, 4), constrained_layout=True, sharey=True
)
opts = dict(orientation="horizontal", baseline=None)

# shear/strain
ax[0].stairs(eps, depth_bin_edges, label="AL", **opts)
ax[0].stairs(eps_PF, depth_bin_edges, label="PF", **opts)
ax[0].legend(loc="upper left")
ax[0].set(
    xscale="log", xlabel=r"$\epsilon$ [W/kg]", ylabel="depth [m]", title="shear/strain"
)
ax[0].invert_yaxis()

# strain only
ax[1].stairs(diag["eps_st"], depth_bin_edges, label="AL", **opts)
ax[1].stairs(diag_PF["eps_st"], depth_bin_edges, label="PF", **opts)
ax[1].set(xscale="log", xlabel=r"$\epsilon$ [W/kg]", title="strain only");
```

Results differ mostly for the upper ocean and the bottom layer. Note that we do not filter out parts of the profile where the shear/strain parameterization may not apply as underlying assumptions are violated, i.e. near the surface mixed layer and in the lowly stratified layer of the abyssal flow through the Samoan Passage.

## Missing Data

Here we pass the data as is, containing NaNs, to `nan_shearstrain`.

```python
eps_nan, krho_nan, diag_nan = mx.shearstrain.nan_shearstrain(
    ctd["depth"],
    ctd["t"],
    ctd["SP"],
    ctd["lon"],
    ctd["lat"],
    ladcp["uz"],
    ladcp["vz"],
    ladcp["depth"],
    **shst_params
)
```

Compare with result of `shearstrain` from above - the results should be the same. Here we apply only the polynomial fit method for obtaining strain.

```python
depth_bin_nan = diag_nan["depth_bin"]
depth_bin_nan_edges = np.concatenate(
    (
        [np.min(depth_bin_nan) - dz / 2],
        0.5 * (depth_bin_nan[1:] + depth_bin_nan[:-1]),
        [np.max(depth_bin_nan) - dz / 2],
    )
)

fig, ax = plt.subplots(
    nrows=1, ncols=2, figsize=(6, 4), constrained_layout=True, sharey=True
)

# shear/strain
ax[0].stairs(eps, depth_bin_edges, label="without NaNs", **opts)
ax[0].stairs(eps_nan, depth_bin_nan_edges, label="with NaNs", linestyle="--", **opts)
ax[0].legend()
ax[0].set(
    xscale="log", xlabel=r"$\epsilon$ [W/kg]", ylabel="depth [m]", title="shear/strain"
)
ax[0].invert_yaxis()

# strain only
ax[1].stairs(diag["eps_st"], depth_bin_edges, label="without NaNs", **opts)
ax[1].stairs(
    diag_nan["eps_st"], depth_bin_nan_edges, label="with NaNs", linestyle="--", **opts
)
ax[1].legend()
ax[1].set(
    xscale="log",
    xlabel=r"$\epsilon$ [W/kg]",
    title="strain only",
);
```

Note that there may be cases where simply removing all missing data from the input, as is happening in `nan_shearstrain`, will lead to unphysical results!


# Using Landlab to explore a diffusive hillslope in the piedmont of North Carolina

This notebook was developed in collaboration with Karl Wegmann at North Carolina State University. This notebook was coded by Nicole Gasparini at Tulane University.

For tutorials on learning Landlab, click here: https://github.com/landlab/landlab/wiki/Tutorials

**What is this notebook?**

This notebook is designed to be an exercise for students in a quantative geomorphology class. This notebook is meant to illustrate to students how a model can be used to understand and interpret real landscapes. This notebook is not designed to teach students how to code, nor is it designed to teach students how to use Landlab.

This notebook provides data from a real landscape in North Carolina (below) in order to compare the shape of this hillslope with those produced by models using linear diffusion. A group of students at NC State have collected elevation data along the yellow transect in the image of the landscape below. This notebook uses that transect data and compares it  with model output.

This notebook steps students through exercises to quantify the diffusivity on this hillslope, assuming that sediment transport on this hillslope follows a linear diffusion law. Students should be introduced to the linear diffusion transport law _before_ using this notebook.

**Application of linear diffusion transport law:**

The tranport law applied here is of the form:
\begin{equation}
q_s = -D \nabla z
\end{equation}
where ${q}_s$ is the transport rate with dimensions of L$^2$T$^{-1}$;  $D$ is a transport coefficient with dimensions of L$^2$T$^{-1}$; and $z$ is elevation. $\nabla z$ is the gradient in the landscape, or change in elevation with change in distance. Landscape slope is $-\nabla z$ (assuming distance increases downhill), hence the negative in the above equation.

Changes in elevation, or erosion, are calculated from conservation of mass:
\begin{equation}
\frac{dz}{dt} = U-\nabla q_s
\end{equation}
where $U$ is the rock uplift rate (relative to a fixed elevation at the boundary), with dimensions LT$^{-1}$, and $t$ is time.

Note that at steady state the rock uplift rate sets the erosion rate. If the erosion rate in a landscape is a known, a modeling scenario in which the landscape is uplifting at the same rate as the known erosion rate will eventually lead to a landscape that is also eroding at that rate.

Although we call $U$ the rate of rock uplift rate, we would get the same solution if $U$ is the rate of river erosion in the river at the base of a hillslope. For this example, it might easier to think of $U$ as a base-level lowering rate.

**What will we do with Landlab?**

Here we will use the Landlab component *LinearDiffuser*, to explore whether hillslopes evolved according to a linear diffusion rule look like the example North Carolina hillslope. We will just be eye-balling similarities. No rigorous comparison between the real and modeled hillslope is performed.

More general background on applying hillslope process models to real landscapes can be found in this paper:

Roering, Joshua J. (2008) "How well can hillslope evolution models “explain” topography? Simulating soil transport and production with high-resolution topographic data." Geological Society of America Bulletin.

This paper has a nice compilation of measured landscape diffusivity values that can help put the values you get from this exercise in context:

Perron, J. T., (2017) "Climate and the pace of erosional landscape evolution" Annual Review of Earth and Planetary Sciences.

**What do you need to know about this Landscape?**

General information on this landscape can be found in this reference:

Wegmann, K. W., Lewis, R. Q., & Hunt, M. C. (2012). "Historic mill ponds and piedmont stream water quality: Making the connection near Raleigh, North Carolina". The Geological Society of America, Field Guide 29.

For now the following information is provided.

- The site is in the Piedmont near Raleigh, North Carolina.
- The site is in the W.B. Umstead State Park which is in the Sycamore Creek watershed. Sycamore Creek drains into the Neuse River.
- The area gets ~ 1.25 m/yr of rainfall.
- The site is currently forested.
- Basin average erosion rates from the area are on the order of 10 m/million years, or 1e-5 m/yr. These rates are averaged over approximately 100,000 years. These rates are from a nearby area similar to this, but not exactly this study area.
- The site has been heavily impacted from agriculture due to European settlers, beginning in the 1600s.

**STUDENTS - Step 0 - What you need to do:**

You will start out by making the assumption that the landscape is eroding at a rate of 1e-5 m/yr.

As you will see, the study area hillslope profile has a form similar to what one would expect from a uniformly eroding diffusive profile. **Your first goal is to find the diffusivity and time required for the profile to reach steady state given the background erosion rate of 1e-5 m/yr.** You can do that by running the code below. **Take special note of Code Block 4 where you set the diffusivity. Make sure choose a resonable initial value, and record what your initial value was.** Code Block 5 will allow you to see the analytical solution for hillslope form using different $D$ Values. You can play with the analystical solution to find the best $D$. Also be prepared to change model run time (Code Block 4) for your initial experiments. Once you iterate to a modeled profile that you feel reasonably matches the DEM profile (just eye-ball the similarity), you will be ready to answer the questions that follow all of the code.

**How to run a Jupyter notebook:**

Read the text boxes and sequentially run each code block (shift - enter OR got to the _Cell_ pulldown menu at the top and choose _Run Cells_). While a code block is running there is a * in the brackets next to it. Once it has completed running there will be a number in the brackets next to it

Remember that you can always go to the _Kernel_ pulldown menu at the top and choose _Restart & Clear Output_ or _Restart & Run All_ if you change things and want to start afresh. If you just change one code block and rerun only that code block, only the parts of the code in that code block will be updated. (E.g. if you change parameters but don't reset the code blocks that initialize run time or topography, then these values will not be reset.)

**Now on to the code...**

We start by importing libraries that we will need to run the code. You should not need to edit this code block.

```python
# Code Block 1

import numpy as np
from landlab.io import read_esri_ascii
from landlab.plot.imshow import imshow_grid
import matplotlib.pyplot as plt
#below is to make plots show up in the notebook
%matplotlib inline
```

Now we will use the survey data from the NC State students and compare it to model output. Below is the information from the survey. You should not need to edit this code block.

```python
# Code Block 2

# distance and elevation data along the survey line

field_dist = np.array([0,1,2,3,4,4.99,5.99,6.99,7.99,8.99,9.99,10.99,11.99,
                       12.99,13.99,14.98,15.98,16.98,17.98,18.98,19.98,
                       20.98,21.98,22.98,23.98,24.97,25.97,26.97,27.97,
                       28.97,29.97,30.97,31.97,32.97,33.97,34.96,35.96,
                       36.96,37.96,38.96,39.96,40.96,41.96,42.96,43.95,
                       44.95,45.95,46.95,49.95,50.95,51.95,52.95,53.94,
                       54.94,55.94,56.94,57.94,58.94,59.94,60.94,61.94,
                       62.94,63.93,64.93,65.93,66.93,67.93,68.93,69.93,
                       70.93,71.93,72.92,73.92,74.92,75.92,76.92,77.92,
                       78.92,79.92,80.92,81.92,82.91,83.91,84.91,85.91,
                       86.91,87.91,88.91])
field_z = np.array([0,0.03,0.3,0.47,0.62,0.83,1.09,1.31,1.54,1.8,2.14,2.38,
                    2.55,2.84,3.15,3.49,3.78,4.05,4.41,4.57,4.77,5.05,5.29,
                    5.46,5.68,5.96,6.4,6.81,6.99,7.21,7.45,7.63,7.79,7.87,
                    8.06,8.24,8.4,8.51,8.65,8.68,8.82,8.98,9.01,9.04,9.05,
                    9.09,9.07,9.07,9.02,8.93,8.9,8.83,8.73,8.62,8.47,8.28,
                    8.22,8,7.82,7.75,7.39,7.2,7.04,6.79,6.6,6.39,6.1,5.77,
                    5.5,5.3,5.11,4.89,4.64,4.5,4.32,4.1,3.96,3.6,3.19,2.92,
                    2.73,2.41,2.12,1.76,1.21,0.95,0.56,0.06])
```

We will create a grid for our model using Landlab's *RasterModelGrid* class, which we need to import. We make a grid with *dx* = *dy* = 1.0 m (same resolution as the survey data). We make a grid that has 5 columns and 90 rows, to match the length of the profile from the real landscape.

```python
# Code Block 3

from landlab import RasterModelGrid
mg = RasterModelGrid((90, 5), 1.0) # make grid
z_vals = mg.add_ones('topographic__elevation', at='node') # initialize z values

# Set initial conditions
initial_elevation = np.multiply(z_vals, -1.0) # this line and next set elevation to 87. m
z_vals += initial_elevation

# Set boundary conditions
mg.set_closed_boundaries_at_grid_edges(True, False, True, False)

# Initialize values for plotting variables down the middle of the hillslope

ycoord_rast = mg.node_vector_to_raster(mg.node_y)
ys_grid = ycoord_rast[:, 2]
```

Now we import and initialize the *LinearDiffuser* component. In this case the units on our diffusivity coefficient, or transport coefficient, are m$^2$yr$^{-1}$.

**NOTE to Students:** You need to chose a reasonable initial value for D (diffusivity). Remember you need to justify your initial guess for D. Supplied references should help with this.

We also initialize a few more parameters.

```python
# Code Block 4

from landlab.components import LinearDiffuser
D = 0.005 # value in m^2/yr
lin_diffuse = LinearDiffuser(mg, linear_diffusivity=D)

# Uniform rate of rock uplift, which drives uniform erosion at steady state
uplift_rate = 0.00001  # m/year, start with 1e-5 m/yr

# Total time in years that the model will run for.
runtime = 500000 # years

# Stability criteria for timestep dt.  Coefficient can be changed
# depending on our tolerance for stability vs tolerance for run time.
# Do not change this.
dt = 0.5 * mg.dx * mg.dx / D

print("dt", dt)

# nt is number of time steps
nt = int(runtime // dt)

# Below is to keep track of time for labeling plots
time_counter = 0

# length of uplift over a single time step, meters
uplift_per_step = uplift_rate * dt
```

Now we figure out the analytical solution for the elevation of the steady-state profile.

Before we evolve the landscape, we look at the model initial topography, the analytical solution, and the field profile.

```python
# Code Block 5

#ANALYTICAL SOLUTION
ys = np.arange(mg.number_of_node_rows*mg.dx-mg.dx)

# location of divide or ridge crest -> middle of grid
# based on boundary conds.
divide_loc = (mg.number_of_node_rows*mg.dx-mg.dx)/2

# half-width of the ridge
half_width = (mg.number_of_node_rows*mg.dx-mg.dx)/2

# analytical solution for elevation under linear diffusion at steady state
zs = (uplift_rate/(2*D)) * \
     (np.power(half_width, 2) - np.power(ys - divide_loc, 2))

#PLOTTING
plt.figure()
imshow_grid(mg, 'topographic__elevation')
plt.title('initial topography, at right is the colorbar')
plt.figure()
elev_rast = mg.node_vector_to_raster(
            mg.at_node['topographic__elevation'])
plt.figure()
plt.plot(ys_grid, elev_rast[:, 2], 'r-', label='model')
plt.plot(ys, zs, 'k--', label='analytical solution')
plt.plot(field_dist, field_z, 'b:', label='field data')
plt.xlabel('horizontal distance (m)')
plt.ylabel('elevation (m)')
plt.legend(loc='lower center')
#plt.title('before running model')
```

Now you can update the values of $D$ in Code Block 4 and rerun Code Block 5 to fit the analytical solution to the field data.

Once you have done that, run the model and evolve the landscape to make sure it also fits the analytical solution. You may need to update $runtime$ in Code Block 4 if the model does not match the analytical solution. Remember that you need to find the $D$ value and approximate time that it takes for the landscape to reach steady state.

Below is the time loop that does all the calculations.

```python
# Code Block 6

for i in range(nt):
    mg['node']['topographic__elevation'][mg.core_nodes] += uplift_per_step
    lin_diffuse.run_one_step(dt)
    time_counter += dt

print('time evolved for ',time_counter,' years')
```

Now we plot the evolved cross-section.

```python
# Code Block 7
plt.figure()
elev_rast = mg.node_vector_to_raster(
            mg.at_node['topographic__elevation'])
plt.plot(ys_grid, elev_rast[:, 2], 'r-', label='model')
plt.plot(ys, zs, 'k--', label='analytical solution')
plt.plot(field_dist, field_z, 'b:', label='field data')
plt.xlabel('horizontal distance (m)')
plt.ylabel('vertical elevation (m)')
plt.legend(loc='lower center')
plt.title('topographic cross section at time %s, with D = %s m^2/yr'%(time_counter,D))
```

**Questions to answer and further model experiments:**

_NOTE_ You should hand in a typed, electronic document that has figures embedded in the document. The document can be any type of readable file. Please include your name in the file name that you turn in.

**Questions:**
1. Your first task was to find the diffusivity and time to steady state required for the model to produce a landscape similar to the field area near Raleigh, NC. (Time to steady state can be approximate, as in within 500,000 years.) What was your initial guess for a diffusivity? What information did you use to choose that initial value? What was the value that produced a good match between the modeled and real hillslope form? (Again, you can eye-ball the match. Just get close visually.) Compared with other values of diffusivity that have been reported in the literature, how does your value compare? Given what you know about this landscape, does this seem like a reasonable diffusivity? Explain.
2. What was the estimated time to steady state using the model? Compare the time to steady state with the time over which erosion rates were measured and the time period over which settlers have dramatically impacted this landscape. What are the implications of the differences among the time to steady state, the time over which erosion rates were measured, and time over which settlers have impacted this system?
3. It is possible that settlers accelerated erosion rates (from the background, pre-settlement rate, which here we can assume is set from "background rock uplift" as 1e-5 m/yr) by almost 500 times. If that happened, the pre-settlement profile likely had a higher peak elevation (and higher total relief), and accelerated erosion would have removed soil and lowered elevations across the profile. This has implications for your estimated diffusivity.
    - Develop a scenario in which you evolve a 'pre-settlement' hillslope to steady state (should have higher relief than the current profile) with a different diffusivity. Think about whether the diffusivity should be higher or lower to create higher relief, using the same background erosion/rock uplift rate. Change Code Block 4 and create this new pre-settlement steady state landscape using Code Blocks 6 and 7.
    - Use this landscape as the initial condition to evolve the landscape another 400 years with a new diffusivity to simulate the impact of settlers. (Once you get your pre-settlement hillslope, go back to Code Block 4 and change $D$ and $runtime$ accordingly, then rerun Code Blocks 6 & 7).
    - You should end up with a simulated hillslope that matches the current topography, but you got there in a different way than you did for the first part of this exercise. Provide plots and describe in detail exactly what you did to simulate the pre- and post-settler landscape. Are there noticeable changes in the landscape? Does this have implications for future evolution of this hillslope?

```python

```