### Daytum +2 Course: Data Analytics, Geostatistics and Machine Learning Deep Dive

### Declustering Demonstration and Exercise

#### Goal

Calculate representative statistics from a sparsely, spatially-biased dataset.

#### Description

Here's a simple workflow with some basic univariate statistics and distribution representativity. This should help you get started data declustering to address spatial sampling bias.

#### Geostatistical Sampling Representativity

In general, we should assume that all spatial data that we work with is biased.

##### Source of Spatial Sampling Bias

Data is collected to answer questions:
* how far does the contaminant plume extend? – sample peripheries
* where is the fault? – drill based on seismic interpretation
* what is the highest mineral grade? – sample the best part
* who far does the reservoir extend? – offset drilling
and to maximize NPV directly:
* maximize production rates

**Random Sampling**: when every item in the population has a equal chance of being chosen. Selection of every item is independent of every other selection.
Is random sampling sufficient for subsurface?  Is it available?
* it is not usually available, would not be economic
* data is collected answer questions
    * how large is the reservoir, what is the thickest part of the reservoir
* and wells are located to maximize future production
    * dual purpose appraisal and injection / production wells!

**Regular Sampling**: when samples are taken at regular intervals (equally spaced).
* less reliable than random sampling.
* Warning: may resonate with some unsuspected environmental variable.

What do we have?
* we usually have biased, opportunity sampling
* we must account for bias (debiasing will be discussed later)

So if we were designing sampling for representativity of the sample set and resulting sample statistics, by theory we have 2 options, random sampling and regular sampling.

* What would happen if you proposed random sampling in the Gulf of Mexico at $150M per well?

We should not change current sampling methods as they result in best economics, we should address sampling bias in the data.

Never use raw spatial data without access sampling bias / correcting.

##### Mitigating Sampling Bias

In this demonstration we will take a biased spatial sample data set and apply declustering using **GeostatsPy** functionality.

#### Cell-based Declustering

Cell Declustering, is robust in 3-D and when the limits / boundary of the volume of interest are poorly defined.  The steps include:

* divide the volume of interest into a grid of cells $l=1,\ldots,L$
* count the occupied cells $L_o$ and the number of data in each cell $n_l, l=1,\ldots, L_o$
* assign weight to each $\alpha$ data sample, scaled inversely to the number of data in the cell (standardize by $\frac{n}{L_o}$ to constrain the weights to sum to 1.0)

\begin{equation}
w_\alpha = \frac{1}{n_l} \times \frac{n}{L_o}, \forall \quad \alpha = 1,\ldots,n
\end{equation}

INSERT FIGURE from SLIDE 21 of 03_DataPrep.pptx

#### Other Declustering Methods

There are a variety of declustering methods that are available. This includes:

* polygonal declustering with voronoi polygons within the area of interest
* kriging for declustering with weights assigned relative to the total weight assigned to each data over all estimates
* geological mapping with the distribution average corrected to the average over the entire map

We will demonstrate cell-based declustering with the original GSLIB (Deutsch and Journel, 1997) re-implemented in Python in GeostatsPy (Pyrcz et al., 2019).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")             # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv')     # load our data table (wrong name!)
```

No error now! It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
print(df.iloc[0:5,:])                   # display first 4 samples in the table as a preview
df.head(n=13)                           # we could also use this command for a table preview
```

         X    Y  Facies  Porosity       Perm
    0  100  900       1  0.115359   5.736104
    1  100  800       1  0.136425  17.211462
    2  100  600       1  0.135810  43.724752
    3  100  500       0  0.094414   1.609942
    4  100  100       0  0.113049  10.886001

      X
      Y
      Facies
      Porosity
      Perm

      0
      100
      900
      1
      0.115359
      5.736104

      1
      100
      800
      1
      0.136425
      17.211462

      2
      100
      600
      1
      0.135810
      43.724752

      3
      100
      500
      0
      0.094414
      1.609942

      4
      100
      100
      0
      0.113049
      10.886001

      5
      200
      800
      1
      0.154648
      106.491795

      6
      200
      700
      1
      0.153113
      140.976324

      7
      200
      500
      1
      0.126167
      12.548074

      8
      200
      400
      0
      0.094750
      1.208561

      9
      200
      100
      1
      0.150961
      44.687430

      10
      300
      800
      1
      0.199227
      1079.709291

      11
      300
      700
      1
      0.154220
      179.491695

      12
      300
      500
      1
      0.137502
      38.164911

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 2 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      289.0
      475.813149
      254.277530
      0.000000
      300.000000
      430.000000
      670.000000
      990.000000

      Y
      289.0
      529.692042
      300.895374
      9.000000
      269.000000
      549.000000
      819.000000
      999.000000

      Facies
      289.0
      0.813149
      0.390468
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      289.0
      0.134744
      0.037745
      0.058548
      0.106318
      0.126167
      0.154220
      0.228790

      Perm
      289.0
      207.832368
      559.359350
      0.075819
      3.634086
      14.908970
      71.454424
      5308.842566

#### Specify the Area of Interest

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;             # range of porosity values
cmap = plt.cm.plasma                    # color map
```

Visualizing Tabular Data with Location Maps¶
Let's try out locmap. This is a reimplementation of GSLIB's locmap program that uses matplotlib. I hope you find it simpler than matplotlib, if you want to get more advanced and build custom plots lock at the source. If you improve it, send me the new code. Any help is appreciated. To see the parameters, just type the command name:

```python
GSLIB.locmap
```

    <function geostatspy.GSLIB.locmap(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, vmin, vmax, title, xlabel, ylabel, vlabel, cmap, fig_name)>

Now we can populate the plotting parameters and visualize the porosity data.

```python
GSLIB.locmap(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap,'locmap_Porosity')
```

Look carefully, and you'll notice the the spatial samples are more dense in the high porosity regions and lower in the low porosity regions.  There is preferential sampling.  We cannot use the naive statistics to represent this region.  We have to correct for the clustering of the samples in the high porosity regions.

Let's try cell declustering. We can interpret that we will want to minimize the declustering mean and that a cell size of between 100 - 200m is likely a good cell size, this is 'an ocular' estimate of the largest average spacing in the sparsely sampled regions.

Let's check out the declus program reimplimented from GSLIB.

```python
geostats.declus
```

    <function geostatspy.geostats.declus(df, xcol, ycol, vcol, iminmax, noff, ncell, cmin, cmax)>

We can now populate the parameters. We will run a very wide range of cell sizes, from 10m to 2,000m ('cmin' and 'cmax') and take the cell size that minimizes the declustered mean ('iminmax' = 1 minimize, and = 0 maximize). Multiple offsets (number of these is 'noff') uses multiple grid origins and averages the results to remove sensitivity to grid position.  The ncell is the number of cell sizes.

The output from this program is:

* wts - an array with the weigths for each data (they sum to the number of data, 1 indicates nominal weight)
* cell_sizes - an array with the considered cell sizes
* dmeans - de an

```python
wts, cell_sizes, dmeans = geostats.declus(df,'X','Y','Porosity',iminmax = 1, noff= 10, ncell=100,cmin=10,cmax=2000)
df['Wts'] = wts                            # add weights to the sample data DataFrame
df.head()                                  # preview to check the sample data DataFrame
```

    There are 289 data with:
       mean of      0.13474387540138408
       min and max  0.058547873 and 0.228790002
       standard dev 0.03767982164385207

      X
      Y
      Facies
      Porosity
      Perm
      Wts

      0
      100
      900
      1
      0.115359
      5.736104
      3.064286

      1
      100
      800
      1
      0.136425
      17.211462
      1.076608

      2
      100
      600
      1
      0.135810
      43.724752
      0.997239

      3
      100
      500
      0
      0.094414
      1.609942
      1.165119

      4
      100
      100
      0
      0.113049
      10.886001
      1.224164

Let's look at the location map of the weights.

```python
GSLIB.locmap(df,'X','Y','Wts',xmin,xmax,ymin,ymax,0.5,2.5,'Well Data Weights','X(m)','Y(m)','Weights',cmap,'locmap_Weights')
```

Does it look correct?  See the weight varies with local sampling density?

Now let's add the distribution of the weights and the naive and declustered porosity distributions. You should see the histogram bars adjusted by the weights. Also note the change in the mean due to the weights. There is a significant change.

```python
plt.subplot(221)
GSLIB.locmap_st(df,'X','Y','Wts',xmin,xmax,ymin,ymax,0.0,2.0,'Declustering Weights','X (m)','Y (m)','Weights',cmap)

plt.subplot(222)
GSLIB.hist_st(df['Wts'],0.5,2.5,log=False,cumul=False,bins=20,weights=None,xlabel="Weights",title="Declustering Weights")
plt.ylim(0.0,60)

plt.subplot(223)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=20,weights=None,xlabel="Porosity",title="Naive Porosity")
plt.ylim(0.0,60)

plt.subplot(224)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=20,weights=df['Wts'],xlabel="Porosity",title="Declustered Porosity")
plt.ylim(0.0,60)

por_mean = np.average(df['Porosity'].values)
por_dmean = np.average(df['Porosity'].values,weights=df['Wts'].values)

print('Porosity naive mean is ' + str(round(por_mean,3))+'.')
print('Porosity declustered mean is ' + str(round(por_dmean,3))+'.')
cor = (por_mean-por_dmean)/por_mean
print('Correction of ' + str(round(cor,4)) +'.')

print('\nSummary statistics of the declsutering weights:')
print(stats.describe(wts))

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.5, wspace=0.2, hspace=0.2)
plt.show()
```

Now let's look at the plot of the declustered porosity mean vs. the declustering cell size over the 100 runs. At very small and very large cell size the declustered mean is the naive mean.

```python
plt.subplot(111)
plt.scatter(cell_sizes,dmeans, s=30, alpha = 0.2, edgecolors = "black", facecolors = 'red')
plt.xlabel('Cell Size (m)')
plt.ylabel('Declustered Porosity Mean (fraction)')
plt.title('Declustered Porosity Mean vs. Cell Size')
plt.plot([0,2000],[por_mean,por_mean],color = 'black')
plt.plot([200,200],[0.10,0.16],color = 'black',linestyle='dashed')
plt.text(300., 0.136, r'Naive Porosity Mean')
plt.text(500., 0.118, r'Declustered Porosity Mean')
plt.text(230., 0.154, r'Minimizing')
plt.text(230., 0.150, r'Cell Size')
plt.ylim(0.10,0.16)
plt.xlim(0,2000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.2, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

The cell size that minimizes the declustered mean is about 200m (estimated from the figure). This makes sense given our previous observation of the data spacing.

#### Comments

 I have other demonstrations on the basics of working with DataFrames, ndarrays and many other workflows availble at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.
#### Comments

This was a basic demonstration of declustering to correct for sampling bias. Much more could be done such as the application of additional declustering methods.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Demo of the robustness score
Sources:
1. [Summary on the robustness of interpretability methods [Medium]](https://medium.com/@mohamadt/summary-on-the-robustness-of-interpretability-methods-ffb26ba22776)
2. [On the Robustness of Interpretability Methods [Arxiv]](https://arxiv.org/pdf/1806.08049.pdf)

```python
%load_ext autoreload
%autoreload 2
```

```python
import matplotlib.pyplot as plt
import numpy as np
from tqdm import tqdm
from lipschitz_metric import (lipschitz_metric,
                              get_perturbations,
                              explainer_func_wrapper)
```

## Combining LIME and Lipschitz robustness metric for classification
Let's import a dataset, train a simple classifier and use LIME to explain the model. Then, let's use the Lipschitz robustness score to check the stability of the explanations. To better illustrate the use case of the Lipschitz metric, notis that we set `tol=0.1`. This is a much higher tolerance than the default, and will stop the training earlier which in term leads to a less reliable model.

```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import classification_report
from lime import lime_tabular

iris = load_iris(as_frame=True)
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y)

model = SVC(probability=True, tol=0.1)  # <-- NB! tol=0.1
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred, target_names=iris.target_names))
```

                  precision    recall  f1-score   support

          setosa       1.00      1.00      1.00         9
      versicolor       1.00      0.94      0.97        16
       virginica       0.93      1.00      0.96        13

        accuracy                           0.97        38
       macro avg       0.98      0.98      0.98        38
    weighted avg       0.98      0.97      0.97        38

```python
lime_explainer = lime_tabular.LimeTabularExplainer(X_train.values, feature_names=iris.feature_names, class_names=list(iris.target_names), discretize_continuous=False, sample_around_instance=True)

x0 = X_test.iloc[0].values

exp = lime_explainer.explain_instance(x0, model.predict_proba)

print(f"Input: {x0}")
print(f"Explanations: {exp.as_list()}")
exp.as_pyplot_figure();

```

### Calculating the Lipschitz robustness metric for one instance

We can use the Lipschitz metric as a measure of the stability of the explanations given by LIME. To do this, we create a functions that does three steps:

1. Wrap the explainer method of LIME so that it input the input to the ML-model and outputs the feature contributions
2. Calculate some small, random perturbations away from the input and the corresponding LIME explanations
3. Use the perturbations to calculate the L-metric

```python
def get_L_metric(x0, input_space, x_range=None, num_perturbations=100):
    # Create a function that accepts the ML-model input and outputs the explainers feature contributions
    explainer_func = explainer_func_wrapper(
        predict_fn=model.predict_proba,
        explainer_handle=lime_explainer,
        explainer='lime'
    )

    # Calculate perturbations and the corresponding explanations
    x0, exp0, x_perturbations, exp_perturbations = get_perturbations(
        func=explainer_func,
        x0=x0,
        input_space=input_space,
        num_perturbations=num_perturbations
    )

    # Calculate the L-metric (and return the input that caused this L-metric)
    L, x_worst_case = lipschitz_metric(x0, exp0, x_perturbations, exp_perturbations, x_range=x_range)
    exp_worst_case = explainer_func(x_worst_case)

    return L, x0, exp0, x_worst_case, exp_worst_case

input_space = [(x_min, x_max) for x_min, x_max in zip(X_train.min(), X_train.max())]
x0 = X_test.iloc[0].values

L, x0, exp0, x_worst_case, exp_worst_case = get_L_metric(x0=x0, input_space=input_space, x_range=input_space)

print(f"Robustness score: {L:.2f}\n")
print("                                                                   Input     Output")
print(f"Original:   {str(x0):>60} --> {exp0}")
print(f"Worst-case: {str(x_worst_case):>60} --> {exp_worst_case}\n")
print(f"Diff:       {str(x_worst_case-x0):>60} --> {exp_worst_case-exp0}")
```

    Robustness score: 11.20

                                                                       Input     Output
    Original:                                            [[6.1 2.8 4.  1.3]] --> [[-0.13059349 -0.0736244   0.04692542 -0.00192733]]
    Worst-case:                [6.12808387 2.79784176 3.94549631 1.28093966] --> [[-0.10115217 -0.05961491  0.05933459 -0.00156356]]

    Diff:                [[ 0.02808387 -0.00215824 -0.05450369 -0.01906034]] --> [[0.02944132 0.01400949 0.01240917 0.00036376]]

### Calculating the Lipschitz robustness metric for one instance
The example above calculated the Lipschitz metric for *one* instance. Let us now do the same thing for the whole test set. This will reveal where LIME has unstable explanations.

(One could also sample the input space and calculate the Lipschitz metric for a larger area.)

```python
input_space = [(x_min, x_max) for x_min, x_max in zip(X_train.min(), X_train.max())]
L_metric = []
x_worst_cases = {col: [] for col in X_test.columns}

for col, row in tqdm(X_test.iterrows(), total=X_test.shape[0]):
    x0 = row.values

    L, x0, exp0, x_worst_case, exp_worst_case = get_L_metric(x0=x0, input_space=input_space, x_range=input_space, num_perturbations=500)

    L_metric.append(L)
    for i, col in enumerate(x_worst_cases):
        x_worst_cases[col].append(x_worst_case[i])

X_test_analysis = X_test.copy()
X_test_analysis['L-metric'] = L_metric
for col, values in x_worst_cases.items():
    X_test_analysis[f'worst_case_{col}'] = values
```

    100%|██████████| 38/38 [10:34<00:00, 16.69s/it]

```python
X_test_analysis.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)
      L-metric
      worst_case_sepal length (cm)
      worst_case_sepal width (cm)
      worst_case_petal length (cm)
      worst_case_petal width (cm)

      71
      6.1
      2.8
      4.0
      1.3
      15.265199
      6.108548
      2.823778
      3.985840
      1.299841

      95
      5.7
      3.0
      4.2
      1.2
      16.965591
      5.718465
      3.013931
      4.165211
      1.205684

      32
      5.2
      4.1
      1.5
      0.1
      18.590519
      5.199954
      4.095338
      1.500680
      0.104436

      116
      6.5
      3.0
      5.5
      1.8
      5.310058
      6.507278
      2.999676
      5.503435
      1.804076

      14
      5.8
      4.0
      1.2
      0.2
      20.741562
      5.808687
      3.997569
      1.196534
      0.200304

```python
X_test_analysis['L-metric'].sort_values(ascending=False).plot.bar(figsize=(15,5), title='L-metric for explanations from X_test');
```

```python
# Look closer at the point with the highest L-metric
worst_idx = X_test_analysis['L-metric'].argmax()
x0 = X_test_analysis.iloc[worst_idx, :4].values
x_perturbed = X_test_analysis.iloc[worst_idx, -4:].values

# Plot feature contribution for x0
lime_explainer.explain_instance(x0, model.predict_proba).as_pyplot_figure();
plt.title('Explanations for original input')
print(f"Input:        {x0}")

# Plot feature contribution for perturbation
lime_explainer.explain_instance(x0, model.predict_proba).as_pyplot_figure();
plt.title('Explanations for small perturbation from input')
print(f"Perturbation: {x_perturbed}")
```

As seen above, the test set containts multiple points where the LIME explanations are unstable. When looking closer at the most unstable point, we see that the feature contributions change dramatically, even though the input has barely changed.

## Educational example
Implement the metric describing the robustness of an explanation model in terms of its constant L in the definition of “local Lipschitz continuity”. For the example, implement using a simple predict method.

**Definition 2.1 (from [2]):**

$$
f \subseteq \mathbb{R}^n \rightarrow \mathbb{R}^m \textrm{is \textbf{Locally Lipschitz} if for every } x_0 \textrm{ there exists } \delta > 0 \textrm{ and } L \in \mathbb{R} \textrm{ such that } \|x - x_0\| < \delta \textrm{ implies } \|f(x) - f(x_0)\| < L\|x - x_0\|
$$

In essence, this formula says that for all perturbations $x$ around a given point $x_0$, a function is Locally Lipschitz if there exists some number $L$ that ensures that the difference in output $\|f(x) - f(x_0)\|$ is less than $L$ multiplied by the difference in input $\|x - x_0\|$.

The paper argues that if we find the minimum value for $L$ that satisfies this condition, we can use that as a measure for stability.

Originally, the $L$ was meant to be calculated analytically. But since our ML-model cannot be expressed as an end-to-end differentiable function, the authors of [2] suggests using the following formula for approximating $L$:

$$
\tilde{L}_X(x_i) = \underset{x_j \in \mathcal{N}_\epsilon(x_i) \le \epsilon}{\mathrm{argmax}} \frac{\|f(x_i)-f(x_j)\|_2}{\|x_i-x_j\|_2}
$$
Where:
$$
\mathcal{N}_\epsilon(x_i) = \{ x_j \in X | \|x_i-x_j\| \le \epsilon \}
$$

The two formulas above basically mean that the stability metric $\tilde{L}_X(x_i)$ can be found by finding the largest deviation between the difference in both input and output, where the input should be based on a perturbations $x_j$ from a finite-sampled neighbourhood around $x_0$, such that the size of the perturbations are less than some small, "human-decided" number $\epsilon$

### An example

Testing the robustness score
Out function consists of two mathematical functions:

$$
\begin{align}
f_1(x) &= 3x_1 \\
f_2(x) &= 5x_2^2 + x_3
\end{align}
$$

The derivatives of these functions give us
$$
\begin{align}
\nabla f_1(x) &= \begin{bmatrix}
           3 \\
           0 \\
           0
         \end{bmatrix} \\

\nabla f_2(x) &= \begin{bmatrix}
           0 \\
           10x_2 \\
           1
         \end{bmatrix} \\
\end{align}
$$

Let's test two points, $(0,0,0)$ and $(0,1,0)$.
The point $(0,0,0)$ gives us the following derivatives:
$$
\begin{align}
\nabla f_1(0,0,0) &= \begin{bmatrix}
           3 \\
           0 \\
           0
         \end{bmatrix} \\

\nabla f_2(0,0,0) &= \begin{bmatrix}
           0 \\
           0 \\
           1
         \end{bmatrix} \\
\end{align}
$$

While the point $(0,1,0)$ gives us the following derivatives:
$$
\begin{align}
\nabla f_1(0,1,0) &= \begin{bmatrix}
           3 \\
           0 \\
           0
         \end{bmatrix} \\

\nabla f_2(0,1,0) &= \begin{bmatrix}
           0 \\
           10 \\
           1
         \end{bmatrix} \\
\end{align}
$$

The Lipschitz robustness score tries to find the largest derivative in any dimention for a given point.
From an analytical sense, this corresponds to `max(max(f1'(x*)), max(f2'(x*)))` for a given point $x^*$.
We should therefor expect the following results, when testing on our function, as $N \rightarrow \infty$:
```
lipschitz_robustness_score([0,0,0]) = 3
lipschitz_robustness_score([0,1,0]) = 10
```

```python
predict = lambda x: np.array([3*x[:,0],
                              5*x[:,1]**2 + x[:,2]]).T
```

## Implementations

### Test using random samples

```python
input_space = input_space = [(0, 100), (0, 3.14), (-5, 5)]
num_perturbations=100_000

x0s = np.array([[0,0,0], [0,1,0]])

for x0 in x0s:
    x0 = x0.reshape(1,-1)

    x0, y0, x_perturbations, y_perturbations = get_perturbations(predict, x0, input_space, num_perturbations=num_perturbations)
    L, L_input = lipschitz_metric(x0, y0, x_perturbations, y_perturbations)

    print(f"For x0={x0}, the robustness metric is {L:.2f} after {num_perturbations} perturbations, where the biggest gradient was found going from x0 to {L_input}")
```

    For x0=[[0 0 0]], the robustness metric is 3.00 after 100000 perturbations, where the biggest gradient was found going from x0 to [ 7.05788851e-01  5.41933926e-05 -6.37760093e-05]
    For x0=[[0 1 0]], the robustness metric is 10.02 after 100000 perturbations, where the biggest gradient was found going from x0 to [ 4.41993306e-04  1.01945420e+00 -1.08918038e-03]

### Test using pre-sampled perturbations
If the `predict()`-method is expensive to run, narrowing down the sample-space in a smart way could significantly reduce the execution time. Rather than sampling the input_space randomly, One could either create a grid-search or design a more deterministic approach based on knowledge about the data that only applied perturbations with meaningfull and desirable directions and magnitudes.

```python
input_perturbations = np.array([
    [  0.1,     0,     0],
    [ -0.1,     0,     0],
    [    0,   0.1,     0],
    [    0,  -0.1,     0],
    [    0,     0,   0.1],
    [    0,     0,  -0.1],
    [ 0.01,  0.01,  0.01],
    [-0.01, -0.01, -0.01],
])

x0 = np.array([[0,0,0]])

x0, y0, x_perturbations, y_perturbations = get_perturbations(predict, x0, input_perturbations=input_perturbations)
L, L_input = lipschitz_metric(x0, y0, x_perturbations, y_perturbations)

print(f"For x0={x0}, the robustness metric is {L:.2f}, where the biggest gradient was found going from x0 to {L_input}")
```

    For x0=[[0 0 0]], the robustness metric is 3.00, where the biggest gradient was found going from x0 to [0.1 0.  0. ]