# Doyle-Fuller-Newman Model (DFN)

## Dimensionless Model Equations

The DFN comprises equations for charge and mass conservation in the solid the solid and electrolyte, and also prescribes behaviour for the electrochemical reactions occurring on the interface between the solid an electrolyte. For more information please see [1] or other standard texts.

Below we summarise the dimensionless form of the DFN, with all parameters give in the table at the end of this notebook. Here we use a roman subscript $\text{k} \in \text{n, s, p}$ is used to denote the regions negative electrode, separator, and positive electrode, respectively.

The model equations for the DFN read:

#### Charge conservation:
\begin{align}
    \frac{\partial i_{\text{e,k}}}{\partial x} &= \begin{cases}
                                                      j_{\text{k}}, \quad &\text{k} = \text{n, p}\\
                                                      0, \qquad &\text{k} = \text{s}
                                                \end{cases} , && \\
    \mathcal{C}_{\text{e}} i_{\text{e,k}} &= \epsilon_{\text{k}}^{\text{b}} \gamma_{\text{e}} \kappa_{\text{e}}(c_{\text{e,k}}) \left( - \frac{\partial \phi_{\text{e,k}}}{\partial x} + 2(1-t^+)\frac{\partial}{\partial x}\left(\log(c_{\text{e,k}})\right)\right), && \text{k} \in \text{n, s, p}, \\
    I-i_{\text{e,k}} &= - \sigma_{\text{k}} \frac{\partial \phi_{\text{e,k}}}{\partial x}, && \text{k} \in \text{n, s, p}.
\end{align}

#### Mass conservation:
\begin{align}
\mathcal{C}_{\text{e}} \epsilon_{\text{k}} \gamma_{\text{e}} \frac{\partial c_{\text{e,k}}}{\partial t} &= -\gamma_{\text{e}} \frac{\partial N_{\text{e,k}}}{\partial x} + \mathcal{C}_{\text{e}} \frac{\partial i_{\text{e,k}}}{\partial x}, && \text{k} \in \text{n, s, p},\\
N_{\text{e,k}} &= -\epsilon_{\text{k}}^{\text{b}} D_{\text{e}}(c_{\text{e,k}}) \frac{\partial c_{\text{e,k}}}{\partial x} +  \frac{\mathcal{C}_{\text{e}} t^+}{\gamma_{\text{e}}} i_{\text{e,k}},
 && \text{k} \in \text{n, s, p}, \\
\mathcal{C}_{\text{k}} \frac{\partial c_{\text{s,k}}}{\partial t} &= -\frac{1}{r_{\text{k}}^2} \frac{\partial}{\partial r_{\text{k}}} \left(r_{\text{k}}^2 N_{\text{s,k}}\right), && \text{k} \in \text{n, p},\\
N_{\text{s,k}} &= -D_{\text{s,k}}(c_{\text{s,k}}) \frac{\partial c_{\text{s,k}}}{\partial r_{\text{k}}}, && \text{k} \in \text{n, p}.
\end{align}

#### Electrochemical reactions:
\begin{align}
j_{\text{k}} &= 2 j_{\text{0,k}} \sinh\left(\frac{ \eta_{\text{k}}}{2} \right), && \text{k} \in \text{n, p}, \\
j_{\text{0,k}} &= \frac{\gamma_{\text{k}}}{\mathcal{C}_{\text{r,k}}} c_{\text{s,k}}^{1/2} (1-c_{\text{s,k}})^{1/2}c_{\text{e,k}}^{1/2}\big|_{r_{\text{k}}=1}, && \text{k} \in \text{n, p}, \\
\eta_{\text{k}} &= \phi_{\text{s,k}} - \phi_{\text{e,k}} - U_{\text{k}}(c_{\text{s,k}}\big|_{r_{\text{k}}=1}),  && \text{k} \in \text{n, p}.
\end{align}

These are to be solved subject to the following boundary conditions:

#### Current:
\begin{gather}
i_{\text{e,n}}\big|_{x=0} = 0, \quad i_{\text{e,p}}\big|_{x=1}=0,  \\
\phi_{\text{e,n}}\big|_{x=L_{\text{n}}} = \phi_{\text{e,s}}\big|_{x=L_{\text{n}}}, \quad i_{\text{e,n}}\big|_{x=L_{\text{n}}} = i_{\text{e,s}}\big\vert_{x=L_{\text{n}}} = I, \\
\phi_{\text{e,s}}\big|_{x=1-L_{\text{p}}} = \phi_{\text{e,p}}\big|_{x=1-L_{\text{p}}}, \quad
    i_{\text{e,s}}\big|_{x=1-L_{\text{p}}} = i_{\text{e,p}}\big|_{x=1-L_{\text{p}}} = I.
\end{gather}

#### Concentration in the electrolyte:
\begin{gather}
N_{\text{e,n}}\big|_{x=0} = 0, \quad N_{\text{e,p}}\big|_{x=1}=0,\\
c_{\text{e,n}}\big|_{x=L_{\text{n}}} = c_{\text{e,s}}|_{x=L_{\text{n}}}, \quad N_{\text{e,n}}\big|_{x=L_{\text{n}}}=N_{\text{e,s}}\big|_{x=L_{\text{n}}}, \\
c_{\text{e,s}}|_{x=1-L_{\text{p}}}=c_{\text{e,p}}|_{x=1-L_{\text{p}}}, \quad N_{\text{e,s}}\big|_{x=1-L_{\text{p}}}=N_{\text{e,p}}\big|_{x=1-L_{\text{p}}}. &&
\end{gather}

####  Concentration in the electrode active material:
\begin{gather}
N_{\text{s,k}}\big|_{r_{\text{k}}=0} = 0, \quad \text{k} \in \text{n, p}, \quad \ \ - \frac{a_{\text{k}}\gamma_{\text{k}}}{\mathcal{C}_{\text{k}}} N_{\text{s,k}}\big|_{r_{\text{k}}=1} = j_{\text{k}}, \quad \text{k} \in \text{n, p}.
\end{gather}

#### Reference potential:
$$\phi_{\text{s,cn}} = 0, \quad  \boldsymbol{x} \in \partial \Omega_{\text{tab,n}}.$$

And the initial conditions:

\begin{align}
&c_{\text{s,k}}(x,r,0) = c_{\text{s,k,0}}, \quad \phi_{\text{s,n}}(x,0) = 0, \quad \phi_{\text{s,p}}(x,0) = \phi_{\text{s,p,0}}, && \text{k} \in \text{n, p},\\
&\phi_{\text{e,k}}(x,0) = \phi_{\text{e,0}}, \quad c_{\text{e,k}}(x,0) = 1, && \text{k} \in \text{n, s, p}.
\end{align}

## Example solving DFN using PyBaMM

Below we show how to solve the DFN model, using the default geometry, mesh, parameters, discretisation and solver provided with PyBaMM. For a more detailed example, see the notebook on the [SPM](https://github.com/pybamm-team/PyBaMM/blob/master/examples/notebooks/models/SPM.ipynb).

First we need to import pybamm, and then change our working directory to the root of the pybamm folder.

```python
%pip install pybamm -q    # install PyBaMM if it is not installed
import pybamm
import numpy as np
import os
import matplotlib.pyplot as plt
os.chdir(pybamm.__path__[0]+'/..')
```

We then get the DFN model equations, and process the model using default settings for the geometry and parameters.

```python
# load model
model = pybamm.lithium_ion.DFN()

# create geometry
geometry = model.default_geometry

# load parameter values and process model and geometry
param = model.default_parameter_values
param.process_model(model)
param.process_geometry(geometry)
```

The next step is to set the mesh and discretise the model. Again, we choose the default settings.

```python
# set mesh
mesh = pybamm.Mesh(geometry, model.default_submesh_types, model.default_var_pts)

# discretise model
disc = pybamm.Discretisation(mesh, model.default_spatial_methods)
disc.process_model(model);
```

The model is now ready to be solved. We select the default DAE solver for the DFN. Note that in order to successfully solve the system of DAEs we are required to give consistent initial conditions. This is handled automatically by PyBaMM during the solve operation.

```python
# solve model
solver = model.default_solver
t_eval = np.linspace(0, 3600, 300) # time in seconds
solution = solver.solve(model, t_eval)

```

To get a quick overview of the model outputs we can use the QuickPlot class, which plots a common set of useful outputs. The method `Quickplot.dynamic_plot` makes a slider widget.

```python
quick_plot = pybamm.QuickPlot(solution)
quick_plot.dynamic_plot();
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=3599.9999999999995, step=35.99999999999999),…

## Dimensionless Parameters

In the table below, we provide the dimensionless parameters in the DFN in terms of the dimensional parameters in mcmb2528_lif6-in-ecdmc_lico2_parameters_Dualfoil.csv. We use a superscript * to indicate dimensional quantities.

| Parameter                 | Expression                              |Interpretation                             |
|:--------------------------|:----------------------------------------|:------------------------------------------|
| $L_{\text{k}}$            | $L_{\text{k}}^*/L^*$                    | Ratio of region thickness to cell thickness|
|$\sigma_{\text{k}}$        | $\sigma_{\text{k}}^*R^* T^*/(I^*F^*L^*)$| Dimensionless solid conductivity |
|$\mathcal{C}_{\text{k}}$   | $\tau_{\text{k}}^*/\tau_{\text{d}}^*$   | Ratio of solid diffusion and discharge timescales |
|$\mathcal{C}_{\text{e}}$   |$\tau_{\text{e}}^*/\tau_{\text{d}}^*$    |Ratio of electrolyte transport and discharge timescales|
|$\mathcal{C}_{\text{r,k}}$ |$\tau_{\text{r,k}}^*/\tau_{\text{d}}^*$  |Ratio of reaction and discharge timescales|
|$a_{\text{k}}$             |$a_{\text{k}}^* R_{\text{k}}^*$          | Product of particle radius and surface area to volume ratio|
|$\gamma_{\text{k}}$        |$c_{\text{k,max}}^*/c_{\text{n,max}}^*$  |Ratio of maximum lithium concentrations in solid|
|$\gamma_{\text{e}}$        |$c_{\text{e,typ}}^*/c_{\text{n,max}}^*$  |Ratio of maximum lithium concentration in the negative electrode solid and typical electrolyte concentration|

Note that the dimensionless parameters $\epsilon_{\text{k}}$, $\text{b}$, and $t^+$ are already provided in the parameter file mcmb2528_lif6-in-ecdmc_lico2_parameters_Dualfoil.csv

## References
[1] M. Doyle,  T.F. Fuller,  and J. Newman.  Modeling of galvanostatic charge and discharge of thelithium/polymer/insertion cell.Journal of the Electrochemical society, 140(6):1526–1533, 1993

```python

```

```python

```


# General Classification Explanations

**Example.** In this example, we ask how the features which distinguish Versicolour from Setosa irises compare to those which distinguish Versicolour from Virginica irises. (This is analogous to asking how the symptoms which distinguish COVID-19 from the flu compare to those which distinguish COVID-19 from the common cold; however, a sufficient COVID-19 dataset is not publicly available.) We find that petal length is the most important feature. Additionally, petal length is relatively less important, and sepal length relatively more important, for distinguishing Versicolour from Setosa (as opposed to distinguishing Versicolour from Virginia).

**General formulation.** More generally, let $f_c(x_i)$ be the probability that observation $x_i$ belongs to class $c\in C$ according to model $f$. Let $C_+\subset C$ be a set of 'positive classes' and $C_-\subset C$ be a set of 'negative classes'. For a single observation $x_i$, we can ask why our model predicts it comes from a positive class rather than a negative class. For a randomly selected set of observations $X$ from positive classes, we can ask which features best distinguish observations which belong to positive classes from observations which belong to negative classes. Here, the positive class is Versicolour, and the negative class is Setosa or Virginica.

```python
import warnings
warnings.simplefilter('ignore')

import gshap
from gshap.probability_distance import ProbabilityDistance

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

import os
import random

np.random.seed(0)
random.seed(0)

# directory for saving figures
# set to `None` to avoid saving
PLOT_DIR = 'plots'
```

**Model.** We train Scikit-learn's support vector classifier with default settings on the Scikit-learn iris dataset ($n=150$), using a 75%-25% train-test split. The purpose of this dataset is to predict an iris's species (Versicolour, Setosa, or Virginica) based on its petal length, petal width, sepal length, and sepal width.

```python
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target)
clf = SVC(probability=True)
clf.fit(X_train, y_train)
print('Test score: % .4f' % clf.score(X_test, y_test))
```

    Test score:  0.9737

**Sample.** To discover which features distinguish Versicolour from Setosa or Virginica, our set of observations $X$ is a random sample of test observations classified as Versicolour. Note that our analysis will not work if $X$ contains a mixture of iris species. Intuitively, we are asking what features lead our model to believe that $X$ contains only Versicolour irises. If $X$ contains a mixture of iris species, the G-SHAP values will be 0 for all features, indicating that our model does not believe $X$ contains only Versicolour irises.

Sepal length and petal length are of most interest to our analysis. To visualize where our observations fall along these dimensions, we create a scatterplot. The selected observations are highlighted in red.

```python
# select Versicolour (class 1)
X = X_test[clf.predict(X_test) == 1]
X = X.reshape((1,-1)) if len(X.shape) == 1 else X

# plot sepal length (feature 0) against petal length (feature 2)
palette = ['blue', 'green', 'purple']
for i in range(3):
    species = iris.data[iris.target==i]
    plt.scatter(species[:,0], species[:,2], c=palette[i], label=iris.target_names[i])
plt.scatter(X[:,0], X[:,2], c='red', label='sample versicolour')
plt.xlabel(iris['feature_names'][0])
plt.ylabel(iris['feature_names'][2])
plt.legend()
if PLOT_DIR is not None:
    plt.savefig(os.path.join(PLOT_DIR, 'iris_scatter.png'))
plt.show()
```

**G-SHAP values.** We compute the G-SHAP values, where $g$ measures the probability that every observation in set $X$ belongs to a positive class given that every observation belongs to a positive class or every observation belongs to a negative class.

\begin{equation}
g(f,X,\Omega) = \frac{\prod_i \sum_{c\in C_+} f_c(x_i)}{\prod_i \sum_{c\in C_+} f_c(x_i) + \prod_i \sum_{c\in C_-} f_c(x_i)}
\end{equation}

Where the additional arguments to $g$ are the sets of positive and negative classes; $\Omega=\{C_+,C_-\}$.

Note that traditional SHAP values are a specific instance of $g(f,X,\Omega)$ in which set $X$ contains a single observation $x_i$, the set of positive classes is the model's classification of $x_i$; $C_+=\{arg\,max_{c\in C} f_c(x_i)\}$, and the negative classes are the complement of the positive classes $C_-=C\setminus C_+$. Also note that, while our generalized function $g(f,X,\Omega)$ uses only two subsets of classes, $C_+$ and $C_-$, it easily extends to accommodate any number of subsets.

In our example, $g$ measures the probability that every iris in this sample is a Versicolour, given that every iris in this sample is a Versicolour or every iris in this sample is a Setosa. Specifically, the set of positive classes $C_+$ is $\{\text{Versicolour}\}$, and the set of negative classes $C_-$ is $\{\text{Setosa}\}$.

\begin{equation}
g(f,X,\Omega)
= \frac{\prod_i f_{\text{Versicolour}}(x_i)}
{\prod_i f_{\text{Versicolour}}(x_i) + \prod_i f_{\text{Setosa}}(x_i)}
\end{equation}

```python
# Versicolour (1) versus Setosa (0)
pos_distribution = lambda y_pred: y_pred[:,1]
neg_distribution = lambda y_pred: y_pred[:,0]
g = ProbabilityDistance(pos_distribution, neg_distribution)
explainer_10 = gshap.KernelExplainer(clf.predict_proba, X_train, g)
gshap_values_10 = explainer_10.gshap_values(X, nsamples=10000)
```

Similarly, to discover which features distinguish Versicolour from Virginica, $C_-$ is $\{\text{Virginica}\}$. As before, $C_+=\{\text{Versicolour}\}$.

```python
# Versicolour (1) versus Virginica (2)
neg_distribution = lambda y_pred: y_pred[:,2]
g = ProbabilityDistance(pos_distribution, neg_distribution)
explainer_12 = gshap.KernelExplainer(clf.predict_proba, X_train, g)
gshap_values_12 = explainer_12.gshap_values(X, nsamples=10000)
```

**Interpretation.** For interpretation, we compare our Versicolour sample $X$ to a background dataset $Z$ (the shuffled training data) in terms of $g$. For the Versicolour-Setosa comparison, our support vector classifier correctly predicts a 100\% chance that our sample contains only Versicolour, as opposed to only Setosa. It also predicts a 65% chance that our background dataset contains only Versicolour, as opposed to only Setosa. Each G-SHAP value $\phi_g^j$ is the number of percentage points of this difference (35 percentage points) explained by feature $j$.

For general classification explanations more broadly, our model predicts that each observation in our sample belongs to a positive class, rather than a negative class, with probability $Pr_X$. It also predicts that each observation in the background dataset comes from a positive class, rather than a negative class, with probability $Pr_Z$. Each G-SHAP value $\phi_g^j$ is the number of percentage points of this difference $Pr_X-Pr_Z$ explained by feature $j$.

```python
g_comparison, g_background = explainer_10.compare(X, bootstrap_samples=10000)
print('Probability that our sample belongs to a positive class: %.4f' % g_comparison)
print('Probability that the background data belongs to a positive class: %.4f' % g_background)
print('Sum of G-SHAP values: %.4f' % gshap_values_10.sum())
```

    Probability that our sample belongs to a positive class: 1.0000
    Probability that the background data belongs to a positive class: 0.6296
    Sum of G-SHAP values: 0.3695

We interpret the G-SHAP values for the Versicolour-Virginica comparison similarly.

```python
g_comparison, g_background = explainer_12.compare(X, bootstrap_samples=10000)
print('Probability that our sample belongs to a positive class: %.4f' % g_comparison)
print('Probability that the background data belongs to a positive class: %.4f' % g_background)
print('Sum of G-SHAP values: %.4f' % gshap_values_12.sum())
```

    Probability that our sample belongs to a positive class: 1.0000
    Probability that the background data belongs to a positive class: 0.3990
    Sum of G-SHAP values: 0.6024

**Results.** We plot our results, normalizing the G-SHAP values to sum to 1 for both comparisons (Versicolour-Setosa and Versicolour-Virginica). We find that petal length is the most important feature. Additionally, petal length is relatively less important, and sepal length relatively more important, for distinguishing Versicolour from Setosa (as opposed to distinguishing Versicolour from Virginia).

```python
df = pd.DataFrame({
    'Features': iris['feature_names']*2,
    'G-SHAP': list(gshap_values_10/gshap_values_10.sum()) + list(gshap_values_12/gshap_values_12.sum()),
    'Comparison': [iris['target_names'][0]]*4 + [iris['target_names'][2]]*4
})
sns.catplot(
    x='G-SHAP',
    y='Features',
    hue='Comparison',
    data=df,
    kind='bar',
    orient='h'
)
if PLOT_DIR is not None:
    plt.savefig(os.path.join(PLOT_DIR, 'iris_gshap.png'))
plt.show()
```

```python
df
```

      Features
      G-SHAP
      Comparison

      0
      sepal length (cm)
      0.059266
      setosa

      1
      sepal width (cm)
      0.041194
      setosa

      2
      petal length (cm)
      0.502755
      setosa

      3
      petal width (cm)
      0.396785
      setosa

      4
      sepal length (cm)
      -0.008588
      virginica

      5
      sepal width (cm)
      0.007735
      virginica

      6
      petal length (cm)
      0.637756
      virginica

      7
      petal width (cm)
      0.363097
      virginica


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Definition

The [Laplace transform](https://en.wikipedia.org/wiki/Laplace_transform) constitutes an integral transform with the complex exponential function $e^{s t}$ as kernel. It maps a complex-valued signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ into its complex-valued Laplace transform $X(s) \in \mathbb{C}$ with complex-valued independent variable $s \in \mathbb{C}$. The Laplace transform is named after its discoverer [Pierre-Simon Laplace](https://en.wikipedia.org/wiki/Pierre-Simon_Laplace).

Complex exponential functions constitute eigenfunctions of linear time-invariant (LTI) systems. The Laplace transform is therefore of special use in the context of signals and LTI systems. It is introduced in the following.

### Two-Sided Laplace Transform

The two-sided (or bilateral) Laplace transform is defined as

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ denotes the Laplace transform of $x(t)$. A complex signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ is represented by its complex valued Laplace transform $X(s) \in \mathbb{C}$ with complex independent variable $s \in \mathbb{C}$. The variable $s$ can be interpreted as [complex frequency](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $s = \sigma + j \omega$ with $\sigma, \omega \in \mathbb{R}$.

Whether a Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ exists depends on the complex frequency $s$ and the signal $x(t)$ itself. All values $s$ for which the Laplace transform converges form a [*region of convergence*](https://en.wikipedia.org/wiki/Radius_of_convergence) (ROC). The Laplace transforms of two different signals may differ only with respect to their ROCs. Consequently, the ROC needs to be explicitly given for an unique inversion of the Laplace transform.

### One-Sided Laplace Transform

Causal signals play an important role in the theory of signals and system. For a causal signal $x(t) = 0$ for $t <0$, the relation $x(t) = x(t) \cdot \epsilon(t)$ holds. Introducing this into the definition of the two-sided Laplace transformation yields

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \cdot \epsilon(t) \, e^{- s t} \; dt = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

This motivates the definition of a one-sided Laplace transform

\begin{equation}
X(s) = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

In the literature both the one- and two-sided Laplace transform are termed as Laplace transform. Both give the same result for causal signals.

### Transformation of the Dirac Impulse

The transform $\mathcal{L} \{ \delta(t) \}$ of the [Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) is derived by introducing $\delta(t)$ into the definition of the two-sided Laplace transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{L} \{ \delta(t) \} = \int_{-\infty}^{\infty} \delta(t) \, e^{- s t} dt = e^{- s t} \Big\vert_{t=0}  =  1
\end{equation}
for $s \in \mathbb{C}$. The ROC covers the entire complex plane.

The transform of the Dirac impulse is equal to one. Hence, all complex frequencies $s$ are present with equal weight. This is an important property in the theory of signals and systems, since the Dirac impulse is used to characterize linear time-invariant (LTI) systems by their [impulse response](../systems_time_domain/impulse_response.ipynb) $h(t) = \mathcal{H} \{ \delta(t) \}$.

**Example - Transformation of the Dirac impulse**

The Laplace transform of the Dirac impulse can be derived in `SymPy` by direct evaluation of above integral

```python
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

X = sym.integrate(sym.DiracDelta(t)*sym.exp(-s*t), (t, -sym.oo, sym.oo))
X
```

**Exercise**

* Derive the two-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$ either by manual evaluation of the integral or by modification of above example.

* Derive the one-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$. Hint: Differentiate between the cases $\tau < 0$ and $\tau > 0$.

### Transformation of the Causal Complex Exponential Signal

The transform $X(s) = \mathcal{L} \{ x(t) \}$ of the causal complex exponential signal

\begin{equation}
x(t) = \epsilon(t) \cdot e^{-s_0 t}
\end{equation}

with complex frequency $s_0 \in \mathbb{C}$ is derived by evaluation of the definition of the one-sided Laplace transform

\begin{equation}
\begin{split}
X(s) &= \int_{0}^{\infty} e^{-s_0 t} \cdot e^{-s t} \; dt = \frac{-e^{-(s+s_0)t}}{s+s_0} \, \bigg\vert_{0}^{\infty} = \frac{-1}{s+s_0} \left[ \lim_{t \to \infty} \left( e^{- (s+s_0) t} \right) -1 \right] \\
&= \frac{1}{s + s_0}
\end{split}
\end{equation}

As can be concluded from the limit case above, the Laplace transform converges only for $\Re \{ s + s_0 \} > 0$. Hence, the ROC is given as

\begin{equation}
\Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

Combining above findings, the transformation of the causal complex exponential signal reads

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

**Example - Transformation of exponential signal**

In `SymPy` the [Laplace transform](http://docs.sympy.org/latest/modules/integrals/integrals.html?highlight=laplace#sympy.integrals.transforms.laplace_transform) is defined as one-sided transformation. The respective function of `SymPy` is utilized in the following to calculate the transform of the causal complex exponential signal.

```python
s0 = sym.symbols('s0', complex=True)

X, a, cond = sym.laplace_transform(sym.exp(-s0*t), t, s)
X, a, cond
```

The function returns three results. The first result is the Laplace transform $X(s)$, the second result $a$ is the half-plane of convergence with $\Re \{ s \} > a$ and the third result represents an auxiliary convergence condition.

### Transformation of the Heaviside Signal

The Laplace transform of the [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) is derived from above transform of the causal complex exponential signal $\epsilon(t) \cdot e^{-s_0 t}$ by setting $s_0 = 0$

\begin{equation}
\mathcal{L} \{ \epsilon(t) \} = \frac{1}{s} \qquad \text{for } \Re \{ s \} > 0
\end{equation}

## Convergence

A sufficient but not necessary condition for the existence of the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of a given signal $x(t)$ is

\begin{equation}
\int_{-\infty}^{\infty} | x(t) \cdot e^{- \sigma t} | \; dt < \infty
\end{equation}

where $\sigma = \Re\{ s \}$ denotes the real part of the complex frequency $s = \sigma + j \omega$ of the Laplace transform. Formally the signal $x(t)$ has to be a [locally integrable function](https://en.wikipedia.org/wiki/Locally_integrable_function).

The imaginary part $\omega = \Im \{ s \}$ of the complex frequency $s$ has no effect in terms of convergence. Consequently, the ROC is determined solely by its real part $\sigma = \Re\{ s \}$. For a right-sided signal with $x(t) = 0$ for $t < T_0$, the ROC is given in the form of $\Re \{ s \} > a$ with $a \in \mathbb{R}$. This can be concluded from the decay of the exponential term $e^{- \sigma t}$ for $t \to \infty$, which ensures convergence for a given $a$. However this reasoning holds only for signals with [exponential growth](https://en.wikipedia.org/wiki/Exponential_growth). Please refer to the [Laplace transform of the causal exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) for an example.

The same reasoning leads to the ROCs for a left-sided and two-sided signal. The resulting ROCs are shown in the following illustration

The gray areas denote the values $s$ for which the Laplace transformation converges. The borders $a$ of these areas (dashed lines) depend on the signal $x(t)$. A more detailed discussion of the ROCs for the Laplace transform can be found in the literature, e.g. [[Girod et al.](index.ipynb#Literature)]

## Representation

The Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ depends on the complex frequency $s \in \mathbb{C}$ and is in general complex valued $X(s) \in \mathbb{C}$. It can be illustrated by plotting its magnitude $|X(s)|$ and phase $\varphi (s)$ or real $\Re \{ X(s) \}$ and imaginary $\Im \{ X(s) \}$ part in the complex $s$-plane. The resulting three-dimensional plots are often not very illustrative.

However, many Laplace transforms of interest in the theory of signals and systems are [rational functions](https://en.wikipedia.org/wiki/Rational_function) in $s$. The polynomials of the numerator and denominator can be represented by their complex [roots](https://en.wikipedia.org/wiki/Zero_of_a_function#Polynomial_roots) and a constant factor. The roots of the numerator are termed as *zeros* while the roots of the denominator are termed as *poles* of $X(s)$. The polynomial and the zero/pole representation of a rational Laplace transform are equivalent

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n} = K \cdot \frac{\prod_{\mu=0}^{Q} (s - s_{0 \mu})}{\prod_{\nu=0}^{P} (s - s_{\infty \nu})}
\end{equation}

where $M$ and $N$ denote the order of the numerator/denominator polynomial, $s_{0 \mu}$ and $s_{\infty \nu}$ the $\mu$-th zero/$\nu$-th pole of $X(s)$, and $Q = M-1$ and $P = N-1$ the total number of zeros and poles, respectively. For $M=N$ the factor $K = \frac{\beta_M}{\alpha_N}$. If $M > N$ at least one pole is located at $|s| = \infty$.

It is common to illustrate the poles and zeros in a [pole-zero plot](https://en.wikipedia.org/wiki/Pole%E2%80%93zero_plot). Here the locations of the complex poles and zeros, their degrees and the factor $K$ are plotted in the $s$-plane

The locations of the poles and zeros provide insights into the composition of a signal. For instance, the Laplace transform of the [complex exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) has a pole at $s = - s_0$. A signal which is composed from a superposition of complex exponential signals will have multiple poles whose positions are related to the complex frequencies it is composed of.

**Exercise**

* Derive $X(s)$ together with its ROC from above pole-zero plot.

**Example - Illustration of Laplace transform of the exponential signal**

The Laplace transform of the causal complex exponential function

\begin{equation}
X(s) = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

is illustrated by plotting its magnitude $|X(s)|$ in the $s$-plane $s = \sigma + j \omega$ for $s_0 = 1 + j$

```python
sigma, omega = sym.symbols('sigma omega')

X = 1/(s + s0)
X1 = X.subs({s: sigma+sym.I*omega, s0: 1+sym.I})

sym.plotting.plot3d(abs(X1), (sigma, -2, 2), (omega, -2, 2),
                    xlabel=r'$\Re\{s\}$', ylabel=r'$\Im\{s\}$', title=r'$|X(s)|$');
```

Alternatively, the Laplace transform is illustrated by its pole-zero plot. First the poles

```python
poles = sym.roots(sym.denom(X), s)
poles
```

and zeros are computed

```python
zeros = sym.roots(sym.numer(X), s)
zeros
```

Above dictionaries are composed of the poles and zeros, and their degrees.

In order to illustrate the location of poles and zeros in the $s$-plane, a function is defined for a pole-zero plot using the Python library [`matplotlib`](http://matplotlib.org/)

```python
import matplotlib.pyplot as plt

def pole_zero_plot(poles, zeros, ax=None):

    for pole in poles:
        plt.plot(complex(pole).real, complex(pole).imag, 'rx', markersize=10)
    for zero in zeros:
        plt.plot(complex(zero).real, complex(zero).imag, 'bo', markersize=10)

    if ax is None:
        ax = plt.gca()
    ax.set_aspect(1)

    plt.axis([-2, 2, -2, 2])
    plt.xlabel(r'$\Re \{ s \}$')
    plt.ylabel(r'$\Im \{ s \}$')
    plt.grid()
```

The pole-zero plot is shown for $s_0 = 1 + j$

```python
X2 = X.subs(s0, 1+sym.I)
pole_zero_plot(sym.roots(sym.denom(X2), s), sym.roots(sym.numer(X2), s))
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Expolinrg the equivalent damping equation $B_e$

```python
# %load ../../imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
#plt.style.use('paper')

#import data
import copy
from rolldecay.bis_system import BisSystem
from rolldecay import database
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sklearn.metrics import r2_score
import rolldecay.paper_writing as paper_writing

```

```python
from rolldecayestimators import measure as measure
```

## Is this a good linearization?

```python
equations.B_e_equation
```

```python
A_44 = 1.0
B_1=0.10
B_2=0.8
C_1=2.0

B_1A=B_1/A_44
B_2A=B_2/A_44
C_1A=C_1/A_44

estimator = EstimatorQuadraticB.load(B_1A=B_1A, B_2A=B_2A, C_1A=C_1A)
```

```python
N=100000
t=np.linspace(0,100.0,N)
phi0=np.deg2rad(10)
phi1d0=0.0

result = estimator.simulate(t=t, phi0=phi0, phi1d0=phi1d0)
```

```python
fig,ax=plt.subplots()
result.plot(y='phi', ax=ax)
ax.grid(True)
```

```python
amplitudes = measure.calculate_amplitudes_and_damping(X=result)
amplitudes['phi_a']=amplitudes['phi']/2
amplitudes['-phi_a']=-amplitudes['phi_a']
amplitudes.head()
```

```python
equations.omega0_equation
```

```python
omega0_lambda = lambdify(sp.solve(equations.omega0_equation,symbols.omega0)[0])
```

```python
omega0=omega0_lambda(A_44=A_44, C=C_1)
omega0
```

```python
2*np.pi/omega0
```

```python
equations.extinction_equation
```

```python
equations.xeta_equation
```

```python
zeta_lambda = lambdify(sp.solve(equations.xeta_equation,symbols.zeta)[0])

phi0=amplitudes.iloc[0]['phi_a']
t_=amplitudes.index-amplitudes.index[0]
amplitudes['zeta']=zeta_lambda(omega0=omega0, phi_0=phi0, phi_a=amplitudes['phi_a'],
                               t=t_)
```

```python
amplitudes
```

```python
fig,ax=plt.subplots()
result.plot(y='phi', ax=ax)

amplitudes.plot( y='phi_a', ax=ax)
amplitudes.plot(y='-phi_a', ax=ax);

```

```python
zeta_lambda
```

```python
fig,ax=plt.subplots()
amplitudes.plot(x='phi_a', y=['B_n'], style='.-', ax=ax)

phi_a = amplitudes['phi_a']
y=B_1+(8/(3*np.pi)*omega0)*B_2*phi_a
ax.plot(phi_a,y,'.--', label='B_e')

ylim=ax.get_ylim()
ax.set_ylim([0,ylim[1]])
ax.legend()
```

```python
zero_crossings = measure.get_peaks(result)

mask = zero_crossings['phi'] < 0
negative_peaks = zero_crossings.loc[mask].copy()
positive_peaks = zero_crossings.loc[~mask].copy()

```

```python
fig,ax=plt.subplots()
result.plot(y='phi', ax=ax)
negative_peaks.plot(y='phi', style='r.', ax=ax)
positive_peaks.plot(y='phi', style='g.', ax=ax)
ax.grid(True)

```

```python
positive_peaks
```

```python
df_decrements_positive = positive_peaks.shift()/positive_peaks
df_decrements_negative = negative_peaks.shift()/negative_peaks
df_decrements = pd.concat([df_decrements_positive,df_decrements_negative])
df_decrements.sort_index(inplace=True)
```

```python
zero_crossings['B_n'] = 2*(1 / (2 * np.pi) * np.log(df_decrements['phi']))
```

```python
zero_crossings['phi_a']=zero_crossings['phi'].abs()*(1+df_decrements['phi'])/2
```

```python
zero_crossings
```

```python
fig,ax=plt.subplots()
zero_crossings.plot(x='phi_a', y=['B_n'], style='.-', ax=ax)

phi_a = zero_crossings['phi_a']
y=B_1+(8/(3*np.pi)*omega0)*B_2*phi_a
ax.plot(phi_a,y,'.--', label='B_e')

ylim=ax.get_ylim()
ax.set_ylim([0,ylim[1]])
ax.legend()
```

```python

```

```python

```

```python

```


##### Copyright 2020 Google LLC.

Licensed under the Apache License, Version 2.0 (the "License");

```
#@title Licensed under the Apache License, Version 2.0 (the "License"); { display-mode: "form" }
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Forward and Backward mode gradients in TFF

    Run in Google Colab

    View source on GitHub

```
#@title Upgrade to TensorFlow nightly
!pip install --upgrade tf-nightly
```

```
#@title Install TF Quant Finance
!pip install tff-nightly
```

### This notebook demonstrates the difference between forward and backward gradient computation

```
#@title Imports { display-mode: "form" }

import tensorflow as tf
import functools
import tf_quant_finance as tff
```

Consider a simple vector-function in two variables $x$ and $y$:

$
\begin{align}
& f = [f_1, f_2, f_3] \\
& where \\
\end{align}
$

$
\begin{align}
f_1 &= x^2 \\
f_2 &= y^2 \\
f_3 &= x y \\
\end{align}
$

```
def func(x):
    func = tf.stack([x[0]**2, x[1]**2, x[0] * x[1]])
    return func

start = tf.constant([1,2], dtype=tf.float64)

```

# Backward mode

For a vector $u = [u_1, u_2, u_3]$, backward gradient computes partial derivatives of the dot product $u \cdot f(x, y)$

$
\begin{align}
\frac {\partial (u \cdot f)}{\partial x} &= u_1 \frac{\partial f_1}{\partial x} + u_2 \frac{\partial f_2}{\partial x} + u_3 \frac{\partial f_3}{\partial x} \\ \\
&= 2 u_1 x + u_3 y\\ \\
\frac {\partial (u \cdot f)}{\partial y} &= u_1 \frac{\partial f_1}{\partial y} + u_2 \frac{\partial f_2}{\partial y} + u_3 \frac{\partial f_3}{\partial y} \\ \\
&= 2 u_1 y + u_3 x
\end{align}
$

In Tensorflow, [$u_1$, $u_2$, $u_3$] is by default set to [1, 1, 1].

Setting [$x$, $y$] to [1, 2], backward mode returns the gradients summed up by components

```
# Note that the output is u  d(u.f(x, y))dx and d(u.f(x, y))dy
tff.math.gradients(func, start)
```

    <tf.Tensor: shape=(2,), dtype=float64, numpy=array([4., 5.])>

The user has access to [$u_1$, $u_2$, $u_3$] as well. Setting the values
to [0, 0, 1] leads to the gradient $[\frac{\partial f_3}{\partial x}, \frac{\partial f_3}{\partial y}]$

```
tff.math.gradients(func, start,
                   output_gradients=tf.constant([0, 0, 1], dtype=tf.float64))
```

    <tf.Tensor: shape=(2,), dtype=float64, numpy=array([1., 1.])>

# Forward mode

TFF provides an opportunity to compute a forward gradient as well.
For a vector $w = [w_1, w_2]$, forward gradient computes differentials for $[f_1, f_2, f_3]$

$
\begin{align}
{\partial f_1} &= w_1 \frac{\partial f_1}{\partial x} + w_2 \frac{\partial f_1}{\partial y} \\ \\
&= 2 w_1 x \\ \\
{\partial f_2} &= w_1 \frac{\partial f_2}{\partial x} + w_2 \frac{\partial f_2}{\partial y} \\ \\
&= 2 w_2 y \\ \\
{\partial f_3} &= w_1 \frac{\partial f_3}{\partial x} + w_2 \frac{\partial f_3}{\partial y} \\ \\
&= w_1 x + w_2 y \\ \\
\end{align}
$

In TFF, [$w_1$, $w_2$] is by default set to [1, 1]. Setting [$x$, $y$] to [1, 2], forward mode returns the differentials by components.

```
tff.math.fwd_gradient(func, start)
```

    <tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 4., 3.])>

Remember, Tensorflow is the tool commonly used in Machine Learning. In Machine Learning, the aim is to minmize the scalar loss function, the loss function being the sum of the gradient with respect to the feature set. This lowest loss is the loss summed up over all training examples which can be computed via backward gradient.

However, let's take the use case where we are valuing a set of options, say ten, against a single spot price $S_0$. We now have ten price functions and we need their gradients against spot $S_0$ (ten deltas).

Using the forward gradients with respect to $S_0$ would give us the ten delta's in a single pass.

Using the backward gradients would result in the sum of the ten delta's, which may not be that useful.

It is useful to note that varying the weights would also give you individual components of the gradients (in other words [1, 0] and [0, 1] as values of [$w_1$, $w_2$], instead of the default [1, 1], similarly for backward. This is, of course, at the expense of more compute.

```
tff.math.fwd_gradient(func, start,
                      input_gradients=tf.constant([1.0, 0.0], dtype=tf.float64))
```

    <tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 0., 2.])>

```
tff.math.fwd_gradient(func, start,
                      input_gradients=tf.constant([0.0, 0.1], dtype=tf.float64))
```

    <tf.Tensor: shape=(3,), dtype=float64, numpy=array([0. , 0.4, 0.1])>

```

```


# Noisy GP Experiments

```python
import numpy as np
import numba
import matplotlib as mpl
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from sklearn.utils import check_random_state, check_array, check_X_y
from sklearn.model_selection import train_test_split
from scipy.spatial.distance import pdist, cdist, squareform
from sklearn.metrics.pairwise import check_pairwise_arrays, euclidean_distances
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import _check_length_scale, WhiteKernel, RBF
from scipy import interpolate
from sklearn.metrics import mean_absolute_error, mean_squared_error
import warnings
mpl.rcParams['savefig.pad_inches'] = 0

%matplotlib inline
```

## 1D Toy Example

```python
figure_location = '/home/emmanuel/projects/2018_igarss/figures/noisy_gp/1d_example/'

```

```python
weights = {}
predictions = {}
variance = {}
scores = {}
```

### Sample Data

```python
seed = 123
rng = np.random.RandomState(seed=seed)

# sample data parameters
n_train, n_test, n_trial = 60, 100, 2000
sigma_y = 0.05
x_cov = 0.8
y_scale = .5
x_min, x_max = -10, 10

# real function
f = lambda x: np.sin(1.0 * np.pi / 1.6 * np.cos(5 + y_scale * x))
# f = lambda x: np.sinc(x)
# f = lambda x: np.sin(2. * x) + np.exp(0.2 * x)

# Training add x, y = f(x)
x = np.linspace(x_min, x_max, n_train + n_test + n_trial)

x, xs, = train_test_split(x, train_size=n_train, random_state=seed)

# add noise
y = f(x)
x_train = x + x_cov * rng.randn(n_train)
y_train = f(x) + sigma_y * rng.randn(n_train)

x_train, y_train = x_train[:, np.newaxis], y_train[:, np.newaxis]

# -----------------
# Testing Data
# -----------------
xs, xt = train_test_split(xs, train_size=n_test, random_state=seed)
xt = np.sort(xt)

ys = f(xs)

# Add noise
x_test = xs + x_cov * rng.randn(n_test)
y_test = ys

x_test, y_test = x_test[:, np.newaxis], y_test[:, np.newaxis]

# -------------------
# Plot Points
# -------------------
x_plot = np.linspace(x_min, x_max, n_test)[:, None]
y_plot = f(x_plot)
```

    /home/emmanuel/.conda/envs/sci_py36/lib/python3.6/site-packages/sklearn/model_selection/_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
      FutureWarning)

```python
print(x.shape, y.shape)
print(x_train.shape, y_train.shape)
print(x_test.shape, x_test.shape)
```

    (60,) (60,)
    (60, 1) (60, 1)
    (100, 1) (100, 1)

```python
# plot toy data
fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, y_plot, linewidth=5, color='k', label='Real Function')
ax.scatter(x_train, y_train, color='r', s=100, label='Training Data')
ax.scatter(x_test, y_test, color='g', s=100, label='Testing Data')

# ax.set_title('Training and Testing Data', fontsize=20)
ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()

save_name = 'training_data.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Naive GP Implementation (to Find Suitable Hyperparameters)

```python
def fit_gp(x_train, y_train, kernel='ard', scale=None,
           length_scale_bounds = (0.001, 100),
           noise_level_bounds = (1e-4, 10),
           n_restarts=3):

    warnings.simplefilter('ignore')
    x_train, y_train = check_X_y(x_train, y_train)
    warnings.simplefilter('default')
    n_train, d_dims = x_train.shape

    if kernel in ['ard', 'ARD']:
        length_scale_init = np.ones(shape=(d_dims))
    elif kernel in ['rbf', 'RBF']:
        length_scale_init = 1.0
    else:
        raise ValueError('Unrecognized kernel function...')

    noise_level_init = 1.0

    gp_kernel = RBF(length_scale=length_scale_init,
                    length_scale_bounds=length_scale_bounds) + \
                WhiteKernel(noise_level=noise_level_init,
                            noise_level_bounds=noise_level_bounds)

    gpr_model = GaussianProcessRegressor(kernel=gp_kernel, random_state=123,
                                         n_restarts_optimizer=n_restarts)

    # Fit the GP Model
    gpr_model.fit(x_train, y_train)

    # the parameters
    length_scale = gpr_model.kernel_.k1.length_scale
    sigma = gpr_model.kernel_.k2.noise_level

    return length_scale, sigma
```

```python
length_scale, sigma_y = fit_gp(x_train, y_train, n_restarts=0)
print('Length Scale: {:.3f}'.format(length_scale))
print('Sigma y: {:.3f}'.format(sigma_y))
```

    Length Scale: 2.069
    Sigma y: 0.092

## Gaussian Covariance Function (ARD/RBF Kernel)

\begin{equation}
    K(x, y) = \nu \cdot \text{exp} \left[ -\frac{1}{2} (x - y)^{\top} \cdot W^{-1} \cdot (x - y) \right]
\end{equation}

* $W^{-1}= \text{diag}\left[ w_1, w_2, \ldots, w_{D} \right]$
* $w_d = \frac{1}{\lambda_d^2}$
* $\nu$ is the signal variance parameter

```python
def ard_kernel(x, y=None, length_scale=None, scale=None, weighted=None, x_cov=None):
    """The Automatic Relevance Determination Kernel.

    Parameters
    ----------
    x : array-like, (n_samples x n_dimensions)
        An array for the left argument for the returned kernel K(X,Y)

    y : array-like, optional (n_samples x n_dimensions), default = None
        The right argument for the returned kernel K(X, Y). If none,
        K(X, X) is evaulated instead.

    length_scale : array (n_dimensions), default: 1.0
        The length scale of the kernel.

    scale : float, default: 1.0
        The vertical scale relative to the zero mean of the process in the
        output space.

    Returns
    -------
    K : array, (n_samples x n_samples)
        The kernel matrix for K(X,Y) or K(X,X)

    Information
    -----------
    Author : Juan Emmanuel Johnson

    References
    ----------
    Scikit-Learn (RBF Kernel): https://goo.gl/Sz5icv
    """
    # check if x, y have the same shape
    if y is not None:
        x, y = check_pairwise_arrays(x, y)

    # grab samples and dimensions
    n_samples, n_dimensions = x.shape

    # check length scale to be appropriate
    length_scale = _check_length_scale(x, length_scale)

    # get default scale value
    if scale is None:
        scale = 1.0

    # compute the euclidean distances
    if y is None:
        dists = pdist(x / length_scale, metric='sqeuclidean')

        K = np.exp(-.5 * dists)

        # convert from upper-triangular matrix to square matrix
        K = squareform(K)
        np.fill_diagonal(K, 1)

    else:
        dists = cdist(x / length_scale, y / length_scale, metric='sqeuclidean')

        # exponentiate the distances
        K = scale * np.exp( -0.5 * dists)

    return K
```

```python
def ard_derivative(x_train, x_test, weights, length_scale, scale=None, n_der=1):
    """Derivative of the GP mean function of the ARD Kernel. This function
    computes the derivative of the mean function that has been trained with an
    ARD kernel with respect to the testing points.

    Parameters
    ----------
    x_train : array-like, (n_train_samples x d_dimensions)
        The training samples used to train the weights and the length scale
        parameters.

    x_test : array-like, (n_test_samples x d_dimensions)
        The test samples that will be used to compute the derivative.

    weights : array-like, (n_train_samples, 1)
        The weights used from the training samples

    length_scale : array, (d_dimensions)
        The length scale for the ARD kernel. This includes a sigma value
        for each dimension.

    n_der : int, default: 1, ('1', '2')
        The nth derivative for the mean GP/KRR function with the ARD kernel

    Returns
    -------
    derivative : array-like, (n_test_samples x d_dimensions)
        The computed derivative.

    Information
    -----------
    Author : Juan Emmanuel Johnson
    Email  : jemanjohnson34@gmail.com

    References
    ----------
    Differenting GPs:
        http://mlg.eng.cam.ac.uk/mchutchon/DifferentiatingGPs.pdf
    """

    # check the sizes of x_train and x_test
    err_msg = "xtrain and xtest d dimensions are not equivalent."
    np.testing.assert_equal(x_test.shape[1], x_train.shape[1], err_msg=err_msg)

    n_train_samples, d_dimensions = x_train.shape
    n_test_samples = x_test.shape[0]
    d_length_scale = np.shape(length_scale)

    length_scale = _check_length_scale(x_train, length_scale)

    # Make the length_scale 1 dimensional
    if np.ndim(length_scale) == 0:
        length_scale = np.array([length_scale])
    if np.ndim(weights) == 1:
        weights = weights[:, np.newaxis]

    if len(length_scale) == 1 and d_dimensions > 1:
        length_scale = length_scale * np.ones(shape=(d_dimensions))
    elif len(length_scale) != d_dimensions:
        raise ValueError('Incorrect Input for length_scale.')

    # check the n_samples for x_train and weights are equal
    err_msg = "Number of training samples for xtrain and weights are not equal."
    np.testing.assert_equal(x_train.shape[0], weights.shape[0], err_msg=err_msg)

    if int(n_der) == 1:
        constant_term = np.diag(- np.power(length_scale**2, -1))

    else:
        constant_term2 = (1 / length_scale)**2
        constant_term4 = (1 / length_scale)**4

    # calculate the ARD Kernel
    kernel_mat = ard_kernel(x_test, x_train, length_scale=length_scale, scale=scale)

    # initialize derivative matrix
    derivative = np.zeros(shape=(n_test_samples, d_dimensions))
    if int(n_der) == 1:
        for itest in range(n_test_samples):

            x_tilde = (x_test[itest, :] - x_train).T

            kernel_term = (kernel_mat[itest, :][:, np.newaxis] * weights)

            derivative[itest, :] = constant_term.dot(x_tilde).dot(kernel_term).squeeze()

    else:
        for itest in range(n_test_samples):

            x_term = np.dot(constant_term2, np.ones(shape=(n_dimensions,
                                                          n_train_samples)))

            x_term += np.dot(constant_term4, (x_test[itest, :] - x_train).T**2)

            derivative[itest, :] = np.dot(x_term, kernel_mat[itest, :] * weights).T

    return derivative
```

### GP/KRR Manually

```python
from sklearn.gaussian_process.kernels import (_check_length_scale)
from sklearn.base import BaseEstimator, RegressorMixin
from sklearn.utils import check_array, check_X_y
from sklearn.gaussian_process.kernels import _check_length_scale

class GP_Simple(BaseEstimator, RegressorMixin):
    def __init__(self, length_scale=None, sigma_y=None, scale=None, kernel='ard',
                 x_covariance=None):
        self.length_scale = length_scale
        self.sigma_y = sigma_y
        self.scale = scale
        self.kernel = kernel
        self.x_covariance = x_covariance

    def fit(self, x, y):

        # check input dimensions
        x_train, y_train = check_X_y(x, y)

        self.n_train, self.d_dim = x_train.shape

        # check if length scale and sigma y are there
        try:
            if self.length_scale.any() is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)
        except:
            if self.length_scale is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)

        if self.scale is None:
            self.scale = 1.0

        # Calculate the training kernel (ARD)
        if self.kernel in ['rbf', 'ard']:
            K_train = ard_kernel(x_train, length_scale=self.length_scale)
        elif self.kernel in ['weighted']:
            K_train = ard_kernel_weighted(x_train, x_cov=self.x_covariance,
                                          length_scale=self.length_scale)
        else:
            raise ValueError('Unrecognized kernel function.')

        # add white noise kernel
        L = np.linalg.cholesky(K_train + self.sigma_y * np.eye(K_train.shape[0]))

        # Calculate the weights
        weights = np.linalg.solve(L.T, np.linalg.solve(L, y_train))

        if weights.ndim == 1:
            weights = weights[:, np.newaxis]

        # save variables
        self.x_train = x_train
        self.y_train = y_train
        self.K_ = K_train
        self.L_ = L
        self.weights_ = weights

    def predict(self, x, return_variance=False):

        x_test = check_array(x)

        # Calculate the train test kernel
        if self.kernel in ['rbf', 'ard']:
            K_traintest = ard_kernel(x_test, self.x_train, length_scale=self.length_scale)

        elif self.kernel in ['weighted']:
            K_traintest = ard_kernel_weighted(x_test, self.x_train, x_cov=self.x_covariance,
                                              length_scale=self.length_scale)
        else:
            raise ValueError('Unrecognized kernel function.')

        if not return_variance:
            return K_traintest.dot(self.weights_)

        else:
            predictions = K_traintest.dot(self.weights_)

            variance = self._calculate_variance(x_test, K_traintest)
            return predictions, variance

    def _calculate_variance(self, x, K_traintest=None):

        x_test = check_array(x)

        K_test = np.diag(ard_kernel(x_test, length_scale=self.length_scale))

        if K_traintest is None:
            if self.kernel in ['rbf', 'ard']:
                K_traintest = ard_kernel(x_test, self.x_train, length_scale=self.length_scale)

            elif self.kernel in ['weighted']:
                K_traintest = ard_kernel_weighted(x_test, self.x_train, x_cov=self.x_covariance,
                                                  length_scale=self.length_scale)
            else:
                raise ValueError('Unrecognized kernel function.')

        v = np.linalg.solve(self.L_, K_traintest.T)

        return self.sigma_y + K_test - np.diag(np.dot(v.T, v))

```

```python
# Initialize GP
gp_simple = GP_Simple(length_scale=length_scale,
                       sigma_y=sigma_y)

# Fit the model
gp_simple.fit(x_train, y_train.ravel());

# Predict
predictions['simple'],  var = gp_simple.predict(x_test, return_variance=True)

scores['simple'] = mean_absolute_error(predictions['simple'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['simple']))
print(var.min(), var.max())
```

    MAE Score: 0.22903

    0.10265435963836944 0.13125602277965664

#### Plot Simple GP Results

```python
# Get Figure for x_plot
predictions['simple'], variance['simple'] = gp_simple.predict(x_plot, return_variance=True)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['simple'], linewidth=5, color='k', label='Predictions (GP)')
ax.fill_between(x_plot.squeeze(),
                predictions['simple'].squeeze() - 2*np.sqrt(variance['simple']).squeeze(),
                predictions['simple'].squeeze() + 2*np.sqrt(variance['simple']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_simple.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

### Gaussian Derivative Term

\begin{align}
\frac{\partial \bar{f}_*}{\partial x_*} &= \frac{\partial k(x_*, X)}{\partial x_*} \cdot \alpha \\
 &= - \Lambda^{-1} \cdot \tilde{X}_*^{\top} \cdot \left( k(x_{*}, X)^{\top} \odot \alpha \right)
\end{align}

where:

* $\Lambda=\left[ \lambda_1, \lambda_2, \ldots, \lambda_D \right]$, the length scale
* $\tilde{X}_* = \left[ x_* - x_1, x_* - x_2, \ldots, x_* - x_N \right]^{\top} \in \mathcal{R}^{N \times D}$
* $\odot$ is the element-wise (hadamard) product.

```python
def ard_derivative(x_train, x_test, weights, length_scale, scale=None, n_der=1):
    """Derivative of the GP mean function of the ARD Kernel. This function
    computes the derivative of the mean function that has been trained with an
    ARD kernel with respect to the testing points.

    Parameters
    ----------
    x_train : array-like, (n_train_samples x d_dimensions)
        The training samples used to train the weights and the length scale
        parameters.

    x_test : array-like, (n_test_samples x d_dimensions)
        The test samples that will be used to compute the derivative.

    weights : array-like, (n_train_samples, 1)
        The weights used from the training samples

    length_scale : array, (d_dimensions)
        The length scale for the ARD kernel. This includes a sigma value
        for each dimension.

    n_der : int, default: 1, ('1', '2')
        The nth derivative for the mean GP/KRR function with the ARD kernel

    Returns
    -------
    derivative : array-like, (n_test_samples x d_dimensions)
        The computed derivative.

    Information
    -----------
    Author : Juan Emmanuel Johnson
    Email  : jemanjohnson34@gmail.com

    References
    ----------
    Differenting GPs:
        http://mlg.eng.cam.ac.uk/mchutchon/DifferentiatingGPs.pdf
    """

    # check the sizes of x_train and x_test
    err_msg = "xtrain and xtest d dimensions are not equivalent."
    np.testing.assert_equal(x_test.shape[1], x_train.shape[1], err_msg=err_msg)

    n_train_samples, d_dimensions = x_train.shape
    n_test_samples = x_test.shape[0]
    d_length_scale = np.shape(length_scale)

    length_scale = _check_length_scale(x_train, length_scale)

    # Make the length_scale 1 dimensional
    if np.ndim(length_scale) == 0:
        length_scale = np.array([length_scale])
    if np.ndim(weights) == 1:
        weights = weights[:, np.newaxis]

    if len(length_scale) == 1 and d_dimensions > 1:
        length_scale = length_scale * np.ones(shape=(d_dimensions))
    elif len(length_scale) != d_dimensions:
        raise ValueError('Incorrect Input for length_scale.')

    # check the n_samples for x_train and weights are equal
    err_msg = "Number of training samples for xtrain and weights are not equal."
    np.testing.assert_equal(x_train.shape[0], weights.shape[0], err_msg=err_msg)

    if int(n_der) == 1:
        constant_term = np.diag(- np.power(length_scale**2, -1))

    else:
        constant_term2 = (1 / length_scale)**2
        constant_term4 = (1 / length_scale)**4

    # calculate the ARD Kernel
    kernel_mat = ard_kernel(x_test, x_train, length_scale=length_scale, scale=scale)

    # initialize derivative matrix
    derivative = np.zeros(shape=(n_test_samples, d_dimensions))
    if int(n_der) == 1:
        for itest in range(n_test_samples):

            x_tilde = (x_test[itest, :] - x_train).T

            kernel_term = (kernel_mat[itest, :][:, np.newaxis] * weights)

            derivative[itest, :] = constant_term.dot(x_tilde).dot(kernel_term).squeeze()

    else:
        for itest in range(n_test_samples):

            x_term = np.dot(constant_term2, np.ones(shape=(n_dimensions,
                                                          n_train_samples)))

            x_term += np.dot(constant_term4, (x_test[itest, :] - x_train).T**2)

            derivative[itest, :] = np.dot(x_term, kernel_mat[itest, :] * weights).T

    return derivative
```

## GP w/ Derivative Term

```python

class GP_Derivative(BaseEstimator, RegressorMixin):
    def __init__(self, length_scale=None, x_covariance=1.0, sigma_y=None, scale=None,
                 variance_func='diagonal'):

        if isinstance(length_scale, float):
            self.length_scale = np.array([length_scale])

        if isinstance(x_covariance, float):
            self.x_covariance = np.asarray([x_covariance])
        self.length_scale = np.asarray(length_scale)
        self.x_covariance = np.asarray(x_covariance)
        self.sigma_y = sigma_y
        self.scale = scale
        self.variance_func = variance_func

    def fit(self, x, y):

        # check input dimensions
        x_train, y_train = check_X_y(x, y)

        self.n_train, self.d_dim = x_train.shape

        self.length_scale = _check_length_scale(x, self.length_scale)
        self.x_covariance = _check_length_scale(x, self.x_covariance)

        if np.ndim(self.x_covariance) == 0:
            self.x_covariance = np.array([self.x_covariance])

        # check if length scale and sigma y are there
        try:
            if self.length_scale.any() is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)
        except:
            if self.length_scale is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)

        if self.scale is None:
            self.scale = 1.0

        # Calculate the training kernel (ARD)
        K_train = ard_kernel(x_train, length_scale=self.length_scale, scale=self.scale)

        # add white noise kernel and diagonal derivative term
        L = np.linalg.cholesky(K_train + self.sigma_y * np.eye(N=self.n_train))

        # Calculate the weights
        weights = np.linalg.solve(L.T, np.linalg.solve(L, y_train))
        if weights.ndim == 1:
            weights = weights[:, np.newaxis]

        # Calculate the derivative for the training points
        if self.variance_func == 'diagonal':
            derivative = ard_derivative(x_train, x_train,
                                        weights=weights,
                                        length_scale=self.length_scale)
            derivative = np.diag(np.diag(
                np.dot(derivative, np.dot(np.diag(self.x_covariance), derivative.T))))

        elif self.variance_func == 'full':
            derivative = ard_derivative(x_train, x_train,
                                        weights=weights,
                                        length_scale=self.length_scale)

            derivative = derivative.dot(np.diag(self.x_covariance)).dot(derivative.T)
        else:
            raise ValueError('Unrecognized variance function type.')

        # add white noise kernel
        L_der = np.linalg.cholesky(K_train + self.sigma_y * np.eye(K_train.shape[0]) + derivative)

        # save variables
        self.x_train = x_train
        self.y_train = y_train
        self.K_ = K_train
        self.L_ = L
        self.derivative = derivative
        self.L_der_ = L_der
        self.weights_ = weights

    def predict(self, x, return_variance=False):

        x_test = check_array(x)

        # Calculate the weights
        K_traintest = ard_kernel(x_test, self.x_train,
                                 length_scale=self.length_scale)

        if not return_variance:
            return K_traintest.dot(self.weights_)

        else:
            predictions = K_traintest.dot(self.weights_)

            variance = self._calculate_variance(x_test, K_traintest)
            return predictions, variance

    def _calculate_derivative(self, x, y, K_train=None):

        # Calculate the training Kernel (ARD)
        if K_train is None:
            K_train = ard_kernel(x, length_scale=self.length_scale, scale=self.scale)

        # Calculate the weights for the initial kernel
        L = np.linalg.cholesky(K_train + self.sigma_y * np.eye(self.n_train))
        initial_weights = np.linalg.solve(L.T, np.linalg.solve(L, y))[:, np.newaxis]

        # Calculate the derivative
        return ard_derivative(x, x, weights=initial_weights, length_scale=self.length_scale)

    def _calculate_variance(self, x, K_traintest=None):

        x_test = check_array(x)

        K_test = np.diag(ard_kernel(x_test, length_scale=self.length_scale))

        if K_traintest is None:
            K_traintest = ard_kernel(x_test, self.x_train, length_scale=self.length_scale)

        v = np.linalg.solve(self.L_der_, K_traintest.T)

        # calculate the derivative for the testing points
        derivative = ard_derivative(self.x_train, x_test, weights=self.weights_,
                                    length_scale=self.length_scale)

        derivative = np.diag(np.dot(derivative, np.dot(np.diag(self.x_covariance), derivative.T)))

        return self.sigma_y + derivative + K_test - np.diag(np.dot(v.T, v))
```

##### Plot Derivative Term Results

```python
# Initialize GP
gp_derivative_full = GP_Derivative(length_scale=length_scale,
                    sigma_y=sigma_y, variance_func='full',
                      x_covariance=x_cov)

# Fit the model
gp_derivative_full.fit(x_train, y_train.ravel());

# Predict
predictions['derivative'],  var = gp_derivative_full.predict(x_test, return_variance=True)

scores['derivative'] = mean_absolute_error(predictions['derivative'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['derivative']))
print(var.min(), var.max())
```

    MAE Score: 0.22903

    0.10614628922364178 0.7439577580294922

```python
# Initialize GP
gp_derivative_diag = GP_Derivative(length_scale=length_scale,
                               sigma_y=sigma_y, variance_func='diagonal',
                               x_covariance=x_cov)

# Fit the model
gp_derivative_diag.fit(x_train, y_train.ravel());

# Predict
predictions['derivative_diag'],  var = gp_derivative_diag.predict(x_test, return_variance=True)

scores['derivative_diag'] = mean_absolute_error(predictions['derivative_diag'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['derivative_diag']))
print(var.min(), var.max())
```

    MAE Score: 0.22903

    0.10858334826254401 0.6934617253473325

```python
# Get Figure for x_plot
# gp_derivative.weights_ = gp_simple.weights_
predictions['derivative'], variance['derivative'] = gp_derivative_full.predict(x_plot, return_variance=True)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['derivative'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['derivative'].squeeze() - 2*np.sqrt(variance['derivative']).squeeze(),
                predictions['derivative'].squeeze() + 2*np.sqrt(variance['derivative']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_derivative.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

```python
predictions['derivative_diag'], variance['derivative_diag'] = gp_derivative_diag.predict(x_plot, return_variance=True)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['derivative_diag'], linewidth=5, color='k', label='Predictions (GP + Error)')
ax.scatter(x_train, y_train, s=100, color='r', label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['derivative_diag'].squeeze() - 2*np.sqrt(variance['derivative_diag']).squeeze(),
                predictions['derivative_diag'].squeeze() + 2*np.sqrt(variance['derivative_diag']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_derivative_diag.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## GP: Corrective Kernel + Derivative Term

\begin{equation}
C(u, x) = |I + W^{-1}\Sigma_x|^{-1/2} \text{exp} \left[-\frac{1}{2}(u-x)^{\top}\Delta^{-1}(u-x) \right]
\end{equation}

* $\Delta^{-1}=W^{-1}(W^{-1} + \Sigma_x^{-1})^{-1}W^{-1}$

```python
def ard_kernel_weighted(x, y=None, x_cov=None, length_scale=None, scale=None):

    # check if x, y have the same shape
    if y is not None:
        x, y = check_pairwise_arrays(x, y)

    # grab samples and dimensions
    n_samples, n_dimensions = x.shape

    # get the default sigma values
    if length_scale is None:
        length_scale = np.ones(shape=n_dimensions)

    else:
        length_scale = _check_length_scale(x, length_scale)

    # check covariance values
    if x_cov is None:
        x_cov = 0.0
    else:
        x_cov = _check_length_scale(x, x_cov)

    # Add dimensions to lengthscale and x_cov
    if np.ndim(length_scale) == 0:
        length_scale = np.array([length_scale])

    if np.ndim(x_cov) == 0:
        x_cov = np.array([x_cov])

    # get default scale values
    if scale is None:
        scale = 1.0

    exp_scale = np.sqrt(x_cov + length_scale**2)

    scale_term = np.diag(x_cov * (length_scale**2)**(-1)) + np.eye(N=n_dimensions)
    scale_term = np.linalg.det(scale_term)
    scale_term = np.power(scale_term, -1/2)

    if y is None:
        dists = pdist(x / exp_scale, metric='sqeuclidean')

        K = np.exp(- 0.5 * dists)

        K = squareform(K)

        np.fill_diagonal(K, 1)

        K *= scale_term

    else:

        dists = cdist(x / exp_scale, y / exp_scale, metric='sqeuclidean')

        K = np.exp(- 0.5 * dists)

        K *= scale_term

    return K
```

### Corrective Derivative Term

\begin{equation}
\nu(u, \Sigma_x) = \sigma^2(u) + \Sigma_{i,j=1}^{N} K_{i,j}^{-1} \cdot K(u, x_{i}) \cdot K(u, x_{j}) \cdot \left(1 - H(u, \bar{x}) \right) + \Sigma_{i,j=1}^{N} \beta_i \cdot \beta_j \cdot K(u, x_{i}) \cdot K(u, x_{j}) \cdot \left[ H(u, \bar{x}) - H(u, x_i) \cdot H(u, x_j) \right]
\end{equation}

where:

\begin{equation}
    H(u, \bar{x}) = \left| \left( \frac{W}{2} \right)^{-1} \Sigma_x + I \right|^{-1/2} \text{exp}
    \left[ \frac{1}{2}(u-\bar{x})^T \Delta^{-1} (u - \bar{x}) \right]
\end{equation}

* $\Delta^{-1} = \left( \frac{W}{2} \right)^{-1} \left[ \left( \frac{W}{2} \right)^{-1} + \Sigma_x^{-1} \right]^{-1/2} \left( \frac{W}{2} \right)^{-1}$
* $\bar{x} = \frac{x_i + x_j}{2}$

#### GP w/ Derivative Corrective Term

```python
@numba.jit(nopython=True, nogil=True)
def calculate_q_numba(x_train, x_test, K, det_term, exp_scale):
    """Calculates the Q matrix used to compute the variance of the
    inputs with a noise covariance matrix. This uses numba to
    speed up the calculations.

    Parameters
    ----------
    x_train : array, (n_samples x d_dimensions)
        The data used to train the weights.

    x_test : array, (d_dimensions)
        A vector of test points.

    K : array, (n_samples)
        The portion of the kernel matrix of the training points at
        test point i, e.g. K = full_kernel_mat[:, i_test]

    det_term : float
        The determinant term that's in from of the exponent
        term.

    exp_scale : array, (d_dimensions)
        The length_scale that's used within the exponential term.

    Returns
    -------
    Q : array, (n_samples x n_samples)
        The Q matrix used to calculate the variance of the samples.

    Information
    -----------
    Author : J. Emmanuel Johnson
    Email  : jemanjohnson34@gmail.com
    Date   : 13 - 06 - 2018

    References
    ----------
    McHutchen et al. - Gaussian Process Training with Input Noise
    http://mlg.eng.cam.ac.uk/pub/pdf/MchRas11.pdf
    """
    n_train, d_dimensions = x_train.shape

    Q = np.zeros(shape=(n_train, n_train), dtype=np.float64)

    # Loop through the row terms
    for iterrow in range(n_train):

        # Calculate the row terms
        x_train_row = 0.5 * x_train[iterrow, :]  - x_test

        K_row = K[iterrow] * det_term

        # Loop through column terms
        for itercol in range(n_train):

            # Z Term
            z_term = x_train_row + 0.5 * x_train[itercol, :]

            # EXPONENTIAL TERM
            exp_term = np.exp( np.sum( z_term**2 * exp_scale) )

            # CONSTANT TERM
            constant_term = K_row * K[itercol]

            # Q Matrix (Corrective Gaussian Kernel)
            Q[iterrow, itercol] = constant_term * exp_term

    return Q
```

```python
class GP_Corrective(object):
    def __init__(self, length_scale=None, x_covariance=1.0, sigma_y=None, scale=None):
        self.length_scale = length_scale
        self.x_covariance = x_covariance
        self.sigma_y = sigma_y
        self.scale = scale

    def fit(self, x, y):

        # check input dimensions
        x_train, y_train = check_X_y(x, y)

        self.n_train, self.d_dim = x_train.shape

        self.length_scale = _check_length_scale(x, self.length_scale)
        self.x_covariance = _check_length_scale(x, self.x_covariance)

        if np.ndim(self.length_scale) == 0:
            self.length_scale = np.array([self.length_scale])

        if np.ndim(self.x_covariance) == 0:
            self.x_covariance = np.array([self.x_covariance])

        # check if length scale and sigma y are there
        try:
            if self.length_scale.any() is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)
        except:
            if self.length_scale is None or self.sigma_y is None:
                self.length_scale, self.sigma_y = fit_gp(x_train, y_train, n_restarts=0)

        if self.scale is None:
            self.scale = 1.0

        # Calculate the training Kernel (ARD)
        K_train = ard_kernel(x_train, length_scale=self.length_scale, scale=self.scale)

        # Calculate the derivative
        derivative = self._calculate_derivative(x_train, y_train, K_train)

        # Add white noise kernel and derivative term
        derivative_term = np.diag(np.diag(derivative.dot(np.diag(self.x_covariance)).dot(derivative.T)))
#         derivative_term = derivative.dot(np.diag(self.x_covariance)).dot(derivative.T)

        K_train += self.sigma_y**2 * np.eye(N=self.n_train) + derivative_term

        K_train_inv = np.linalg.inv(K_train)

        # Calculate the weights
        weights = K_train_inv.dot(y_train)

        # save variables
        self.x_train = x_train
        self.y_train = y_train
        self.K_ = K_train
        self.K_inv_ = K_train_inv
        self.derivative_ = derivative
        self.weights_ = weights

        return self

    def predict(self, x, return_std=False):

        x_test = check_array(x)

        # Calculate the weights
        K_traintest = ard_kernel_weighted(x_test, self.x_train,
                                          x_cov=self.x_covariance,
                                          length_scale=self.length_scale)

        if not return_std:
            return K_traintest.dot(self.weights_)

        else:
            predictions = K_traintest.dot(self.weights_)
            variance = self._calculate_variance(x_test, predictions=predictions)
            return predictions, variance

    def _calculate_derivative(self, x, y, K_train=None):

        # Calculate the training Kernel (ARD)
        if K_train is None:
            K_train = ard_kernel(x, length_scale=self.length_scale, scale=self.scale)

        # Calculate the weights for the initial kernel
        L = np.linalg.cholesky(K_train + self.sigma_y**2 * np.eye(self.n_train))
        initial_weights = np.linalg.solve(L.T, np.linalg.solve(L, y))[:, np.newaxis]
        self.der_weights_ = initial_weights

        # Calculate the derivative
        return ard_derivative(x, x, weights=initial_weights, length_scale=self.length_scale)

    def _calculate_variance(self, x, predictions=None):

        x_test = check_array(x)
        n_test = x_test.shape[0]

        if predictions is None:
            predictions = self.predict(x_test, return_std=False)

        # Determinant Term
        det_term = 2 * self.x_covariance * np.power(self.length_scale, -2) + 1
        det_term = 1 / np.sqrt(np.linalg.det(np.diag(det_term)))

        # Exponential Term
        exp_scale = np.power(np.power(self.length_scale, 2)
                             + 0.5 * np.power(self.length_scale, 4)
                             * np.power(self.x_covariance, -1), -1)

        K = ard_kernel(self.x_train, x_test, length_scale=self.length_scale)

        variance = np.zeros(shape=(n_test))
        trace_term = variance.copy()
        q_weight_term = variance.copy()
        pred_term = variance.copy()

        # Loop through test points
        for itertest in range(n_test):

            # Calculate Q matrix
            Q = calculate_q_numba(self.x_train, x_test[itertest, :], K[:, itertest], det_term, exp_scale)

            # Terms
            trace_term[itertest] = float(np.trace(np.dot(self.K_inv_, Q)))
            q_weight_term[itertest] = float(self.weights_.T.dot(Q).dot(self.weights_))
            pred_term[itertest] = float(predictions[itertest]**2)
            # calculate the final predictive variance
            variance[itertest] = self.scale - trace_term[itertest] + \
                q_weight_term[itertest] - pred_term[itertest]

        # Negative variances due to numerical issues.
        # Set those variances to 0.
        var_negative = variance < 0
        if np.any(var_negative):
            warnings.warn("Predicted variances smaller than 0. "
                          "Setting those varinaces to 0.")

            variance[var_negative] = 0.0
        return variance

```

```python
# Initialize GP
gp_corrective = GP_Corrective(length_scale=length_scale,
                       x_covariance=x_cov,
                       sigma_y=sigma_y)

# Fit the model
gp_corrective.fit(x_train, y_train.ravel());

# Predict
predictions['corrective'],  var = gp_corrective.predict(x_test, return_std=True)

scores['corrective'] = mean_absolute_error(predictions['corrective'], y_test)
print('\nMAE Score: {:.5f}\n'.format(scores['corrective']))
print(var.min(), var.max())
```

    MAE Score: 0.24083

    0.011152590348052382 0.3583540820915454

```python
# Get Figure for x_plot
predictions['corrective'], variance['corrective'] = gp_corrective.predict(x_plot, return_std=True)

# Plot Figure

fig, ax = plt.subplots(figsize=(10, 7))

ax.plot(x_plot, predictions['corrective'], linewidth=5, color='k', label='Predictions')
ax.scatter(x_train, y_train, color='r', s=100, label='Training Data')
ax.fill_between(x_plot.squeeze(),
                predictions['corrective'].squeeze() - 2*np.sqrt(variance['corrective']).squeeze(),
                predictions['corrective'].squeeze() + 2*np.sqrt(variance['corrective']).squeeze(),
                color = 'red',
                alpha=0.5, label='Standard Deviation')

ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax.patch.set_visible(False)
plt.show()

save_name = 'gp_corrective.png'

fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

## Empirical Variance Comparison

### Calculate the Empirical Variance (Get Scores)

```python
%%time

from sklearn.metrics import mean_absolute_error, mean_squared_error
rng = np.random.RandomState(None)
n_trials = 100

mae_score = {
    'simple': list(),
    'derivative': list(),
    'derivative_diag': list(),
    'corrective': list()
}

mse_score = {
    'simple': list(),
    'derivative': list(),
    'derivative_diag': list(),
    'corrective': list()
}

abs_error = {
    'simple': list(),
    'derivative': list(),
    'derivative_diag': list(),
    'corrective': list()
}
squared_error = {
    'simple': list(),
    'derivative': list(),
    'derivative_diag': list(),
    'corrective': list()
}

# testing set (noise-less)
ys = f(xt)
y_test = ys.squeeze()
y_test = np.atleast_2d(y_test).T

# fig, ax = plt.subplots()

for itrial in range(n_trials):

    # Generate x samples with random error
    x_test = xt + 0.3 * rng.randn(n_trial)
    x_test = x_test[:, np.newaxis]

#     ax.scatter(x_test, y_test)

    # -------------------
    # Model I - Simple GP
    # -------------------
    mean = gp_simple.predict(x_test, return_variance=False).squeeze()

    abs_error['simple'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['simple'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['simple'].append(mean_absolute_error(mean, y_test))
    mse_score['simple'].append(mean_squared_error(mean, y_test))

    # -------------------
    # Model II - Derivative GP
    # -------------------
    mean = gp_derivative_full.predict(x_test, return_variance=False).squeeze()

    abs_error['derivative'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['derivative'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['derivative'].append(mean_absolute_error(mean, y_test))
    mse_score['derivative'].append(mean_squared_error(mean, y_test))

    # -------------------
    # Model II - Derivative GP
    # -------------------
    mean = gp_derivative_diag.predict(x_test, return_variance=False).squeeze()

    abs_error['derivative_diag'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['derivative_diag'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['derivative_diag'].append(mean_absolute_error(mean, y_test))
    mse_score['derivative_diag'].append(mean_squared_error(mean, y_test))
    # -------------------
    # Model V - GP: Corrective
    # -------------------
    mean = gp_corrective.predict(x_test, return_std=False).squeeze()

    abs_error['corrective'].append(np.abs(mean.squeeze() - y_test.squeeze()))
    squared_error['corrective'].append((mean.squeeze() - y_test.squeeze())**2)
    mae_score['corrective'].append(mean_absolute_error(mean, y_test))
    mse_score['corrective'].append(mean_squared_error(mean, y_test))

#     break

# ax.scatter(x_train, y_train, label='training samples')
# ax.plot(x_plot, f(x_plot), color='black', label='real function')
# ax.legend()
# plt.show()
# Make lists into arrays
abs_error['simple'] = np.mean(np.array(abs_error['simple']).squeeze(), axis=0)
squared_error['simple'] = np.mean(np.array(squared_error['simple']).squeeze(), axis=0)
mae_score['simple'] = np.mean(np.array(mae_score['simple']))
mse_score['simple'] = np.mean(np.array(mse_score['simple']))

abs_error['derivative'] = np.mean(np.array(abs_error['derivative']).squeeze(), axis=0)
squared_error['derivative'] = np.mean(np.array(squared_error['derivative']).squeeze(), axis=0)
mae_score['derivative'] = np.mean(np.array(mae_score['derivative']))
mse_score['derivative'] = np.mean(np.array(mse_score['derivative']))

abs_error['derivative_diag'] = np.mean(np.array(abs_error['derivative_diag']).squeeze(), axis=0)
squared_error['derivative_diag'] = np.mean(np.array(squared_error['derivative_diag']).squeeze(), axis=0)
mae_score['derivative_diag'] = np.mean(np.array(mae_score['derivative_diag']))
mse_score['derivative_diag'] = np.mean(np.array(mse_score['derivative_diag']))

abs_error['corrective'] = np.mean(np.array(abs_error['corrective']).squeeze(), axis=0)
squared_error['corrective'] = np.mean(np.array(squared_error['corrective']).squeeze(), axis=0)
mae_score['corrective'] = np.mean(np.array(mae_score['corrective']))
mse_score['corrective'] = np.mean(np.array(mse_score['corrective']))
```

    CPU times: user 9.48 s, sys: 1min 44s, total: 1min 54s
    Wall time: 2.39 s

```python
fig, ax = plt.subplots()

ax.scatter(x_train, y_train, label='training samples')
ax.plot(x_plot, f(x_plot), color='black', label='real function')
ax.legend()
plt.show()
# ax.scatter(xs, abs_error['simple'], color='green', label='Empirical Absolute Error (Points)')
```

```python
print(xs.shape, abs_error['simple'].shape)
```

    (100,) (2000,)

```python
print(mse_score['simple'])
```

    0.051286455493815

### Calculate the Variance and Standard Deviation

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['simple'], kind='slinear')(xnew)

print('MSE: {:.5f}'.format(mse_score['simple']))
print('MAE: {:.5f}'.format(mae_score['derivative']))

pred, var = gp_simple.predict(xt[:, np.newaxis], return_variance=True)

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Variance')

# ax.set_title('Squared Error, Variance')
ax.grid(True)
ax.legend(fontsize=14)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()

save_name = 'exp_empirical_simple_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)

# abs_error_line = interpolate.interp1d(xt, abs_error['simple'], kind='quadratic')(xnew)

# fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

# ax.plot(xnew, abs_error_line, linewidth=3, color='r', label='Empirical Absolute Error')
# ax.plot(x_plot, np.sqrt(variance['simple']), linewidth=4, color='k', label='GP: Standard Deviation')

# # ax.set_title('Absolute Error, Standard Deviation')
# ax.legend(fontsize=12)
# ax.grid(True)
# plt.show()

# save_name = 'exp_empirical_simple_std.png'

# fig.savefig(figure_location + save_name, bbox_inhces='tight')
```

```python
xnew = np.linspace(xt.min(), xt.max(), 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['derivative'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['derivative'], kind='quadratic')(xnew)

pred, var = gp_derivative_full.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['derivative']))
print('MAE: {:.5f}'.format(mae_score['derivative']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

# save_name = 'exp_empirical_derivative_var.png'
# fig.savefig(figure_location + save_name, bbox_inhces='tight',
#             dpi=100, transparent=True, frameon=None)

# fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

# ax.plot(xnew, abs_error_line, linewidth=3, color='r', label='Empirical Absolute Error')
# ax.plot(x_plot, np.sqrt(variance['derivative']), linewidth=4, color='k', label='GP: Standard Deviation')

# # ax.set_title('Absolute Error, Standard Deviation')
# ax.legend(fontsize=12)
# ax.grid(True)
# plt.show()

# save_name = 'exp_empirical_derivative_std.png'

# fig.savefig(figure_location + save_name, bbox_inhces='tight')
```

```python
xnew = np.linspace(x_min, x_max, 1000)

abs_error_line = interpolate.interp1d(xt, abs_error['derivative_diag'], kind='cubic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['derivative_diag'], kind='slinear')(xnew)

pred, var = gp_derivative_diag.predict(xt[:, np.newaxis], return_variance=True)

print('MSE: {:.5f}'.format(mse_score['derivative_diag']))
print('MAE: {:.5f}'.format(mae_score['derivative_diag']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Error Variance')

# ax.set_title('Squared Error, Variance')
ax.legend(fontsize=14)
# ax.get_xaxis().set_visible(False)
# ax.get_yaxis().set_visible(False)
ax.grid(True)
plt.show()

save_name = 'exp_empirical_derivative_diag_var.png'
fig.savefig(figure_location + save_name, bbox_inhces='tight',
            dpi=100, transparent=True, frameon=None)
```

```python
xnew = np.linspace(xt.min(), xt.max(), 300)

abs_error_line = interpolate.interp1d(xt, abs_error['corrective'], kind='quadratic')(xnew)

# Interpolation function for squared error
squared_error_line = interpolate.interp1d(xt, squared_error['corrective'], kind='quadratic')(xnew)

pred, var = gp_corrective.predict(xt[:, np.newaxis], return_std=True)

print('MSE: {:.5f}'.format(mse_score['corrective']))
print('MAE: {:.5f}'.format(mae_score['corrective']))

fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

ax.plot(xnew, squared_error_line, linewidth=3, color='r', label='Empirical Squared Error')
ax.plot(xt, var, linewidth=4, color='k', label='GP: Variance')

# ax.set_title('Squared Error, Variance')
ax.grid(True)
ax.legend(fontsize=12)
plt.show()

save_name = 'exp_empirical_corrective_var.png'

# fig.savefig(figure_location + save_name, bbox_inhces='tight')

# fig, ax = plt.subplots(nrows=1, figsize=(10, 7))

# ax.plot(xnew, abs_error_line, linewidth=3, color='r', label='Empirical Absolute Error')
# ax.plot(x_plot, np.sqrt(variance['corrective']), linewidth=4, color='k', label='GP: Standard Deviation')

# # ax.set_title('Absolute Error, Standard Deviation')
# ax.legend(fontsize=12)
# ax.grid(True)
# plt.show()

# save_name = 'exp_empirical_corrective_std.png'

# fig.savefig(figure_location + save_name, bbox_inhces='tight')
```

```python
xtrain = np.array([1, 5, 3])[:, np.newaxis]
xtest = np.array([1])[:, np.newaxis]
print(xtrain.shape, xtest.shape)
n_samples = xtrain.shape[0]
d_dimensions = xtrain.shape[1]
```

    (3, 1) (1, 1)

```python
xrow = xtrain[:, np.newaxis, :]
xrow = np.tile(xrow, [1, n_samples, d_dimensions])
print(xrow)
print(xrow.shape)
```

    [[[1]
      [1]
      [1]]

     [[5]
      [5]
      [5]]

     [[3]
      [3]
      [3]]]
    (3, 3, 1)

```python
def zterm_matrix(xtrain):
    n_samples, d_dimensions = xtrain.shape

    xrow = xtrain[:, np.newaxis, :]
    xrow = np.tile(xrow, [1, n_samples, d_dimensions])
    zterm = 0.5 * (xrow + np.transpose(xrow, axes=(1,0, 2)))
    return zterm
```

```python
xcol = xrow.copy() #xtrain[:, np.newaxis, :]
xcol = np.transpose(xcol, axes=(1,0, 2))
print(xcol)
print(xcol.shape)

```

    [[[1]
      [5]
      [3]]

     [[1]
      [5]
      [3]]

     [[1]
      [5]
      [3]]]
    (3, 3, 1)

```python
zterm = 0.5 * (xrow + xcol)
zterm = zterm_matrix(xtrain)
print(zterm == zterm2)
print(zterm.shape)
```

```python
zterm = zterm_matrix(xtrain)
zterm = zterm - xtest
print(zterm.shape, zterm.T.shape)
```

    (3, 3, 1) (1, 3, 3)

```python
diag_term = np.array([1.0])
print(zterm.shape, diag_term.shape)
exp_term = zterm.dot(diag_term).dot(zterm.T)
print(exp_term.shape)
```

    (3, 3, 1) (1,)
    (3, 1, 3)

```python

```


```python
from __future__ import print_function
from sympy import Symbol, symbols, sin, cos, Rational, expand, simplify, collect, S
from galgebra.printer import Eprint, Get_Program, Print_Function, Format
from galgebra.ga import Ga, one, zero
from galgebra.mv import Nga
Format()
```

```python
X = (x, y, z) = symbols('x y z')
o3d = Ga('e_x e_y e_z', g=[1, 1, 1], coords=X)
(ex, ey, ez) = o3d.mv()
grad = o3d.grad
```

```python
c = o3d.mv('c', 'scalar')
```

```python
a = o3d.mv('a', 'vector')
b = o3d.mv('b', 'vector')
```

```python
A = o3d.mv('A','mv')
B = o3d.mv('B','mv')
```

The inner product of blades in GAlgebra is zero if either operand is a scalar:

$$\begin{split}\begin{aligned}
      {\boldsymbol{A}}_{r}{\wedge}{\boldsymbol{B}}_{s} &\equiv {\left <{{\boldsymbol{A}}_{r}{\boldsymbol{B}}_{s}} \right >_{r+s}} \\
      {\boldsymbol{A}}_{r}\cdot{\boldsymbol{B}}_{s} &\equiv {\left \{ { \begin{array}{cc}
      r\mbox{ and }s \ne 0: & {\left <{{\boldsymbol{A}}_{r}{\boldsymbol{B}}_{s}} \right >_{{\left |{r-s}\right |}}}  \\
      r\mbox{ or }s = 0: & 0 \end{array}} \right \}}
   \end{aligned}\end{split}$$

This definition comes from _David Hestenes and Garret Sobczyk, “Clifford Algebra to Geometric Calculus,” Kluwer Academic Publishers, 1984_.

In some other literature, the inner product is defined without the exceptional case for scalar part and the definition above is known as "the modified Hestenes inner product" (this name comes from the source code of [GAViewer](http://www.geometricalgebra.net/gaviewer_download.html)).

```python
c|a
```

\begin{equation*} 0 \end{equation*}

```python
a|c
```

\begin{equation*} 0 \end{equation*}

```python
c|A
```

\begin{equation*} 0 \end{equation*}

```python
A|c
```

\begin{equation*} 0 \end{equation*}

$ab=a \wedge b + a \cdot b$ holds for vectors:

```python
a*b
```

\begin{equation*}\left ( a^{x} b^{x} + a^{y} b^{y} + a^{z} b^{z}\right )  + \left ( a^{x} b^{y} - a^{y} b^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( a^{x} b^{z} - a^{z} b^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( a^{y} b^{z} - a^{z} b^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
a^b
```

\begin{equation*}\left ( a^{x} b^{y} - a^{y} b^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( a^{x} b^{z} - a^{z} b^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( a^{y} b^{z} - a^{z} b^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
a|b
```

\begin{equation*}a^{x} b^{x} + a^{y} b^{y} + a^{z} b^{z}\end{equation*}

```python
(a*b)-(a^b)-(a|b)
```

\begin{equation*} 0 \end{equation*}

$aA=a \wedge A + a \cdot A$ holds for the products between vectors and multivectors:

```python
a*A
```

\begin{equation*}\left ( A^{x} a^{x} + A^{y} a^{y} + A^{z} a^{z}\right )  + \left ( A a^{x} - A^{xy} a^{y} - A^{xz} a^{z}\right ) \boldsymbol{e}_{x} + \left ( A a^{y} + A^{xy} a^{x} - A^{yz} a^{z}\right ) \boldsymbol{e}_{y} + \left ( A a^{z} + A^{xz} a^{x} + A^{yz} a^{y}\right ) \boldsymbol{e}_{z} + \left ( - A^{x} a^{y} + A^{xyz} a^{z} + A^{y} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - A^{x} a^{z} - A^{xyz} a^{y} + A^{z} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( A^{xyz} a^{x} - A^{y} a^{z} + A^{z} a^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z} + \left ( A^{xy} a^{z} - A^{xz} a^{y} + A^{yz} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
a^A
```

\begin{equation*}A a^{x} \boldsymbol{e}_{x} + A a^{y} \boldsymbol{e}_{y} + A a^{z} \boldsymbol{e}_{z} + \left ( - A^{x} a^{y} + A^{y} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - A^{x} a^{z} + A^{z} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( - A^{y} a^{z} + A^{z} a^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z} + \left ( A^{xy} a^{z} - A^{xz} a^{y} + A^{yz} a^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
a|A
```

\begin{equation*}\left ( A^{x} a^{x} + A^{y} a^{y} + A^{z} a^{z}\right )  + \left ( - A^{xy} a^{y} - A^{xz} a^{z}\right ) \boldsymbol{e}_{x} + \left ( A^{xy} a^{x} - A^{yz} a^{z}\right ) \boldsymbol{e}_{y} + \left ( A^{xz} a^{x} + A^{yz} a^{y}\right ) \boldsymbol{e}_{z} + A^{xyz} a^{z} \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} - A^{xyz} a^{y} \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + A^{xyz} a^{x} \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
(a*A)-(a^A)-(a|A)
```

\begin{equation*} 0 \end{equation*}

$AB=A \wedge B + A \cdot B$ does NOT hold for the products between multivectors and multivectors:

```python
A*B
```

\begin{equation*}\left ( A B + A^{x} B^{x} - A^{xy} B^{xy} - A^{xyz} B^{xyz} - A^{xz} B^{xz} + A^{y} B^{y} - A^{yz} B^{yz} + A^{z} B^{z}\right )  + \left ( A B^{x} + A^{x} B + A^{xy} B^{y} - A^{xyz} B^{yz} + A^{xz} B^{z} - A^{y} B^{xy} - A^{yz} B^{xyz} - A^{z} B^{xz}\right ) \boldsymbol{e}_{x} + \left ( A B^{y} + A^{x} B^{xy} - A^{xy} B^{x} + A^{xyz} B^{xz} + A^{xz} B^{xyz} + A^{y} B + A^{yz} B^{z} - A^{z} B^{yz}\right ) \boldsymbol{e}_{y} + \left ( A B^{z} + A^{x} B^{xz} - A^{xy} B^{xyz} - A^{xyz} B^{xy} - A^{xz} B^{x} + A^{y} B^{yz} - A^{yz} B^{y} + A^{z} B\right ) \boldsymbol{e}_{z} + \left ( A B^{xy} + A^{x} B^{y} + A^{xy} B + A^{xyz} B^{z} - A^{xz} B^{yz} - A^{y} B^{x} + A^{yz} B^{xz} + A^{z} B^{xyz}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( A B^{xz} + A^{x} B^{z} + A^{xy} B^{yz} - A^{xyz} B^{y} + A^{xz} B - A^{y} B^{xyz} - A^{yz} B^{xy} - A^{z} B^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( A B^{yz} + A^{x} B^{xyz} - A^{xy} B^{xz} + A^{xyz} B^{x} + A^{xz} B^{xy} + A^{y} B^{z} + A^{yz} B - A^{z} B^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z} + \left ( A B^{xyz} + A^{x} B^{yz} + A^{xy} B^{z} + A^{xyz} B - A^{xz} B^{y} - A^{y} B^{xz} + A^{yz} B^{x} + A^{z} B^{xy}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
A|B
```

\begin{equation*}\left ( A^{x} B^{x} - A^{xy} B^{xy} - A^{xyz} B^{xyz} - A^{xz} B^{xz} + A^{y} B^{y} - A^{yz} B^{yz} + A^{z} B^{z}\right )  + \left ( A^{xy} B^{y} - A^{xyz} B^{yz} + A^{xz} B^{z} - A^{y} B^{xy} - A^{yz} B^{xyz} - A^{z} B^{xz}\right ) \boldsymbol{e}_{x} + \left ( A^{x} B^{xy} - A^{xy} B^{x} + A^{xyz} B^{xz} + A^{xz} B^{xyz} + A^{yz} B^{z} - A^{z} B^{yz}\right ) \boldsymbol{e}_{y} + \left ( A^{x} B^{xz} - A^{xy} B^{xyz} - A^{xyz} B^{xy} - A^{xz} B^{x} + A^{y} B^{yz} - A^{yz} B^{y}\right ) \boldsymbol{e}_{z} + \left ( A^{xyz} B^{z} + A^{z} B^{xyz}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - A^{xyz} B^{y} - A^{y} B^{xyz}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( A^{x} B^{xyz} + A^{xyz} B^{x}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
(A*B)-(A^B)-(A|B)
```

\begin{equation*}\left ( - A^{xz} B^{yz} + A^{yz} B^{xz}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( A^{xy} B^{yz} - A^{yz} B^{xy}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( - A^{xy} B^{xz} + A^{xz} B^{xy}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python
(A<B)+(A|B)+(A>B)-A*B
```

\begin{equation*}\left ( A B + 2 A^{x} B^{x} - 2 A^{xy} B^{xy} - 2 A^{xyz} B^{xyz} - 2 A^{xz} B^{xz} + 2 A^{y} B^{y} - 2 A^{yz} B^{yz} + 2 A^{z} B^{z}\right )  + \left ( A^{xy} B^{y} - A^{xyz} B^{yz} + A^{xz} B^{z} - A^{y} B^{xy} - A^{yz} B^{xyz} - A^{z} B^{xz}\right ) \boldsymbol{e}_{x} + \left ( A^{x} B^{xy} - A^{xy} B^{x} + A^{xyz} B^{xz} + A^{xz} B^{xyz} + A^{yz} B^{z} - A^{z} B^{yz}\right ) \boldsymbol{e}_{y} + \left ( A^{x} B^{xz} - A^{xy} B^{xyz} - A^{xyz} B^{xy} - A^{xz} B^{x} + A^{y} B^{yz} - A^{yz} B^{y}\right ) \boldsymbol{e}_{z} + \left ( - A^{x} B^{y} + A^{xyz} B^{z} + A^{xz} B^{yz} + A^{y} B^{x} - A^{yz} B^{xz} + A^{z} B^{xyz}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y} + \left ( - A^{x} B^{z} - A^{xy} B^{yz} - A^{xyz} B^{y} - A^{y} B^{xyz} + A^{yz} B^{xy} + A^{z} B^{x}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{z} + \left ( A^{x} B^{xyz} + A^{xy} B^{xz} + A^{xyz} B^{x} - A^{xz} B^{xy} - A^{y} B^{z} + A^{z} B^{y}\right ) \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z} + \left ( - A^{x} B^{yz} - A^{xy} B^{z} + A^{xz} B^{y} + A^{y} B^{xz} - A^{yz} B^{x} - A^{z} B^{xy}\right ) \boldsymbol{e}_{x}\wedge \boldsymbol{e}_{y}\wedge \boldsymbol{e}_{z}\end{equation*}

```python

```