# GP Regression with Uncertain Inputs

## Introduction

In this notebook, we're going to demonstrate one way of dealing with uncertainty in our training data. Let's say that we're collecting training data that models the following function.

\begin{align}
y &= \sin(2\pi x) + \epsilon \\
  \epsilon &\sim \mathcal{N}(0, 0.2)
\end{align}

However, now assume that we're a bit uncertain about our features. In particular, we're going to assume that every `x_i` value is not a point but a distribution instead. E.g.

$$ x_i \sim \mathcal{N}(\mu_i, \sigma_i). $$

### Using a distributional kernel to deal with uncertain inputs

Rather than using a variational method (see the GP Regression with Uncertian Inputs tutorial in the variational examples), if we explicitly know the type of uncertainty in our inputs we can pass that into our kernel.

More specifically, assuming Gaussian inputs, we will compute the symmetrized KL divergence between the Gaussian inputs.

```python
import math
import torch
import tqdm
import gpytorch
from matplotlib import pyplot as plt

%matplotlib inline
%load_ext autoreload
%autoreload 2
```

```python
# Training data is 100 points in [0,1] inclusive regularly spaced
train_x_mean = torch.linspace(0, 1, 20)
# We'll assume the variance shrinks the closer we get to 1
train_x_stdv = torch.linspace(0.03, 0.01, 20)

# True function is sin(2*pi*x) with Gaussian noise
train_y = torch.sin(train_x_mean * (2 * math.pi)) + torch.randn(train_x_mean.size()) * 0.2
```

To effectively pass in the training distributional data, we will need to stack the mean and log variances.

```python
train_x_distributional = torch.stack((train_x_mean, (train_x_stdv**2).log()), dim=1)
```

```python
f, ax = plt.subplots(1, 1, figsize=(8, 3))
ax.errorbar(train_x_mean, train_y, xerr=(train_x_stdv * 2), fmt="k*", label="Train Data")
ax.legend()
```

We train the hyperparameters of the resulting distributional GP via type-II gradient descent, as is standard in many settings. We could also do fully Bayesian inference.

```python
from gpytorch.models import ExactGP
from gpytorch.kernels import GaussianSymmetrizedKLKernel, ScaleKernel
from gpytorch.means import ConstantMean

class ExactGPModel(ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(ExactGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = ConstantMean()
        self.covar_module = ScaleKernel(GaussianSymmetrizedKLKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# initialize likelihood and model
likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = ExactGPModel(train_x_distributional, train_y, likelihood)
```

```python
# this is for running the notebook in our testing framework
import os
smoke_test = ('CI' in os.environ)
training_iter = 2 if smoke_test else 500

# Find optimal model hyperparameters
model.train()
likelihood.train()

# Use the adam optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.25)  # Includes GaussianLikelihood parameters

# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

for i in range(training_iter):
    # Zero gradients from previous iteration
    optimizer.zero_grad()
    # Output from model
    output = model(train_x_distributional)
    # Calc loss and backprop gradients
    loss = -mll(output, train_y)
    loss.backward()
    print('Iter %d/%d - Loss: %.3f   lengthscale: %.3f   noise: %.3f' % (
        i + 1, training_iter, loss.item(),
        model.covar_module.base_kernel.lengthscale.item(),
        model.likelihood.noise.item()
    ))
    optimizer.step()
```

    Iter 1/500 - Loss: 0.460   lengthscale: 58.355   noise: 0.047
    Iter 2/500 - Loss: 0.494   lengthscale: 58.605   noise: 0.037
    Iter 3/500 - Loss: 0.461   lengthscale: 58.855   noise: 0.044
    Iter 4/500 - Loss: 0.468   lengthscale: 59.105   noise: 0.053
    Iter 5/500 - Loss: 0.476   lengthscale: 59.353   noise: 0.057
    Iter 6/500 - Loss: 0.467   lengthscale: 59.601   noise: 0.057
    Iter 7/500 - Loss: 0.459   lengthscale: 59.849   noise: 0.053
    Iter 8/500 - Loss: 0.458   lengthscale: 60.096   noise: 0.049
    Iter 9/500 - Loss: 0.461   lengthscale: 60.344   noise: 0.045
    Iter 10/500 - Loss: 0.462   lengthscale: 60.592   noise: 0.042
    Iter 11/500 - Loss: 0.460   lengthscale: 60.840   noise: 0.041
    Iter 12/500 - Loss: 0.455   lengthscale: 61.089   noise: 0.042
    Iter 13/500 - Loss: 0.453   lengthscale: 61.338   noise: 0.044
    Iter 14/500 - Loss: 0.453   lengthscale: 61.586   noise: 0.047
    Iter 15/500 - Loss: 0.454   lengthscale: 61.834   noise: 0.049
    Iter 16/500 - Loss: 0.454   lengthscale: 62.081   noise: 0.051
    Iter 17/500 - Loss: 0.451   lengthscale: 62.327   noise: 0.052
    Iter 18/500 - Loss: 0.449   lengthscale: 62.572   noise: 0.051
    Iter 19/500 - Loss: 0.449   lengthscale: 62.817   noise: 0.049
    Iter 20/500 - Loss: 0.449   lengthscale: 63.061   noise: 0.047
    Iter 21/500 - Loss: 0.448   lengthscale: 63.306   noise: 0.045
    Iter 22/500 - Loss: 0.447   lengthscale: 63.550   noise: 0.044
    Iter 23/500 - Loss: 0.446   lengthscale: 63.794   noise: 0.044
    Iter 24/500 - Loss: 0.445   lengthscale: 64.038   noise: 0.044
    Iter 25/500 - Loss: 0.444   lengthscale: 64.282   noise: 0.045
    Iter 26/500 - Loss: 0.444   lengthscale: 64.525   noise: 0.046
    Iter 27/500 - Loss: 0.443   lengthscale: 64.767   noise: 0.048
    Iter 28/500 - Loss: 0.442   lengthscale: 65.009   noise: 0.049
    Iter 29/500 - Loss: 0.441   lengthscale: 65.250   noise: 0.049
    Iter 30/500 - Loss: 0.441   lengthscale: 65.490   noise: 0.049
    Iter 31/500 - Loss: 0.440   lengthscale: 65.730   noise: 0.048
    Iter 32/500 - Loss: 0.440   lengthscale: 65.970   noise: 0.047
    Iter 33/500 - Loss: 0.439   lengthscale: 66.209   noise: 0.045
    Iter 34/500 - Loss: 0.438   lengthscale: 66.448   noise: 0.045
    Iter 35/500 - Loss: 0.437   lengthscale: 66.686   noise: 0.045
    Iter 36/500 - Loss: 0.437   lengthscale: 66.924   noise: 0.045
    Iter 37/500 - Loss: 0.436   lengthscale: 67.161   noise: 0.046
    Iter 38/500 - Loss: 0.436   lengthscale: 67.398   noise: 0.047
    Iter 39/500 - Loss: 0.435   lengthscale: 67.634   noise: 0.047
    Iter 40/500 - Loss: 0.434   lengthscale: 67.870   noise: 0.048
    Iter 41/500 - Loss: 0.434   lengthscale: 68.105   noise: 0.047
    Iter 42/500 - Loss: 0.433   lengthscale: 68.339   noise: 0.047
    Iter 43/500 - Loss: 0.432   lengthscale: 68.573   noise: 0.046
    Iter 44/500 - Loss: 0.432   lengthscale: 68.806   noise: 0.046
    Iter 45/500 - Loss: 0.431   lengthscale: 69.039   noise: 0.045
    Iter 46/500 - Loss: 0.431   lengthscale: 69.272   noise: 0.045
    Iter 47/500 - Loss: 0.430   lengthscale: 69.504   noise: 0.045
    Iter 48/500 - Loss: 0.430   lengthscale: 69.735   noise: 0.046
    Iter 49/500 - Loss: 0.429   lengthscale: 69.966   noise: 0.046
    Iter 50/500 - Loss: 0.428   lengthscale: 70.196   noise: 0.047
    Iter 51/500 - Loss: 0.428   lengthscale: 70.426   noise: 0.047
    Iter 52/500 - Loss: 0.427   lengthscale: 70.655   noise: 0.047
    Iter 53/500 - Loss: 0.427   lengthscale: 70.884   noise: 0.046
    Iter 54/500 - Loss: 0.426   lengthscale: 71.112   noise: 0.046
    Iter 55/500 - Loss: 0.426   lengthscale: 71.339   noise: 0.045
    Iter 56/500 - Loss: 0.425   lengthscale: 71.566   noise: 0.045
    Iter 57/500 - Loss: 0.425   lengthscale: 71.793   noise: 0.045
    Iter 58/500 - Loss: 0.424   lengthscale: 72.019   noise: 0.046
    Iter 59/500 - Loss: 0.424   lengthscale: 72.245   noise: 0.046
    Iter 60/500 - Loss: 0.423   lengthscale: 72.470   noise: 0.046
    Iter 61/500 - Loss: 0.423   lengthscale: 72.694   noise: 0.046
    Iter 62/500 - Loss: 0.422   lengthscale: 72.918   noise: 0.046
    Iter 63/500 - Loss: 0.422   lengthscale: 73.141   noise: 0.046
    Iter 64/500 - Loss: 0.421   lengthscale: 73.364   noise: 0.046
    Iter 65/500 - Loss: 0.420   lengthscale: 73.587   noise: 0.045
    Iter 66/500 - Loss: 0.420   lengthscale: 73.808   noise: 0.045
    Iter 67/500 - Loss: 0.420   lengthscale: 74.030   noise: 0.045
    Iter 68/500 - Loss: 0.419   lengthscale: 74.251   noise: 0.045
    Iter 69/500 - Loss: 0.419   lengthscale: 74.471   noise: 0.046
    Iter 70/500 - Loss: 0.418   lengthscale: 74.691   noise: 0.046
    Iter 71/500 - Loss: 0.418   lengthscale: 74.910   noise: 0.046
    Iter 72/500 - Loss: 0.417   lengthscale: 75.129   noise: 0.046
    Iter 73/500 - Loss: 0.417   lengthscale: 75.347   noise: 0.045
    Iter 74/500 - Loss: 0.416   lengthscale: 75.565   noise: 0.045
    Iter 75/500 - Loss: 0.416   lengthscale: 75.782   noise: 0.045
    Iter 76/500 - Loss: 0.415   lengthscale: 75.999   noise: 0.045
    Iter 77/500 - Loss: 0.415   lengthscale: 76.216   noise: 0.045
    Iter 78/500 - Loss: 0.414   lengthscale: 76.432   noise: 0.045
    Iter 79/500 - Loss: 0.414   lengthscale: 76.647   noise: 0.045
    Iter 80/500 - Loss: 0.413   lengthscale: 76.862   noise: 0.045
    Iter 81/500 - Loss: 0.413   lengthscale: 77.076   noise: 0.045
    Iter 82/500 - Loss: 0.412   lengthscale: 77.290   noise: 0.045
    Iter 83/500 - Loss: 0.412   lengthscale: 77.504   noise: 0.045
    Iter 84/500 - Loss: 0.412   lengthscale: 77.717   noise: 0.045
    Iter 85/500 - Loss: 0.411   lengthscale: 77.929   noise: 0.045
    Iter 86/500 - Loss: 0.411   lengthscale: 78.141   noise: 0.045
    Iter 87/500 - Loss: 0.410   lengthscale: 78.353   noise: 0.045
    Iter 88/500 - Loss: 0.410   lengthscale: 78.564   noise: 0.045
    Iter 89/500 - Loss: 0.409   lengthscale: 78.774   noise: 0.045
    Iter 90/500 - Loss: 0.409   lengthscale: 78.984   noise: 0.045
    Iter 91/500 - Loss: 0.409   lengthscale: 79.194   noise: 0.045
    Iter 92/500 - Loss: 0.408   lengthscale: 79.403   noise: 0.045
    Iter 93/500 - Loss: 0.408   lengthscale: 79.612   noise: 0.045
    Iter 94/500 - Loss: 0.407   lengthscale: 79.820   noise: 0.045
    Iter 95/500 - Loss: 0.407   lengthscale: 80.028   noise: 0.045
    Iter 96/500 - Loss: 0.407   lengthscale: 80.236   noise: 0.045
    Iter 97/500 - Loss: 0.406   lengthscale: 80.443   noise: 0.045
    Iter 98/500 - Loss: 0.406   lengthscale: 80.649   noise: 0.045
    Iter 99/500 - Loss: 0.405   lengthscale: 80.855   noise: 0.045
    Iter 100/500 - Loss: 0.405   lengthscale: 81.061   noise: 0.045
    Iter 101/500 - Loss: 0.405   lengthscale: 81.266   noise: 0.045
    Iter 102/500 - Loss: 0.404   lengthscale: 81.471   noise: 0.045
    Iter 103/500 - Loss: 0.404   lengthscale: 81.675   noise: 0.045
    Iter 104/500 - Loss: 0.403   lengthscale: 81.879   noise: 0.045
    Iter 105/500 - Loss: 0.403   lengthscale: 82.082   noise: 0.045
    Iter 106/500 - Loss: 0.403   lengthscale: 82.285   noise: 0.045
    Iter 107/500 - Loss: 0.402   lengthscale: 82.488   noise: 0.045
    Iter 108/500 - Loss: 0.402   lengthscale: 82.690   noise: 0.045
    Iter 109/500 - Loss: 0.401   lengthscale: 82.892   noise: 0.045
    Iter 110/500 - Loss: 0.401   lengthscale: 83.093   noise: 0.045
    Iter 111/500 - Loss: 0.401   lengthscale: 83.294   noise: 0.045
    Iter 112/500 - Loss: 0.400   lengthscale: 83.494   noise: 0.045
    Iter 113/500 - Loss: 0.400   lengthscale: 83.694   noise: 0.045
    Iter 114/500 - Loss: 0.400   lengthscale: 83.894   noise: 0.045
    Iter 115/500 - Loss: 0.399   lengthscale: 84.093   noise: 0.045
    Iter 116/500 - Loss: 0.399   lengthscale: 84.292   noise: 0.045
    Iter 117/500 - Loss: 0.399   lengthscale: 84.490   noise: 0.045
    Iter 118/500 - Loss: 0.398   lengthscale: 84.688   noise: 0.045
    Iter 119/500 - Loss: 0.398   lengthscale: 84.886   noise: 0.045
    Iter 120/500 - Loss: 0.397   lengthscale: 85.083   noise: 0.045
    Iter 121/500 - Loss: 0.397   lengthscale: 85.280   noise: 0.045
    Iter 122/500 - Loss: 0.397   lengthscale: 85.476   noise: 0.045
    Iter 123/500 - Loss: 0.396   lengthscale: 85.672   noise: 0.045
    Iter 124/500 - Loss: 0.396   lengthscale: 85.867   noise: 0.044
    Iter 125/500 - Loss: 0.396   lengthscale: 86.062   noise: 0.044
    Iter 126/500 - Loss: 0.395   lengthscale: 86.257   noise: 0.044
    Iter 127/500 - Loss: 0.395   lengthscale: 86.451   noise: 0.044
    Iter 128/500 - Loss: 0.395   lengthscale: 86.645   noise: 0.044
    Iter 129/500 - Loss: 0.394   lengthscale: 86.839   noise: 0.044
    Iter 130/500 - Loss: 0.394   lengthscale: 87.032   noise: 0.044
    Iter 131/500 - Loss: 0.394   lengthscale: 87.225   noise: 0.044
    Iter 132/500 - Loss: 0.393   lengthscale: 87.417   noise: 0.044
    Iter 133/500 - Loss: 0.393   lengthscale: 87.609   noise: 0.044
    Iter 134/500 - Loss: 0.393   lengthscale: 87.801   noise: 0.044
    Iter 135/500 - Loss: 0.392   lengthscale: 87.992   noise: 0.044
    Iter 136/500 - Loss: 0.392   lengthscale: 88.183   noise: 0.044
    Iter 137/500 - Loss: 0.392   lengthscale: 88.373   noise: 0.044
    Iter 138/500 - Loss: 0.391   lengthscale: 88.564   noise: 0.044
    Iter 139/500 - Loss: 0.391   lengthscale: 88.753   noise: 0.044
    Iter 140/500 - Loss: 0.391   lengthscale: 88.943   noise: 0.044
    Iter 141/500 - Loss: 0.391   lengthscale: 89.132   noise: 0.044
    Iter 142/500 - Loss: 0.390   lengthscale: 89.320   noise: 0.044
    Iter 143/500 - Loss: 0.390   lengthscale: 89.508   noise: 0.044
    Iter 144/500 - Loss: 0.390   lengthscale: 89.696   noise: 0.044
    Iter 145/500 - Loss: 0.389   lengthscale: 89.884   noise: 0.044
    Iter 146/500 - Loss: 0.389   lengthscale: 90.071   noise: 0.044
    Iter 147/500 - Loss: 0.389   lengthscale: 90.258   noise: 0.044
    Iter 148/500 - Loss: 0.388   lengthscale: 90.444   noise: 0.044
    Iter 149/500 - Loss: 0.388   lengthscale: 90.630   noise: 0.044
    Iter 150/500 - Loss: 0.388   lengthscale: 90.816   noise: 0.044
    Iter 151/500 - Loss: 0.388   lengthscale: 91.001   noise: 0.044
    Iter 152/500 - Loss: 0.387   lengthscale: 91.186   noise: 0.044
    Iter 153/500 - Loss: 0.387   lengthscale: 91.371   noise: 0.044
    Iter 154/500 - Loss: 0.387   lengthscale: 91.555   noise: 0.044
    Iter 155/500 - Loss: 0.386   lengthscale: 91.739   noise: 0.044
    Iter 156/500 - Loss: 0.386   lengthscale: 91.923   noise: 0.044
    Iter 157/500 - Loss: 0.386   lengthscale: 92.106   noise: 0.044
    Iter 158/500 - Loss: 0.385   lengthscale: 92.289   noise: 0.044
    Iter 159/500 - Loss: 0.385   lengthscale: 92.471   noise: 0.044
    Iter 160/500 - Loss: 0.385   lengthscale: 92.654   noise: 0.044
    Iter 161/500 - Loss: 0.385   lengthscale: 92.836   noise: 0.044
    Iter 162/500 - Loss: 0.384   lengthscale: 93.017   noise: 0.044
    Iter 163/500 - Loss: 0.384   lengthscale: 93.198   noise: 0.044
    Iter 164/500 - Loss: 0.384   lengthscale: 93.379   noise: 0.044
    Iter 165/500 - Loss: 0.384   lengthscale: 93.560   noise: 0.044
    Iter 166/500 - Loss: 0.383   lengthscale: 93.740   noise: 0.044
    Iter 167/500 - Loss: 0.383   lengthscale: 93.920   noise: 0.044
    Iter 168/500 - Loss: 0.383   lengthscale: 94.099   noise: 0.044
    Iter 169/500 - Loss: 0.382   lengthscale: 94.278   noise: 0.044
    Iter 170/500 - Loss: 0.382   lengthscale: 94.457   noise: 0.044
    Iter 171/500 - Loss: 0.382   lengthscale: 94.636   noise: 0.044
    Iter 172/500 - Loss: 0.382   lengthscale: 94.814   noise: 0.044
    Iter 173/500 - Loss: 0.381   lengthscale: 94.992   noise: 0.044
    Iter 174/500 - Loss: 0.381   lengthscale: 95.169   noise: 0.044
    Iter 175/500 - Loss: 0.381   lengthscale: 95.346   noise: 0.044
    Iter 176/500 - Loss: 0.381   lengthscale: 95.523   noise: 0.044
    Iter 177/500 - Loss: 0.380   lengthscale: 95.700   noise: 0.044
    Iter 178/500 - Loss: 0.380   lengthscale: 95.876   noise: 0.044
    Iter 179/500 - Loss: 0.380   lengthscale: 96.052   noise: 0.044
    Iter 180/500 - Loss: 0.380   lengthscale: 96.228   noise: 0.044
    Iter 181/500 - Loss: 0.379   lengthscale: 96.403   noise: 0.044
    Iter 182/500 - Loss: 0.379   lengthscale: 96.578   noise: 0.044
    Iter 183/500 - Loss: 0.379   lengthscale: 96.752   noise: 0.044
    Iter 184/500 - Loss: 0.379   lengthscale: 96.927   noise: 0.044
    Iter 185/500 - Loss: 0.378   lengthscale: 97.101   noise: 0.044
    Iter 186/500 - Loss: 0.378   lengthscale: 97.275   noise: 0.044
    Iter 187/500 - Loss: 0.378   lengthscale: 97.448   noise: 0.044
    Iter 188/500 - Loss: 0.378   lengthscale: 97.621   noise: 0.044
    Iter 189/500 - Loss: 0.377   lengthscale: 97.794   noise: 0.044
    Iter 190/500 - Loss: 0.377   lengthscale: 97.966   noise: 0.044
    Iter 191/500 - Loss: 0.377   lengthscale: 98.139   noise: 0.044
    Iter 192/500 - Loss: 0.377   lengthscale: 98.310   noise: 0.044
    Iter 193/500 - Loss: 0.376   lengthscale: 98.482   noise: 0.044
    Iter 194/500 - Loss: 0.376   lengthscale: 98.653   noise: 0.044
    Iter 195/500 - Loss: 0.376   lengthscale: 98.824   noise: 0.044
    Iter 196/500 - Loss: 0.376   lengthscale: 98.995   noise: 0.044
    Iter 197/500 - Loss: 0.375   lengthscale: 99.165   noise: 0.044
    Iter 198/500 - Loss: 0.375   lengthscale: 99.336   noise: 0.044
    Iter 199/500 - Loss: 0.375   lengthscale: 99.505   noise: 0.043
    Iter 200/500 - Loss: 0.375   lengthscale: 99.675   noise: 0.043
    Iter 201/500 - Loss: 0.375   lengthscale: 99.844   noise: 0.043
    Iter 202/500 - Loss: 0.374   lengthscale: 100.013   noise: 0.043
    Iter 203/500 - Loss: 0.374   lengthscale: 100.182   noise: 0.043
    Iter 204/500 - Loss: 0.374   lengthscale: 100.350   noise: 0.043
    Iter 205/500 - Loss: 0.374   lengthscale: 100.518   noise: 0.043
    Iter 206/500 - Loss: 0.373   lengthscale: 100.686   noise: 0.043
    Iter 207/500 - Loss: 0.373   lengthscale: 100.853   noise: 0.043
    Iter 208/500 - Loss: 0.373   lengthscale: 101.020   noise: 0.043
    Iter 209/500 - Loss: 0.373   lengthscale: 101.187   noise: 0.043
    Iter 210/500 - Loss: 0.373   lengthscale: 101.354   noise: 0.043
    Iter 211/500 - Loss: 0.372   lengthscale: 101.520   noise: 0.043
    Iter 212/500 - Loss: 0.372   lengthscale: 101.686   noise: 0.043
    Iter 213/500 - Loss: 0.372   lengthscale: 101.852   noise: 0.043
    Iter 214/500 - Loss: 0.372   lengthscale: 102.018   noise: 0.043
    Iter 215/500 - Loss: 0.372   lengthscale: 102.183   noise: 0.043
    Iter 216/500 - Loss: 0.371   lengthscale: 102.348   noise: 0.043
    Iter 217/500 - Loss: 0.371   lengthscale: 102.513   noise: 0.043
    Iter 218/500 - Loss: 0.371   lengthscale: 102.677   noise: 0.043
    Iter 219/500 - Loss: 0.371   lengthscale: 102.841   noise: 0.043
    Iter 220/500 - Loss: 0.370   lengthscale: 103.005   noise: 0.043
    Iter 221/500 - Loss: 0.370   lengthscale: 103.169   noise: 0.043
    Iter 222/500 - Loss: 0.370   lengthscale: 103.332   noise: 0.043
    Iter 223/500 - Loss: 0.370   lengthscale: 103.495   noise: 0.043
    Iter 224/500 - Loss: 0.370   lengthscale: 103.658   noise: 0.043
    Iter 225/500 - Loss: 0.369   lengthscale: 103.820   noise: 0.043
    Iter 226/500 - Loss: 0.369   lengthscale: 103.982   noise: 0.043
    Iter 227/500 - Loss: 0.369   lengthscale: 104.144   noise: 0.043
    Iter 228/500 - Loss: 0.369   lengthscale: 104.306   noise: 0.043
    Iter 229/500 - Loss: 0.369   lengthscale: 104.468   noise: 0.043
    Iter 230/500 - Loss: 0.368   lengthscale: 104.629   noise: 0.043
    Iter 231/500 - Loss: 0.368   lengthscale: 104.790   noise: 0.043
    Iter 232/500 - Loss: 0.368   lengthscale: 104.950   noise: 0.043
    Iter 233/500 - Loss: 0.368   lengthscale: 105.111   noise: 0.043
    Iter 234/500 - Loss: 0.368   lengthscale: 105.271   noise: 0.043
    Iter 235/500 - Loss: 0.367   lengthscale: 105.431   noise: 0.043
    Iter 236/500 - Loss: 0.367   lengthscale: 105.590   noise: 0.043
    Iter 237/500 - Loss: 0.367   lengthscale: 105.750   noise: 0.043
    Iter 238/500 - Loss: 0.367   lengthscale: 105.909   noise: 0.043
    Iter 239/500 - Loss: 0.367   lengthscale: 106.068   noise: 0.043
    Iter 240/500 - Loss: 0.367   lengthscale: 106.226   noise: 0.043
    Iter 241/500 - Loss: 0.366   lengthscale: 106.385   noise: 0.043
    Iter 242/500 - Loss: 0.366   lengthscale: 106.543   noise: 0.043
    Iter 243/500 - Loss: 0.366   lengthscale: 106.701   noise: 0.043
    Iter 244/500 - Loss: 0.366   lengthscale: 106.858   noise: 0.043
    Iter 245/500 - Loss: 0.366   lengthscale: 107.016   noise: 0.043
    Iter 246/500 - Loss: 0.365   lengthscale: 107.173   noise: 0.043
    Iter 247/500 - Loss: 0.365   lengthscale: 107.330   noise: 0.043
    Iter 248/500 - Loss: 0.365   lengthscale: 107.486   noise: 0.043
    Iter 249/500 - Loss: 0.365   lengthscale: 107.643   noise: 0.043
    Iter 250/500 - Loss: 0.365   lengthscale: 107.799   noise: 0.043
    Iter 251/500 - Loss: 0.364   lengthscale: 107.955   noise: 0.043
    Iter 252/500 - Loss: 0.364   lengthscale: 108.110   noise: 0.043
    Iter 253/500 - Loss: 0.364   lengthscale: 108.266   noise: 0.043
    Iter 254/500 - Loss: 0.364   lengthscale: 108.421   noise: 0.043
    Iter 255/500 - Loss: 0.364   lengthscale: 108.576   noise: 0.043
    Iter 256/500 - Loss: 0.364   lengthscale: 108.731   noise: 0.043
    Iter 257/500 - Loss: 0.363   lengthscale: 108.885   noise: 0.043
    Iter 258/500 - Loss: 0.363   lengthscale: 109.039   noise: 0.043
    Iter 259/500 - Loss: 0.363   lengthscale: 109.193   noise: 0.043
    Iter 260/500 - Loss: 0.363   lengthscale: 109.347   noise: 0.043
    Iter 261/500 - Loss: 0.363   lengthscale: 109.500   noise: 0.043
    Iter 262/500 - Loss: 0.363   lengthscale: 109.654   noise: 0.043
    Iter 263/500 - Loss: 0.362   lengthscale: 109.807   noise: 0.043
    Iter 264/500 - Loss: 0.362   lengthscale: 109.960   noise: 0.043
    Iter 265/500 - Loss: 0.362   lengthscale: 110.112   noise: 0.043
    Iter 266/500 - Loss: 0.362   lengthscale: 110.265   noise: 0.043
    Iter 267/500 - Loss: 0.362   lengthscale: 110.417   noise: 0.043
    Iter 268/500 - Loss: 0.362   lengthscale: 110.569   noise: 0.043
    Iter 269/500 - Loss: 0.361   lengthscale: 110.720   noise: 0.043
    Iter 270/500 - Loss: 0.361   lengthscale: 110.872   noise: 0.043
    Iter 271/500 - Loss: 0.361   lengthscale: 111.023   noise: 0.043
    Iter 272/500 - Loss: 0.361   lengthscale: 111.174   noise: 0.043
    Iter 273/500 - Loss: 0.361   lengthscale: 111.325   noise: 0.043
    Iter 274/500 - Loss: 0.361   lengthscale: 111.475   noise: 0.043
    Iter 275/500 - Loss: 0.360   lengthscale: 111.625   noise: 0.043
    Iter 276/500 - Loss: 0.360   lengthscale: 111.775   noise: 0.043
    Iter 277/500 - Loss: 0.360   lengthscale: 111.925   noise: 0.043
    Iter 278/500 - Loss: 0.360   lengthscale: 112.075   noise: 0.043
    Iter 279/500 - Loss: 0.360   lengthscale: 112.224   noise: 0.043
    Iter 280/500 - Loss: 0.360   lengthscale: 112.374   noise: 0.043
    Iter 281/500 - Loss: 0.359   lengthscale: 112.523   noise: 0.043
    Iter 282/500 - Loss: 0.359   lengthscale: 112.671   noise: 0.043
    Iter 283/500 - Loss: 0.359   lengthscale: 112.820   noise: 0.043
    Iter 284/500 - Loss: 0.359   lengthscale: 112.968   noise: 0.043
    Iter 285/500 - Loss: 0.359   lengthscale: 113.116   noise: 0.043
    Iter 286/500 - Loss: 0.359   lengthscale: 113.264   noise: 0.043
    Iter 287/500 - Loss: 0.359   lengthscale: 113.412   noise: 0.043
    Iter 288/500 - Loss: 0.358   lengthscale: 113.559   noise: 0.043
    Iter 289/500 - Loss: 0.358   lengthscale: 113.707   noise: 0.043
    Iter 290/500 - Loss: 0.358   lengthscale: 113.854   noise: 0.043
    Iter 291/500 - Loss: 0.358   lengthscale: 114.001   noise: 0.043
    Iter 292/500 - Loss: 0.358   lengthscale: 114.147   noise: 0.043
    Iter 293/500 - Loss: 0.358   lengthscale: 114.294   noise: 0.043
    Iter 294/500 - Loss: 0.357   lengthscale: 114.440   noise: 0.043
    Iter 295/500 - Loss: 0.357   lengthscale: 114.586   noise: 0.043
    Iter 296/500 - Loss: 0.357   lengthscale: 114.732   noise: 0.043
    Iter 297/500 - Loss: 0.357   lengthscale: 114.877   noise: 0.043
    Iter 298/500 - Loss: 0.357   lengthscale: 115.023   noise: 0.043
    Iter 299/500 - Loss: 0.357   lengthscale: 115.168   noise: 0.043
    Iter 300/500 - Loss: 0.357   lengthscale: 115.313   noise: 0.043
    Iter 301/500 - Loss: 0.356   lengthscale: 115.458   noise: 0.043
    Iter 302/500 - Loss: 0.356   lengthscale: 115.602   noise: 0.043
    Iter 303/500 - Loss: 0.356   lengthscale: 115.747   noise: 0.043
    Iter 304/500 - Loss: 0.356   lengthscale: 115.891   noise: 0.043
    Iter 305/500 - Loss: 0.356   lengthscale: 116.035   noise: 0.043
    Iter 306/500 - Loss: 0.356   lengthscale: 116.179   noise: 0.043
    Iter 307/500 - Loss: 0.356   lengthscale: 116.322   noise: 0.043
    Iter 308/500 - Loss: 0.355   lengthscale: 116.465   noise: 0.043
    Iter 309/500 - Loss: 0.355   lengthscale: 116.609   noise: 0.043
    Iter 310/500 - Loss: 0.355   lengthscale: 116.752   noise: 0.043
    Iter 311/500 - Loss: 0.355   lengthscale: 116.894   noise: 0.043
    Iter 312/500 - Loss: 0.355   lengthscale: 117.037   noise: 0.043
    Iter 313/500 - Loss: 0.355   lengthscale: 117.179   noise: 0.043
    Iter 314/500 - Loss: 0.355   lengthscale: 117.322   noise: 0.043
    Iter 315/500 - Loss: 0.354   lengthscale: 117.464   noise: 0.043
    Iter 316/500 - Loss: 0.354   lengthscale: 117.605   noise: 0.043
    Iter 317/500 - Loss: 0.354   lengthscale: 117.747   noise: 0.043
    Iter 318/500 - Loss: 0.354   lengthscale: 117.888   noise: 0.043
    Iter 319/500 - Loss: 0.354   lengthscale: 118.030   noise: 0.043
    Iter 320/500 - Loss: 0.354   lengthscale: 118.171   noise: 0.043
    Iter 321/500 - Loss: 0.354   lengthscale: 118.312   noise: 0.043
    Iter 322/500 - Loss: 0.354   lengthscale: 118.452   noise: 0.043
    Iter 323/500 - Loss: 0.353   lengthscale: 118.593   noise: 0.043
    Iter 324/500 - Loss: 0.353   lengthscale: 118.733   noise: 0.043
    Iter 325/500 - Loss: 0.353   lengthscale: 118.873   noise: 0.043
    Iter 326/500 - Loss: 0.353   lengthscale: 119.013   noise: 0.043
    Iter 327/500 - Loss: 0.353   lengthscale: 119.153   noise: 0.043
    Iter 328/500 - Loss: 0.353   lengthscale: 119.292   noise: 0.043
    Iter 329/500 - Loss: 0.353   lengthscale: 119.432   noise: 0.043
    Iter 330/500 - Loss: 0.352   lengthscale: 119.571   noise: 0.043
    Iter 331/500 - Loss: 0.352   lengthscale: 119.710   noise: 0.043
    Iter 332/500 - Loss: 0.352   lengthscale: 119.849   noise: 0.043
    Iter 333/500 - Loss: 0.352   lengthscale: 119.987   noise: 0.043
    Iter 334/500 - Loss: 0.352   lengthscale: 120.126   noise: 0.043
    Iter 335/500 - Loss: 0.352   lengthscale: 120.264   noise: 0.043
    Iter 336/500 - Loss: 0.352   lengthscale: 120.402   noise: 0.043
    Iter 337/500 - Loss: 0.352   lengthscale: 120.540   noise: 0.043
    Iter 338/500 - Loss: 0.351   lengthscale: 120.678   noise: 0.042
    Iter 339/500 - Loss: 0.351   lengthscale: 120.815   noise: 0.042
    Iter 340/500 - Loss: 0.351   lengthscale: 120.953   noise: 0.042
    Iter 341/500 - Loss: 0.351   lengthscale: 121.090   noise: 0.042
    Iter 342/500 - Loss: 0.351   lengthscale: 121.227   noise: 0.042
    Iter 343/500 - Loss: 0.351   lengthscale: 121.364   noise: 0.042
    Iter 344/500 - Loss: 0.351   lengthscale: 121.500   noise: 0.042
    Iter 345/500 - Loss: 0.351   lengthscale: 121.637   noise: 0.042
    Iter 346/500 - Loss: 0.350   lengthscale: 121.773   noise: 0.042
    Iter 347/500 - Loss: 0.350   lengthscale: 121.909   noise: 0.042
    Iter 348/500 - Loss: 0.350   lengthscale: 122.045   noise: 0.042
    Iter 349/500 - Loss: 0.350   lengthscale: 122.181   noise: 0.042
    Iter 350/500 - Loss: 0.350   lengthscale: 122.317   noise: 0.042
    Iter 351/500 - Loss: 0.350   lengthscale: 122.452   noise: 0.042
    Iter 352/500 - Loss: 0.350   lengthscale: 122.588   noise: 0.042
    Iter 353/500 - Loss: 0.350   lengthscale: 122.723   noise: 0.042
    Iter 354/500 - Loss: 0.350   lengthscale: 122.858   noise: 0.042
    Iter 355/500 - Loss: 0.349   lengthscale: 122.992   noise: 0.042
    Iter 356/500 - Loss: 0.349   lengthscale: 123.127   noise: 0.042
    Iter 357/500 - Loss: 0.349   lengthscale: 123.261   noise: 0.042
    Iter 358/500 - Loss: 0.349   lengthscale: 123.396   noise: 0.042
    Iter 359/500 - Loss: 0.349   lengthscale: 123.530   noise: 0.042
    Iter 360/500 - Loss: 0.349   lengthscale: 123.664   noise: 0.042
    Iter 361/500 - Loss: 0.349   lengthscale: 123.798   noise: 0.042
    Iter 362/500 - Loss: 0.349   lengthscale: 123.931   noise: 0.042
    Iter 363/500 - Loss: 0.348   lengthscale: 124.065   noise: 0.042
    Iter 364/500 - Loss: 0.348   lengthscale: 124.198   noise: 0.042
    Iter 365/500 - Loss: 0.348   lengthscale: 124.331   noise: 0.042
    Iter 366/500 - Loss: 0.348   lengthscale: 124.464   noise: 0.042
    Iter 367/500 - Loss: 0.348   lengthscale: 124.597   noise: 0.042
    Iter 368/500 - Loss: 0.348   lengthscale: 124.729   noise: 0.042
    Iter 369/500 - Loss: 0.348   lengthscale: 124.862   noise: 0.042
    Iter 370/500 - Loss: 0.348   lengthscale: 124.994   noise: 0.042
    Iter 371/500 - Loss: 0.348   lengthscale: 125.126   noise: 0.042
    Iter 372/500 - Loss: 0.347   lengthscale: 125.258   noise: 0.042
    Iter 373/500 - Loss: 0.347   lengthscale: 125.390   noise: 0.042
    Iter 374/500 - Loss: 0.347   lengthscale: 125.522   noise: 0.042
    Iter 375/500 - Loss: 0.347   lengthscale: 125.653   noise: 0.042
    Iter 376/500 - Loss: 0.347   lengthscale: 125.785   noise: 0.042
    Iter 377/500 - Loss: 0.347   lengthscale: 125.916   noise: 0.042
    Iter 378/500 - Loss: 0.347   lengthscale: 126.047   noise: 0.042
    Iter 379/500 - Loss: 0.347   lengthscale: 126.178   noise: 0.042
    Iter 380/500 - Loss: 0.347   lengthscale: 126.309   noise: 0.042
    Iter 381/500 - Loss: 0.346   lengthscale: 126.439   noise: 0.042
    Iter 382/500 - Loss: 0.346   lengthscale: 126.570   noise: 0.042
    Iter 383/500 - Loss: 0.346   lengthscale: 126.700   noise: 0.042
    Iter 384/500 - Loss: 0.346   lengthscale: 126.830   noise: 0.042
    Iter 385/500 - Loss: 0.346   lengthscale: 126.960   noise: 0.042
    Iter 386/500 - Loss: 0.346   lengthscale: 127.090   noise: 0.042
    Iter 387/500 - Loss: 0.346   lengthscale: 127.220   noise: 0.042
    Iter 388/500 - Loss: 0.346   lengthscale: 127.349   noise: 0.042
    Iter 389/500 - Loss: 0.346   lengthscale: 127.479   noise: 0.042
    Iter 390/500 - Loss: 0.346   lengthscale: 127.608   noise: 0.042
    Iter 391/500 - Loss: 0.345   lengthscale: 127.737   noise: 0.042
    Iter 392/500 - Loss: 0.345   lengthscale: 127.866   noise: 0.042
    Iter 393/500 - Loss: 0.345   lengthscale: 127.995   noise: 0.042
    Iter 394/500 - Loss: 0.345   lengthscale: 128.123   noise: 0.042
    Iter 395/500 - Loss: 0.345   lengthscale: 128.252   noise: 0.042
    Iter 396/500 - Loss: 0.345   lengthscale: 128.380   noise: 0.042
    Iter 397/500 - Loss: 0.345   lengthscale: 128.508   noise: 0.042
    Iter 398/500 - Loss: 0.345   lengthscale: 128.636   noise: 0.042
    Iter 399/500 - Loss: 0.345   lengthscale: 128.764   noise: 0.042
    Iter 400/500 - Loss: 0.345   lengthscale: 128.892   noise: 0.042
    Iter 401/500 - Loss: 0.344   lengthscale: 129.019   noise: 0.042
    Iter 402/500 - Loss: 0.344   lengthscale: 129.147   noise: 0.042
    Iter 403/500 - Loss: 0.344   lengthscale: 129.274   noise: 0.042
    Iter 404/500 - Loss: 0.344   lengthscale: 129.401   noise: 0.042
    Iter 405/500 - Loss: 0.344   lengthscale: 129.528   noise: 0.042
    Iter 406/500 - Loss: 0.344   lengthscale: 129.655   noise: 0.042
    Iter 407/500 - Loss: 0.344   lengthscale: 129.782   noise: 0.042
    Iter 408/500 - Loss: 0.344   lengthscale: 129.909   noise: 0.042
    Iter 409/500 - Loss: 0.344   lengthscale: 130.035   noise: 0.042
    Iter 410/500 - Loss: 0.344   lengthscale: 130.161   noise: 0.042
    Iter 411/500 - Loss: 0.343   lengthscale: 130.288   noise: 0.042
    Iter 412/500 - Loss: 0.343   lengthscale: 130.414   noise: 0.042
    Iter 413/500 - Loss: 0.343   lengthscale: 130.540   noise: 0.042
    Iter 414/500 - Loss: 0.343   lengthscale: 130.665   noise: 0.042
    Iter 415/500 - Loss: 0.343   lengthscale: 130.791   noise: 0.042
    Iter 416/500 - Loss: 0.343   lengthscale: 130.917   noise: 0.042
    Iter 417/500 - Loss: 0.343   lengthscale: 131.042   noise: 0.042
    Iter 418/500 - Loss: 0.343   lengthscale: 131.167   noise: 0.042
    Iter 419/500 - Loss: 0.343   lengthscale: 131.292   noise: 0.042
    Iter 420/500 - Loss: 0.343   lengthscale: 131.417   noise: 0.042
    Iter 421/500 - Loss: 0.342   lengthscale: 131.542   noise: 0.042
    Iter 422/500 - Loss: 0.342   lengthscale: 131.667   noise: 0.042
    Iter 423/500 - Loss: 0.342   lengthscale: 131.791   noise: 0.042
    Iter 424/500 - Loss: 0.342   lengthscale: 131.916   noise: 0.042
    Iter 425/500 - Loss: 0.342   lengthscale: 132.040   noise: 0.042
    Iter 426/500 - Loss: 0.342   lengthscale: 132.164   noise: 0.042
    Iter 427/500 - Loss: 0.342   lengthscale: 132.288   noise: 0.042
    Iter 428/500 - Loss: 0.342   lengthscale: 132.412   noise: 0.042
    Iter 429/500 - Loss: 0.342   lengthscale: 132.536   noise: 0.042
    Iter 430/500 - Loss: 0.342   lengthscale: 132.659   noise: 0.042
    Iter 431/500 - Loss: 0.342   lengthscale: 132.783   noise: 0.042
    Iter 432/500 - Loss: 0.341   lengthscale: 132.906   noise: 0.042
    Iter 433/500 - Loss: 0.341   lengthscale: 133.029   noise: 0.042
    Iter 434/500 - Loss: 0.341   lengthscale: 133.152   noise: 0.042
    Iter 435/500 - Loss: 0.341   lengthscale: 133.275   noise: 0.042
    Iter 436/500 - Loss: 0.341   lengthscale: 133.398   noise: 0.042
    Iter 437/500 - Loss: 0.341   lengthscale: 133.521   noise: 0.042
    Iter 438/500 - Loss: 0.341   lengthscale: 133.644   noise: 0.042
    Iter 439/500 - Loss: 0.341   lengthscale: 133.766   noise: 0.042
    Iter 440/500 - Loss: 0.341   lengthscale: 133.888   noise: 0.042
    Iter 441/500 - Loss: 0.341   lengthscale: 134.011   noise: 0.042
    Iter 442/500 - Loss: 0.341   lengthscale: 134.133   noise: 0.042
    Iter 443/500 - Loss: 0.341   lengthscale: 134.255   noise: 0.042
    Iter 444/500 - Loss: 0.340   lengthscale: 134.376   noise: 0.042
    Iter 445/500 - Loss: 0.340   lengthscale: 134.498   noise: 0.042
    Iter 446/500 - Loss: 0.340   lengthscale: 134.620   noise: 0.042
    Iter 447/500 - Loss: 0.340   lengthscale: 134.741   noise: 0.042
    Iter 448/500 - Loss: 0.340   lengthscale: 134.862   noise: 0.042
    Iter 449/500 - Loss: 0.340   lengthscale: 134.984   noise: 0.042
    Iter 450/500 - Loss: 0.340   lengthscale: 135.105   noise: 0.042
    Iter 451/500 - Loss: 0.340   lengthscale: 135.226   noise: 0.042
    Iter 452/500 - Loss: 0.340   lengthscale: 135.346   noise: 0.042
    Iter 453/500 - Loss: 0.340   lengthscale: 135.467   noise: 0.042
    Iter 454/500 - Loss: 0.340   lengthscale: 135.588   noise: 0.042
    Iter 455/500 - Loss: 0.339   lengthscale: 135.708   noise: 0.042
    Iter 456/500 - Loss: 0.339   lengthscale: 135.829   noise: 0.042
    Iter 457/500 - Loss: 0.339   lengthscale: 135.949   noise: 0.042
    Iter 458/500 - Loss: 0.339   lengthscale: 136.069   noise: 0.042
    Iter 459/500 - Loss: 0.339   lengthscale: 136.189   noise: 0.042
    Iter 460/500 - Loss: 0.339   lengthscale: 136.309   noise: 0.042
    Iter 461/500 - Loss: 0.339   lengthscale: 136.428   noise: 0.042
    Iter 462/500 - Loss: 0.339   lengthscale: 136.548   noise: 0.042
    Iter 463/500 - Loss: 0.339   lengthscale: 136.668   noise: 0.042
    Iter 464/500 - Loss: 0.339   lengthscale: 136.787   noise: 0.042
    Iter 465/500 - Loss: 0.339   lengthscale: 136.906   noise: 0.042
    Iter 466/500 - Loss: 0.339   lengthscale: 137.025   noise: 0.042
    Iter 467/500 - Loss: 0.338   lengthscale: 137.144   noise: 0.042
    Iter 468/500 - Loss: 0.338   lengthscale: 137.263   noise: 0.042
    Iter 469/500 - Loss: 0.338   lengthscale: 137.382   noise: 0.042
    Iter 470/500 - Loss: 0.338   lengthscale: 137.501   noise: 0.042
    Iter 471/500 - Loss: 0.338   lengthscale: 137.619   noise: 0.042
    Iter 472/500 - Loss: 0.338   lengthscale: 137.738   noise: 0.042
    Iter 473/500 - Loss: 0.338   lengthscale: 137.856   noise: 0.042
    Iter 474/500 - Loss: 0.338   lengthscale: 137.974   noise: 0.042
    Iter 475/500 - Loss: 0.338   lengthscale: 138.093   noise: 0.042
    Iter 476/500 - Loss: 0.338   lengthscale: 138.211   noise: 0.042
    Iter 477/500 - Loss: 0.338   lengthscale: 138.328   noise: 0.042
    Iter 478/500 - Loss: 0.338   lengthscale: 138.446   noise: 0.042
    Iter 479/500 - Loss: 0.338   lengthscale: 138.564   noise: 0.042
    Iter 480/500 - Loss: 0.337   lengthscale: 138.681   noise: 0.042
    Iter 481/500 - Loss: 0.337   lengthscale: 138.799   noise: 0.042
    Iter 482/500 - Loss: 0.337   lengthscale: 138.916   noise: 0.042
    Iter 483/500 - Loss: 0.337   lengthscale: 139.033   noise: 0.042
    Iter 484/500 - Loss: 0.337   lengthscale: 139.151   noise: 0.042
    Iter 485/500 - Loss: 0.337   lengthscale: 139.268   noise: 0.042
    Iter 486/500 - Loss: 0.337   lengthscale: 139.384   noise: 0.042
    Iter 487/500 - Loss: 0.337   lengthscale: 139.501   noise: 0.042
    Iter 488/500 - Loss: 0.337   lengthscale: 139.618   noise: 0.042
    Iter 489/500 - Loss: 0.337   lengthscale: 139.734   noise: 0.042
    Iter 490/500 - Loss: 0.337   lengthscale: 139.851   noise: 0.042
    Iter 491/500 - Loss: 0.337   lengthscale: 139.967   noise: 0.042
    Iter 492/500 - Loss: 0.337   lengthscale: 140.083   noise: 0.042
    Iter 493/500 - Loss: 0.336   lengthscale: 140.200   noise: 0.042
    Iter 494/500 - Loss: 0.336   lengthscale: 140.316   noise: 0.042
    Iter 495/500 - Loss: 0.336   lengthscale: 140.432   noise: 0.042
    Iter 496/500 - Loss: 0.336   lengthscale: 140.547   noise: 0.042
    Iter 497/500 - Loss: 0.336   lengthscale: 140.663   noise: 0.042
    Iter 498/500 - Loss: 0.336   lengthscale: 140.779   noise: 0.042
    Iter 499/500 - Loss: 0.336   lengthscale: 140.894   noise: 0.042
    Iter 500/500 - Loss: 0.336   lengthscale: 141.009   noise: 0.042

Now, we test predictions. For simplicity, we will assume a fixed variance of $0.01.$

```python
# Get into evaluation (predictive posterior) mode
model.eval()
likelihood.eval()

# Test points are regularly spaced along [0,1]
# Make predictions by feeding model through likelihood
with torch.no_grad(), gpytorch.settings.fast_pred_var():
    test_x = torch.linspace(0, 1, 51)
    test_x_distributional = torch.stack((test_x, (1e-2 * torch.ones_like(test_x)).log()), dim=1)
    observed_pred = likelihood(model(test_x_distributional))

with torch.no_grad():
    # Initialize plot
    f, ax = plt.subplots(1, 1, figsize=(12, 12))

    # Get upper and lower confidence bounds
    lower, upper = observed_pred.confidence_region()
    # Plot training data as black stars
    ax.errorbar(train_x_mean.numpy(), train_y.numpy(), xerr=train_x_stdv, fmt='k*')
    # Plot predictive means as blue line
    ax.plot(test_x.numpy(), observed_pred.mean.numpy(), 'b')
    # Shade between the lower and upper confidence bounds
    ax.fill_between(test_x.numpy(), lower.numpy(), upper.numpy(), alpha=0.5)
    ax.set_ylim([-3, 3])
    ax.legend(['Observed Data', 'Mean', 'Confidence'])
```

As a final note, we've made it very easy to extend the distributional kernel class by exposing a generic `DistributionalInputKernel` class that takes as input any distance function over probability distributions.

```python

```


# DiffEqBiological Tutorial I: Introduction
### Samuel Isaacson

DiffEqBiological.jl is a domain specific language (DSL) for writing chemical
reaction networks in Julia. The generated chemical reaction network model can
then be translated into a variety of mathematical models which can be solved
using components of the broader
[DifferentialEquations.jl](http://juliadiffeq.org/) ecosystem.

In this tutorial we'll provide an introduction to using DiffEqBiological to
specify chemical reaction networks, and then to solve ODE, jump, tau-leaping and
SDE models generated from them. Let's start by using the DiffEqBiological
`reaction_network` macro to specify a simply chemical reaction network; the
well-known Repressilator.

We first import the basic packages we'll need, and use Plots.jl for making
figures:

```julia
# If not already installed, first hit "]" within a Julia REPL. Then type:
# add DifferentialEquations DiffEqBiological PyPlot Plots Latexify

using DifferentialEquations, DiffEqBiological, Plots, Latexify
pyplot(fmt=:svg);
```

We now construct the reaction network. The basic types of arrows and predefined
rate laws one can use are discussed in detail within the DiffEqBiological
[Chemical Reaction Models
documentation](http://docs.juliadiffeq.org/latest/models/biological.html). Here
we use a mix of first order, zero order and repressive Hill function rate laws.
Note, $\varnothing$ corresponds to the empty state, and is used for zeroth order
production and first order degradation reactions:

```julia
repressilator = @reaction_network begin
    hillr(P₃,α,K,n), ∅ --> m₁
    hillr(P₁,α,K,n), ∅ --> m₂
    hillr(P₂,α,K,n), ∅ --> m₃
    (δ,γ), m₁ ↔ ∅
    (δ,γ), m₂ ↔ ∅
    (δ,γ), m₃ ↔ ∅
    β, m₁ --> m₁ + P₁
    β, m₂ --> m₂ + P₂
    β, m₃ --> m₃ + P₃
    μ, P₁ --> ∅
    μ, P₂ --> ∅
    μ, P₃ --> ∅
end α K n δ γ β μ;
```

We can use Latexify to look at the corresponding reactions and understand the
generated rate laws for each reaction

```julia
latexify(repressilator; env=:chemical)
```

\begin{align}
\require{mhchem}
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_3^{n}}] m_{1}}\\
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_1^{n}}] m_{2}}\\
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_2^{n}}] m_{3}}\\
\ce{ m_{1} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{2} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{3} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{1} &->[\beta] m_{1} + P_{1}}\\
\ce{ m_{2} &->[\beta] m_{2} + P_{2}}\\
\ce{ m_{3} &->[\beta] m_{3} + P_{3}}\\
\ce{ P_{1} &->[\mu] \varnothing}\\
\ce{ P_{2} &->[\mu] \varnothing}\\
\ce{ P_{3} &->[\mu] \varnothing}
\end{align}

We can also use Latexify to look at the corresponding ODE model for the chemical
system

```julia
latexify(repressilator)
```

\begin{align}
\frac{dm_1}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_3^{n}} - \delta \cdot m_1 + \gamma \\
\frac{dm_2}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_1^{n}} - \delta \cdot m_2 + \gamma \\
\frac{dm_3}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_2^{n}} - \delta \cdot m_3 + \gamma \\
\frac{dP_1}{dt} =& \beta \cdot m_1 - \mu \cdot P_1 \\
\frac{dP_2}{dt} =& \beta \cdot m_2 - \mu \cdot P_2 \\
\frac{dP_3}{dt} =& \beta \cdot m_3 - \mu \cdot P_3 \\
\end{align}

To solve the ODEs we need to specify the values of the parameters in the model,
the initial condition, and the time interval to solve the model on. To do this
it helps to know the orderings of the parameters and the species. Parameters are
ordered in the same order they appear after the `end` statement in the
`@reaction_network` macro. Species are ordered in the order they first appear
within the `@reaction_network` macro. We can see these orderings using the
`speciesmap` and `paramsmap` functions:

```julia
speciesmap(repressilator)
```

    OrderedCollections.OrderedDict{Symbol,Int64} with 6 entries:
      :m₁ => 1
      :m₂ => 2
      :m₃ => 3
      :P₁ => 4
      :P₂ => 5
      :P₃ => 6

```julia
paramsmap(repressilator)
```

    OrderedCollections.OrderedDict{Symbol,Int64} with 7 entries:
      :α => 1
      :K => 2
      :n => 3
      :δ => 4
      :γ => 5
      :β => 6
      :μ => 7

## Solving the ODEs:
Knowing these orderings, we can create parameter and initial condition vectors,
and setup the `ODEProblem` we want to solve:

```julia
# parameters [α,K,n,δ,γ,β,μ]
p = (.5, 40, 2, log(2)/120, 5e-3, 20*log(2)/120, log(2)/60)

# initial condition [m₁,m₂,m₃,P₁,P₂,P₃]
u₀ = [0.,0.,0.,20.,0.,0.]

# time interval to solve on
tspan = (0., 10000.)

# create the ODEProblem we want to solve
oprob = ODEProblem(repressilator, u₀, tspan, p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 10000.0)
    u0: [0.0, 0.0, 0.0, 20.0, 0.0, 0.0]

At this point we are all set to solve the ODEs. We can now use any ODE solver
from within the DiffEq package. We'll just use the default DifferentialEquations
solver for now, and then plot the solutions:

```julia
sol = solve(oprob, saveat=10.)
plot(sol, fmt=:svg)
```

We see the well-known oscillatory behavior of the repressilator! For more on
choices of ODE solvers, see the JuliaDiffEq
[documentation](http://docs.juliadiffeq.org/latest/solvers/ode_solve.html).

---

## Stochastic Simulation Algorithms (SSAs) for Stochastic Chemical Kinetics
Let's now look at a stochastic chemical kinetics model of the repressilator,
modeling it with jump processes. Here we will construct a DiffEqJump
`JumpProblem` that uses Gillespie's `Direct` method, and then solve it to
generate one realization of the jump process:

```julia
# first we redefine the initial condition to be integer valued
u₀ = [0,0,0,20,0,0]

# next we create a discrete problem to encode that our species are integer valued:
dprob = DiscreteProblem(repressilator, u₀, tspan, p)

# now we create a JumpProblem, and specify Gillespie's Direct Method as the solver:
jprob = JumpProblem(dprob, Direct(), repressilator, save_positions=(false,false))

# now let's solve and plot the jump process:
sol = solve(jprob, SSAStepper(), saveat=10.)
plot(sol, fmt=:svg)
```

Here we see that oscillations remain, but become much noiser. Note, in
constructing the `JumpProblem` we could have used any of the SSAs that are part
of DiffEqJump instead of the `Direct` method, see the list of SSAs (i.e.
constant rate jump aggregators) in the
[documentation](http://docs.juliadiffeq.org/latest/types/jump_types.html#Constant-Rate-Jump-Aggregators-1).

---
## $\tau$-leaping Methods:
While SSAs generate exact realizations for stochastic chemical kinetics jump
process models, [$\tau$-leaping](https://en.wikipedia.org/wiki/Tau-leaping)
methods offer a performant alternative by discretizing in time the underlying
time-change representation of the stochastic process. The DiffEqJump package has
limited support for $\tau$-leaping methods in the form of the basic Euler's
method type approximation proposed by Gillespie. We can simulate a $\tau$-leap
approximation to the repressilator by using the  `RegularJump` representation of
the network to construct a `JumpProblem`:

```julia
rjs = regularjumps(repressilator)
lprob = JumpProblem(dprob, Direct(), rjs)
lsol = solve(lprob, SimpleTauLeaping(), dt=.1)
plot(lsol, plotdensity=1000, fmt=:svg)
```

---
## Chemical Langevin Equation (CLE) Stochastic Differential Equation (SDE) Models:
At an intermediary physical scale between macroscopic ODE models and microscopic
stochastic chemical kinetic models lies the CLE, a SDE version of the model. The
SDEs add to each ODE above a noise term. As the repressilator has species that
get very close to zero in size, it is not a good candidate to model with the CLE
(where solutions can then go negative and become unphysical). Let's create a
simpler reaction network for a birth-death process that will stay non-negative:

```julia
bdp = @reaction_network begin
  c₁, X --> 2X
  c₂, X --> 0
  c₃, 0 --> X
end c₁ c₂ c₃
p = (1.0,2.0,50.)
u₀ = [5.]
tspan = (0.,4.);
```

The corresponding Chemical Langevin Equation SDE is then

$$
dX_t = \left(c_1 X - c_2 X + c_3 \right) dt + \left( \sqrt{c_1 X} - \sqrt{c_2 X} + \sqrt{c_3} \right)dW_t,
$$

where $W_t$ denotes a standard Brownian Motion. We can solve the CLE SDE model
by creating an SDEProblem and solving it similar to what we did for ODEs above:

```julia
# SDEProblem for CLE
sprob = SDEProblem(bdp, u₀, tspan, p)

# solve and plot, tstops is used to specify enough points
# that the plot looks well-resolved
sol = solve(sprob, tstops=range(0., step=4e-3, length=1001))
plot(sol, fmt=:svg)
```

We again have complete freedom to select any of the
StochasticDifferentialEquations.jl SDE solvers, see the
[documentation](http://docs.juliadiffeq.org/latest/solvers/sde_solve.html).

---
## What information can be queried from the reaction_network:
The generated `reaction_network` contains a lot of basic information. For example
- `f=oderhsfun(repressilator)` is a function `f(du,u,p,t)` that given the current
  state vector `u` and time `t` fills `du` with the time derivatives of `u`
  (i.e. the right hand side of the ODEs).
- `jac=jacfun(repressilator)` is a function `jac(J,u,p,t)` that evaluates and
  returns the Jacobian of the ODEs in `J`. A corresponding Jacobian matrix of
  expressions can be accessed using the `jacobianexprs` function:

```julia
latexify(jacobianexprs(repressilator))
```

\begin{equation}
\left[
\begin{array}{cccccc}
 - \delta & 0 & 0 & 0 & 0 & \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_3^{-1 + n}}{\left( K^{n} + P_3^{n} \right)^{2}} \\
0 &  - \delta & 0 & \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_1^{-1 + n}}{\left( K^{n} + P_1^{n} \right)^{2}} & 0 & 0 \\
0 & 0 &  - \delta & 0 & \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_2^{-1 + n}}{\left( K^{n} + P_2^{n} \right)^{2}} & 0 \\
\beta & 0 & 0 &  - \mu & 0 & 0 \\
0 & \beta & 0 & 0 &  - \mu & 0 \\
0 & 0 & \beta & 0 & 0 &  - \mu \\
\end{array}
\right]
\end{equation}

- `pjac = paramjacfun(repressilator)` is a function `pjac(pJ,u,p,t)` that
  evaluates and returns the Jacobian, `pJ`, of the ODEs *with respect to the
  parameters*. This allows `reaction_network`s to be used in the
  DifferentialEquations.jl local sensitivity analysis package
  [DiffEqSensitivity](http://docs.juliadiffeq.org/latest/analysis/sensitivity.html).

By default, generated `ODEProblems` will be passed the corresponding Jacobian
function, which will then be used within implicit ODE/SDE methods.

The [DiffEqBiological API
documentation](http://docs.juliadiffeq.org/latest/apis/diffeqbio.html) provides
a thorough description of the many query functions that are provided to access
network properties and generated functions. In DiffEqBiological Tutorial II
we'll explore the API.

---
## Getting Help
Have a question related to DiffEqBiological or this tutorial? Feel free to ask
in the DifferentialEquations.jl [Gitter](https://gitter.im/JuliaDiffEq/Lobby).
If you think you've found a bug in DiffEqBiological, or would like to
request/discuss new functionality, feel free to open an issue on
[Github](https://github.com/JuliaDiffEq/DiffEqBiological.jl) (but please check
there is no related issue already open). If you've found a bug in this tutorial,
or have a suggestion, feel free to open an issue on the [DiffEqTutorials Github
site](https://github.com/JuliaDiffEq/DiffEqTutorials.jl). Or, submit a pull
request to DiffEqTutorials updating the tutorial!

---


```python
import numpy as np
import sympy as sm
import sympy.vector

a,b,c = sm.symbols('a b c')
C = sm.vector.CoordSys3D('')
C.i.dot(C.j)
C.i.cross(C.j)

v = a*C.j + b*C.k + 3*a*C.i
w = (sm.sin(a)**2 + sm.cos(a)**2)*C.i + (2*sm.cos(b) - 1)*C.j + 2*c*C.k
w.factor()
w.simplify()
w.dot(v)
w.cross(w).simplify()
w.diff(b,1)

sm.Derivative(w,a)
sm.Derivative(w,c).doit()
sm.Integral(w,a).doit()
sm.Integral(w,a)
```

$\displaystyle (\int \left(\sin^{2}{\left(a \right)} + \cos^{2}{\left(a \right)}\right)\, da)\mathbf{\hat{i}_{}} + (\int \left(2 \cos{\left(b \right)} - 1\right)\, da)\mathbf{\hat{j}_{}} + (\int 2 c\, da)\mathbf{\hat{k}_{}}$

```python
# O = C.origin
P = O.locate_new('P', C.i + C.j + C.k)
P = C.origin.locate_new('P', C.i + C.j + C.k)
Q = P.locate_new('Q', 3*C.i + -3*C.j)
R = P.position_wrt(Q)
#Q.express_coordinates(C)
type(R)
```

    sympy.vector.vector.VectorAdd

```python

```