**Copyright 2020 Marcel Arpogaus**

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

# Implementation of Bernstein-Polynomials as TensorFlow Probability Bijector

    Run in Google Colab

    View source on GitHub

    Download notebook

Table of Contents

1  Introduction

1.1  Motivation: Need for Complex Distributions

1.2  Probabilistic Regression to Express Aleatoric Uncertainty

2  Dependencies & Prerequisites

2.1  Install Dependencies

2.2  Import Libaries

2.3  Ensure Reproducibility

2.4  Function Definitions

3  Bernstein Polynomials

3.1  Bernstein Basis Polynomials

3.2  Bernstein Polynomial

3.3  Bernstein Bijector

4  Conditioned Normalizing Flow using Bernstein Polynomials

4.1  Building up the Flow

4.2  Illustration of the Flow

4.3  Example: Bimodal Distribution

4.3.1  Evaluating

4.3.2  Sampling

5  Normalizing Flow Network

5.1  Data

5.2  Transformed Distribution as Keras Layer

5.2.1  Model Definition

5.2.2  Training

5.2.3  Result

5.3  Transformed Distribution as Keras Loss Function

5.3.1  Model Definition

5.3.2  Training

5.3.3  Result

## Introduction

### Motivation: Need for Complex Distributions

The shape of data distributions are often very complex.
Transformation models are an easy way to replace simple Distributions (like a Gaussian) with more expressive ones.

The following example of a classical data set containing the waiting time between eruptions of the [Old Faithful Geyser](https://en.wikipedia.org/wiki/Old_Faithful) in [Yellowstone National Park](https://en.wikipedia.org/wiki/Yellowstone_National_Park) is used as an example.

| Gaussian                                                     | Normalizing Flow                           |
|:-------------------------------------------------------------|:-------------------------------------------|
|                                       |                       |

As shown in the left figure, the normality assumption is clearly violated by the bimodal nature of the data.
The proposed transformation model, however, has the flexibility to adapt to this complexity.

### Probabilistic Regression to Express Aleatoric Uncertainty

Real-world regression problems are often not deterministic, meaning that for every possible input vector *x* a whole range of outputs *y* are probable.
In such cases it is not sufficient to just predict a single point, but rather the conditioned probability density function *p(y|x)* for *y* given the input *x*.
This class of machine learning problems is commonly referred to as conditional density estimation (CDE).

Normalizing flows (NF) [16] are a type of transformation model that can be used for density estimation.
Instead of parameterizing a density function to fit the shape of the desired conditional probability distribution, they use a single simple *base distribution*, usually a Gaussian, and pass it through a series of special transformation functions until its shape is most similar to that of the *data distribution*.
This composition of simple transformation functions allows NF based models to be very expressive, while staying computational efficient [10].

Conditioned normalizing flows can be used to model complex distributions, without assuming much about their shape [4, 14].
Their expressiveness, however, is highly dependent on the flexibility of the used transformation functions.
A recent study [14] presented a new type of flow-based transformation models especially optimized for CDE.
The authors joined ideas from statistical transformation models [4] as well as deep normalizing flows [16] and were able to outperform existing models for complex distributions far away from Gaussian.
Very expressive Bernstein polynomials are used as basis transformations [4] combined in a composition of four different transformation functions to build a NF.

Compared to the statistical transformation models the proposed deep transformation model does not require predefined features and can be trained in an end-to-end fashion from complex data.

##  Dependencies & Prerequisites

Before we start, lets import all the packages we require late.

### Install Dependencies

```python
try:
    from google.colab import drive
    !pip install git+https://github.com/MArpogaus/TensorFlow-Probability-Bernstein-Polynomial-Bijector.git
except:
    print('Not running in Colab')
```

    Not running in Colab

### Import Libaries

```python
import os
import pandas as pd
import numpy as np

from functools import partial

import scipy.interpolate as I
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="ticks", context="paper")  # improved plt styles
```

```python
import tensorflow as tf
import tensorflow_probability as tfp

from tensorflow_probability import distributions as tfd

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense, Input, Reshape, Concatenate
from tensorflow.keras.optimizers import Adam

from sklearn.preprocessing import MinMaxScaler
```

Import The Bernstein flow and loss classes.

```python
from bernstein_flow.bijectors import BernsteinBijector
from bernstein_flow.distributions import BernsteinFlow
from bernstein_flow.losses import BernsteinFlowLoss
```

Import utility functions to visualize the flow

```python
from bernstein_flow.util.visualization import vizualize_flow_from_z_domain
from bernstein_flow.util.visualization import plot_chained_bijectors
from bernstein_flow.util.visualization import plot_flow
```

### Ensure Reproducibility

```python
np.random.seed(2)
tf.random.set_seed(2)
print("TFP Version", tfp.__version__)
print("TF  Version", tf.__version__)
```

    TFP Version 0.11.1
    TF  Version 2.3.0

### Function Definitions

We want to directly minimize the negative log-likelihood:

```python
def negloglik(y_true, y_hat):
    nll = -y_hat.log_prob(y_true)
    return nll
```

## Bernstein Polynomials

In [14] the expressive Bernstein polynomials are used as basis transformations.
So lets first cover some theory about these kinds of polynomials.

### Bernstein Basis Polynomials

The Bernstein polynomial basis has been introduced by Sergei Natanovich Bernstein over 100 years ago to facilitate a constructive proof of the Weierstrass approximation theorem.
It turned out as a remarkable method for approximating any continuous function, to the desired accuracy over finite domains, and firstly found application in computer-aided geometric design.
Due to its excellent numerical stability properties, the diversification of its repertoire of applications steadily increased.
A comprehensible survey of the historical development and current state of theory, algorithms, and applications can be found in [3].
The Bernstein basis of degree $M$ on $x\in[0,1]$ is defined by:

\begin{equation}
    \operatorname{Be}_{k}^{M}(x):=
        \left(%
            \begin{array}{l}
                M \\
                k
            \end{array}%
        \right)
        (1-x)^{M-k} x^{k}, \ \text{with} \ k=0, \ldots, M
\end{equation}

The Bernstein basis polynomials have two fundamental properties, firstly they are non–negative on $x\in[0,1]$ and secondly they form a partition of unity, hence

\begin{equation}
    \sum_{i=0}^{n} Be_{i}^{M}(t)=1.
\end{equation}

The five Bernstein basis polynomials of degree $M=5$ on $x\in[0,1]$ along with the cumulative sum are are shown in the following plot.

```python
M = 5
x = np.linspace(0, 1, 100)
d = {}
for i in range(M):
    c = np.zeros(M).reshape(-1, 1)
    c[i] = 1
    bp = I.BPoly(c, [0, 1])
    d[f'$Be^{M}_{i}$'] = bp(np.linspace(0, 1, 100))

df = pd.DataFrame(d, index=x)
df[f'$\sum^{M}_{{i=0}} Be^{M}_i$'] = df.sum(axis='columns')
df.plot(figsize=(16, 8))
```

### Bernstein Polynomial

The Bernstein basis polynomials of degree $M$ form a basis of polynomials of degree at most $M$ with real coefficients. A linear combination of Bernstein basis polynomials

\begin{equation}
    b_{M}(x)=\sum_{i=0}^{M} \vartheta_{i} \operatorname{Be}_{i}^{M}
\end{equation}

is then called a Bernstein polynomial of degree $M$, with the Bernstein coefficients $\vartheta_\nu$[3].
The following plot sows the Bernstein polynomials with increasing order $M=1,\ldots,5$, and random integer coefficients $\theta$.

```python
M = 5
x = np.linspace(0, 1, 100)
d = {}
for i in range(1, M):
    c = np.random.randint(1, M, (i, 1))
    bp = I.BPoly(c, [0, 1])
    d[f'$b_{i}(x); \\vartheta=[{",".join(c.flatten().astype(str))}]$'] = bp(
        np.linspace(0, 1, 100))

df = pd.DataFrame(d, index=x)
df.plot(figsize=(16, 8))
```

### Bernstein Bijector

The provided `BernsteinBijector` implements a the [`tfb.Bijector`](https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors/Bijector) for convenient transformation of samples using [`tfd.TransformedDistribution`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/TransformedDistribution).

The monotonicity of a Bernstein polynomial can been ensured via linear constraints (`BernsteinBijector.constrain_theta`).
If $\tilde\vartheta_k$ is the $k$-th Bernstein coefficients, choosing $\vartheta_k = \vartheta_{k-1} + \exp(\tilde\vartheta_k)$ for $k>0$, dos the trick.
The first output can stay unchanged and defines $\vartheta_0=\tilde\vartheta_0$.

These Constrains make the bijector technically invertible.
However, a closed-form solution for the inversion of higher-order Bernstein polynomials is not known.
Since extensions of the Bernstein polynomials for interpolations with *piecewise-polynomial functions* (B-Splines) exists [3] they are used to approximate the inverse of the Bernstein polynomial.
In the implementation cubic splines have been used for the approximation using a linear interval of $10$ times the polynomial order $M$ points.

```python
M = 9
# ensure monotonicity
theta = BernsteinBijector.constrain_theta(
    np.random.normal(0, 1, (1, M)).astype(np.float32)
)
print(theta)
bs = BernsteinBijector(
    theta=theta
)
yy = np.linspace(0, 1, 200, dtype=np.float32)
zz = bs.forward(yy)
# Prevent caching -> use spline interpolation
zi = zz + 1e-15 * np.random.random(zz.shape)
yyy = bs.inverse(zi)

plt.figure(figsize=(16, 16))
plt.plot(yy, zz)
plt.plot(yyy, zi)
```

```python
f'The MSE of the interpolation in this example is {np.sum((np.squeeze(yyy)-yy)**2):.3e}$.'
```

    'The MSE of the interpolation in this example is 2.017e-10$.'

## Conditioned Normalizing Flow using Bernstein Polynomials

Models based on NF have only gained little attention in the research community for CDE and have predominantly been used for variational inference [6, 12] or modeling of complex multi-modal unconditional distributions, for example for image generation [5, 2] or speech synthesis [8, 9].
Compared to conditional VAE [15] and conditional GAN [7] NF based models explicitly estimate the distribution and hence the negative logarithmic likelihood becomes tractable and can be used as loss-function, what makes them attractive for CDE.
Very recent research applied NF in that context and has shown very promising results [13, 11, 17].

In the original formulation of NF by [16], rich density approximations were constructed, by transforming a simple initial density, into a more complex one.
This implementation of flows is particularly efficient for generating new samples from the learned data distribution but limited by the inefficiency of evaluating the likelihood of externally provided data since it requires inverting each $f_n$.
Depending on the specific implementation it may be sufficient to reverse that direction, to allow the evaluation of likelihoods in constant time.
The following illustration  shows a inverse conditioned normalizing flow model, transforming a complex conditioned distribution $p_y(y|x)$ step by step to a simple Gaussian distribution $p_z(z|x)$, conditioned on some value $x$. The visualization was inspired by [18].

Defining the forward mapping from the target $y$ to the base distribution, allows the density to be evaluated in constant time, even for complex transformation functions, which leads to

\begin{equation}
    z = t(y,\boldsymbol{\theta}_x)\qquad\text{with}\qquad y \sim p_z(z)
\end{equation}

with $t$ being a composition of $K$ bijectors $f_1(y, \boldsymbol{\theta}_{1}) \cdots f_K(y, \boldsymbol{\theta}_{K})$, parameterized with the corresponding parameter vector $\boldsymbol{\theta}_n=[\vartheta_1,\ldots,\vartheta_l]$.
Conditioning the transformed *base distribution* $p_z(z|\mathbf{x})$ on the vector $\mathbf{x}$ is then achieved by, conditioning the parameters $\boldsymbol{\theta}_n$ on them.

A neural network can be trained to learn the mapping from the input $\mathbf{x}$ to the parameter vectors $h(\mathbf{x},\omega)=\boldsymbol{\theta}_x=[\theta_{1,x},\ldots,\theta_{K,x}]^T$, which is then used to parametrize the transformer functions $f_n(y,\theta_{n,x})$.

\begin{equation}
    \begin{aligned}
        z &= t(y|x,\boldsymbol{\theta}_x) = (f_K \circ f_{K-1} \circ \cdots \circ f_1)(y|x,\boldsymbol{\theta}_x)\\
          &= t(y,h(\mathbf{x},\omega)) = f_K(\cdots(f_1(y,\theta_{1,x} )\cdots)\theta_{M,x}
    )
    \end{aligned}
\end{equation}.

This way each input $\mathbf{x}$ yields a different parametrization of $t$ and thus results in a different transformation.

The expressiveness of the NF is highly dependent on the used bijectors.
Building upon the works from [4] we will use the very expressive Bernstein polynomials as basis transformations combined in a composition of four different transformation functions to define the flow [14].

### Building up the Flow

First a affine transformation $f_1$ is used, which scales and shifts $y$ followed by a sigmoid function to ensure the restriction to the valid range $[0, 1]$:

\begin{equation}
   \sigma \circ  f_1: \quad \tilde{y} = \sigma(a(\mathbf{x}) \cdot y - b(\mathbf{x}))
\end{equation}

To condition the scale and shift parameters $a$ and $b$ in the input vector $\mathbf{x}$, they are controlled by the output of an artificial neural network with input $\mathbf{x}$.
To ensure monotonicity the softplus activation function is used for $a$, the activation of $b$ is linear.

The second transformation $f_2$ is the transformation consisting of a Bernstein polynomial with $M+1$
parameters $\vartheta_0 , \vartheta_1,\ldots \vartheta_M$ which again are controlled by a neural network.

\begin{equation}
    f_{2}: \quad \tilde{z}=\frac{1}{M+1} \sum_{i=0}^{M} \operatorname{Be}_{i}^M(\tilde{y}) \vartheta_{i}(\mathbf{x})
\end{equation}

$Be$ is the Bernstein polynomials of order $M$, generated by the $M + 1$ beta-densities

\begin{equation}
    \operatorname{Be}_i^M(\tilde{y})=f_{i+1, M-i+1}(\tilde{y})
\end{equation}

Maintaining the monotonicity for the invertibility of $f_2$ can then easily be achieved via a linear constraint on the basis coefficients, a circumstance that the implementation in [14] also took advantage of by simply enforcing that the coefficients $\theta=[\vartheta_1,\ldots\vartheta_M]^T$ of the polynomial are in increasing order.

The last transformation in the chain $f_3$ is again a scale and shift transformation to force the output of $f_3$ into the range of the standard normal distribution.

\begin{equation}
    f_{3}: \quad z=\alpha(\mathbf{x}) \cdot \tilde{z}-\beta(\mathbf{x})
\end{equation}

The parameters $\alpha$ and $\beta$ are again controlled by a neural network, where the
softplus transformation is used to guarantee a positive $\alpha$.
The first experiments showed that the third bijector is not necessary for the performance, but accelerates training.

The final transformation function $t(y,h(\mathbf{x},\omega)): y \rightarrow z$ is then obtained by chaining the single transformation functions.

\begin{equation}\label{eq-sick-flow}
    \begin{aligned}
        z &= t(y,h(\mathbf{x},\omega)) = (f_3 \circ f_2 \circ \sigma \circ f_1)(y,h(\mathbf{x},\omega))\\
          &= (f_{3, \alpha(x), \beta(x)} \circ f_{2, \vartheta_{0}(x), \ldots, \vartheta_{M}(x)} \circ \sigma \circ f_{1, a(x), b(x)})(y,x)
    \end{aligned}
\end{equation}.

### Illustration of the Flow

Illustration of a flow transforming from a bimodal distribution, to a standard Gaussian. Image taken from [14]:

### Example: Bimodal Distribution

In the following we will use fixed values to parametrize a example Flow.
It will transform a simple Gaussian Distribution to a more complex bimodal distribution.

```python
a1 = np.array([6], dtype=np.float32)
b1 = np.array([-3], dtype=np.float32)
theta = np.array([-2 ,  2,  3, -7, -7, -7, -7, -7, 7], dtype=np.float32)
a2 = np.array([0.2], dtype=np.float32)
b2 = np.array([-2.5], dtype=np.float32)
theta
```

    array([-2.,  2.,  3., -7., -7., -7., -7., -7.,  7.], dtype=float32)

#### Evaluating

\begin{equation}
    p_y(y|\mathbf{x}) = p_z(t(\mathbf{y,\boldsymbol{\theta}_x})) \left|\det\nabla{t}(\mathbf{x},\boldsymbol{\theta}_x)\right|
\end{equation}

```python
flow = BernsteinFlow(np.concatenate([a1,b1,theta,a2,b2]))
flow
```

    <tfp.distributions.BernsteinFlow 'BernsteinFlow' batch_shape=[1] event_shape=[] dtype=float32>

This Visualization shows how the distribution shapes when flowing through the chain of bijectors from the z-Domain:

```python
plot_chained_bijectors(flow)
```

This is the shape of the Bernstein polynomial:

```python
bs=flow.bijector#.bijector.bijectors[2]
zz = bs.forward(yy)
plt.figure(figsize=(16, 16))
plt.plot(yy, zz)
```

The final density function along with the transformed mean (black line), variance (green dashed) and quantiles (blue shades):

```python
n = 1000
y_samples = np.linspace(0, 1, n, dtype=np.float32)
y_probs = flow.prob(y_samples[...,None])

plt.figure(figsize=(16,8))
plot_flow(flow,y_samples[...,None])
```

The Jacobean matrix $\left|\det\nabla{t}(\mathbf{x},\boldsymbol{\theta}_x)\right|$ *normalizes* the transformed distribution such that it integrates to 1:

```python
dy = y_samples[1]-y_samples[0]
(sum(y_probs)*dy).numpy()
```

    array([0.8512521], dtype=float32)

#### Sampling
\begin{equation}
    y\sim p_y(y|x) = t^{-1}(z,\boldsymbol{\theta}_x)\qquad\text{with}~z \sim p_z(z)
\end{equation}

With the approximated inverse sampling from the distribution becomes possible:

```python
samples=flow.sample(10000)
```

```python
fig = plt.figure(figsize=(16,8))
sns.distplot(samples.numpy().flatten(),bins=100,label='samples')

plt.plot(y_samples, y_probs,label='flow')
plt.legend()
```

The following visualization shows how samples, drawn from the z-domain, get mapped to the y-domain.

```python
fig=vizualize_flow_from_z_domain(flow)
```

## Normalizing Flow Network

All the parameters $a,b,\theta,\alpha$ and $\beta$ can be obtained from the outputs of a artificial neural network with input $\mathbf{x}$.

### Data

We will define a simple artifical dataset for a probabilistic regression task.

```python
from sklearn.utils import shuffle

def gen_data(t):
    # x=1: y(t)=sin(t) + Normal(loc=0, scale=0.1*|y(t)|)
    t1=t
    y1=1.0*t1
    y1+=np.random.normal(0,0.05*np.abs(t1))

    # x=1: y(t)=-sin(t) + Normal(loc=0, scale=0.1*|y(t)|)
    t2=t
    y2=-.2*t2
    y2+=np.random.normal(0,0.2*np.abs(t2))

    t=np.concatenate([t1,t2])
    y=np.concatenate([y1,y2])

    return shuffle(t[...,np.newaxis], y[...,np.newaxis], random_state=0)
```

```python
def gen_test_data(n=5, observations=100):
    t = np.linspace(0,1,n, dtype=np.float32)
    t=np.repeat([t],observations)

    return gen_data(t)
```

```python
def gen_train_data(n=100):
    # x=1: y(t)=sin(t) + Normal(loc=0, scale=0.1*|y(t)|)
    t = np.random.uniform(0,1,n//2).astype(np.float32)

    return gen_data(t)
```

```python
n = 2000
train_x,train_y = gen_train_data(n = n)
val_x, val_y = gen_train_data(n = n//10)
train_x.shape,train_y.shape, val_x.shape,val_y.shape
```

    ((2000, 1), (2000, 1), (200, 1), (200, 1))

```python
plt.scatter(train_x,train_y, alpha=0.1)
```

```python
t_test, y_test = gen_test_data(5, 200)
x_test = np.unique(t_test)
```

```python
plt.scatter(t_test, y_test, alpha=0.1)
```

### Transformed Distribution as Keras Layer

#### Model Definition

```python
bernstein_order = 12 # Order of Bernstein polynomial

flow_model = tf.keras.Sequential([
    Input(1),
    Dense(8, activation='relu'),
    Dense(4 + bernstein_order), # Bernstein coefficients and 2 times scale and shift
    tfp.layers.DistributionLambda(BernsteinFlow) # Add the Trasnformed Distribution
])
flow_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense (Dense)                (None, 8)                 16
    _________________________________________________________________
    dense_1 (Dense)              (None, 16)                144
    _________________________________________________________________
    distribution_lambda (Distrib ((None,), (None,))        0
    =================================================================
    Total params: 160
    Trainable params: 160
    Non-trainable params: 0
    _________________________________________________________________

```python
reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss', factor=0.1,
    patience=3, min_lr=0.00001)
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=6, restore_best_weights=True)
callbacks=[reduce_lr, early_stop]
```

```python
flow_model.compile(
    optimizer=tf.optimizers.Adam(learning_rate=0.01),
    loss=negloglik
)
```

#### Training

```python
hist = flow_model.fit(
    train_x,train_y,
    validation_data=(val_x, val_y),
    epochs=150,
    shuffle=True,
    callbacks=callbacks,
    batch_size=1,
    verbose=False
)
```

```python
hist_df = pd.DataFrame(hist.history)
hist_df.plot(figsize=(16,12), subplots=True), hist_df.loss.min()
```

#### Result

```python
flow = flow_model(x_test)
yy = np.linspace(-1, 1.5, 1000, dtype=np.float32)[...,None]
ps = flow.prob(yy)
```

```python
fig, ax = plt.subplots(len(x_test), figsize=(16, len(x_test)*4))
for i,x in enumerate(x_test):
    ax[i].set_title(f'x={x}')
    sampl=y_test[(t_test.flatten()==x)].flatten()
    ax[i].scatter(sampl, [0]*len(sampl), marker='|')
    #ax[i].hist(sampl, bins=30, density=True)
    ax[i].plot(yy,ps[:,i])
```

```python
fig=vizualize_flow_from_z_domain(flow_model(tf.ones(1)))
```

```python
plot_chained_bijectors(flow_model(tf.ones(1)))
```

```python
b2 = flow.bijector.bijector.bijectors[0].shift.numpy()
a2 = flow.bijector.bijector.bijectors[1].scale.numpy()
theta = flow.bijector.bijector.bijectors[2].theta.numpy()
b1 = flow.bijector.bijector.bijectors[4].shift.numpy()
a1 = flow.bijector.bijector.bijectors[5].scale.numpy()
```

```python
print(f"""
a1 = {a1}
b1 = {b1}
theta = {theta}
a2 = {a2}
b2 = {b2}
""")
```

    a1 = [27.867687  10.643073   6.5445995  4.051173   3.7937753]
    b1 = [-0.14596826 -0.36557338 -0.48864335 -0.60996133 -0.7784065 ]
    theta = [[-4.2067842  -4.2066836  -1.1863723  -0.8709214  -0.8707732   1.8681883
       1.8682883   1.8683883   1.8684883  10.393819   21.296143   56.071537  ]
     [-4.0907526  -4.089123    0.4110489   0.449197    0.44930068  0.44965032
       0.44975033  0.44985035  0.44995037  0.45006946  0.45030737 32.438946  ]
     [-4.6784105  -4.5946994   0.7007847   0.7071469   0.70724714  0.70735115
       0.70745116  0.7075512   0.7076512   0.7077512   0.70785165 34.241676  ]
     [-5.3824997  -4.21073     1.8717513   1.8729513   1.8730513   1.8731514
       1.8732514   1.8733515   1.8734515   1.8735515   1.8736515  37.958954  ]
     [-6.4065275  -5.062344    2.166798    2.1671426   2.1672425   2.1673424
       2.1674423   2.1675422   2.167642    2.167742    2.167842   44.627327  ]]
    a2 = [1.6726568  0.7610469  0.56367743 0.42921513 0.28369522]
    b2 = [ 0.50604945 -0.29888433 -0.5368844  -0.7020801  -0.7439234 ]

### Transformed Distribution as Keras Loss Function

The Bernstein bijector can also be used as a Keras loss function.

#### Model Definition

```python
bernstein_order = 12 # Order of Bernstin polynomial

flow_parameter_model = tf.keras.Sequential([
    Input(1),
    Dense(8, activation='relu'),
    Dense(4 + bernstein_order) # Bernstein coefficients and 2 times scale and shift
])
flow_parameter_model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_2 (Dense)              (None, 8)                 16
    _________________________________________________________________
    dense_3 (Dense)              (None, 16)                144
    =================================================================
    Total params: 160
    Trainable params: 160
    Non-trainable params: 0
    _________________________________________________________________

```python
flow_parameter_model.compile(
    optimizer=tf.optimizers.Adam(learning_rate=0.01),
    loss=BernsteinFlowLoss()
)
```

#### Training

```python
# Do inference.
hist = flow_parameter_model.fit(
    train_x, train_y,
    validation_data=(val_x, val_y),
    epochs=150,
    batch_size=1,
    verbose=False,
    callbacks=callbacks
)
```

```python
hist_df = pd.DataFrame(hist.history)
hist_df.plot(figsize=(16,12), subplots=True), hist_df.loss.min()
```

#### Result

```python
pvs = flow_parameter_model(x_test[...,None])
flow = BernsteinFlow(pvs)
yy = np.linspace(-1, 1.5, 1000, dtype=np.float32)
ps = flow.prob(yy[...,None])
```

```python
fig, ax = plt.subplots(len(x_test), figsize=(16, len(x_test)*4))
for i,x in enumerate(x_test):
    ax[i].set_title(f'x={x}')
    sampl=y_test[(t_test.flatten()==x)].flatten()
    ax[i].scatter(sampl, [0]*len(sampl), marker='|')
    #ax[i].hist(sampl, bins=30, density=True)
    ax[i].plot(yy,ps[:,i])
```

```python
fig=vizualize_flow_from_z_domain(BernsteinFlow(flow_parameter_model(tf.ones((1)))))
```

```python
plot_chained_bijectors(BernsteinFlow(flow_parameter_model(tf.ones((1)))))
```

```python
b2 = flow.bijector.bijector.bijectors[0].shift.numpy()
a2 = flow.bijector.bijector.bijectors[1].scale.numpy()
theta = flow.bijector.bijector.bijectors[2].theta.numpy()
b1 = flow.bijector.bijector.bijectors[4].shift.numpy()
a1 = flow.bijector.bijector.bijectors[5].scale.numpy()
```

```python
print(f"""
a1 = {a1}
b1 = {b1}
theta = {theta}
a2 = {a2}
b2 = {b2}
""")
```

    a1 = [30.387337  11.90539    6.3490133  4.3184857  3.1653957]
    b1 = [ 0.29992062 -0.5315632  -0.60218966 -0.6236345  -0.66010576]
    theta = [[-4.7898064  -4.778853   -1.1840863  -0.59784234 -0.540962   -0.5408594
       1.0341911   1.0342911   4.156374    8.362941   14.018232   67.138214  ]
     [-3.8011754  -3.721397    0.46536112  0.51523393  0.51663285  0.51673305
       0.5169688   0.5170688   0.5171885   0.5173284   0.51793027 40.409943  ]
     [-3.763159   -3.6315348   1.1652138   1.1736287   1.1737968   1.1738968
       1.1739969   1.174097    1.174197    1.174297    1.1743985  39.117733  ]
     [-3.9755173  -3.8326576   1.6368983   1.6388649   1.6389686   1.6390686
       1.6391686   1.6392686   1.6393687   1.6394687   1.6395687  41.564297  ]
     [-4.2286787  -4.08363     2.0876021   2.0881298   2.08823     2.0883298
       2.0884297   2.0885296   2.0886295   2.0887294   2.0888293  45.13305   ]]
    a2 = [2.3247056  0.686284   0.5773292  0.5003731  0.42071748]
    b2 = [ 0.3763702  -0.33933124 -0.7272833  -0.85299563 -0.89877206]

# References

[1] A. Azzalini and A. W. Bowman, “A Look at Some Data on the Old Faithful Geyser,” Journal of the Royal Statistical Society. Series C (Applied Statistics), vol. 39, no. 3, pp. 357–365, 1990, doi: 10.2307/2347385.

[2] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using Real NVP,” arXiv:1605.08803 [cs, stat], Feb. 2017. [Online](http://arxiv.org/abs/1605.08803).

[3] R. T. Farouki, “The Bernstein polynomial basis: A centennial retrospective,” Comput. Aided Geom. Des., vol. 29, no. 6, pp. 379–419, Aug. 2012, doi: 10.1016/j.cagd.2012.03.001.

[4] T. Hothorn, L. Möst, and P. Bühlmann, “Most Likely Transformations,” Scandinavian Journal of Statistics, vol. 45, no. 1, pp. 110–134, 2018, doi: 10.1111/sjos.12291.

[5] D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” arXiv:1807.03039 [cs, stat], Jul. 2018. [Online](http://arxiv.org/abs/1807.03039).

[6] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improving Variational Inference with Inverse Autoregressive Flow,” arXiv:1606.04934 [cs, stat], Jan. 2017. [Online](http://arxiv.org/abs/1606.04934).

[7] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784 [cs, stat], Nov. 2014. [Online](http://arxiv.org/abs/1411.1784).

[8] A. van den Oord et al., “WaveNet: A Generative Model for Raw Audio,” arXiv:1609.03499 [cs], Sep. 2016. [Online](http://arxiv.org/abs/1609.03499).

[9] A. van den Oord et al., “Parallel WaveNet: Fast High-Fidelity Speech Synthesis,” arXiv:1711.10433 [cs], Nov. 2017. [Online](http://arxiv.org/abs/1711.10433).

[10] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing Flows for Probabilistic Modeling and Inference,” arXiv:1912.02762 [cs, stat], Dec. 2019. [Online](http://arxiv.org/abs/1912.02762).

[11] K. Rasul, A.-S. Sheikh, I. Schuster, U. Bergmann, and R. Vollgraf, “Multi-variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows,” arXiv:2002.06103 [cs, stat], Feb. 2020. [Online](http://arxiv.org/abs/2002.06103).

[12] D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” arXiv:1505.05770 [cs, stat], Jun. 2016. [Online](http://arxiv.org/abs/1505.05770).

[13] J. Rothfuss et al., “Noise Regularization for Conditional Density Estimation,” arXiv:1907.08982 [cs, stat], Feb. 2020. [Online](http://arxiv.org/abs/1907.08982).

[14] B. Sick, T. Hothorn, and O. Dürr, “Deep transformation models: Tackling complex regression problems with neural network based transformation models,” arXiv:2004.00464 [cs, stat], Apr. 2020. [Online](http://arxiv.org/abs/2004.00464).

[15] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” in Advances in neural information processing systems 28, 2015, pp. 3483–3491, [Online](http://papers.nips.cc/paper/5775-learning-structured-output-representation-using-deep-conditional-generative-models.pdf).

[16] E. G. Tabak and C. V. Turner, “A Family of Nonparametric Density Estimation Algorithms,” Communications on Pure and Applied Mathematics, vol. 66, no. 2, pp. 145–164, 2013, doi: 10.1002/cpa.21423.

[17] B. L. Trippe and R. E. Turner, “Conditional Density Estimation with Bayesian Normalising Flows,” arXiv:1802.04908 [stat], Feb. 2018. [Online](http://arxiv.org/abs/1802.04908).

[18] L. Weng, “Flow-based deep generative models,” lilianweng.github.io/lil-log, Oct. 2018, [Online](http://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html).


```python
from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# *Circuitos Elétricos I*

## Semana 2 - Exercícios

### Exercício 1: circuito com fonte controlada de corrente

No circuito abaixo, dado que $\frac{V_0}{V_s}=9$, determine o valor de $A$.

### Exercício 2: circuito com fonte controlada de tensão

Determine o valor de todas as correntes no circuito abaixo.

### Solução das equações

```python
import sympy as sp
from utils import symdisp
```

```python
# define as N variáveis desconhecidas
i1, i2, i3, i4 = sp.symbols('i1, i2, i3, i4')

# define os sistema de N equações
eq1 = sp.Eq()
eq2 = sp.Eq()
eq3 = sp.Eq()
eq4 = sp.Eq()

# resolve o sistema
soluc = sp.solve((eq1, eq2, eq3, eq4), dict=True)
soluc = soluc[0]

i1 = soluc[i1]
i2 = soluc[i2]
i3 = soluc[i3]
i4 = soluc[i4]
ix = 3

print('Solução do sistema:')
symdisp('i_1 = ', round(i1, 2), ' A')
symdisp('i_2 = ', round(i2, 2), ' A')
symdisp('i_3 = ', round(i3, 2), ' A')
symdisp('i_4 = ', round(i4, 2), ' A')
symdisp('i_x = ', round(ix, 2), ' A')
```

#### Cálculo das potências

```python
# expressões para as potências
pR =
p10v =
p15v =
p2ix =

print('Potências:')
symdisp('p_R = ', round(pR, 2), ' W')
symdisp('p_{10V} = ', round(p10v, 2), ' W')
symdisp('p_{15V} = ', round(p15v, 2), ' W')
symdisp('p_{2i_x} = ', round(p2ix, 2), ' W')
```

```python
# calcula somatório das potências
print('Somatório das potências : %.2f W\n' %(pR+p10v+p15v+p2ix))
```

Jupyter notebooks/Circuitos Eletricos I - Semana 2.1.ipynb


```python
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns

from IPython.display import Image
```

Chapter 7 n-step Bootstrapping
=================

MC <-- n-step TD --> one-step TD

```python
Image('./res/fig7_1.png')
```

Monte Carlo return:

$G_t \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots + \gamma^{T-t-1} R_T$

one-step return:

$G_{t:t+1} \doteq R_{t+1} + \gamma V_t(S_{t+1})$

two-step return:

$G_{t:t+2} \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 V_{t+1}(S_{t+2})$

n-step return:

$G_{t:t+n} \doteq R_{t+1} + \gamma R_{t+2} + \cdots + \gamma^{n-1} R_{t+n} + \gamma^n V_{t+n-1}(S_{t+n})$

The natural state-value learning algorithm for using n-step returns is thus:

$V_{t+n}(S_t) \doteq V_{t+n-1}(S_t) + \alpha \left [ G_{t:t+n} - V_{t+n-1}(S_t) \right ]$

```python
Image('./res/n_step_predict.png')
```

error reduction property of n-step returns: their expectation is guaranteed to be a better estimate of $v_\pi$ than $V_{t+n-1}$ is, in a worst-state sense:

$\max_s \left | \mathbb{E}_\pi [G_{t:t+n} \mid S_t = s] - v_\pi(s) \right | \leq \gamma^n \max_s \left | V_{t+n-1}(s) - v_\pi(s) \right |$

### 7.2 n-step Sarsa

The main idea is to simply switch states for actions (state-action pairs) and then use an $\epsilon$-greedy policy.

$Q_{t+n}(S_t, A_t) \doteq Q_{t+n-1}(S_t, A_t) + \alpha [G_{t:t+n} - Q_{t+n-1}(S_t, A_t)] \quad \text{ , $0 \leq t < T$}$

```python
Image('./res/fig7_3.png')
```

Expected Sarsa (like n-step sarsa):

\begin{align}
    G_{t:t+n} & \doteq R_{t+1} + \cdots + \gamma^{n-1} R_{t+n} + \gamma^n \bar{V}_{t+n-1}(S_{t+n}) \qquad t + n < T \\
    \bar{V}_t(s) & \doteq \sum_a \pi(a \mid s) Q_t(s, a) \qquad \text{for all $s \in \mathcal{S}$}
\end{align}

### 7.3 n-step Off-policy Learning by Importance Sampling

simply be weighted by $\rho_{t:t+n-1}$, importance sampling ratio:

\begin{align}
    V_{t+n}(S_t) & \doteq V_{t+n-1}(S_t) + \alpha \rho_{t:t+n-1} [G_{t:t+n} - V_{t+n-1}(S_t)] \qquad 0 \leq t < T \\
    \rho_{t:h} & \doteq \prod_{k=t}^{\min(h, T-1)} \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)} \\
    Q_{t+n}(S_t, A_t) & \doteq Q_{t+n-1}(S_t, A_t) + \alpha \rho_{t+1:t+n-1} [G_{t:t+n} - Q_{t+n-1}(S_t, A_t)] \quad \text{ , $0 \leq t < T$}
\end{align}

### 7.4 Per-decision Off-policy Methods with Control Variates

### 7.5 Off-policy Learning Without Importance Sampling: The n-step Tree Backup Algorithm

Tree-backup algorithm:

In the tree-backup update, the target includes all these things plus the estimated values of the dangling action nodes hanging off the sides, at all levels.

```python
Image('./res/tree_update.png')
```

### 7.6 A Unifying Algorithm: n-step $Q(\sigma)$

dicide on a step-by-step basis whether one wanted to take the action as a sample (as in Sarsa), or consider the expectation over all actions instead (as in the tree-backup update).

Let $\sigma(t) \in [0, 1]$ denote the degree of sampling on step $t$, with $\sigma = 1$ denoting full sampling and $\sigma = 0$ denoting a pure expectation with no sampling. We call this proposed new algorithm n-step $Q(\sigma)$.

```python
Image('./res/fig7_5.png')
```

```python

```


 Training of a closure for the uncertainty ropagation in the Burgers equation

    Olivier Pannekoucke

Abstract

    Estimation of the closure for the uncertainty propagation in the Burgers dynamics

---
 Table of contents

 1. [Introduction](#introduction)
 1. [The Burgers dynamics and its unclosed PKF formulation](#burgers-pkf)
  1. [Set of the function and symbols](#burgers-pkf-sympy-definition)
  1. [Set constants for numerical experiments](#burgers-pkf-num-definition)
  1. [Set of the Burgers equation](#burgers-pkf-dyn-burgers)
  1. [Set of the PKF equations for the Burgers equation](#burgers-pkf-dyn-PKF)
 1. [Application numerique](#num)
  1. [Définition et utilisation de la dynamique fermée](#num-closed)
  1. [Définition et apprentissage de la dynamique non-fermée](#num-unclosed)
---

```python
import numpy as np
import matplotlib.pyplot as plt
#%matplotlib inline
#%matplotlib notebook
```

## Introduction

The aim is to design a NN which merges known and unknown physics.

```python
import sympy
from sympy import (Function, symbols, init_printing, Derivative,
                   latex, Add, Mul, Pow,
                   Integer, Rational, Float, Symbol, symbol,
                   srepr, Tuple
                  )
init_printing()
```

### Functions for plot

```python
import matplotlib.pyplot as plt
```

```python
def plot_results(data, label=None, labelx=True, title=None, save_file=None, normalisation=None,
                 selected_times=None,style=None, name=None, alpha=1., bolds=[0., 1.]):

    normalisation = 1. if normalisation is None else normalisation

    selected_times = [time for time in data] if selected_times is None else selected_times

    style = 'k' if style is None else style

    for time in selected_times:
        lalpha = alpha if time in bolds else 0.2
        lname = name if time==selected_times[-1] else None
        plt.plot(domain.x[0],data[time]/normalisation, style, alpha = lalpha, label=lname)

    if labelx:
        plt.xlabel('$x/D$', fontsize=15)
    if label:
        plt.ylabel(label, fontsize=15)
    if title:
        plt.title(title)
    if save_file:
        plt.savefig(save_file)
```

## The Burgers dynamics and its unclosed PKF formulation

```python
from pdenetgen import NNModelBuilder, Eq
```

```python
def display_system(system):
    print(50*'*')
    for equation in system:
        display(equation)
        print(50*'*')
```

#### Set of the function and symbols

```python
t, x = symbols('t x')

u = Function('u')(t,x)
closure = sympy.Function('closure')(t,x)
V = Function('{V_{u}}')(t,x)
nu = Function('{\\nu_{u,xx}}')(t,x)
Kappa = symbols('\\kappa')
```

#### Set constants for numerical experiments

```python
# Constant setting following Pannekoucke et al. (2018)
n = 241
kappa = 0.0025
dt = 0.002
```

#### Set of the Burgers equation

```python
burgers_dynamics = [
        Eq(
        Derivative(u,t),
        Kappa*Derivative(u,x,2)-u*Derivative(u,x)
      ),
]
display_system(burgers_dynamics)
```

```python
burgers_NN_builder = NNModelBuilder(burgers_dynamics, "Burgers")
print(burgers_NN_builder.code)
exec(burgers_NN_builder.code)
burgers = Burgers(shape=(n,), kappa=kappa)
```

##### Example of forecast from a given initial condition

```python
domain = burgers
# Set initial condition for 'u'
U0=0.25*( 1+np.cos(2*np.pi/ domain.lengths[0]  *(domain.x[0]-0.25)) )
Umax = U0.max()
```

```python
burgers.set_dt(dt)
end_time_forecast = 1.
times = burgers.window(end_time_forecast)
saved_times = times[::50]
print('saved_times :' ,saved_times)
```

```python
forecast = burgers.forecast(times, np.array([U0.reshape((1,)+U0.shape+(1,)) ]))
```

```python
for time in times:
    plt.plot(domain.x[0], forecast[time][0,0,:,0])
```

#### Set of the PKF equations for the Burgers equation

```python
# From Pannekoucke et al. (2018)

pkf_dynamics = [
    # Trend of the expectation of 'u'
    Eq(
        Derivative(u,t),
        Kappa*Derivative(u,x,2)-u*Derivative(u,x)-Derivative(V,x)/Integer(2)
      ),
    # Trend of the variance
    Eq(
        Derivative(V,t),
        -Kappa*V/nu + Kappa*Derivative(V,x,2)-Kappa*Derivative(V,x)**Integer(2)/(Integer(2)*V)
        -u*Derivative(V,x)-Integer(2)*V*Derivative(u,x)
      ),
    # Trend of the diffusion
    Eq(
        Derivative(nu,t),
        Integer(4)*Kappa*nu**Integer(2)*closure
        -Integer(3)*Kappa*Derivative(nu,x,2)
        -Kappa
        +Integer(6)*Kappa*Derivative(nu,x)**Integer(2)/nu
        -Integer(2)*Kappa*nu*Derivative(V,x,2)/V
        +Kappa*Derivative(V,x)*Derivative(nu,x)/V
        +Integer(2)*Kappa*nu*Derivative(V,x)**Integer(2)/V**Integer(2)
        -u*Derivative(nu,x)
        +Integer(2)*nu*Derivative(u,x)
    )
]

display_system(pkf_dynamics)
```

```python
pkf_NN_builder = NNModelBuilder(pkf_dynamics,'NN_Unclosed_PKF_Burgers')
```

```python
print(pkf_NN_builder.code)
exec(pkf_NN_builder.code)
```

#### Construction of a closure as a NN from parameterized form

  L'objectif est de calculer le terme
  $$a\frac{\frac{\partial^{2}}{\partial x^{2}} \operatorname{{\nu_{u,xx}}}{\left(t,x \right)}}{\operatorname{{\nu_{u,xx}}}^{2}{\left(t,x \right)}} +b \frac{1}{ \operatorname{{\nu_{u,xx}}}^{2}{\left(t,x \right)}} +c\frac{ \left(\frac{\partial}{\partial x} \operatorname{{\nu_{u,xx}}}{\left(t,x \right)}\right)^{2}}{\operatorname{{\nu_{u,xx}}}^{3}{\left(t,x \right)}},$$
  sous la forme d'une fonction exogène.

```python
class ClosedPKFBurgers(NN_Unclosed_PKF_Burgers):
    def _make_exogenous_model(self):

        u = keras.layers.Input(shape=(self.input_shape_x,1))
        V = keras.layers.Input(shape=(self.input_shape_x,1))
        nu_u_xx = keras.layers.Input(shape=(self.input_shape_x,1))

        #
        # Computation of the spatial derivatives
        #

        kernel_Dnu_u_xx_x_o2 = np.asarray([self.dx[self.coordinates.index('x')]**(-2),
        -2/self.dx[self.coordinates.index('x')]**2,
        self.dx[self.coordinates.index('x')]**(-2)]).reshape((3,)+(1,1))
        Dnu_u_xx_x_o2 = DerivativeFactory((3,),kernel=kernel_Dnu_u_xx_x_o2,name='Dnu_u_xx_x_o2')(nu_u_xx)

        kernel_Dnu_u_xx_x_o1 = np.asarray([-1/(2*self.dx[self.coordinates.index('x')]),0.0,
        1/(2*self.dx[self.coordinates.index('x')])]).reshape((3,)+(1,1))
        Dnu_u_xx_x_o1 = DerivativeFactory((3,),kernel=kernel_Dnu_u_xx_x_o1,name='Dnu_u_xx_x_o1')(nu_u_xx)

        #
        # Design of the unknown closure to train
        #

        # Terme 1
        div_14 = keras.layers.Lambda(lambda x: 1/x,name='DivLayer_14')(nu_u_xx)
        pow_12 = keras.layers.multiply([div_14,div_14,] ,name='PowLayer_12')
        term1 = keras.layers.multiply([pow_12,Dnu_u_xx_x_o2],name='MulLayer_25')

        # Terme 2
        div_13 = keras.layers.Lambda(lambda x: 1/x,name='DivLayer_13')(nu_u_xx)
        term2 = keras.layers.multiply([div_13,div_13,] ,name='PowLayer_11')

        # Terme 3
        pow_13 = keras.layers.multiply([Dnu_u_xx_x_o1,Dnu_u_xx_x_o1,] ,name='PowLayer_13')
        div_15 = keras.layers.Lambda(lambda x: 1/x,name='DivLayer_15')(nu_u_xx)
        pow_14 = keras.layers.multiply([div_15,div_15,div_15,] ,name='PowLayer_14')
        term3 = keras.layers.multiply([pow_13,pow_14],name='MulLayer_26')

        # Product by (a,b,c), implemented as Conv1D
        term1 = keras.layers.Conv1D(1,1,name='times_a',padding='same',use_bias=False,activation='linear')(term1)
        term2 = keras.layers.Conv1D(1,1,name='times_b',padding='same',use_bias=False,activation='linear')(term2)
        term3 = keras.layers.Conv1D(1,1,name='times_c',padding='same',use_bias=False,activation='linear')(term3)

        closure = keras.layers.add([term1, term2, term3],name='Closure')

        self._exogenous_model = keras.models.Model(inputs=[u,V,nu_u_xx], outputs=[closure])

    def compute_exogenous(self, t, state):

        if self._exogenous_model is None:
            self._make_exogenous_model()

        u,V,nu = state
        closure = self._exogenous_model.predict([u,V,nu])
        if not isinstance(closure, list):
            closure = [closure]
        return closure

    def _make_full_trend(self):

        if self._trend_model is None:
            self._make_trend_model()
        if self._exogenous_model is None:
            self._make_exogenous_model()

        state = keras.layers.Input(shape=(3,self.input_shape_x,1))

        u = keras.layers.Lambda(lambda x : x[:,0,:,:])(state)
        V = keras.layers.Lambda(lambda x : x[:,1,:,:])(state)
        nu_u_xx = keras.layers.Lambda(lambda x : x[:,2,:,:])(state)

        closure = self._exogenous_model([u,V,nu_u_xx])
        trend_u, trend_V, trend_nu = self._trend_model([u,V,nu_u_xx,closure])

        trend_u = keras.layers.Reshape((1,self.input_shape_x,1))(trend_u)
        trend_V = keras.layers.Reshape((1,self.input_shape_x,1))(trend_V)
        trend_nu = keras.layers.Reshape((1,self.input_shape_x,1))(trend_nu)

        trend = keras.layers.Concatenate(axis=1)([trend_u,trend_V,trend_nu])
        self._full_trend = keras.models.Model(inputs=state,outputs=trend)
```

```python
closed_burgers = ClosedPKFBurgers(shape=(241,),kappa=kappa)
```

```python
closed_burgers._make_full_trend()
```

**Set initial PKF fields**

```python
# Set initial condition for the variance parameter 'V_u'
V0 = (0.01*Umax)**2 + 0*U0

# Set the initial condition for the diffusion
# L**2 = 2nu t => nu = 0.5*L**2
lh = 0.02*domain.lengths[0]
nu0 = 0.5*lh**2 + 0*U0

state0 = np.asarray([U0, V0,nu0])
normalization = {
                'Velocity':U0.max(),
                'Variance':V0.max(),
                'Length-scale':lh
                }
```

```python
length_scale = lambda nu: np.sqrt(2*nu)
plt.figure(figsize=(12,12))
for k,field in enumerate(normalization):
    plt.subplot(221+k)
    if field=='Length-scale':
        data = {0:length_scale(state0[k])}
    else:
        data = {0:state0[k]}
    plot_results(data, label=field)

```

## Application numérique

```python
def plot_pkf_traj_ensemble(traj):
    plt.figure(figsize=(15,5))
    for k,field in enumerate(normalization):
        if field=='Length-scale':
            data = {time:length_scale(traj[time][k]) for time in traj}
        else:
            data = {time:traj[time][k] for time in traj}
        plt.subplot(131+k)
        plot_results(data,label=field,normalisation=normalization[field])
```

### Définition et utilisation de la dynamique fermée

```python
state0 = np.asarray([U0.reshape((1,)+U0.shape+(1,)),
                     V0.reshape((1,)+V0.shape+(1,)),
                     nu0.reshape((1,)+nu0.shape+(1,))])
```

```python
def plot_pkf_traj_NN(traj):
    plt.figure(figsize=(15,5))
    for k,field in enumerate(normalization):
        if field=='Length-scale':
            data = {time:length_scale(traj[time][k][0,:,0]) for time in traj}
        else:
                data = {time:traj[time][k][0,:,0] for time in traj}
        plt.subplot(131+k)
        plot_results(data,label=field,normalisation=normalization[field])

#plt.savefig("./figures/NN-PKF-closure_loc-gaussian.jpg")
```

## Generation of a database

##### **Gaussian random vector of Gaussian correlation function**

```python
# Création d'une matrice de covariance d'erreur de prévision initiale: $P_0$
#   Cette matrice est construite comme une matrice homogène de corrélation Gaussienne et de longueur de portée l_h

# 1) Définition de la fonction de corrélation homogène
gauss = lambda x : np.exp(-0.5*x**2/lh**2) # lh has been previously specified
correlation = gauss(domain.x[0]-domain.x[0][domain.shape[0]//2])
spectrum = np.abs(np.fft.fft(correlation))

# 2) Construction de B^(1/2)
std_spectrum = np.sqrt(spectrum)
def make_sample():
    zeta = np.random.normal(size=domain.shape)
    zeta = np.fft.fft(zeta)
    ef = np.fft.ifft(std_spectrum * zeta)
    ef = np.real(ef)
    return ef
```

```python
plt.figure(figsize=(12,5))
plt.subplot(121)
plt.plot(domain.x[0], correlation)
plt.title('Homogenous correlation function');
plt.subplot(122)
for k in range(10):
    plt.plot(domain.x[0], make_sample())
plt.title("Example of sample errors");
```

##### **Diagnosis tool for ensemble estimation of expectation/variance/diffusion tensor**

```python
def make_init_ensemble(Ne):
    return np.array([make_sample() for k in range(Ne)])

def estimate_covariance(ensemble):
    mean = ensemble.mean(0)
    error = (ensemble - mean)/np.sqrt(len(ensemble))
    return error.T @ error

class EnsembleDiagnosis(object):

    def __init__(self, ensemble, base_space):
        self.base_space = base_space

        if isinstance(ensemble, list):
            ensemble = np.array(ensemble)

        if len(ensemble.shape)==3:
            ensemble = np.array([elm[0] for elm in ensemble])

        # 1) Computation of the mean
        self.mean = ensemble.mean(axis=0)

        # 2) Computation of the variance
        self.std = ensemble.std(axis=0)
        self.variance = self.std*self.std

        # 3) Computation of the metric terms
        #  we use the formula g_ij = E[(D_i eps)(D_j eps)]

        #  a) Computation of the normalized error
        epsilon = (ensemble-self.mean)/self.std

        #  b) Computation of derivatives
        n = self.base_space.shape[0]
        K = np.arange(n)
        kp = (K+1)%n
        km = (K-1)%n
        dx = self.base_space.dx[0]
        Depsilon = np.array([(eps[kp]-eps[km])/(2*dx) for eps in epsilon])
        self.metric = (Depsilon*Depsilon).mean(axis=0)     # see Pannekoucke et al. (2018) for details

        # Computation of the diffusion tensor
        self.diffusion = 0.5*1/self.metric
        self.length_scale = np.sqrt(2*self.diffusion)
```

##### **Ensemble validation for the covariance setting**

```python
Ne = 1600

ensemble = make_init_ensemble(Ne)

mean = ensemble.mean(axis=0)
std = ensemble.std(axis=0)

print(f"Validation of the mean (=0): {mean.mean()} +/- {mean.std()}" )
print(f"Validation of the standard-deviation (=1): {std.mean()} +/- {std.std()}" )

ens_diagnosis = EnsembleDiagnosis(ensemble, domain)
nu_h = 0.5*lh**2

plt.figure(figsize=(15,5))

plt.subplot(131)
plt.plot(ens_diagnosis.mean)
plt.title('Moyenne')

plt.subplot(132)
plt.plot(ens_diagnosis.variance)
plt.title('Variance')

plt.subplot(133)
plt.plot(ens_diagnosis.diffusion/nu_h)
plt.title('diffusion (normalisée par $nu_h$)')
```

**Computation of a large ensemble (1600 members) to build a reference**

```python
# Standard deviation for the initial perturbation
sigma_f = 0.01*U0.max()
```

```python
# Set parameters for ensemble estimation
large_Ne = 1600

# 1. Set the initial background state
random_U0 = U0 + sigma_f*make_init_ensemble(1)[0]

# 2. Build an ensemble of initial perturbed state
ensemble = make_init_ensemble(large_Ne)
ensemble_ea = np.array([random_U0+sigma_f*ea for ea in ensemble])
ensemble_ea = ensemble_ea.reshape((1,)+ensemble_ea.shape+(1,))
print(f"shape of ensemble_ea: {ensemble_ea.shape}")

# 3. Build the ensemble of forecast using the NN architecture
ensemble_forecast = burgers.forecast(times,ensemble_ea)
```

```python
# 4. Compute diagnosis from ensemble
ensemble_traj = {}
for time in times[::50]:
    diagnosis = EnsembleDiagnosis(ensemble_forecast[time][0,:,:,0], domain)
    ensemble_traj[time] = [diagnosis.mean, diagnosis.variance, diagnosis.diffusion]
```

```python
plot_pkf_traj_ensemble(ensemble_traj)
```

#### **Generation of the training data set**

```python
def generate_data(k, Ne=400):
    # 1. Set the initial background state
    random_U0 = U0 + sigma_f*make_init_ensemble(1)[0]

    # 2. Build an ensemble of initial perturbed state
    ensemble = make_init_ensemble(Ne)
    ensemble_ea = np.array([random_U0+sigma_f*ea for ea in ensemble])
    ensemble_ea = ensemble_ea.reshape((1,)+ensemble_ea.shape+(1,))

    # 3. Compute the ensemble of forecasts
    ensemble_forecast = burgers.forecast(times,ensemble_ea)

    # 4. Compute the diagnosis
    diagnosis_list = []
    for time in times:
        diagnosis = EnsembleDiagnosis(ensemble_forecast[time][0,:,:,0], domain)
        diagnosis_list.append( np.array([diagnosis.mean, diagnosis.variance, diagnosis.diffusion]))

    return diagnosis_list
```

```python
data_size = 400  # for Ne=400, this takes 1h09'01'' so take care with this..
save_file = "pkf-dataset.npy"

generate_data_set = False
parallel_diagnosis = False

try:
    # load data
    data = np.load(save_file)
    data = data.reshape(data.shape+(1,))
except:
    # 1. Generate data
    #data = [generate_data(k) for k in range(data_size)]
    data = []
    for k in range(data_size):
        if k%5==0:
            print(k)
        data.append(generate_data(k))

    # 2. Save data
    data = np.array(data)
    np.save(save_file,data)

```

```python
data.shape
```

```python
def plot_training_data():
    plt.figure(figsize=(15,5))

    title = ['Ensemble mean','Variance','Diffusion']
    normalization = [1, sigma_f**2, 0.5*lh**2]

    for trajectory in data[:10]:
        for date in  trajectory[::100]:
            for k,field in enumerate(date):
                plt.subplot(131+k)
                plt.plot(domain.x[0], field/normalization[k] )
                plt.title(title[k])
    plt.savefig('./figures/NN-burgers-training-data.pdf')
plot_training_data()
```

#### Construction d'un schéma d'intégration RK4

```python
# Schéma temporelle de type RK4
def make_time_scheme(dt, trend):
    """ Implémentation d'un schéma de RK4 sous forme de réseau de neurones """
    import keras

    state = keras.layers.Input(shape = trend.input_shape[1:])

    # k1
    k1 = trend(state)
    # k2
    _tmp_1 = keras.layers.Lambda(lambda x : 0.5*dt*x)(k1)
    input_k2 = keras.layers.add([state,_tmp_1])
    k2 = trend(input_k2)
    # k3
    _tmp_2 = keras.layers.Lambda(lambda x : 0.5*dt*x)(k2)
    input_k3 = keras.layers.add([state,_tmp_2])
    k3 = trend(input_k3)
    # k4
    _tmp_3 = keras.layers.Lambda(lambda x : dt*x)(k3)
    input_k4 = keras.layers.add([state,_tmp_3])
    k4 = trend(input_k4)

    # output
    # k2+k3
    add_k2_k3 = keras.layers.add([k2,k3])
    add_k2_k3_mul2 = keras.layers.Lambda(lambda x:2.*x)(add_k2_k3)
    # Add k1,k4
    _sum = keras.layers.add([k1,add_k2_k3_mul2,k4])
    # *dt
    _sc_mul = keras.layers.Lambda(lambda x:dt/6.*x)(_sum)
    output = keras.layers.add([state, _sc_mul])

    time_scheme = keras.models.Model(inputs =[state], outputs=[output])
    return time_scheme
```

```python
closed_pkf_burgers = ClosedPKFBurgers(shape=(241,),kappa=kappa)
```

```python
closed_pkf_burgers._make_full_trend()
closed_pkf_burgers._full_trend.summary()
```

```python
closed_pkf_burgers._exogenous_model.summary()
```

```python
time_scheme = make_time_scheme(dt, closed_pkf_burgers._full_trend)
#time_scheme.summary()
```

#### Constitution de la base de données d'apprentissage

```python
data[0].shape
```

```python
select_from = 400 # 200
X = np.array([elm[select_from:-1] for elm in data])
Y = np.array([elm[select_from+1:] for elm in data])
```

```python
X = X.reshape((np.prod(X.shape[:2]),3,241,1))
Y = Y.reshape((np.prod(Y.shape[:2]),3,241,1))
```

```python
X.shape
```

#### **Training of the NN**

```python
# Expérience d'apprentissage:
# 2. Adam
lr = 0.1 # old value: 0.001
epochs = 30

for iteration in range(3):
    # 1. Set the learning
    time_scheme.compile(optimizer=keras.optimizers.Adam(lr=lr),
        loss='mean_squared_error') # Ne permet pas la convergence

    # 2. Train
    history = time_scheme.fit(X,Y,epochs=epochs, batch_size=32,verbose=0)
    print(f"iteration {iteration} is complet")

    # 3. Plot history
    plt.figure()
    plt.plot(history.history['loss'])

    # 4. Update the learning rate for next iteration
    lr = lr/10

```

**Compare the weights with the previous theoretical closure**

The weights of the theoretical closure are : 1, 3/4, -2

```python
trained = unclosed_burgers._full_trend.get_weights()
trained = np.array((trained[0], trained[1], trained[2])).flatten()
trained
```

```python
theoretical = np.array([1,3/4,-2])
```

```python
relative_error = (trained - theoretical)/theoretical
```

```python
relative_error*100
```

**Exemple de prévision réalisée avec le modèle calibré**

```python
# default
unclosed_burgers.set_dt(dt)
times = unclosed_burgers.window(1)
#saved_times = times[::25]
saved_times = times[::50]
print('saved_times :' ,saved_times)
```

```python
trained_unclosed_traj = unclosed_burgers.forecast(times, state0, saved_times)
```

```python
normalization
```

```python
# PKF using trained closure
plot_pkf_traj_NN(trained_unclosed_traj)
plt.savefig('./figures/burgers-b.pdf')
```

```python
# ensemble of forecast statistics
plot_pkf_traj_ensemble(ensemble_traj)
plt.savefig('./figures/burgers-a.pdf')
```

## Conclusion

    In this notebook, a closure for the Burgers dynamics has been learned from the data.

## Appendix

```python
from IPython.core.display import HTML
with open("css/lecture.css",'r') as css_file:
    css_style = css_file.read()
HTML(css_style)
```


Παρακάτω προσπαθώ να παρουσιάσω την ανάλυση του *Hogg et al 2010 - Data analysis Recipes* η οποία μας λέει από το abstract:

Χρησιμοποιούμε πάρα πολλές *considerations involved* ώστε να προσαρμόσουμε ένα μοντέλο σε δεδομένα, όπως για παράδειγμα τη προσαρμογή μιας ευθείας σε σημεία δύο διαστάσεων. Η standard μέθοδος των ελαχίστων τετραγώνων (wls) θεωρεί οτι τα σφάλματα των σημείων στη μια διάσταση είναι *neglible* ενώ στην άλλη περιγράφονται απο Gaussians με γνωστή απόκλιση, υποθέσεις που στη πραγματικότητα συναντώνται εξαιρετικά σπάνια.

Θεωρούμε τη περίπτωση ενός ετερογενούς δείγματος δεδομένων με άγνωστα σφάλματα σε δύο διαστάσεις και περιπτώσεις όπου συμπεριλαμβάνονται κακά δεδομένα (outliers) τα οποία εμφανίζουν γραμμική συσχέτιση.

*Above all we emphasize the importance of having a “generative model” for the data, even an approx-
imate one. Once there is a generative model, the subsequent fitting
is non-arbitrary because the model permits direct computation of the
likelihood of the parameters or the posterior probability distribution.
Construction of a posterior probability distribution is indispensible if
there are “nuisance parameters” to marginalize away.*

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import rc,rcParams
rc('text', usetex=True)
rcParams['figure.figsize'] = (8, 6.5)
rcParams['ytick.labelsize'],rcParams['xtick.labelsize'] = 17.,17.
rcParams['axes.labelsize']=19.
rcParams['legend.fontsize']=17.
rcParams['axes.titlesize']=20.
rcParams['text.latex.preamble'] = ['\\usepackage{siunitx}']
import seaborn
seaborn.despine()
seaborn.set_style('white', {'xes.linewidth': 0.5, 'axes.edgecolor':'black'})
seaborn.despine(left=True)
%load_ext autoreload
from astroML.linear_model import TLS_logL
from astroML.plotting.mcmc import convert_to_stdev
from astroML.datasets import fetch_hogg2010test
from astropy.table import Table
from scipy import polyfit,linalg
from scipy.optimize import curve_fit
import scipy.stats as sts
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

    <matplotlib.figure.Figure at 0x7f4778569a90>

Η βιβλιοθήκη astroML (που μπορει να εγκατασταθεί έυκολα μέσω της ```sudo pip install astroml``` εμπεριέχει τα δεδομένα που χρησιμοποιούνται στο paper. Παρόλαυτα όποιος τα θέλει αλλιώς ή δεν θέλει/μπορεί να εγκαταστήσει τη βιβλιοθήκη μπορεί να μου τα ζητήσει.

```python
data = fetch_hogg2010test()
data = data[5:]  # no outliers
x = data['x']
y = data['y']
sigma_x = data['sigma_x']
sigma_y = data['sigma_y']
rho_xy = data['rho_xy']
```

```python
np.savetxt('data',data)
```

```python
t = Table([x, y, sigma_x,sigma_y,rho_xy], names=('X', 'Y', '$\sigma _x$','$\sigma _y$','$\\rho _{xy}$'))#, meta={'name': 'first table'})
t
```

Table length=15

X	Y	$\sigma _x$	$\sigma _y$	$\rho _{xy}$

float64	float64	float64	float64	float64

58.0	173.0	9.0	15.0	0.67

210.0	479.0	4.0	27.0	-0.02

202.0	504.0	4.0	14.0	-0.05

198.0	510.0	11.0	30.0	-0.84

158.0	416.0	7.0	16.0	-0.69

165.0	393.0	5.0	14.0	0.3

201.0	442.0	5.0	25.0	-0.46

157.0	317.0	5.0	52.0	-0.03

131.0	311.0	6.0	16.0	0.5

166.0	400.0	6.0	34.0	0.73

160.0	337.0	5.0	31.0	-0.52

186.0	423.0	9.0	42.0	0.9

125.0	334.0	8.0	26.0	0.4

218.0	533.0	6.0	16.0	-0.78

146.0	344.0	5.0	22.0	-0.56

```python
plt.errorbar(x,y,yerr=sigma_y,xerr=sigma_x,fmt='o')
```

Έχουμε ένα σύνολο $N>2$ σημείων $(x_i,y_i)$ με γνωστές gaussian αποκλίσεις $\sigma _{yi}$ στη $y$ διεύθυνση και χωρίς καμία απροσδιοριστία στη $x$ διεύθυνση.

Αναζητούμε τη συνάρτηση $f(x)$ της μορφής: $ f(x)=\theta _1 x+\theta _0 $ η οποία προσαρμόζεται καλύτερα στα σημεία.

Κατασκευάζουμε τους πίνακες
\begin{align}
\mathbf{Y}&=\begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_N\end{pmatrix} \\
\mathbf{A}&=\begin{pmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_N \end{pmatrix} \\
\mathbf{C}&=\begin{pmatrix}
\frac{1}{\sigma ^2_{y1}} & 0 & 0 & \cdots & 0 \\
0 & \frac{1}{\sigma ^2_{y2}} & 0 & \cdots & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & 0 & \cdots & \frac{1}{\sigma ^2_{yN}} \end{pmatrix}
\end{align}

Η λύση καλύτερης προσαρμογης δίνεται από τη:
\begin{equation}
\begin{pmatrix} b \\ m \end{pmatrix} = \mathbf{S}= [\mathbf{A}^T \mathbf{C} \mathbf{A}]^{-1} [\mathbf{A}^T \mathbf{C} \mathbf{Y}]
\end{equation}

```python
Y=y
A=np.stack((np.ones(x.shape[0]),x),axis=-1)
C=np.diagflat(1./sigma_y**2)
```

```python
first=np.linalg.inv(A.T.dot(C.dot(A)))
second=A.T.dot(C.dot(Y))
M=np.dot(first,second)
print(M)
```

    [34.84594131  2.23253622]

***Ερώτηση:*** Πως μπορούμε να υπολογίσουμε τα σφάλματα μέσω αναλυτικής έκφρασης?

```python
cov = np.linalg.inv(np.dot(A.T, np.linalg.solve(C, A)))
b_ls, m_ls = np.dot(cov, np.dot(A.T, np.linalg.solve(C, y)))
```

```python
b_ls,m_ls
```

    (-8.187190205908337, 2.335838244213832)

## Εκτίμηση των παραμέτρων προσαρμογής μέσω optimization
Μέσω της βιβλιοθήκης ```scipy.optimization```

## Εκτίμηση των παραμέτρων προσαρμογής μέσω της μεθόδου ελαχίστων τετραγώνων
Μέσω της βιβλιοθήκης ```scipy.curve_fit```

```python
def f(x,b,m): return b+m*x
Mcf,dMcf2=curve_fit(f,x,y,[2.,30.],sigma=sigma_y)
dMcf=np.sqrt(np.diag(dMcf2))
print(Mcf, dMcf)
print('Y=({:.1f} +/- {:.1f}) + ({:.1f} +/- {:.2f}) X'.format(Mcf[0],dMcf[0],Mcf[1],dMcf[1]))
#----PLOT----------------
xx=np.linspace(x.min(),x.max())
plt.errorbar(x,y,yerr=sigma_y,fmt='o')
plt.plot(xx,f(xx,Mcf[0],Mcf[1]),label='Scipy Curve Fit')
plt.fill_between(xx,f(xx,Mcf[0]-2.*dMcf[0],Mcf[1]-2.*dMcf[1]),f(xx,Mcf[0]+2.*dMcf[0],Mcf[1]+2.*dMcf[1]),alpha=0.25,label='95\% Confidence')
#plt.fill_between(xx,f(xx,Mcf[0]-2.*dMcf[0],Mcf[1]),f(xx,Mcf[0]+2.*dMcf[0],Mcf[1]),alpha=0.25,label='95\% Confidence')
#plt.fill_between(xx,f(xx,Mcf[0],Mcf[1]-2.*dMcf[1]),f(xx,Mcf[0],Mcf[1]+2.*dMcf[1]),alpha=0.25,label='95\% Confidence')
plt.legend(loc='best')
```

```python
N=len(x)
d=N-2
```

```python
NN=200
bb=np.linspace(-40,110,NN)
mm=np.linspace(1.75,2.7,NN)
LogL=np.zeros((len(bb),len(mm)))
for i,b in enumerate(bb):
    for j,m in enumerate(mm):
        LogL[i,j]=np.sum((y-f(x,b,m))**2/sigma_y**2)
print(LogL.min(),LogL.min()/d,(LogL.min()/d-1.)/np.sqrt(2./d))
ax=plt.subplot()
c=ax.contour(LogL,np.linspace(LogL.min(),1.8*LogL.min(),5),extent=[mm.min(),mm.max(),bb.min(),bb.max()])
c2=ax.contourf(LogL,[LogL.min(),LogL.min()+1.],extent=[mm.min(),mm.max(),bb.min(),bb.max()],label='$\chi ^2+1$')
ax.set_xlabel('slope')
ax.set_ylabel('intercept')
ax.vlines(Mcf[1],bb.min(),bb.max(),linestyles='--',alpha=0.2)
ax.hlines(Mcf[0],mm.min(),mm.max(),linestyles='--',alpha=0.2)
ax.vlines(Mcf[1]+dMcf[1],Mcf[0],Mcf[0]+dMcf[0],linestyles='--',alpha=0.7)#,label=r'Intercept $\sigma$')
ax.hlines(Mcf[0]-dMcf[0],Mcf[1],Mcf[1]-dMcf[1],linestyles='--',alpha=0.7)#,label=r'Slope $\sigma$')
ax.plot([Mcf[1]],[Mcf[0]],'o',label='Minimum at [{:.2f},{:.2f}]'.format(Mcf[1],Mcf[0]))
ax.set_title('$\chi ^2$ Contours')
plt.colorbar(c,ax=ax)
ax.legend()
```

Εδώ προσπαθώ να υπολογίσω τη παράμετρο p. Η παράμετρος p ουσιαστικά είναι η πιθανότητα η κάθε παράμετρος να μην παίζει ρόλο. Δηλαδή αν για παράδειγμα η ευθεία ειχε κλίση αλλα το σφάλμα της κλίση είναι μέγαλο τότε υπάρχει μια πιθανότητα η κλιση να είναι 0 άρα ουσιαστικά τα δεδομένα να μην έχουν αυτο το βαθμό ελευθερίας που εμεις θεωρούμε.
Τέλος Πάντων είναι λίγο άσχετο με την όλη θεματολογια και μπορεί να το βγάλω στο τέλος.

```python
from scipy.stats import chisquare
```

```python
y,f(x,Mcf[0],Mcf[1])
```

    (array([173., 479., 504., 510., 416., 393., 442., 317., 311., 400., 337.,
            423., 334., 533., 344.]),
     array([164.3330419 , 503.67854688, 485.81825715, 476.88811228,
            387.5866636 , 403.21441712, 483.58572093, 385.35412738,
            327.30818574, 405.44695334, 392.05173603, 450.09767768,
            313.91296844, 521.53883662, 360.796229  ]))

```python
chisquare(f_obs=y,f_exp=f(x,Mcf[0],Mcf[1]),ddof=1)
```

    Power_divergenceResult(statistic=35.25487445723004, pvalue=0.0007736005394805199)

```python
s=np.random.standard_t(d, size=100000)
tt=np.array([Mcf[0]/dMcf[0]/np.sqrt(N),Mcf[1]/dMcf[1]/np.sqrt(N)])
print np.sum(s>tt[0])/float(len(s))+np.sum(s<-tt[0])/float(len(s))
print np.sum(s>tt[1])/float(len(s))+np.sum(s<-tt[1])/float(len(s))
```

    0.68883
    0.00068

```python
def f2(x,b,m1,m2): return b+m1*x+m2*x**2
M2cf,dM2cf2=curve_fit(f2,x,y,[2.,30.,2],sigma=sigma_y)
dM2cf=np.sqrt(np.diag(dM2cf2))
print(M2cf, dM2cf)
print('Y=({:.1f} +- {:.1f}) + ({:.1f} +- {:.2f}) X+({:.3f} +- {:.4f}) X^2'.format(M2cf[0],dM2cf[0],M2cf[1],dM2cf[1],M2cf[2],dM2cf[2]))
xx=np.linspace(x.min(),x.max())
plt.errorbar(x,y,yerr=sigma_y,fmt='o')
plt.plot(xx,f2(xx,M2cf[0],M2cf[1],M2cf[2]),label='Scipy Curve Fit')
plt.fill_between(xx,f2(xx,M2cf[0]-2.*dM2cf[0],M2cf[1]-2.*dM2cf[1],M2cf[2]-2.*dM2cf[2]),
                 f2(xx,M2cf[0]+2.*dM2cf[0],M2cf[1]+2.*dM2cf[1],M2cf[2]+2.*dM2cf[2]),
                 alpha=0.25,label='95\% Confidence')
plt.legend(loc='best')
#plt.ylim([0,700])
```

```python
tt2
```

    array([0.39977454, 0.59171963, 0.23587458])

```python
N=x.shape[0]
d=N-3
s=np.random.standard_t(d, size=100000)
tt2=np.array([M2cf[0]/dM2cf[0]/np.sqrt(N),M2cf[1]/dM2cf[1]/np.sqrt(N),M2cf[2]/dM2cf[2]/np.sqrt(N)])
plt.hist(s, bins=100, normed=True)
plt.vlines(tt2[2],0,0.4)
print(tt2)
print(np.sum(s>tt2[0])/float(len(s))+np.sum(s<-tt2[0])/float(len(s)))
print(np.sum(s>tt2[1])/float(len(s))+np.sum(s<-tt2[1])/float(len(s)))
print(np.sum(s>tt2[2])/float(len(s))+np.sum(s<-tt2[2])/float(len(s)))
```

# Data with outliers

```python
data = fetch_hogg2010test()
x = data['x']
y = data['y']
sigma_x = data['sigma_x']
sigma_y = data['sigma_y']
rho_xy = data['rho_xy']
plt.errorbar(x,y,yerr=sigma_y,xerr=sigma_x,fmt='o')
```

```python
Mcft,dMcf2t=curve_fit(f,x,y,[2.,30.],sigma=sigma_y)
dMcft=np.sqrt(np.diag(dMcf2t))
print Mcft, dMcft
print 'Y=({:.1f} +- {:.1f}) + ({:.1f} +- {:.2f}) X'.format(Mcft[0],dMcft[0],Mcft[1],dMcft[1])
xx=np.linspace(x.min(),x.max())
plt.errorbar(x,y,yerr=sigma_y,fmt='o')
plt.plot(xx,f(xx,Mcft[0],Mcft[1]),label='Scipy Curve Fit')
plt.fill_between(xx,f(xx,Mcft[0]-2.*dMcft[0],Mcft[1]),f(xx,Mcft[0]+2.*dMcft[0],Mcft[1]),color='g',alpha=0.3,label='95\% Confidence')
plt.fill_between(xx,f(xx,Mcft[0],Mcft[1]-2.*dMcft[1]),f(xx,Mcft[0],Mcft[1]+2.*dMcft[1]),color='g',alpha=0.3,label='95\% Confidence')
plt.plot(xx,f(xx,Mcf[0],Mcf[1]),'--',label='Scipy Curve Fit (No outliers)')
plt.legend(loc='best')
```

```python
N=200
bb=np.linspace(-40,320,N)
mm=np.linspace(0.5,2.4,N)
LogL=np.zeros((len(bb),len(mm)))
for i,b in enumerate(bb):
    for j,m in enumerate(mm):
        LogL[i,j]=np.sum((y-f(x,b,m))**2/sigma_y**2)
print LogL.min(),LogL.mean()
ax=plt.subplot()
c=ax.contour(LogL,np.linspace(LogL.min(),1.8*LogL.min(),5),extent=[mm.min(),mm.max(),bb.min(),bb.max()])
c2=ax.contourf(LogL,[LogL.min(),LogL.min()+5],extent=[mm.min(),mm.max(),bb.min(),bb.max()],label='$\chi ^2+1$')
ax.set_xlabel('slope')
ax.set_ylabel('intercept')
ax.vlines(Mcft[1],bb.min(),bb.max(),linestyles='--',alpha=0.2)
ax.hlines(Mcft[0],mm.min(),mm.max(),linestyles='--',alpha=0.2)
ax.vlines(Mcft[1]+dMcft[1],Mcft[0],Mcft[0]+dMcft[0],linestyles='--',alpha=0.7)#,label=r'Intercept $\sigma$')
ax.hlines(Mcft[0]-dMcft[0],Mcft[1],Mcft[1]-dMcft[1],linestyles='--',alpha=0.7)#,label=r'Slope $\sigma$')
ax.plot([Mcft[1]],[Mcft[0]],'o',label='Minimum at [{:.2f},{:.2f}]'.format(Mcft[1],Mcft[0]))
ax.set_title('$\chi ^2$ Contours')
plt.colorbar(c,ax=ax)
ax.legend()
```

```python
def f2(x,b,m1,m2): return b+m1*x+m2*x**2
M2cf,dM2cf2=curve_fit(f2,x,y,[2.,30.,2],sigma=sigma_y)
dM2cf=np.sqrt(np.diag(dM2cf2))
print M2cf, dM2cf
print 'Y=({:.1f} +- {:.1f}) + ({:.1f} +- {:.2f}) X+({:.3f} +- {:.4f}) X^2'.format(M2cf[0],dM2cf[0],M2cf[1],dM2cf[1],M2cf[2],dM2cf[2])
xx=np.linspace(x.min(),x.max())
plt.errorbar(x,y,yerr=sigma_y,fmt='o')
plt.plot(xx,f2(xx,M2cf[0],M2cf[1],M2cf[2]),label='Scipy Curve Fit')
plt.fill_between(xx,f2(xx,M2cf[0]-2.*dM2cf[0],M2cf[1],M2cf[2]),f2(xx,M2cf[0]+2.*dM2cf[0],M2cf[1],M2cf[2]),alpha=0.25,label='95\% Confidence')
plt.fill_between(xx,f2(xx,M2cf[0],M2cf[1]-2.*dM2cf[1],M2cf[2]),f2(xx,M2cf[0],M2cf[1]+2.*dM2cf[1],M2cf[2]),alpha=0.25,label='95\% Confidence')
plt.fill_between(xx,f2(xx,M2cf[0],M2cf[1],M2cf[2]-2.*dM2cf[2]),f2(xx,M2cf[0],M2cf[1],M2cf[2]+2.*dM2cf[2]),alpha=0.25,label='95\% Confidence')
plt.legend(loc='best')
#plt.ylim([0,700])
```

```python

```