# Lecture 24 - Bayesian Model Selection using Sequential Monte Carlo

## Objectives

+ Compute the model evidence using [pysmc](https://github.com/PredictiveScienceLab/pysmc).
+ Do model selection with [pysmc](https://github.com/PredictiveScienceLab/pysmc).

```python
import numpy as np
import math
import scipy.stats as st
import scipy
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
mpl.rcParams['figure.dpi'] = 300
import seaborn as sns
sns.set_style('white')
sns.set_context('talk')
#import design
#import orthpol
import warnings
warnings.filterwarnings('ignore')
#from tqdm import tqdm # pip install tqdm (or conda)
import sklearn
import pymc as pm
import pysmc as ps

# We need a generic function that computes the design matrix
def compute_design_matrix(X, phi):
    """
    Arguments:

    X   -  The observed inputs (1D array)
    phi -  The basis functions.
    """
    num_observations = X.shape[0]
    num_basis = phi.num_basis
    Phi = np.ndarray((num_observations, num_basis))
    for i in xrange(num_observations):
        Phi[i, :] = phi(X[i, :])
    return Phi

class PolynomialBasis(object):
    """
    A set of linear basis functions.

    Arguments:
    degree  -  The degree of the polynomial.
    """
    def __init__(self, degree):
        self.degree = degree
        self.num_basis = degree + 1
    def __call__(self, x):
        return np.array([x[0] ** i for i in range(self.degree + 1)])

class FourierBasis(object):
    """
    A set of linear basis functions.

    Arguments:
    num_terms  -  The number of Fourier terms.
    L          -  The period of the function.
    """
    def __init__(self, num_terms, L):
        self.num_terms = num_terms
        self.L = L
        self.num_basis = 2 * num_terms
    def __call__(self, x):
        res = np.ndarray((self.num_basis,))
        for i in xrange(num_terms):
            res[2 * i] = np.cos(2 * i * np.pi / self.L * x[0])
            res[2 * i + 1] = np.sin(2 * (i+1) * np.pi / self.L * x[0])
        return res

class RadialBasisFunctions(object):
    """
    A set of linear basis functions.

    Arguments:
    X   -  The centers of the radial basis functions.
    ell -  The assumed lengthscale.
    """
    def __init__(self, X, ell):
        self.X = X
        self.ell = ell
        self.num_basis = X.shape[0]
    def __call__(self, x):
        return np.exp(-.5 * (x - self.X) ** 2 / self.ell ** 2).flatten()
```

## Sanity check - Does the calculation of the evidence with PySMC work?

Let
$$
p(\theta) = \mathcal{N}(\theta|0, 1),
$$
and
$$
p(y|\theta) = \mathcal{N}(y|\theta,0).
$$
The posterior of $\theta$ given $y$ is:
$$
p(\theta|y) = \frac{p(y|\theta)p(\theta)}{Z},
$$
where
$$
Z = \int_{-\infty}^{\infty} p(y|\theta)p(\theta)d\theta.
$$
Let's first calculate $Z$ analytically.

```python
import sympy
import sympy.stats
sympy.init_printing()
y, t = sympy.symbols('y \\theta')
q = 1. / sympy.sqrt(2. * sympy.pi) * sympy.exp(-0.5 * (y - t) ** 2) * \
    1. / sympy.sqrt(2. * sympy.pi) * sympy.exp(-0.5 * t ** 2)
sympy.simplify(sympy.integrate(q, (t, -sympy.oo, sympy.oo)))
```

So, if the observed $y$ was zero, then the Z should be:
$$
Z = \frac{1}{2\sqrt{\pi}}.
$$

```python
Z = 1 /  2. / np.sqrt(np.pi)
print 'log Z = {0:1.3e}'.format(np.log(Z))
```

    log Z = -1.266e+00

All, right. Now, let's program this thing in pysmc and compare the results.

We start with the model:

```python
def make_model(y=0.):
    # The gamma parameter
    gamma = 1.

    # The prior over theta
    theta = pm.Normal('$\theta$', mu=0., tau=1., value=0.)

    # The likelihood term - which must contain gamma in it
    @pm.stochastic(observed=True)
    def likelihood(value=y, gamma=gamma, theta=theta):
        return gamma * pm.normal_like(value, theta, 1.)

    return locals()
```

Now we initialize the model and run PySMC.

```python
model = make_model(y=0.)
mcmc = pm.MCMC(model)
mcmc.use_step_method(ps.RandomWalk, model['theta'])
smc = ps.SMC(mcmc, num_particles=100, num_mcmc=1, verbose=4, ess_reduction=0.9)
smc.initialize(0.)
smc.move_to(1.)
```

    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.9
    - tuning the MCMC parameters:
    	- tuning step method: <pysmc._step_methods.RandomWalk object at 0x107b1a990>
    - finding next gamma.
    - moving to gamma : 0.885585220454
    - performing 1 MCMC step(s) per particle
    - ESS  = 90.00 %
    - logZ = -1.089e+00
     [-----------------100%-----------------] 99 of 99 complete in 0.0 sec
    - acceptance rate for each step method:
    	- <pysmc._step_methods.RandomWalk object at 0x107b1a990> : 0.93
    - tuning the MCMC parameters:
    	- tuning step method: <pysmc._step_methods.RandomWalk object at 0x107b1a990>

    		adaptive_scale_factor: 1.1

    		SUCCESS
    - finding next gamma.
    - 	we can move directly to the target gamma...
    - moving to gamma : 1.0
    - performing 1 MCMC step(s) per particle
    - ESS  = 88.75 %
    - logZ = -1.299e-01
     [-----------------100%-----------------] 99 of 99 complete in 0.0 sec
    - acceptance rate for each step method:
    	- <pysmc._step_methods.RandomWalk object at 0x107b1a990> : 0.98
    - total number of MCMC steps: 200
    ---------------
    END SMC MOVE TO
    ---------------

Here are the log Z ratios that we gathered along the way:

```python
print smc.log_Zs
```

    [-1.0886775023625104, -0.12993101938408813]

```python

```

The evidence is the sum of these:

```python
print 'Log Z SMC estiamte: {0:1.6e}'.format(np.sum(smc.log_Zs))
```

    Log Z SMC estiamte: -1.218609e+00

Which is close to the truth.

## Questions
+ Repeat the calculations above for a varying number of SMC particles. Start from 10 and go up to 10,000.

## Polynomial Regression

We are going to implement a standard Bayesian linear regression using pymc and train it with pysmc.
We will compute the evidence in order to select the best class of basis functions.

The model is as follows. The output $y$ conditioned on the input $x$, the weights of the basis functions $w$ and
the noise variance $\sigma^2$ has likelihood:
$$
p(y|x,w,\sigma, \mathcal{M}) = \mathcal{N}(y|w^T\phi_{\mathcal{M}}(x), \sigma^2),
$$
where $\phi_{\mathcal{M},1}(\cdot), \dots, \phi_{\mathcal{M},m_{\mathcal{M}}}(\cdot)$ are the
$m_{\mathcal{M}}$ basis functions of the model $\mathcal{M}$.
We put a normal prior on the weights:
$$
p(w|\alpha) = \mathcal{N}(w|0, \alpha I_{m_{\mathcal{M}}}),
$$
and an inverse Gamma prior for $\sigma$ and $\alpha$:
$$
p(\sigma^2) = \mbox{IG}(\sigma^2|1, 1),
$$
and
$$
p(\alpha) = \mbox{IG}(\alpha|1,1).
$$

Assume that the data we have observed are:
$$
x_{1:n} = \{x_1,\dots,x_n\},\;\mbox{and}\;y_{1:n} = \{y_1,\dots,y_n\}.
$$
Consider the design matrix $\Phi_{\mathcal{M}}\in\mathbb{R}^{n\times m}$:
$$
\Phi_{\mathcal{M},ij} = \phi_{\mathcal{M},j}(x_i).
$$
The likelihood of the data is:
$$
p(y_{1:n} | x_{1:n}, w, \sigma, \mathcal{M}) = \mathcal{N}(y_{1:n}|\Phi_{\mathcal{M}}w, \sigma^2I_n).
$$
Let's turn this into pymc code.

```python
def make_model(Phi, y):
    # The parameter gamma
    gamma = 1.
    # The prior for sigma
    sigma2 = pm.InverseGamma('sigma2', .5, 0.5)
    # The prior for alpha
    alpha = pm.InverseGamma('alpha', .5, .5)
    # The prior of the weights
    w = pm.Normal('w', mu=np.zeros(Phi.shape[1]), tau=alpha)
    # The likelihood
    @pm.stochastic(observed=True)
    def likelihood(value=y, Phi=Phi, gamma=gamma, w=w, sigma2=sigma2):
        return gamma * pm.normal_like(value, np.dot(Phi, w), 1. / sigma2)
    return locals()
```

Let's generate some fake data to play with:

```python
np.random.seed(12345)
num_samples = 50
sigma2 = 1e-6
X = 2 * np.random.rand(num_samples) - 1.
y = 0.5 * X  ** 3 - 0.3 * X ** 2 + np.sqrt(sigma2) * np.random.rand(num_samples)
plt.plot(X, y, 'kx', markeredgewidth=2)
```

Now, let's create a function that trains the model using pysmc for a polynomial basis with a given order.

```python
def fit_poly(phi, X, y, num_particles=100, verbose=0):
    Phi = compute_design_matrix(X[:, None], phi)
    model = make_model(Phi, y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['w'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['alpha'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma2'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=10, verbose=verbose, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc
```

Now, let's define a function that plots the posteriors of everything.

```python
def plot_posteriors(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    fig, ax = plt.subplots()
    for j in xrange(W.shape[1]):
        ax.hist(W[:, j], normed=True, alpha=0.5, label='$w_{%d}$' % j)
    plt.legend(loc='best')
    fig, ax = plt.subplots()
    alpha = pa.alpha
    ax.hist(alpha, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('alpha')
    fig, ax = plt.subplots()
    sigma2 = pa.sigma2
    ax.hist(sigma2, normed=True, label='$\sigma^2$', alpha=0.5)
    ax.set_xlabel('sigma2')
```

Let's also have a function that plots the predictive distribution:

```python
def plot_predictive(phi, smc, num_samples=100):
    Xt = np.linspace(-1, 1, 100)
    Phit = compute_design_matrix(Xt[:, None], phi)
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    sigma2 = pa.sigma2
    # We find the percantiles numerically
    Ys = np.ndarray((num_samples * pa.num_particles, Phit.shape[0]))
    for i in xrange(W.shape[0]):
        w = W[i, :]
        sigma = np.sqrt(sigma2[i])
        Yt = np.dot(Phit, w)
        for j in xrange(num_samples):
            Ys[i * num_samples + j, :] = Yt + sigma * np.random.randn(*Yt.shape)
    Ym = np.percentile(Ys, 50, axis=0)
    Yu = np.percentile(Ys, 97.5, axis=0)
    Yl = np.percentile(Ys, 2.5, axis=0)
    fig, ax = plt.subplots()
    ax.plot(Xt, Ym, color=sns.color_palette()[0])
    ax.fill_between(Xt, Yl, Yu, color=sns.color_palette()[0], alpha=0.25)
    ax.plot(X, y, 'kx', markeredgewidth=2)
```

```python
phi = PolynomialBasis(3)
smc = fit_poly(phi, X, y, num_particles=100, verbose=1)
plot_posteriors(smc)
plot_predictive(phi, smc)
```

Now, we can compute and compare the evidence of all  models:

```python
# Evaluate the evidence for the various degrees
log_Zs = []
for d in [1, 2, 3, 4, 5]:
    phi = phi = PolynomialBasis(d)
    smc = fit_poly(phi, X, y, num_particles=100)
    log_Z = np.sum(smc.log_Zs)
    print 'degree {0:1d} gives {1:1.3e}'.format(d, log_Z)
    log_Zs.append(log_Z)
```

    degree 1 gives 8.251e+00
    degree 2 gives 1.179e+01
    degree 3 gives 1.696e+01
    degree 4 gives 1.629e+01
    degree 5 gives 1.568e+01

```python
X.shape
```

```python
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))
```

## Questions
+ The model with degree 3 polynomials has the gradest evidence. However, the degree 4 and 5 seem also very plaussible. Is this a problem for the theory of Bayesian model selection? What complicates things here, is that model 3 is included in model 4 which is included in model 5. This requires us to design special priors for the models being right. They have to be consistent in some sense. For example, if model 3 is right then model 4 must be right, etc.

+ In the space below, we load the motorcycle data. By copy pasting the code above and by making small modifications, evaluate the model evidence for a 1) Polynomial basis; 2) a Fourier basis; and 3) a Radial basis function basis. Feel free to pick the parameters of these basis functions to be the best we found in [Lecture 9](./lec_09.ipynb).

# Revisiting Challenger Disaster Problem (Model Selection)

```python
from IPython.core.pylabtools import figsize
figsize(12.5, 3.5)
np.set_printoptions(precision=3, suppress=True)
challenger_data = np.genfromtxt("challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
# drop the NA values
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]

# plot it, as a function of temperature (the first column)
print("Temp (F), O-Ring failure?")
print(challenger_data)

plt.scatter(challenger_data[:, 0], challenger_data[:, 1])
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Fahrenheit)")
plt.title("Defects of the Space Shuttle O-Rings vs temperature");
```

```python
def make_model(feat, y):
    """
    PyMC model (wrapping all the data and variables into a single function)
    Inputs:
    challenger data (23 x 2): Pandas dataframe ( Outside Temp vs. Damage Incident)
    Outputs:
    alpha, beta : stochastic parameters associated with the given model
    """
    gamma = 1.
    # Load data

    feat_scaled = (feat - np.mean(feat))/np.std(feat) # to ensure proper mixing of MCMC chains
    #D = challenger_data[:, 1]  # defect or not?

    # Define Prior
    beta = pm.Normal("beta", 0, 0.1, value = 0) # 0 mean and small 0.001 precision - vague prior
    alpha = pm.Normal("alpha", 0, 0.1, value = 0)

    @pm.deterministic
    def p(t=feat_scaled, alpha=alpha, beta=beta):
        return 1.0 / (1. + np.exp(beta * t + alpha))

    # Define likelihood
    # connect the probabilities in `p` with our observations through a
    # Bernoulli random variable.
    @pm.stochastic(observed = True)
    def likelihood(value = y, p = p, gamma = gamma):
        return gamma*pm.bernoulli_like(value, p)
    return locals()

import mpi4py.MPI as mpi

def fit_smc(feat,y, num_particles=100, verbose=0):
    model = make_model(feat,y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['beta'])
    mcmc.use_step_method(ps.RandomWalk, model['alpha'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=20,
                 verbose=verbose,mpi=mpi, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc
```

```python
temp = challenger_data[:,0]
y = challenger_data[:,1]
smc = fit_smc(temp,y, num_particles= 500, verbose=1)
```

    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.95
    - moving to gamma : 0.00865564018366
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.434e-01
     [-----------------61%---               ] 6162 of 9980 complete in 0.5 sec
    - moving to gamma : 0.0133321960377
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.627e-01
     [-----------------99%----------------- ] 9962 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0176104267615
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.383e-01
     [-----------------85%------------      ] 8562 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0218126630496
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.274e-01
     [-----------------89%--------------    ] 8942 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0260481897488
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.210e-01
     [-----------------78%----------        ] 7882 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0304810345915
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.202e-01
     [-----------------73%-------           ] 7302 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0351855505451
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.194e-01
     [-----------------75%--------          ] 7582 of 9980 complete in 1.0 sec
    - resampling
    - moving to gamma : 0.0402061770975
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.210e-01
     [-----------------95%----------------  ] 9482 of 9980 complete in 1.5 sec
    - moving to gamma : 0.0615975079386
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -4.471e-01
     [-----------------89%--------------    ] 8962 of 9980 complete in 1.5 sec
    - moving to gamma : 0.0748712742064
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -2.496e-01
     [-----------------83%-----------       ] 8382 of 9980 complete in 1.5 sec
    - moving to gamma : 0.088150027908
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -2.358e-01
     [-----------------78%---------         ] 7842 of 9980 complete in 1.5 sec
    - moving to gamma : 0.102637465085
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -2.414e-01
     [-----------------75%--------          ] 7582 of 9980 complete in 1.5 sec
    - moving to gamma : 0.118510487533
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.546e-01
     [-----------------95%----------------  ] 9522 of 9980 complete in 2.0 sec
    - moving to gamma : 0.136408391547
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.749e-01
     [-----------------87%-------------     ] 8722 of 9980 complete in 2.0 sec
    - moving to gamma : 0.156444844031
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.986e-01
     [-----------------86%------------      ] 8602 of 9980 complete in 2.0 sec
    - resampling
    - moving to gamma : 0.178483582282
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.158e-01
     [-----------------82%-----------       ] 8242 of 9980 complete in 2.0 sec
    - moving to gamma : 0.248166897111
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -9.521e-01
     [-----------------83%-----------       ] 8322 of 9980 complete in 2.0 sec
    - moving to gamma : 0.294186783124
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.961e-01
     [-----------------99%----------------- ] 9942 of 9980 complete in 2.5 sec
    - moving to gamma : 0.340593739596
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -5.874e-01
     [-----------------87%-------------     ] 8742 of 9980 complete in 2.5 sec
    - moving to gamma : 0.392369549345
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -6.423e-01
     [-----------------84%------------      ] 8462 of 9980 complete in 2.5 sec
    - moving to gamma : 0.452487311888
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -7.325e-01
     [-----------------82%-----------       ] 8262 of 9980 complete in 2.5 sec
    - moving to gamma : 0.515164099767
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -7.508e-01
     [-----------------92%---------------   ] 9222 of 9980 complete in 2.5 sec
    - moving to gamma : 0.582607848414
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -7.973e-01
     [-----------------85%------------      ] 8522 of 9980 complete in 2.5 sec
    - resampling
    - moving to gamma : 0.657474070489
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.693e-01
     [-----------------95%----------------  ] 9502 of 9980 complete in 3.0 sec
    - moving to gamma : 0.873012235206
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.442e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 3.0 sec
    - moving to gamma : 1.0
    - performing 20 MCMC step(s) per particle
    - ESS  = 91.31 %
    - logZ = -1.409e+00
     [-----------------94%---------------   ] 9402 of 9980 complete in 3.0 sec
    - total number of MCMC steps: 260000
    ---------------
    END SMC MOVE TO
    ---------------

```python
def plot_posterior(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    fig, ax = plt.subplots()
    alpha = pa.alpha
    ax.hist(alpha, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('alpha')
    fig, ax = plt.subplots()
    beta = pa.beta
    ax.hist(beta, normed=True, label='$\beta^2$', alpha=0.5)
    ax.set_xlabel('beta')

def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

def plot_predictive(challenger_data, smc, num_samples=500):
    pa = smc.get_particle_approximation()
    pa.resample()
    alpha = pa.alpha
    beta = pa.beta

    temp = challenger_data[:, 0]
    D = challenger_data[:, 1]
    t = np.linspace(temp.min() - 5, temp.max() + 5, 50)[:, None]
    t_scaled = (t - np.mean(temp))/np.std(temp)
    p_t = logistic(t_scaled.T, beta[:,None], alpha[:,None])

    mean_prob_t = p_t.mean(axis=0)
    # vectorized bottom and top 2.5% quantiles for "confidence interval"
    qs = mquantiles(p_t, [0.025, 0.975], axis=0)
    plt.fill_between(t[:, 0], *qs, alpha=0.7,
                     color="#7A68A6")

    plt.plot(t[:, 0], qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

    plt.plot(t, mean_prob_t, lw=1, ls="--", color="k",
             label="average posterior \nprobability of defect")

    plt.xlim(t.min(), t.max())
    plt.ylim(-0.02, 1.02)
    plt.legend(loc="lower left")
    plt.scatter(temp, D, color="k", s=50, alpha=0.5)
    plt.xlabel("temp, $t$")

    plt.ylabel("probability estimate")
    plt.title("Posterior probability estimates given temp. $t$");
```

```python
plot_posterior(smc)
```

```python
from scipy.stats.mstats import mquantiles
plot_predictive(challenger_data, smc, num_samples=500)
```

# Feature Selection

We knew that temperature was a relevant feature for predicting the probability of failure of O-rings since there are numerous studies published which illustrates this fact.

But when dealing with your own problem, you might not know which feature is relevant/important. For example, we have not considered Leak-check pressure (psi) for predicting the probability of failure (which is available in whole Challenger disaster data UCI data repository (https://archive.ics.uci.edu/ml/datasets/Challenger+USA+Space+Shuttle+O-Ring). Let us see if we can predict the probability of failure using leak check pressure feature.

```python
c2_data = np.loadtxt('challenger2.txt')
leak_pressure = c2_data[:,3]
smc2 = fit_smc(leak_pressure,y, num_particles= 500, verbose=1)
```

    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.95
    - moving to gamma : 0.0108245514789
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -4.274e-01
     [-----------------62%---               ] 6262 of 9980 complete in 0.5 sec
    - moving to gamma : 0.0163899737888
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.989e-01
     [-----------------54%                  ] 5442 of 9980 complete in 0.5 sec
    - moving to gamma : 0.0213098254705
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.666e-01
     [-----------------95%----------------  ] 9562 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0260483815351
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.529e-01
     [-----------------92%---------------   ] 9222 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0307911181215
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.461e-01
     [-----------------84%------------      ] 8482 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0356708537074
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.454e-01
     [-----------------77%---------         ] 7742 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0407594109513
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.470e-01
     [-----------------96%----------------  ] 9642 of 9980 complete in 1.5 sec
    - resampling
    - moving to gamma : 0.046167323868
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.509e-01
     [-----------------68%------            ] 6882 of 9980 complete in 1.0 sec
    - moving to gamma : 0.0665401312668
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -5.319e-01
     [-----------------85%------------      ] 8522 of 9980 complete in 1.5 sec
    - moving to gamma : 0.0783268183555
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -2.834e-01
     [-----------------82%-----------       ] 8202 of 9980 complete in 1.5 sec
    - moving to gamma : 0.089690829153
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -2.604e-01
     [-----------------84%------------      ] 8442 of 9980 complete in 1.5 sec
    - moving to gamma : 0.101325759532
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -2.546e-01
     [-----------------75%--------          ] 7582 of 9980 complete in 1.5 sec
    - moving to gamma : 0.113534807204
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.581e-01
     [-----------------80%----------        ] 8022 of 9980 complete in 1.5 sec
    - moving to gamma : 0.126678735754
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.720e-01
     [-----------------99%----------------- ] 9902 of 9980 complete in 2.0 sec
    - moving to gamma : 0.140197786021
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.689e-01
     [-----------------90%--------------    ] 9002 of 9980 complete in 2.0 sec
    - resampling
    - moving to gamma : 0.155111274837
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.880e-01
     [-----------------87%-------------     ] 8762 of 9980 complete in 2.0 sec
    - moving to gamma : 0.203424174022
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.981e-01
     [-----------------85%------------      ] 8582 of 9980 complete in 2.0 sec
    - moving to gamma : 0.232802675323
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.224e-01
     [-----------------83%-----------       ] 8362 of 9980 complete in 2.0 sec
    - moving to gamma : 0.261745152305
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -4.939e-01
     [-----------------80%----------        ] 8042 of 9980 complete in 2.0 sec
    - moving to gamma : 0.293068466277
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.224e-01
     [-----------------99%----------------- ] 9942 of 9980 complete in 2.5 sec
    - moving to gamma : 0.327691803884
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -5.700e-01
     [-----------------94%----------------  ] 9462 of 9980 complete in 2.5 sec
    - moving to gamma : 0.368013099831
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -6.521e-01
     [-----------------90%--------------    ] 9002 of 9980 complete in 2.5 sec
    - moving to gamma : 0.412343950188
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -6.998e-01
     [-----------------99%----------------- ] 9942 of 9980 complete in 3.0 sec
    - resampling
    - moving to gamma : 0.467500449554
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.539e-01
     [-----------------94%----------------  ] 9462 of 9980 complete in 3.0 sec
    - moving to gamma : 0.612241074257
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.180e+00
     [-----------------87%-------------     ] 8702 of 9980 complete in 2.5 sec
    - moving to gamma : 0.722832641022
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.632e+00
     [-----------------83%-----------       ] 8382 of 9980 complete in 2.5 sec
    - moving to gamma : 0.84199107311
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.732e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 3.0 sec
    - moving to gamma : 0.97766098638
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.939e+00
     [-----------------94%----------------  ] 9462 of 9980 complete in 3.0 sec
    - moving to gamma : 1.0
    - performing 20 MCMC step(s) per particle
    - ESS  = 80.97 %
    - logZ = -3.185e-01
     [-----------------91%--------------    ] 9082 of 9980 complete in 3.0 sec
    - total number of MCMC steps: 290000
    ---------------
    END SMC MOVE TO
    ---------------

```python
log_Zs = [np.sum(smc.log_Zs), np.sum(smc2.log_Zs)]
print 'When we use temp as feature, log evidence is :', str(log_Zs[0])
print 'When we use leak pressure as feature, log evidence is :', str(log_Zs[1])
```

    When we use temp as feature, log evidence is : -13.350311386
    When we use leak pressure as feature, log evidence is : -16.9670236784

```python
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))
```

# Motor Cycle Data Visualization

```python
data = np.loadtxt('motor.dat')
X = data[:, 0][:, None]
Y = data[:, 1]
fig, ax = plt.subplots()
ax.plot(X, Y, 'x', markeredgewidth=2)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
```

```python
import pymc as pm
def make_model(Phi, y):
    # The parameter gamma
    gamma = 1.
    # The prior for sigma
    sigma1 = pm.Exponential('sigma1', 0.1)
    sigma2 = pm.Exponential('sigma2', 0.1)
    # The prior of the weights
    w = pm.Normal('w', mu=np.zeros(Phi.shape[1]), tau= 1./sigma1**2)
    # The likelihood
    @pm.stochastic(observed=True)
    def likelihood(value=y, Phi=Phi, gamma=gamma, w=w, sigma2=sigma2):
        return gamma * pm.normal_like(value, np.dot(Phi, w), 1. / sigma2**2)
    return locals()

def fit_radial(ell, X, y, num_particles=100, verbose=0):
    phi = RadialBasisFunctions(X, ell)
    Phi = compute_design_matrix(X, phi)
    model = make_model(Phi, y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['w'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma1'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma2'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=20,
                 verbose=verbose,mpi=mpi, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc
```

```python
def plot_posterior(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    fig, ax = plt.subplots()
    for j in xrange(W.shape[1]):
        ax.hist(W[:, j], normed=True, alpha=0.5, label='$w_{%d}$' % j)
    #plt.legend(loc='best')
    fig, ax = plt.subplots()
    sigma1 = pa.sigma1
    ax.hist(sigma1, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('sigma1')
    fig, ax = plt.subplots()
    sigma2 = pa.sigma2
    ax.hist(sigma2, normed=True, label='$\sigma^2$', alpha=0.5)
    ax.set_xlabel('sigma2')

def plot_predictive(phi, smc, num_samples=100):
    Xt = np.linspace(0, 60, 100)
    Phit = compute_design_matrix(Xt[:,None], phi)
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w

    p_t = np.dot(W,Phit.T)
    mean_prob_t = np.mean(p_t, axis = 0)

    # vectorized bottom and top 2.5% quantiles for "confidence interval"
    qs = mquantiles(p_t, [0.025, 0.975], axis=0)
    plt.fill_between(Xt, *qs, alpha=0.7,
                     color="#7A68A6")

    plt.plot(Xt, qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

    plt.plot(Xt, mean_prob_t, lw=1, ls="--", color="k",
             label="average posterior")

    plt.xlim(Xt.min(), Xt.max())
    plt.legend(loc="best")
    plt.scatter(X, Y, color="k", s=50, alpha=0.5)
```

```python
ell = 5.
smc = fit_radial(ell, X, Y, num_particles=500, verbose=1)
```

    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.95
    - moving to gamma : 1.3603447533e-06
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.736e-02
     [-----------------95%----------------  ] 9542 of 9980 complete in 1.5 sec
    - moving to gamma : 8.11196843751e-06
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.084e-01
     [-----------------92%---------------   ] 9222 of 9980 complete in 1.5 sec
    - moving to gamma : 2.37181354045e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.401e-01
     [-----------------80%----------        ] 8062 of 9980 complete in 1.5 sec
    - moving to gamma : 5.50638957792e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.575e-01
     [-----------------82%-----------       ] 8282 of 9980 complete in 1.5 sec
    - moving to gamma : 0.000108064666015
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.907e-01
     [-----------------76%---------         ] 7642 of 9980 complete in 1.5 sec
    - moving to gamma : 0.000190734569713
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.045e-01
     [-----------------96%----------------  ] 9622 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000309546456794
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.150e-01
     [-----------------95%----------------  ] 9542 of 9980 complete in 2.0 sec
    - resampling
    - moving to gamma : 0.00051192202063
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.023e-01
     [-----------------83%-----------       ] 8342 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000831624231596
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.724e-01
     [-----------------94%----------------  ] 9462 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00116861638489
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -3.644e-01
     [-----------------96%----------------  ] 9622 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00155502123572
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.577e-01
     [-----------------84%------------      ] 8422 of 9980 complete in 2.5 sec
    - moving to gamma : 0.0020641027251
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -4.260e-01
     [-----------------84%------------      ] 8462 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00260420046848
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -4.111e-01
     [-----------------96%----------------  ] 9582 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00326797071155
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -4.664e-01
     [-----------------90%--------------    ] 9042 of 9980 complete in 3.0 sec
    - moving to gamma : 0.0040698074566
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -5.366e-01
     [-----------------88%-------------     ] 8802 of 9980 complete in 3.0 sec
    - resampling
    - moving to gamma : 0.00500127333903
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -6.076e-01
     [-----------------90%--------------    ] 9042 of 9980 complete in 4.1 sec
    - moving to gamma : 0.00667260523917
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.055e+00
     [-----------------89%-------------     ] 8902 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00826974386229
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -9.606e-01
     [-----------------95%----------------  ] 9562 of 9980 complete in 3.5 sec
    - moving to gamma : 0.00989246274416
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -9.471e-01
     [-----------------94%----------------  ] 9462 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0115283647359
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -9.307e-01
     [-----------------91%--------------    ] 9082 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0135369559899
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.115e+00
     [-----------------93%---------------   ] 9302 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0158033827628
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.248e+00
     [-----------------90%--------------    ] 9022 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0177154047795
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.028e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 3.5 sec
    - resampling
    - moving to gamma : 0.0197210788445
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.071e+00
     [-----------------93%---------------   ] 9302 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0255939025876
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.094e+00
     [-----------------89%-------------     ] 8922 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0291244159352
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.836e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0319836055827
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.473e+00
     [-----------------92%---------------   ] 9202 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0345358649799
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.306e+00
     [-----------------92%--------------    ] 9182 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0369732226234
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.241e+00
     [-----------------89%-------------     ] 8902 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0389243781522
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -9.871e-01
     [-----------------93%---------------   ] 9282 of 9980 complete in 4.5 sec
    - moving to gamma : 0.0406624948835
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -8.786e-01
     [-----------------94%---------------   ] 9442 of 9980 complete in 4.5 sec
    - resampling
    - moving to gamma : 0.0422733259
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.109e-01
     [-----------------92%--------------    ] 9182 of 9980 complete in 4.6 sec
    - moving to gamma : 0.0495819252723
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.653e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 5.0 sec
    - moving to gamma : 0.0531485196324
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.760e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 5.1 sec
    - moving to gamma : 0.0559608194059
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.379e+00
     [-----------------93%---------------   ] 9302 of 9980 complete in 5.0 sec
    - moving to gamma : 0.0583652692894
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.176e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 5.6 sec
    - moving to gamma : 0.0604524543387
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.015e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 5.6 sec
    - moving to gamma : 0.0624142938268
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -9.516e-01
     [-----------------95%----------------  ] 9482 of 9980 complete in 5.6 sec
    - moving to gamma : 0.064326789181
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -9.249e-01
     [-----------------92%---------------   ] 9202 of 9980 complete in 5.6 sec
    - resampling
    - moving to gamma : 0.0661243235327
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.668e-01
     [-----------------99%----------------- ] 9922 of 9980 complete in 6.1 sec
    - moving to gamma : 0.074559227845
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -4.043e+00
     [-----------------97%----------------  ] 9702 of 9980 complete in 6.1 sec
    - moving to gamma : 0.0786113153522
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.925e+00
     [-----------------95%----------------  ] 9502 of 9980 complete in 6.1 sec
    - moving to gamma : 0.0819066523589
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.557e+00
     [-----------------92%---------------   ] 9262 of 9980 complete in 6.1 sec
    - moving to gamma : 0.0848915611346
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.405e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 6.6 sec
    - moving to gamma : 0.0876493153619
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.293e+00
     [-----------------96%----------------  ] 9622 of 9980 complete in 6.6 sec
    - moving to gamma : 0.0903187544305
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.249e+00
     [-----------------94%---------------   ] 9382 of 9980 complete in 6.6 sec
    - moving to gamma : 0.0930066823411
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.253e+00
     [-----------------93%---------------   ] 9322 of 9980 complete in 6.6 sec
    - resampling
    - moving to gamma : 0.0957403134797
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.271e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 7.1 sec
    - moving to gamma : 0.107576106793
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -5.460e+00
     [-----------------93%---------------   ] 9362 of 9980 complete in 7.1 sec
    - moving to gamma : 0.113829832322
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -2.867e+00
     [-----------------94%---------------   ] 9402 of 9980 complete in 7.1 sec
    - moving to gamma : 0.119511525027
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -2.595e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 7.6 sec
    - moving to gamma : 0.125245213638
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -2.608e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 7.6 sec
    - moving to gamma : 0.131069008908
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.641e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 7.7 sec
    - moving to gamma : 0.137250962436
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.795e+00
     [-----------------94%----------------  ] 9462 of 9980 complete in 7.6 sec
    - moving to gamma : 0.143665695679
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.892e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 8.1 sec
    - resampling
    - moving to gamma : 0.150624233726
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.130e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 8.1 sec
    - moving to gamma : 0.176068588647
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.140e+01
     [-----------------96%----------------  ] 9622 of 9980 complete in 8.1 sec
    - moving to gamma : 0.191491137682
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -6.886e+00
     [-----------------95%----------------  ] 9482 of 9980 complete in 8.1 sec
    - moving to gamma : 0.207096307934
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.959e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 8.7 sec
    - moving to gamma : 0.222203453295
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -6.726e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 8.6 sec
    - moving to gamma : 0.237050834908
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -6.602e+00
     [-----------------96%----------------  ] 9582 of 9980 complete in 8.7 sec
    - moving to gamma : 0.25151895383
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -6.426e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 9.1 sec
    - moving to gamma : 0.266090939106
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -6.461e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 9.2 sec
    - resampling
    - moving to gamma : 0.279964573416
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -6.145e+00
     [-----------------96%----------------  ] 9622 of 9980 complete in 9.2 sec
    - moving to gamma : 0.33251025217
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.322e+01
     [-----------------95%----------------  ] 9522 of 9980 complete in 9.1 sec
    - moving to gamma : 0.359816340066
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.205e+01
     [-----------------99%----------------- ] 9922 of 9980 complete in 9.7 sec
    - moving to gamma : 0.382465617297
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -9.982e+00
     [-----------------97%----------------- ] 9722 of 9980 complete in 9.7 sec
    - moving to gamma : 0.402459000896
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -8.805e+00
     [-----------------96%----------------  ] 9582 of 9980 complete in 9.7 sec
    - moving to gamma : 0.421608105208
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -8.430e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 10.2 sec
    - moving to gamma : 0.439930528942
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -8.060e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 10.2 sec
    - moving to gamma : 0.457655552985
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -7.794e+00
     [-----------------97%----------------  ] 9702 of 9980 complete in 10.2 sec
    - resampling
    - moving to gamma : 0.475298478019
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -7.753e+00
     [-----------------96%----------------  ] 9582 of 9980 complete in 10.2 sec
    - moving to gamma : 0.543700862261
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.003e+01
     [-----------------99%----------------- ] 9902 of 9980 complete in 10.7 sec
    - moving to gamma : 0.575784245511
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.407e+01
     [-----------------98%----------------- ] 9782 of 9980 complete in 10.8 sec
    - moving to gamma : 0.601265781215
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.117e+01
     [-----------------99%----------------- ] 9942 of 9980 complete in 11.3 sec
    - moving to gamma : 0.623934012475
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -9.927e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 11.3 sec
    - moving to gamma : 0.645359633053
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -9.380e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 11.3 sec
    - moving to gamma : 0.665415311608
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -8.775e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 11.8 sec
    - moving to gamma : 0.684334598997
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -8.274e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 12.3 sec
    - resampling
    - moving to gamma : 0.702726526264
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.040e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 12.3 sec
    - moving to gamma : 0.783864826785
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.544e+01
     [-----------------97%----------------  ] 9702 of 9980 complete in 12.2 sec
    - moving to gamma : 0.821658199682
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.649e+01
     [-----------------98%----------------- ] 9782 of 9980 complete in 12.3 sec
    - moving to gamma : 0.851988099832
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.322e+01
     [-----------------99%----------------- ] 9882 of 9980 complete in 12.8 sec
    - moving to gamma : 0.878895127561
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.172e+01
     [-----------------97%----------------  ] 9682 of 9980 complete in 12.3 sec
    - moving to gamma : 0.903563072284
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.074e+01
     [-----------------97%----------------  ] 9682 of 9980 complete in 12.8 sec
    - moving to gamma : 0.926425373229
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -9.950e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 12.8 sec
    - moving to gamma : 0.948814116939
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -9.741e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 13.4 sec
    - resampling
    - moving to gamma : 0.970149734629
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -9.280e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 13.8 sec
    - moving to gamma : 1.0
    - performing 20 MCMC step(s) per particle
    - ESS  = 99.46 %
    - logZ = -1.298e+01
     [-----------------99%----------------- ] 9922 of 9980 complete in 13.3 sec
    - total number of MCMC steps: 890000
    ---------------
    END SMC MOVE TO
    ---------------

```python
plot_posterior(smc)
```

```python
phi = RadialBasisFunctions(X, ell)
plot_predictive(phi, smc, num_samples=500)
```

```python
# Evaluate the evidence for the various degrees
log_Zs = []
for ell in [2, 5]:
    smc = fit_radial(ell, X, Y, num_particles=200, verbose=1)
    log_Z = np.sum(smc.log_Zs)
    log_Zs.append(log_Z)
```

    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.95
    - moving to gamma : 1.07613662955e-06
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.245e-02
     [-----------------93%---------------   ] 9342 of 9980 complete in 1.5 sec
    - moving to gamma : 7.60155818624e-06
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.192e-01
     [-----------------82%-----------       ] 8262 of 9980 complete in 1.5 sec
    - moving to gamma : 2.4147235304e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.404e-01
     [-----------------81%-----------       ] 8162 of 9980 complete in 1.5 sec
    - moving to gamma : 5.60806000119e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.557e-01
     [-----------------76%---------         ] 7622 of 9980 complete in 1.5 sec
    - moving to gamma : 0.000119337340186
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.915e-01
     [-----------------96%----------------  ] 9582 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000212487983834
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.111e-01
     [-----------------87%-------------     ] 8702 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000334673765166
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.187e-01
     [-----------------87%-------------     ] 8782 of 9980 complete in 2.0 sec
    - resampling
    - moving to gamma : 0.000529065746313
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.671e-01
     [-----------------88%-------------     ] 8822 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000854326625779
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.795e-01
     [-----------------90%--------------    ] 9042 of 9980 complete in 2.0 sec
    - moving to gamma : 0.00122456360649
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -3.540e-01
     [-----------------94%---------------   ] 9442 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00173720926307
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -4.367e-01
     [-----------------94%---------------   ] 9402 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00238378829172
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.113e-01
     [-----------------93%---------------   ] 9282 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00318401724175
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -5.903e-01
     [-----------------92%---------------   ] 9222 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00396015757757
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -5.431e-01
     [-----------------95%----------------  ] 9522 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00507724465295
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -7.164e-01
     [-----------------87%-------------     ] 8702 of 9980 complete in 3.0 sec
    - resampling
    - moving to gamma : 0.00644749215233
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -8.590e-01
     [-----------------98%----------------- ] 9822 of 9980 complete in 3.5 sec
    - moving to gamma : 0.00874313517107
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.378e+00
     [-----------------87%-------------     ] 8762 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0108800727628
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.239e+00
     [-----------------95%----------------  ] 9482 of 9980 complete in 3.5 sec
    - moving to gamma : 0.013157802395
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.289e+00
     [-----------------95%----------------  ] 9522 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0156384336641
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.380e+00
     [-----------------94%---------------   ] 9402 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0183111511815
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.463e+00
     [-----------------90%--------------    ] 8982 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0217094394944
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.851e+00
     [-----------------93%---------------   ] 9302 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0249236233409
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.728e+00
     [-----------------88%-------------     ] 8842 of 9980 complete in 3.5 sec
    - resampling
    - moving to gamma : 0.0283691152668
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.832e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0367913176227
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -4.432e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0430955432542
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -3.285e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0490206467248
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.073e+00
     [-----------------90%--------------    ] 9042 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0550175818269
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.087e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 4.5 sec
    - moving to gamma : 0.0606824430284
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.906e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 5.0 sec
    - moving to gamma : 0.0656908688964
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.558e+00
     [-----------------91%--------------    ] 9142 of 9980 complete in 4.5 sec
    - moving to gamma : 0.0704626300506
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.430e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 5.1 sec
    - resampling
    - moving to gamma : 0.0747258601346
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.164e+00
     [-----------------97%----------------  ] 9702 of 9980 complete in 5.1 sec
    - moving to gamma : 0.0909775032029
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.222e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 5.1 sec
    - moving to gamma : 0.0989660375345
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -4.023e+00
     [-----------------92%---------------   ] 9202 of 9980 complete in 5.1 sec
    - moving to gamma : 0.105813920666
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.440e+00
     [-----------------95%----------------  ] 9542 of 9980 complete in 5.5 sec
    - moving to gamma : 0.112172746116
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.185e+00
     [-----------------93%---------------   ] 9342 of 9980 complete in 5.5 sec
    - moving to gamma : 0.118052876558
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.941e+00
     [-----------------94%---------------   ] 9382 of 9980 complete in 5.6 sec
    - moving to gamma : 0.123416978787
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.678e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 6.1 sec
    - moving to gamma : 0.128406141436
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.486e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 6.1 sec
    - resampling
    - moving to gamma : 0.133198922372
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.386e+00
     [-----------------95%----------------  ] 9522 of 9980 complete in 6.1 sec
    - moving to gamma : 0.153400899215
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.003e+01
     [-----------------93%---------------   ] 9282 of 9980 complete in 6.1 sec
    - moving to gamma : 0.163093429437
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -4.793e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 6.6 sec
    - moving to gamma : 0.170947792218
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.874e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 7.6 sec
    - moving to gamma : 0.177887016809
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.418e+00
     [-----------------93%---------------   ] 9362 of 9980 complete in 6.6 sec
    - moving to gamma : 0.18403500675
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -3.023e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 7.1 sec
    - moving to gamma : 0.189710772808
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.787e+00
     [-----------------94%---------------   ] 9382 of 9980 complete in 8.2 sec
    - moving to gamma : 0.195172230546
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.679e+00
     [-----------------93%---------------   ] 9362 of 9980 complete in 7.1 sec
    - resampling
    - moving to gamma : 0.200451841827
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.586e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 7.6 sec
    - moving to gamma : 0.226118363893
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.254e+01
     [-----------------95%----------------  ] 9502 of 9980 complete in 7.6 sec
    - moving to gamma : 0.237644707366
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.615e+00
     [-----------------93%---------------   ] 9362 of 9980 complete in 7.6 sec
    - moving to gamma : 0.24715807517
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -4.626e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 8.2 sec
    - moving to gamma : 0.255601288503
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -4.101e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 8.1 sec
    - moving to gamma : 0.262860235543
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -3.521e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 8.6 sec
    - moving to gamma : 0.269333672217
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -3.135e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 8.7 sec
    - moving to gamma : 0.27523913078
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.858e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 8.7 sec
    - resampling
    - moving to gamma : 0.280818155748
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.696e+00
     [-----------------96%----------------  ] 9582 of 9980 complete in 8.7 sec
    - moving to gamma : 0.306853212072
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.256e+01
     [-----------------95%----------------  ] 9502 of 9980 complete in 8.7 sec
    - moving to gamma : 0.318389330862
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.545e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 9.1 sec
    - moving to gamma : 0.327986584585
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -4.605e+00
     [-----------------95%----------------  ] 9562 of 9980 complete in 9.2 sec
    - moving to gamma : 0.336333522647
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.999e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 9.7 sec
    - moving to gamma : 0.34398277631
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -3.660e+00
     [-----------------95%----------------  ] 9502 of 9980 complete in 9.2 sec
    - moving to gamma : 0.351243114948
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -3.470e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 9.7 sec
    - moving to gamma : 0.358179965071
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -3.312e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 9.7 sec
    - resampling
    - moving to gamma : 0.364855549641
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.183e+00
     [-----------------95%----------------  ] 9542 of 9980 complete in 9.7 sec
    - moving to gamma : 0.39593771428
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.479e+01
     [-----------------100%-----------------] 9982 of 9980 complete in 10.2 sec
    - moving to gamma : 0.409778997402
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -6.570e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 10.3 sec
    - moving to gamma : 0.420964744352
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -5.303e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 10.2 sec
    - moving to gamma : 0.430761759472
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -4.638e+00
     [-----------------95%----------------  ] 9562 of 9980 complete in 10.2 sec
    - moving to gamma : 0.439671064542
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -4.212e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 10.7 sec
    - moving to gamma : 0.448097149958
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -3.979e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 12.8 sec
    - moving to gamma : 0.456107205911
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -3.779e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 10.8 sec
    - resampling
    - moving to gamma : 0.463770038297
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.612e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 10.7 sec
    - moving to gamma : 0.500699576087
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.737e+01
     [-----------------99%----------------- ] 9962 of 9980 complete in 11.2 sec
    - moving to gamma : 0.517875271471
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -8.056e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 11.3 sec
    - moving to gamma : 0.532066139788
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.646e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 12.2 sec
    - moving to gamma : 0.544754717583
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.935e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 11.8 sec
    - moving to gamma : 0.556692916203
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -5.577e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 11.8 sec
    - moving to gamma : 0.567839221562
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -5.201e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 11.8 sec
    - moving to gamma : 0.578412170931
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -4.928e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 11.9 sec
    - resampling
    - moving to gamma : 0.588170189924
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -4.543e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 11.9 sec
    - moving to gamma : 0.629851194404
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.937e+01
     [-----------------99%----------------- ] 9942 of 9980 complete in 12.3 sec
    - moving to gamma : 0.647771921581
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -8.307e+00
     [-----------------98%----------------- ] 9822 of 9980 complete in 12.3 sec
    - moving to gamma : 0.661705751645
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.449e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 12.3 sec
    - moving to gamma : 0.673441598871
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.424e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 12.8 sec
    - moving to gamma : 0.683921005597
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -4.838e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 12.8 sec
    - moving to gamma : 0.693404850241
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -4.373e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 12.9 sec
    - moving to gamma : 0.702124680062
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -4.016e+00
     [-----------------97%----------------- ] 9722 of 9980 complete in 12.8 sec
    - resampling
    - moving to gamma : 0.710277425735
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.752e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 13.3 sec
    - moving to gamma : 0.75177661954
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.905e+01
     [-----------------99%----------------- ] 9882 of 9980 complete in 13.3 sec
    - moving to gamma : 0.768788921052
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -7.791e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 13.3 sec
    - moving to gamma : 0.78215159247
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.113e+00
     [-----------------98%----------------- ] 9802 of 9980 complete in 13.4 sec
    - moving to gamma : 0.793554227736
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.209e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 13.9 sec
    - moving to gamma : 0.803665406648
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -4.613e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 13.9 sec
    - moving to gamma : 0.812928467182
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -4.222e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 13.8 sec
    - moving to gamma : 0.821663575507
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -3.977e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 13.9 sec
    - resampling
    - moving to gamma : 0.830243300686
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.903e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 13.9 sec
    - moving to gamma : 0.872046802456
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.897e+01
     [-----------------99%----------------- ] 9962 of 9980 complete in 14.5 sec
    - moving to gamma : 0.889886651444
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -8.076e+00
     [-----------------97%----------------- ] 9722 of 9980 complete in 14.3 sec
    - moving to gamma : 0.9040135694
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.385e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 14.4 sec
    - moving to gamma : 0.916410299475
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.596e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 14.9 sec
    - moving to gamma : 0.927600839569
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -5.045e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 14.8 sec
    - moving to gamma : 0.937988683214
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -4.677e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 14.9 sec
    - moving to gamma : 0.947804689272
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -4.414e+00
     [-----------------97%----------------  ] 9702 of 9980 complete in 15.0 sec
    - resampling
    - moving to gamma : 0.957253723964
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -4.244e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 15.5 sec
    - moving to gamma : 1.0
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.03 %
    - logZ = -1.915e+01
     [-----------------99%----------------- ] 9882 of 9980 complete in 15.4 sec
    - total number of MCMC steps: 1050000
    ---------------
    END SMC MOVE TO
    ---------------
    ------------------------
    START SMC Initialization
    ------------------------
    - initializing at gamma : 0.0
    - initializing by sampling from the prior: SUCCESS
    ----------------------
    END SMC Initialization
    ----------------------
    -----------------
    START SMC MOVE TO
    -----------------
    initial  gamma : 0.0
    final gamma : 1.0
    ess reduction:  0.95
    - moving to gamma : 9.02599464112e-07
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.457e-02
     [-----------------97%----------------- ] 9742 of 9980 complete in 1.5 sec
    - moving to gamma : 6.37115202075e-06
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -9.933e-02
     [-----------------92%---------------   ] 9202 of 9980 complete in 1.5 sec
    - moving to gamma : 2.31247718354e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.343e-01
     [-----------------87%-------------     ] 8702 of 9980 complete in 1.5 sec
    - moving to gamma : 5.77622089007e-05
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.695e-01
     [-----------------80%----------        ] 8002 of 9980 complete in 1.5 sec
    - moving to gamma : 0.000105017383743
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.552e-01
     [-----------------79%----------        ] 7902 of 9980 complete in 1.5 sec
    - moving to gamma : 0.000179641600029
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.077e-01
     [-----------------99%----------------- ] 9962 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000290986629876
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.168e-01
     [-----------------93%---------------   ] 9322 of 9980 complete in 2.0 sec
    - resampling
    - moving to gamma : 0.000440304333646
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.340e-01
     [-----------------89%-------------     ] 8902 of 9980 complete in 2.0 sec
    - moving to gamma : 0.000708177942193
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.348e-01
     [-----------------87%-------------     ] 8722 of 9980 complete in 2.0 sec
    - moving to gamma : 0.00104089309319
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -3.525e-01
     [-----------------82%-----------       ] 8262 of 9980 complete in 2.0 sec
    - moving to gamma : 0.00145879626779
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.997e-01
     [-----------------99%----------------- ] 9882 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00195946504442
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -4.258e-01
     [-----------------95%----------------  ] 9502 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00255728605022
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -4.600e-01
     [-----------------95%----------------  ] 9522 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00321776638236
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -4.834e-01
     [-----------------90%--------------    ] 9002 of 9980 complete in 2.5 sec
    - moving to gamma : 0.00398918341027
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -5.320e-01
     [-----------------87%-------------     ] 8742 of 9980 complete in 2.5 sec
    - resampling
    - moving to gamma : 0.00484141030461
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -5.662e-01
     [-----------------98%----------------- ] 9802 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00651497926182
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.056e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 3.0 sec
    - moving to gamma : 0.00832377391145
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.079e+00
     [-----------------95%----------------  ] 9522 of 9980 complete in 3.0 sec
    - moving to gamma : 0.0104327619559
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -1.221e+00
     [-----------------89%-------------     ] 8902 of 9980 complete in 3.0 sec
    - moving to gamma : 0.0128421107902
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -1.361e+00
     [-----------------86%------------      ] 8662 of 9980 complete in 3.0 sec
    - moving to gamma : 0.0154546363394
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -1.437e+00
     [-----------------91%--------------    ] 9102 of 9980 complete in 3.5 sec
    - moving to gamma : 0.0179140605163
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.341e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 3.5 sec
    - moving to gamma : 0.020581040352
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.443e+00
     [-----------------92%---------------   ] 9222 of 9980 complete in 3.5 sec
    - resampling
    - moving to gamma : 0.0234545821671
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.533e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 4.0 sec
    - moving to gamma : 0.030332535202
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -3.654e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0353527143968
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -2.623e+00
     [-----------------93%---------------   ] 9322 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0401762324564
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -2.499e+00
     [-----------------88%-------------     ] 8882 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0449649484385
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -2.467e+00
     [-----------------89%-------------     ] 8922 of 9980 complete in 4.0 sec
    - moving to gamma : 0.0492265451174
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.188e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 4.5 sec
    - moving to gamma : 0.0530348050365
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -1.947e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 4.6 sec
    - moving to gamma : 0.0567145142835
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -1.873e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 5.0 sec
    - resampling
    - moving to gamma : 0.0604268243438
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -1.883e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 5.0 sec
    - moving to gamma : 0.0731445707251
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -6.413e+00
     [-----------------94%---------------   ] 9422 of 9980 complete in 5.1 sec
    - moving to gamma : 0.0802641682108
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -3.565e+00
     [-----------------92%---------------   ] 9242 of 9980 complete in 5.0 sec
    - moving to gamma : 0.0861484915944
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -2.937e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 5.6 sec
    - moving to gamma : 0.0914872700025
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -2.654e+00
     [-----------------94%---------------   ] 9442 of 9980 complete in 5.6 sec
    - moving to gamma : 0.0964346774476
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.455e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 6.1 sec
    - moving to gamma : 0.101137937623
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.327e+00
     [-----------------96%----------------  ] 9582 of 9980 complete in 6.1 sec
    - moving to gamma : 0.105671081359
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.239e+00
     [-----------------97%----------------- ] 9722 of 9980 complete in 6.1 sec
    - resampling
    - moving to gamma : 0.109959962843
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.115e+00
     [-----------------94%---------------   ] 9402 of 9980 complete in 6.1 sec
    - moving to gamma : 0.128000338208
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -8.852e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 6.6 sec
    - moving to gamma : 0.137101526098
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -4.441e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 6.6 sec
    - moving to gamma : 0.144549768608
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.625e+00
     [-----------------94%----------------  ] 9462 of 9980 complete in 6.6 sec
    - moving to gamma : 0.151558660813
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.405e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 7.1 sec
    - moving to gamma : 0.158084876162
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -3.166e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 7.1 sec
    - moving to gamma : 0.16416682748
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.948e+00
     [-----------------95%----------------  ] 9522 of 9980 complete in 7.1 sec
    - moving to gamma : 0.169876351952
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.762e+00
     [-----------------94%---------------   ] 9382 of 9980 complete in 7.1 sec
    - resampling
    - moving to gamma : 0.17515510539
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.551e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 7.6 sec
    - moving to gamma : 0.198418027292
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.120e+01
     [-----------------96%----------------  ] 9582 of 9980 complete in 7.6 sec
    - moving to gamma : 0.208971487967
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.061e+00
     [-----------------94%---------------   ] 9382 of 9980 complete in 7.6 sec
    - moving to gamma : 0.216996525985
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.840e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 8.1 sec
    - moving to gamma : 0.223809425037
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.254e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 8.1 sec
    - moving to gamma : 0.229886824938
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.898e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 8.1 sec
    - moving to gamma : 0.235452240553
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.650e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 8.7 sec
    - moving to gamma : 0.240486790748
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.394e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 8.6 sec
    - resampling
    - moving to gamma : 0.245222268654
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.250e+00
     [-----------------96%----------------  ] 9602 of 9980 complete in 8.6 sec
    - moving to gamma : 0.267361287956
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.049e+01
     [-----------------95%----------------  ] 9522 of 9980 complete in 8.7 sec
    - moving to gamma : 0.276895179271
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -4.502e+00
     [-----------------94%----------------  ] 9462 of 9980 complete in 8.7 sec
    - moving to gamma : 0.284584304552
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -3.624e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 9.2 sec
    - moving to gamma : 0.291387208258
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.201e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 9.1 sec
    - moving to gamma : 0.297669482416
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -2.951e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 9.7 sec
    - moving to gamma : 0.303705999995
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -2.832e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 10.7 sec
    - moving to gamma : 0.309442182847
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -2.686e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 10.2 sec
    - resampling
    - moving to gamma : 0.314936185577
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -2.569e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 10.2 sec
    - moving to gamma : 0.340567201861
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.196e+01
     [-----------------98%----------------- ] 9802 of 9980 complete in 10.2 sec
    - moving to gamma : 0.352357282412
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -5.480e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 10.2 sec
    - moving to gamma : 0.362152350984
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -4.545e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 10.3 sec
    - moving to gamma : 0.370696696386
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -3.958e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 10.7 sec
    - moving to gamma : 0.378652570439
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -3.679e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 10.7 sec
    - moving to gamma : 0.386444965166
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -3.599e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 10.7 sec
    - moving to gamma : 0.393998226817
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -3.483e+00
     [-----------------99%----------------- ] 9922 of 9980 complete in 11.3 sec
    - resampling
    - moving to gamma : 0.401414649365
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -3.415e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 11.3 sec
    - moving to gamma : 0.433951005664
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -1.495e+01
     [-----------------97%----------------- ] 9722 of 9980 complete in 11.2 sec
    - moving to gamma : 0.449793304691
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -7.259e+00
     [-----------------97%----------------  ] 9682 of 9980 complete in 11.2 sec
    - moving to gamma : 0.463167708794
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -6.118e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 12.3 sec
    - moving to gamma : 0.475446868669
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -5.609e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 12.3 sec
    - moving to gamma : 0.487239261506
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -5.381e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 12.3 sec
    - moving to gamma : 0.498552681204
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -5.154e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 11.3 sec
    - moving to gamma : 0.509668672476
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -5.060e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 11.3 sec
    - resampling
    - moving to gamma : 0.520219768006
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -4.798e+00
     [-----------------99%----------------- ] 9942 of 9980 complete in 11.3 sec
    - moving to gamma : 0.565696711446
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.065e+01
     [-----------------96%----------------  ] 9622 of 9980 complete in 11.3 sec
    - moving to gamma : 0.586963657245
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -9.637e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 11.8 sec
    - moving to gamma : 0.604450582053
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -7.914e+00
     [-----------------99%----------------- ] 9882 of 9980 complete in 11.8 sec
    - moving to gamma : 0.620030766596
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -7.042e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 11.7 sec
    - moving to gamma : 0.634145308617
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -6.372e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 12.4 sec
    - moving to gamma : 0.647572210941
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -6.056e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 12.3 sec
    - moving to gamma : 0.659804500482
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -5.511e+00
     [-----------------98%----------------- ] 9782 of 9980 complete in 12.3 sec
    - resampling
    - moving to gamma : 0.671253804984
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -5.155e+00
     [-----------------96%----------------  ] 9662 of 9980 complete in 12.3 sec
    - moving to gamma : 0.722848715713
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.319e+01
     [-----------------99%----------------- ] 9922 of 9980 complete in 12.8 sec
    - moving to gamma : 0.745653026655
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.023e+01
     [-----------------99%----------------- ] 9902 of 9980 complete in 12.8 sec
    - moving to gamma : 0.763991894902
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -8.216e+00
     [-----------------96%----------------  ] 9642 of 9980 complete in 12.9 sec
    - moving to gamma : 0.78034036673
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -7.317e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 13.3 sec
    - moving to gamma : 0.79580650561
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -6.915e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 13.3 sec
    - moving to gamma : 0.810029354689
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -6.354e+00
     [-----------------97%----------------- ] 9762 of 9980 complete in 13.4 sec
    - moving to gamma : 0.823764430797
    - performing 20 MCMC step(s) per particle
    - ESS  = 69.83 %
    - logZ = -6.131e+00
     [-----------------97%----------------- ] 9742 of 9980 complete in 13.4 sec
    - resampling
    - moving to gamma : 0.836674925253
    - performing 20 MCMC step(s) per particle
    - ESS  = 100.00 %
    - logZ = -5.757e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 13.9 sec
    - moving to gamma : 0.891680716398
    - performing 20 MCMC step(s) per particle
    - ESS  = 95.00 %
    - logZ = -2.450e+01
     [-----------------99%----------------- ] 9922 of 9980 complete in 13.9 sec
    - moving to gamma : 0.917805652025
    - performing 20 MCMC step(s) per particle
    - ESS  = 90.25 %
    - logZ = -1.161e+01
     [-----------------97%----------------  ] 9702 of 9980 complete in 13.7 sec
    - moving to gamma : 0.939023511141
    - performing 20 MCMC step(s) per particle
    - ESS  = 85.74 %
    - logZ = -9.419e+00
     [-----------------100%-----------------] 9982 of 9980 complete in 14.4 sec
    - moving to gamma : 0.957684315327
    - performing 20 MCMC step(s) per particle
    - ESS  = 81.45 %
    - logZ = -8.275e+00
     [-----------------98%----------------- ] 9842 of 9980 complete in 15.4 sec
    - moving to gamma : 0.975175653723
    - performing 20 MCMC step(s) per particle
    - ESS  = 77.38 %
    - logZ = -7.750e+00
     [-----------------98%----------------- ] 9862 of 9980 complete in 15.4 sec
    - moving to gamma : 0.991605900574
    - performing 20 MCMC step(s) per particle
    - ESS  = 73.51 %
    - logZ = -7.274e+00
     [-----------------99%----------------- ] 9902 of 9980 complete in 15.5 sec
    - moving to gamma : 1.0
    - performing 20 MCMC step(s) per particle
    - ESS  = 71.58 %
    - logZ = -3.714e+00
     [-----------------99%----------------- ] 9962 of 9980 complete in 15.5 sec
    - total number of MCMC steps: 1030000
    ---------------
    END SMC MOVE TO
    ---------------

```python
ells = [2, 5]
for i in xrange(2):
    print 'For ell = ' + str(ell) + ', evidence is : ' + str(log_Zs[i])
```

    For ell = 5, evidence is : -476.953643092
    For ell = 5, evidence is : -466.748153197

```python
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))
```

```python

```