# Basic python

```python
# Running code
results = [1, 1, 2, 3]
print(max(results))
```

    3

```python
min_value = min(results)
max_value = max(results)
range_values = max_value - min_value
```

```python
range_values
```

    2

```python
scaled = [(r - min_value)/range_values for r in results]
```

```python
scaled
```

    [0.0, 0.0, 0.5, 1.0]

# Executing bash commands

```python
!pip3 freeze
```

    appnope==0.1.0
    attrs==19.1.0
    backcall==0.1.0
    bleach==3.1.0
    certifi==2019.3.9
    chardet==3.0.4
    cycler==0.10.0
    decorator==4.3.2
    defusedxml==0.5.0
    entrypoints==0.3
    idna==2.8
    ipykernel==5.1.0
    ipympl==0.2.1
    ipython==7.3.0
    ipython-genutils==0.2.0
    ipywidgets==7.4.2
    jedi==0.13.3
    Jinja2==2.10
    jsonschema==3.0.1
    jupyter==1.0.0
    jupyter-client==5.2.4
    jupyter-console==6.0.0
    jupyter-core==4.4.0
    jupyterlab==0.35.4
    jupyterlab-server==0.2.0
    kiwisolver==1.0.1
    MarkupSafe==1.1.1
    matplotlib==3.0.3
    mistune==0.8.4
    nbconvert==5.4.1
    nbformat==4.4.0
    notebook==5.7.6
    numpy==1.16.2
    pandas==0.24.1
    pandocfilters==1.4.2
    parso==0.3.4
    pexpect==4.6.0
    pickleshare==0.7.5
    pkg-resources==0.0.0
    plotly==3.7.0
    prometheus-client==0.6.0
    prompt-toolkit==2.0.9
    ptyprocess==0.6.0
    Pygments==2.3.1
    pyparsing==2.3.1
    pyrsistent==0.14.11
    python-dateutil==2.8.0
    pytz==2018.9
    pyzmq==18.0.1
    qgrid==1.1.1
    qtconsole==4.4.3
    requests==2.21.0
    retrying==1.3.3
    scipy==1.2.1
    seaborn==0.9.0
    Send2Trash==1.5.0
    six==1.12.0
    terminado==0.8.1
    testpath==0.4.2
    tornado==6.0.1
    traitlets==4.3.2
    urllib3==1.24.1
    wcwidth==0.1.7
    webencodings==0.5.1
    widgetsnbextension==3.4.2
    You are using pip version 8.1.1, however version 19.1.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

```python
! head README.md
```

    head: cannot open 'README.md' for reading: No such file or directory

# Jupyter Magics

```python
%time [(r - min_value)/range_values for r in results]
```

    CPU times: user 5 µs, sys: 1 µs, total: 6 µs
    Wall time: 9.06 µs

    [0.0, 0.0, 0.5, 1.0]

# Rich text format

## LaTeX !!

%%latex
\begin{align}
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

$$c = \sqrt{a^2 + b^2}$$

## Free text + images

Supports markdown syntax:
- lists
- headings
- links

# Manipulate data

```python
import qgrid
import pandas as pd
```

```python
df = pd.read_csv('../test_data.csv')
```

```python
qwidget = qgrid.show_grid(df)
qwidget
```

    QgridWidget(grid_options={'boldIndex': True, 'maxVisibleRows': 15, 'defaultColumnWidth': 150, 'rowHeight': 28,…

# Data Visualisation

```python
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/zoo/zoo.data -O zoo.data
%ls
```

    --2019-06-17 20:20:04--  https://archive.ics.uci.edu/ml/machine-learning-databases/zoo/zoo.data
    Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252
    Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 4126 (4.0K) [application/x-httpd-php]
    Saving to: ‘zoo.data’

    zoo.data            100%[===================>]   4.03K  --.-KB/s    in 0s

    2019-06-17 20:20:05 (8.22 MB/s) - ‘zoo.data’ saved [4126/4126]

    1_love_jupyter.ipynb     3_buggy_notebook.ipynb  5_sanity_checks.ipynb
    2_research_python.ipynb  4_unittests.ipynb       zoo.data

```python
import seaborn as sns
import matplotlib.pyplot as plt
```

```python
column_names = ["hair", "feathers", "eggs", "milk", "airborne", "aquatic",
                "predator", "toothed", "backbone", "breathes", "venomous",
                "fins", "legs", "tail", "domestic", "catsize", "type"]

zoo_data = pd.read_csv('zoo.data', names=column_names)
```

```python
zoo_data.head()
```

      hair
      feathers
      eggs
      milk
      airborne
      aquatic
      predator
      toothed
      backbone
      breathes
      venomous
      fins
      legs
      tail
      domestic
      catsize
      type

      aardvark
      1
      0
      0
      1
      0
      0
      1
      1
      1
      1
      0
      0
      4
      0
      0
      1
      1

      antelope
      1
      0
      0
      1
      0
      0
      0
      1
      1
      1
      0
      0
      4
      1
      0
      1
      1

      bass
      0
      0
      1
      0
      0
      1
      1
      1
      1
      0
      0
      1
      0
      1
      0
      0
      4

      bear
      1
      0
      0
      1
      0
      0
      1
      1
      1
      1
      0
      0
      4
      0
      0
      1
      1

      boar
      1
      0
      0
      1
      0
      0
      1
      1
      1
      1
      0
      0
      4
      1
      0
      1
      1

```python
correlation_matrix = zoo_data.corr()
plt.figure(figsize=(20,14))
sns.heatmap(correlation_matrix, cbar=True,  square=True, annot=True,annot_kws={'size': 12}, linewidths=0.15, linecolor='white')
plt.title('Correlation of ZOO Features', y=1.05, size=15)
```

```python

```


```python
!jupyter nbconvert Lecture-22-Phase-Field-Basics.ipynb --to slides --post serve
```

# Lecture 22:  Phase Field Models

### Sections

* [Introduction](#Introduction)
* [Learning Goals](#Learning-Goals)
* [On Your Own](#On-Your-Own)
    * [The Order Parameter](#The-Order-Parameter)
* [In Class](#In-Class)
    * [The Free Energy Functional](#The-Free-Energy-Functional)
    * [The Equation of Motion](#The-Equation-of-Motion)
    * [Analytical Solution](#Analytical-Solution)
    * [Numerical Solution (FiPy)](#Numerical-Solution-(FiPy)
    * [In 2D](#In-2D)
* [Homework](#Homework)
* [Summary](#Summary)
* [Looking Ahead](#Looking-Ahead)
* [Reading Assignments and Practice](#Reading-Assignments-and-Practice)

### Introduction
----

This workbook/lecture is derived from Boettinger, et al. in Annual Review of Materials Research, v32, p163-194 (2002).

doi: 10.1146/annurev.matsci.32.101901.155803

The phase field method makes possible the study of complex microstructural morphologies such as dendritic and eutectic solidification as well as polycrystalline growth.

The major contribution of the method is the introduction of an order parameter used to delineate phases such as solid/liquid, $\alpha~/~\beta$, etc.  The concept of an order parameter is not new.  However, smoothly varying this order parameter through an interphase interface frees us from tracking the interface position and applying boundary conditions at interfaces having complex morphologies.

[Top of Page](#Sections)

### Learning Goals
----

*  Introduction to the idea of an "order parameter".
*  Observe a practical use for the Calculus of Variations.
*  Introduction to what is meant by a non-homogeneous thermodynamic system.
*  Code a simple microstructure simulation.

[Top of Page](#Sections)

### On Your Own
----

Read this small excerpt from Boettinger's paper:

>The method employs a phase-field variable, e.g., $\phi$, which is a function of position and time, to describe whether the material is liquid or solid. The behavior
of this variable is governed by an equation that is coupled to equations for heat
and solute transport. Interfaces between liquid and solid are described by smooth
but highly localized changes of this variable between fixed values that represent
solid and liquid, (in this review, 0 and 1, respectively).

Therein is the key feature of phase field models.  The order (or the phase) is described by a field variable ($\phi$) coupled to heat and mass transfer.  The result is that complex interface shapes do not require tracking of the position of the interface.

This may not have significance to you, however there was a time when knowledge of the position of the interface was required for solidification calculations.  This boundary condition made dendrite computation difficult if not impossible.

#### The Order Parameter

The order parameter can be thought of as an envelope around probability amplitudes of atomic positions.  In the picture below we have a probability density of finding an atom at a particular position.  In this picture $\phi = 0$ might be considered the solid, and $\phi = 1$ would be the liquid.

Using the order parameter in this way makes it easier to calculate solidification microstructures - we no longer have to track the interface (you'll see how this works below).

The shape of this interface is a balance between two forces.  The energy increase for intermediate states between solid and liquid (from the bulk free energy) and energy costs associated with steep gradients in the phase-field order parameter.

[Top of Page](#Sections)

#### Review:  Calculus of Variations

The calculus of variations is a rich mathematical subject.  There are many books on the topic.  One of the canonical problems in the subject area is to compute the shortest arc between to points on a plane.  This, is a good place to begin your study of the topic.  For now I'll describe the major output of the CoV and the points relevant to phase field.

The analogy between calculus and CoV is good.  If finding the minimum of a function can be done by inspecting derivatives then the minimum of a functional can be found by inspecting the so-called 'variational derivative'.  In particular the first chapter of Lev. D. Elsgolc's book "Calculus of Variations" presents this idea nicely.

The CoV gives us the Euler-Lagrange equation.  This is the main tool of CoV:

$$ \frac{\delta F}{\delta \phi} = \frac{\partial F}{\partial \phi} - \frac{\partial}{\partial x} \frac{\partial F}{\partial \nabla \phi} = 0$$

The scalar and gradient terms in $\phi$ are treated as independent variables.  This equation is telling us that the function that minimizes the functional is the solution to the above differential equation.

[Top of Page](#Sections)

### In Class
----

#### The Free Energy Functional

The Langragian is constructed from the free energy functional (and integrated over all space, V) and that, in turn is constructed from the bulk free energy and the gradient energy thus:

$$L(\phi,\nabla\phi) = \int_V \Big[ ~~f(\phi,T) + \frac{\epsilon^2_\phi}{2}|\nabla \phi|^2~\Big]~ dV$$

Where L is the Lagrangian, the functional (hereafter F) is in the square brackets, V is the volume of the system, $\phi$ is the order parameter, T is the temperature, $\epsilon$ is the gradient energy coefficient and $\nabla$ has the usual meaning.

$f(\phi,T)$ is the free energy density.  This is often referred to as the 'bulk' free energy term and the terms in $\nabla\phi$ are the gradient energy terms.  You may also hear these terms referred to as the 'non-classical' terms.

At equilibrium the variational derivatives must satisfy the following:

$$ \frac{\delta F}{\delta \phi} = \frac{\partial f}{\partial \phi} - \epsilon^2_\phi \nabla^2 \phi = 0$$

Recall the definition of the Euler-Lagrange equation:

$$ \frac{\delta F}{\delta \phi} = \frac{\partial F}{\partial \phi} - \frac{\partial}{\partial x} \frac{\partial F}{\partial \nabla \phi} = 0$$

and the free energy functional:

$$f(\phi,T) + \frac{\epsilon^2_\phi}{2}|\nabla \phi|^2$$

The first equation above tells us that the function $\phi(x,t)$ is unchanging.  We will compute this interface profile below.  To develop a kinetic expression we make an educated guess (also known as an "ansatz" in the phase field literature) about the relaxation of a system towards equilibrium.

[Top of Page](#Sections)

#### The Equation of Motion

We assume the following functional form for the equation of motion:

$$\frac{\partial \phi}{\partial t} = - M_\phi \frac{\delta F}{\delta \phi}$$

This is the simplest expression that guarantees the free energy of the system will decrease over time.  In this form the phase-field variable, $\phi$, is non-conserved.  The conserved form takes the divergence of the expression above as is done when expression accumulation in a control volume (as in Fick's second law).

Our equation of motion is therefore:

$$\frac{\partial \phi}{\partial t} = - M_\phi \Big[\frac{\partial f}{\partial \phi} - \epsilon^2_\phi \nabla^2 \phi \Big]$$

[Top of Page](#Sections)

#### Building the Bulk Free Energy '$f$'

There are two so-called helper functions that homogenize the free energy.  The interpolating function, $p(\phi)$ and the double well function $g(\phi)$.

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib notebook

def plot_p_and_g():
    phi = np.linspace(-0.1, 1.1, 200)
    g=phi**2*(1-phi)**2
    p=phi**3*(6*phi**2-15*phi+10)

    # Changed 3 to 1 in the figure call.
    plt.figure(1, figsize=(12,6))
    plt.subplot(121)
    plt.plot(phi, g, linewidth=1.0);
    plt.xlabel('$\phi$', fontsize=18)
    plt.ylabel('$g(\phi)$', fontsize=18)

    plt.subplot(122)
    plt.plot(phi, p, linewidth=1.0);
    plt.xlabel('$\phi$', fontsize=18)
    plt.ylabel('$p(\phi)$', fontsize=18)

    return
```

```python
plot_p_and_g()
```

    <IPython.core.display.Javascript object>

We start by using the ordinary free energy of the pure components:

 * Pure A, liquid phase - $f_A^L(T)$
 * Pure A, solid phase - $f_A^S(T)$

As we will be limiting ourselves to a pure material at this time, these are the only two free energies we need.  Near the melting point these free energies are often modeled as straight lines using the relationship for the Gibbs free energy:

$$G = H - TS$$

taking H and S to be constants.  Following conventions of the phase diagram modeling community we take the reference state of the component A to be the equilibrium phase at STP.  If this were a metal like Cu then the reference state would be the FCC phase.  For us, that will be the SOLID.  This sets:

$$f_A^S(T) = 0$$

Expanding the difference in free energy between the solid and the liquid around the melting point results in:

$$f_A^L(T)-f_A^S(T) = L_A \frac{(T_M^A - T)}{T_M^A}$$

The next step is to homogenize the free energy for component A.  We build the free energy $f(\phi,T)_A$ as follows:

$$f(\phi,T)_A = W_A~g(\phi) + f_L p(\phi) + f_S (1-p(\phi))$$

so that:

$$f(\phi,T)_A = W_A~g(\phi) + L_A \frac{(T_M^A - T)}{T_M^A}p(\phi)$$

Let us plot this and see what it looks like.

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.ticker import LinearLocator, FormatStrFormatter

def plot_homogeneous_F():
    plt.fig = plt.figure(2, figsize=(10,10))
    plt.ax = plt.fig.gca(projection='3d')

    phi = np.linspace(0.0, 1.0, 100)
    temperature = np.linspace(0.0, 1.0, 100)
    phi,temperature = np.meshgrid(phi,temperature)

    W=30.0
    L=1.0
    Tm=0.5
    g=phi**2*(1-phi)**2
    p=phi**3*(6*phi**2-15*phi+10)

    f = W*g+L*p*(Tm-temperature)/Tm

    energyPlot = plt.ax.plot_surface(phi, temperature, f, label=None,
                    cmap=plt.cm.coolwarm, rstride=5, cstride=5, alpha=0.5)
    energyPlot = plt.contour(phi, temperature, f,20)

    plt.clabel(energyPlot, inline=1, fontsize=10)
    plt.ax.set_xlabel('$\phi$')
    plt.ax.set_ylabel('T')
    plt.ax.set_zlabel('$f(\phi,t)$')

    return
```

```python
plot_homogeneous_F()
```

    <IPython.core.display.Javascript object>

$$L(\phi,\nabla\phi) = \int_V \Big[ ~~f(\phi,T) + \frac{\epsilon^2_\phi}{2}|\nabla \phi|^2~\Big]~ dV$$

From here it should be clear that the free energy space is homogeneous in the order parameter and temperature.  Now that the description of the bulk energy is complete we can return to the Euler-Lagrange equations and proceed to develop our equilibrium solution and our equations of motion.

[Top of Page](#Sections)

#### Analytical Solution

The full expression for the equation of motion is:

$$\frac{\partial \phi}{\partial t} = - M_\phi \epsilon^2
\Big[\nabla^2\phi- \frac{2W_A}{\epsilon^2} \phi(1-\phi)(1-2\phi)\Big]-\frac{30 M_\phi L_A}{T_M^A}(T_M^A - T)\phi^2(1-\phi)^2$$

While this is correct - it is often not explicitly written out like this when solving the equations numerically.  Further, it is better if you don't fully expand the derivatives when attempting the analytical solution.

There is a fair bit of algebra and a few assumptions that enable the solution to the Euler-Lagrange equation above.  I will state the procedue and leave out the gory detail.  Remember we are after the expression that gives us $\phi(x)$.  Keep this in mind as you read through the following bullet points.

 * First - assume that you are at the melting temperature.  The rationale is that this is the only temperature where BOTH phases CO-EXIST.  Any other temperature and it does not make sense to discuss an interface.  This removes the second term on the RHS of the above expression.
 * Second - you can use $\frac{d\phi}{dx}$ as an integrating factor to take the first integral of the Euler-Lagrange equation.
 * Third - after evaluating the constant (C=0 is the answer, but, why it is zero will test your reasoning skills) the equation is seperable and can be integrated.

 The result is:

 $$\phi(x) = \frac{1}{2} \Big[ 1 + \tanh \Big( \frac{x}{2\delta} \Big) \Big]$$

 where $\delta$ is related to the W and $\epsilon$ parameters.

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interact, fixed

fig = None
def plot_equilibrium(W=500.0, epsilon=1.0):
    global fig
    if fig: plt.close(fig)
    fig = plt.figure()
    x = np.linspace(-1.0, 1.0, 200)
    phi = 0.5*(1+np.tanh(x*np.sqrt(2*W)/(2*epsilon)))
    plt.plot(x, phi, linewidth=1.0)
    plt.xlabel('$x$', fontsize=24)
    plt.ylabel('$\phi(x)$', fontsize=24)
    return
```

```python
print 'Hello!'
```

    Hello!

```python
interact(plot_equilibrium, W=(1,1000,10), epsilon=fixed(1.0))
```

    <IPython.core.display.Javascript object>

W and $\epsilon$ can be parameterized in terms of the surface energy and the interface thickness to make a connection with the physical world.

[Top of Page](#Sections)

#### Numerical Solution (FiPy)

```python
%matplotlib osx
from fipy import *
```

```python
L = 1.
nx = 400
dx = L/nx
mesh = Grid1D(dx=dx, nx=nx)
phase = CellVariable(name="phase",mesh=mesh)
viewer = MatplotlibViewer(vars=(phase,),datamin=-0.1, datamax=1.1, legend=None)
```

This cell sets the initial conditions.  There is a helper attribute cellCenters that fetches a list of the x points.  The setValue helper functions and the 'where' keyword help you to set the initial conditions.  FiPy is linked to Matplotlib and once you created the viewer object you call .plot() to update.

```python
x = mesh.cellCenters
phase.setValue(1.)
phase.setValue(0., where=x > L/2)
viewer.plot()
```

[Top of Page](#Sections)

#### DIY:  Code the terms to complete the phase field description.

$$\frac{\partial \phi}{\partial t} = - M_\phi \epsilon^2
\Big[\nabla^2\phi- \frac{2W_A}{\epsilon^2} \phi(1-\phi)(1-2\phi)\Big]-\frac{30 M_\phi L_A}{T_M^A}(T_M^A - T)\phi^2(1-\phi)^2$$

```python
import sympy as sp
phi = sp.symbols('phi')
sp.init_printing()
```

```python
((1-phi)**2*(phi**2)).diff(phi).simplify()
```

```python
(phi**3*(6*phi**2-15*phi+10)).diff(phi).simplify()
```

$$\frac{\partial \phi}{\partial t} = - M_\phi \Big[\frac{\partial f}{\partial \phi} - \epsilon^2_\phi \nabla^2 \phi \Big]$$

$$f(\phi,T)_A = W_A~g(\phi) + L_A \frac{(T_M^A - T)}{T_M^A}p(\phi)$$

```python
eps_sqrd = 0.00025
M = 1.0
W = 0.5
Lv = 1.
Tm = 1.
T = 1.0
enthalpy = Lv*(Tm-T)/Tm
S0 = W*2.0*phase*(phase-1.0)*(2*phase-1.0) + 30*phase**2*(phase**2-2*phase+1)*enthalpy
```

This is our general statement of a diffusive PDE.  There is a transient term and a source term.  Translate from the description of the phase field model above.

```python
eq = TransientTerm() == DiffusionTerm(coeff=eps_sqrd*M) - S0
```

```python
for i in range(50):
    eq.solve(var = phase, dt=0.1)
    viewer.plot()
```

Just re-execute this cell after you change parameters.  You can execute it over and over until you are satisfied that you've reached equilibrium.

You can try changing $\kappa$, W, and T.  Changing T from the melting tmperature will result in a moving interface.  This is where things get interesting!

[Top of Page](#Sections)

#### In 2D

This one is important.  We will simulate a pair of curved particles (each with a different radius) at the melting temperature.  What do you think will happen?

```python
%matplotlib osx
from fipy import *

L = 1.
nx = 200
dx = L/nx
dy = L/nx
mesh = Grid2D(dx=dx, dy=dx, nx=nx, ny=nx)
phase = CellVariable(name="phase", mesh=mesh)
x = mesh.cellCenters()[0]
y = mesh.cellCenters()[1]
```

```python
phase.setValue(1.)
x0 = 0.0
y0 = 0.0
#phase.setValue(0., where=(
#                          ((x-x0)**2+(y-y0)**2 > L/3) & ((x-L)**2+(y-L)**2 > 0.2)
#                          )
#               )
phase.setValue(ExponentialNoiseVariable(mesh=mesh, mean=0.5))
viewer = Matplotlib2DGridViewer(vars=phase, datamin=0.0, datamax=1.0)
viewer.plot()
```

The strength of FiPy is that you can use the same code here in 2D as above.

```python
eps_sqrd = 0.00025
M = 1.0
W = 0.5
Lv = 1.
Tm = 1.
T = 1.
enthalpy = Lv*(Tm-T)/Tm
S0 = W*2.0*phase*(phase-1.0)*(2*phase-1.0) + 30*phase**2*(phase**2-2*phase+1)*enthalpy
```

```python
eq = TransientTerm() == DiffusionTerm(coeff=eps_sqrd) - S0
for i in range(500):
    eq.solve(var = phase, dt=0.05)
    viewer.plot()
```

[Top of Page](#Sections)


```python
from resources.workspace import *
from IPython.display import display
from scipy.integrate import odeint
import copy

%matplotlib inline
```

# Dynamics of ensembles and perturbations

In the following, we will neglect the important issues of model deficiencies and the inherent stochasticity of certain dynamics.  Instead, we will focus on the simplified scenario where we assume:

 we can perfectly model and compute the purely deterministic dynamics; and

 prediction error originates soley from the uncertainty in initial conditions.

Understanding that perturbations rapidly diverge even in a chaotic system as described above, this led to the transition from single-trajectory forecasts to ensemble-based forecasts.   Ensembles are used to "average out" our initialization errors, and to understand the variability and uncertainty in forecasts.

The advantages of ensemble based forecasts over single trajectory forecasts historically led to a search for perturbations that are most representative of the error growth in operational forecasts.  Prediction centers have sought to initialize ensemble-based forecasts in a way to best capture the variability induced by the dynamical chaos.  Two major techniques emerged,

 "bred vectors", and

 forcing singular vectors".

These lead to different formulations of the classical "Lyapunov vectors".  We do not stress here what a "Lyapunov vector" is, rather we will discover their nature experimentally in the following work.  This will lead to a formal definition of one type of Lyapunov vectors by the end of the exercises.

### "Breeding" growing modes

Suppose we have a smooth, nonlinear dynamical system,

$$
\begin{align}
\dot{\mathbf{x}} = f(\mathbf{x}) & & \mathbf{x} \in \mathbb{R}^n,
\end{align}
$$

and a precise estimate of an initial condition $\mathbf{x}^c_0$, from which we want to make a forecast.  Suppose, also, that there are future observations that we will assimilate.

It was suggested by Toth and Kalnay to use the evolution of the initial estimate $\mathbf{x}^c_0$ as a control trajectory, while introducing small perturbations to generate an ensemble,

$$\begin{align}
\mathbf{x}^i_0 = \mathbf{x}^c_0 + \boldsymbol{\delta}^i_0
\end{align}$$

where $\left \rvert \boldsymbol{\delta}^i \right \rvert  = \epsilon \ll 1$.

The ensemble is evolved in parallel to the control trajectory.  Between times $t_{k-1}$ and $t_k$, this takes the form

$$\begin{align}
\widehat{\mathbf{x}}^c_k &= \mathbf{x}_{k-1}^c + \int_{t_{k-1}}^{t_k} f(x) {\rm d}t \\
\widehat{\mathbf{x}}^i_k &= \mathbf{x}_{k-1}^i + \int_{t_{k-1}}^{t_k} f(x) {\rm d}t.
\end{align}
$$

At the point of analyzing new observations we form a new estimate for the control trajectory, taking $\widehat{\mathbf{x}}_k^c$ to $\mathbf{x}_k^c$. the perturbations are rescaled back to their original small size while maintaining their directions.  That is to say,

$$
\begin{align}
\widehat{\boldsymbol{\delta}}_k^i \triangleq \mathbf{x}_k^c - \widehat{\mathbf{x}}_k^i, & &
\boldsymbol{\delta}_k^i \triangleq \frac{\epsilon}{\left\rvert \widehat{\boldsymbol{\delta}}^i_k\right\rvert} \widehat{\boldsymbol{\delta}}^i_k.
\end{align}
$$

"Breeding growing modes" is designed to simulate how the modes of fast growing error are maintained and propagated through the successive use of short range forecasts.  The resulting perturbations are thus meant to represent a perturbation field of the "errors of the day", i.e., uncertainties in the initial condition at the present time that result from the repeated cycle of forecasts and analyses.

**Exc 4.14**: Run the code below and use the sliders to examine behavior of successive "breeding" of growing modes. The parameter B stands for the number of breeding cycles.  The parameter eps stands for the re-scaling parameter $\epsilon$ defined above.  The parameter N is the number of perturbations.

The the plots on the left hand side show the evolution of the control trajectory and the perturbed trajectories along each breeding cycle.  The right hand side plots $\pm 1$ times the normalized perturbations,

$$
 \frac{ \pm\widehat{\boldsymbol{\delta}}_k^i}{\left\rvert \widehat{\boldsymbol{\delta}}^i_k\right\rvert},
$$

giving the directions of the perturbations, plotted as lines through the unit sphere.

**Answer the following questions**:

 For small values of $\epsilon$, what is significant about the long term behavior of the directions of the perturbations?

 Does this behavior change with large N, i.e., more directions for the pertubations?

 How does this behavior change when $\epsilon$ is increased?  Do the directions of the perturbations depend on N for large $\epsilon$?

```python
SIGMA = 10.0
BETA  = 8/3
RHO   = 28.0

def dxdt(xyz, t0, sigma=SIGMA, beta=BETA, rho=RHO):
    """Compute the time-derivative of the Lorenz-63 system."""
    x, y, z = xyz
    return array([
        sigma * (y - x),
        x * (rho - z) - y,
        x * y - beta * z
    ])

def animate_bred_vectors(B=0, eps=0.01, N=10):

    # Initial conditions: perturbations around some "proto" state
    sigma=SIGMA
    beta=BETA
    rho=RHO
    T=0.05

    seed(1)
    x_0 = array([-6.1, 1.2, 32.5])               # define the control

    # define the perturbations, randomly generated but of fixed norm epsilon
    perts = randn([N,3])
    perts = array([eps * perts[i] / sqrt(perts[i] @ perts[i]) for i in range(N)])
    delta_x = x_0 + perts

    tt = linspace(0, T, 10)           # Time instances for trajectory
    d2 = lambda x,t: dxdt(x,t, sigma,beta,rho)  # Define dxdt(x,t) with fixed params.

    # for each breeding cycle
    for kk in range(B):
        # Compute trajectories
        x_traj = array([odeint(d2, x_0, tt)])        # integrate the control trajectory
        x_0 = np.squeeze(x_traj[:, -1, :])

        delta_x_traj = array([odeint(d2, delta_xi, tt) for delta_xi in delta_x]) # Integrate the perturbations
        perts = delta_x_traj[:, -1, :] - x_0                                     # redefine the perts
        perts = array([eps * perts[i] / sqrt(perts[i] @ perts[i]) for i in range(N)])
        delta_x = x_0 + perts # redefine the initialization of the perturbed trajectories

    # PLOTTING
    fig = plt.figure(figsize=(16,8))
    ax1 = plt.subplot(121, projection='3d')
    ax2 = plt.subplot(122, projection='3d')

    if B==0:
        ax1.scatter3D(*x_0, s=40, c='k')

    else:
        ax1.plot(*x_traj[0,:,:].T, '-', c='k')
        ax1.scatter3D(*x_traj[0,-1,:].T, '-', s=40, c='k')

    colors = plt.cm.jet(linspace(0,1,N))
    for i in range(N):
        # for each breeding cycle
        if B==0:
            # if just the initial conditions, we plot these
            ax1.scatter3D(*delta_x[i,:],s=40,c=colors[i])

        else:
            # otherwise, plot the trajectories over a breeding cycle
            ax1.plot(*(delta_x_traj[i,:,:].T),'-'  ,c=colors[i])
            ax1.scatter3D(*delta_x_traj[i,-1,:],s=40,c=colors[i])

        # we plot the normalized perturbations on the unit sphere
        tmp = perts[i,:]/sqrt(perts[i,:] @ perts[i, :])
        p_vect = np.concatenate([np.reshape([0,0,0],[1,3]), np.reshape(tmp,[1,3])], axis=0)

        # delta * +1
        ax2.plot(p_vect[:,0], p_vect[:,1], p_vect[:,2],'-'  ,c=colors[i])
        ax2.scatter3D(*tmp[:],s=40,c=colors[i], marker='o')

        # delta * -1
        ax2.plot(-1*p_vect[:,0], -1*p_vect[:,1], -1*p_vect[:,2],'-'  ,c=colors[i])
        ax2.scatter3D(*tmp[:]*(-1),s=40,c=colors[i], marker='o')

    ax1.axis('off')
    ax2.set_xlim((-.9, .9))
    ax2.set_ylim((-.9, .9))
    ax2.set_zlim((-.9, .9))
    plt.show()

w = interactive(animate_bred_vectors,B=(0,175,25), eps=(0.01,1.61, .2), N=(1, 161, 20))
w
```

**Exc 4.16**: If the "breeding" of perturbations is meant to represent the unstable growth of initial perturbations, what can we learn from their growth rates?  In the following code, fill in the missing lines to define a function that will compute the log-growth rate of the perturbation $\boldsymbol{\delta^i}_k$, i.e., the log growth relative to the length of time in the breeding interval.  This function should return

$$\begin{align}
\frac{1}{T}\log \left( \frac{\left\rvert \widehat{\boldsymbol{\delta}}^i_k\right \rvert}{\left\rvert \boldsymbol{\delta}_{k-1}^i \right\rvert}\right) \equiv \frac{1}{T}\log \left( \frac{\left\rvert \widehat{\boldsymbol{\delta}}^i_k\right \rvert}{\epsilon}\right),
\end{align}$$

for a single perturbation.

```python
def log_growth(x_control_k, x_pert_k, T, eps):
    """function returns array of the log growth values for a single perturbation"""
    nrm = sqrt( (x_pert_k - x_control_k) @ (x_pert_k - x_control_k).T )

    log_growth_rate = (1.0 / T) * log(nrm / eps)

    ### Fill in missing line(s) here ###

    return log_growth_rate
```

```python
## Example solution

# show_answer('log_growth')
```

**Exc 4.18**: Test your answer to **Exc 4.16**.  Using the code and slider below, investigate the distributions of the log-growth rates of the bred vectors as a function of the number of breeding cycles.  Answer the following questions:

 What is the long term behaviour of this distribution?

 The leading Lyapunov exponent of the Lorenz-63 system is $\approx 0.9050$, what do you notice about the mean of the log-growth rates over a long number of breeding cycles?

 Consider the behavior of small perturbations from Exc 4.14. Can you conjecture what the perturbations are converging to?

 What does this suggest about the "representative perturbations" for the error growth in chaotic systems?

```python
def animate_bred_growth_rates(B=1000):

    # Initial conditions: perturbations around some "proto" state
    sigma=SIGMA
    beta=BETA
    rho=RHO
    eps=0.01
    N=1
    T=0.01

    seed(1)
    x_0 = array([-6.1, 1.2, 32.5])               # define the control

    # define the perturbation, randomly generated but of fixed norm epsilon
    perts = randn(3)
    perts = eps * perts / sqrt(perts @ perts)
    delta_x = x_0 + perts

    tt = linspace(0, T, 20)           # Time instances for trajectory
    d2 = lambda x,t: dxdt(x,t, sigma,beta,rho)  # Define dxdt(x,t) with fixed params.
    grwt = np.zeros(B)

    # for each breeding cycle
    for kk in range(B):
        # Compute trajectories
        x_traj = array([odeint(d2, x_0, tt)])        # integrate the control trajectory
        x_0 = np.squeeze(x_traj[:, -1, :])

        delta_x_traj = array([odeint(d2, delta_x, tt)]) # Integrate the perturbation

        # compute the log growth from the code defined earlier
        grwt[kk] = log_growth(x_0, delta_x_traj[0, -1, :], T, eps)

        # redefine perts
        perts = delta_x_traj[0, -1, :] - x_0
        perts = eps * perts / sqrt(perts @ perts.T)
        delta_x = x_0 + perts

    # PLOTTING
    fig = plt.figure(figsize=(16,8))
    ax = plt.subplot(111)
    ax.hist(grwt, bins=linspace(-20,20,4001), density=True)
    ax.set_xlim((-12, 12))
    ax.set_ylim((0, 0.8))
    ax.text(4, 0.6, 'Mean log-growth rate=' + str(np.round(mean(grwt),decimals=4)).zfill(4), size=20)

    plt.show()

w = interactive(animate_bred_growth_rates,B=(1000,30000, 5800))
w
```

### Next: [Lyapunov exponents and eigenvalues](T3 - Lyapunov exponents and eigenvalues.ipynb)

tutorials/DA and the Dynamics of Ensemble Based Forecasting/T2 - Dynamics of ensembles and perturbations.ipynb


```
# This notebook illustrates how to use numpyro
# https://github.com/pyro-ppl/numpyro

# Speed comparison with TFP
# https://rlouf.github.io/post/jax-random-walk-metropolis/
# Speed comparison with pymc3
# https://www.kaggle.com/s903124/numpyro-speed-benchmark

```

# Installation

```
# Standard Python libraries
from __future__ import absolute_import, division, print_function, unicode_literals

import os
import time
#import numpy as np
#np.set_printoptions(precision=3)
import glob
import matplotlib.pyplot as plt
import PIL
import imageio

from IPython import display
%matplotlib inline

import sklearn

import seaborn as sns;
sns.set(style="ticks", color_codes=True)

import pandas as pd
pd.set_option('precision', 2) # 2 decimal places
pd.set_option('display.max_rows', 20)
pd.set_option('display.max_columns', 30)
pd.set_option('display.width', 100) # wide windows
```

    /usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```
# As of 5/25/20, colab has jax=0.1.67 and jaxlib=0.1.47 builtin

import jax
import jax.numpy as np
import numpy as onp # original numpy
from jax import grad, hessian, jit, vmap, random

print("jax version {}".format(jax.__version__))
```

    jax version 0.1.67

```
# Check if GPU is available
!nvidia-smi

# Check if JAX is using GPU

print("jax backend {}".format(jax.lib.xla_bridge.get_backend().platform))
```

    Mon May 25 21:46:40 2020
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 440.82       Driver Version: 418.67       CUDA Version: 10.1     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   41C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+
    jax backend gpu

```

#https://github.com/pyro-ppl/numpyro/issues/531

# https://github.com/pyro-ppl/numpyro
!pip install numpyro # requires jax=0.1.57, jaxlib=0.1.37

print("jax version {}".format(jax.__version__))
print("jax backend {}".format(jax.lib.xla_bridge.get_backend().platform))

```

    Collecting numpyro
      Downloading https://files.pythonhosted.org/packages/b8/58/54e914bb6d8ee9196f8dbf28b81057fea81871fc171dbee03b790336d0c5/numpyro-0.2.4-py3-none-any.whl (159kB)
         |████████████████████████████████| 163kB 6.4MB/s
    Collecting jaxlib==0.1.37
      Downloading https://files.pythonhosted.org/packages/24/bf/e181454464b866f30f09b5d74d1dd08e8b15e032716d8bcc531c659776ab/jaxlib-0.1.37-cp36-none-manylinux2010_x86_64.whl (25.4MB)
         |████████████████████████████████| 25.4MB 4.8MB/s
    Collecting jax==0.1.57
      Downloading https://files.pythonhosted.org/packages/ae/f2/ea981ed2659f70a1d8286ce41b5e74f1d9df844c1c6be6696144ed3f2932/jax-0.1.57.tar.gz (255kB)
         |████████████████████████████████| 256kB 42.9MB/s
    Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from numpyro) (4.41.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37->numpyro) (1.4.1)
    Requirement already satisfied: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37->numpyro) (1.18.4)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37->numpyro) (1.12.0)
    Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37->numpyro) (3.10.0)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37->numpyro) (0.9.0)
    Requirement already satisfied: opt_einsum in /usr/local/lib/python3.6/dist-packages (from jax==0.1.57->numpyro) (3.2.1)
    Collecting fastcache
      Downloading https://files.pythonhosted.org/packages/5f/a3/b280cba4b4abfe5f5bdc643e6c9d81bf3b9dc2148a11e5df06b6ba85a560/fastcache-1.1.0.tar.gz
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.0->jaxlib==0.1.37->numpyro) (46.3.0)
    Building wheels for collected packages: jax, fastcache
      Building wheel for jax (setup.py) ... done
      Created wheel for jax: filename=jax-0.1.57-cp36-none-any.whl size=297709 sha256=c2bea348ae9097f522298dfd173b13b80d87b6b3a8694218cc7f41561b5baef6
      Stored in directory: /root/.cache/pip/wheels/8a/b4/75/859bcdaf181569124306615bd9b68c747725c60bfa68826378
      Building wheel for fastcache (setup.py) ... done
      Created wheel for fastcache: filename=fastcache-1.1.0-cp36-cp36m-linux_x86_64.whl size=39211 sha256=8ccffe72cb0f057afa1a0020eba9ee41bfde1c83b251e4a4f8c5051f751b9233
      Stored in directory: /root/.cache/pip/wheels/6a/80/bf/30024738b03fa5aa521e2a2ac952a8d77d0c65e68d92bcd3b6
    Successfully built jax fastcache
    Installing collected packages: jaxlib, fastcache, jax, numpyro
      Found existing installation: jaxlib 0.1.47
        Uninstalling jaxlib-0.1.47:
          Successfully uninstalled jaxlib-0.1.47
      Found existing installation: jax 0.1.67
        Uninstalling jax-0.1.67:
          Successfully uninstalled jax-0.1.67
    Successfully installed fastcache-1.1.0 jax-0.1.57 jaxlib-0.1.37 numpyro-0.2.4

    jax version 0.1.67
    jax backend gpu

```
'''
#https://github.com/pyro-ppl/numpyro/issues/531

#!pip install --upgrade jax==0.1.57
#!pip install --upgrade jaxlib==0.1.37

#!pip install --upgrade -q https://storage.googleapis.com/jax-releases/cuda$(echo $CUDA_VERSION | sed -e 's/\.//' -e 's/\..*//')/jaxlib-$(pip search jaxlib | grep -oP '[0-9\.]+' | head -n 1)-cp36-none-linux_x86_64.whl
#!pip install --upgrade -q jax

ver = !echo $CUDA_VERSION
print(ver)

# install jaxlib
PYTHON_VERSION='cp36'  # alternatives: cp36, cp37, cp38
CUDA_VERSION='cuda101'  # alternatives: cuda92, cuda100, cuda101, cuda102
PLATFORM='linux_x86_64'  # alternatives: linux_x86_64
BASE_URL='https://storage.googleapis.com/jax-releases'
fname = f'{BASE_URL}/{CUDA_VERSION}/jaxlib-0.1.37-{PYTHON_VERSION}-none-{PLATFORM}.whl'
print(fname)
#!pip install --upgrade $BASE_URL/$CUDA_VERSION/jaxlib-0.1.37-$PYTHON_VERSION-none-$PLATFORM.whl
!pip install --upgrade $fname

!pip install numpyro
!pip install --upgrade jax==0.1.57
print("jax backend {}".format(jax.lib.xla_bridge.get_backend().platform))
'''

```

    ['cuda101']
    https://storage.googleapis.com/jax-releases/cuda101/jaxlib-0.1.37-cp36-none-linux_x86_64.whl
    Collecting jaxlib==0.1.37
      Downloading https://storage.googleapis.com/jax-releases/cuda101/jaxlib-0.1.37-cp36-none-linux_x86_64.whl (48.3MB)
         |████████████████████████████████| 48.3MB 66kB/s
    Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37) (1.4.1)
    Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37) (1.12.0)
    Requirement already satisfied, skipping upgrade: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37) (1.18.4)
    Requirement already satisfied, skipping upgrade: absl-py in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37) (0.9.0)
    Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from jaxlib==0.1.37) (3.10.0)
    Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.0->jaxlib==0.1.37) (46.3.0)
    ERROR: numpyro 0.2.4 has requirement jax>=0.1.65, but you'll have jax 0.1.57 which is incompatible.
    ERROR: numpyro 0.2.4 has requirement jaxlib>=0.1.45, but you'll have jaxlib 0.1.37 which is incompatible.
    Installing collected packages: jaxlib
      Found existing installation: jaxlib 0.1.47
        Uninstalling jaxlib-0.1.47:
          Successfully uninstalled jaxlib-0.1.47
    Successfully installed jaxlib-0.1.37

    Requirement already satisfied: numpyro in ./numpyro (0.2.4)
    Processing /root/.cache/pip/wheels/3d/8d/d8/b0463ab20eb85b4ae7c602f7fbc0bd890f2af483b61e6d6096/jax-0.1.68-cp36-none-any.whl
    Collecting jaxlib>=0.1.45
      Using cached https://files.pythonhosted.org/packages/ea/c0/64c0e5a2c6da1d3ffdec95da74abf14df2c7508776ff5f155461fec1ef1d/jaxlib-0.1.47-cp36-none-manylinux2010_x86_64.whl
    Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from numpyro) (4.41.1)
    Requirement already satisfied: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro) (1.18.4)
    Requirement already satisfied: opt-einsum in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro) (3.2.1)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro) (0.9.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from jaxlib>=0.1.45->numpyro) (1.4.1)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from absl-py->jax>=0.1.65->numpyro) (1.12.0)
    Installing collected packages: jax, jaxlib
      Found existing installation: jax 0.1.57
        Uninstalling jax-0.1.57:
          Successfully uninstalled jax-0.1.57
      Found existing installation: jaxlib 0.1.37
        Uninstalling jaxlib-0.1.37:
          Successfully uninstalled jaxlib-0.1.37
    Successfully installed jax-0.1.68 jaxlib-0.1.47

    Processing /root/.cache/pip/wheels/8a/b4/75/859bcdaf181569124306615bd9b68c747725c60bfa68826378/jax-0.1.57-cp36-none-any.whl
    Requirement already satisfied, skipping upgrade: fastcache in /usr/local/lib/python3.6/dist-packages (from jax==0.1.57) (1.1.0)
    Requirement already satisfied, skipping upgrade: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from jax==0.1.57) (1.18.4)
    Requirement already satisfied, skipping upgrade: absl-py in /usr/local/lib/python3.6/dist-packages (from jax==0.1.57) (0.9.0)
    Requirement already satisfied, skipping upgrade: opt-einsum in /usr/local/lib/python3.6/dist-packages (from jax==0.1.57) (3.2.1)
    Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from absl-py->jax==0.1.57) (1.12.0)
    ERROR: numpyro 0.2.4 has requirement jax>=0.1.65, but you'll have jax 0.1.57 which is incompatible.
    Installing collected packages: jax
      Found existing installation: jax 0.1.68
        Uninstalling jax-0.1.68:
          Successfully uninstalled jax-0.1.68
    Successfully installed jax-0.1.57

    jax backend cpu

```
'''

# The latest version uses jax >= 0.1.65, jaxlib >= 0.1.45
# https://github.com/pyro-ppl/numpyro/blob/master/setup.py

#https://medium.com/@ashwindesilva/how-to-use-google-colaboratory-to-clone-a-github-repository-e07cf8d3d22b

!git clone https://github.com/pyro-ppl/numpyro.git
%cd numpyro
!pip install -e .[dev]

print("jax version {}".format(jax.__version__))
print("jax backend {}".format(jax.lib.xla_bridge.get_backend().platform))
'''

```

    fatal: destination path 'numpyro' already exists and is not an empty directory.
    /content/numpyro
    Obtaining file:///content/numpyro
    Requirement already satisfied: jax>=0.1.65 in /usr/local/lib/python3.6/dist-packages (from numpyro==0.2.4) (0.1.68)
    Requirement already satisfied: jaxlib>=0.1.45 in /usr/local/lib/python3.6/dist-packages (from numpyro==0.2.4) (0.1.47)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from numpyro==0.2.4) (4.41.1)
    Requirement already satisfied: ipython in /usr/local/lib/python3.6/dist-packages (from numpyro==0.2.4) (5.5.0)
    Requirement already satisfied: isort in /usr/local/lib/python3.6/dist-packages (from numpyro==0.2.4) (4.3.21)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro==0.2.4) (0.9.0)
    Requirement already satisfied: opt-einsum in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro==0.2.4) (3.2.1)
    Requirement already satisfied: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from jax>=0.1.65->numpyro==0.2.4) (1.18.4)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from jaxlib>=0.1.45->numpyro==0.2.4) (1.4.1)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (0.7.5)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (0.8.1)
    Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (4.3.3)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (1.0.18)
    Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (4.4.2)
    Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (2.1.3)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (4.8.0)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.6/dist-packages (from ipython->numpyro==0.2.4) (46.3.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from absl-py->jax>=0.1.65->numpyro==0.2.4) (1.12.0)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.2->ipython->numpyro==0.2.4) (0.2.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->numpyro==0.2.4) (0.1.9)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != "win32"->ipython->numpyro==0.2.4) (0.6.0)
    Installing collected packages: numpyro
      Found existing installation: numpyro 0.2.4
        Can't uninstall 'numpyro'. No files were found to uninstall.
      Running setup.py develop for numpyro
    Successfully installed numpyro

# Distributions

```
import numpyro
import numpyro.distributions as dist
from numpyro.diagnostics import hpdi
from numpyro.distributions.transforms import AffineTransform
from numpyro.infer import MCMC, NUTS, Predictive

rng_key = random.PRNGKey(0)
rng_key, rng_key_ = random.split(rng_key)
```

## 1d Gaussian

```
# 2 independent 1d gaussians (ie 1 diagonal Gaussian)
mu = 1.5
sigma = 2
d = dist.Normal(mu, sigma)
dir(d)
```

    ['__call__',
     '__class__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__module__',
     '__ne__',
     '__new__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '__weakref__',
     '_batch_shape',
     '_event_shape',
     '_validate_args',
     '_validate_sample',
     'arg_constraints',
     'batch_shape',
     'event_shape',
     'icdf',
     'loc',
     'log_prob',
     'mean',
     'reparametrized_params',
     'sample',
     'sample_with_intermediates',
     'scale',
     'set_default_validate_args',
     'support',
     'to_event',
     'transform_with_intermediates',
     'variance']

```
#rng_key, rng_key_ = random.split(rng_key)
nsamples = 1000
ys = d.sample(rng_key_, (nsamples,))
print(ys.shape)
mu_hat = np.mean(ys,0)
print(mu_hat)
sigma_hat = np.std(ys, 0)
print(sigma_hat)
```

    (1000,)
    1.4788736
    2.0460527

## Multivariate Gaussian

```
mu = np.array([-1, 1])
sigma = np.array([1, 2])
Sigma = np.diag(sigma)
d2 = dist.MultivariateNormal(mu, Sigma)
```

```
#rng_key, rng_key_ = random.split(rng_key)
nsamples = 1000
ys = d2.sample(rng_key_, (nsamples,))
print(ys.shape)
mu_hat = np.mean(ys,0)
print(mu_hat)
Sigma_hat = onp.cov(ys, rowvar=False) #jax.np.cov not implemented
print(Sigma_hat)
```

    (1000, 2)
    [-0.9644672   0.99415004]
    [[0.93275181 0.0756547 ]
     [0.0756547  1.91598212]]

## Shape semantics

Numpyro, [Pyro](https://pyro.ai/examples/tensor_shapes.html) and [TFP](https://www.tensorflow.org/probability/examples/Understanding_TensorFlow_Distributions_Shapes) all distinguish between 'event shape' and 'batch shape'.
For a D-dimensional Gaussian, the event shape is (D,), and the batch shape
will be (), meaning we have a single instance of this distribution.
If the covariance is diagonal, we can view this as D independent
1d Gaussians, stored along the batch dimension; this will have event shape () but batch shape (2,).

When we sample from a distribution, we also specify the sample_shape.
Suppose we draw N samples  from a single D-dim diagonal Gaussian,
and N samples from D 1d Gaussians. These samples will have the same shape.
However, the semantics of logprob differs.
We illustrate this below.

```
d2 = dist.MultivariateNormal(mu, Sigma)
print(d2.event_shape)
print(d2.batch_shape)
nsamples = 1000
ys2 = d2.sample(rng_key_, (nsamples,))
print(ys2.shape)

# 2 independent 1d gaussians (same as one 2d diagonal Gaussian)
d3 = dist.Normal(mu, np.diag(Sigma))
print(d3.event_shape)
print(d3.batch_shape)
ys3 = d3.sample(rng_key_, (nsamples,))
print(ys3.shape)

print(np.allclose(ys2, ys3))
```

    (2,)
    ()
    (1000, 2)
    ()
    (2,)
    (1000, 2)
    True

```
y = ys2[0,:] # 2 numbers
print(d2.log_prob(y)) # log prob of a single 2d distribution on 2d input
print(d3.log_prob(y)) # log prob of two 1d distributions on 2d input

```

    -2.1086864
    [-1.1897303 -0.9189563]

We can turn a set of independent distributions into a single product
distribution using the [Independent class](http://num.pyro.ai/en/stable/distributions.html#independent)

```
d4 = dist.Independent(d3, 1) # treat the first batch dimension as an event dimensions
print(d4.event_shape)
print(d4.batch_shape)
print(d4.log_prob(y))
```

    (2,)
    ()
    -2.1086864

# Posterior inference with MCMC

## Example: 1d Gaussian with unknown mean.

We use the simple example from the [Pyro intro](https://pyro.ai/examples/intro_part_ii.html#A-Simple-Example). The goal is to infer the weight $\theta$ of an object, given noisy measurements $y$. We assume the following model:
$$
\begin{align}
\theta &\sim N(\mu=8.5, \tau^2=1.0)\\
y \sim &N(\theta, \sigma^2=0.75^2)
\end{align}
$$

Where $\mu=8.5$ is the initial guess.

By Bayes rule for Gaussians, we know that the exact posterior,
given a single observation $y=9.5$, is given by

$$
\begin{align}
\theta|y &\sim N(m, s^s) \\
m &=\frac{\sigma^2 \mu + \tau^2 y}{\sigma^2 + \tau^2}
  = \frac{0.75^2 \times 8.5 + 1 \times 9.5}{0.75^2 + 1^2}
  = 9.14 \\
s^2 &= \frac{\sigma^2 \tau^2}{\sigma^2  + \tau^2}
= \frac{0.75^2 \times 1^2}{0.75^2 + 1^2}= 0.6^2
\end{align}
$$

```
mu = 8.5; tau = 1.0; sigma = 0.75; y = 9.5
m = (sigma**2 * mu + tau**2 * y)/(sigma**2 + tau**2)
s2 = (sigma**2 * tau**2)/(sigma**2 + tau**2)
s = np.sqrt(s2)
print(m)
print(s)
```

    9.14
    0.6

```

def model(guess, measurement=None):
    weight = numpyro.sample("weight", dist.Normal(guess, tau))
    return numpyro.sample("measurement", dist.Normal(weight, sigma), obs=measurement)

```

    9.14

```
nuts_kernel = NUTS(model)
mcmc = MCMC(nuts_kernel, num_warmup=100, num_samples=1000)
guess = mu
measurement = y
mcmc.run(rng_key_, guess, measurement=measurement)

mcmc.print_summary()
samples  = mcmc.get_samples()

```

    sample: 100%|██████████| 1100/1100 [00:04<00:00, 229.85it/s, 1 steps of size 8.84e-01. acc. prob=0.96]

                    mean       std    median      5.0%     95.0%     n_eff     r_hat
        weight      9.09      0.62      9.11      8.05     10.03    325.40      1.01

    Number of divergences: 0

```

```