###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth. Partly based on David Ketcheson's pendulum lesson, also under CC-BY.

# Phugoid Oscillation

Welcome back! This is the second Jupyter Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/info)

In the [first notebook](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_01_Phugoid_Theory.ipynb), _"Phugoid Motion"_, we described the physics of an aircraft's oscillatory trajectory, seen as an exchange of kinetic and potential energy. This analysis goes back to Frederick Lanchester, who published his book _"Aerodonetics"_ on aircraft stability in 1909. We concluded that first exposure to our problem of interest by plotting the flight paths predicted by Lanchester's analysis, known as _phugoids_.

Here, we will look at the situation when an aircraft is initially moving on the straight-line phugoid (obtained with the parameters $C=2/3$, $\cos\theta=1$, and $z=z_t$ in the previous analysis), and experiences a small upset, a wind gust that slightly perturbs its path. It will then enter into a gentle oscillation around the previous straight-line path: a _phugoid oscillation_.

If the aircraft experiences an upward acceleration of $-d^2z/dt^2$, and we assume that the perturbation is small, then $\cos\theta=1$ is a good approximation and Newton's second law in the vertical direction is:

$$
\begin{equation}
L - W = - \frac{W}{g}\frac{d^2 z}{dt^2}
\end{equation}
$$

In the previous notebook, we saw that the following relation holds for the ratio of lift to weight, in terms of the trim velocity $v_t$:

$$
\begin{equation}
\frac{L}{W}=\frac{v^2}{v_t^2}
\end{equation}
$$

This will be useful: we can divide Equation (1) by the weight and use Equation (2) to replace $L/W$. Another useful relation from the previous notebook expressed the conservation of energy (per unit mass) as $v^2 = 2 gz$. With this, Equation (1) is rearranged as:

$$
\begin{equation}
\frac{d^2z}{dt^2} + \frac{gz}{z_t} = g
\end{equation}
$$

Look at Equation (3) for a moment. Does it ring a bell? Do you recognize it?

If you remember from your physics courses the equation for _simple harmonic motion_, you should see the similarity!

Take the case of a simple spring. Hooke's law is $F=-kx$, where $F$ is a restoring force, $x$ the displacement from a position of equilibrium and $k$ the spring constant. This results in the following ordinary differential equation for the displacement:

$$
\begin{equation}
 \frac{d^2 x}{dt^2}= -\frac{k}{m}x
\end{equation}
$$

which has the solution $x(t) = A \cos(\omega t- \phi)$, representing simple harmonic motion with an angular frequency $\omega=\sqrt{k/m}=2\pi f$ and phase angle $\phi$.

Now look back at Equation (3): it has nearly the same form and it represents simple harmonic motion with angular frequency $\omega=\sqrt{g/z_t}$ around mean height $z_t$.

Think about this for a moment ... we can immediately say what the period of the oscillation is: exactly $2 \pi \sqrt{z_t/g}$ — or, in terms of the trim velocity, $\pi \sqrt{2} v_t/g$.

_This is a remarkable result!_ Think about it: we know nothing about the aircraft, or the flight altitude, yet we can obtain the period of the phugoid oscillation simply as a function of the trim velocity. For example, if trim velocity is 200 knots, we get a phugoid period of about 47 seconds—over that time, you really would not notice anything if you were flying in that aircraft.

Next, we want to be able to compute the trajectory of the aircraft for a given initial perturbation. We will do this by numerically integrating the equation of motion.

## Prepare to integrate

We want to integrate the differential equation and plot the trajectory of the aircraft. Are you ready?

The equation for the phugoid oscillation is a second-order, ordinary differential equation (ODE). Let's represent the time derivative with a prime, and write it like this:

$$
\begin{equation}
z(t)'' + \frac{g \,z(t)}{z_t}=g
\end{equation}
$$

There's a convenient trick when we work with ODEs: we can turn this 2nd-order equation into a system of two 1st-order equations. Like this:

$$
\begin{eqnarray}
z'(t) &=& b(t)\\
b'(t) &=& g\left(1-\frac{z(t)}{z_t}\right)
\end{eqnarray}
$$

Are you following? Make sure you are following the derivations, even if it means writing the equations down in your own notes! (Yes, the old-fashioned paper way.)

Another way to look at a system of two 1st-order ODEs is by using vectors. You can make a vector with your two independent variables,

$$
\begin{equation}
\vec{u}  = \begin{pmatrix} z \\ b \end{pmatrix}
\end{equation}
$$

and write the differential system as a single vector equation:

$$
\begin{equation}
\vec{u}'(t)  = \begin{pmatrix} b\\ g-g\frac{z(t)}{z_t} \end{pmatrix}
\end{equation}
$$

If you call the right-hand-side $\vec{f}(\vec{u})$, then the equation is very short: $\vec{u}'(t) = \vec{f}(\vec{u})$—but let's drop those arrows to denote vectors from now on, as they are a bit cumbersome: just remember that $u$ and $f$ are vectors in the phugoid equation of motion.

Next, we'll prepare to solve this problem numerically.

## Initial value problems

Let's step back for a moment. Suppose we have a first-order ODE $u'=f(u)$. You know that if we were to integrate this, there would be an arbitrary constant of integration. To find its value, we do need to know one point on the curve $(t, u)$. When the derivative in the ODE is with respect to time, we call that point the _initial value_ and write something like this:

$$
u(t=0)=u_0
$$

In the case of a second-order ODE, we already saw how to write it as a system of first-order ODEs, and we would need an initial value for each equation: two conditions are needed to determine our constants of integration. The same applies for higher-order ODEs: if it is of order $n$, we can write it as $n$ first-order equations, and we need $n$ known values. If we have that data, we call the problem an _initial value problem_.

Remember the definition of a derivative? The derivative represents the slope of the tangent at a point of the curve $u=u(t)$, and the definition of the derivative $u'$ for a function is:

$$
u'(t) = \lim_{\Delta t\rightarrow 0} \frac{u(t+\Delta t)-u(t)}{\Delta t}
$$

If the step $\Delta t$ is already very small, we can _approximate_ the derivative by dropping the limit. We can write:

$$
\begin{equation}
u(t+\Delta t) \approx u(t) + u'(t) \Delta t
\end{equation}
$$

With this equation, and because we know $u'(t)=f(u)$, if we have an initial value, we can step by $\Delta t$ and find the value of $u(t+\Delta t)$, then we can take this value, and find $u(t+2\Delta t)$, and so on: we say that we _step in time_, numerically finding the solution $u(t)$ for a range of values: $t_1, t_2, t_3 \cdots$, each separated by $\Delta t$. The numerical solution of the ODE is simply the table of values $t_i, u_i$ that results from this process.

## Discretization

In order to execute the process described above and find the numerical solution of the ODE, we start by choosing the values $t_1,t_2,t_3 \cdots t_n$—we call these values our *grid* in time. The first point of the grid is given by our _initial value_, and the small difference between two consecutive times is called the _time step_, denoted by $\Delta t$.  The solution value at time $t_n$ is denoted by $u_n$.

Let's build a time grid for our problem. We first choose a final time $T$ and the time step $\Delta t$. In code, we'll use readily identifiable variable names: `T` and `dt`, respectively. With those values set, we can calculate the number of time steps that will be needed to reach the final time; we call that variable `N`.

Let's write some code. The first thing we do in Python is load our favorite libraries: NumPy for array operations, and the Pyplot module in Matplotlib, to later on be able to plot the numerical solution. The line `%matplotlib inline` tells Jupyter Notebook to show the plots inline.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

Now, initialize `T` and `dt`, calculate `N` and build a NumPy array with all the values of time that make up the grid.

```python
# Create the time grid.
T = 100.0  # length of the time-interval
dt = 0.02  # time-step size
N = int(T / dt) + 1  # number of time steps
t = numpy.linspace(0.0, T, num=N)  # time grid
```

We have our grid! Now it's time to apply the numerical time stepping represented by Equation (10).

##### Challenge!

* Write the code above using the NumPy function `arange()` instead of `linspace()`. If you need to, read the documentation for these functions.

##### Pro tip:

Enter a question mark followed by any function, e.g., `?numpy.linspace`, into a code cell and execute it, to get a help pane on the notebook.

## Euler's method

The approximate solution at time $t_n$ is $u_n$, and the numerical solution of the differential equation consists of computing a sequence of approximate solutions by the following formula, based on Equation (10):

$$
\begin{equation}
u_{n+1} = u_n + \Delta t \,f(u_n)
\end{equation}
$$

This formula is called **Euler's method**.

For the equations of the phugoid oscillation, Euler's method gives the following algorithm that we need to implement in code:

$$
\begin{align}
z_{n+1} & = z_n + \Delta t \, b_n \\
b_{n+1} & = b_n + \Delta t \left(g - \frac{g}{z_t} \, z_n \right)
\end{align}
$$

### And solve!

To apply the numerical solution method, we need to set things up in code: define the parameter values needed in the model, initialize a NumPy array to hold the discrete solution values, and initialize another array for the elevation values.

```python
# Set the initial conditions.
z0 = 100.0  # altitude
b0 = 10.0  # upward velocity resulting from gust
zt = 100.0  # trim altitude
g = 9.81  # acceleration due to gravity

# Set the initial value of the numerical solution.
u = numpy.array([z0, b0])

# Create an array to store the elevation value at each time step.
z = numpy.zeros(N)
z[0] = z0
```

You should pay attention to a couple of things: (1) See how there is a dot after the numbers used to define our parameters? We just want to be explicit (as a good habit) that these variables are real numbers, called "floats." (2) We both _created_ and _initialized_ with zeros everywhere the solution vector `z`. Look up the documentation for the handy NumPy function `zeros()`, if you need to. (3) In the last line above, we assign the _initial value_ to the first element of the solution vector: `z[0]`.

Now we can step in time using Euler's method. Notice how we are time stepping the two independent variables at once in the time iterations.

```python
# Temporal integration using Euler's method.
for n in range(1, N):
    rhs = numpy.array([u[1], g * (1 - u[0] / zt)])
    u = u + dt * rhs
    z[n] = u[0]
```

Make sure you understand what this code is doing. This is a basic pattern in numerical methods: iterations in a time variable that apply a numerical scheme at each step.

## Plot the solution

If the code is correct, we have stored in the array `z` the position of the glider at each time. Let's use Matplotlib to examine the flight path of the aircraft.

You should explore the [Matplotlib tutorial](https://matplotlib.org/users/pyplot_tutorial.html) (if you need to) and familiarize yourself with the command-style functions that control the size, labels, line style, and so on. Creating good plots is a useful skill: it is about communicating your results effectively.

Here, we set the figure size, the limits of the vertical axis, the format of tick-marks, and axis labels. The final line actually produces the plot, with our chosen line style (continuous black line).

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16

# Plot the solution of the elevation.
pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title('Elevation of the phugoid over the time')  # set the title
pyplot.xlabel('Time [s]')  # set the x-axis label
pyplot.ylabel('Elevation [m]')  # set the y-axis label
pyplot.xlim(t[0], t[-1])  # set the x-axis limits
pyplot.ylim(40.0, 160.0)  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.plot(t, z, color='C0', linestyle='-', linewidth=2);
```

##### Explore and think

Try changing the value of `v` in the initial conditions.

* What happens when you have a larger gust?
* What about a smaller gust?
* What happens if there isn't a gust (`v = 0`)?

## Exact solution

The equation for phugoid oscillations is a 2nd-order, linear ODE and it has an exact solution of the following form:

$$
\begin{equation}
z(t) = A \sin \left(\sqrt{\frac{g}{z_t}} t \right) + B \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

where $A$ and $B$ are constants that we solve for using initial conditions.

Our numerical solution used the initial conditions:

$$
\begin{eqnarray}
z(0) = z_0 \\
b(0) = b_0
\end{eqnarray}
$$

Applying these to the exact solution and solving for $A$ and $B$, we get:

$$
\begin{equation}
z(t) = b_0 \sqrt{\frac{z_t}{g}} \sin \left(\sqrt{\frac{g}{z_t}} t \right) + (z_0-z_t) \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

We already defined all of these variables for our numerical solution, so we can immediately compute the exact solution.

##### Pro tip:

The expression is a bit long —if you don't feel like scrolling left and right, you can cast the value of the variable between parenthesis and insert line breaks and Python will treat the next line as a continuation of the first.

```python
z_exact = (b0 * (zt / g)**0.5 * numpy.sin((g / zt)**0.5 * t) +
           (z0 - zt) * numpy.cos((g / zt)**0.5 * t) + zt)
```

## Plot the exact solution

Now we can plot our exact solution!  Even better, we can plot _both_ the numerical solution *and* the exact solution to see how well Euler's method approximated the phugoid oscillations.

To add another curve to a plot, simply type a second `pyplot.plot()` statement.  We also add a legend using `pyplot.legend()`.

```python
# Plot the numerical solution and the exact solution.
pyplot.figure(figsize=(9.0, 4.0))  # set the size of the figure
pyplot.title('Elevation of the phugoid over the time')  # set the title
pyplot.xlabel('Time [s]')  # set the x-axis label
pyplot.ylabel('Elevation [m]')  # set the y-axis label
pyplot.xlim(t[0], t[-1])  # set the x-axis limits
pyplot.ylim(40.0, 160.0)  # set the y-axis limits
pyplot.grid()  # set a background grid to improve readability
pyplot.plot(t, z, label='Numerical',
            color='C0', linestyle='-', linewidth=2)
pyplot.plot(t, z_exact, label='Analytical',
            color='C1', linestyle='-', linewidth=2)
pyplot.legend();  # set the legend
```

That looks like pretty good agreement, but what's happening towards the end? We'll come back to this. For now, re-run the previous steps with a different time step, say $dt=0.01$ and pay attention to the difference.

Euler's method, like all numerical methods, introduces some errors.  If the method is *convergent*, the approximation will get closer and closer to the exact solution as we reduce the size of the step, $\Delta t$. The error in the numerical method should tend to zero, in fact, when $\Delta t\rightarrow 0$—when this happens, we call the method _consistent_. We'll define these terms more carefully in the theory components of this course.

## Convergence

To compare the two solutions, we need to use a **norm** of the difference, like the $L_1$ norm, for example.

$$
E = \Delta t \sum_{n=0}^N \left|z(t_n) - z_n\right|
$$

The $L_1$ norm is the sum of the individual differences between the exact and the numerical solutions, at each mesh point. In other words, $E$ is the discrete representation of the integral over the interval $T$ of the (absolute) difference between the computed $z$ and $z_{\rm exact}$:

$$
E = \int \vert z-z_\rm{exact}\vert dt
$$

We check for convergence by calculating the numerical solution using progressively smaller values of `dt`. We already have most of the code that we need.  We just need to add an extra loop and an array of different $\Delta t$ values to iterate through.

##### Warning

The cell below can take a little while to finish (the last $\Delta t$ value alone requires 1 million iterations!).  If the cell is still running, the input label will say `In [*]`.  When it finishes, the `*` will be replaced by a number.

```python
# Set the list of time-step sizes.
dt_values = [0.1, 0.05, 0.01, 0.005, 0.001, 0.0001]

# Create an empty list that will contain the solution of each grid.
z_values = []

for dt in dt_values:
    N = int(T / dt) + 1  # number of time-steps
    t = numpy.linspace(0.0, T, num=N)  # time grid
    # Set the initial conditions.
    u = numpy.array([z0, b0])
    z = numpy.empty_like(t)
    z[0] = z0
    # Temporal integration using Euler's method.
    for n in range(1, N):
        rhs = numpy.array([u[1], g * (1 - u[0] / zt)])
        u = u + dt * rhs
        z[n] = u[0]  # store the elevation at time-step n+1
    z_values.append(z)  # store the elevation over the time
```

### Calculate the error

We now have numerical solutions for each $\Delta t$ in the array `z_values`.  To calculate the error corresponding to each $\Delta t$, we can write a function!

```python
def l1_error(z, z_exact, dt):
    """
    Computes and returns the error
    (between the numerical and exact solutions)
    in the L1 norm.

    Parameters
    ----------
    z : numpy.ndarray
        The numerical solution as an array of floats.
    z_exact : numpy.ndarray
        The analytical solution as an array of floats.
    dt : float
        The time-step size.

    Returns
    -------
    error: float
        L1-norm of the error with respect to the exact solution.
    """
    error = dt * numpy.sum(numpy.abs(z - z_exact))
    return error
```

**Note**: in the first line of the function, we perform an 'array operation':

`z - z_exact`

We are *not* subtracting one value from another.  Instead, we are taking the difference between elements at each corresponding index in both arrays.  Here is a quick example:

```python
a = numpy.array([1, 2, 3])
b = numpy.array([4, 4, 4])

b - a
```

    array([3, 2, 1])

Now, we iterate through each $\Delta t$ value and calculate the corresponding error.
In the following code cell, we use the built-in function [`zip`](https://docs.python.org/3/library/functions.html#zip) to iterate over the two lists `z_values` and `dt_values`.

```python
# Create an empty list to store the errors on each time grid.
error_values = []

for z, dt in zip(z_values, dt_values):
    N = int(T / dt) + 1  # number of time-steps
    t = numpy.linspace(0.0, T, num=N)  # time grid
    # Compute the exact solution.
    z_exact = (b0 * (zt / g)**0.5 * numpy.sin((g / zt)**0.5 * t) +
               (z0 - zt) * numpy.cos((g / zt)**0.5 * t) + zt)
    # Calculate the L1-norm of the error for the present time grid.
    error_values.append(l1_error(z, z_exact, dt))
```

Remember, *if* the method is convergent then the error should get smaller as  $\Delta t$ gets smaller.  To visualize this, let's plot $\Delta t$ vs. error.  If you use `pyplot.plot` you won't get a very useful result.  Instead, use `pyplot.loglog` to create the same plot with a log-log scale.  This is what we do almost always to assess the errors of a numerical scheme graphically.

```python
# Plot the error versus the time-step size.
pyplot.figure(figsize=(6.0, 6.0))
pyplot.title('L1-norm error vs. time-step size')  # set the title
pyplot.xlabel('$\Delta t$')  # set the x-axis label
pyplot.ylabel('Error')  # set the y-axis label
pyplot.grid()
pyplot.loglog(dt_values, error_values,
              color='C0', linestyle='--', marker='o')  # log-log plot
pyplot.axis('equal');  # make axes scale equally
```

This is the kind of result we like to see!  As $\Delta t$ shrinks (towards the left), the error gets smaller and smaller, like it should.

##### Challenge!

We calculated the solution for several different time-step sizes using two nested `for` loops.
That worked, but whenever possible, we like to re-use code (and not just copy and paste it!).

Create a function that implements Euler's method and re-write the code cell that computes the solution for different time-step sizes using your function.

---

###### The cell below loads the style of this notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


## Problem Statement

Use PINNs to solve the parameterized fluid flow for the given geometry and flow parameters

Simulate the same chip flow, but now with variable chip height and width in a single parameteric run.

## Challenge

The main challenge in this problem is to correctly formulate the problem using PINNs. In order to achieve that, you will have to complete the following parts successfully:
1. Define the correct *parameterized* geometry for the problem
2. Set-up the correct boundary conditions and equations
3. Create the neural network and solve the *parameteric* problem

A successful completion of the problem should result a network that can be inferenced at any geometry parameter combination of our choice. For example, the model should be able to create plots for a geometry configuraions where the chip height is of 0.4 and width is 1.4. An example is shown below. Remember, once the model is trained, you can infer any geometry combination without the need to sovle the problem again. Quite cool isn't it!?

In this template, we will have give you a skeleton code which you will fill in to define and solve the parametreric problem. If you have completed the challenge 1 successfully, moving to this part should be fairly easy as there are only a few minor edits. Let us start with importing the required packages.

**Note: You need to edit the `chip_2d_parameterized_template.py` script that is placed in the ../source_code/chip_2d/ directory.**

From the top menu, click on File, and Open `chip_2d_parameterized_template.py` from the current directory at `../source_code/chip_2d` directory. Remember to SAVE your code after changes, before running below cells.

```python
from sympy import Symbol
import numpy as np
import tensorflow as tf
from simnet.solver import Solver
from simnet.dataset import TrainDomain, ValidationDomain, InferenceDomain
from simnet.data import Validation, Inference
from simnet.sympy_utils.geometry_2d import Rectangle, Line, Channel2D
from simnet.sympy_utils.functions import parabola
from simnet.csv_utils.csv_rw import csv_to_dict
from simnet.PDES.navier_stokes import IntegralContinuity, NavierStokes
from simnet.controller import SimNetController
from simnet.architecture import FourierNetArch
from simnet.learning_rate import ExponentialDecayLRWithWarmup
```

Now we have defined the simulation parameters and the symbolic variables to parameterize the geometry below. As seen in the earlier tutorials, we have define a range in which data points will be sampled during the trainig. Feel free to try different ranges for the variables.

```python
# simulation params
channel_length = (-2.5, 2.5)
channel_width = (-0.5, 0.5)
chip_pos = -1.0
#chip_height = 0.6         # Not fixed anymore
#chip_width = 1.0          # Not fixed anymore
inlet_vel = 1.5

# paramteric variables
chip_height = Symbol('chip_height')
chip_width = Symbol('chip_width')

chip_height_range = (0.4, 0.8)
chip_width_range  = (0.6, 1.4)

param_ranges = {chip_height: chip_height_range,
                chip_width: chip_width_range}
```

```python
#TODO: Replace all the placeholders with appropriate geometry constructions
# define geometry here
# you may use the geometry generated in the previous challenge problem as a reference

channel = placeholder
# define inlet and outlet
inlet = placeholder
outlet = placeholder
# define the chip
rec = placeholder
# create a geometry for higher sampling of point cloud near the fin
flow_rec = placeholder

# fluid area
geo = placeholder
geo_hr = placeholder
geo_lr = placeholder

x_pos = Symbol('x_pos')
integral_line = placeholder
x_pos_range = {x_pos: lambda batch_size: np.full((batch_size, 1), np.random.uniform(channel_length[0], channel_length[1]))}
```

Now complete the `Chip2DTrain` class below. Remember that this time, you will have to add the dictionary of the parameter keys and their ranges as an input to the `param_ranges` parameter. An example of the inlet boundary condition is already shown. *Note: For the integral continuity planes, you have two dictionaries. One for the randomly sampled* `x_pos` *and the other for the parameteric variables itself. Make sure to include both of these as input to the* `param_ranges`

```python
#TODO: Replace all the placeholders with appropriate values

# define sympy variables to parametrize domain curves
x, y = Symbol('x'), Symbol('y')

class Chip2DTrain(TrainDomain):
  def __init__(self, **config):
    super(Chip2DTrain, self).__init__()

    # inlet
    inlet_parabola = parabola(y, channel_width[0], channel_width[1], inlet_vel)
    inlet_bc = inlet.boundary_bc(outvar_sympy={'u': inlet_parabola, 'v': 0},
                                 batch_size_per_area=64,
                                 param_ranges=param_ranges)
    self.add(inlet_bc, name="Inlet")

    # outlet
    outlet_bc = outlet.boundary_bc(outvar_sympy={placeholder},
                                   batch_size_per_area=placeholder,
                                   param_ranges=placeholder)
    self.add(outlet_bc, name="Outlet")

    # noslip
    noslip = geo.boundary_bc(outvar_sympy={placeholder},
                             batch_size_per_area=placeholder,
                             param_ranges=placeholder)
    self.add(noslip, name="ChipNS")

    # interior lr
    interior_lr = geo_lr.interior_bc(outvar_sympy={placeholder},
                                     bounds={placeholder},
                                     lambda_sympy={placeholder},
                                     batch_size_per_area=placeholder,
                                     param_ranges=placeholder)
    self.add(interior_lr, name="InteriorLR")

    # interior hr
    interior_hr = geo_hr.interior_bc(outvar_sympy={placeholder},
                                     bounds={placeholder},
                                     lambda_sympy={placeholder},
                                     batch_size_per_area=placeholder,
                                     param_ranges=placeholder)
    self.add(interior_hr, name="InteriorHR")

    # integral continuity
    for i in range(4):
      IC = integral_line.boundary_bc(outvar_sympy={placeholder},
                                     batch_size_per_area=placeholder,
                                     lambda_sympy={placeholder},
                                     criteria=placeholder,
                                     param_ranges={placeholder},
                                     fixed_var=placeholder)
      self.add(IC, name="IntegralContinuity_"+str(i))
```

Now, add validation data to the problem. Similar to the tutorial examples, validate this parameteric solution for a single configuration. Remember that the network for this problem has four inputs (x, y, chip_height, and chip_width). So your validation data also must have those corresponding inputs for each point in the CSV file.

```python
# validation data
mapping = {'Points:0': 'x', 'Points:1': 'y',
           'U:0': 'u', 'U:1': 'v', 'p': 'p'}
openfoam_var = csv_to_dict('openfoam/2D_chip_fluid0.csv', mapping)
openfoam_var['x'] -= 2.5 # normalize pos
openfoam_var['y'] -= 0.5

#TODO: Add the arrays for 'chip_height' and 'chip_width'

openfoam_invar_numpy = {key: value for key, value in openfoam_var.items() if key in ['x', 'y', 'chip_height', 'chip_width']}
openfoam_outvar_numpy = {key: value for key, value in openfoam_var.items() if key in ['u', 'v', 'p']}

class Chip2DVal(ValidationDomain):
  def __init__(self, **config):
    super(Chip2DVal, self).__init__()
    val = Validation.from_numpy(openfoam_invar_numpy, openfoam_outvar_numpy)
    self.add(val, name='Val')
```

As discussed in the begining, we would like to visualize multiple designs without traing again and again. Let's define a `InferenceDomain` to do this. By changing the values in the this domain alone, and executing the script with `--run_mode=eval` commandline argument will help you to visualize multiple designs using the already trained network.

```python
class Chip2DInf(InferenceDomain):
  def __init__(self, **config):
    super(Chip2DInf, self).__init__()
    inf = Inference(geo.sample_interior(2048, bounds={x: channel_length, y: channel_width},
                                        param_ranges={chip_height: 0.4, chip_width: 1.4}),
                    ['u', 'v', 'p'])
    self.add(inf, name='Inference')
```

Now finish the problem by defining the `ChipSolver` to solve our parametric problem. We have used the same fourier networks we used in the previous template. The important parameters of the neural network are defined for you. Feel free to tweak them and observe its behavior on the results and speed of convergence.

```python
#TODO: Replace all the placeholders with appropriate values
class ChipSolver(Solver):
  train_domain = placeholder
  val_domain = placeholder
  arch = FourierNetArch
  lr = ExponentialDecayLRWithWarmup
  inference_domain = placeholder

  def __init__(self, **config):
    super(ChipSolver, self).__init__(**config)

    self.frequencies = ('axis,diagonal', [i/5. for i in range(25)])

    self.equations = (placeholder)
    flow_net = self.arch.make_node(name='flow_net',
                                   inputs=[placeholder],
                                   outputs=[placeholder])
    self.nets = [flow_net]

  @classmethod
  def update_defaults(cls, defaults):
    defaults.update({
        'network_dir': './network_checkpoint_chip_2d_parameterized',
        'rec_results': True,
        'rec_results_freq': 5000,
        'max_steps': 20000,
        'decay_steps': 400,
        'warmup_type': 'gradual',
        'warmup_steps': 2000,
        'xla': True
        })
if __name__ == '__main__':
  ctr = SimNetController(ChipSolver)
  ctr.run()
```

## Bonus: Multi-GPU speedup

Nvida SimNet library is optimized to achieve fast and scalable performance with features like accelerated linear algebra (XLA), automatic mixed precision (AMP) support and multi-GPU/multi-node implementation. You would have already observed that we had the `'xla'` option set to `True` for most of our examples. Here, we will discuss about the multi-GPU capabilies. For a detailed discussion on the performance upgrades available in Simnet, please refer to the *SimNet User Guide Chapter 16: Case Study: Performance Upgrades and Parallel Processing using Multi-GPU Configurations*

To boost performance and to run larger problems, SimNet supports multi-GPU and multi-node scaling using Horovod. This allows for multiple processes, each targeting a single GPU, to perform independent forward and backward passes and aggregate the gradients collectively before updating the model weights. The figure below shows the scaling performance
of SimNet on a annular ring test problem up to 128 V100 GPUs on 16 nodes. The scaling efficiency from 1 to 32 GPUs is more than 95%.

This data parallel fashion of multi-GPU training keeps the number of points sampled per GPU constant while increasing
the total effective batch size. We can use this to our advantage to increase the number of points sampled by increasing
the number of GPUs allowing us to handle much larger problems.

We can run the same scripts using multi-GPU to achieve larger batch sizes. And doing so, the time per iteration remains fairly constant and the benefits are mostly in terms of large problem solution (refer the figure above). But what if we want to also achieve faster time to convergence? If you noticed carefully, we have used a different learning rate schedule in this problem. This basically allows us to decrease the total time to convergence when training using multiple GPUs. This is done by scaling the learning rate linearly with the number of GPUs. As described in this [paper](https://arxiv.org/abs/1706.02677), simply increasing the learning rate can cause the model to diverge at large batch
sizes. This can be fixed by using an initial learning rate warm-up. In this code we have made use of this feature of SimNet (with gradual warm-up). Below figure shows the learning rate schedule and the loss function evolution as the number of GPUs is increased from 1 to 16 for the NVSwitch heatsink case. For the multi-GPU cases, the learning rate is gradually increased from the baseline case and this allows the model to train without diverging early on and allows the model to converge faster as a result of the increased global batch size coupled with the increased learning rate.

For more details, refer to *SimNet User Guide Chapter 1 and Chapter 16*.

To run your code using multiple GPUs, simply execute the python script using the horovod command as follows

`horovodrun -np #GPUs python_script.py `

# Licensing
This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)


## Subsurface Data Analytics

### Multidimensional Scaling for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Subsurface Machine Learning: Multidimensional Scaling for Subsurface Data Analytics

Here's a demonstration of mulidimensional scaling for subsurface modeling in Python. This is part of my Subsuface Machine Learning Course at the Cockrell School of Engineering at the University of Texas at Austin.

#### Multidimensional Scaling

A powerful ordination method in inferential statistics / information visualization for exploring / visualizing the similarity (conversely the difference) between individual samples from a high dimensional dataset.

* beyond 2 or 3 features it is difficult to visualize the relationship between samples

* for 2 features we can easily visualize the relationships between samples with a scatter plot

* for 3 features we can either visualize in 3D or include color or matrix scatter plots

Multidimensional scaling projects the $m$ dimensional data to $p$ dimensions such that $p << m$.

* ideally we are able to project to $p=2$ to easily explore the relationships between the samples

While principal component analysis (PCA) operates with the covariance matrix, multidimensional scaling operates with the distance / dissimilarity matrix.

* you don't need to know the actual feature values, just the distance or dissimilarity between the samples

* as with any distance in feature space, we consider feature standardization and weighting

* we may also work with a variety of dissimilarity measures

#### Classical Multidimensional Scaling

Based on Euclidian distance between samples.

The Steps:

1. calculate the square distance matrix, $D^{(2)} = \left[ \delta_{i,j}^2 \right]$.

2. apply double centering $B = - \frac{1}{2} J D^{(2)} J$

3. solve for the Eigen Values, $\lambda_1,\ldots,\lambda_p$

4. solve for the projected coordinates, $x^{'}_1, \dots ,x^{'}_p$

\begin{equation}
X^{'} = E_m \land^{\frac{1}{2}}_m
\end{equation}

where $E_m$ is the matrix of Eigen vectors and $\land_m$ is the diagonal matrix of Eigen values.

General comments about classical multidimensional scaling:

* nonlinear dimensionality reduction

* no distribution assumption

* the transform may not be unique, and may be arbitrarily be translated, rotated and transformed (these do not change the pairwise distances $||x_i - x_j||$.

#### Metric Multidimensional Scaling

A generalization of classical multidimensional scaling with a variety of metrics and a loss function optimization.

* formulated as an optimization problem to minimize the squate difference between the original and projected pairwise distances

\begin{equation}
min_{x_1,\ldots,x_m} \sum_{i<j} \left( ||x_i - x_j|| - \delta_{i,j} \right)^2
\end{equation}

where $||x_i - x_j||$ are the pairwise distances in the projected space ($p$ dimensional) and $\delta_{i,j}$ are the pairwise distances in the original feature space.

General comments about metric multidimensional scaling:

* dissimilarity measure must be meaningful

* dimensionality reduction is performed such that the error in the sample pairwise distance is minimized

* there is a variant known as Nonmetric Multidimensional Scaling for ordinal features (categorical with ordering).

#### Checking Multidimensional Scaling Results

The multidimensional scaling approach minimizes the square difference of the pairwise distances between all of the data samples and eachother between the projected, lower dimensional, and original feature space.

* **stress** is defined as:

\begin{equation}
Stress_P(x_1,\ldots,x_n) = \left( \sum_{i \ne j = 1,\ldots,n} \left( ||x_i - x_j|| - \delta_{i,j} \right)^2 \right)^{\frac{1}{2}}
\end{equation}

* it is also useful to visualize the scatterplot of projected vs. original pairwise distances

#### Comparison with Principal Component Analysis

Principal component analysis takes the covariance matrix ($m \times m$) between all the features and finds the orthogonal rotation such that the **variance is maximized** over the ordered principle components.

Multidimensional scaling takes the matrix of the pairwise distances ($n \times n$) between all the samples in feature space and finds the nonlinear projects such that the **error in the pairwise distances is minimized**.

#### Random Projection

An alternative multidimensional scaling that relies on an random ($p \times n$) projection matrix, with columns with unit length.

* first row is a random unit vector

* second row is orthogonal to the first row

* continue

This random approach works practically because of the The Johnson-Lindenstrauss lemma:

*In mathematics, the Johnson-Lindenstrauss lemma is a result concerning low-distortion embeddings of points from high-dimensional into low-dimensional Euclidean space. The lemma states that a small set of points in a high-dimensional space can be embedded into a space of much lower dimension in such a way that distances between the points are nearly preserved. The map used for the embedding is at least Lipschitz, and can even be taken to be an orthogonal projection.* - Wikipedia article.

#### Objective

In the Stochastic Machine Learning class, I want to provide hands-on experience with solving complicated subsurface modeling problems with data analytics, machine learning. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

There are examples below with GeostatsPy functions. You can go here to see a list of the available functions, https://git.io/fh4eX, and for other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # for randon numbers
import seaborn as sns                     # for matrix scatter plots
from scipy import linalg                  # for linear regression

from sklearn.manifold import MDS          # multidimensional scaling
from sklearn.random_projection import GaussianRandomProjection # random projection
from sklearn.random_projection import johnson_lindenstrauss_min_dim
from sklearn.random_projection import SparseRandomProjection
from sklearn.metrics.pairwise import euclidean_distances
```

Here's a couple of functions that I found online for metrics that we will calculate below.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('unconv_MV_v4.csv')      # load our data table
df['TOC'] = np.where(df['TOC']<0.0, 0.0, df['TOC']) # set TOC < 0.0 as 0.0, otherwise leave the same
```

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=13)                             # we could also use this command for a table preview
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

      5
      6
      14.53
      4.81
      2.69
      53.60
      0.94
      1.67
      4017.374438

      6
      7
      13.49
      3.60
      2.93
      63.71
      0.80
      1.85
      2952.812773

      7
      8
      11.58
      3.03
      3.25
      53.00
      0.69
      1.93
      2670.933846

      8
      9
      12.52
      2.72
      2.43
      65.77
      0.95
      1.98
      2474.048178

      9
      10
      13.25
      3.94
      3.71
      66.20
      1.14
      2.65
      2722.893266

      10
      11
      15.04
      4.39
      2.22
      61.11
      1.08
      1.77
      3828.247174

      11
      12
      16.19
      6.30
      2.29
      49.10
      1.53
      1.86
      5095.810104

      12
      13
      16.82
      5.42
      2.80
      66.65
      1.17
      1.98
      4091.637316

This dataset has features from 200 unconventional wells including:

0. well index
1. well average porosity (%)
2. permeability (mD)
3. accoustic impedance (kg/m2s*10^6)
4. brittness ratio (%)
5. total organic carbon (%)
6. vitrinite reflectance (%)
8. normalized initial production 90 day average (MCFPD).

Note, the dataset is synthetic, but has realistic ranges and general multivariate relationships.

Ranking features is really an effort to understand the features and their relationships with eachother.  We will start with basic data visualization and move to more complicated methods such are partial correlation and recursive feature elimination.

#### Summary Statistics

Let's check the summary statistics of our data.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Well
      200.0
      100.500000
      57.879185
      1.000000
      50.750000
      100.500000
      150.250000
      200.000000

      Por
      200.0
      14.991150
      2.971176
      6.550000
      12.912500
      15.070000
      17.402500
      23.550000

      Perm
      200.0
      4.330750
      1.731014
      1.130000
      3.122500
      4.035000
      5.287500
      9.870000

      AI
      200.0
      2.968850
      0.566885
      1.280000
      2.547500
      2.955000
      3.345000
      4.630000

      Brittle
      200.0
      48.161950
      14.129455
      10.940000
      37.755000
      49.510000
      58.262500
      84.330000

      TOC
      200.0
      0.991950
      0.478264
      0.000000
      0.617500
      1.030000
      1.350000
      2.180000

      VR
      200.0
      1.964300
      0.300827
      0.930000
      1.770000
      1.960000
      2.142500
      2.870000

      Prod
      200.0
      3864.407081
      1553.277558
      839.822063
      2686.227611
      3604.303507
      4752.637556
      8590.384044

Summary statistics are a critical first step in data checking.

* this includes the number of valid (non-null) values for each feature (count removes all np.NaN from the totals for each variable).

* we can see the general behavoirs such as central tendency, mean, and dispersion, variance.

* we can identify issue with negative values, extreme values, and values that are outside the range of plausible values for each property.

* We can also establish the feature ranges for plotting.  We could calculate the feature range directly from the data with code like this:

```p
Pormin = np.min(df['Por'].values)          # extract ndarray of data table column
Pormax = np.max(df['Por'].values)          # and calculate min and max
```

but, this would not result in easy to understand color bars and axis scales, let's pick convenient round numbers. We will also declare feature labels for ease of plotting.

```python
pormin = 6.0; pormax = 24.0; porname = 'Porosity (%)'; portitle = 'Porosity' # user specified min and max values, and labels for plotting
permmin = 0.0; permmax = 10; permname = 'Permeability (mD)'; permtitle = 'Permeability'
AImin = 1.0; AImax = 5.0; AIname = 'Acoustic Impedance (kg/m2s*10^6)'; AItitle = 'Acoustic Impedance'
brmin = 10.0; brmax = 85.0; brname = 'Brittleness Ratio (%)'; brtitle = 'Brittleness'
TOCmin = 0.0; TOCmax = 2.2; TOCname = 'Total Organic Carbon (%)'; TOCtitle = 'Total Organic Carbon'
VRmin = 0.9; VRmax = 2.9; VRname = 'Vitrinite Reflectance (%)'; VRtitle = 'Vitrinite Reflectance'
prodmin = 500.0; prodmax = 9000.0; prodname = 'Normalized Initial Production (MCFPD)'; prodtitle = 'Normalized Initial Production'
```

The data looks to be in pretty good shape and for brevity we skip outlier detection. Let's look at the distributions with a martix scatter plot from the Seaborn package.

```python
sns.pairplot(df,vars=['Por','Perm','AI','Brittle','TOC','VR','Prod'],markers='o',plot_kws={'alpha': 0.2})
plt.subplots_adjust(left=0.0, bottom=0.0, right=0.6, top=0.6, wspace=0.3, hspace=0.2)
plt.show()
```

#### Feature Standardization

Since our model works with dissimilarity (analogous to distance in feature space) measures, we need ot standardize the features.

```python
from sklearn.preprocessing import StandardScaler

features = ['Por','AI','TOC']
x = df.loc[:,features].values
mu = np.mean(x, axis=0)
sd = np.std(x, axis=0)
xs = StandardScaler().fit_transform(x)

ns_features = []
for i in range(0,len(features)):
    df['NS_'+features[i]] = xs[:,i]
    ns_features.append('NS_'+features[i])

df.head(n=13)
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod
      NS_Por
      NS_AI
      NS_TOC

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819
      -0.982256
      -0.298603
      0.352257

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063
      -0.881032
      0.444147
      -0.213702

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259
      -0.327677
      1.841224
      -0.213702

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854
      0.903875
      -0.599240
      0.184565

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624
      0.853263
      0.373409
      1.085907

      5
      6
      14.53
      4.81
      2.69
      53.60
      0.94
      1.67
      4017.374438
      -0.155597
      -0.493133
      -0.108895

      6
      7
      13.49
      3.60
      2.93
      63.71
      0.80
      1.85
      2952.812773
      -0.506505
      -0.068704
      -0.402355

      7
      8
      11.58
      3.03
      3.25
      53.00
      0.69
      1.93
      2670.933846
      -1.150962
      0.497200
      -0.632930

      8
      9
      12.52
      2.72
      2.43
      65.77
      0.95
      1.98
      2474.048178
      -0.833795
      -0.952930
      -0.087933

      9
      10
      13.25
      3.94
      3.71
      66.20
      1.14
      2.65
      2722.893266
      -0.587484
      1.310688
      0.310334

      10
      11
      15.04
      4.39
      2.22
      61.11
      1.08
      1.77
      3828.247174
      0.016483
      -1.324305
      0.184565

      11
      12
      16.19
      6.30
      2.29
      49.10
      1.53
      1.86
      5095.810104
      0.404506
      -1.200514
      1.127830

      12
      13
      16.82
      5.42
      2.80
      66.65
      1.17
      1.98
      4091.637316
      0.617075
      -0.298603
      0.373218

#### Data Preparation

Let's make an ordinal feature from the continuous production:

1. low
2. medium
3. high
4. very high

production rates.  This will help us visualize the results as we proceed, we can look at wells with different levels of production projected into a variety of lower dimensional spaces with multidimensional scaling.

```python
bins = [0,2500,5000,7500,10000]                # assign the production bins (these are the fence posts)
labels = ['low', 'med', 'high', 'vhigh']       # assign the labels
category = pd.cut(df['Prod'],bins,labels=labels)     # make the 1D array with the labels for our data
df['tProd'] = category                                # add the new ordinal production feature to our DataFrames
df.head()
dpalette = sns.color_palette("rocket_r",n_colors = 4)
palette = sns.color_palette("rocket")
```

Let's take a look at the matrix scatter plot of our 3 features and the production levels.

```python
plot = sns.pairplot(df[ns_features + ['tProd']],markers='o',hue = 'tProd', palette = dpalette,diag_kws={'edgecolor':'black'},plot_kws=dict(s=50, edgecolor="black", linewidth=0.5))

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.5, wspace=0.3, hspace=0.2)
plt.show()
```

#### Multidimensional Scaling

The multidimensional scaling method follows the sample pattern as other scikit-learn methods, we instantiate, fit and then apply or transform.

Let's run multidimensional scaling on our subset of features ($m = 3$) and project to only 2 features ($p = 2$).

* we set the random_state for repeatability, everyone gets the same result from the iterative solution

* we use 20 random initializations, the best solution is selected to improve likelihood of selection of (or search resulting in) the global optimum and not a local optimum

* we use an increased number of max_iter to improve the convergence

```python
np.random.seed(1867)                               # set the random number seed, so we all get the same answer
n_components = 2                                   # p, reduced dimensionality space
embedding = MDS(n_components=2,n_init = 20,max_iter = 1000,random_state = 73073) # instantiate and set the hyperparameter
MDS_transformed = embedding.fit_transform(df[ns_features])
MDS_transformed.shape
```

    (200, 2)

The output is 2 multidimensional scaling components.  We have projected our 3 features to 2 features to minimize the error in pairwise distance between the samples.  Let's add the 2 components to our DataFrame.

```python
df['MDS1'] = MDS_transformed[:,0]
df['MDS2'] = MDS_transformed[:,1]
df.head()
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod
      NS_Por
      NS_AI
      NS_TOC
      tProd
      MDS1
      MDS2

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819
      -0.982256
      -0.298603
      0.352257
      low
      0.610034
      -0.645613

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063
      -0.881032
      0.444147
      -0.213702
      med
      0.877231
      0.299790

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259
      -0.327677
      1.841224
      -0.213702
      med
      0.478352
      1.842171

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854
      0.903875
      -0.599240
      0.184565
      high
      -0.884147
      -0.479161

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624
      0.853263
      0.373409
      1.085907
      med
      -1.382521
      0.425827

Let's take a look at the samples projected into the new 2 dimensional feature space.

* note the rotation, translation is arbitrary in this space, only the sample pairwise distances are relevant.

```python
plt.subplot(121)
pairplot = sns.scatterplot(x = df['MDS1'],y = df['MDS2'],hue = df['tProd'],markers='o',palette = dpalette,edgecolor="black")

plt.subplot(122)
pairplot = sns.scatterplot(x = df['MDS1'],y = df['MDS2'],hue = df['Well'],markers='o',edgecolor="black")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2., top=1.3, wspace=0.3, hspace=0.2,)
plt.show()
```

Some comments:

* these are the actual samples in this space, we know the values, including the sample number to assist with interpretations

* the general relationship between samples is preserved.

* the general production transition from low to very high production is preserved.

Let's check our model:

* we will calculate the original and projected pairwise distances between all the samples

* we will cross plot the original vs the projects pairwise distances

* we will plot the distribution of the ratio between projects / original pariwise distances

```python
dists = euclidean_distances(df[ns_features], squared=False).ravel()
nonzero = dists != 0   # select only non-identical samples pairs
dists = dists[nonzero]
projected_dists = euclidean_distances(MDS_transformed, squared=False).ravel()[nonzero]

plt.subplot(221)
plt.scatter(dists,projected_dists,c='red',alpha=0.2,edgecolor = 'black')
plt.arrow(0,0,200,200,width=0.02,color='black',head_length=0.0,head_width=0.0)
plt.xlim(0,15); plt.ylim(0,15)
plt.xlabel("Pairwise Distance: original space")
plt.ylabel("Pairwise Distance: projected space")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

rates = projected_dists / dists
print("Distance Ratio, mean: %0.2f, standard deviation %0.2f." % (np.mean(rates), np.std(rates)))

plt.subplot(222)
plt.hist(rates, bins=50, range=(0.5, 1.5),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Distance Ratio: projected / original")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplot(223)
plt.hist(dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Original Data")

plt.subplot(224)
plt.hist(projected_dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.7, top=2.3, wspace=0.2, hspace=0.3)
plt.show()
```

We projected to a 2 dimensional feature space and did a pretty good job preserving the pairwise distances between the samples.

#### Observing Specific Samples

Let's reduce the number of wells and actually post the well numbers and observe their spacings in the original and projected features spaces.

```python
df_subset = df.iloc[:30,:]
```

Let's look at out dataset, we will visualize the 3 possible scatter plots between the three features with the sample points labeled by well number (1 through 30).

```python
plt.subplot(131)
pairplot = sns.scatterplot(x = df_subset[ns_features[0]],y = df_subset[ns_features[1]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[0]][i], df_subset[ns_features[1]][i]))

plt.subplot(132)
pairplot = sns.scatterplot(x = df_subset[ns_features[1]],y = df_subset[ns_features[2]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[1]][i], df_subset[ns_features[2]][i]))

plt.subplot(133)
pairplot = sns.scatterplot(x = df_subset[ns_features[0]],y = df_subset[ns_features[2]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[0]][i], df_subset[ns_features[2]][i]))

plt.subplots_adjust(left=0.0, bottom=0.0, right=2., top=1., wspace=0.3, hspace=0.2)
plt.show()
```

We can now see our 30 wells with their indices plots on each of the 3 possible bivariate plots.  Let's perform multidimensional scaling down to 2 components.

```python
embedding_subset = MDS(n_components=2)
MDS_transformed_subset = embedding_subset.fit_transform(df_subset[ns_features])
MDS_transformed_subset.shape
```

    (30, 2)

Now let's visualize the wells in the projected feature space.

```python
plt.subplot(221)
pairplot = sns.scatterplot(x = df_subset[ns_features[0]],y = df_subset[ns_features[1]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[0]][i], df_subset[ns_features[1]][i]))
pairplot.legend(loc='upper left')

plt.subplot(222)
pairplot = sns.scatterplot(x = df_subset[ns_features[1]],y = df_subset[ns_features[2]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[1]][i], df_subset[ns_features[2]][i]))
pairplot.legend(loc='lower right')

plt.subplot(223)
pairplot = sns.scatterplot(x = df_subset[ns_features[0]],y = df_subset[ns_features[2]],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset[ns_features[0]][i], df_subset[ns_features[2]][i]))
pairplot.legend(loc='upper left')

plt.subplot(224)
pairplot = sns.scatterplot(x = MDS_transformed_subset[:,0],y = MDS_transformed_subset[:,1],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (MDS_transformed_subset[i,0], MDS_transformed_subset[i,1]))
pairplot.set_xlabel('MDS1'); pairplot.set_ylabel('MDS2')
pairplot.legend(loc='lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.8, top=1.8, wspace=0.3, hspace=0.2)
plt.show()
```

Some observations:

* the transformation is not a rotation (like PCA)

* the intersample distance is approximately preserved

\begin{equation}
\delta_{i,j} = \left( (\delta Por_{i,j}^{2}) + (\delta AI_{i,j}^{2}) + (\delta TOC_{i,j}^{2}) \right)^{\frac{1}{2}}
\end{equation}

#### Applications

The main benefit with multiple dimensional scaling is the ability to inspect high dimensional feature spaces for relationships between samples.

* we may observe that specific wells cluster together
* we may observe systematic transitions

#### Random Projection

Let's demonstrate the use of random projection.

* given the low dimensionalilty you will see quite poor performance for some randon seeds

```python
n_components = 2; seed = 73058
rp = GaussianRandomProjection(n_components=n_components,random_state = seed)

dists = euclidean_distances(df[['Por','AI','TOC']], squared=False).ravel()
nonzero = dists != 0   # select only non-identical samples pairs
dists = dists[nonzero]

projected_data = rp.fit_transform(df[['Por','AI','TOC']])
projected_dists = euclidean_distances(projected_data, squared=False).ravel()[nonzero]

plt.subplot(221)
plt.scatter(dists,projected_dists,c='red',alpha=0.2,edgecolor = 'black')
plt.arrow(0,0,200,200,width=0.02,color='black',head_length=0.0,head_width=0.0)
plt.xlim(0,15); plt.ylim(0,15)
plt.xlabel("Pairwise Distance: original space")
plt.ylabel("Pairwise Distance: projected space")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

rates = projected_dists / dists
print("Distance Ratio, mean: %0.2f, standard deviation %0.2f." % (np.mean(rates), np.std(rates)))

plt.subplot(222)
plt.hist(rates, bins=50, range=(0., 2.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Distance Ratio: projected / original")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplot(223)
plt.hist(dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Original Data")

plt.subplot(224)
plt.hist(projected_dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.7, top=2.3, wspace=0.2, hspace=0.3)
plt.show()
```

We should see improved performance (lower stress) for high dimensional problems (large $m$).

#### Comments

This was a basic demonstration of multidimensional scaling. A lot more could be done:

* comparison to principal components analysis
* use of a dataset with larger number of features
* use of a dataset with natural clusters in multivariate space

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Time evolution of the entanglement entroy of a product state

This project is part of the Computational Physics course held at Perimeter, Fall 2018.

```julia
using LinearAlgebra
using Arpack
using Plots
```

The following functions are used to compute the entanglement entropy, given a state separated in sub-states A and B.

```julia
function compute_rdmA(PsiAB)
    @assert ndims(PsiAB) == 2 # did we input a matrix?
    @assert size(PsiAB,1) < 2^13 # make sure the resulting matrix is not too large!
    rdm = PsiAB*PsiAB'
end

function compute_entropy(dm)
    p = eigvals(dm) # we diagonalize and use the eigenvalues. the more Julia way
    # has a problem when the log is very large
    entropy = 0.0
    #entropy = - p'*log2.(p) # problems when p[n] is of order 0 (possibly negative)
    for n in 1:size(p,1)
        if abs(p[n]) > 1e-12
            entropy = entropy - p[n]*log2(p[n])
        end
    end
    entropy
end
```

    compute_entropy (generic function with 1 method)

## Product state

We use the outer product of 1 spin states to construct a separable state:

\begin{equation}
| \Psi_{\text{sep}} > = |\Psi>_1 \otimes |\Psi>_2 \otimes \cdots \otimes |\Psi>_N
\end{equation}

```julia
N=2*5 # > 2

function normalizeVec(Vec)
    Vec = Vec/sqrt(abs(Vec'*Vec))
end

# here we implement the outer product
function separable(N)
    PsiInitial= reshape( (randn(2) + im*randn(2))*(randn(2) + im*randn(2))', 2^2 )
    for i in 3:N
        PsiInitial = reshape( PsiInitial*(randn(2) + im*randn(2))', 2^i )
    end
    return normalizeVec(PsiInitial)
end

PsiInitial = separable(N);
```

```julia
# checking separability
nA = Int(N/2) # number spins/qubits in A
nB = Int(N/2) # number of spins/qubits in B

PsiAB = reshape(PsiInitial, (2^(nA),2^(nB)))

rdmA = compute_rdmA(PsiAB)

isapprox(compute_entropy(rdmA), 0, atol=10^-14)  # checking that entropy is zero to 10^-14 accuracy
```

    true

## Time evolution

To obtain $|\Psi(t)\rangle \equiv \exp (-i t H) |\Psi(0)\rangle$, we can use
 $|\Psi(t+\delta t)\rangle = |\Psi(t)\rangle - i \delta t H |\Psi(t)\rangle$. We want to keep the accuracy within $10^{-5}$ at each time step.

We want the code to be able to deal with large spin chains, so we do not want to record the full Hamiltonian matrix.

```julia
# Ising model
function buildIsing(theta=pi/4) # Ising model with transverse magnetic field h (critical theta=pi/4 by default)
    X = [0. 1; 1 0]
    Z = [1. 0; 0 -1]
    E = diagm(0=>ones(2))
    XX = kron(X,X)
    HXX = XX
    HZ = kron(Z,E) + kron(E,Z)
    H2 = -(cos(theta)*XX + sin(theta)/2*HZ)
    return H2
end

# Energy shift
theta = pi/4 # critical magnetic field
H2 = buildIsing(theta) # Hamiltonian
D,U = eigen(H2)
shiftE = D[end]
H2 = H2 - shiftE*diagm(0=>ones(4))

# Sparse multiplication
function multiplyHPsi(Psi, H2)
    L = length(Psi)  # Dimension of the vector space
    N = convert(Int64,log2(L))  # Number of spins
    HPsi = zeros(L)
    for n=1:N                              # This multiplies by the Hamiltonian H
        Psi = reshape(Psi,(4,2^(N-2)))
        HPsi = reshape(HPsi,(4,2^(N-2)))
        HPsi += H2*Psi
        Psi = reshape(Psi,(2,2^(N-1)))
        HPsi = reshape(HPsi,(2,2^(N-1)))
        Psi = permutedims(Psi,(2,1))
        HPsi = permutedims(HPsi,(2,1))
    end
    Psi = reshape(Psi,L)
    HPsi = reshape(HPsi, L)
    return HPsi
end
```

    multiplyHPsi (generic function with 1 method)

## Example: N=12

```julia
# Initialization
init_steps = 1
entropy = zeros(1)
normdiff = zeros(1)
initial_step=1
Nsteps = 0;

N = 2 * 6
Psi = separable(N);
deltat = 0.0001
```

    0.0001

The for loop implements $|\Psi(t+\delta t)\rangle = |\Psi(t)\rangle - i \delta t H |\Psi(t)\rangle$.

```julia
# Loop over the time steps
initial_step += Nsteps
Nsteps = 80000  # change this number if the entanglement entropy has not reached its plateau
final_step = initial_step + Nsteps-1

for n=initial_step:final_step
    Psi = Psi .- im * deltat .* multiplyHPsi(Psi,H2)
    if n%400==0
    normdiff = [normdiff; abs(1 - sqrt(abs(Psi'*Psi)))]
    #Psi = normalizeVec(Psinew)
    PsiAB = reshape(Psi, (2^(Int(N/2)),2^(Int(N/2))))
    rdmA = compute_rdmA(PsiAB)
    entropynew = compute_entropy(rdmA)
    entropy = [entropy; entropynew]
    end
    if n%4000==0 print(n-initial_step+1, ":", Nsteps, " ") end
end
```

    4000:80000 8000:80000 12000:80000 16000:80000 20000:80000 24000:80000 28000:80000 32000:80000 36000:80000 40000:80000 44000:80000 48000:80000 52000:80000 56000:80000 60000:80000 64000:80000 68000:80000 72000:80000 76000:80000 80000:80000

We plot the difference from 1 of the normalization, to check the precision of our time integration.

```julia
tarray = collect(range(0, stop = deltat * Nsteps, length = length(normdiff)))
standard = [10^-5 for i in 1:length(tarray)]
plot(tarray, standard, label = "goal", xlabel = "t", ylabel = "normalization loss")
plot!(tarray, normdiff, label = "numerical", markershape = :hexagon, markerstrokealpha = 0,
    markersize = 2)
```

```julia
savefig("12spins_Ising_norm.png")
```

## Evolution of the entanglement entropy

We can finally plot the evolution of the entanglement entropy of the half chain with time.

```julia
tarray = collect(range(0, stop = deltat * Nsteps, length = length(normdiff)))
asympt = [N/2 for i in 1:length(tarray)]
plot(tarray, asympt, xlabel = "t", ylabel = "entanglement entropy", label = "N/2")
plot!(tarray, entropy, label = "numerical", markershape = :hexagon, markerstrokealpha = 0,
    markersize = 2)
#(length(tarray),length(entropy))
```

For the Ising model, with similar parameters as the ones used for the modified Hamiltonian, we do not saturate at the same point.

```julia
savefig("12spins_Ising_ee.png")
```


# Demo 1: Flow

## Background/Motivation

Flow is a fundamental driver of processes in the surface and subsurface. In this set of demonstrations we will introduce

  * Richards Model for variably saturated subsurface flow
  * Diffusion Wave model for overland flow (surface flow)
  * A fully coupled (integrated hydrology) model

For each model we will provide the mathematical equations, highlight information about the discretization and provide tips on setting boundary conditions, initial conditions, source terms and key parameters.

## Subsurface Flow (Richards Equation)

We can provide the essentials of the mathematical models using LaTeX.  For example, Richards (1931) model for variably saturated flow:

$\newcommand{porosity}{\phi}$
$\newcommand{dens}{n_l}$
$\newcommand{sat}{s}$
$\newcommand{flux}{\mathbf{q}}$
$\newcommand{krel}{k_r}$
$\newcommand{kabs}{\kappa}$
$\newcommand{viscosity}{\mu}$

$$
\begin{align}
  \frac{\partial}{\partial t} (\porosity \dens \sat) &+ \nabla \cdot \flux = 0 \\
  \flux &= - \dens \frac{\krel \kabs}{\viscosity} (\nabla p + \rho \mathbf{g})
\end{align}
$$

**Note that this uses latex macros defined in this cell.**
Water retention models

Blah, Blah, lots more about the model and equations ...

These tests demonstrate solution of Richards equation on a single column. A single column of cells, oriented vertically, and are initialized as the hydrostatic solution. Infilitration is turned on, and water fills the domain.

```python
%matplotlib inline
```

```python
import sys,os
sys.path.append(os.path.join(os.environ['ATS_SRC_DIR'],'tools', 'utils'))
import ats_xdmf
import plot_column_data
import numpy as np
from matplotlib import pyplot as plt
```

```python
def plot(dirname, title):
    """A generic plotting function that plots saturation as a function of depth."""
    fig = plt.figure()

    vis = ats_xdmf.VisFile(dirname, time_unit='d')
    vis.loadMesh(columnar=True)

    # plot the solution
    ax = fig.subplots(1,1)
    plot_column_data.plot_subsurface(vis, ['saturation_liquid',], ax, None, cmap='jet')
    ax.set_title(title)
    ax.set_xlabel('saturation [-]')
    ax.set_ylabel('z [m]')
```

```python
plot("./richards/run/", "infiltration example")
```

## Overland Flow (Diffusion Wave Equation)

## Integrated Hydrology


# Mixed Dirichlet

A mixed Dirichlet random variable $Y$ takes on values in the probability simplex $\Delta_{K-1}$, an assignment $Y=y$ has probability density given by

\begin{align}
P_{Y}(y|\alpha, w) &= \sum_{f} \mathrm{Gibbs}(f|w) \times \mathrm{Dirichlet}(y|\alpha \odot f)
\end{align}

where $w \in \mathbb R^K$, $\alpha \in \mathbb R^K_{>0}$, $f$ is one of the non-empty faces of the simplex,  by $\alpha \odot f$ we mean the sub-vector of $\alpha$ whose coordinates are associated with the vertices in $f$.

The distribution over proper faces has probability mass function:
\begin{align}
\mathrm{Gibbs}(f|w) = \frac{\exp(w^\top \phi(f))}{\sum_{f'} \exp(w^\top \phi(f'))}
\end{align}
where $\phi(f) \in \mathbb {-1, 1}^K$ is such that $\phi_k(f) = 1$ if the vertex $\mathbf e_k$ is in the face, and $-1$, otherwise.

```python
import probabll.distributions as pd
```

```python
import torch
import torch.distributions as td
```

```python
import matplotlib.pyplot as plt
```

```python
def plot_marginals(samples, bins=100):
    D = samples.shape[-1]
    fig, ax = plt.subplots(D, 1, figsize=(4, 2*D), sharex=True)
    for d in range(D):
        _ = ax[d].hist(samples[...,d].flatten().numpy(), bins=bins, density=True)
    return fig, ax
```

# Uniform F and Uniform Y|f

```python
p3d = pd.MixedDirichlet(concentration=torch.ones(3), scores=torch.zeros(3))
```

```python
p3d.sample([10])
```

```python
p3d.entropy(), p3d.cross_entropy(p3d), td.kl_divergence(p3d, p3d)
```

```python
_ = plot_marginals(p3d.sample([1000]), bins=100)
```

```python
_p = p3d.expand([2, 1])
_p.sample().shape
```

```python
_p.entropy(), _p.cross_entropy(_p), td.kl_divergence(_p, _p)
```

```python
_p.faces.cross_entropy(_p.faces).shape
```

# Max-Ent F and Uniform Y|f

```python
pm3d = pd.MixedDirichlet(concentration=torch.ones(3), pmf_n=pd.MaxEntropyFaces.pmf_n(3, 1))
```

```python
_ = plot_marginals(pm3d.sample(torch.Size([1000])), bins=100)
```

```python
pm3d.entropy(), pm3d.cross_entropy(pm3d), td.kl_divergence(pm3d, pm3d)
```

```python
_pm = pm3d.expand([2, 1])
_pm.sample().shape
```

```python
_pm.entropy(), _pm.cross_entropy(_pm), td.kl_divergence(_pm, _pm)
```

# VI

```python
p = pd.MixedDirichlet(concentration=torch.ones(5), pmf_n=pd.MaxEntropyFaces.pmf_n(5, 1))
q = pd.MixedDirichlet(concentration=torch.ones(5)/10, scores=torch.zeros(5))
```

```python
p.batch_shape, p.event_shape
```

```python
p.sample(torch.Size([10]))
```

```python
f = p.faces.enumerate_support()
```

```python
p.faces.log_prob(f).exp(), f.sum(-1)
```

```python
p.cross_entropy(q)
```

```python
p.Y(f).cross_entropy(q.Y(f))
```

```python
p.Y(f).entropy()
```

```python
p.cross_entropy(q)
```

```python
td.kl_divergence(p, q)
```

```python
td.kl_divergence(p.faces, q.faces)
```

```python

```

```python

```


# HMC Monte-Carlo sampling from a power-law mass function

We assume a set of masses drawn from a power-law distribution. The distribution
is defined by 3 parameters: the index $\alpha$, the lower and upper mass range
$M_{min}$, $M_{max}$.
    $$\theta = { \alpha, M_{min}, M_{max} }$$

Let's suppose known $M_{min}$ and $M_{max}$ such that we only look at the posterior of
$\alpha$. We know that in fact $M_{max}$ and $M_{min}$ cannot be infered because they are
most likely outside from the mass range covered by the data set.
    $$dN/dM = Z \times M ^ -\alpha$$
where
\begin{equation}
     Z = \int_{M_{min}}^{M_{max}} dN/dM dM \\
       = \frac{(1 - \alpha)}{(M_{max} ^ {1 - \alpha} - M_{min} ^ {1 - \alpha})}
\end{equation}

we define $\beta = 1 - \alpha$
    $$Z = \frac{\beta}{M_{max} ^ \beta - M_{min} ^ \beta}$$

For this test, we neglect errors of stellar mass observations.
Least-square likelihood could be used in this case. However instead we can make
use that the mass function is the PDF of our data. Thus, We can derive the likelihood as
    $$L(Data; \alpha) = \prod_k Z \times M_k ^ \alpha$$
and because numerically more stable and usually makes the math simpler we
define the  log-likelhood $\ln L(Data; \alpha)$ as
    $$\ln L(Data; \alpha) = N \ln(Z) - \alpha  \sum_k \ln(M_k)$$

Note that the data set only enters via $D = \sum_k \ln(M_k)$, term that is
completely independent of the fit parameter. Hence this term can be computed
once and for all at beginning. Therefore, the length of the data set does not
slow down the computations. (This may not be true for non-perfectly known masses)

This experiment aims at implementing Hamiltonian Monte-Carlo method. This
method involves a stepsize as the traditional MH (Metropolis Hastings) method
but improves efficiency by updating this value using the local gradient of the
objective function $\ln L$.

Hamiltonian Monte-Carlo makes use of the fact, that we can write our
log-likelihood as an energy similarly to dynamic systems:
    $$L = \exp( \ln(L) ) = \exp( -E )$$
So that $E = - \ln(L)$ is the energy of any point in the parameter space.

Maximazing the likelihood becomes finding the minimum state of energy.
The algorithm then uses Hamiltonian dynamics to modify the way how candidates
are proposed by computing the total energy of a local point by:
    $$H = \frac{1}{2} p ^ 2 + E$$
where $p ^ 2$ is the impulse (movement between 2 steps)
$p$ is proposed by doing random leap frog steps locally using the gradient of the
objective function.

In our case the gradient function of the log-likelihood is:
\begin{eqnarray}
    \frac{d \ln L}{d \theta} &=& \frac{d \ln L}{d \alpha}\\
    &=& -D - \frac{N}{\beta} \times \left[ 1 + Z \times (\ln(M_{min}) \times M_{min} ^ \beta - \ln(M_{max}) \times M_{max} ^ \beta)\right]
\end{eqnarray}

```python
%matplotlib inline
```

```python
import numpy as np
import random as random
import matplotlib.pyplot as plt
import time
from nuts import nuts6, NutsSampler_fn_wrapper

# set the seed
random.seed(1)

def random_PowerLaw(N, alpha, M_min, M_max):
    """
    Draw random samples from a power-law defined over M_min, M_max.
        dN/dM = Z x M ** -alpha
    INPUTS
    ------
    N: int
        number of samples.
    alpha: float
        power-law index.
    M_min: float
        lower bound of mass interval.
    M_max: float
        upper bound of mass interval.
    OUTPUTS
    -------
    masses: ndarray[float, ndim=1]
        list of N masses drawn from the power-law
    """
    beta = 1. - alpha
    x = np.random.uniform(0., 1., N)
    if beta == 0:
        y = M_min * np.exp( np.log(M_max / M_min) * x )
    else:
        y = ((M_max ** beta - M_min ** beta) * x + M_min ** beta) ** (1. / beta)
    return y

def logLikelihood(theta, D, N, M_min, M_max):
    """
    Define logarithmic likelihood function.
    theta: ndarray[float, ndim=1]
        array of fit params
    D: float
        data
        normalization constant, sum_n log(M_n)
    N: int
        number of data points
    M_min: float
        lower limit of mass interval
    M_max: float
        upper limit of mass interval
    """
    # extract alpha
    alpha = theta[0]
    beta = 1.0 - alpha

    # Compute normalisation constant.
    if beta == 0:
        c = np.log(M_max / M_min)
    else:
        c = beta / (M_max ** beta - M_min ** beta)

    # return log likelihood.
    return N * np.log(c) - alpha * D

def grad_logLikelihood(theta, D, N, M_min, M_max):
    """Define gradient of log-likelihood
    d lnL / d theta (here only one dim alpha
    theta: ndarray[float, ndim=1]
        array of fit params
    D: float
        data
        normalization constant, sum_n log(M_n)
    N: int
        number of data points
    M_min: float
        lower limit of mass interval
    M_max: float
        upper limit of mass interval
    """
    alpha = theta[0]  # extract alpha
    beta = 1.0 - alpha

    #Convert limits from M to logM.
    logMmin = np.log(M_min)
    logMmax = np.log(M_max)

    if beta != 0:
        grad = logMmin * M_min ** beta - logMmax * M_max ** beta
        grad = 1.0 + grad * beta / (M_max ** beta - M_min ** beta)
        grad = -D - N * grad / beta
    else:
        grad = float(N)

    return np.array([grad])
```

```python
# Generate toy data.
Nstars = int(1e5)
alpha  = 2.35
M_min  = 1.0
M_max  = 100.0
Masses = random_PowerLaw(Nstars, alpha, M_min, M_max)
LogM   = np.log(Masses)
D      = np.mean(LogM) * Nstars

#NUTS pars
M, Madapt = 1000, 1000
theta0 = np.asarray([3.0])
delta = 0.5

nuts_fn = NutsSampler_fn_wrapper(logLikelihood, grad_logLikelihood, D, Nstars, M_min, M_max)
#nuts_fn.verbose = True

t_start = time.time()
print("Starting Sampling at %s" % time.ctime(t_start))
A, lnprob, epsilon = nuts6(nuts_fn, M, Madapt, theta0, delta)
t_stop = time.time()
print("Sampling Completed in %0.2f seconds" % (t_stop - t_start))
```

    Starting Sampling at Mon Jul 16 16:01:59 2018
    find_reasonable_epsilon= 0.0078125

    /srv/conda/lib/python3.6/site-packages/ipykernel_launcher.py:62: RuntimeWarning: overflow encountered in double_scalars
    /srv/conda/lib/python3.6/site-packages/ipykernel_launcher.py:65: RuntimeWarning: divide by zero encountered in log
    /srv/conda/lib/python3.6/site-packages/ipykernel_launcher.py:91: RuntimeWarning: overflow encountered in double_scalars
    /srv/conda/lib/python3.6/site-packages/ipykernel_launcher.py:92: RuntimeWarning: overflow encountered in double_scalars
    /srv/conda/lib/python3.6/site-packages/ipykernel_launcher.py:92: RuntimeWarning: invalid value encountered in double_scalars
    /home/jovyan/nuts/nuts.py:177: RuntimeWarning: overflow encountered in exp
      alphaprime = min(1., np.exp(joint - joint0))

    Sampling Completed in 0.21 seconds

```python
plt.figure(1)

# Print Monte-Carlo estimate of alpha.
print("Mean:  " + str(np.mean(A)))
per = np.percentile(A, [16, 50, 84])
print("Alpha = {} (+{} / - {})".format( per[1], per[2] - per[1], per[1] - per[0] ))

n, b = np.histogram(A, 30)
x = 0.5 * (b[:-1] + b[1:])
y = n.astype(float) / n.sum()
plt.step(x, y, color='b', lw=3, where='mid')
plt.vlines(per, 0., max(y), linestyle='--', color='b', lw=1)

ylim = plt.ylim()
plt.vlines(alpha, 0, ylim[1], color='r', lw=3)
plt.ylim(ylim)
plt.xlabel(r'$\alpha$', fontsize=24)
plt.ylabel(r'$\cal L($Data$;\alpha)$', fontsize=24)
plt.show()
```

```python

```