Copyright statement?

# A New 'Transition'
Welcome back! At this point in the course you are all likley aces at numerical analysis, expecially after 5 modules ranging from Diffusion, to Convection, to Conduction, in 1 or 2D, with various Neumann and Dirichlet Boundary conditions!

But now it's time for a new "phase". In this lesson we begin looking at what happens when your nice rigid boundary conditions decide to start moving on you, essentially creating a moving boundary interface!

A moving boundary interface is represented by numerous physical behaviors in real-world applications, from the polar ice caps melting, to the phase transformations of metal alloys, to the varying oxygen content of  muscles near a clotted bloodvessel [2].

 #### Real World Applications of a Moving Boundary Interface

### The Stefan Problem
This new type of problem is known as the "Stefan Problem" as it was first studied by Slovene physicist Jozef Stefan around 1890 [3]. Though his focus was primarily on the analysis of ice formations, nowadays his name is synonymous with the particular type of boundary value problem for PDEs where the boundary can move with time.  Since the classic Stefan problem concentrated on the temperature distribution of a homogeneous material undergoing a phase transition, one of the most commonly studied Stefan problems today is the melting of Ice to Water!

 #### Jozef Stefan pioneered work into phase transitions of materials (ie Ice)

## A Review: 1D Heat Conduction

Recall from both Modules 2 and 4 we took a loook at the Diffusion equation in 1D:

$$\begin{equation}
\frac{\partial U}{\partial t} = \alpha \frac{\partial^2U}{\partial x^2}
\end{equation}$$

Where we have the temperature distribution $U(x,t)$ and the thermal diffusivity $\alpha$. While before we looked at the conduction of heat through a graphite rod of length 1 meter, in this scenario we will analyze heat conduction though a 1D rod of ice.  Let's first list some basic coefficients associated with the new material:

##### Thermal Properties of Ice at ~$0^{\circ}C$:
Thermal Conductivity: $k = 2.22 \frac{W}{mK}$

Density: $\rho \approx 1000 \frac{kg}{m^3}$

Specific Heat: $c_{p}= 2.05x10^{-3} \frac{J}{kgK}$

and lastly, Thermal Diffusivity: $\alpha = \frac{k_{ice}}{\rho_{ice}c_{p_{ice}}} = 1.083x10^{-6} \frac{m^2}{sec}$

Melting Temperature of Ice is: $T_{melt}=0^{\circ}C$

Okay! With that out of the way, let's look at the temperature distribution across the a rod of ice of length 1m with some basic initial and boundary conditions. For this first scenario, we will not take into account phase transition if temperatures hit $T_{melt}$, and will therefore assume a static boundary condition.

##### Problem Setup
Governing Equation: $$\begin{equation}
\frac{\partial U}{\partial t} = \alpha \frac{\partial^2U}{\partial x^2}
\end{equation}$$

Boundary Conditions:

LHS: $\frac{\partial U}{\partial x} = -e^{\beta t}$,   @x=0, t>0, time-dependant (increasing) heat flux in

RHS: $\frac{\partial U}{\partial x} = 0$,  @x=L, t>0,   Insulated end, no heat flux in/out.

Initial Conditions:

$U(x,t) = -10^{\circ}C$, for 0<x<L, t=0

Lets start coding!!

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family']= 'serif'
rcParams['font.size']=16
from IPython.display import Image
```

Now let's define a function to run the governing equations using the Forward-Time/Central-Difference Method discretization

```python
def FTCS(U, nt, dt, dx, alpha, beta):

    for n in range(nt):

        Un=U.copy()
        U[1:-1] = Un[1:-1] + alpha*(dt/dx**2)*(Un[2:]-2*Un[1:-1]+Un[0:-2])

        #Boundary Conditions

        U[-1]=U[-2]                  #RHS Insulated BC

        t=n*dt                                    #Increasing time = n*timestep(dt)
        U[0]=numpy.exp(beta*t)     #LHS Time dependant heat input in, BC

    return U
```

```python
#Basic Parameters and Initialization

# Temperature scale: Celsius
# Length scale: meters
# Mass scale: kg
# Time scale: seconds
# Energy scale: Joules
# Power scale: Watts

L = 1                                                         # Length of my ice rod
nt = 40000                                                  # Number of timesteps
nx = 51                                                     # Number of grid space steps
alpha = 1.083e-6                                    # Thermal Diffusivity
dx = L/(nx-1)                                           # grid spacing in "x"
Ui = numpy.ones(nx)*(-10)                    # initialized Temperature array
beta = 0.001                                               #Growth Factor of my Temperature input

sigma = 1/2                                             # Stability*
dt=0.1                                                      # Timestep chosen to be 0.1 seconds*
```

A word on Stability:

Recall from module 2 we had the Courant-Friedrichs-Lewy (CFL) Stability condition for the FTCS Diffusion equation in the form:
$$\sigma = \alpha \frac{\Delta t}{\Delta x^{2}} \leq \frac{1}{2} $$

re-arranging to determine the upper limit for a time-step (dt) we have:
$$\Delta t \leq \frac{\sigma \Delta x^{2}}{\alpha} \approx 923 seconds$$

As you can see, chosing a time-step (dt) equal to $0.1$ seconds, we more than satisfy this CFL Stability condition.

```python
x = numpy.linspace(0, 1, nx)
print('Initial ice rod temperature distribution')
pyplot.plot(x, Ui, color = '#003366', ls = '-', lw =4)
pyplot.ylim(-20, 60)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')

```

```python
U = FTCS(Ui.copy(), nt, dt, dx, alpha, beta)
print('Total elapsed time is ', nt*dt, 'seconds, or', nt*dt/60, 'minutes' )
pyplot.plot(x, U, color = '#003366', ls = '-', lw =3)
pyplot.ylim(-20, 60)
pyplot.xlim(0, 1)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
```

The above figure shows us the temperature distribution of the ice rod at a point in time approximately 1 hour into applying our heat source. Now this would be all well and good if ice didn't melt (and therefore change phase) at zero degrees. But it does!!

Let's build a rudimentary function to see what portion of the rod should be water by now, and which part should still be ice:

```python
def Phase_graph(U, x, nx):
    phase=numpy.ones(nx)*(-100)
    for n in range(nx):
        if U[n]>0:
            phase[n]=100
    return phase
```

```python
pyplot.plot(x, U, color = '#003366')
pyplot.ylabel('Temperature', color ='#003366')
Phase1=numpy.ones(nx)
Phase=Phase_graph(U, x, nx)
pyplot.plot(x, Phase, color = '#654321', ls = '-', lw =3)
pyplot.ylim(-12, 10)
pyplot.xlim(0, 0.3)
pyplot.xlabel('Approx Water-Ice Interface', color = '#654321')
```

As you can see, the ice SHOULD have melted about 0.07 meters  (or 2.75 inches) into our rod. In reality, our boundary interface has moved from x=0 to the right as time elapsed.  Not only should our temperature distribution profile change due to the differences in properties ($\rho, k, c_{p}$), but also the feedback from the moving boundary condition.

## Solutions to the Dimensionless Stefan problem:
### Analytical Solution:

Before we continue we need to make some simplfications:

1) No convection, heat transfer is limited to conduction,

2) Pressure is constant,

3) Density does not change between the solid and liquid phase (Ice/Water), ie $\rho_{ice}=\rho_{water}\approx 1000 \frac{kg}{m^3}$,

4) The phase change interface at $s(t)$ has no thickness

Looking closer at the problem at hand, we see that our temperature distribution must conform to the below figure:

 #### Phase Change Domain

  As you can see, our temperature distribution experiences a discontinuity at our solid-liquid interface at x=s(t). Furthermore, this boundary moves to the right as time elapses. If we wish to analyze the distribution in one region or the other, we need to take into account a growing or shrinking domain, and therefore, the boundary of the domain has to be found as part of the solution [2].  In the Melting Problem depicted above, a moving interface separates the liquid and solid phases. The displacement of the boundary interface (denoted as $\dot{s}$) is driven by the heat transport through it [2]. The relationship  between the moving interface, s(t), and the temperature distribution through it was developed by and is known as the Stefan Condition (or Stefan Equation) and takes the form of the following boundary conditions:

when x =s(t),
######  [1, 3, 2]
$$\dot{s} = \frac{ds}{dt} = -\frac{\partial U}{\partial x}$$
$$U=0$$

There are many solution methods for the Stefan problem, for this lesson we will focus on finding the temperature distribution in the liquid region only (ie 0<x<s(t)). Critical parameters to track are both temperature, U(x,t), and the position of the interface, s(t). Let's first solve this problem analytically:

To make our lives easier we want a simplified version of the Stefan Problem, In his paper, Crowley [4] presented that both Oleinik and Jerome demonstrated how using an appropriate dimensionless model, and appropriate boundary conditions, and initial conditions, not only could one determine a general solution for the diffusion equation with the Stefan condition, but also that an explicit finite difference scheme converges to this solution [4]. For our dimensionless model we will set $\alpha=1, \beta=1$ to get the following governing equations, and boundary conditions:

#### Dimensionless Stefan Problem Equations

$$\frac{\partial U}{\partial t} = \frac{\partial^{2} U}{\partial x^{2}}$$

$$s(t)$$

(1) $\frac{\partial U(x=0, t)}{\partial x} = -e^{t}$, LHS BC (Heat input into the system)

(2) $\frac{\partial U(x=s(t), t)}{\partial x} = -\frac{d s(t)}{d t}$, RHS BC, Stefan Condition

(3) $U(x=s(t), t) = T_{melt}=0$, By definition of the Melting interface

(4) s(t=0)=0, initial condition

These equations set up the new below figure:

Because we want to be able to judge the accuracy of our numerical analysis (and thanks to Crowley, Oleinik, and Jerome!) we next find the general solution of this problem by means of Separation of Variables:

SOV: $U(x,t)=X(x)T(t)$, gives us a general solution of the form:

$$U(x,t) = c_{1}e^{c_{2}^{2}t-c_{2}x} +c_{3}$$

from BC(1) we get $c_{2}=1, c_{1}=1, yielding: U(x,t) = e^{t-x} +c_{3}$

from BC(2) we get:  $\dot{s}=\frac{\partial s(t)}{\partial t} = -\frac{\partial U(x=s(t), t)}{\partial x}\rightarrow -e^{t-s(t)}= \frac{ds(t)}{d t}$, this yields that the only solution fo s(t) is: $$s(t)=t$$

from BC(3) we get: $U(s(t),t) + c_{3} = 0, \rightarrow e^{t-t} +c_{3}=0, \rightarrow c_{3}=-1$,

Finally we get an exact solution for the temperature distribution:

$$U(x,t) = e^{t-x}-1$$,    $$s(t) = t$$

and this satisfies the initial condition IC(4) that s(t=0)=0!!

And now you see why we chose the heat input function to be a time-dependant exponential, if you don't believe me that the analytical solution would be much nastier if you had chosesn say, a constant heat input, give SOV a try and let me know how it goes! Let's graph this solution and see if it makes sense.

```python
def Exact_Stefan( nt, dt, x):
    U=numpy.ones(nx)
    for n in range(nt):
        U = numpy.exp(n*dt-x)-1
    return U
```

```python
dt_A=0.002
nt_A=500
ExactS=Exact_Stefan(nt_A, dt_A, x)
Max_tempA=max(ExactS)

pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
print('Analytically, this is our temperature profile after:', nt_A*dt_A,'seconds')
print('The temperature at our LHS boundary is:', Max_tempA, 'degrees')
```

Does the solution look right? YES! Remember, this temperature profile is "Dimensionless" so we can't compare it to our previous example (not only is the diffusivity constant vastly different, but we are looking 1 second into the diffusion vs 1 hr!). Also it accounts ONLY for the temperature distribution in the liquid, assuming an ever increasing domain due to an ever expanding RHS boundary. Not only did we expect temperature to be highest at the input side, but our moving boundary interface, s(t), moves to the right with time and always hits our melting temperature ($0^{\circ} c$) at x=1, just as one would expect.

### Numerical Solution: The Variable Grid Method

What makes this problem unique is that we must now track the time-dependant moving boundary. Now, for previous numerical analysis in a 1D domain, you had a constant number of spatial intervals nx (or N) between your two boundaries x=0, and x=L. Thus your spatial grid steps dx were constant and defined as $dx = L/(nx-1)$. But now you have one fixed bounary and one moving boundary, and your domain increases as time passes. [1]

One of the key tenements of the Variable Grid Method for solving the Stefan probem is that you keep the NUMBER of grid points (nx) fixed, thus your grid SIZE (dx) will increase as the domain increases. Your grid size now varies dependant on the location of the interaface front, dx = s(t)/N.

$$dx = \frac{L}{N-1}     \longrightarrow      dx = \frac{s(t)}{N}$$

Now, while one might be tempted to view the figure above as the new "Variable Grid" stencil, one must remember that the "Y"-axis is time, and therefore as you move up you are moving forward in time, therefore a more accurate depiction of the FTCS method in stencil form for the Variable Grid method would be:

and so it becomes clear - that the our spatial step ($dx$) will be depentant on our time step ($m$)!!

####  Derivation of the new Governing Equation

   Now let's set up the new governing equations to be discretized for our code. We know that the 1D Diffusion equation must be valid for all points on our spatial grid so we can re-write the LHS of equation:
   $$\frac{\partial U}{\partial t} = \frac{\partial^{2} U}{\partial x^{2}}$$
   to be $$ \frac{\partial U_{i}}{\partial t} = \frac{\partial U_{@t}}{\partial x}\frac{dx_{i}}{dt} + \frac{\partial U_{@x}}{\partial t}$$

  we can use the expression: $$\frac{dx_{i}}{dt} = \frac{x_{i}}{s(t)}\frac{ds}{dt}$$   to track the movement of the node $i$.

  Substituting these into the diffusion equation (and droping the i,t, and x indices since they are constant) we obtain a new governing equation for diffusion:

  $$\frac{\partial U}{\partial t} = \frac{x_{i}}{s}\frac{ds}{dt}\frac{\partial U}{\partial x} + \frac{\partial^{2}U}{\partial x^{2}}$$

  This is subject to the boundary and initial conditions (BC1-BC3, IC4) as stated for the 1D dimensionless Stefan problem above.

#### Discretization

and now we seek to discretize the new governing equations. For this code we will implement an explicit, FTCS scheme, with parameters taking a Taylor expansion centered about the node ($x_{i}^{m}$), and time ($t_{m}$) just like before. In the above equations we discritized the time derivatives of U using forward time, and the spatial derivitaives using centered space. We re-write ds/dt as $\dot{s}$ and leave it a variable for now. (EXERCISE: See if you can discretize this equation form memory!!)

Taylor Expand and Discretize: (moving from left to right)

$\frac{\partial U}{\partial t} \longrightarrow  \frac{U_{i}^{m+1}-U_{i}^{m}}{\Delta t}$,

$\frac{x_{i}}{s}  \longrightarrow  \frac{x_{i}^{m}}{s_{m}}$,

$\frac{ds}{dt} \longrightarrow \dot{s_{m}}$,

$\frac{\partial U }{\partial x} \longrightarrow \frac{U_{i+1}^{m}-U_{i-1}^{m}}{2\Delta x^{m}}$,

$\frac{\partial^{2}U}{\partial x^{2}} \longrightarrow \frac{U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m}}{(\Delta x^{m})^{2}}$

And now we substitute, rearrange and solve for $U_{i}^{m+1}$ to get:

$U_{i}^{m+1}=U_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(U_{i+1}^{m}-U_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m})$

Great!! We're almost ready to start coding, but before we begin, do you notice a problem with this expression? What about if i=0? Plug it in and you will see that we have expressions of the form $U_{-1}^{m}$ in both right hand terms, but that can't be right, looking at our stencil we see that $i=-1$ is off our grid! This is where the boundary conditions come in!

Generate discretization expressions for the boundary conditions at x=0 (LHS) and x=s(t) (RHS). For the RHS and the temperature gradient across the moving boundary interface we will use a  three-term backward difference scheme:

LHS:  $\frac{\partial U(x=0, t)}{\partial x} = -e^{t} \longrightarrow    \frac{U_{i+1}^{m}(0,t)-U_{i-1}^{m}(0,t)}{2\Delta x^{m}} = e^{t_{m}}$, and

RHS: $\frac{\partial U(x=s(t), t)}{\partial x} = -\frac{d s(t)}{d t} \longrightarrow \frac{\partial U(x=s(t),t)}{\partial x} = \frac{3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m}}{2\Delta x^{m}}  $

from our LHS boundary condition expression, if we set $i=0$ and solve for $U_{-1}^{m}$ we get: $$U_{-1}^{m}=U_{1}^{m} + 2\Delta x^{m} e^{t_{m}}$$

We can now combine this expression and substitute into the governing equation to get expressions for diffusion at  $i=0, i=1$ to $(N-1)$, and $i=N$:

$U_{i}^{m+1}=(1-2\frac{\Delta t}{(\Delta x^{m})^{2}})U_{i}^{m} + 2\frac{\Delta t}{(\Delta x^{m})^{2}}U_{i+1}^{m} + (2\frac{\Delta t}{\Delta x^{m}}-\frac{\Delta t x_{i}^{m}\dot{s_{m}}}{s_{m}})e^{t_{m}}$

$U_{i}^{m+1}=U_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(U_{i+1}^{m}-U_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(U_{i+1}^{m}-2U_{i}^{m}+U_{i-1}^{m})$

$U_{i}^{m}=0$, $i=N$

Phew!! Now that is one impressive looking set of discretization expressions. But, sorry, we aren't done yet..... In the above expressions what are we supposed to do about $\dot{s_{m}}$ and $s_{m}$?

Well luckily, that is where the Stefan condition comes in, remember:

$\dot{s_{m}} = \frac{ds}{dt}= -\frac{\partial U_{@x=s}}{\partial x} = -\frac{3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m}}{2\Delta x^{m}} $, and as for $s_{m}$ that is just a Heat Balance equation:

$s_{m+1} = s_{m}+\dot{s_{m}}\Delta t = s_{m} - \frac{\Delta t}{2\Delta x^{m}}(3U_{N}^{m}-4U_{N-1}^{m}+U_{N-2}^{m})$, where $s_{0}=0$

To make our lives easier, instead of inserting these expressions for $\dot{s_{m}}$ and $s_{m}$ into our governing equations, lets keep them as coupled expressions to be calculated during the time loops and calculated at every grid point $i$.

lastly, lets not forget that the updated interface location $s_{m+1}$ and grid size $\Delta x^{m}$ are calculated at every timestep $m$, and have the relationship:

$$\Delta x^{m+1} = \frac{s_{m+1}}{N}$$

Don't worry, we are almost ready to code, I promise! There is just one more expression we need to consider, and that relates to Stability! Following the calculations of Caldwell and Savovic [5], for now, we will use something simple in order to limit the size of the timestep ($\Delta t$), that is: $$\frac{\Delta t}{(\Delta x^{m})^{2}} \leq 1$$, or

$$\Delta t \leq (\Delta x^{m})^{2}$$

## Let's Code!!

To summarize our governing equations we need to code these 7 coupled equations:

(A) $u_{i}^{m+1}=(1-2\frac{\Delta t}{(\Delta x^{m})^{2}})u_{i}^{m} + 2\frac{\Delta t}{(\Delta x^{m})^{2}}u_{i+1}^{m} + (2\frac{\Delta t}{\Delta x^{m}}-\frac{\Delta t x_{i}^{m}\dot{s_{m}}}{s_{m}})e^{t_{m}}$

(B) $u_{i}^{m+1}=u_{i}^{m}+\frac{\Delta t x_{i}^{m} \dot{s_{m}}}{2 \Delta x^{m}s_{m}}(u_{i+1}^{m}-u_{i-1}^{m}) + \frac{\Delta t}{(\Delta x^{m})^{2}}(u_{i+1}^{m}-2u_{i}^{m}+u_{i-1}^{m})$

(C) $u_{i}^{m}=0$, $i=N$

(D) $\dot{s_{m}} = \frac{ds}{dt}= -\frac{\partial u_{@x=s}}{\partial x} = -\frac{3u_{N}^{m}-4u_{N-1}^{m}+u_{N-2}^{m}}{2\Delta x^{m}} $

(E) $s_{m+1} = s_{m}+\dot{s_{m}}\Delta t = s_{m} - \frac{\Delta t}{2\Delta x^{m}}(3u_{N}^{m}-4u_{N-1}^{m}+u_{N-2}^{m}), s_{0}=0$

(F) $\Delta x^{m+1} = \frac{s_{m+1}}{N}$

(G) $\Delta t \leq (\Delta x^{m})^{2}$

```python
def VGM(nt, N, U0, dx, s, dt):

    Uo=0                      #Initial Temperature input
    U0 = numpy.ones(N)*(Uo)
    s0=0.02                   #Initial Interface Position
                              #(cannot chose zero or our expressions would blow up!)
    s=numpy.ones(nt)*(s0)
    dx=numpy.ones(nt)*(s[1]/N)
    sdot=(s[1]-s[0])/dt

    s[1]= s[0] - (dt/(2*dx[0]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])
    dx[0]=(s[1]/N)

    for m in range(0, nt-1):

        for i in range(0, N-1):

         #LHS BC (x=0, i=0)
            if(i==0):U0[i]= (1-2*(dt/dx[m]**2))*U0[i]+2*(dt/dx[m]**2)*U0[i+1] + (2*(dt/dx[m])-dt*(dx[m]*i)*sdot/s[m])*numpy.exp(m*dt)

        #Governing Equation (B)
            else: U0[i]= U0[i]+ ((dt*(dx[m]*i)*sdot)/(2*dx[m]*s[m]))*(U0[i+1]-U0[i-1]) + (dt/(dx[m]**2))*(U0[i+1]-2*U0[i]+U0[i-1])

        #RHS BC (x=L, L=dx{m*N})
            U0[N-1]=0

        s[m+1]= s[m] - (dt/(2*dx[m]))*(3*U0[N-1]-4*U0[N-2]+U0[N-3])         #Heat Balance Equation, (E)
        sdot=(s[m+1]-s[m])/dt                                                                                #Updating Speed of Interface
        dx[m+1]=(s[m+1]/N)                                                                                #Updating dx for each time step m

        if (U0[i]>0):
            Location=s[m]

    return U0, s, dx
```

```python
# First we set up some new initial parameters.These values differ from our first example since we are now dealing with
# a new non-dimensionalized heat equation, and new governing equations.

#1.0 Numerical Solution Parameters:
dt=2.0e-6        # Size of our Time-step, Set constant, Chosen in accordance with Caldwell and Savovic [5]
nt=500000      # Number of time steps, chosen so that elapsed solution will determine heat diffusion after
                       # (nt*dt= 1.0 seconds) Just like in our Analytical Solution.
N = 10            #  Number of spatial grid points, Chosen, for now, in accordance with Caldwell and Savovic [5]

```

```python
U_VGM, s_VGM, dx_VGM = VGM(nt, N, U0, dx, s, dt)
print('Initial Position of the interface (So):', s_VGM[0])
print('Initial X_step(dx):', dx_VGM[0])
print('Time Step (dt):', dt)
```

    Initial Position of the interface (So): 0.02
    Initial X_step(dx): 0.002
    Time Step (dt): 2e-06

### Results and Discussions

Ok! Now lets see what we get, and more importantly, if it makes any sense:

```python
XX = numpy.linspace(0, 1, N)
Max_tempVGM=max(U_VGM)

pyplot.plot(XX,U_VGM, color = '#654322', ls = '--', lw =4)
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)

print('This is our VGM temperature profile after:', nt*dt, 'seconds')
print('The temperature at our LHS boundary is:', Max_tempVGM, 'degrees')
print('Final position of our interface is:', s_VGM[m])
print('The final speed of our interface is:', sdot)
print('Our grid spaceing (dx) after', nt*dt, 'seconds is:', dx_VGM[m] )
```

Hey now! That looks pretty good! Let's compare this result to our earlier Analytical (exact) solution to the 1D Dimensionless problem, for 1 second into the diffusion:

```python
pyplot.figure(figsize=(12,10))
pyplot.plot(XX,U_VGM, color = '#654322', ls = '--',  lw =4, label = 'Variable Grid Method')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical Solution')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(0, 1.8)
pyplot.xlim(0, 1.0)
pyplot.legend();

print('Max error (@x=0) is:',abs(Max_tempVGM - Max_tempA)*(100/Max_tempA),'percent')
```

Under 2% error is pretty good! What else should we verify? How about the change in the interface location s(t) and the size of our spatial grid (dx) over time?

```python
Time=numpy.linspace(0,dt*nt,nt)
pyplot.figure(figsize=(10,6))

pyplot.plot(Time,dx_VGM, color = '#660033', ls = '--', lw =2, label='Spatial Grid Size')
pyplot.plot(Time,s_VGM,color = '#222222', ls = '-', lw =4, label='Interface Position, s(t)')
pyplot.xlabel('Time (Seconds)')
pyplot.ylabel('Length')
pyplot.legend();
```

and so we see that both the spatial grid size, dx, and the interface location, s(t), increase with time over the course of 1 second. This is exactly what we would expect! Infact, remember back to our anayltical solution, we solved the interface function  to be: $s(t)=t$, which is exactly the graph that our numerical solution has given us!

Well, then! That's a wrap! Looks like everything is all accounted for, right?. . . . . . . . . . . .  . .well, not exactly. There are still quite a few questions left unanswered, and now we must discuss the limitations in the Variable Grid Method of numerical implementation.

### Limitations of the Variable Grid Method

A number of questions should be coming to mind, among those are:

1) Why do we only analyze for 1 second of diffusion?

2) Why is your VGM timestep (dt) so small at 2.0e-6?

3) Why is your number of spatial grid points (N) only 10?

These are all VERY good questions, do you already know the answer? Here is a hint: it all has to do with stability!

You see, we have a relatively benign stability statement: $\Delta t \leq (\Delta x^{m})^{2}$, but still, it is a crutial aspect of our numerical analysis. This statement limits the size of our timestep, dt, and is the key reason why we chose our initial, constant, dt to be 2.0e-6 (lets not also forget the fact that this dt and N were also chosen by Caldwell and Savovic [5]) . Remember, because we are using a Central Difference scheme, our stability criteria essentially comes from the CFL condition:

$$\frac{\Delta t}{(\Delta x^{m})^{2}} \leq \sigma = 1$$.

The parameters dt and dx are not just arbitrarilty chosen. They determine the speed of your numerical solution. At all times you need the speed at which your solution progresses to be faster than the speed at which the problem (in this case, thermal diffusion) propagates. Afterall, how can you calculate the solution numerically when the solution is faster than the numerical analysis?!

Here is a pertinent question for you: Which do you determine first: dt, nt, N, or L? Does it matter which one you chose first?
The answer is normally, yes, we chose N and L which gives us dx, we then use a stability statement with dx (and $\alpha$) to limit dt, we then choose our nt to determine the elapsed time into our solution in the form of (nt*dt) seconds, easy right?

Well this is where we get to limitation #1 with the Variable Grid Method:

Because the end of our domain "L" is in actuality the position of our interface, s(t), and our interface at time t=0 is approximately at the origin (x=0.02), then if we want a constant number of spatial grids N (say N=10), our initial dx is extremely small!. Remember governing equation (F) says: $\Delta x^{m+1} = \frac{s_{m+1}}{N}$, well at t=0 (m=0) s[0] is 0.02, meaning our initial dx, dx[0] is: 0.002. Plug this back into our stability statement and you see that dt must be smaller than or equal to 4e-6! That is VERY harsh criterion for a timestep dt. In essence, with SUCH a small time step, we would need hundreds of thousands of time loops, nt, in order to get just 1 second of data. Now you see why nt is chosen to be 500,000 in order to get just 1 second of data. If I had wanted more time elapsed, we would have needed iterations in the millions. (This is also why the analytical solution was calculated only for 1 second, we wanted to be able to compare results at the end!)

Now limitation #2 of the Variable Grid Method: Because the interface starts at the origin, the initial spatial step is very small, and forces a very very small timestep, dt. Thus a very very large number of iterations is needed to get into the "Seconds" scale for simulations, this is a slow numerical process...

Not only this, but we have limitation #3 of the Variable Grid Method: If we want a longer dimensional domain, we need to choose a larger N. This will give us a final domain of $L=N*dx_{final}$, but if N is larger, then going back to our governing equation (F),  $\Delta x^{m+1} = \frac{s_{m+1}}{N}$ this means that dx[0] is even smaller, which continues to force a much smaller dt. Thus for the Variable Grid Method, if you want a large domain, you need an even smaller timestep, which forces more time in your numerical calculation.

Let's test this out for ourselves, lets redo the VGM calculations but increase N to say, 14:
(The cell below is in Raw-NB Convert since we didn't want it to slow you down the first time you ran this program. Notice the N is now 14, go ahead and run this below cell and see what happens)

```python
dt=2.0e-6
nt=500000

N = 14

U_VGM14, s_VGM14, dx_VGM14 = VGM(nt, N, U0, dx, s, dt)

XX = numpy.linspace(0, 1, N)
Max_tempVGM14=max(U_VGM14)
pyplot.plot(XX,U_VGM14, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx_VGM14[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM14-Max_tempA)*(100/Max_tempA),'Percent')
```

As you can see, our stability criteria is still met, since $\Delta t \leq (\Delta x_{0})^{2}$. Infact, with a 40% larger N we see that we have less error as well! But what if we set N=15?
(The cell below is in Raw-NB Convert since we didn't want it to slow you down the first time you ran this program. Notice the N is now 15, go ahead and run this below cell and see what happens THIS time)

```python
dt=2.0e-6
nt=500000

N = 15

U_VGM15, s_VGM15, dx_VGM15 = VGM(nt, N, U0, dx, s, dt)

XX = numpy.linspace(0, 1, N)
Max_tempVGM15=max(U_VGM15)
pyplot.plot(XX,U_VGM15, color = '#654322', ls = '--', lw =4, label = 'VGM')
pyplot.plot(x, ExactS, color = '#003366', ls = '--', lw =1, label='Analytical')
pyplot.xlabel('Length of Domain')
pyplot.ylabel('Temperature')
pyplot.ylim(-.5, 2)
pyplot.xlim(0, 1.0)
pyplot.legend();
print('Time Step dt is:', dt)
print('Initial spatial step, dx[0]^2 is:', dx_VGM15[0]**2 )
print('Max error (@x=0) is:',abs(Max_tempVGM15-Max_tempA)*(100/Max_tempA),'Percent')
```

Wow! That blew up! Just as we expect it would now that with N=15 our time step is larger than our initial spatial step!

#### Final Thoughts

And now we know, that the Stefan Problem, a boundary value PDE with  a time-dependant moving boundary, CAN be solved. We have just demonstrated the solution for a 1D, Dimensionless Stefan Problem with specifically chosen (time-dependant) input heat flux in order to give us a simplified exact solution to compare with. You have also seen implementation of the Variable Grid Method, one of many ways inwhich one can numerical simulate the Stefan problem. But we have also seen the limitations, namely in small time-steps ($dt$), smaller numerical domain ($N\cdot dx_{f}$), and large number of iterations ($nt$) for just a small amount of analytical time (t=$nt \cdot dt$ seconds). Perhaps when it comes time for you to model the melting of the Polar Ice caps, you'll choose a more "expedient" method? Just make sure you get the answer BEFORE the caps melt...)

EXERCISE #1: Can you implement the Variable Grid Method discretization governing equations to give us a solution faser? Can it handle millions of time iterations without putting us to sleep?

EXERCISE #2: Can you write a stability condition statement that will maximize our time-step (dt) for a given N and still keep it constant?

# References

1. Kutluay S., The numerical solution of one-phase classical Stefan problem, Journal of Computational and Applied Mathematics 81 (1997) 135-144

2. Javierre, E., A Comparison of Numeical Models for one-dimensional Stefan problems, Journal of Compuational and Applied Mathematics 192 (2006) 445-459

3. Vuik, C., "Some historical notes about the Stefan problem". Nieuw Archief voor Wiskunde, 4e serie 11 (2): 157-167 (1993)

4. Crowley, A. B., Numerical Solution of Stefan Problems, Brunel University, Department of Mathematics, TR/69 December 1976

5. Caldwell, J., Numerical Solution of Stefan Problem By Variable Space Grid Method and Boundary Immodbilisation Method, Jour. of Mathematical Sciences, Vol. 13 No.1 (2002) 67-79.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file ='numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>


## Markov Networks

author: Jacob Schreiber <br>
contact: jmschreiber91@gmail.com

Markov networks are probabilistic models that are usually represented as an undirected graph, where the nodes represent variables and the edges represent associations. Markov networks are similar to Bayesian networks with the primary difference being that Bayesian networks can be represented as directed graphs with known parental relations. Generally, Bayesian networks are easier to interpret and can be used to calculate probabilities faster but, naturally, require that causality is known. However, in many settings, one can only know the associations between variables and not necessarily the direction of causality.

The underlying implementation of inference in pomegranate for both Markov networks and Bayesian networks is the same, because both get converted to their factor graph representations. However, there are many important differences between the two models that should be considered before choosing one.

In this tutorial we will go over how to use Markov networks in pomegranate and what some of the current limitations of the implementation are.

```python
%matplotlib inline
import numpy
import itertools

from pomegranate import *

numpy.random.seed(0)
numpy.set_printoptions(suppress=True)

%load_ext watermark
%watermark -m -n -p numpy,scipy,pomegranate
```

    Mon Dec 02 2019

    numpy 1.17.2
    scipy 1.3.1
    pomegranate 0.11.3

    compiler   : GCC 7.3.0
    system     : Linux
    release    : 4.15.0-66-generic
    machine    : x86_64
    processor  : x86_64
    CPU cores  : 8
    interpreter: 64bit

### Defining a Markov Network

A Markov network is defined by passing in a list of the joint probability tables associated with a series of cliques rather than an explicit graph structure. This is because the probability distributions for a particular variable are not defined by themselves, but rather through associations with other variables. While a Bayesian network has root variables that do not hav e parents, the undirected nature of the edges in a Markov networks means that variables are generally grouped together.

Let's define a simple Markov network where the cliques are A-B, B-C-D, and C-D-E. B-C-D-E is almost a clique but are missing the connection between B and E.

```python
d1 = JointProbabilityTable([
    [0, 0, 0.1],
    [0, 1, 0.2],
    [1, 0, 0.4],
    [1, 1, 0.3]], [0, 1])

d2 = JointProbabilityTable([
    [0, 0, 0, 0.05],
    [0, 0, 1, 0.15],
    [0, 1, 0, 0.07],
    [0, 1, 1, 0.03],
    [1, 0, 0, 0.12],
    [1, 0, 1, 0.18],
    [1, 1, 0, 0.10],
    [1, 1, 1, 0.30]], [1, 2, 3])

d3 = JointProbabilityTable([
    [0, 0, 0, 0.08],
    [0, 0, 1, 0.12],
    [0, 1, 0, 0.11],
    [0, 1, 1, 0.19],
    [1, 0, 0, 0.04],
    [1, 0, 1, 0.06],
    [1, 1, 0, 0.23],
    [1, 1, 1, 0.17]], [2, 3, 4])

model = MarkovNetwork([d1, d2, d3])
model.bake()
```

We can see that the initialization is fairly straightforward. An important note is that the JointProbabilityTable object requires as the second argument a list of variables that are included in that clique in the order that they appear in the table, from left to right.

### Calculating the probability of examples

Similar to the other probabilistic models in pomegranate, Markov networks can be used to calculate the probability or log probability of examples. However, unlike the other models, calculating the log probability for Markov networks is generally computationally intractable for data with even a modest number of variables (~30).

The process for calculating the log probability begins by calculating the "unnormalized" log probability $\hat{P}$, which is just the product of the probabilities for the variables $c$ in each clique $c \in C$ under their joint probability table $JPT(c)$. This step is easy because it just involves, for each clique, taking the columns corresponding to that clique and performing a table lookup.

\begin{equation}
\hat{P}(X=x) = \prod\limits_{c \in C} JPT(c)
\end{equation}

The reason this is called the unnormalized probability is because the sum of all combinations of variables that $X$ can take $\sum\limits_{x \in X} \hat{P}(X=x)$ does not sum to 1; thus, it is not a true probability distribution.

We can calculate the normalized probability $P(X)$ by dividing by the sum of probabilities under all combinations of variables, frequently referred to as the "partition function". Calculating the partition function $Z$ is as simple as summing the unnormalized probabilities over all possible combinations of variables $x \in X$.

\begin{equation}
Z = \sum\limits_{x \in X} \hat{P}(X=x)
\end{equation}

Finally, we can divide any unnormalized probability calculation by the partition function to get the correct probability.

\begin{equation}
P(X = x) = \frac{1}{Z} \hat{P}(X = x)
\end{equation}

The `probability` method returns the normalized probability value. We can check this by seeing that it is different than simply passing the columns of data in to the distributions for their respective cliques.

```python
model.probability([0, 1, 0, 0, 1])
```

    0.020425530968183916

And the probability if we simply passed the columns into the corresponding cliques:

```python
d1.probability([0, 1]) * d2.probability([1, 0, 0]) * d3.probability([0, 0, 1])
```

    0.0028800000000000006

However, by passing the `unnormalized=True` parameter in to the `probability` method we can return the unnormalized probability values.

```python
model.probability([0, 1, 0, 0, 1], unnormalized=True)
```

    0.0028799999999999997

We can see that the two are identical, subject to machine precision.

### Calculating the partition function

Calculating the partition function involves summing the unnormalized probabilities of all combinations of variables that an example can take. Unfortunately, the time it takes to calculate the partition function grows exponentially with the number of dimensions. This means that it may not be able to calculate the partition function exactly for more than ~25 variables, depending on your machine. While pomegranate does not currently support any methods for calculating the partition function other than the exact method, it is flexible enough to allow users to get around this limitation.

The partition function itself is calculated in the `bake` method because, at that point, all combinations of variables are known to the model. This value is then cached so that calls to `probability` or `log_probability` are just as fast regardless of if the normalized or unnormalized probabilities are calculated. However, if the user passes in `calculate_partition=False` the model will not spend time calculating the partition function. We can see the difference in time here:

```python
X = numpy.random.randint(2, size=(100, 14))
model2 = MarkovNetwork.from_samples(X)

%timeit model2.bake()
%timeit model2.bake(calculate_partition=False)
```

    2.67 s ± 50.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    1.06 ms ± 31.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

There are two main reasons that one might not want to calculate the partition function when creating the model. The first is when the user will only be inspecting the model, such as after structure learning, or only doing inference, which uses the approximate loopy belief propogation. The second is if the user wants to estimate the partition function themselves using an approximate algorithm.

Let's look at how one would manually calculate the exact partition function to see how an approximate algorithm could be substituted in. First, what happens if we don't calculate the partition but try to calculate probabilities?

```python
model.bake(calculate_partition=False)
model.probability([0, 1, 0, 0, 1])
```

Looks like we get an error. We can still calculate unnormalized probabilities though.

```python
model.probability([0, 1, 0, 0, 1], unnormalized=True)
```

    0.0028799999999999997

Now we can calculate the partition function by calculating the unnormalized probability of all combinations.

```python
Z = model.probability(list(itertools.product(*model.keys_)), unnormalized=True).sum()
Z
```

    0.141

We can set the stored partition function in the model to be this value (or specifically the log partition function) and then calculate normalized probabilities as before.

```python
model.partition = numpy.log(Z)
model.probability([0, 1, 0, 0, 1])
```

    0.020425530968183916

Now you can calculate $Z$ however you'd like and simply plug it in.

### Inference

Similar to Bayesian networks, Markov networks can be used to infer missing values in data sets. In pomegranate, inference is done for Markov networks by converting them to their factor graph representations and then using loopy belief propagation. This results in fast, but approximate, inference.

```python
model.predict([[None, 1, 0, None, None], [None, None, 1, None, 0]])
```

    [array([1, 1, 0, 1, 1], dtype=object), array([1, 1, 1, 1, 0], dtype=object)]

If we go back and inspect the joint probability tables we can easily see that the inferred values are the correct ones.

If we want the full probability distribution rather than just the most likely value we can use the `predict_proba` method.

```python
model.predict_proba([[None, None, 1, None, 0]])
```

    [array([{
         "class" :"Distribution",
         "dtype" :"int",
         "name" :"DiscreteDistribution",
         "parameters" :[
             {
                 "0" :0.37654171704957684,
                 "1" :0.623458282950423
             }
         ],
         "frozen" :false
     },
            {
         "class" :"Distribution",
         "dtype" :"int",
         "name" :"DiscreteDistribution",
         "parameters" :[
             {
                 "0" :0.11729141475211632,
                 "1" :0.8827085852478838
             }
         ],
         "frozen" :false
     },
            1,
            {
         "class" :"Distribution",
         "dtype" :"int",
         "name" :"DiscreteDistribution",
         "parameters" :[
             {
                 "0" :0.08222490931076197,
                 "1" :0.917775090689238
             }
         ],
         "frozen" :false
     },
            0], dtype=object)]

### Structure Learning

Markov networks can be learned from data just as Bayesian networks can. While there are many algorithms that have been proposed for Bayesian network structure learning, there are fewer for Markov networks. Currently, pomegranate only supports the Chow-Liu tree-building algorithm for learning a tree-like network. Let's see a simple example where we learn a Markov network over a chunk of features from the digits data set.

```python
from sklearn.datasets import load_digits

X = load_digits().data[:, 22:40]
X = (X > numpy.median(X)).astype(int)

model = MarkovNetwork.from_samples(X)
icd = IndependentComponentsDistribution.from_samples(X, distributions=DiscreteDistribution)

model.log_probability(X).sum(), icd.log_probability(X).sum()
```

    (-12699.71644426486, -13141.625038482453)

It looks like the Markov network is somewhat better than simply modeling each pixel individually for this set of features.


```python
import import_ipynb
from Data_Analysis_1 import *
```

    importing Jupyter notebook from Data_Analysis_1.ipynb

# $e^x$

```python
from sympy import *
import numpy as np
x = Symbol('x')
y = exp(x)
# yprime = y.diff(x)
def factorial(i):
    if i == 1 or i == 0:
        return 1
    else:
        return i*factorial(i-1)
def Taylor(point_of_expansion, order_of_polynomial):
    x = Symbol('x')
    y = exp(x)
    x_0 = point_of_expansion
    i = order_of_polynomial
    polinomio = 0
    for j in range(0, i+1):
        f_1 = lambdify(x, y)
        y_0 = f_1(x_0)
        polinomio += pow((x-x_0),j)*Rational(y_0, factorial(j))
        y = y.diff(x)
    return polinomio
```

```python
polinomio_1 = Taylor(0, 1)
f = lambdify(x, polinomio_1, 'numpy')
xaxis = np.linspace(-2, 2, 100)
yaxis_1 = f(xaxis)
```

```python
polinomio_2 = Taylor(0, 2)
f = lambdify(x, polinomio_2, 'numpy')
yaxis_2 = f(xaxis)
```

```python
polinomio_3 = Taylor(0, 3)
f = lambdify(x, polinomio_3, 'numpy')
yaxis_3 = f(xaxis)
```

```python
polinomio_4 = Taylor(0, 4)
f = lambdify(x, polinomio_4, 'numpy')
yaxis_4 = f(xaxis)
```

```python
polinomio_5 = Taylor(0, 5)
f = lambdify(x, polinomio_5, 'numpy')
yaxis_5 = f(xaxis)
```

```python
polinomio_6 = Taylor(0, 6)
f = lambdify(x, polinomio_6, 'numpy')
yaxis_6 = f(xaxis)
```

```python
layout = Layout()
traccia_1 = traccia_relazione(yaxis_1, None, xaxis, None, "lines", "Taylor I")
traccia_2 = traccia_relazione(yaxis_2, None, xaxis, None, "lines", "Taylor II")
traccia_3 = traccia_relazione(yaxis_3, None, xaxis, None, "lines", "Taylor III")
traccia_4 = traccia_relazione(yaxis_4, None, xaxis, None, "lines", "Taylor IV")
traccia_5 = traccia_relazione(yaxis_5, None, xaxis, None, "lines", "Taylor V")
traccia_6 = traccia_relazione(yaxis_6, None, xaxis, None, "lines", "Taylor VI")
```

```python
e = exp(x)
f_real_exp = lambdify(x, e, 'numpy')
yaxis_prime = f_real_exp(xaxis)
traccia_e = traccia_relazione(yaxis_prime, None ,xaxis, None, "lines", "e^x")
```

```python
data = [traccia_1, traccia_2, traccia_3, traccia_4, traccia_5, traccia_6, traccia_e, X(-2, 2), Y(-2, 7.4)]
Grafico(data, layout)
```

# $log(x)$

```python
x = Symbol('x')
y = log(x)
def Taylor(point_of_expansion, order_of_polynomial):
    x = Symbol('x')
    y = log(x)
    x_0 = point_of_expansion
    i = order_of_polynomial
    polinomio = 0
    for j in range(0, i+1):
        f_1 = lambdify(x, y)
        y_0 = f_1(x_0)
        polinomio += pow((x-x_0),j)*Rational(y_0, factorial(j))
        y = y.diff(x)
    return polinomio
```

```python
polinomio_1 = Taylor(1, 1)
f = lambdify(x, polinomio_1, 'numpy')
xaxis = np.linspace(0.5, 2, 100)
yaxis_1 = f(xaxis)
```

```python
polinomio_2 = Taylor(1, 2)
f = lambdify(x, polinomio_2, 'numpy')
yaxis_2 = f(xaxis)
```

```python
polinomio_3 = Taylor(1, 3)
f = lambdify(x, polinomio_3, 'numpy')
yaxis_3 = f(xaxis)
```

```python
polinomio_4 = Taylor(1, 4)
f = lambdify(x, polinomio_4, 'numpy')
yaxis_4 = f(xaxis)
```

```python
polinomio_5 = Taylor(1, 5)
f = lambdify(x, polinomio_5, 'numpy')
yaxis_5 = f(xaxis)
```

```python
polinomio_6 = Taylor(1, 6)
f = lambdify(x, polinomio_6, 'numpy')
yaxis_6 = f(xaxis)
```

```python
layout = Layout()
traccia_1 = traccia_relazione(yaxis_1, None, xaxis, None, "lines", "Taylor I")
traccia_2 = traccia_relazione(yaxis_2, None, xaxis, None, "lines", "Taylor II")
traccia_3 = traccia_relazione(yaxis_3, None, xaxis, None, "lines", "Taylor III")
traccia_4 = traccia_relazione(yaxis_4, None, xaxis, None, "lines", "Taylor IV")
traccia_5 = traccia_relazione(yaxis_5, None, xaxis, None, "lines", "Taylor V")
traccia_6 = traccia_relazione(yaxis_6, None, xaxis, None, "lines", "Taylor VI")
```

```python
logarithm = log(x)
f_real_log = lambdify(x, logarithm, 'numpy')
yaxis_prime = f_real_log(xaxis)
traccia_log = traccia_relazione(yaxis_prime, None ,xaxis, None, "lines", "log(x)")
```

```python
data = [traccia_1, traccia_2, traccia_3, traccia_4, traccia_5, traccia_6, traccia_log, X(0, 2), Y(-0.7, 1)]
Grafico(data, layout)
```


# Trust Simulation

### initial settings and classes

```python
# modules
import random
import sympy
from sympy import *
import numpy as np
import matplotlib.pyplot as plt

# for nice printing if we want to do symbolic stuff
# init_printing(use_unicode=True)
# N, M, beta, mu, nu, c, r, T, b = symbols('N, M, beta, mu, nu, c, r, T, b')

# initial settings
N = 2  # number of players
M = 2  # number of groups
beta = .1  # intensity of selector
mu = .0001  # probability that we mutate
nu = .01  # probability that we migrate
c = 1 # cash
r = .5 # return
T = 100 # total time steps
b = .25 # bin size

# classes
class Player:
    random.seed()
    p = random.uniform(0, 1)  # proportion invested in players from the same group
    q = random.uniform(0, 1)  # proportion returned to players from the same group
    g = random.randint(1, M)  # group

# list of all players
P = [Player() for i in range(N)]
```

### some preliminary functions

```python
# input: P1, P2, 2 players in sequence
# output: payoff for P1 interacting with P2
def interaction(P1, P2):
    payoff = c-P1.p  # initial investment in P2
    payoff += P1.p*r*P2.q  # gain for having invested in P2
    payoff += P2.p*r*(1-P1.q)  # gain by not returning investment to P2
    return payoff

# input: 1 player
# output: the player's fitness
def fitness(P1):
    payoff = 0
    for P2 in P:
        if P2.g == P1.g and P2 != P1:
            payoff += interaction(P1, P2)
    return np.exp(beta*payoff)
```

### iteration

```python
# compute total fitness
f = 0
for P1 in P:
    f += fitness(P1)

# compute scaled fitness for each player
F = [fitness(P[i])/f for i in range(N)]

```

### plots


```
import sympy as sy
from dolfin import *
import numpy as np

def polarx(u, rho, phi):
    return sy.cos(phi)*sy.diff(u, rho) - (1./rho)*sy.sin(phi)*sy.diff(u, phi)

def polary(u, rho, phi):
    return sy.sin(phi)*sy.diff(u, rho) + (1./rho)*sy.cos(phi)*sy.diff(u, phi)

def polarr(u, x, y):
    return (1./sqrt(x**2 + y**2))*(x*sy.diff(u,x)+y*sy.diff(u,y))

def polart(u, x, y):
    return -y*sy.diff(u,x)+x*sy.diff(u,y)

l = 0.54448373678246
omega = (3./2)*np.pi

z = sy.symbols('z')

x = sy.symbols('x[0]')
y = sy.symbols('x[1]')
rho = sy.sqrt(x**2 + y**2)
phi = sy.atan2(y,x)

# looked at all the exact solutions and they seems to be the same as the paper.....
psi = (sy.sin((1+l)*phi)*sy.cos(l*omega))/(1+l) - sy.cos((1+l)*phi) - (sy.sin((1-l)*phi)*sy.cos(l*omega))/(1-l) + sy.cos((1-l)*phi)

psi_prime = polart(psi, x, y)
psi_3prime = polart(polart(psi_prime, x, y), x, y)

u = rho**l*((1+l)*sy.sin(phi)*psi + sy.cos(phi)*psi_prime)
v = rho**l*(-(1+l)*sy.cos(phi)*psi + sy.sin(phi)*psi_prime)

uu0 = Expression((sy.ccode(u),sy.ccode(v)))
ub0 = Expression((str(sy.ccode(u)).replace('atan2(x[1], x[0])','(atan2(x[1], x[0])+2*pi)'),str(sy.ccode(v)).replace('atan2(x[1], x[0])','(atan2(x[1], x[0])+2*pi)')))

p = -rho**(l-1)*((1+l)**2*psi_prime + psi_3prime)/(1-l)
pu0 = Expression(sy.ccode(p))
pb0 = Expression(str(sy.ccode(p)).replace('atan2(x[1], x[0])','(atan2(x[1], x[0])+2*pi)'))

f = rho**(2./3)*sy.sin((2./3)*phi)
b = sy.diff(f,x)
d = sy.diff(f,y)
bu0 = Expression((sy.ccode(b),sy.ccode(d)))
bb0 = Expression((str(sy.ccode(b)).replace('atan2(x[1], x[0])','(atan2(x[1], x[0])+2*pi)'),str(sy.ccode(d)).replace('atan2(x[1], x[0])','(atan2(x[1], x[0])+2*pi)')))

r = sy.diff(x,y)
ru0 = Expression(sy.ccode(r))

#Laplacian
L1 = sy.diff(u,x,x)+sy.diff(u,y,y)
L2 = sy.diff(v,x,x)+sy.diff(v,y,y)

A1 = u*sy.diff(u,x)+v*sy.diff(u,y)
A2 = u*sy.diff(v,x)+v*sy.diff(v,y)

P1 = sy.diff(p,x)
P2 = sy.diff(p,y)

# Curl-curl
C1 = sy.diff(d,x,y) - sy.diff(b,y,y)
C2 = sy.diff(b,x,y) - sy.diff(d,x,x)

R1 = sy.diff(r,x)
R2 = sy.diff(r,y)

NS1 = -d*(sy.diff(d,x)-sy.diff(b,y))
NS2 = b*(sy.diff(d,x)-sy.diff(b,y))

M1 = sy.diff(u*d-v*b,y)
M2 = -sy.diff(u*d-v*b,x)

```

```
velocity = open('Solution/velocity.t','w')
magnetic = open('Solution/magnetic.t','w')
pressure = open('Solution/pressure.t','w')
advection = open('Solution/advection.t','w')
laplacian = open('Solution/laplacian.t','w')
pressureGrad = open('Solution/pressureGrad.t','w')
curlcurl = open('Solution/curlcurl.t','w')
Mcouple = open('Solution/Mcouple.t','w')
NScouple = open('Solution/NScouple.t','w')

u = velocity.write(str(sy.ccode(u)))
v = velocity.write(str(sy.ccode(v)))
b = magnetic.write(str(sy.ccode(b)))
d = magnetic.write(str(sy.ccode(d)))
p = pressure.write(str(sy.ccode(p)))

L1 = laplacian.write(str(sy.ccode(L1)))
L2 = laplacian.write(str(sy.ccode(L2)))
A1 = advection.write(str(sy.ccode(A1)))
A2 = advection.write(str(sy.ccode(A2)))
P1 = pressureGrad.write(str(sy.ccode(P1)))
P2 = pressureGrad.write(str(sy.ccode(P2)))

C1 = curlcurl.write(str(sy.ccode(C1)))
C2 = curlcurl.write(str(sy.ccode(C2)))
M1 = Mcouple.write(str(sy.ccode(M1)))
M2 = Mcouple.write(str(sy.ccode(M2)))
NS1 = NScouple.write(str(sy.ccode(NS1)))
NS2 = NScouple.write(str(sy.ccode(NS2)))
```

```
V1.readline()
```

    ''

```

```

```

```

```

```


Computational Seismology

Finite Volume Method - 1D Advection Equation

Seismo-Live: http://seismo-live.org

##### Authors:
* David Vargas ([@dvargas](https://github.com/davofis))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))

## Basic Equations
We want to solve the linear advection equation as a scalar hyperbolic equation:

\begin{equation}
\partial_t q + a \partial_x q = 0
\end{equation}

This equation arises from conservation principles considering the case of advective fluxes. Let us partitioned the space in $n$ cells centered at $x = x_i$, the average of the quantity $q$ inside the cell is

\begin{equation}
Q_{i}^{n} = \int_{\mathscr{c}} q(x,t) dx
\end{equation}

Any change inside each cell implies a net flux through the cell boundaries, then

\begin{equation}
\frac{Q_{i}^{n+1} - Q_{i}^{n}}{dt} = \frac{F_{i-1/2}^{n} - F_{i+1/2}^{n}}{dx}
\end{equation}

in an advection problem, the flux terms simply are

\begin{equation}
F_{i-1/2}^{n} =  a Q_{i-1}^{n}
\qquad\text{and}\qquad
F_{i+1/2}^{n} =  a Q_{i}^{n}
\end{equation}

Using the above definitions we obtain a complete discrete extrapolation scheme as

\begin{equation}
Q_{i}^{n+1}  = Q_{i}^{n} + a \frac{dt}{dx}(Q_{i-1}^{n} - Q_{i}^{n})
\end{equation}

This is the so call upwind scheme. Alternatively, one uses the Taylor expansion to extrapolate $Q(x, t)$ in time. Successive differentiation of the advection equation allows us to express time derivatives in terms of space derivatives. Finally, finite differences are used to approximate space derivatives leading to the Lax-Wendroff Scheme.

\begin{equation}
Q_{i}^{n+1}  = Q_{i}^{n} - \frac{adt}{2dx}(Q_{i+1}^{n} - Q_{i-1}^{n}) + \frac{1}{2}(\frac{adt}{dx})^2(Q_{i+1}^{n} - 2Q_{i}^{n} + Q_{i-1}^{n})
\end{equation}

This notebook implements both upwind and Lax-Wendroff schemes for solving the scalar advection equation. To keep the problem simple we use as spatial initial condition a Gauss function with half-width $\sigma$

\begin{equation}
Q(x,t=0)  = e^{-1/\sigma^2 (x - x_{o})^2}
\end{equation}

```python
# Import all necessary libraries, this is a configuration step for the exercise.
# Please run it before the simulation code!
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation

# Show the plots in the Notebook.
plt.switch_backend("nbagg")
```

### 1. Initialization of setup

```python
# Initialization of setup
# --------------------------------------------------------------------------
nx     = 2000       # number of grid points
xmax   = 8000       # in m
c      = 2500       # Advected speed
eps    = 0.5        # CFL
tmax   = 2          # simulation time in s
sig    = 200        # Gaussian width, in m
x0     = 1000       # Gaussian position, in m
method = 'Lax-Wendroff'   # 'Lax-Wendroff', 'upwind'
isnap  = 10
```

### 2. Finite Volumes setup

```python
# Initialize Space
x  = np.linspace(0,xmax,nx)
dx = min(np.diff(x))

# use wave based CFL criterion
dt = eps*dx/c    # calculate tim step from stability criterion

# Simulation time
nt = int(np.floor(tmax/dt))

# Initialize shape of fields
Q   = np.zeros(nx)
dQ  = np.zeros(nx)
dQ1 = np.zeros(nx)
dQ2 = np.zeros(nx)
Qa  = np.zeros(nx)
```

### 3. Initial condition

```python
# Spatial initial condition
#---------------------------------------------------------------
sx = np.exp(-1./sig**2 * (x-x0)**2)

# Set Initial condition
Q = sx

# ---------------------------------------------------------------
# Plot initial condition
# ---------------------------------------------------------------
plt.plot(x, sx, color='b', lw=2, label='Initial condition')
plt.ylabel('Amplitude', size=16)
plt.xlabel('x', size=16)
plt.legend()
plt.grid(True)
plt.show()
```

### 4. Solution for the scalar advection problem

```python
# ---------------------------------------------------------------
# Initialize animated plot
# ---------------------------------------------------------------
fig = plt.figure(figsize=(12,6))
line = plt.plot(x, Q, 'k', x, Qa, 'r--')
plt.ylabel('Amplitude')
plt.xlabel('x')
plt.title('Scalar Advection - %s method'%method, size=16)

plt.ion()    # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for i in range(nt):
    # upwind method
    if method == 'upwind':
        for j in range(1, nx-1):
            # Forward (upwind) (c>0)
            dQ[j] = Q[j] - Q[j-1]
        # Time extrapolation
        Q = Q - dt/dx*c*dQ

    # Lax wendroff method
    if method == 'Lax-Wendroff':
        for j in range(1, nx-1):
            # Forward (upwind) (c>0)
            dQ1[j] = Q[j+1] - 2*Q[j] + Q[j-1]
            dQ2[j] = Q[j+1] - Q[j-1]
        # Time extrapolation
        Q = Q - 0.5*c*dt/dx*dQ2 + 0.5*(c*dt/dx)**2 *dQ1

    # Boundary condition
    Q[0] = Q[nx-2]    # Periodic
    Q[nx-1] = Q[nx-2] # Absorbing

    # --------------------------------------
    # Animation plot. Display solution
    if not i % isnap:
        for l in line:
            l.remove()
            del l
        # --------------------------------------
        # Analytical solution
        xd = c*i*dt+x0
        Qa = np.exp(-1./sig**2 * (x - xd)**2)

        # --------------------------------------
        # Display lines
        line = plt.plot(x, Q, 'k', x, Qa, 'r--', lw=1.5)
        plt.legend(iter(line), ('F. Volume', 'Analytic'))
        plt.gcf().canvas.draw()

```

```python

```


## Subsurface Data Analytics

### Spatial Bootstrap with the Number of Effective Data

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Exercise: Spatial Bootstrap for Subsurface Data Analytics in Python

Here's a simple workflow, demonstration of spatial bootstrap for subsurface modeling workflows. This should help you get started with building subsurface models that integrate uncertainty in the sample statistics.

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The **Bootstrap Approach** (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.

#### Spatial Bootstrap

Journel (1993) developed methods for spatial bootstrap based on bootstrap resampling accounting for the locations of the data and the spatial continuity model.

* One method to perform spatial bootstrap for uncertainty of a statistic is to adjust the number of data, $n$, to the **number of effective data**, then use the number of effective data instead of number of data of resamples with replacement for each bootstrap realization.

This number of effectve data may be calculated by:

* building multiple unconditional simulated realizations at the data locations only

* calculating the variance of the average of each simulated realization

* then calculate the number effective data by manipulating the standard error in the average equation:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2}{n}
\end{equation}

to be expressed as:

\begin{equation}
n^{'} = \frac{\sigma^2}{\sigma^2_\overline{x}}
\end{equation}

where $\sigma^2_\overline{x}$ is the variance of the average over bootstrap realizations $\ell = 1,\ldots,L$, $\sigma^2$ is the variance / sill of the problem.

#### Simulating Only at the Data Locations

We ultilize LU Simulation to efficiently calculate Gaussian realizations only at the data locations, based on the Lower-Upper Decomposition of the left-hand / redudancy martix of the simple kriging.

* We multiple the Lower matrix ($n \times n$) with a random Gaussian vector ($1 \times n$) to calculate each realization at the data locations.

* We then take the average of each realization and then calculate the variance of the average over enough realizations.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.geostats as geostats
import geostatspy.GSLIB as GSLIB
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
import scipy                              # statistics
import scipy.linalg                       # linear algebra library
import math                               # trig, etc.
import random                             # bootstrap
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
 cmap = plt.cm.inferno
```

##### Declare Functions

I have included the n_effective function and dependencies here.  I will add this shortly to the GeostatsPy Package.

```python
from numba import jit # for numerical speed up

@jit(nopython=True)
def cova2(x1, y1, x2, y2, nst, c0, pmx, cc, aa, it, ang, anis, rotmat, maxcov):
    """Calculate the covariance associated with a variogram model specified by a
    nugget effect and nested variogram structures.
    :param x1: x coordinate of first point
    :param y1: y coordinate of first point
    :param x2: x coordinate of second point
    :param y2: y coordinate of second point
    :param nst: number of nested structures (maximum of 4)
    :param c0: isotropic nugget constant (TODO: not used)
    :param pmx: TODO
    :param cc: multiplicative factor of each nested structure
    :param aa: parameter `a` of each nested structure
    :param it: TODO
    :param ang: TODO: not used
    :param anis: TODO
    :param rotmat: rotation matrices
    :param maxcov: TODO
    :return: TODO
    """
    EPSLON = 0.000001

    # Check for very small distance
    dx = x2 - x1
    dy = y2 - y1

    if (dx * dx + dy * dy) < EPSLON:
        cova2_ = maxcov
        return cova2_

    # Non-zero distance, loop over all the structures
    cova2_ = 0.0
    for js in range(0, nst):
        # Compute the appropriate structural distance
        dx1 = dx * rotmat[0, js] + dy * rotmat[1, js]
        dy1 = (dx * rotmat[2, js] + dy * rotmat[3, js]) / anis[js]
        h = math.sqrt(max((dx1 * dx1 + dy1 * dy1), 0.0))
        if it[js] == 1:
            # Spherical model
            hr = h / aa[js]
            if hr < 1.0:
                cova2_ = cova2_ + cc[js] * (1.0 - hr * (1.5 - 0.5 * hr * hr))
        elif it[js] == 2:
            # Exponential model
            cova2_ = cova2_ + cc[js] * np.exp(-3.0 * h / aa[js])
        elif it[js] == 3:
            # Gaussian model
            hh = -3.0 * (h * h) / (aa[js] * aa[js])
            cova2_ = cova2_ + cc[js] * np.exp(hh)
        elif it[js] == 4:
            # Power model
            cov1 = pmx - cc[js] * (h ** aa[js])
            cova2_ = cova2_ + cov1
    return cova2_

@jit(nopython=True)
def setup_rotmat(c0, nst, it, cc, ang, pmx):
    """Setup rotation matrix.
    :param c0: nugget constant (isotropic)
    :param nst: number of nested structures (max. 4)
    :param it: TODO
    :param cc: multiplicative factor of each nested structure
    :param ang: TODO
    :param pmx: TODO
    :return: TODO
    """
    PI = 3.141_592_65
    DTOR = PI / 180.0

    # The first time around, re-initialize the cosine matrix for the variogram
    # structures
    rotmat = np.zeros((4, nst))
    maxcov = c0
    for js in range(0, nst):
        azmuth = (90.0 - ang[js]) * DTOR
        rotmat[0, js] = math.cos(azmuth)
        rotmat[1, js] = math.sin(azmuth)
        rotmat[2, js] = -1 * math.sin(azmuth)
        rotmat[3, js] = math.cos(azmuth)
        if it[js] == 4:
            maxcov = maxcov + pmx
        else:
            maxcov = maxcov + cc[js]
    return rotmat, maxcov

def n_effective(df,xcol,ycol,seed,nreal,vario):
    """Calculate the number of effective data from spatial locations and spatial continuity model
    Used in bootstrap to account for spatial continuity, use n effective instead of number of data
    :param df: source DataFrame
    :param xcol: column with the X locations
    :param ycol: column with the Y locations
    :param seed: random number seed for the random sampling
    :param nreal: number of realizations to sample the variance of the average
    :param vario: variogram model as a dictionary, see the GeostatsPy Package's GSLIB.make_variogram() function
    :return: n_eff as effective number of data
    """

# Set constants
    np.random.seed(seed)
    PMX = 9999.9

# load the variogram
    nst = vario['nst']
    cc = np.zeros(nst); aa = np.zeros(nst); it = np.zeros(nst)
    ang = np.zeros(nst); anis = np.zeros(nst)
    c0 = vario['nug'];
    cc[0] = vario['cc1']; it[0] = vario['it1']; ang[0] = vario['azi1'];
    aa[0] = vario['hmaj1']; anis[0] = vario['hmin1']/vario['hmaj1'];
    if nst == 2:                                   # include 2nd structure if present (optional)
        cc[1] = vario['cc2']; it[1] = vario['it2']; ang[1] = vario['azi2'];
        aa[1] = vario['hmaj2']; anis[1] = vario['hmin2']/vario['hmaj2'];

# Set up the rotation matrix
    rotmat, maxcov = setup_rotmat(c0,nst,it,cc,ang,PMX)

# Load the data
    nd = len(df)
    x = df[xcol].values
    y = df[ycol].values

# Calculate Symmetric Covariance Array - assuming variogram with spherical structure with range specified
    cov = np.zeros((nd,nd))
    var_range = 100.0
    for i in range(0, nd):
        x1 = x[i]; y1 = y[i]
        for j in range(0, nd):
            x2 = x[j]; y2 = y[j]
            cova = cova2(x1, y1, x2, y2, nst, c0, PMX, cc, aa, it, ang, anis, rotmat, maxcov)
            cov[i,j] = cova

# Lower and upper deconvolution
    P, L, U = scipy.linalg.lu(cov)

# Build realization and calculate the average
    average_array = np.zeros(nreal)
    rand = np.zeros((nd))
    for l in range(0, nreal):
        rand = np.random.normal(loc = 0.0, scale = 1.0, size = nd)
        realization = np.matmul(L,rand)
        average_array[l] = np.average(realization)

# Back out the number of effecitve data useing the standard error in the average
    var_average = np.var(average_array)
    n_eff = max(min(1.0/var_average, nd),1.0)    # filter n effective less than 1.0 or greater than number of data

    return n_eff

```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

* Add a null feature for plotting the data locations, data values are not needed in this calculation / workflow for number of effective data.

```python
df = pd.read_csv("12_sample_data.csv")                      # read a .csv file in as a DataFrame
df['null'] = np.zeros(len(df))
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

* We will resample 25% of the original data file.

* I use a random number seed so that we all get the same sample to work with.

```python
df.sample(frac = 0.1, random_state = 73073)
```

where *frac* is the fraction to select and *random_state* is the random number seed.

To get different random 25% subset of the data, just drop the 'rand_state' parameter or change the value from '73073' to any other integer.

```python
df = df.sample(frac = 0.1, random_state = 73073)                  # extract 50 random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 48 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We plot the data locations below.

```python
plt.subplot(111)
im = plt.scatter(df['X'],df['Y'],s=None,c=df['null'],marker=None,cmap=cmap,norm=None,vmin=0,vmax=1.0,alpha=0.8,
        linewidths=0.8,verts=None,edgecolors="black",)
plt.title('Input Data Locations')
plt.xlim(0, 1000); plt.ylim(0, 1000)
plt.xlabel('X (m)'); plt.ylabel('Y (m)')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.5, wspace=0.3, hspace=0.3)
plt.show()
```

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n = 13)
```

      X
      Y
      Facies
      Porosity
      Perm
      AI
      null

      72
      250
      50
      0
      0.139637
      0.347182
      4747.274043
      0.0

      153
      650
      750
      0
      0.170732
      10.720560
      4535.625583
      0.0

      258
      80
      669
      1
      0.244345
      3222.716042
      2696.102930
      0.0

      56
      200
      150
      0
      0.167125
      3.042590
      5500.997419
      0.0

      303
      60
      929
      1
      0.216253
      400.298484
      3959.934912
      0.0

      374
      750
      699
      0
      0.159171
      9.224292
      5263.064063
      0.0

      333
      340
      549
      0
      0.170881
      84.471492
      2918.232227
      0.0

      1
      50
      850
      1
      0.237154
      39.837129
      3074.562617
      0.0

      54
      200
      250
      1
      0.188043
      57.804784
      4997.078597
      0.0

      311
      250
      579
      1
      0.215039
      625.510541
      2693.691341
      0.0

      8
      50
      450
      1
      0.200389
      265.636019
      3454.389302
      0.0

      269
      540
      859
      0
      0.173662
      39.910675
      3414.962471
      0.0

      279
      580
      779
      0
      0.168371
      11.171392
      3646.918994
      0.0

#### Number of Effective Data

Let's try out our program for a variety of spatial continuity models.

* **100% nugget** - should have number of effective data equal to the number of data as they are all independent.

* **Zonal Anisotropy** - should have number of effective data equal to 1.0 as they are all perfectly redundnat.

* **Range of Fraction of Model Size** - should have a low number of effective data then increasing as the fraction decreases.

* **Other** - testing the code.

```python
vario = GSLIB.make_variogram(nug=1.0,nst=1,it1=1,cc1=0.0,azi1=0.0,hmaj1=100,hmin1=100)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('100% Nugget:                  Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=1,it1=1,cc1=1.0,azi1=0.0,hmaj1=1000000,hmin1=1000000)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Zonal Anisotropy:             Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=1,it1=1,cc1=1.0,azi1=0.0,hmaj1=1000,hmin1=1000)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Range of Model Size:          Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=1,it1=1,cc1=1.0,azi1=0.0,hmaj1=500,hmin1=500)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Range of 1/2 Model Size:      Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=1,it1=1,cc1=1.0,azi1=0.0,hmaj1=250,hmin1=250)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Range of 1/4 Model Size:      Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=2,it1=1,cc1=0.5,azi1=0.0,hmaj1=50,hmin1=50,it2=1,cc2=0.5,azi2=0.0,hmaj2=250,hmin2=250)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Nested Variogram:             Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

vario = GSLIB.make_variogram(nug=0.0,nst=2,it1=1,cc1=0.5,azi1=0.0,hmaj1=50,hmin1=5,it2=1,cc2=0.5,azi2=0.0,hmaj2=250,hmin2=25)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)
print('Nested Anisotropic Variogram: Number of effective data = ' + str(round(n_eff,2)) + ' / ' + str(len(df)) + '.')

```

    100% Nugget:                  Number of effective data = 48 / 48.
    Zonal Anisotropy:             Number of effective data = 1.0 / 48.
    Range of Model Size:          Number of effective data = 2.14 / 48.
    Range of 1/2 Model Size:      Number of effective data = 5.69 / 48.
    Range of 1/4 Model Size:      Number of effective data = 15.89 / 48.
    Nested Variogram:             Number of effective data = 24.32 / 48.
    Nested Anisotropic Variogram: Number of effective data = 42.5 / 48.

```python

```

```python
vario = GSLIB.make_variogram(nug=0.0,nst=1,it1=1,cc1=1.0,azi1=0.0,hmaj1=250,hmin1=250)
n_eff = n_effective(df,'X','Y',seed=73073,nreal=1000,vario=vario)

L = 10000                                      # set the number of realizations for uncertainty calculation
mean = np.zeros(L); spatial_mean = np.zeros(L) # declare arrays to hold the realizations of the statistics
P10 = np.zeros(L)
P50 = np.zeros(L); P90 = np.zeros(L)
for l in range(0, L):                          # loop over realizations
    samples = random.choices(df['Porosity'].values, weights=None, cum_weights=None, k=len(df))
    spatial_samples = random.choices(df['Porosity'].values, weights=None, cum_weights=None, k=int(n_eff))

    mean[l] = np.average(samples)
    spatial_mean[l] = np.average(spatial_samples)

plt.subplot(121)
GSLIB.hist_st(mean,0.16,0.22,False,False,50,None,'Average Porosity (fraction)','Bootstrap Uncertainty in Porosity Average')

plt.subplot(122)
GSLIB.hist_st(spatial_mean,0.16,0.22,False,False,50,None,'Average Porosity (fraction)','Spatial Bootstrap Uncertainty in Porosity Standard Deviation')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

```

Note the expansion of variance when we account for spatial continuity.  Experiment with the variogram model and rerun this result.

* What happens when the range is increased?
* What happens when the nugget is added and increased?

#### Comments

This was a basic demonstration of spatial bootstrap with a comparison to traditional bootstrap with a developed function to calculate the number of effective data with LU-simulation.

Much more could be done, you could replace the statistics, average and standard deviation with any other statistics, for example P90, kurtosis, P13 etc. I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

```python

```


```
%matplotlib inline
```

```
from __future__ import print_function, division, unicode_literals
import numpy as N
import libstempo as T
import libstempo.plot as LP, libstempo.toasim as LT
import os
import matplotlib.pyplot as plt
```

# A fake pulsar

Let's start by simulating a series of TOAs, based on a fake pulsar. The contents of the file `pulsar.par` are:
```
PSRJ	       Fake
RAJ            09:55:42.1440
DECJ           69:40:26.0040
F0             72.866368000367956367
F1	           0
F2	           0
F3             0
PEPOCH         56691.600000000000001
POSEPOCH       56691.600000000000001
DMEPOCH        56691.600000000000001
DM             0
BINARY         BT
PB             2.5323984426832435485
T0             56694.732722311168896
A1             22.225008573439934967
OM             0
ECC            0
TRACK          1
TZRSITE        @
TRES           5879.148
EPHVER         5
CLK            TT(TAI)
EPHEM          DE200
```

```
T.data = os.getcwd() # Directory where the parameter file is found
# a bunch of observations, randomly taken
obstimes = N.random.uniform(56600,57200,100)
obstimes.sort()

# Fake the arrival times!
psr = LT.fakepulsar(parfile=os.path.join(T.data,'pulsar.par'),
                    obstimes=obstimes,
                    toaerr=100)

# Model observing times and pulsar
model_obstimes = N.linspace(56600,57200,6001)
model_psr = LT.fakepulsar(parfile=os.path.join(T.data,'pulsar.par'),
                          obstimes=model_obstimes,
                          toaerr=0.0001)

LT.make_ideal(model_psr)

LT.add_efac(psr,efac=1.0)

# save these values
psr.savetim('./pulsar.tim')
model_psr.savetim('./model.tim')

# Convenience function to plot data and model
def plot_models_and_errors(psr, model_psr, xrange=None, ax=None, title=None):
    res, t, errs = psr.residuals(), psr.toas(), psr.toaerrs
    mres, mt, merrs = model_psr.residuals(), model_psr.toas(), model_psr.toaerrs

    if ax is None:
        ax = plt.gca()
    ax.plot(mt, mres, lw=2, color='r', zorder=1)
    ax.errorbar(t, res, yerr=errs / 1e6, color='b', fmt='none', zorder=10)

    if xrange is not None:
        ax.set_xlim(xrange)
    if title is not None:
        ax.set_title(title)

plot_models_and_errors(psr, model_psr)
```

## Let's play!
Now we're going to modify slightly single parameters of the timing model, and we'll see the kind of residuals that will come out.
First of all, we start with a very slightly different value of the pulse frequency (one part in ten billions, or ~$7\times10^{-9}$ Hz).
Remember that
\begin{equation}
\phi(t) = \phi_0 + \nu t + \frac{1}{2} \dot{\nu} t^2 + \frac{1}{6} \ddot{\nu} t^3 + \dots
\end{equation}
So, if instead of $\nu$ we use $\tilde{\nu}$, we expect a new $\tilde{\phi}$ that differs from $\phi$ of a quantity $\Delta\phi = \tilde{\phi} - \phi = (\tilde{\nu} - \nu) t$. This means that the residuals will have an additional _linear_ trend.

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))
model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))

```

```
rel_error = 1e-10
new_F0 = psr['F0'].val * (1 - rel_error)
psr['F0'].val = new_F0
model_psr['F0'].val = new_F0
print('dF = {} Hz'.format(rel_error * psr['F0'].val))

plot_models_and_errors(psr, model_psr)
```

## Effect of frequency derivative

Now let us try with an error on the frequency derivative. We use a mismatch of ~$5\cdot 10^{-16}$ Hz/s. According to the points above, this wrong frequency derivative will produce a spurious term proportional to $(\tilde{\dot{\nu}} - \dot{\nu}) t^2$, so a _quadratical_ term.

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))
model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))

dFdot = -5e-16
newval = psr['F1'].val + dFdot
psr['F1'].val = newval
model_psr['F1'].val = newval

plot_models_and_errors(psr, model_psr)
```

## Effect of a wrong orbital period
A slightly wrong orbital period will produce structure in the residuals proportional to

\begin{align}
\tilde{\phi} - \phi &= A [\sin(\tilde{\Omega} t) - \sin(\Omega t)] \\
                    &= 2A [\sin(\alpha t)\cos(\beta t)]
\end{align}

where $A$ is a constant, $\Omega = 2\pi / P_{orb}$ is the orbital angular velocity, $\alpha = (\tilde{\Omega} - \Omega) / 2$ and $\beta = (\tilde{\Omega} + \Omega) / 2 \approx \Omega$. So, it will be a sinusoid at a very low frequency (the difference between the right and the wrong orbital frequencies), modulated by another sinusoid at approximately the orbital frequency. The smaller the error, the lower the frequency of this modulation.

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))

model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))

from matplotlib import gridspec
fig = plt.figure(figsize=(12,6))
dPB = 2e-7

fig.suptitle('P$_{orb}$ uncertainty = ' + '{} ms'.format(dPB * 86400 * 1000))
gs = gridspec.GridSpec(1,3)
ax = plt.subplot(gs[0])
print('dPB = {} ms'.format(dPB * 86400 * 1000))
newval = psr['PB'].val - dPB
psr['PB'].val = newval
model_psr['PB'].val = newval

plot_models_and_errors(psr, model_psr, ax=ax, title='Full')

ax = plt.subplot(gs[1])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[56700, 56750], title='Early Zoom')

ax = plt.subplot(gs[2])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[57100, 57130], title='Late Zoom')
```

## Effect of a wrong $a \sin i$
In this case, as we expect, we'll have a spurious sinusoidal modulation due to the non complete correction, or the over correction, of the orbital modulation. The amplitude of this modulation will be constant.

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))

model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))
newval = psr['A1'].val * 0.99995
psr['A1'].val = newval
model_psr['A1'].val = newval

res, t, errs = psr.residuals(), psr.toas(), psr.toaerrs

fig = plt.figure(figsize=(8,6))
gs = gridspec.GridSpec(1,2)
ax = plt.subplot(gs[0])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[56700, 56750], title='Early Zoom')
ax = plt.subplot(gs[1])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[57100, 57130], title='Late Zoom')
```

## Wrong $T_0$
Let's add one _second_ to $T_0$.
For a small addition $\delta$ to $T_0$, we have something like
\begin{align}
\tilde{\phi} - \phi &= A [\sin(\Omega(t + \delta)) - \sin(\Omega t)] \\
                    &= A [\sin(\Omega t)\cos(\Omega \delta) + \cos(\Omega t)\sin(\Omega \delta) - \sin(\Omega t)]\\
                    &\approx A \cos(\Omega t) \epsilon
\end{align}
where $\epsilon=\sin(\Omega \delta)$ is a constant quantity that drives the amplitude of the spurious modulation

The sum $\sin(\Omega t)\cos(\Omega \delta) - \sin(\Omega t)$ vanishes approximately like $\sin^2(\Omega \delta)$

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))

model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))
#I add one second to T0!
newval = psr['T0'].val + 1 / 3600 / 24
psr['T0'].val = newval
model_psr['T0'].val = newval

res, t, errs = psr.residuals(), psr.toas(), psr.toaerrs

fig = plt.figure(figsize=(8,6))
gs = gridspec.GridSpec(1,2)
ax = plt.subplot(gs[0])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[56700, 56750], title='Early Zoom')
ax = plt.subplot(gs[1])
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[57100, 57150], title='Late Zoom')
```

## Wrong eccentricity

```
psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                    timfile = os.path.join(T.data,'pulsar.tim'))

model_psr = T.tempopulsar(parfile = os.path.join(T.data,'pulsar.par'),
                          timfile = os.path.join(T.data,'model.tim'))
#Let's increase the eccentricity
newval = 0.0001
psr['ECC'].val = newval
model_psr['ECC'].val = newval

res, t, errs = psr.residuals(), psr.toas(), psr.toaerrs

fig = plt.figure(figsize=(8,6))
ax = plt.subplot()
plot_models_and_errors(psr, model_psr, ax=ax, xrange=[56700, 56705])

```

```

```

```

```