# 3D elastic VTI wave equation implementation on a staggered grid

This is a first attempt at implementing the elastic wave equations in 3D vertical transversely isotropic (VTI) media as described in [1].

The notebook implements various Finite Difference schemes to solve this PDE and recreate the results from various forward simulations.

 Chirayu Khimji

 Independent Research Project

 MSc Applied Computational Science and Engineering - Imperial College London

 IRP Supervisors: Dr. Gerard Gorman, Dr. Rhodri Nelson and the Devito Project Research Group

## Introduction

To be added in proper academic syntax

## Outline
this cell needs to be completed in more detail when tasks below are completed

1. Scalar PDE Implementation for 3D elastic VTI wave propagation using temporal high-accuracy staggered-grid finite-difference scheme

2. Vectorial PDE Implementation for 3D elastic VTI wave propagation (Vectorized Approach) using temporal high-accuracy staggered-grid finite-difference scheme

3. Implementation Testing: Comparing the results of these two implementations against an pure Elastic Isotropic Implementation

4. Accuracy Testing:

# Scalar PDE Implementation: 3D Elastic VTI wave-equations
The paper [1] expresses the elastic wave equations as a set of scalar PDEs in conventional velocity-stress format:

$$
\begin{aligned}
\frac{\partial v_x}{\partial t} &= \frac{1}{\rho} \left( \frac{\partial \tau_{xx}}{\partial x} + \frac{\partial \tau_{xy}}{\partial y} + \frac{\partial \tau_{xz}}{\partial z} \right) \\
\frac{\partial v_y}{\partial t} &= \frac{1}{\rho} \left( \frac{\partial \tau_{xy}}{\partial x} + \frac{\partial \tau_{yy}}{\partial y} + \frac{\partial \tau_{yz}}{\partial z} \right) \\
\frac{\partial v_z}{\partial t} &= \frac{1}{\rho} \left( \frac{\partial \tau_{xz}}{\partial x} + \frac{\partial \tau_{yz}}{\partial y} + \frac{\partial \tau_{zz}}{\partial z} \right) \\
\frac{\partial \tau_{xx}}{\partial t} &= c_{11}\frac{\partial v_{x}}{\partial x} + c_{11}\frac{\partial v_{y}}{\partial y} - 2c_{66}\frac{\partial v_{y}}{\partial y} + c_{13}\frac{\partial v_{z}}{\partial z} \\
\frac{\partial \tau_{yy}}{\partial t} &= c_{11}\frac{\partial v_{x}}{\partial x} - 2c_{66}\frac{\partial v_{x}}{\partial x} + c_{11}\frac{\partial v_{y}}{\partial y} + c_{13}\frac{\partial v_{z}}{\partial z} \\
\frac{\partial \tau_{zz}}{\partial t} &= c_{13}\frac{\partial v_{x}}{\partial x} + c_{13}\frac{\partial v_{y}}{\partial y} + c_{33}\frac{\partial v_{z}}{\partial z} \\
\frac{\partial \tau_{zz}}{\partial t} &= c_{44}\frac{\partial v_{z}}{\partial x} + c_{44}\frac{\partial v_{x}}{\partial z} \\
\frac{\partial \tau_{yz}}{\partial t} &= c_{44}\frac{\partial v_{z}}{\partial y} + c_{44}\frac{\partial v_{y}}{\partial z} \\
\frac{\partial \tau_{xy}}{\partial t} &= c_{66}\frac{\partial v_{y}}{\partial x} + c_{66}\frac{\partial v_{x}}{\partial y} \\
\end{aligned}
$$

Stiffness coefficients in elastic VTI media can be given by (Thomsen 1986):

$
\begin{aligned}
c_{11} &= \rho \left(1 + 2\epsilon\right)v_{p0}^{2} \\
c_{33} &= \rho v_{p0}^{2} \\
c_{44} &= \rho v_{s0}^{2} \\
c_{66} &= \rho \left(1 + 2\gamma\right)v_{s0}^{2} \\
c_{13} &= \rho v_{p0}^{2}\sqrt{f\left(f + 2\delta\right)} - \rho v_{s0}^{2} \\
f &= 1 - v_{s0}^{2}/v_{p0}^{2} \\
\end{aligned}
$

### Imports
For consistency, all required imports used by this notebook are below:

```python
# Required imports:
import numpy as np

from devito import configuration, norm
from examples.seismic import plot_velocity, RickerSource, TimeAxis, Receiver, plot_image, AcquisitionGeometry, setup_geometry, seismic_args
from model_VTI import ModelElasticVTI
from elastic_VTI_solvers import *

import matplotlib.pyplot as plt

from sympy import init_printing, latex
init_printing(use_latex='mathjax')

%matplotlib inline

plt.rc('font', size=14)

# default language set to openmp i.e. Code Generated by Devito will be in openmp
#configuration['language'] = 'openmp'

# logging set to debug to capture statistics on the performance of operators
configuration['log-level'] = 'DEBUG'

# define precision type: 32bit floating point
dtype = np.float32

```

### Instantiate Devito grid for a three dimensional problem
For initial implementation testing, grid setup equivalent to devito/examples/seismic/tutorials/06_elastic.ipynb

```python
# Define dimensions for the interior of the model
nx, ny, nz = 201, 201, 201          # Number of grid points
dx, dy, dz = 7.5, 7.5, 7.5          # Grid spacing in m
shape = (nx, ny, nz)                # Domain shape
spacing = (dx, dy, dz)              # Domain size is now 5 km by 5 km
origin = (0., 0., 0.)               # Origin of coordinate system, specified in m.
extent = tuple([i*(n-1) for i, n in zip(spacing, shape)])
```

### Multilayered Model

Run this code block if you would like to use a Multilayered Model

```python
#Thomsen's anisotropy parameters
epsilon = 0.25
delta = 0.10
gamma = 0.05

# Define a physical size
nlayers = 3
npad = 50

so = 4 #spatial order
to = 1 #temporal order

# Model physical parameters:
vp = np.zeros(shape)
vs = np.zeros(shape)
rho = np.zeros(shape)

# Define a Vp profile. The velocity is in km/s
vp_top = 1.5
vp_bottom = 3.5

# Define a Vs profile. The velocity is in km/s
vs_top = 1.5
vs_bottom = 3.5

# Define a velocity profile in km/s
v_p = np.empty(shape, dtype=dtype)
v_p[:] = vp_top  # Top velocity (background)
vp_i = np.linspace(vp_top, vp_bottom, nlayers)
for i in range(1, nlayers):
    v_p[..., i*int(shape[-1] / nlayers):] = vp_i[i]  # Bottom velocity

v_s = np.empty(shape, dtype=dtype)
v_s[:] = vs_top  # Top velocity (background)
vs_i = np.linspace(vs_top, vs_bottom, nlayers)
for i in range(1, nlayers):
    v_s[..., i*int(shape[-1] / nlayers):] = vs_i[i]  # Bottom velocity

rho[:] = 0.31*(v_p[:]*1000.)**0.25 # Gardner's relation

model = ModelElasticVTI(origin=origin, spacing = spacing, shape=shape, space_order=so,
vp=v_p, vs=v_s, rho=rho, epsilon=epsilon, delta=delta, gamma=gamma, nbl = npad, dtype = dtype, bcs = "mask")

print("")
print("Grid Setup:")
print("Shape:       ", model.grid.shape)
print("Origin:      ", origin)
print("Grid Spacing: ", model.grid.spacing_map)
print("Extent:      ", model.grid.extent)
```

    Operator `initdamp` generated in 0.17 s
      * lowering.Clusters: 0.08 s (47.3 %)
         * specializing.Clusters: 0.05 s (29.6 %)
      * lowering.Expressions: 0.05 s (29.6 %)
      * lowering.IET: 0.05 s (29.6 %)
    Flops reduction after symbolic optimization: [102 --> 102]
    Allocating memory for damp(303, 303, 303)
    Operator `initdamp` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/45d77e2623ece35a7ebe4724e555e318bf6bea06.c` in 0.08 s from jit-cache
    Operator `initdamp` run in 0.20 s
    * section0<<301,301,301>,<50,301,301>,<50,301,301>,<301,50,301>,<301,50,301>,<301,301,50>,<301,301,50>> with OI=0.01 computed in 0.20 s [2.42 GFlops/s]
    Performance[mode=advanced] arguments: {}
    Allocating memory for vp(309, 309, 309)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (41.6 %)
         * specializing.IET: 0.03 s (25.0 %)
      * lowering.Clusters: 0.05 s (41.6 %)
      * lowering.Expressions: 0.03 s (25.0 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/d1fb67ec328162eff2c72c057cfa87d03cbb44d8.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for vs(309, 309, 309)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (40.6 %)
         * specializing.IET: 0.03 s (24.4 %)
      * lowering.Clusters: 0.05 s (40.6 %)
      * lowering.Expressions: 0.03 s (24.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/f1cab5f51af97008f357d66314b715484f6d37d9.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for rho(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.2 %)
         * specializing.IET: 0.03 s (25.4 %)
      * lowering.Clusters: 0.04 s (33.8 %)
      * lowering.Expressions: 0.03 s (25.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/c887fe6a3f29f23770a9e643a9b110f42768b95e.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for irho(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.3 %)
         * specializing.IET: 0.03 s (25.4 %)
      * lowering.Clusters: 0.05 s (42.3 %)
      * lowering.Expressions: 0.03 s (25.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/b611bd7b0822b11c757ff73a94eec73b0ef3d725.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c11(309, 309, 309)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (41.2 %)
         * specializing.IET: 0.03 s (24.8 %)
      * lowering.Clusters: 0.04 s (33.0 %)
      * lowering.Expressions: 0.03 s (24.8 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/5abd0c50c7ebfea9e3155b2aa27bd408843d642a.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c33(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.4 %)
         * specializing.IET: 0.03 s (25.5 %)
      * lowering.Clusters: 0.04 s (33.9 %)
      * lowering.Expressions: 0.03 s (25.5 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/b766dbd75cc24476139b5e39ae3cebca1961a233.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c44(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.7 %)
         * specializing.IET: 0.03 s (25.6 %)
      * lowering.Clusters: 0.04 s (34.2 %)
      * lowering.Expressions: 0.03 s (25.6 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/95d2b61f6a3378d5d46367ef45c70fc844fc968b.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c66(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.1 %)
         * specializing.IET: 0.03 s (25.3 %)
      * lowering.Clusters: 0.04 s (33.7 %)
      * lowering.Expressions: 0.03 s (25.3 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/1502c177c0033cd182f371b6085859aa40f9a2c6.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.06 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c13(309, 309, 309)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.3 %)
         * specializing.IET: 0.03 s (25.4 %)
      * lowering.Clusters: 0.04 s (33.9 %)
      * lowering.Expressions: 0.03 s (25.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/88269c2a3dd38174fa9d9fadeaa738759d778e0a.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.05 s
    Performance[mode=advanced] arguments: {}

    Grid Setup:
    Shape:        (301, 301, 301)
    Origin:       (0.0, 0.0, 0.0)
    Grid Spacing:  {h_x: 7.5, h_y: 7.5, h_z: 7.5}
    Extent:       (2250.0, 2250.0, 2250.0)

```python
# Code to plot model attributes
def plot_model(model):
    print("Velocity Model Plots:")

    plt_options_model = {'cmap': 'jet', 'extent': [model.origin[0], model.origin[0] + model.domain_size[0],
                                                model.origin[-1] + model.domain_size[-1], model.origin[-1]]}
    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(15, 5))

    slices = [slice(model.nbl, -model.nbl), slice(model.nbl, -model.nbl)]
    img1 = ax[0].imshow(np.transpose(model.vp.data[0][slices]), vmin=0.9, vmax=5, **plt_options_model)
    fig.colorbar(img1, ax=ax[0])
    ax[0].set_title(r"Vp (km/s)", fontsize=20)
    ax[0].set_xlabel('X (m)', fontsize=20)
    ax[0].set_ylabel('Depth (m)', fontsize=20)
    ax[0].set_aspect('auto')

    img2 = ax[1].imshow(np.transpose(model.vs.data[0][slices]), vmin=0.9, vmax=5, **plt_options_model)
    fig.colorbar(img2, ax=ax[1])
    ax[1].set_title(r"Vs (km/s)", fontsize=20)
    ax[1].set_xlabel('X (m)', fontsize=20)
    ax[1].set_ylabel('Depth (m)', fontsize=20)
    ax[1].set_aspect('auto')

    img3 = ax[2].imshow(np.transpose(model.rho.data[0][slices]), vmin=0.9, vmax=3.4, **plt_options_model)
    fig.colorbar(img3, ax=ax[2])
    ax[2].set_title(r"Density $\rho$ (g/cm^3)", fontsize=20)
    ax[2].set_xlabel('X (m)', fontsize=20)
    ax[2].set_ylabel('Depth (m)', fontsize=20)
    ax[2].set_aspect('auto')

    plt.tight_layout()

## Plot Multilayered Model
plot_model(model)
```

### Single Layered Model
Run the code block below instead for a single layered model

```python
#Model Absorbing Boundaries
npad = 20

#Thomsen's anisotropy parameters
epsilon = 0.25
delta = 0.10
gamma = 0.05

#initial conditions
v_p = np.zeros(shape)
v_s = np.zeros(shape)
rho = np.zeros(shape)

v_p[:] = 2.0 #km/s
v_s[:] = 1.0 #km/s
rho[:] = 1.8 #kg/m**3

so = 4 #spatial order
to = 1 #temporal order

model = ModelElasticVTI(origin=origin, spacing = spacing, shape=shape, space_order=so,
vp=v_p, vs=v_s, rho=rho, epsilon=epsilon, delta=delta, gamma=gamma, nbl = npad, dtype = dtype, bcs = "mask")

print("")
print("Grid Setup:")
print("Shape:       ", model.grid.shape)
print("Origin:      ", origin)
print("Grid Spacing: ", model.grid.spacing_map)
print("Extent:      ", model.grid.extent)
```

    Operator `initdamp` generated in 0.17 s
      * lowering.Clusters: 0.08 s (47.5 %)
         * specializing.Clusters: 0.05 s (29.7 %)
      * lowering.Expressions: 0.05 s (29.7 %)
      * lowering.IET: 0.05 s (29.7 %)
    Flops reduction after symbolic optimization: [102 --> 102]
    Allocating memory for damp(243, 243, 243)
    Operator `initdamp` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/5818e98a502780ac5c5da091323ca30a180fa0bd.c` in 0.06 s from jit-cache
    Operator `initdamp` run in 0.07 s
    * section0<<241,241,241>,<20,241,241>,<20,241,241>,<241,20,241>,<241,20,241>,<241,241,20>,<241,241,20>> with OI=0.01 computed in 0.07 s [1.81 GFlops/s]
    Performance[mode=advanced] arguments: {}
    Allocating memory for vp(249, 249, 249)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.06 s (48.7 %)
         * specializing.IET: 0.03 s (24.4 %)
      * lowering.Clusters: 0.05 s (40.6 %)
      * lowering.Expressions: 0.03 s (24.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/fe04162359efa23048d643bc5b413543273a4c04.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.03 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for vs(249, 249, 249)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (40.9 %)
         * specializing.IET: 0.03 s (24.6 %)
      * lowering.Clusters: 0.05 s (40.9 %)
      * lowering.Expressions: 0.03 s (24.6 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/2e7c1fa3566510d4bf9cd3bb9adf96c525bcfabf.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.03 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for rho(249, 249, 249)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (40.9 %)
         * specializing.IET: 0.03 s (24.6 %)
      * lowering.Clusters: 0.05 s (40.9 %)
      * lowering.Expressions: 0.03 s (24.6 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/b757e66d09fa494d252ddf2f2246af53e91d0298.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.03 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for irho(249, 249, 249)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.9 %)
         * specializing.IET: 0.03 s (25.8 %)
      * lowering.Clusters: 0.05 s (42.9 %)
      * lowering.Expressions: 0.03 s (25.8 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/e19e839bb20ed0d68372b7c73c042e986f155602.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.02 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c11(249, 249, 249)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.6 %)
         * specializing.IET: 0.03 s (25.6 %)
      * lowering.Clusters: 0.04 s (34.1 %)
      * lowering.Expressions: 0.03 s (25.6 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/bce3354659d9d2a2d6a8791cabfcf84c8126c6cf.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.06 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c33(249, 249, 249)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.7 %)
         * specializing.IET: 0.03 s (25.7 %)
      * lowering.Clusters: 0.04 s (34.2 %)
      * lowering.Expressions: 0.03 s (25.7 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/553cdccdc21143d5fe225c69f151d65407d90b61.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.02 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c44(249, 249, 249)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.3 %)
         * specializing.IET: 0.03 s (25.4 %)
      * lowering.Clusters: 0.05 s (42.3 %)
      * lowering.Expressions: 0.03 s (25.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/0c5333998f23921c8a749a3095a16d34741d0c11.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.02 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c66(249, 249, 249)
    Operator `padfunc` generated in 0.13 s
      * lowering.IET: 0.05 s (41.2 %)
         * specializing.IET: 0.03 s (24.8 %)
      * lowering.Clusters: 0.05 s (41.2 %)
      * lowering.Expressions: 0.03 s (24.8 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/cd9813d01425699c4d2313c7004b4cf313ea892f.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.06 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for c13(249, 249, 249)
    Operator `padfunc` generated in 0.12 s
      * lowering.IET: 0.05 s (42.3 %)
         * specializing.IET: 0.03 s (25.4 %)
      * lowering.Clusters: 0.04 s (33.8 %)
      * lowering.Expressions: 0.03 s (25.4 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `padfunc` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/1946ed3522d9de88d18596ed1be3133edadb6b41.c` in 0.05 s from jit-cache
    Operator `padfunc` run in 0.06 s
    Performance[mode=advanced] arguments: {}

    Grid Setup:
    Shape:        (241, 241, 241)
    Origin:       (0.0, 0.0, 0.0)
    Grid Spacing:  {h_x: 7.5, h_y: 7.5, h_z: 7.5}
    Extent:       (1800.0, 1800.0, 1800.0)

```python
plot_model(model)
```

### Devito's Symbolic Objects are now defined on the `grid`
The symbolic objects `Function`, `VectorTimeFunction` and `TensorTimeFunction` define the physics of Wavefields onto the grid. These objects account for both spatially varying and time dependent features of Wavefields.

```python
# PDE functions assigned to model:
x, y, z = model.grid.dimensions
# dampening function assigned to model:
damp = model.damp

# Now we create the velocity and pressure fields

#Velocity
v = VectorTimeFunction(name='v', grid=model.grid, space_order=so, time_order=to)

#Stress
tau = TensorTimeFunction(name='t', grid=model.grid, space_order=so, time_order=to)

# Symbolic definition of the model grid spacing for a staggered grid
ts = model.grid.stepping_dim.spacing
```

### Define the simulation time range

```python
t0 = 0.     # Simulation time start
tn = 500. #300.  # Simulation time end (1 second = 1000 msec)

dt = model.critical_dt #time step computed from elastic VTI CFL condition

time_range = TimeAxis(start=t0, stop=tn, step=dt)
print("Time range: ", time_range)
```

    Allocating memory for n(1,)
    Time range:  TimeAxis: start=0, stop=500.52, step=1.164, num=431

### Setting up the acquisition geometry: locations of explosive source and recievers

For an example simulation:
A 10 Hz center frequency Ricker wavelet source, located at (x, y, z) = (750m, 750m, 750m) and is added into $v_x$ component to generate vibration.

For Ricker wavelet, the source signature $g(t)$ is the derivative of a Gaussian pulse:

$$g(t) = -2 \alpha(t - t_0)e^{-\alpha(t-t_0)^2}$$

```python
# Centered Source with 10 Hz center frequency
fpeak = 0.010
src = RickerSource(name='src', grid=model.grid, f0=fpeak, time_range=time_range)
src.coordinates.data[:] = np.array([750., 750., 750.])

#rec = Receiver(name='rec', grid=model.grid, npoint = shape[0], time_range=time_range)
#rec.coordinates.data[:,0] = dx * (nx/2)
#rec.coordinates.data[:,1] = 6
#rec.coordinates.data[:,2] = np.linspace(0.0, dz*(nz-1), nz)

# The receiver
nrec = 301
rec = Receiver(name="rec", grid=model.grid, npoint=nrec, time_range=time_range)
rec.coordinates.data[:, 0] = np.linspace(0., model.domain_size[0], num=nrec)
rec.coordinates.data[:, 1] = 0 #model.grid.extent[0]/2
rec.coordinates.data[:, -1] = 5.

rec2 = Receiver(name="rec2", grid=model.grid, npoint=nrec, time_range=time_range)
rec2.coordinates.data[:, 0] = np.linspace(0., model.domain_size[0], num=nrec)
rec2.coordinates.data[:, 1] = 0 #model.grid.extent[0]/2
rec2.coordinates.data[:, -1] = 5.

rec3 = Receiver(name="rec3", grid=model.grid, npoint=nrec, time_range=time_range)
rec3.coordinates.data[:, 0] = np.linspace(0., model.domain_size[0], num=nrec)
rec3.coordinates.data[:, 1] = 0 #model.grid.extent[0]/2
rec3.coordinates.data[:, -1] = 5.

print("src_coords [x,y,z]:", src.coordinates.data)
#print("rec_coords [x,y,z]:", rec.coordinates.data)

#plot time signature to view wavelet
src.show()
```

```python
## Symbolic Parameters used from the model
irho = model.irho #irho = 1/rho

c11 = model.c11
c33 = model.c33
c44 = model.c44
c66 = model.c66
c13 = model.c13
```

```python
#3D source injection term
src_xx = src.inject(field=tau[0, 0].forward, expr=src*ts)
src_yy = src.inject(field=tau[1, 1].forward, expr=src*ts)
src_zz = src.inject(field=tau[2, 2].forward, expr=src*ts)

# 3D interpolation expression for receivers
rec_xx = rec.interpolate(expr=v[0])
rec_yy = rec2.interpolate(expr=v[1])
rec_zz = rec3.interpolate(expr=v[2])
#rec_term = rec.interpolate(expr=tau.forward)

src_term = src_xx + src_yy + src_zz
rec_term = rec_xx + rec_yy + rec_zz
src_rec = src_term + rec_term
```

### Implementation with Absorbing BCs
Assemble scalar PDEs individually

```python
# Define Scalar stencil for VTI operators with absorbing BCs:

# Particle Velocity for each direction
u_vx = Eq(v[0].forward, damp*v[0] + damp*ts*irho*(tau[0,0].dx + tau[1,0].dy + tau[2,0].dz) )
u_vy = Eq(v[1].forward, damp*v[1] + damp*ts*irho*(tau[0,1].dx + tau[1,1].dy + tau[1,2].dz) )
u_vz = Eq(v[2].forward, damp*v[2] + damp*ts*irho*(tau[0,2].dx + tau[2,1].dy + tau[2,2].dz) )

# Stress for each direction in VTI Media:
u_txx = Eq(tau[0,0].forward, damp*tau[0,0] + damp*ts*(c11*v[0].forward.dx + c11*v[1].forward.dy - 2*c66*v[1].forward.dy + c13*v[2].forward.dz) )
u_tyy = Eq(tau[1,1].forward, damp*tau[1,1] + damp*ts*(c11*v[0].forward.dx - 2*c66*v[0].forward.dx + c11*v[1].forward.dy + c13*v[2].forward.dz) )
u_tzz = Eq(tau[2,2].forward, damp*tau[2,2] + damp*ts*(c13*v[0].forward.dx + c13*v[1].forward.dy + c33*v[2].forward.dz) )

u_txz = Eq(tau[0,2].forward, damp*tau[0,2] + damp*ts*(c44*v[2].forward.dx + c44*v[0].forward.dz) )
u_tyz = Eq(tau[1,2].forward, damp*tau[1,2] + damp*ts*(c44*v[2].forward.dy + c44*v[1].forward.dz) )
u_txy = Eq(tau[0,1].forward, damp*tau[0,1] + damp*ts*(c66*v[1].forward.dx + c66*v[0].forward.dy) )

stencil = [u_vx, u_vy, u_vz, u_txx, u_tyy, u_tzz, u_txz, u_tyz, u_txy]

pde = stencil + src_rec
```

### Forward Simulation

Define forward propogator using the Devito `Operator`. `Operator` will generate optimized C code which is JIT-compiled and ready to run.

```python
#Assemble the operator matrix with the correct model spacing

op_fwd = Operator(pde, subs=model.spacing_map)
```

    Operator `Kernel` generated in 3.22 s
      * lowering.Expressions: 1.37 s (42.7 %)
      * lowering.Clusters: 0.86 s (26.8 %)
      * lowering.IET: 0.82 s (25.6 %)
    Flops reduction after symbolic optimization: [306 --> 166]

```python
#Uncomment to Print generated c++ code
#print(op_fwd_prop)
```

This demonstration shows that `Operator` makes it very trivial to create a fully functional computational kernal for elastic wave propogation in VTI media in just a few lines of python.
Now we run `op_fwd` for forward simulation.

```python
#NBVAL_IGNORE_OUTPUT

#Forward Simulation with the critical computed timestep
op_fwd(dt=dt)
```

    Allocating memory for rec(431, 301)
    Allocating memory for rec2(431, 301)
    Allocating memory for rec3(431, 301)
    Allocating memory for t_xx(2, 249, 249, 249)
    Allocating memory for t_xy(2, 249, 249, 249)
    Allocating memory for t_xz(2, 249, 249, 249)
    Allocating memory for t_yy(2, 249, 249, 249)
    Allocating memory for t_yz(2, 249, 249, 249)
    Allocating memory for t_zz(2, 249, 249, 249)
    Allocating memory for v_x(2, 249, 249, 249)
    Allocating memory for v_y(2, 249, 249, 249)
    Allocating memory for v_z(2, 249, 249, 249)
    Operator `Kernel` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/aae26e86d5e049c86fb01fd2de8626c8c29313cf.c` in 0.41 s from jit-cache

```python
#NBVAL_SKIP

# Mid-points:
mid_x = int(0.5*(v[0].data.shape[1]-1))+1
mid_y = int(0.5*(v[0].data.shape[2]-1))+1

# Plot some selected results:

plot_image(v[0].data[1, :, mid_y, :], cmap="seismic")
plot_image(v[0].data[1, mid_x, :, :], cmap="seismic")

plot_image(tau[2, 2].data[1, :, mid_y, :], cmap="seismic")
plot_image(tau[2, 2].data[1, mid_x, :, :], cmap="seismic")
```

```python
assert np.isclose(norm(v[0]), 0.037024777, atol=1e-4, rtol=0)
```

```python
def plot_v(model, v):
    slices = [slice(model.nbl, -model.nbl), slice(model.nbl, -model.nbl)]
    scale = .5*1e-7

    plt_options_model = {'extent': [model.origin[0] , model.origin[0] + model.domain_size[0],
                                    model.origin[-1] + model.domain_size[-1], model.origin[-1]]}

    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(15, 7))
    ax[0].imshow(np.transpose(v[0].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[0].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[0].set_aspect('auto')
    ax[0].set_xlabel('X (m)', fontsize=20)
    ax[0].set_ylabel('Depth (m)', fontsize=20)
    ax[0].set_title(r"$v_{x}$", fontsize=20)

    ax[1].imshow(np.transpose(v[1].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[1].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[1].set_aspect('auto')
    ax[1].set_xlabel('X (m)', fontsize=20)
    ax[1].set_title(r"$v_{y}$", fontsize=20)

    ax[2].imshow(np.transpose(v[-1].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[2].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[2].set_aspect('auto')
    ax[2].set_xlabel('X (m)', fontsize=20)
    ax[2].set_title(r"$v_{z}$", fontsize=20)

def plot_tau(model, tau):
    slices = [slice(model.nbl, -model.nbl), slice(model.nbl, -model.nbl)]
    scale = .5*1e-7

    plt_options_model = {'extent': [model.origin[0] , model.origin[0] + model.domain_size[0],
                                    model.origin[-1] + model.domain_size[-1], model.origin[-1]]}

    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(15, 7))
    ax[0].imshow(np.transpose(tau[0,0].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[0].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[0].set_aspect('auto')
    ax[0].set_xlabel('X (m)', fontsize=20)
    ax[0].set_ylabel('Depth (m)', fontsize=20)
    ax[0].set_title(r"$\tau_{xx}$", fontsize=20)

    ax[1].imshow(np.transpose(tau[1,1].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[1].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[1].set_aspect('auto')
    ax[1].set_xlabel('X (m)', fontsize=20)
    ax[1].set_title(r"$\tau_{yy}$", fontsize=20)

    ax[2].imshow(np.transpose(tau[2,2].data[0][:,-1][slices]), vmin=-scale, vmax=scale, cmap="RdGy", **plt_options_model)
    ax[2].imshow(np.transpose(model.vp.data[0][slices]), vmin=1.5, vmax=3.5, cmap="jet", alpha=.5, **plt_options_model)
    ax[2].set_aspect('auto')
    ax[2].set_xlabel('X (m)', fontsize=20)
    ax[2].set_title(r"$\tau_{zz}$", fontsize=20)

plot_v(model, v)
plot_tau(model, tau)
```

### Reciever Shot Record to go here -- not working

```python
# Data on a standard 2ms tim axis
rec_plot = rec.resample(num=1001)
rec2_plot = rec2.resample(num=1001)
rec3_plot = rec3.resample(num=1001)
```

```python
scale_for_plot = np.diag(np.linspace(1.0, 2.5, 1001)**2.0)
```

```python
rec_plot.coordinates.data.shape
```

```python
def plot_receiver(rec): ## This function is not giving desired results
    rec_plot = rec.resample(num=1001)
    rec2_plot = rec2.resample(num=1001)
    rec3_plot = rec3.resample(num=1001)

    extent = [rec_plot.coordinates.data[0, 0], rec_plot.coordinates.data[-1, 0], 1e-3*tn, t0]
    aspect = rec_plot.coordinates.data[-1, 0]/(1e-3*tn)/.5

    plt.figure(figsize=(10, 10))
    #plt.imshow(rec_plot.data, vmin=-.001, vmax=.1, cmap="seismic",
    plt.imshow(rec_plot.data, cmap="seismic",
           interpolation='lanczos', extent=extent, aspect=aspect)
    plt.ylabel("Time (s)", fontsize=20)
    plt.xlabel("Receiver position (m)", fontsize=20)

print(np.unique(rec.data), np.unique(rec2.data), np.unique(rec3.data)) ## Why are these numbers so small ?
plot_receiver(rec)
```

### Function to reset Pressure and Velocity fields between operator runs

```python
#Reset Fields
def reset_fields(v, tau):
    v[0].data[:] = 0
    v[1].data[:] = 0
    v[2].data[:] = 0

    tau[0,0].data[:] = 0
    tau[0,1].data[:] = 0
    tau[0,2].data[:] = 0

    tau[1,0].data[:] = 0
    tau[1,1].data[:] = 0
    tau[1,2].data[:] = 0

    tau[2,0].data[:] = 0
    tau[2,1].data[:] = 0
    tau[2,2].data[:] = 0
    return v, tau
```

### Testing Scalar Implementation: Compute Norm of Elastic Wave Equation in Vectorial form for Isotropic Media

We use the material from the examples/seismic/acoustic/accuracy.ipynb notebook.

```python
#Thomsen's anisotropy parameters
epsilon = 0
delta = 0
gamma = 0

#Initial Conditions
v_p = 2.0 #km/s
v_s = 1.0 #km/s
rho = 1.8 #kg/m**3

model = ModelElasticVTI(origin=origin, spacing = spacing, shape=shape, space_order=so,
vp=v_p, vs=v_s, rho=rho, epsilon=epsilon, delta=delta, gamma=gamma, nbl = npad, dtype = dtype, bcs = "mask")

# PDE functions assigned to model:
x, y, z = model.grid.dimensions

#Velocity
v = VectorTimeFunction(name='v', grid=model.grid, space_order=so, time_order=to)

#Stress
tau = TensorTimeFunction(name='t', grid=model.grid, space_order=so, time_order=to)

# Symbolic definition of the model grid spacing for a staggered grid
ts = model.grid.stepping_dim.spacing

t0 = 0.     # Simulation time start
tn = 300.  # Simulation time end (1 second = 1000 msec)

dt = model.critical_dt #time step computed from elastic VTI CFL condition

time_range = TimeAxis(start=t0, stop=tn, step=dt)

# Centered Source with 10 Hz center frequency
fpeak = 0.010
src = RickerSource(name='src', grid=model.grid, f0=fpeak, time_range=time_range)
src.coordinates.data[:] = np.array([750., 750., 750.])

#3D source injection term
src_xx = src.inject(field=tau[0, 0].forward, expr=src*ts)
src_yy = src.inject(field=tau[1, 1].forward, expr=src*ts)
src_zz = src.inject(field=tau[2, 2].forward, expr=src*ts)

src_term = src_xx + src_yy + src_zz

```

```python
def vectorial_elastic(model, v, tau):
    v, tau = reset_fields(v, tau)

    # Thorbecke's parameter notation
    irho = model.irho #irho = 1/rho
    rho = model.rho
    cp2 = model.vp*model.vp
    cs2 = model.vs*model.vs

    mu = cs2*rho
    l = (cp2*rho - 2*mu)

    # fdelmodc reference implementation
    u_v = Eq(v.forward, v + ts*irho*div(tau))
    u_t = Eq(tau.forward, tau + ts * l * diag(div(v.forward)) + ts * mu * (grad(v.forward) + grad(v.forward).T))
    stencil = [u_v, u_t]

    op = Operator(stencil + src_term,  subs=model.spacing_map)

    #Forward Simulation
    op(dt = dt)
    return v, tau
```

```python
#NBVAL_IGNORE_OUTPUT

v, tau = vectorial_elastic(model, v, tau)
```

```python
norm_elastic = norm(v[0])
print("Normalized Velocity:", norm_elastic)
```

### Testing Scalar Implementation: Reduce $C_{VTI}$ to $C_{ISO}$

In order to test that the Scale PDE Implementation is correct, the stress tensor for VTI media can be made isotropic and should retrieve Elastic wave equations for Isotropic media. This is done by setting the Thomsen's anisotropy parameters to zero. The results generated from the converted VTI stress tensor should reproduce equivalent results as a purely Isotropic elastic problem. The elastic test case should have an identical problem setup as the elastic VTI case i.e. equivalent grid setup, model, sources etc. This test will be used to verify that the Scalar PDE Implementation of 3D Elastic VTI wave-equations is correct.

Elastic Coefficient Matrix for VTI Media [2]:

$C_{VTI} = \begin{bmatrix}c_{11} & c_{11} - 2c_{66} & c_{13} & 0 & 0 & 0 \\ c_{11} - 2c_{66} & c_{11} & c_{13} & 0 & 0 & 0 \\ c_{13} & c_{13} & c_{33} & 0 & 0 & 0 \\ 0 & 0 & 0 & c_{44} & 0 & 0 \\ 0 & 0 & 0 & 0 & c_{44} & 0 \\ 0 & 0 & 0 & 0 & 0 & c_{66} \end{bmatrix} \\ $
Where the compenents are stiffness coefficients as explained previously.

For Isotropic media, Elastic Coefficient Matrix reduces to [3]:

$ C_{ISO} = \begin{bmatrix} \lambda + 2\mu & \lambda & \lambda & 0 & 0 & 0 \\ \lambda & \lambda + 2\mu & \lambda & 0 & 0 & 0 \\ \lambda & \lambda & \lambda + 2\mu & 0 & 0 & 0 \\ 0 & 0 & 0 & \mu & 0 & 0 \\ 0 & 0 & 0 & 0 & \mu & 0 \\ 0 & 0 & 0 & 0 & 0 & \mu \end{bmatrix} \\ $

where $\lambda$ and $\mu$ are Lamé parameters.

### Make the stress tensor for VTI media isotropic:

This can either be done by setting $\ep

By comparing both the matrices above, it is convenient to express the stiffness coefficients for VTI media in terms of generalized Lamé parameters:

$
\begin{aligned}
c_{11} &= \lambda + 2\mu \\
c_{33} &= \lambda + 2\mu \\
c_{44} &= \mu \\
c_{66} &= \mu \\
c_{13} &= \lambda \\
c_{11} - 2c_{66} &= \lambda \\
\end{aligned}
$

Hence:

$
\begin{aligned}
c_{11} &= c_{33} \\
c_{44} &= c_{66} \\
c_{13} &= c_{11} - 2c_{66} \\
\end{aligned}
$

This should cancel out all the anisotropy parameters and therefore make the Elastic VTI stress tensor isotropic

```python
# Setup VTI Stress tensor for Isotropic media
c11 = c33
c44 = c66
c13 = c11 - 2*c66
```

### Implementation without absorbing BCs

```python
def elastic_VTI_stencil_testing(model, v, tau):
    v, tau = reset_fields(v, tau)

    ## Symbolic Parameters used from the model
    irho = model.irho #irho = 1/rho
    c11 = model.c11
    c33 = model.c33
    c44 = model.c44
    c66 = model.c66
    c13 = model.c13

    # Particle Velocity for each direction
    u_vx = Eq(v[0].forward, v[0] + ts*irho*(tau[0,0].dx + tau[1,0].dy + tau[2,0].dz) )
    u_vy = Eq(v[1].forward, v[1] + ts*irho*(tau[0,1].dx + tau[1,1].dy + tau[1,2].dz) )
    u_vz = Eq(v[2].forward, v[2] + ts*irho*(tau[0,2].dx + tau[2,1].dy + tau[2,2].dz) )

    # Stress for each direction in VTI Media:
    u_txx = Eq(tau[0,0].forward, tau[0,0] + ts*(c11*v[0].forward.dx + c11*v[1].forward.dy - 2*c66*v[1].forward.dy + c13*v[2].forward.dz) )
    u_tyy = Eq(tau[1,1].forward, tau[1,1] + ts*(c11*v[0].forward.dx - 2*c66*v[0].forward.dx + c11*v[1].forward.dy + c13*v[2].forward.dz) )
    u_tzz = Eq(tau[2,2].forward, tau[2,2] + ts*(c13*v[0].forward.dx + c13*v[1].forward.dy + c33*v[2].forward.dz) )

    u_txz = Eq(tau[0,2].forward, tau[0,2] + ts*(c44*v[2].forward.dx + c44*v[0].forward.dz) )
    u_tyz = Eq(tau[1,2].forward, tau[1,2] + ts*(c44*v[2].forward.dy + c44*v[1].forward.dz) )
    u_txy = Eq(tau[0,1].forward, tau[0,1] + ts*(c66*v[1].forward.dx + c66*v[0].forward.dy) )

    stencil = [u_vx, u_vy, u_vz, u_txx, u_tyy, u_tzz, u_txz, u_tyz, u_txy]

    pde = stencil + src_term

    #Assemble the operator
    op = Operator(pde, subs=model.spacing_map)
    #Forward Simulation
    op(dt=dt)

    return v, tau
```

```python
v, tau = elastic_VTI_stencil_testing(model, v, tau)
```

```python
#NBVAL_SKIP

# Mid-points:
mid_x = int(0.5*(v[0].data.shape[1]-1))+1
mid_y = int(0.5*(v[0].data.shape[2]-1))+1

# Plot some selected results:

plot_image(v[0].data[1, :, mid_y, :], cmap="seismic")
plot_image(v[0].data[1, mid_x, :, :], cmap="seismic")

plot_image(tau[2, 2].data[1, :, mid_y, :], cmap="seismic")
plot_image(tau[2, 2].data[1, mid_x, :, :], cmap="seismic")
```

```python
assert np.isclose(norm(v[0]), norm_elastic, atol=1e-6, rtol=0)
```

# Verification

Compute the error between numerical and reference solutions for increasing spatial discretization order and grid spacing. Additionally compare the time to solution with the error for these src_coords. from accuracy.ipynb

```python
# Model with fixed time step value
class ModelBench(ModelElasticVTI):
    """
    Physical model used for accuracy benchmarking.
    The critical dt is made small enough to ignore
    time discretization errors
    """

    @property
    def critical_dt(self):
        """Critical computational time step value."""
        return .1
```

```python
# Discretization order
orders = (2, 4, 6, 8, 10)
norder = len(orders)

```

```python
# Number of time steps
nt = 2001
# Time axis defined
dt = 0.1
t0 = 0.
tn = dt * (nt-1)
time = np.linspace(t0, tn, nt)
# Source peak frequency in KHz
fpeak = 0.09
f0 = fpeak

print("Time Start: ", t0 )
print("Time End: ", tn )
print("Time Step: ", dt )
print("Number of Time Steps: ", nt )
print("Source Peak Frequency: ", f0 )

```

    Time Start:  0.0
    Time End:  200.0
    Time Step:  0.1
    Number of Time Steps:  2001
    Source Peak Frequency:  0.09

```python
# Increasing grid spacing and Domain size
sizes = ((201, 2.0), (161, 2.5), (101, 4.0))
dx = [2.0, 2.5, 4.0]
nsizes = len(sizes)
```

```python
# Fine grid model
c0 = 1.5
rho= 1.8 #kg/m**3

#Thomsen's anisotropy parameters
epsilon = 0.25
delta = 0.10
gamma = 0.05

model = ModelBench(vp=c0, vs=c0, rho=rho, epsilon=epsilon, delta=delta, gamma=gamma, origin=(0., 0., 0.), spacing=(.5, .5, .5),                     bcs="damp", shape=(201, 201, 201), space_order=20, nbl=40, dtype=np.float32)
```

    Operator `initdamp` generated in 0.17 s
      * lowering.Clusters: 0.07 s (42.4 %)
         * specializing.Clusters: 0.05 s (30.3 %)
      * lowering.Expressions: 0.05 s (30.3 %)
      * lowering.IET: 0.05 s (30.3 %)
    Flops reduction after symbolic optimization: [102 --> 102]
    Allocating memory for damp(283, 283, 283)
    Operator `initdamp` fetched `/var/folders/kb/bc2lwpvd4kj2k3qjll_cmwp80000gn/T/devito-jitcache-uid501/624b4455001f27d77e8620a18c0893867bdf4357.c` in 0.08 s from jit-cache
    Operator `initdamp` run in 0.15 s
    * section0<<40,281,281>,<40,281,281>,<281,40,281>,<281,40,281>,<281,281,40>,<281,281,40>> with OI=0.01 computed in 0.15 s [2.28 GFlops/s]
    Performance[mode=advanced] arguments: {}

```python
# Source and receiver geometries
src_coordinates = np.empty((1, 3))
src_coordinates[0, :] = 100.

# Single receiver offset 100 m from source
rec_coordinates = np.empty((1, 3))
rec_coordinates[:, :] = 150.

print("The computational Grid has (%s, %s, %s) grid points "
       "and a physical extent of (%sm, %sm, %sm)" % (*model.grid.shape, *model.grid.extent))
print("Source is at the center with coordinates", src_coordinates)
print("Receiver (single receiver) is located at", rec_coordinates)

# Note: gets time sampling from model.critical_dt
geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates,
                                   t0=t0, tn=tn, src_type='Ricker', f0=f0, t0w=1.5/f0)

src_coordinates.shape

```

    The computational Grid has (281, 281, 281) grid points and a physical extent of (140.0m, 140.0m, 140.0m)
    Source is at the center with coordinates [[100. 100. 100.]]
    Receiver (single receiver) is located at [[150. 150. 150.]]

$\displaystyle \left( 1, \  3\right)$

```python
solver = ElasticVTIWaveSolver(model, geometry, space_order=4)
ref_rec, ref_u, _ = solver.forward()
```

    Allocating memory for src_coords(1, 3)
    Allocating memory for src(2001, 1)
    Allocating memory for rec_coords(1, 3)
    Trying to allocate more memory for symbol v_x than available on physical device, this will start swapping
    Trying to allocate more memory for symbol v_y than available on physical device, this will start swapping
    Trying to allocate more memory for symbol v_z than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_xx than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_xy than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_xz than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_yy than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_yz than available on physical device, this will start swapping
    Trying to allocate more memory for symbol tau_zz than available on physical device, this will start swapping

```python
# Source and receiver coordinates
sx, sz = src_coordinates[0, :]
rx, rz = rec_coordinates[0, :]
```

### Convergence in time

```python
#NBVAL_IGNORE_OUTPUT
dt = [0.1000, 0.0800, 0.0750, 0.0625, 0.0500]
nnt = (np.divide(150.0, dt) + 1).astype(int)

for i in range(1, 5):
    # Time axis
    t0 = 0.0
    tn = 150.0
    time = np.linspace(t0, tn, nnt[i])

    # Source geometry
    src_coordinates = np.empty((1, 3))
    src_coordinates[0, :] = 100.

    # Single receiver offset 100 m from source
    rec_coordinates = np.empty((1, 3))
    rec_coordinates[:, :] = 150.

    geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates,
                                   t0=t0, tn=tn, src_type='Ricker', f0=f0, t0w=1.5/f0)

    # Note: incorrect data size will be generated here due to AcquisitionGeometry bug ...
    # temporarily fixed below by resizing the output from the solver
    geometry.resample(dt[i])
    print("geometry.time_axes; ", geometry.time_axis)

    solver = ElasticVTIWaveSolver(model, geometry, time_order=1, space_order=8)
    ref_rec1, ref_u1, _ = solver.forward(dt=dt[i])
    ref_rec1_data = ref_rec1.data[0:nnt[i],:]

    time1 = np.linspace(0.0, 3000., 20*(nnt[i]-1) + 1)
    U_t1 = analytical(20*(nnt[i]-1) + 1, model, time1, dt=time1[1] - time1[0])
    U_t1 = U_t1[0:nnt[i]]

    error_time[i] = np.linalg.norm(U_t1[:-1] - ref_rec1_data[:-1, 0], 2) / np.sqrt(nnt[i]-1)

    ratio_d = dt[i-1]/dt[i] if i > 0 else 1.0
    ratio_e = error_time[i-1]/error_time[i] if i > 0 else 1.0
    print("error for dt=%.4f is %12.6e -- ratio dt^2,ratio err; %12.6f %12.6f \n" %
          (dt[i], error_time[i], ratio_d**2, ratio_e))
    errors_plot.append((geometry.time_axis.time_values, U_t1[:-1] - ref_rec1_data[:-1, 0]))
```

# References

[1] Xu, S. and Liu, Y., 2019. Modeling 3D elastic VTI wave propagation using an optimal k-space operator-based temporal high-accuracy staggered-grid finite-difference scheme. Journal of Applied Geophysics, 170, p.103847.

[2] Wang, Y., Mu, P., Duan, Y. and Wang, T., 2018. Numerical Simulation of Elastic Wave Equation and Analysis of Wave Field Characteristics in 2-D VTI Medium. Open Journal of Yangtze Oil and Gas, 3(03), p.153.

[3] Bloot, R., Schleicher, J. and Santos, L.T., 2013. On the elastic wave equation in weakly anisotropic VTI media. Geophysical Journal International, 192(3), pp.1144-1155.


# Sampling implementation tests

```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import time
from planar_ising import PlanarIsingModelGenerator, InferenceAndSampling, PlanarIsingModel

np.random.seed(42)
```

# Problem statement

Let $V = \{ 1, ..., N \}, E \subseteq \binom{V}{2}, G = (V, E)$ be an undirected planar graph. For each $v \in V$ we assign a random binary variable (a spin) $s_v \in \{ -1, +1 \}$, $S = (s_1, ..., s_N)$. For each $e \in E$ we define a number $J_e \in \mathbb{R}$. We define a probability of the assignment $X = (x_1, ..., x_N) \in \{ -1, +1 \}^N$ as follows:

\begin{equation}
\mathbb{P} (S = X) = \frac{1}{Z} \exp{ \sum_{e = \{i, j\} \in E} J_e x_i x_j }
\label{eq:distr}
\end{equation}

Where

\begin{equation}
Z = \sum_{X \in \{ -1, +1 \}^N} \exp{ \sum_{e = \{i, j\} \in E} J_e x_i x_j }
\end{equation}

Given $G$'s plane embedding and a set of $J_e$, a goal is to draw samples from $\mathbb{P}(S = X)$ efficiently.

We will test two slightly different inference/sampling algorithms - `EfficientInferenceAndSampling` and `WilsonInferenceAndSampling`. Both are numerically correct, but the first one tends to draw samples faster on practice.

In all the experiments conducted further we work with the following model generating parameters (check `PlanarIsingModelGenerator` docstrings):

```python
graph_density = 1.0
interaction_values_std = 0.1
```

# Validation of sampling routine

It appears that having a sample $D = \{ (X_1, m_{X_1}), ..., (X_m, m_{X_m}) \}$ ($\{ X_i \}$ are distinct, $X_i$ enters sample $m_{X_i}$ times, $m_{X} = 0$ for $X$ not in sample) of spin configurations drawn from the distribution $\mathbb{P}(S = X)$, the $L_2$ error $\alpha$ between discrete distribution $\mathbb{P}(S = X)$ and empirical distribution $\mathbb{P}_{emp} (S = X) = \frac{m_X}{\sum_{i = 1}^m m_{X_i}}$ can be computed analytically. Indeed,

\begin{equation}
\begin{aligned}
\alpha^2 &= \sum_{X} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 \\
	&= \sum_{X \in D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 + \sum_{X \notin D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 \\
    &= \sum_{X \in D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 + \sum_{X \notin D} \mathbb{P}(S = X)^2 \\
    &= \sum_{X \in D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 + \sum_{X \notin D} \frac{1}{Z^2} \exp{ 2 \sum_{e = \{i, j\} \in E} J_e x_i x_j } \\
    &= \sum_{X \in D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 + \sum_{X \notin D} \frac{1}{Z^2} \exp{ \sum_{e = \{i, j\} \in E} (2 J_e) x_i x_j } \\
    &= \sum_{X \in D} (\mathbb{P}(S = X) - \mathbb{P}_{emp}(S = X))^2 + \frac{Z_2}{Z^2} - \sum_{X \in D} \frac{1}{Z^2} \exp{ \sum_{e = \{i, j\} \in E} (2 J_e) x_i x_j }
\label{eq:dist}
\end{aligned}
\end{equation}

where $Z_2$ is a partition function value associated with Ising Model on the same graph $G$ but with interaction values equal to $\{ 2J_e\}_{e \in E}$. Since we have implemented and tested partition function computation independently (check [notebook](https://github.com/ValeryTyumen/planar_ising/blob/master/tests/inference_tests.ipynb)), we can assume that we can compute $Z$ and $Z_2$ exactly. Two other sum terms in the formula are computed by one pass through the sample.

Showing that $\alpha$ approaches zero with the growth of sample size, we assure that our algorithm produces correct samples from $\mathbb{P}(S = X)$.

The next method implements empirical distribution L2 error for a given model and sample size.

```python
def compute_l2_error_between_empirical_and_theoretical_distribution(ising_model,
        sampled_configurations):

    sample_size = len(sampled_configurations)

    inference_and_sampling = InferenceAndSampling(ising_model)
    inference_and_sampling.prepare()
    log_partition_function = inference_and_sampling.compute_logpf()

    ising_model_doubled_interactions = PlanarIsingModel(ising_model.graph,
            2*ising_model.interaction_values)
    inference_doubled_interactions = InferenceAndSampling(ising_model_doubled_interactions)
    inference_doubled_interactions.prepare()
    log_partition_function_doubled_interactions = \
            inference_doubled_interactions.compute_logpf()

    l2_error_squared = np.exp(log_partition_function_doubled_interactions - \
            2*log_partition_function)

    configuration_statistics = {}

    for configuration in sampled_configurations:

        configuration_index = \
                (configuration + 1).dot(2**np.arange(ising_model.graph.size))//2

        if configuration_index not in configuration_statistics:

            minus_energy = ising_model.get_minus_energy(configuration)
            minus_energy_doubled_interactions = \
                    ising_model_doubled_interactions.get_minus_energy(configuration)

            configuration_statistics[configuration_index] = (1, minus_energy, \
                    minus_energy_doubled_interactions)

        else:

            configuration_statistics[configuration_index] = \
                    (configuration_statistics[configuration_index][0] + 1,
                     configuration_statistics[configuration_index][1],
                     configuration_statistics[configuration_index][2])

    for frequency, minus_energy, \
            minus_energy_doubled_interactions in configuration_statistics.values():

        l2_error_squared -= np.exp(minus_energy_doubled_interactions - \
                2*log_partition_function)
        l2_error_squared += (np.exp(minus_energy - log_partition_function) - \
                frequency/sample_size)**2

    return np.sqrt(l2_error_squared)
```

Now let's generate 3 Planar Ising Models of different scale and plot behaviour of $\alpha$ depending on sample size. Evident convergence to zero proves correctness of the sampling routine.

```python
plt.figure(figsize=(9, 6))

graph_sizes = [3, 10, 50]
log2_samples_sizes_per_graph_size = [np.arange(4, 17), np.arange(4, 12),
                                     np.arange(4, 10)]

for graph_size, log2_sample_sizes, color in zip(graph_sizes, log2_samples_sizes_per_graph_size,
                                                ['r', 'g', 'b']):

    ising_model = PlanarIsingModelGenerator.generate_random_model(graph_size,
            graph_density, interaction_values_std)

    sampling = InferenceAndSampling(ising_model)
    sampling.prepare(sampling=True)

    l2_errors = []

    for log2_sample_size in log2_sample_sizes:

        sample_size = 2**log2_sample_size

        sampled_configurations = sampling.sample_spin_configurations(sample_size)

        l2_error = compute_l2_error_between_empirical_and_theoretical_distribution(
                ising_model, sampled_configurations)

        l2_errors.append(l2_error)

    plt.plot(log2_sample_sizes, l2_errors, color, label='N={}'.format(graph_size))

plt.title('Empirical distribution L2-error')
plt.xlabel('$\log_2$(sample size)')
plt.ylabel('L2-error')
plt.legend()

plt.show()
```

# Execution time

The next plot shows time complexity of preprocessing and sampling steps depending on the size of the model. Each point of the scatter plot corresponds to the random Ising model of the given size. Logarithmic axes unveil solid $O(N^{\frac32})$ complexity of both preprocessing and sampling. The maximal model size exposed on the plot is $4096$, which speaks of framework scalability potential.

```python
log2_graph_sizes = np.arange(2, 13)
samples_count = 5

log2_graph_sizes_per_sample = []
preprocessing_times_per_sample = []
sampling_times_per_sample = []

for log2_graph_size in log2_graph_sizes:

    graph_size = 2**log2_graph_size

    for sample_index in range(samples_count):

        log2_graph_sizes_per_sample.append(log2_graph_size)

        ising_model = PlanarIsingModelGenerator.generate_random_model(graph_size,
                graph_density, interaction_values_std)

        start_moment = time.time()
        sampling = InferenceAndSampling(ising_model)
        sampling.prepare(sampling=True)
        preprocessing_time = time.time() - start_moment

        start_moment = time.time()
        sampling.sample_spin_configurations(1)
        sampling_time = time.time() - start_moment

        preprocessing_times_per_sample.append(preprocessing_time)
        sampling_times_per_sample.append(sampling_time)

    if log2_graph_size > 8:
        print('Done with size {}'.format(graph_size))
```

    Done with size 512
    Done with size 1024
    Done with size 2048
    Done with size 4096

```python
plt.figure(figsize=(9, 6))

plt.scatter(log2_graph_sizes_per_sample, np.log2(preprocessing_times_per_sample),
        c='r', s=16, label='preprocessing')

plt.scatter(log2_graph_sizes_per_sample, np.log2(sampling_times_per_sample),
        c='b', s=16, label='sampling')

plt.plot(log2_graph_sizes, 3*log2_graph_sizes - 10, '--', label='$O(N^3)$')
plt.plot(log2_graph_sizes, 1.5*log2_graph_sizes - 10, '--', label='$O(N^{1.5})$')

plt.title('Execution time - log scale')
plt.xlabel('$\log_2 (N)$')
plt.ylabel('$\log_2 (sec.)$')
plt.legend()

plt.show()
```

```python

```


## Chapter 4 -  Training Models

### Logistic Regression

Logistic regression / Logit regression is commonly used to estimate the probability that an instance belongs to a particular class. If the estimated probability is greater than 50%, then the model predicts that the instance belongs to that class. Otherwise it belongs to the complement class. This makes logistic regression a binary classifier.

The central problem is to model the relationship with a probability of an event occuring: $p(\mathbf x) = \Pr (y=1|\mathbf x)$ from the predictors $\mathbf x$. If we use the linear model

$$p(\mathbf x) = \beta_0 + \beta_1 x_1$$

then we could get values of the probability, $\Pr(\cdot)$ that is outside $[0,1]$.

Hence, we transform the linear model to the logistic model:

$$p(\mathbf x) = \frac{\exp(\beta_0 + \beta_1 x_1)}{1+\exp(\beta_0 + \beta_1 x_1)} = \frac{1}{1+\exp[-(\beta_0 + \beta_1 x_1)]}$$

from there, after the evaluation of the RHS, we obtain the result:

$$y = \begin{cases}1 \text{ if } \hat{p}(\mathbf x)\geq0.5\\0 \text{ if } \hat{p}(\mathbf x)<0.5\end{cases}$$

Note that $0.5$ is an arbitrary threshold. It can be shifted up and down in the range $(0,1)$.

After manipulation of the logit model, we find:
$$\frac{p(\mathbf x)}{1-p(\mathbf x)} = \exp(\beta_0 + \beta_1 x_1)$$

where the LHS is the odds and can take on any value between $0$ and $\infty$. If the odds are close to zero then there is a low probability of $y=1$. If the odds are very large then there $\Pr(y=1)$ is high.

By taking log on both sides, we obtain

$$\log \frac{p(\mathbf x)}{1-p(\mathbf x)} = \beta_0 + \beta_1 x_1$$

and so we say the logistic regression model has a logit or log-odds that is linear in $\mathbf x$.

In a logistic regression model, increasing $x_1$ by 1 unit changes the log odds by $\beta_1$. However, because the relationship between $p$ and $\mathbf x$ is not linear, a one unit increase in $x_1$ does not correspond to a $\beta_1$ change in $p$. The amount that $p$ changes due to a one unit change in $x_1$ will depend on the current value of $\mathbf x$.

#### Estimation using Maximum Likelihood

In order to obtain the estimates $\hat{\beta_0}$ and $\hat{\beta_1}$, we use maximum likelihood estimation. The intuition is to fit a model - seeking coefficient estimates such that the predicted probability of each observation in the training set, for each type of event, is the highest. We first obtain the likelihood function of the observed data:

$$L(\beta) = L(\beta_0, \beta_1) = \prod_{i=1}^n p(\mathbf x_i)^{y_i} \cdot(1-p(\mathbf x_i))^{1-y_i}$$

Observe that for each sample, the term in the product reduces to $p(\mathbf x_i)$ if $y_i=1$ and $1-p(\mathbf x_i)$ if $y_i=0$. From the likelihood function, we obtain the cost function in this problem - the log-likelihood function:

$$\begin{align}J(\Theta) = l(\beta) = l(\beta_0, \beta_1)
&= \sum_{i=1}^n y_i \log p(\mathbf x_i) + (1-y_i) \log (1-p(\mathbf x_i)) \\
&= \sum_{i=1}^n y_i \log p(\mathbf x_i) + \log (1-p(x_i)) - y_i \log(1-p(\mathbf x_i))  \\
&= \sum_{i=1}^n \log (1-p(\mathbf x_i)) + \sum_{i=1}^m y_i \log \frac{p(\mathbf x_i)}{1 - p(\mathbf x_i)}\\
&= \sum_{i=1}^n -\log (1+e^{\beta_0 + \beta_1x_i}) + \sum_{i=1}^m y_i ({\beta_0 + \beta_1x_i})\end{align}$$

There is no closed-form solution to the log likelihood for logistic regression so we perform gradient descent for each of the parameters to estimate.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
import statsmodels.api as sm
```

```python
df = pd.read_csv('Default.csv', index_col=0)
df['y'] = df['default'].apply(lambda x: 1 if x == 'Yes' else 0)
df['is_student'] = df['student'].apply(lambda x: 1 if x == 'Yes' else 0)
df['income_000'] = df['income'].apply(lambda x: x/1000)
```

```python
X1 = df[['balance']].copy()
train_df1 = df[['balance', 'y']].copy()
X2 = df[['balance', 'income_000', 'is_student']].copy()

y = df['y'].copy()
```

```python
# Logistic Regression using sklearn
reg = LogisticRegression()
reg.fit(X1, y)
print(reg.intercept_, reg.coef_)
```

    [-10.65132824] [[0.00549892]]

```python
# Logistic Regression using statsmodels
reg2 = sm.Logit(y, sm.add_constant(X1))
results2 = reg2.fit()
Theta_hat2 = list(results2.params)
print(["{:.6f}".format(t) for t in Theta_hat2])
```

    Optimization terminated successfully.
             Current function value: 0.079823
             Iterations 10
    ['-10.651331', '0.005499']

```python
print(results2.summary().tables[1])
```

    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const        -10.6513      0.361    -29.491      0.000     -11.359      -9.943
    balance        0.0055      0.000     24.952      0.000       0.005       0.006
    ==============================================================================

In this example, we see that $\hat{\beta_1}=0.0055$. This indicates that a one-unit increase in `balance` results in a 0.0055 unit increase in the log odds. An increase in balance is an associated with an increase in the probability of default.

Hypothesis testing can also be applied to logistic regression. The $z$-statistic is similar in concept to the $t$-statistic in linear regression. A large value of the $z$-statistic (and correspondingly a small $p$-value) indicates evidence in the null hypothesis, where $\beta_1$ is insignificant and near zero. Since the $p$-value for `balance` is small, we reject the null in favour of the alternate hypothesis. There is indeed an association between `balance` and `default`.

```python
# Prediction
results2.predict([[1,1000], [1,2000]])
```

    array([0.00575215, 0.58576937])

For prediction, the probability of someone defaulting with a balance of \\$1000 is $0.00576$ which is below 1\%. This increases drastically to $59\%$ when the balance increases to \\$2000.

```python
# Plotting for different alpha values
lin_X = np.linspace(X1['balance'].min(),X1['balance'].max(),1000).reshape(-1,1)
y_prob = results2.predict(sm.add_constant(lin_X))
y_prob1 = 1-y_prob
pred_df = pd.DataFrame({'X' : lin_X.flatten(), 'y0' : y_prob, 'y1' : y_prob1})
# display(pred_df.head())
fig, ax = plt.subplots(figsize=(10, 5))
train_df1[train_df1.y==0].plot(kind='scatter', x='balance', y='y', ax=ax, color='red')
train_df1[train_df1.y==1].plot(kind='scatter', x='balance', y='y', ax=ax, color='green', marker='X')
pred_df.plot(kind='line', x='X', y='y0', ax=ax, color='maroon')
pred_df.plot(kind='line', x='X', y='y1', ax=ax, color='green', style='--')
plt.show()
```

```python
# Using dummy variables for prediction
X3 = sm.add_constant(df[['is_student']].copy())
reg3 = sm.Logit(y, X3)
results3 = reg3.fit()
print(results3.summary2().tables[1])
```

    Optimization terminated successfully.
             Current function value: 0.145434
             Iterations 7
                   Coef.  Std.Err.          z     P>|z|    [0.025    0.975]
    const      -3.504128  0.070713 -49.554094  0.000000 -3.642723 -3.365532
    is_student  0.404887  0.115019   3.520177  0.000431  0.179454  0.630320

We can use dummy variables to assess for the `is_student` variable. Also note that for this variable `is_student`, the associated $p$-value is also small, indicating it is significant.

```python
results3.predict([[1,1], [1,0]])
# We can see that students have a higher probability of default compared to non-students.
```

    array([0.04313859, 0.02919501])

### Multivariate Logistic Regression

Multivariate logistic regression extends the above to have more than 1 predictor:

$$p(\mathbf x) = \frac{\exp(\Theta^T\mathbf x)}{1+\exp(\Theta^T\mathbf x)} = \frac{1}{1+\exp[-(\Theta^T\mathbf x)]}$$

where $\Theta = \begin{pmatrix}\beta_0, \beta_1, \cdots, \beta_p\end{pmatrix}^T$ and $\mathbf x = \begin{pmatrix}1, x_{1}, \cdots, x_{p}\end{pmatrix}^T$

Similarly, the cost function of multivariate logistic regression is:

$$\begin{align}J(\Theta) = l(\Theta)
&= \sum_{i=1}^n -\log (1+e^{\Theta^T\mathbf x}) + \sum_{i=1}^m y_i (\Theta^T\mathbf x)\end{align}$$

```python
# Multivar Logistic Regression using statsmodels
multivarreg = sm.Logit(y, sm.add_constant(X2))
results22 = multivarreg.fit()
Theta_hat21 = list(results22.params)
print(["{:.6f}".format(t) for t in Theta_hat21])
```

    Optimization terminated successfully.
             Current function value: 0.078577
             Iterations 10
    ['-10.869045', '0.005737', '0.003033', '-0.646776']

```python
print(results22.summary().tables[1])
```

    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const        -10.8690      0.492    -22.079      0.000     -11.834      -9.904
    balance        0.0057      0.000     24.737      0.000       0.005       0.006
    income_000     0.0030      0.008      0.370      0.712      -0.013       0.019
    is_student    -0.6468      0.236     -2.738      0.006      -1.110      -0.184
    ==============================================================================

In this model, we see that the $p$-value associated with `income` is very large. The coefficient for the dummy variable `is_student` is also negative, indicating students are less likely to default.

It is likely that the variables `is_student` and `balance` are correlated.

Note that logistic regression assume that the model and the output are linearly correlated.

### Softmax Regression

Logistic regression can be generalised to the softmax regression. Generalising the function, given that there are now $K$ different classes, the probability that the instance is in class $k$ is:

$$
\hat{p_k(\mathbf x)} = \frac{e^{\Theta^T_k \mathbf x_i}}{\sum_{j \in K} \exp{\Theta^T_j \mathbf x_i}}
$$

For the cost function, use the cross entropy cost function:

$$J(\Theta) = \sum_{i=1}^n\sum_{k=1}^K y_k^{(i)}\log \begin{pmatrix}\hat{p_k}^{(i)}\end{pmatrix}$$

and the gradient is:

$$\nabla_{\theta_k} J(\Theta) = \frac1m \sum_{i=1}^m \begin{pmatrix}\hat{p_k}^{(i)}-\hat{y_k}^{(i)}\end{pmatrix}\mathbf x_i$$

Similarly, compute the gradient vector for each class and use Gradient Descent to find the parameter matrix $\Theta$ that minimises the cost function.

```python
# Ingest
iris = datasets.load_iris()
Xiris = pd.DataFrame(iris['data'], columns=iris['feature_names'])
display(Xiris.head())
yiris = pd.Series([1 if t else 0 for t in list(iris['target']==2) ])
display(yiris.head())
feat2 = Xiris.iloc[:,2:4]
y2iris = iris['target']
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

      2
      4.7
      3.2
      1.3
      0.2

      3
      4.6
      3.1
      1.5
      0.2

      4
      5.0
      3.6
      1.4
      0.2

    0    0
    1    0
    2    0
    3    0
    4    0
    dtype: int64

```python
# Train
softmax_reg = LogisticRegression(multi_class='multinomial', solver='lbfgs', C=10)
softmax_reg.fit(feat2, y2iris)
```

    LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='multinomial', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)

```python
# Train using statsmodels
mnlogit = sm.MNLogit(y2iris, sm.add_constant(feat2))
results3 = mnlogit.fit(maxiter=90)
print(results3.summary())
```

    Warning: Maximum number of iterations has been exceeded.
             Current function value: 0.068545
             Iterations: 90
                              MNLogit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  150
    Model:                        MNLogit   Df Residuals:                      144
    Method:                           MLE   Df Model:                            4
    Date:                Tue, 19 May 2020   Pseudo R-squ.:                  0.9376
    Time:                        12:02:24   Log-Likelihood:                -10.282
    converged:                      False   LL-Null:                       -164.79
    Covariance Type:            nonrobust   LLR p-value:                 1.227e-65
    =====================================================================================
                  y=1       coef    std err          z      P>|z|      [0.025      0.975]
    -------------------------------------------------------------------------------------
    const               -74.7912   3.63e+04     -0.002      0.998   -7.12e+04    7.11e+04
    petal length (cm)    28.0872   1.53e+04      0.002      0.999   -2.99e+04       3e+04
    petal width (cm)     22.4123   2.17e+04      0.001      0.999   -4.25e+04    4.26e+04
    -------------------------------------------------------------------------------------
                  y=2       coef    std err          z      P>|z|      [0.025      0.975]
    -------------------------------------------------------------------------------------
    const              -120.0636   3.63e+04     -0.003      0.997   -7.12e+04     7.1e+04
    petal length (cm)    33.8417   1.53e+04      0.002      0.998   -2.99e+04       3e+04
    petal width (cm)     32.8590   2.17e+04      0.002      0.999   -4.25e+04    4.26e+04
    =====================================================================================

    /Users/bryanlim/.pyenv/versions/3.7.2/envs/botanic/lib/python3.7/site-packages/statsmodels/base/model.py:568: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
      "Check mle_retvals", ConvergenceWarning)

```python
# Predict
print(softmax_reg.predict([[5,2]]))
print(softmax_reg.predict_proba([[5,2]]))
```

    [2]
    [[6.38014896e-07 5.74929995e-02 9.42506362e-01]]

```python
results3.predict([[1,5,2]])
```

    array([[1.28938767e-50, 1.22039372e-02, 9.87796063e-01]])