# TTI pure qP-wave equation implementation

The aim of this notebook is to show how to solve the pure qP-wave equation using the finite-difference (FD) scheme. The 2D TTI pure qP-wave equation can be written as ([Mu et al., 2020](https://library.seg.org/doi/10.1190/geo2019-0320.1))

$$\begin{align}
\frac{1}{v_{p}^{2}}\frac{\partial^{2}p(\textbf{x},t)}{\partial t^{2}} = & \,\, (1+2\delta\sin^{2}\theta\cos^{2}\theta + 2\epsilon\cos^{4}\theta)\frac{\partial^{4}q(\textbf{x},t)}{\partial x^{4}} \\
& + (1+2\delta\sin^{2}\theta\cos^{2}\theta + 2\epsilon\sin^{4}\theta)\frac{\partial^{4}q(\textbf{x},t)}{\partial z^{4}} \\
&  + (2 - \delta\sin^{2}2\theta+3\epsilon\sin^{2}2\theta+2\delta\cos^{2}\theta)\frac{\partial^{4}q(\textbf{x},t)}{\partial x^{2}\partial z^{2}} \\
& +(\delta\sin4\theta-4\epsilon\sin2\theta\cos^{2}\theta)\frac{\partial^4 q(\textbf{x},t)}{\partial x^{3}\partial z} \\
& +(-\delta\sin4\theta-4\epsilon\sin2\theta\cos^{2}\theta)\frac{\partial^4 q(\textbf{x},t)}{\partial x\partial z^{3}} \\
& + f(\textbf{x}_{s},t),
\end{align}$$

$$
\frac{\partial^{2}q(\textbf{x},t)}{\partial x^{2}} + \frac{\partial^{2}q(\textbf{x},t)}{\partial z^{2}} = p(\textbf{x},t),
$$

where $q(\textbf{x},t)$ is an auxiliary wavefield, which is introduced for implementing the FD scheme.

First of all, it is necessary to import some Devito modules and other packages that will be used in the implementation. We set Devito logging `configuration ['log-level'] = 'DEBUG'` to view all processing times (i,e. compilation and execution of `Operators`)

```python
import numpy as np

from devito import (Function, TimeFunction, cos, sin, solve,
                    Eq, Operator, configuration, norm)
from examples.seismic import TimeAxis
from examples.seismic import RickerSource
from examples.seismic import Receiver
from examples.seismic import demo_model
from matplotlib import pyplot as plt

# Set logging to debug, captures statistics on the performance of operators
#configuration['log-level'] = 'INFO'
configuration['log-level']='DEBUG'
```

We will start with the definitions of the grid and the physical parameters $v_{p}, \theta, \epsilon, \delta$. For simplicity, we don't use any absorbing boundary conditions. We use a homogeneous model. The model is discretized with a grid of $101 \times 101$ and spacing of 10 m. The $v_{p}, \epsilon, \delta$ and $\theta$ parameters of this model are 3600 m∕s, 0.23, 0.17, and 45°, respectively.

```python
# NBVAL_IGNORE_OUTPUT

dtype = np.float32  # 32 bit floating point as the precision type
space_order = 8

shape   = (101,101) # 101x101 grid
spacing = (10.,10.) # spacing of 10 meters
origin  = (0.,0.)
nbl = 0  # number of pad points

model = demo_model('constant-tti', spacing=spacing, space_order=8,
                   shape=shape, nbl=nbl, dtype=dtype)

# initialize thomsem parameters to those used in Mu et al., (2020)
model.update('vp', np.ones(shape)*3.6)
model.update('epsilon', np.ones(shape)*0.23)
model.update('delta', np.ones(shape)*0.17)
model.update('theta', np.ones(shape)*(45.*(np.pi/180.)))
```

    Allocating memory for vp(117, 117)
    Operator `pad_vp` generated in 0.10 s
      * lowering.IET: 0.04 s (43.1 %)
         * specializing.IET: 0.02 s (21.6 %)
      * lowering.Clusters: 0.03 s (32.3 %)
         * specializing.Clusters: 0.02 s (21.6 %)
      * lowering.Expressions: 0.03 s (32.3 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `pad_vp` fetched `/tmp/devito-jitcache-uid1001/eb0b4836d1cb5c11bc20c5ed0a1f10bf88a26987.c` in 0.07 s from jit-cache
    Operator `pad_vp` ran in 0.01 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for epsilon(117, 117)
    Operator `pad_epsilon` generated in 0.10 s
      * lowering.IET: 0.04 s (42.3 %)
         * specializing.IET: 0.02 s (21.2 %)
      * lowering.Clusters: 0.04 s (42.3 %)
         * specializing.Clusters: 0.02 s (21.2 %)
      * lowering.Expressions: 0.03 s (31.8 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `pad_epsilon` fetched `/tmp/devito-jitcache-uid1001/5ade86409010323ebd717c48274e6de7214c63cd.c` in 0.04 s from jit-cache
    Operator `pad_epsilon` ran in 0.01 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for delta(117, 117)
    Operator `pad_delta` generated in 0.11 s
      * lowering.IET: 0.04 s (39.3 %)
      * lowering.Clusters: 0.04 s (39.3 %)
      * lowering.Expressions: 0.03 s (29.5 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `pad_delta` fetched `/tmp/devito-jitcache-uid1001/fd1f464d5ebb87cfc9c48d75069221f17c342191.c` in 0.06 s from jit-cache
    Operator `pad_delta` ran in 0.01 s
    Performance[mode=advanced] arguments: {}
    Allocating memory for theta(117, 117)
    Operator `pad_theta` generated in 0.11 s
      * lowering.IET: 0.04 s (38.8 %)
      * lowering.Clusters: 0.04 s (38.8 %)
      * lowering.Expressions: 0.03 s (29.1 %)
    Flops reduction after symbolic optimization: [0 --> 0]
    Operator `pad_theta` fetched `/tmp/devito-jitcache-uid1001/8e5b46910a68f6290d320d47ab24fbbf97232ca9.c` in 0.02 s from jit-cache
    Operator `pad_theta` ran in 0.01 s
    Performance[mode=advanced] arguments: {}

In cell below, symbols used in the PDE definition are obtained from the `model` object. Note that trigonometric functions proper of Devito are exploited.

```python
# Get symbols from model
theta = model.theta
delta = model.delta
epsilon = model.epsilon
m = model.m

# Use trigonometric functions from Devito
costheta  = cos(theta)
sintheta  = sin(theta)
cos2theta = cos(2*theta)
sin2theta = sin(2*theta)
sin4theta = sin(4*theta)
```

Accordingly to [Mu et al., (2020)](https://library.seg.org/doi/10.1190/geo2019-0320.1), the time sampling can be chosen as
$$
\Delta t < \frac{\Delta d}{\pi \cdot (v_{p})_{max}}\sqrt{\dfrac{1}{(1+\eta_{max}|\cos\theta-\sin\theta|_{max}^{2})}}
$$,

where $\eta_{max}$ denotes the maximum value between $|\epsilon|_{max}$ and $|\delta|_{max}$, $|cos\theta − sin\theta|_{max}$ is the maximum value of $|cos\theta − sin\theta|$.

```python
# NBVAL_IGNORE_OUTPUT

# Values used to compute the time sampling
epsilonmax = np.max(np.abs(epsilon.data[:]))
deltamax = np.max(np.abs(delta.data[:]))
etamax = max(epsilonmax, deltamax)
vmax = model._max_vp
max_cos_sin = np.amax(np.abs(np.cos(theta.data[:]) - np.sin(theta.data[:])))
dvalue = min(spacing)
```

    Allocating memory for n(1,)

The next step is to define the simulation time. It has to be small enough to avoid reflections from borders. Note we will use the `dt` computed below rather than the one provided by the property() function `critical_dt` in the `SeismicModel` class, as the latter only works for the coupled pseudoacoustic equation.

```python
# Compute the dt and set time range
t0 = 0.   #  Simulation time start
tn = 160. #  Simulation time end (0.16 second = 160 msec)
dt = (dvalue/(np.pi*vmax))*np.sqrt(1/(1+etamax*(max_cos_sin)**2)) # eq. above (cell 3)
time_range = TimeAxis(start=t0,stop=tn,step=dt)
print("time_range; ", time_range)
```

    time_range;  TimeAxis: start=0, stop=160.039, step=0.884194, num=182

In exactly the same form as in the [Cavity flow with Navier-Stokes]() tutorial, we will use two operators, one for solving the Poisson equation in pseudotime and one for advancing in time. But unlike what was done in such tutorial, in this case, we write the FD solution of the poisson equation in a manually way, without using the `laplace` shortcut and `solve` functionality (just to break up the routine and try to vary). The internal time loop can be controlled by supplying the number of pseudotime steps (`niter_poisson` iterations) as a `time` argument to the operator. A Ricker wavelet source with peak frequency of 20 Hz is located at center of the model.

```python
# NBVAL_IGNORE_OUTPUT

# time stepping
p = TimeFunction(name="p", grid=model.grid, time_order=2, space_order=2)
q = Function(name="q", grid=model.grid, space_order=8)

# Main equations
term1_p = (1 + 2*delta*(sintheta**2)*(costheta**2) + 2*epsilon*costheta**4)*q.dx4
term2_p = (1 + 2*delta*(sintheta**2)*(costheta**2) + 2*epsilon*sintheta**4)*q.dy4
term3_p = (2-delta*(sin2theta)**2 + 3*epsilon*(sin2theta)**2 + 2*delta*(cos2theta)**2)*((q.dy2).dx2)
term4_p = ( delta*sin4theta - 4*epsilon*sin2theta*costheta**2)*((q.dy).dx3)
term5_p = (-delta*sin4theta - 4*epsilon*sin2theta*sintheta**2)*((q.dy3).dx)

stencil_p = solve(m*p.dt2 - (term1_p + term2_p + term3_p + term4_p + term5_p), p.forward)
update_p = Eq(p.forward, stencil_p)

# Poisson eq. (following notebook 6 from CFD examples)
b = Function(name='b', grid=model.grid, space_order=2)
pp = TimeFunction(name='pp', grid=model.grid, space_order=2)

# Create stencil and boundary condition expressions
x, z = model.grid.dimensions
t = model.grid.stepping_dim

update_q = Eq( pp[t+1,x,z],((pp[t,x+1,z] + pp[t,x-1,z])*z.spacing**2 + (pp[t,x,z+1] + pp[t,x,z-1])*x.spacing**2 -
         b[x,z]*x.spacing**2*z.spacing**2) / (2*(x.spacing**2 + z.spacing**2)))

bc = [Eq(pp[t+1,x, 0], 0.)]
bc += [Eq(pp[t+1,x, shape[1]+2*nbl-1], 0.)]
bc += [Eq(pp[t+1,0, z], 0.)]
bc += [Eq(pp[t+1,shape[0]-1+2*nbl, z], 0.)]

# set source and receivers
src = RickerSource(name='src',grid=model.grid,f0=0.02,npoint=1,time_range=time_range)
src.coordinates.data[:,0] = model.domain_size[0]* .5
src.coordinates.data[:,1] = model.domain_size[0]* .5
# Define the source injection
src_term = src.inject(field=p.forward,expr=src * dt**2 / m)

rec  = Receiver(name='rec',grid=model.grid,npoint=shape[0],time_range=time_range)
rec.coordinates.data[:, 0] = np.linspace(model.origin[0],model.domain_size[0], num=model.shape[0])
rec.coordinates.data[:, 1] = 2*spacing[1]
# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=p.forward)

# Operators
optime=Operator([update_p] + src_term + rec_term)
oppres=Operator([update_q] + bc)

# you can print the generated code for both operators by typing print(optime) and print(oppres)
```

    Allocating memory for src(182, 1)
    Allocating memory for src_coords(1, 2)
    Allocating memory for rec_coords(101, 2)
    Operator `Kernel` generated in 1.68 s
      * lowering.Expressions: 0.73 s (43.6 %)
      * lowering.Clusters: 0.70 s (41.8 %)
         * specializing.Clusters: 0.59 s (35.2 %)
            * cire: 0.37 s (22.1 %)
    Flops reduction after symbolic optimization: [1047 --> 155]
    Operator `Kernel` generated in 0.12 s
      * lowering.Clusters: 0.05 s (44.8 %)
         * specializing.Clusters: 0.03 s (26.9 %)
      * lowering.IET: 0.05 s (44.8 %)
         * specializing.IET: 0.03 s (26.9 %)
    Flops reduction after symbolic optimization: [17 --> 10]

The time steps are advanced through a Python loop where both operators `optime` and `oppres`are called. Note the use of module indices to get proper buffers. As the operators will be applied at each step, the logging detail is expected to be too much. So, it is more convenient to switch from `DEBUG` to `INFO`

```python
# NBVAL_IGNORE_OUTPUT
configuration['log-level'] = 'INFO'

psave =np.empty ((time_range.num,model.grid.shape[0],model.grid.shape[1]))
niter_poisson = 1200

# This is the time loop.
for step in range(0,time_range.num-2):
    q.data[:,:]=pp.data[(niter_poisson+1)%2,:,:]
    optime(time_m=step, time_M=step, dt=dt)
    pp.data[:,:]=0.
    b.data[:,:]=p.data[(step+1)%3,:,:]
    oppres(time_M = niter_poisson)
    psave[step,:,:]=p.data[(step+1)%3,:,:]
```

    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.03 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s

```python
# Some useful definitions for plotting if nbl is set to any other value than zero
nxpad,nzpad = shape[0] + 2 * nbl, shape[1] + 2 * nbl
shape_pad   = np.array(shape) + 2 * nbl
origin_pad  = tuple([o - s*nbl for o, s in zip(origin, spacing)])
extent_pad  = tuple([s*(n-1) for s, n in zip(spacing, shape_pad)])
```

We can plot equally spaced snaps (by `factor`) from the full history saved in `psave` using matplotlib.

```python
# NBVAL_IGNORE_OUTPUT

# Note: flip sense of second dimension to make the plot positive downwards
plt_extent = [origin_pad[0], origin_pad[0] + extent_pad[0],
              origin_pad[1] + extent_pad[1], origin_pad[1]]

# Plot the wavefields, each normalized to scaled maximum of last time step
kt = (time_range.num - 2) - 1
amax = 0.05 * np.max(np.abs(psave[kt,:,:]))
print("amax; %12.6f" % (amax))

nsnaps = 10
factor = round(time_range.num/nsnaps)

fig, axes = plt.subplots(2, 5, figsize=(18, 7), sharex=True)
fig.suptitle("Snapshots", size=14)
for count, ax in enumerate(axes.ravel()):
    snapshot = factor*count
    ax.imshow(np.transpose(psave[snapshot,:,:]), cmap="seismic",
               vmin=-amax, vmax=+amax, extent=plt_extent)
    ax.plot(model.domain_size[0]* .5, model.domain_size[1]* .5, \
         'red', linestyle='None', marker='*', markersize=15, label="Source")
    ax.grid()
    ax.tick_params('both', length=2, width=0.5, which='major',labelsize=10)
    ax.set_title("Wavefield at t=%.2fms" % snapshot,fontsize=12)
for ax in axes[1, :]:
    ax.set_xlabel("X Coordinate (m)",fontsize=10)
for ax in axes[:, 0]:
    ax.set_ylabel("Z Coordinate (m)",fontsize=10)
```

## References

- **Least-squares reverse time migration in TTI media using a pure qP-wave equation** (2020)
<br> Xinru Mu, Jianping Huang, Jidong Yang, Xu Guo, and Yundong Guo
<br> Geophysics, Vol. 85, No. 4
<br> https://doi.org/10.1190/geo2019-0320.1


# Random walks on the spanning cluster

In this notebook we'll explore diffusion on the spanning cluster.

```python
import sys
sys.path.append("..")

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.ndimage
import tqdm
import sklearn.linear_model

from generate_spanning_cluster import get_spanning_cluster
from percwalk import percwalk, parallel_percwalk
from log_binning import log_bin
```

```python
sns.set(color_codes=True)
```

```python
# Critical percolation probability
p_c = 0.59275
```

## Distance moved on the percolating cluster

We'll start by exploring diffusion on the percolating cluster for $p > p_c$. We'll measure the second moment of the average distance, $\langle R^2 \rangle$, as a function of the number of steps performed. Below we'll demonstrate what the diffusion on the percolating cluster looks like.

```python
spanning_cluster = get_spanning_cluster(100, p_c)
```

Having created a system with a percolating cluster, we start by visualizing the spanning cluster.

```python
plt.figure(figsize=(14, 10))

plt.imshow(spanning_cluster, origin="lower")
plt.show()
```

Here we can see the sites contained in the percolating cluster. Moving on, we'll start a walker on a random site in the percolating cluster.

```python
num_walks = int(1e5)
num_steps = 0
```

```python
while num_steps <= 1:
    walker_map, displacements, num_steps = percwalk(spanning_cluster, num_walks)
```

```python
plt.figure(figsize=(14, 10))

plt.imshow(spanning_cluster, origin="lower")
# walker_map is oriented as row-column (ix, iy)
plt.plot(walker_map[1], walker_map[0])
plt.show()
```

In the above plot we can see how the walker moves in the spanning cluster. A thick line signifies several walks over a site.

## Mean squared distance

Next, we'll look at the mean squared distance, $\langle r^2 \rangle$, as a function of the number of walks, $N$, for varying $p > p_c$.
A free random walker behaves according to
\begin{align}
    \langle r^2 \rangle \propto D t,
\end{align}
as we've seen in the molecular dynamics simulations.
On the percolating cluster we expect that the random walker at times gets stuck in dangling ends or slows down through singly connected bonds.
We introduce the scaling theory
\beging{align}
    \langle r^2 \rangle = t^{2k} f\left[ (p - p_c) t^x \right],
\end{align}
for the mean squared distance on a system with random set sites.
We can categorize the function $f(u)$ for three different situations.
\begin{align}
    f(u) = \begin{cases}
        \text{const} & |u| \ll 1, \\
        u^{\mu} & u \gg 1, \\
        (-u)^{\beta - 2\nu} &  u \ll -1.
    \end{cases}
\end{align}
The first situation arises when $p \approx p_c$ and yields the scaling
\begin{align}
    \langle r^2 \rangle \propto t^{2k},
\end{align}
where $k < 1/2$.
This is known as _subdiffusion_ as opposed to the _superdiffusion_ observed for the initial molecular dynamics simulations.
The second situation $u \gg 1 \implies p > p_c$ and $t > t_0$, where $t_0$ is some threshold time.
This will yield the scaling theory
\begin{align}
    \langle r^2 \rangle \propto (p - p_c)^{\mu} t \propto t^{2k} \left[(p - p_c) t^{x} \right]^{\mu}.
\end{align}
The final situation is found when $p < p_c$ and $t > t_0$ yielding the scaling theory
\begin{align}
    \langle r^2 \rangle \propto (p_c - p)^{\beta - 2\nu} \propto t^{2k} \left[ (p_c - p) t^x \right]^{\beta - 2\nu}.
\end{align}
We will not be exploring the latter situation.
When $p > p_c$ and after a certain time $t_0$ has passed we expect the random walker on the cluster to behave as a normal free random walker.
That is, when $t \gg t_0$ we get
\begin{align}
    \langle r^2 \rangle \propto D t,
\end{align}
when $p > p_c$.
Stated differently, when $\langle r^2 \rangle \gg \xi^2$ the walker behaves as a free random walker.
The interesting fact here is that when $p = p_c$ we know that $\xi \to \infty$ and this condition will never be fullfilled.
This means that the scaling theory described above for the mean squared distance when $p = p_c$ will be true indefinitely.
We can summarize it by the function
\begin{align}
    \langle r^2 \rangle \propto \begin{cases}
        t^{2k} & \langle r^2 \rangle \ll \xi^2, \\
        D t & \langle r^2 \rangle \gg \xi^2.
    \end{cases}
\end{align}

We now simulate a system of size $L$ and compute the mean squared displacement on the spanning cluster several times for every system.

```python
L = 2048
num_walks = 1 * int(1e6)
num_walks_arr = np.arange(num_walks)
p_arr = np.linspace(p_c, 0.8, 12)
num_walkers = 500
num_systems = 20

r_squared = np.zeros((len(p_arr), num_walks))
```

It is important to note that the implementation for the percolation walker we use require $2L^2 > N_w$, where $N_w$ is the number of walks (or, time if you like) we let the walkers perform. This is so that the walker won't start backtracking from whence it came thereby flattening the displacement curve.

```python
for i, p in tqdm.tqdm_notebook(enumerate(p_arr), total=len(p_arr)):
    spanning_cluster = get_spanning_cluster(L, p, num_attempts=1000)
    r_squared[i] += parallel_percwalk(spanning_cluster, num_walks, num_walkers, num_systems)
```

    HBox(children=(IntProgress(value=0, max=12), HTML(value='')))

Having computed the mean squared displacement for several different values of $p \geq p_c$ we look at the diffusion behavior.
For $p = p_c$ we know that $\xi \to \infty$ and we know that
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k},
\end{align}
for all values of $t$ as we will never have $\langle r^2(t) \rangle \gg \xi^2$.
We therefore estimate $k$ by taking the logarithm on both sides of the equation and using linear regression.
\begin{gather}
    \log\left( \langle r^2(t) \rangle \right)
    = \log(C) + 2k \log(t),
\end{gather}
where $C$ is a constant serving as the intercept.

```python
plt.figure(figsize=(14, 10))

plt.loglog(num_walks_arr, r_squared[0])
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of $t$ for $p = p_c$")
plt.show()
```

In this figure we can see how the log-log plot of the mean squared distance as a function of time for $p = p_c$ becomes linear.
We now use linear regression to find an estimate for $k$.

```python
log_walks = np.log(num_walks_arr[1:])
log_r = np.log(r_squared[0, 1:])

clf = sklearn.linear_model.LinearRegression().fit(
    log_walks[:, np.newaxis], log_r[:, np.newaxis]
)

k = clf.coef_[0, 0] / 2
C = clf.intercept_[0]

print(f"k = {k}")
```

    k = 0.35000464022009997

Having found the intercept and the coefficient $k$ we plot $\langle r^2(t) \rangle$ as a function of the number of walks (time, in a sense) and compare it to the theoretical estimate.

```python
plt.figure(figsize=(14, 10))

plt.plot(
    num_walks_arr,
    r_squared[0],
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

plt.plot(
    num_walks_arr,
    np.exp(C) * num_walks_arr ** (2 * k),
    "--",
    label=r"$\langle r^2(t) \rangle = \exp(C) t^{2k}$",
)

plt.title(r"Plot of the mean squared displacement as a function of time $t$ for $p = p_c$")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.legend(loc="best")
plt.show()
```

In this figure we have plotted the mean squared displacement as a function of "time", i.e., the number of walks, on the spanning cluster for $p = p_c$.
We observe the expected subdiffusion.
We also see how the theoretical estimate of the subdiffusion follows the data closely.
Below we plot the mean squared distance for $p \geq p_c$.

```python
plt.figure(figsize=(14, 10))

plt.loglog(
    num_walks_arr,
    r_squared[0],
    "--",
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

for i, p in enumerate(p_arr[1:]):
    plt.loglog(
        num_walks_arr,
        r_squared[i + 1],
        label=rf"$p = {p:.3f}$"
    )

plt.legend(loc="best")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of time for $p \geq p_c$")
plt.show()
```

In the above figure we see how after a time $t_0$ the mean squared distance for $p > p_c$ deviates from the situation when $p = p_c$.
Reformulating the scaling theory for $p > p_c$ to create a data collapse we get
\begin{gather}
    \langle r^2 \rangle \propto t^{2k} (p - p_c) t^{x}
    \implies
    t^{-2k} \langle r^2 \rangle \propto (p - p_c) t^{x}.
\end{gather}
We plot this on a log-log axis when creating the data collapse.
However, the challenge is to find an expression for $x$.
We have previously found that
\begin{align}
    \langle r^2(t) \rangle \propto D(p) t,
\end{align}
where $D(p) = (p - p_c)^{\mu}$ when $p > p_c$ and $r \gg \xi$.
Inserted into the scaling theory we can write
\begin{gather}
    (p - p_c)^{\mu} t \propto t^{2k} \left[(p - p_c) t^{x}\right]^{\mu}
    \implies
    1 = 2k + \mu x
    \implies
    x = \frac{1 - 2k}{\mu}.
\end{gather}
Having already found $k$ we are thus tasked with finding an expression for $\mu$ and we can be on our merry way to plotting a data collapse.

We know that when $t < t_0$ the mean squared distance scales as $t^{2k}$ when $p = p_c$ and $p > p_c$.
However, when $t > t_0$ this scaling continues only for $p = p_c$ whereas for $p > p_c$ the mean squared distance will increase faster in time.
We are interested in locating $t_0$ and we do this by finding the point where
\begin{align}
    \langle r^2(t; p_c) \rangle \leq \frac{1}{2} \langle r^2(t; p > p_c) \rangle,
\end{align}
that is, we find the point where the scaling is more than twice as high.
At $t = t_0$ we know that $\langle r^2(t_0) \rangle = \xi^2$.
We are lucky enough to have a scaling ansatz for the characteristic cluster length stating
\begin{align}
    \xi = \xi_0 (p - p_c)^{-\nu}.
\end{align}

```python
plt.figure(figsize=(14, 10))

plt.loglog(
    num_walks_arr,
    r_squared[0],
    "--",
    label=rf"$p = p_c = {p_c}$",
)
plt.fill_between(num_walks_arr, r_squared[0], alpha=0.2)

xi_squared_list = []
t_list = []
ind_list = []
p_list = []

for i, p in enumerate(p_arr[1:]):
    plt.loglog(
        num_walks_arr,
        r_squared[i + 1],
        label=rf"$p = {p:.3f}$"
    )

    mask = r_squared[0, 1:] <= 0.5 * r_squared[i + 1, 1:]

    if not np.any(mask):
        continue

    index = np.argmax(mask)

    xi_squared_list.append(r_squared[i + 1, index])
    t_list.append(num_walks_arr[index])
    ind_list.append(i + 1)
    p_list.append(p)

    plt.scatter(num_walks_arr[index], r_squared[i + 1, index])

plt.legend(loc="best")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.title(r"Plot of the mean squared distance as a function of time for $p \geq p_c$")
plt.show()
```

It is interesting to note that for a larger value of $p > p_c$ the cross-over time $t_0$ becomes smaller.
That is, for a higher percolation probability $p$ the time it takes for the system to behave as a "normal" random walker becomes shorter.
This also makes sense as there are fewer sites not contained in the spanning cluster.

Having found the points $(t_0, \xi^2)$ we now find an expression for $\nu$.
Taking the logarithm on both sides of the scaling ansatz for the characteristic length yields
\begin{align}
    \log(\xi) = \log(\xi_0) - \nu \log(p - p_c).
\end{align}
We use linear regression to find an expression for the intercept $\log(\xi_0)$ and the exponent $\nu$.

```python
p_min_pc_arr = np.abs(np.array(p_list) - p_c)

clf = sklearn.linear_model.LinearRegression().fit(
    np.log(p_min_pc_arr)[:, np.newaxis],
    np.log(np.sqrt(xi_squared_list))[:, np.newaxis],
)

nu = -clf.coef_[0, 0]
log_xi_0 = clf.intercept_[0]

print(f"nu = {nu}")
```

    nu = 1.2067707882240395

The exact value of $\nu$ in two dimensions is $\nu = 4/3$.

```python
nu_exact = 4 / 3
print(f"nu_exact = {nu_exact}")
```

    nu_exact = 1.3333333333333333

We therefore see that we are in ballpark range.
We plot the characteristic length as a function of $p$ and compare it to the theoretical scaling using our estimate of $\nu$.

```python
fig = plt.figure(figsize=(14, 10))

plt.plot(
    p_list,
    np.sqrt(xi_squared_list),
    "-o",
    label=r"$Measured$",
)
plt.plot(
    p_list,
    np.exp(log_xi_0) * p_min_pc_arr ** (-nu),
    "-o",
    label=r"$(p - p_c)^{-%.3f}$" % nu
)
#plt.plot(
#    p_list,
#    np.exp(log_xi_0) * p_min_pc_arr ** (-nu_exact),
#    "-o",
#    label=r"$(p - p_c)^{-%.3f}$" % nu_exact
#)
plt.xlabel(r"$p$")
plt.ylabel(r"$\xi$")
plt.title(r"Plot of the characteristic length as a function of the percolation probability $p$")
plt.legend(loc="best")
plt.show()
```

In this figure we see how our estimate of $\nu$ follows the measured data quite well.
And, more importantly, they follow the theoretical scaling using the exact value of $\nu$ closely.
Looking at the point where the cross-over time occurs, we see that this scales inversely with increasing $p$.
We know that for $p > p_c$ and $t < t_0$ we have the relation
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k},
\end{align}
that is the system experiences subdiffusion.
When $t > t_0$ the relation takes on the form
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k}\left[(p - p_c) t^x \right]^{\mu}.
\end{align}
However, at the cross-over time, that is when $t = t_0$, we expect these to be proportional.
\begin{gather}
    t_0^{2k} \propto
    t_0^{2k}\left[(p - p_c) t_0^{x} \right]^{\mu}
    \implies
    (p - p_c) t_0^{x} \propto 1
    \implies
    t_0 \propto |p - p_c|^{-1/x}.
\end{gather}

```python
neg_inv_x, log_C = np.polyfit(np.log(p_min_pc_arr), np.log(t_list), deg=1)

x = -1 / neg_inv_x
print(f"x = {x}")
```

    x = 0.29554963750661734

```python
fig = plt.figure(figsize=(14, 10))

plt.plot(
    p_list,
    t_list,
    "-o",
    label=r"Measured",
)

plt.plot(
    p_list,
    np.exp(log_C) * p_min_pc_arr ** (-1 / x),
    "-o",
    label=r"$t_0 \propto |p - p_c|^{-1/x}$"
)

plt.xlabel(r"$p$")
plt.ylabel(r"$t_0$")
plt.title(r"Plot of the scaling of the cross-over time as a function for $p$")
plt.legend(loc="best")
plt.show()
```

In order to find an estimate for $x$ we need to find a value for $\mu$.
As we've stated earlier, when $p > p_c$ and $t \gg t_0$ which means that $r \gg \xi$ the random walker moves away from the anomalous diffusion and instead behaves as a free random walker with the relation
\begin{align}
    \langle r^2(t) \rangle = D(p) t.
\end{align}
The Einstein relation for diffusion relates the diffusion constant to the conductance through
\begin{align}
    D(p) \propto \sigma(p) \propto (p - p_c)^{\mu}.
\end{align}

```python
plt.figure(figsize=(14, 10))

diffusion_list = []
intercept_list = []
p_min_pc_diff_list = []

for i, p_i in enumerate(ind_list):
    t_0 = t_list[i]
    p = p_arr[p_i]

    p_min_pc_diff_list.append(p - p_c)

    cross = num_walks_arr > t_0

    t_cross = num_walks_arr[cross]
    r_cross = r_squared[p_i, cross]

    d, intercept = np.polyfit(t_cross, r_cross, deg=1)
    diffusion_list.append(d)
    intercept_list.append(intercept)

    plt.plot(t_cross, r_cross, label=fr"$p = {p:.3f}$")

plt.title(r"Plot of the mean squared distance as a function of $t > t_0$")
plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t) \rangle$")
plt.legend(loc="best")
plt.show()
```

Ideally we expect the mean squared distance to be completely linear, however, we see that even though $t > t_0$ there are slight deviations to this trend.

Taking the logarithm on both sides of the scaling ansatz we are able to estimate $\mu$.
\begin{align}
    \log(D) \propto \mu \log(p - p_c).
\end{align}
Again we use the least squares method to estimate $\mu$.

```python
mu, intercept = np.polyfit(np.log(p_min_pc_diff_list), np.log(diffusion_list), deg=1)
```

```python
print(mu)
```

    0.8923720197088654

From the lecture notes we know that the exact value of $\mu$ is given by $\mu = 1.3$.

```python
mu_exact = 1.3
```

This shows that we miss by quite a lot when it comes to estimating $\mu$.

```python
plt.figure(figsize=(14, 10))

plt.plot(
    p_min_pc_diff_list,
    diffusion_list,
    "-o",
    label=r"Measured",
)
plt.plot(
    p_min_pc_diff_list,
    np.exp(intercept) * np.array(p_min_pc_diff_list) ** (mu),
    "-o",
    label=r"$(p - p_c)^{%.3f}$" % mu,
)
plt.xlabel(r"$p - p_c$")
plt.ylabel(r"$D(p)$")
plt.title(r"Plot of the diffusion constant as a function of $p - p_c$")
plt.legend(loc="best")
plt.show()
```

In this figure we've plotted the diffusion constant as a function of $p - p_c$.
We compare the measured values with the theoretical scaling using our estimate of $\mu$.

```python
x = (1 - 2 * k) / mu
print(f"x = {x}")
```

    x = 0.3361722610460954

```python
p_min_pc = p_arr[1:] - p_c
r_gg_xi = r_squared[1:, 1:]
t_sq = num_walks_arr[1:] ** (-2 * k)
```

```python
plt.figure(figsize=(14, 10))

for i in range(len(p_min_pc)):
    plt.plot(
        p_min_pc[i] * num_walks_arr[1:] ** (x),
        t_sq * r_gg_xi[i],
        label=fr"$p = {p_arr[i + 1]:.3f}$",
    )
    #plt.plot(
    #    p_min_pc[i] * num_walks_arr[1:] ** (x),
    #    np.exp(intercept) * (p_min_pc[i] * num_walks_arr[1:] ** (x)) ** mu,
    #    label=fr"Theoretical: $p = {p_arr[i + 1]:.3f}$",
    #)

plt.legend(loc="best")
plt.xlabel(r"$(p - p_c) t^{x}$")
plt.ylabel(r"$t^{-2k} \langle r^{2} (t)\rangle$")
plt.title(r"Data-collapse plot for mean squared distance")
plt.show()
```

In this figure we can see the data-collapse of the mean squared displacement.
We have compared to the theoretical scaling
\begin{gather}
    \langle r^2 \rangle \propto t^{2k} (p - p_c) t^{x}
    \propto t^{2k}\left[(p - p_c) t^{x}\right]^{\mu},
\end{gather}
where we use the intercept from the diffusion constant as the constant of proportionality.
We note that collapse isn't completely perfect, especially when $p$ is close to $p_c$.
A possible explanation to this is due to the percolation walker being restricted to a finite spanning cluster.
We might be able to alleviate some of this discrepancy if we allow the spanning cluster to have periodic boundaries, however this will require quite a lot of manipulation when labelling the spanning cluster and for the percolation walker and has therefore not been tested.

Finally we are interested in estimating the dimension of the random walk, $d_w$.
We have that
\begin{align}
    d_w = \frac{1}{k}.
\end{align}
We can also estimate $d_w$ from the scaling relation
\begin{align}
    \langle r^2(t) \rangle \propto t^{2k} = t^{2/d_w},
\end{align}
when we $t < t_0$ for $p > p_c$.

```python
d_w_list = []
d_w_intercept_list = []
p_min_pc_d_w_list = []

for i, p_i in enumerate(ind_list):
    t_0 = t_list[i]
    p = p_arr[p_i]

    p_min_pc_d_w_list.append(p - p_c)

    pre_cross = num_walks_arr < t_0

    t_pre_cross = num_walks_arr[pre_cross]
    r_pre_cross = r_squared[p_i, pre_cross]

    d_w_2_inv, d_w_intercept = np.polyfit(t_pre_cross, r_pre_cross, deg=1)
    d_w_list.append(2 / d_w_2_inv)
    d_w_intercept_list.append(d_w_intercept)
```

```python
fig = plt.figure(figsize=(14, 10))

plt.plot(p_list, d_w_list, "-o")
plt.xlabel(r"$p$")
plt.ylabel(r"$d_w$")
plt.title(r"Plot of the dimension of the random walk as a function of $p$")
plt.show()
```


```python
import pandas as pd
import sympy as sym
sym.init_printing()
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import control
```

# Tarea: LGDR
**Nombre completo**:

*Fulano de Tal*

Considere el proceso modelado por

$$G_p(s) =  \frac{5}{s(s+4)(s+10)}$$

Suponga que los polos dominantes en lazo cerrado están en $-3 \pm j3$

1. Diseñe un compensador en adelanto que cumpla con el requerimiento.

```python
# código necesario para su desarrollo

```

Respuestas

2.Diseñe un compensador en atraso que disminuya a la décima parte el error de estado estable ante una entrada rampa.

```python
# código necesario para su desarrollo

```

Respuestas

3. Analice y compare el LGDR del proceso con el del sistema controlado con el compensador en adelanto y el sistema controlado con el  compensador en adelanto-atraso.

```python
# código necesario para su desarrollo

```

Respuestas

4. Analice y compare la respuesta temporal del proceso con el del sistema controlado con el compensador en adelanto y el sistema controlado con el  compensador en adelanto-atraso.

```python
# código necesario para su desarrollo

```

Respuestas

5. Conclusiones
- conclusión 1.
- conclusión 2.


# Appendix B.1 $\quad$ Complex Numbers

A **complex number** $c$ is of the form $\underline{\hspace{1in}}$, where
-

-

-

### Example 1

For symbol $i = \sqrt{-1}$, we have the following:
\begin{equation*}
  i^2 = ~~~~~~~~~i^3 = ~~~~~~~~~i^4 = ~~~~~~~~~i^5 = ~~~~~~~~~i^6 = ~~~~~~~~~i^7 = ~~~~~~~~~
\end{equation*}

```python
1j**2, 1j**3, 1j**4, 1j**5, 1j**6, 1j**7
```

    ((-1+0j), (-0-1j), (1+0j), 1j, (-1+0j), (-0-1j))

## Operations

Let $c_1 = a_1+b_1i$ and $c_2 = a_2+b_2i$ be complex numbers.
- $c_1+c_2 = $

- $c_1-c_2 = $

- $c_1c_2 = $

If $c = a+bi$ is a complex number, then the **conjugate** of $c$ is
- $\bar{\bar{c}} = $

- $\overline{c_1+c_2} = $

- $\overline{c_1c_2} = $

- if $\overline{c} = c$, then

- $c\bar{c} = $

If $c_1 = a_1+b_1i$ and $c_2 = a_2+b_2i$ , then

$\dfrac{c_1}{c_2} = $

### Example 2

Let $c_1 = 2-5i$, $c_2 = -3+4i$.
\begin{equation*}
  c_1 + c_2 = \hspace{4in}
\end{equation*}
\begin{equation*}
  c_1 - c_2 = \hspace{4in}
\end{equation*}

\begin{equation*}
  c_1 c_2 = \hspace{4in}
\end{equation*}

\begin{equation*}
  \frac{c_1}{c_2}= \hspace{4in}
\end{equation*}

```python
c1 = 2 - 5j; c2 = -3 + 4j;

c1 + c2, c1 - c2, c1 * c2, c1 / c2
```

    ((-1-1j), (5-9j), (14+23j), (-1.04+0.28j))

### Example 3

Find the value of following quantities
\begin{equation*}
  \frac{1}{2+3i} = \hspace{4in}
\end{equation*}

\begin{equation*}
  \frac{1}{i} =\hspace{4in}
\end{equation*}

```python
1 / (2 + 3j), 1 / 1j
```

    ((0.15384615384615385-0.23076923076923078j), -1j)

The **absolute value** or **modulus** of a complex number $c = a+bi$ is

**Remark:** $c\bar{c} = $

## Matrices with Complex Entries

### Example 4

Let
\begin{equation*}
  A = \left[
        \begin{array}{cc}
          4+i & -2+3i\\
          6+4i & -3i \\
        \end{array}
      \right]\quad
  B = \left[
        \begin{array}{cc}
          2-i & 3-4i\\
          5+2i & -7+5i \\
        \end{array}
      \right]\quad
  C = \left[
        \begin{array}{cc}
          1+2i & i\\
          3-4i & 8 \\
          4+2i & 1-i\\
        \end{array}
      \right]
\end{equation*}
Find $A+B$, $CA$, and $(2+i)B$.

```python
from sympy import *

A = Matrix([[4+1j, -2+3j], [6+4j, -3j]]);
B = Matrix([[2-1j, 3-4j], [5+2j, -7+5j]]);
C = Matrix([[1+2j, 1j], [3-4j, 8], [4+2j, 1-1j]]);

A + B, C.multiply(A), (2+1j) * B
```

    (Matrix([
     [         6.0,  1.0 - 1.0*I],
     [11.0 + 6.0*I, -7.0 + 2.0*I]]), Matrix([
     [        1.0*I*(6.0 + 4.0*I) + (1.0 + 2.0*I)*(4.0 + 1.0*I),                  3.0 + (-2.0 + 3.0*I)*(1.0 + 2.0*I)],
     [              48.0 + (3.0 - 4.0*I)*(4.0 + 1.0*I) + 32.0*I,              -24.0*I + (-2.0 + 3.0*I)*(3.0 - 4.0*I)],
     [(1.0 - 1.0*I)*(6.0 + 4.0*I) + (4.0 + 1.0*I)*(4.0 + 2.0*I), -3.0*I*(1.0 - 1.0*I) + (-2.0 + 3.0*I)*(4.0 + 2.0*I)]]), Matrix([
     [(2.0 - 1.0*I)*(2.0 + 1.0*I),  (2.0 + 1.0*I)*(3.0 - 4.0*I)],
     [(2.0 + 1.0*I)*(5.0 + 2.0*I), (-7.0 + 5.0*I)*(2.0 + 1.0*I)]]))

If $A = [a_{ij}]$, then the **conjugate** of $A$ is

## Properties of Conjugate of a Matrix

- $\overline{\overline{A}}=$

- $\overline{A+B}=$

- $\overline{AB}=$

- For any complex number $c$, $\overline{cA}=$

- $(\overline{A})^T=$

An $n\times n$ complex matrix $A$ is called **Hermitian** if

For example,
$A = \left[
        \begin{array}{cc}
          2 & 3+i\\
          3-i & 5 \\
        \end{array}
      \right]$


# Iris data processing

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

### import load_iris function from datasets module

```python
from sklearn.datasets import load_iris
```

### save "bunch" object containing iris dataset and iits attributes

```python
iris = load_iris()
type(iris)
```

### print the names of the four features

```python
print (iris.feature_names)
```

### print the integers representing the species of each observation

```python
print (iris.target)
```

```python
# Extract the values for features and create a list called featuresAll
featuresAll=[]
features = iris.data[: , [0,1,2,3]]
features.shape
```

```python
# Extract the values for targets
targets = iris.target
targets.reshape(targets.shape[0],-1)
targets.shape
```

```python

```


# hw 6: estimators
learning objectives:
* solidify "what is an estimator"
* evaluate the effectiveness of an estimator computationally (through simulation)
* understand the notion of unbiasedness, consistency, and efficiency of an estimator and evaluate these qualities computationally (through simulation)

```julia
using StatsBase
using Random
using Statistics
using PyPlot
using PyCall

sns = pyimport("seaborn")
sns.set_context("talk");
```

    ┌ Info: Recompiling stale cache file C:\Users\cartemic\.julia\compiled\v1.2\StatsBase\EZjIG.ji for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
    └ @ Base loading.jl:1240

### what is the most important problem in your field?

(0) It is important to effectively communicate with people from different fields. Introduce yourself to someone in the class from a different field of study (someone you haven't met). Ask them, "what is the most important problem in your field of study?". Then, argue/explain what you think is the most important problem in *your* field of study.

Write the name of the student you spoke with and their field of study.

I'd also be interested to read about what you think is the most important problem in your field (a few sentences), but this is not required.

("field of study" can be as general as "chemistry" or as specific as "self-assembly of nanoparticles".)

I spoke with Heather Miller, whose field is humanitarian engineering

### counting
(1) A biologist takes a random sample of six fish from a lake. The lake has ten distinct species of fish. It is possible for the biologist to select more than one fish of any given species. e.g., one outcome is: one fish of species $X$, two fish of species $Y$, and four fish of species $Z$. How many different samples could the biologist draw from the lake?
* the sampling is done without replacement, but this is irrelevant since the lake contains more than six fish of each species
* the order in which the fish are sampled does not matter
* fish of a given species are indistinguishable

Essentially what this question comes down to is that we have 10 distinct bins (i.e. species), and we want to know how many ways we can split up 6 particles (samples) between them. Pretty neat how partitioning just kind of... works for multisets like that. Anyway, from our class notes there are ${n + k - 1 \choose k - 1}$ ways to partition $n$ particles into $k$ bins. Interestingly, there are also ${n + k - 1 \choose n}$ ways to partition the same $n$ particles into the same $k$ bins because

$${{n + k - 1} \choose k - 1} = \frac{(n + k - 1)!}{(k-1)!(n+k-1-k+1)!} = \frac{(n+k-1)!}{(k-1)!n!}$$
is equivalent to
$${n + k - 1 \choose n} = \frac{(n + k - 1)!}{n!(n+k-1-n)!} = \frac{(n+k-1)!}{n!(k-1)!}$$
which means that, generally, $${m + n \choose m} = {m + n \choose n}$$ which is pretty cool.

```julia
function n_choose_k(n::Int, k::Int)
    # using multiple divide operations to avoid overflow for larger values
    # of n, k
    return Int(factorial(n) / factorial(k) / factorial(n - k))
end
@assert n_choose_k(7, 5) == 21

sample_size = 6
num_species = 10

n = sample_size + num_species - 1
k = sample_size

fish_partitions = n_choose_k(n, k)
println("There are $fish_partitions samples of fish that the biologist could collect.")
```

    There are 5005 samples of fish that the biologist could collect.

### capture, mark, release, recapture

In ecology, one wishes to estimate the size of a population (e.g. turtles). It is too costly and impractical to count *every* member of a population. One strategy is to:
1. capture a random sample from the population, without replacement
2. mark/tag each member of this random sample
3. release the marked sample back into the population
4. after sufficient time has passed, recapture another sample
5. count the number of marked members from the recaptured sample

We assume that:
* when the captured and marked sample is released back into the wild, they randomly (homogenously) mix with the rest of the (unmarked) population before we recapture
* marking a member of the population does not change its likelihood of being recaptured
* the time between capture/mark/release and recapture/count is short enough to neglect deaths, births, and migration out of the population

define the variables:
* $n$: the total number of turtles in the population (unknown)
* $k$: number of turtles captured, marked, and released in the first phase
* $k_r$: number of turtles recaptured
* $m$: the number of recaptured turtles that had marks on them

##### Lincoln-Petersen estimator
A very intuitive estimator is found by imposing that the proportion found marked in the recaptured sample is equal to the proportion of the population that was captured/marked/released in the first phase.

\begin{equation}
\frac{k}{n}=\frac{m}{k_r}
\end{equation}

giving the Lincoln-Petersen estimator for the population size $n$:
\begin{equation}
\hat{n} = \frac{k k_r}{m}
\end{equation}

##### Chapmen estimator
Chapmen derived a different estimator that we will compare to the Lincoln-Petersen estimator below.
\begin{equation}
\hat{n} = \frac{(k + 1) (k_r +1)}{m+1} - 1
\end{equation}

#### estimating the population of turtles on a small island
We wish to estimate the population of turtles on a small island by a capture, mark, release, and recapture strategy.

(2) create a mutable data structure `Turtle` that represents a turtle in the population. It should have a single attribute, `marked`, that indicates whether it has been marked or not.

```julia
mutable struct Turtle
    marked::Bool

    function Turtle()
        # new turtles aren't marked
        new(false)
    end
end

@assert !Turtle().marked
```

(3) write a function `create_population(nb_turtles::Int)` that creates a population of `nb_turtles` unmarked `Turtle`s. return the turtles as an `Array{Turtle}`.

```julia
function create_population(nb_turtles::Int)
    return [Turtle() for _ in 1:nb_turtles]
end

@assert create_population(4) isa Array{Turtle}
```

(4) write a function `count_marked(turtles::Array{Turtle})` that takes in a population of turtles and returns the number of these turtles that are marked.

```julia
turtles = create_population(500)
count_marked(turtles) # should return zero
```

```julia
function count_marked(turtles::Array{Turtle})
    sum([t.marked for t in turtles])
end

turtles = create_population(500)
@assert count_marked(turtles) == 0
[turtles[i].marked = true for i in 1:5]
@assert count_marked(turtles) == 5
# put the test turtles back in the pond
[t.marked = false for t in turtles];
```

(5) write a function `capture_mark_release!` that takes in two arguments:
* `turtles::Array{Turtle}` the population of turtles
* `nb_capture_mark_release::Int` the number of turtles to randomly capture (select without replacement) and mark

and modifies the `marked` attribute of `nb_capture_mark_release` randomly selected turtles in `turtles` to denote that they have been marked. think about why the function has an `!`.

```julia
turtles = create_population(500)
capture_mark_release!(turtles, 45)
count_marked(turtles) # should return 45
```

```julia
function capture_mark_release!(
        turtles::Array{Turtle},
        nb_capture_mark_release
    )
    selected = sample(turtles, nb_capture_mark_release, replace=false)

    for t in selected
        t.marked = true
    end
end

turtles = create_population(100)
capture_mark_release!(turtles, 30)
@assert count_marked(turtles) == 30
```

(6) write a function `recapture(turtles::Array{Turtle}, nb_recapture::Int)` that returns a random sample (without replacement) of `nb_recapture` turtles from `turtles` in the form of an `Array{Turtle}`.

```julia
turtles = create_population(500)
capture_mark_release!(turtles, 45)
recaptured_turtles = recapture(turtles, 50) # should return Array{Turtle} with 50 elements
```

```julia
function recapture(turtles::Array{Turtle}, nb_recapture::Int)
    return sample(turtles, nb_recapture, replace=false)
end

turtles = create_population(500)
capture_mark_release!(turtles, 45)
@assert length(recapture(turtles, 50)) == 50
```

(7) write two functions, one for each estimator:

* `chapman_estimator(nb_capture_mark_release::Int, nb_recapture::Int, nb_marked_in_recaptured::Int)`
* `lincoln_petersen_estimator(nb_capture_mark_release::Int, nb_recapture::Int, nb_marked_in_recaptured::Int)`

that each take in the entire population of turtles (marked and unmarked), `turtles`, and the array of recaptured turtles, `recaptured_turtles`, and returns the respective estimate $\hat{n}$ of the number of turtles. here, `nb_capture_mark_release` is $k$, `nb_recapture` is $k_r$, and `nb_marked_in_recaptured` is $m$.

```julia
function chapman_estimator(
        nb_capture_mark_release::Int,
        nb_recapture::Int,
        nb_marked_in_recaptured::Int
)
    return (nb_capture_mark_release + 1) * (nb_recapture + 1) / (nb_marked_in_recaptured + 1) - 1
end
```

    chapman_estimator (generic function with 1 method)

```julia
function lincoln_petersen_estimator(
        nb_capture_mark_release::Int,
        nb_recapture::Int,
        nb_marked_in_recaptured::Int
)
    return nb_capture_mark_release * nb_recapture / nb_marked_in_recaptured
end
```

    lincoln_petersen_estimator (generic function with 1 method)

(8) write a function `sim_capture_mark_release_recapture` that takes in:

* `nb_turtles::Int` $=n$
* `nb_capture_mark_release::Int` $=k$
* `nb_recapture::Int` $=k_r$
* `estimator::Function` either `chapman_estimator` or `lincoln_peterson_estimator` that you wrote above

and returns $\hat{n}$, the estimate of the number of turtles in this simulation.

use all of the functions you wrote above.

```julia
nb_turtles = 200
nb_capture_mark_release = 50
nb_recapture = 42
estimator = lincoln_petersen_estimator
n̂ = sim_capture_mark_release_recapture(
    nb_turtles,
    nb_capture_mark_release,
    nb_recapture,
    estimator
)
```

```julia
function sim_capture_mark_release_recapture(
        nb_turtles::Int,
        nb_capture_mark_release::Int,
        nb_recapture::Int,
        estimator::Function
    )
    turtles = create_population(nb_turtles)
    capture_mark_release!(turtles, nb_capture_mark_release)
    new_turtles = recapture(turtles, nb_recapture)
    nb_marked_in_recaptured = count_marked(new_turtles)
    estimator(nb_capture_mark_release, nb_recapture, nb_marked_in_recaptured)
end
```

    sim_capture_mark_release_recapture (generic function with 1 method)

(9) evaluate/compare the biasedness of the Chapman and Lincoln-Peterson estimators by plotting the distribution of $\hat{n}$ over many capture, mark, release, recapture simulations. Label which histogram corresponds to which estimator (in a legend if in the same histogram panel or in a title if in two subplots). plot as a vertical line the true number of turtles for comparison. print off the average and standard deviation of $\hat{n}$ over the simulations.

```julia
nb_turtles = 200
nb_capture_mark_release = 50
nb_recapture = 42
sims_range = 1:5000 # number of simulations
face_color = "#fcf8eb"

estimators = Dict(
    "Chapman" => Dict(
        "function" => chapman_estimator,
        "̂n" => [NaN for _ in sims_range]
    ),
    "Lincoln-Petersen" => Dict(
        "function" => lincoln_petersen_estimator,
        "̂n" => [NaN for _ in sims_range]
    )
)

fig, ax = subplots(2, 1, figsize=(18, 8), sharey=true)
bins = range(0, stop=2*nb_turtles, length=16)

est_keys = collect(keys(estimators))
num_keys = length(est_keys)
avgs = [NaN for _ in 1:num_keys]
stds = [NaN for _ in 1:num_keys]
for (plot, name) in enumerate(est_keys)
    for i in sims_range
        estimators[name]["̂n"][i] = sim_capture_mark_release_recapture(
            nb_turtles,
            nb_capture_mark_release,
            nb_recapture,
            estimators[name]["function"]
        )
    end
    ax[plot].hist(
        estimators[name]["̂n"],
        alpha=0.7,
        color="C$plot",
        label=name,
        bins=bins,
        density=true
    )
    ax[plot].axvline(nb_turtles, color="k")
    ax[plot].set_xlabel("Estimated # Turtles")
    ax[plot].set_ylabel("Simulation Density")
    ax[plot].set_title("$name Estimator", weight="bold")
    ax[plot].set_xlim([0, 2*nb_turtles])
    ax[plot].set_facecolor(face_color)
    avgs[plot] = mean(estimators[name]["̂n"])
    stds[plot] = std(estimators[name]["̂n"])
    println(name)
    println("=" ^ length(name))
    println("avg ̂n:\t", avgs[plot])
    println("std ̂n:\t", stds[plot])
    println()
end
fig.tight_layout()
sns.despine()

println()
println("least biased:\t", est_keys[argmin(avgs .- nb_turtles)])
println("most efficient:\t", est_keys[argmin(stds)])
```

(10) comment on which estimator, Chapman or Lincoln-Petersen, that appears to be most unbiased.

Based on the above simulations, the Chapman estimator appears unbiased, as well as more efficient.

(11) evaluate the consistency of the Lincoln-Petersen estimator by plotting the simulated distribution of $\hat{n}$ for several different values of `nb_recapture`. Use 10000 simulations for each value of `nb_recapture`.

```julia
recapture_range = 50:25:150
fig, ax = subplots(1, 1, figsize=(18, 8))
bins = range(0, stop=2*nb_turtles, length=16)
sims_range = 1:10000

color_norm = PyPlot.matplotlib.colors.Normalize(vmin=50, vmax=150)
kr_to_color = PyPlot.cm.ScalarMappable(
    norm=color_norm,
    cmap=get_cmap("viridis")
).to_rgba

for nb_recapture in recapture_range
    n = [sim_capture_mark_release_recapture(
            nb_turtles,
            nb_capture_mark_release,
            nb_recapture,
            chapman_estimator
        ) for _ in sims_range]
    ax.hist(
        n,
        label=nb_recapture,
        alpha=0.4,
        density=true,
        color=kr_to_color(nb_recapture)
    )
end
ax.axvline(nb_turtles, color="k")
ax.set_xlabel("Estimated # Turtles")
ax.set_ylabel("Simulation Density")
ax.set_title("Chapman Consistency Evaluation", weight="bold")
ax.legend(title="# Recaptured", frameon=false)
ax.set_xlim([0, 2*nb_turtles])
ax.set_facecolor(face_color)
sns.despine()
```


---
layout:     post
title:      "Inverted Pendulum Dynamics"
date:       2017-04-11 12:00:00
author:     Andrew
header-img: img/posts/pendulum_dynamics/balance_rock.jpg
header-credit: https://unsplash.com/@nbmat
tags:       programming dynamics sympy python
---

Today, we're talking double pendulums, kinematics and more!  Before you get scared off, I promise to make it accessible (why not give physics a chance).

The task at hand is to balance an inverted [double pendulum](https://en.wikipedia.org/wiki/Double_pendulum), but in this post I'm going to focus solely on modeling the system.  Modeling the system is important to understand what's happening, but I also plan to use this model to test out different strategies for actually controlling the balancing.

Testing different controls strategies with a physical setup is expensive, but I don't mean monetarily.  Time-wise, I could be running dozens or even hundreds of trials virtually before attempting anything with the physical motor and pendulum.
<!--break-->
So, with that said, lets get down to business.  Thankfully, the double pendulum is a well studied mechanical system, and myriad [academic papers](https://scholar.google.com/scholar?as_vis=1&q=double+pendulum&hl=en&as_sdt=1,16) are available on the subject.  One great resource I found was *A Mathematical Introduction to Robotic Manipulation* [^mls_textbook]

One reason the double pendulum is popular is because it's a simple example of [chaotic motion](https://en.wikipedia.org/wiki/Chaos_theory).  Chaotic motion means that the system is highly sensitive to input conditions.  Practically, this means if you were to release the pendulum from almost exactly the same spot many times in a row you would see far different outcomes each time.

This post is going to be a bit technical, but as always, I'll try my best to make it interesting for all audiences.

First things first, let's agree on what everything's going to be called:

Here is our double pendulum system.  Each link has a length $$l$$, a mass $$m$$, and it's center of mass has coordinates $$x$$, $$y$$.  The center of mass is a distance $$r$$ from the base of each link.  Motion will only be considered in the $$x-y$$ plane.

Mechanical systems such as these can be described by the number of [degrees of freedom](https://en.wikipedia.org/wiki/Degrees_of_freedom_(mechanics)) that it has.  The degrees of freedom of a system are the independant parameters that are needed to fully describe the configuration.

If there were a physical double pendulum in front of you, image moving the links around.  You would eventually determine that there are really only two things that you would need to tell someone for them to agree that you're both looking at the same configuration of the links.

In this system, those two degrees of freedom are the angles $$\theta_0$$ and $$\theta_1$$.  The state of the system can be described completely with these two angles.

What that means for us is that those two angles are what we want to simulate given some initial conditions.

#### WARNING: MATH

This is where we take a slight leap into some mechanics that may be a bit advanced, but try to stick it out!

There are two types of energy present in our double pendulum system:
* **[kinetic](https://en.wikipedia.org/wiki/Kinetic_energy)** ($$K$$), which comes from motion.  In this case, that motion is the swinging action of links.
* **[potential](https://en.wikipedia.org/wiki/Potential_energy)** ($$U$$), which is the stored energy of the system.  In this case, the stored energy is contained in the link's height, from gravity.  Gravitational potential energy is why things roll downhill (basically).

We're going to use a function called the [Lagrangian](https://en.wikipedia.org/wiki/Lagrangian_mechanics) to work some magic.

Using just the potential and kinetic energy of the system, and describing the coordinates of the links, the Lagrangain can yield the equations of motion for the system.

> Any function which generates the correct equations of motion, in agreement with physical laws, can be taken as a Lagrangian. It is nevertheless possible to construct general expressions for large classes of applications.

## System State Equations

$$ \begin{align}
x_0=& r_{0} \cos{\left (\theta_0 \right )} & \\
y_0=& r_{0} \sin{\left (\theta_0 \right )} & \\
x_1=& l_{0} \cos{\left (\theta_0 \right )} + r_{1} \cos{\left (\theta_0 + \theta_1 \right )} & \\
y_1=& l_{0} \sin{\left (\theta_0 \right )} + r_{1} \sin{\left (\theta_0 + \theta_1 \right )} & \\
\end{align}
$$

## Kinetic Energy

$$
\begin{align}
K =& \tfrac{1}{2}I_0\omega_0^2 + \tfrac{1}{2}m_0v_0^2 + \tfrac{1}{2}I_1\omega_1^2 + \tfrac{1}{2}m_1v_1^2 & \\
\omega_0 =& \dot{\theta_0} & \\
\omega_1 =& \dot{\theta_0} + \dot{\theta_1} & \\
K =& \tfrac{1}{2}I_0\dot{\theta_0}^2 + \tfrac{1}{2}m_0v_0^2 + \tfrac{1}{2}I_1(\dot{\theta_0} + \dot{\theta_1})^2 + \tfrac{1}{2}m_1v_1^2 & \\
v =& \dot{x} + \dot{y} & \\
K =& \tfrac{1}{2}I_0\dot{\theta_0}^2 + \tfrac{1}{2}m_0(\dot{x_0}^2 + \dot{y_0}^2) + \tfrac{1}{2}I_1(\dot{\theta_0} + \dot{\theta_1})^2 + \tfrac{1}{2}m_1(\dot{x_1}^2 + \dot{y_1}^2) & \\
\end{align}
$$

We'll let sympy handle the last substitution to get the equation in terms of only $$\theta$$

## Potential Energy

$$
\begin{align}
U =& m_0gy_0 + m_1gy_1 & \\
\end{align}
$$

## Lagrange

$$ L = K - U $$

Now we can substitute numbers into our equation to perform the actual calculations.

```python
from sympy import symbols, init_printing, S, Derivative, diff, simplify, solve, lambdify, cos, sin
from sympy.physics.vector import vlatex
import numpy as np
import scipy.integrate as integrate
from matplotlib import pyplot as plt
from matplotlib import animation, rc
from itertools import chain
from IPython.display import HTML, display, Math

rc('animation', html='html5')

init_printing(latex_printer=vlatex, latex_mode='equation')
```

```python
def disp(expr):
    display(Math(vlatex(simplify(expr))))

def disp_eq(lhs,expr):
    display(Math("$${0} = {1}$$".format(lhs, vlatex(simplify(expr)))))
```

```python
def generate_double_pendulum_odes():
    """
    :return:
     List of ODE describing system (Number = DOF of system)
     List of plotting position functions (Number = DOF of system)
    """
    t = symbols('t')
    g = symbols('g')
    l = symbols('l0:2')
    m = symbols('m0:2')
    r = symbols('r0:2')
    i = symbols('I0:2')
    tau = symbols('tau0:2')
    b = symbols('b0:2')

    g_val = S(9.8)
    l_val = [S(1.0), S(1.0)]
    m_val = [S(1.0), S(1.0)]
    r_val = [temp_l / 2 for temp_l in l_val]
    i_val = [(temp_m * temp_l ** 2) / 12 for temp_m, temp_l in zip(m_val, l_val)]
    tau_val = [S(0.0), S(0.0)]
    b_val = [S(0.0), S(0.0)]

    theta = [w(t) for w in symbols('theta0:2')]
    theta_dot = [Derivative(w, t) for w in theta]
    theta_ddot = [Derivative(w, t, t) for w in theta]

    x = [None] * 2
    y = [None] * 2
    x_dot = [None] * 2
    y_dot = [None] * 2

    x[0] = r[0] * cos(theta[0])
    y[0] = r[0] * sin(theta[0])
    x[1] = l[0] * cos(theta[0]) + r[1] * cos(theta[0] + theta[1])
    y[1] = l[0] * sin(theta[0]) + r[1] * sin(theta[0] + theta[1])

    x_dot[0] = diff(x[0], t)
    y_dot[0] = diff(y[0], t)
    x_dot[1] = diff(x[1], t)
    y_dot[1] = diff(y[1], t)

    kinetic = (m[0] * (x_dot[0] ** 2 + y_dot[0] ** 2)
               + m[1] * (x_dot[1] ** 2 + y_dot[1] ** 2)
               + i[0] * (theta_dot[0]) ** 2
               + i[1] * (theta_dot[0] + theta_dot[1]) ** 2) / 2

    potential = (m[0] * g * y[0]) + (m[1] * g * y[1])

    lagrange = kinetic - potential

    lagrangian = [None] * 2
    lagrangian[0] = diff(lagrange, theta_dot[0], t) - diff(lagrange, theta[0])
    lagrangian[1] = diff(lagrange, theta_dot[1], t) - diff(lagrange, theta[1])

    solution = solve(lagrangian, theta_ddot)

    values = [(g, g_val),
              (l[0], l_val[0]),
              (l[1], l_val[1]),
              (m[0], m_val[0]),
              (m[1], m_val[1]),
              (r[0], r_val[0]),
              (r[1], r_val[1]),
              (i[0], i_val[0]),
              (i[1], i_val[1]),
              (tau[0], tau_val[0]),
              (tau[1], tau_val[1]),
              (b[0], b_val[0]),
              (b[1], b_val[1])]

    temp_vars = symbols('z0:4')

    inputs = list(zip((theta_dot[0], theta[0], theta_dot[1], theta[1]), temp_vars))

    ode_equations = [None] * 2
    ode_equations[0] = lambdify(temp_vars, simplify(solution[theta_ddot[0]]).subs(values).subs(inputs))
    ode_equations[1] = lambdify(temp_vars, simplify(solution[theta_ddot[1]]).subs(values).subs(inputs))

    def double_pendulum_position(pos):
        result = []

        for _, theta0, _, theta1 in pos:
            x1_pos = float(l_val[0]) * np.cos(theta0)
            y1_pos = float(l_val[0]) * np.sin(theta0)

            x2_pos = x1_pos + float(l_val[1]) * np.cos(theta0 + theta1)
            y2_pos = y1_pos + float(l_val[1]) * np.sin(theta0 + theta1)

            result.append(((0, x1_pos, x2_pos), (0, y1_pos, y2_pos)))

        return result

    return ode_equations, double_pendulum_position
```

```python
def generic_deriv_handler(this_state, _, deriv_functions):
    # x_dot, x pairs
    result = [(float(func(*this_state)), this_state[(i * 2)]) for i, func in enumerate(deriv_functions)]

    flattened = chain.from_iterable(result)
    float_flattened = list(map(float, flattened))

    return np.array(float_flattened)
```

```python
def animate_system(time, time_step, initial_conditions, derivation_functions, position_function, plot_limit):

    pos = integrate.odeint(generic_deriv_handler, np.radians(initial_conditions), np.arange(0.0, time, time_step),
                           args=(derivation_functions,))

    plot_positions = position_function(pos)

    fig = plt.figure()
    ax = fig.add_subplot(111, autoscale_on=False, xlim=(-plot_limit, plot_limit), ylim=(-plot_limit, plot_limit))
    ax.grid()
    ax.set_aspect('equal', adjustable='box')

    line, = ax.plot([], [], 'k-', lw=4, solid_capstyle='round')
    time_template = 'time = {:0.2f}s'
    time_text = ax.text(0.05, 0.9, '', transform=ax.transAxes)

    def init():
        line.set_data([], [])
        time_text.set_text('')
        return line, time_text

    def animate(i):
        # TODO position func should be generator
        # TODO dependance on DOF
        thisx, thisy = plot_positions[i]

        line.set_data(thisx, thisy)
        time_text.set_text(time_template.format(time_step))
        return line, time_text

    return animation.FuncAnimation(fig, animate, frames=len(pos), interval=25, blit=True, init_func=init)
```

```python
ani = animate_system(5, 0.05, [0, 90, 0, 90], *generate_double_pendulum_odes(), 2)
```

```python
HTML(ani.to_html5_video())
```

```python
def generate_triple_pendulum_odes():
    """
    :return:
     List of ODE describing system (Number = DOF of system)
     List of plotting position functions (Number = DOF of system)
    """
    t = symbols('t')
    g = symbols('g')
    l = symbols('l0:3')
    m = symbols('m0:3')
    r = symbols('r0:3')
    i = symbols('I0:3')
    tau = symbols('tau0:3')
    b = symbols('b0:3')

    g_val = S(9.8)
    l_val = [S(1.0)] * 3
    m_val = [S(1.0)] * 3
    r_val = [temp_l / 2 for temp_l in l_val]
    i_val = [(temp_m * temp_l ** 2) / 12 for temp_m, temp_l in zip(m_val, l_val)]
    tau_val = [S(0.0)] * 3
    b_val = [S(0.0)] * 3

    theta = [w(t) for w in symbols('theta0:3')]
    theta_dot = [Derivative(w, t) for w in theta]
    theta_ddot = [Derivative(w, t, t) for w in theta]

    x = [None] * 3
    y = [None] * 3
    x_dot = [None] * 3
    y_dot = [None] * 3

    x[0] = r[0] * cos(theta[0])
    y[0] = r[0] * sin(theta[0])
    x[1] = l[0] * cos(theta[0]) + r[1] * cos(theta[0] + theta[1])
    y[1] = l[0] * sin(theta[0]) + r[1] * sin(theta[0] + theta[1])
    x[2] = l[0] * cos(theta[0]) + l[1] * cos(theta[0] + theta[1]) + r[2] * cos(theta[0] + theta[1] + theta[2])
    y[2] = l[0] * sin(theta[0]) + l[1] * sin(theta[0] + theta[1]) + r[2] * sin(theta[0] + theta[1] + theta[2])

    x_dot[0] = diff(x[0], t)
    y_dot[0] = diff(y[0], t)
    x_dot[1] = diff(x[1], t)
    y_dot[1] = diff(y[1], t)
    x_dot[2] = diff(x[2], t)
    y_dot[2] = diff(y[2], t)

    kinetic = (m[0] * (x_dot[0] ** 2 + y_dot[0] ** 2)
               + m[1] * (x_dot[1] ** 2 + y_dot[1] ** 2)
               + m[2] * (x_dot[2] ** 2 + y_dot[2] ** 2)
               + i[0] * (theta_dot[0]) ** 2
               + i[1] * (theta_dot[0] + theta_dot[1]) ** 2
               + i[2] * (theta_dot[0] + theta_dot[1] + theta_dot[2]) ** 2) / 2

    potential = (m[0] * g * y[0]) + (m[1] * g * y[1]) + (m[2] * g * y[2])

    lagrange = kinetic - potential

    lagrangian = [diff(lagrange, th_d, t) - diff(lagrange, th) for th_d,th in zip(theta_dot,theta)]

    solution = solve(lagrangian, theta_ddot)

    values = [(g, g_val),
              (l[0], l_val[0]),
              (l[1], l_val[1]),
              (l[2], l_val[2]),
              (m[0], m_val[0]),
              (m[1], m_val[1]),
              (m[2], m_val[2]),
              (r[0], r_val[0]),
              (r[1], r_val[1]),
              (r[2], r_val[2]),
              (i[0], i_val[0]),
              (i[1], i_val[1]),
              (i[2], i_val[2]),
              (tau[0], tau_val[0]),
              (tau[1], tau_val[1]),
              (tau[2], tau_val[2]),
              (b[0], b_val[0]),
              (b[1], b_val[1]),
              (b[2], b_val[2])]

    temp_vars = symbols('z0:6')

    inputs = list(zip((theta_dot[0], theta[0], theta_dot[1], theta[1], theta_dot[2], theta[2]), temp_vars))

    ode_equations = [lambdify(temp_vars, simplify(solution[th_ddot]).subs(values).subs(inputs)) for th_ddot in theta_ddot]

    def triple_pendulum_position(pos):
        result = []

        for _, theta0, _, theta1, _, theta2 in pos:
            x1_pos = float(l_val[0]) * np.cos(theta0)
            y1_pos = float(l_val[0]) * np.sin(theta0)

            x2_pos = x1_pos + float(l_val[1]) * np.cos(theta0 + theta1)
            y2_pos = y1_pos + float(l_val[1]) * np.sin(theta0 + theta1)

            x3_pos = x2_pos + float(l_val[2]) * np.cos(theta0 + theta1 + theta2)
            y3_pos = y2_pos + float(l_val[2]) * np.sin(theta0 + theta1 + theta2)

            result.append(((0, x1_pos, x2_pos, x3_pos), (0, y1_pos, y2_pos, y3_pos)))

        return result

    return ode_equations, triple_pendulum_position
```

```python
ani = animate_system(5, 0.05, [0, 0, 0, 0, 0, 0], *generate_triple_pendulum_odes(), 3)
```

```python
HTML(ani.to_html5_video())
```

```python

```

Author. Title. Title of container (self contained if book), Other contributors (translators or editors), Version (edition), Number (vol. and/or no.), Publisher, Publication Date, Location (pages, paragraphs URL or DOI). 2nd container’s title, Other contributors, Version, Number, Publisher, Publication date, Location, Date of Access (if applicable).

[^mls_textbook]: Richard M. Murray , S. Shankar Sastry , Li Zexiang, A Mathematical Introduction to Robotic Manipulation, CRC Press, Inc., Boca Raton, FL, 1994

```python

```


$\Huge{\text{Logistic Regression}}$

In this session, we shall learn about the basics of logistic regression.

Let us first consider a sample data, which will be useful for our study of logistic regression.

```python
#First, we import the required packages
import pandas as pd #the pandas library is useful for data processing
import numpy as np #numpy package will be useful for numerical computations
import matplotlib.pyplot as plt #the matplotlib library is useful for plotting purposes

# The following python directive helps to plot the graph in the notebook directly
%matplotlib inline
```

Now let us consider some open source data sets available in the internet.

```python
#Let us explore the data and the description

#Get the data from the web
data = pd.read_csv('http://users.stat.ufl.edu/~winner/data/challenger.dat', delim_whitespace=True, index_col=False, header=None)

#print the first few rows of the data
data.head()

```

      0
      1
      2
      3

      0
      1
      66
      0
      0

      1
      2
      70
      1
      1

      2
      3
      69
      0
      0

      3
      4
      68
      0
      0

      4
      5
      67
      0
      0

```python
#if you wish to print the entire data, then use the following
data
```

      0
      1
      2
      3

      0
      1
      66
      0
      0

      1
      2
      70
      1
      1

      2
      3
      69
      0
      0

      3
      4
      68
      0
      0

      4
      5
      67
      0
      0

      5
      6
      72
      0
      0

      6
      7
      73
      0
      0

      7
      8
      70
      0
      0

      8
      9
      57
      1
      1

      9
      10
      63
      1
      1

      10
      11
      70
      1
      1

      11
      12
      78
      0
      0

      12
      13
      67
      0
      0

      13
      14
      53
      2
      1

      14
      15
      67
      0
      0

      15
      16
      75
      0
      0

      16
      17
      70
      0
      0

      17
      18
      81
      0
      0

      18
      19
      76
      0
      0

      19
      20
      79
      0
      0

      20
      21
      75
      2
      1

      21
      22
      76
      0
      0

      22
      23
      58
      1
      1

What do the numbers in the above data mean? Let us understand the data by seeing its description.

```python
import urllib.request  #this package is useful for accessing text files over internet
data_description_file = urllib.request.urlopen('http://users.stat.ufl.edu/~winner/data/challenger.txt')

for line in data_description_file:
  decoded_line = line.decode("utf-8")
  print(decoded_line)
```

    Dataset: challenger.dat

    Source: S.R. Dalal, E.B. Fowlkes, B. Hoadley (1989). "Risk

    Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure,"

    Journal of the American Statistical Association, Vol. 84, #408, pp. 945-957.

    Description: Temperature and Failure of field-joint O-Ring failures

    for 23 pre-Challenger space shuttle flights.

    Variables/Columns

    Flight sequence number  7-8

    Temperature      15-16

    # O-Ring Field joints with erosion or blowby   24

    Indicator of erosion or blowby    32

Having known the data description, let us insert the descriptions into the data now.

```python
data.columns = ['Flight sequence number', 'Temperature', 'Number of O-Ring Field joints with erosion','Erosion Indicator']

#check by printing the data again
data.head()
```

      Flight sequence number
      Temperature
      Number of O-Ring Field joints with erosion
      Erosion Indicator

      0
      1
      66
      0
      0

      1
      2
      70
      1
      1

      2
      3
      69
      0
      0

      3
      4
      68
      0
      0

      4
      5
      67
      0
      0

We will consider Temperature and Erosion Indicator for analysis, therefore we will drop other columns.

```python
# drop columns 'Flight sequence number' and 'Number of O-Ring Field Joints with erosion'
data.pop('Flight sequence number')
data.pop('Number of O-Ring Field joints with erosion')

print('After dropping Flight sequence number and Number of O-Ring Field joints with erosion columns')
data.head()

```

    After dropping Flight sequence number and Number of O-Ring Field joints with erosion columns

      Temperature
      Erosion Indicator

      0
      66
      0

      1
      70
      1

      2
      69
      0

      3
      68
      0

      4
      67
      0

Seeing the data as mere numbers might not be interesting. So, let us use some graphical ways to visualize the data.

```python
#We will plot a scatter plot of Temperature vs Erosion Indicator

plt.scatter(data['Temperature'], data['Erosion Indicator'])
#plt.scatter(data.iloc[:,0], data.iloc[:,1])
plt.title("Temperature vs Erosion Indicator")
plt.xlabel("Temperature")
plt.ylabel("Erosion Indicator")
plt.show()
```

We see that the $y$ coordinates of the points (corresponding to the Erosion Indicator) are simply 0 and 1. Hence it would be unsuitable to model the given data using linear regression model.

$\large{\text{Question:}}$ How do we model the response variable in this situation?

$\large{\text{One Possible Solution:}}$ We will discuss logistic regression as a possible approach to model such situations where the response variable is dichotomous (or) binary-valued.

$\Large{\text{Probabilistic interpretation}}$

$\textbf{Recall:}$ In linear regression, we assumed that in terms of probability, the response variable $Y \in {\mathbb{R}}$ was dependent on the predictor variable $X \in {\mathbb{R}}$ using the relation:

$
E[Y|X=x] =  \beta_0 + \beta_1 x.
$

However $\textbf{note}$ that the response variable $Y$ in linear regression was a real-valued quantity.

However for our current data, $Y$ takes only the values $0$ and $1$. Hence if we wish to use a probabilistic idea for such binary-valued response variable, one straightforward idea comes from Bernoulli random variables.

$\large{\text{Bernoulli Random Variable}}$

Let $Z$ be a Bernoulli random variable taking values $0$ and $1$. Then the distribution of $Z$ is given by:

$
\begin{align}
P(Z=1) &= p, \\
P(Z=0) &= 1-p
\end{align}
$

where $p$ is the probability with which $Z$ takes the value $1$. $p$ is called the parameter of the Bernoulli distribution.

Also note that the expected value of $Z$ is given by: $E[Z] = p$.

Now considering our response random variable $Y$ as a Bernoulli random variable,we would have: $E[Y]=p$, where $p$ is the probability that $Y$ takes the value $1$.

However in the presence of a predictor random variable $X$, we can write the conditional expectation of $Y$ given an observation $x$ of $X$ as:

$
\begin{align}
E[Y|X=x] = p(x)
\end{align}
$

where we have assumed that the probability now has a dependence on the observation $x$.

For example, in our data, since $Y$ denotes the Erosion Indicator, and $X$ denotes the Temperature, we can assume that the probability that Erosion happens is somehow related to the observed value of Temperature.

There are multiple ways to model $p(x)$. In logistic regression, we assume the following:

$
p(x)=p(x;\beta_0,\beta_1) = \frac{e^{(\beta_0 + \beta_1 x)}}{1+e^{(\beta_0 + \beta_1 x)}}.
$

$\textbf{Note:}$ The form of $p(x)$ is assumed to be parametric, dependent on parameters $\beta_0, \beta_1$. The form of $p(x)$ in the previous equation is specifically called a $\textbf{(logistic) sigmoid}$ function.

```python
#let us plot the logistic sigmoid function
x = np.linspace(-5,5,100) #creates a series of points in x axis
y = np.exp(x)/(1.0+np.exp(x))
y1 = np.exp(0.5*x)/(1.0+np.exp(0.5*x))
y2=np.exp(50*x)/(1.0+np.exp(50*x))
plt.plot(x, y, '-g', label='logistic sigmoid function')
plt.plot(x, y1, '-r', label='logistic sigmoid function - 1')
plt.plot(x, y2, '-b', label='logistic sigmoid function - 2')

plt.title("Logistic sigmoid")
plt.xlabel("x")
plt.ylabel("exp(x)/(1+exp(x))")
plt.legend(loc='lower right')
#plt.grid()
plt.show()
```

We can further write $p(x)$ equivalently as:

$
p(x)=p(x;\beta_0,\beta_1) = \frac{1}{1+e^{-(\beta_0 + \beta_1 x)}}.
$

If we denote $p(x)$ simply as $p$ and if we have the notations $\mathbf{x}=\begin{bmatrix}
x \\ 1
\end{bmatrix}, \beta=\begin{bmatrix}
\beta_1 \\ \beta_0
\end{bmatrix}$
then we can write:

$
\begin{align}
p = \frac{1}{1+e^{-\beta^\top \mathbf{x}}}.
\end{align}
$

Then note the following that the following algebraic manipulations hold:

$
\begin{align}
p &= \frac{1}{1+e^{-\beta^\top \mathbf{x}}} \\
\implies 1 + e^{-\beta^\top \mathbf{x}} &= \frac{1}{p} \\
\implies  e^{-\beta^\top \mathbf{x}} &= \frac{1}{p} - 1 \\
\implies  e^{-\beta^\top \mathbf{x}} &= \frac{1-p}{p} \\
\implies \ln e^{-\beta^\top \mathbf{x}} &= \ln \frac{1-p}{p} \\
\implies -\beta^\top \mathbf{x} &= \ln \frac{1-p}{p} \\
\implies \beta^\top \mathbf{x} &= \ln \frac{p}{1-p} \\
\implies \beta_0 + \beta_1 x &= \ln \frac{p}{1-p}.
\end{align}
$

Thus, even if we did not have a straightforward dependence of $Y$ on an observation $x$ of $X$ as a linear relation, we see that the linear relation $\beta_0 + \beta_1 x$ is related to the probability $p$ using:

$
\ln\frac{p}{1-p}=\beta^\top \mathbf{x}.
$

Note that the ratio $\frac{p}{1-p}$ is called $\textbf{odds}$ that the event $Y=1$ occurs, and hence $\ln \frac{p}{1-p}$ denotes the $\textbf{log odds}$.

More popularly, the log odds $\ln \frac{p}{1-p}$ is called the $\textbf{logit}$ function.

```python
#let us plot the logit function
x = np.linspace(0,1,100) #creates a series of points in x axis
y = np.log(x/(1-x))
plt.plot(x, y, '-g', label='logit function')

plt.title("Logit")
plt.xlabel("p")
plt.ylabel("ln p/(1-p)")
plt.legend(loc='upper left')
#plt.grid()
plt.show()
```

$\Large{\text{Likelihood function}}$

We now define a quantity which will be useful in the estimation of the parameters $\beta_0, \beta_1$ used to model $p$.

Given an observation $X=x$ we define the $\textbf{likelihood function}$ as:

$
L(y;p) = p^y(1-p)^{(1-y)}
$

where recall that $p=p(x)=p(x;\beta_0,\beta_1)=\frac{1}{1+e^{-{(\beta_0 + \beta_1 x)}}}$. Note that likelihood function is simply an equivalent way to represent $P[Y=y]$, when $Y$ is assumed to be Bernoulli random variable.

Then observe that the natural goal is to maximize the likelihood function with respect to parameters $\beta_0, \beta_1$.

Now given a data set $D$ containing $n$ observations of the form $\{({x}^1,y^1), ({x}^2,y^2), \ldots, ({x}^n,y^n)\}$, and assuming that the pairs $({x}^i,y^i)$ are independent observations, then it is possible to extend the likelihood function as:

$
\begin{align}
L(y^1,\ldots,y^n;p^1,\ldots,p^n) = \Pi_{i=1}^{n} {(p^i)}^{y^i}(1-p^i)^{(1-y^i)}.
\end{align}
$

We can now write the $\textbf{log likelihood}$ function as:

$
\begin{align}
\ln L(y^1,\ldots,y^n;p^1,\ldots,p^n) = \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
\end{align}
$

Since log likelihood function is monotonic, maximizing the likelihood function is equivalent to maximizing the log likelihood function.

Hence the concerned optimization problem is:

$
\max_{\beta_0, \beta_1} \ln L(y^1,\ldots,y^n;p^1;\ldots,p^n)= \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
$

Note that $p^i = p(x^i) = p(x^i; \beta_0, \beta_1) = \frac{1}{1+e^{-{(\beta_0 + \beta_1 x^i)}}}, \forall i = 1,\ldots,n$.

$\Large{\text{Solving the likelihood maximization problem}}:$

To solve

$
\max_{\beta_0, \beta_1} \ln L(y^1,\ldots,y^n;p^1;\ldots,p^n)= \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
$

we can find the derivatives of the objective function with respect to $\beta_0$ and $\beta_1$ as:

$
\begin{align}
\frac{\partial{ \ln L }} {\partial \beta_0} &= \sum_{i=1}^{n} y^i (1-p^i) - (1-y^i) p^i \\
\frac{\partial{ \ln L }} {\partial \beta_1} &= \sum_{i=1}^{n} x^i(y^i (1-p^i) - (1-y^i) p^i)
\end{align}.
$

Equating these derivatives to zero will lead to system of non-linear equations which is generally difficult to solve to obtain closed form expression.

Hence we generally adopt an iterative procedure as follows to find the optimal $\beta_0, \beta_1$.

$\large{\text{Gradient ascent for solving the likelihood maximization problem}}:$

$
\begin{align}
&\textbf{Step 0:}  \text{Input data set $D$, tolerances $\epsilon_1, \epsilon_2$.} \\
&\textbf{Step 1:}  \text{Start with arbitrary $\beta_0, \beta_1$.} \\
&\textbf{Step 2:}  \text{For $k=1,2,\ldots$} \\
&\quad \quad \textbf{Step 2.1:} \text{Compute differentials  $\frac{\partial{ \ln L }} {\partial \beta_0}$ and $\frac{\partial{ \ln L }} {\partial \beta_1}$} \\
&\quad \quad \textbf{Step 2.2:}  \text{Compute step length $\eta$ using line search procedure} \\
&\quad \quad \textbf{Step 2.3:}  \beta_0 = \beta_0 + \eta \frac{\partial{ \ln L }} {\partial \beta_0} \\
&\quad \quad \textbf{Step 2.4:}  \beta_1 = \beta_1 + \eta \frac{\partial{ \ln L }} {\partial \beta_1} \\
&\quad \quad \textbf{Step 2.5:}  \text{if $\|\nabla_{\beta} \ln L\|_2 \leq \epsilon_1$ break from loop} \\
&\quad \quad \textbf{Step 2.6:}  \text{if relative change in function value is $\leq \epsilon_2$ break from loop} \\
&\textbf{Step 3:}  \text{ Output $\beta_0, \beta_1$}
\end{align}
$

Note that $\nabla_{\beta} \ln L$ denotes the gradient of $\ln L$ with respect to $\beta$ given by:

$
\nabla_{\beta} \ln L = \begin{bmatrix}
\frac{\partial{ \ln L }} {\partial \beta_0}\\
\frac{\partial{ \ln L }} {\partial \beta_1}
\end{bmatrix}
$

and $\eta$ denotes the learning rate.

$\Large{\text{Computing } \beta \text{ for the erosion data set}}:$

```python
#Let us now compute beta
n = len(data.index) #number of data points in the data set
print('number of data points in the data set:',n)
```

    number of data points in the data set: 23

```python
import numpy as np
#computing log likelihood function
def log_likelihood(beta_0,beta_1):
  log_likelihood = 0.0
  for i in range(n):
    x_i = float(data.iat[i,0]) #access element at Temperature column
    y_i = float(data.iat[i,1]) #access element at Erosion Indicator column
    p_i = 1.0/(1.0+np.exp(-beta_0 - beta_1*x_i)) #probability with the current beta_0, beta_1
    log_likelihood += y_i * np.log(p_i) + (1.0-y_i)*np.log(1-p_i)
  return log_likelihood
```

```python
#gradient ascent for likelihood maximization
beta_0 = 0.0
beta_1 = 0.0
max_iter = 500000
eps_1 = 0.1 #tolerance for gradient norm
eps_2 = 1e-9 #tolerance for relative function value difference

f_val = log_likelihood(beta_0,beta_1)

#store the objective function values for plotting purposes
f_val_list = []
f_val_list.append(f_val)

print('Initial values: beta_0:',beta_0, ' beta_1:',beta_1, ' log likeihood:', f_val)
#the loop
for k in range(max_iter):

  grad_beta_0 = 0.0
  grad_beta_1= 0.0
  #pass through the data to compute gradients
  for i in range(n):
    x_i = float(data.iat[i,0]) #access element at Temperature column
    y_i = float(data.iat[i,1]) #access element at Erosion Indicator column
    p_i = 1.0/(1.0+np.exp(-beta_0 - beta_1*x_i)) #probability with the current beta_0, beta_1
    grad_beta_0 +=  (y_i * (1.0-p_i)) - ((1.0-y_i) * p_i)
    grad_beta_1 +=  x_i * (y_i * (1.0-p_i) - (1.0-y_i) * p_i)

  #lr = linesearch(beta_0, beta_1, grad_beta_0, grad_beta_1, float(f_val_list[-1]))
  lr = 0.0001
  beta_0 += lr * grad_beta_0
  beta_1 += lr * grad_beta_1
  #print('k: ', k, ' grad beta_0: ', grad_beta_0, 'grad beta_1: ',grad_beta_1, 'beta_0:', beta_0, 'beta_1:',beta_1)
  grad_norm = np.sqrt(grad_beta_0**2 + grad_beta_1**2)
  f_val = log_likelihood(beta_0,beta_1)
  f_val_list.append(f_val)

  rel_change_in_fval = np.abs((f_val - f_val_list[-2])/f_val_list[-2])

  if k%1000+1 == 1:
    print('k: ', k,  ' beta_0:', beta_0, ' beta_1:',beta_1, ' grad norm:', grad_norm, 'log likeihood:', f_val)
  if grad_norm <= eps_1 or rel_change_in_fval <= eps_2:
    break
print('Final: k: ', k,  ' beta_0:', beta_0, ' beta_1:',beta_1, ' grad norm:', grad_norm, 'log likeihood:', f_val)

#plot the function values during optimization
plt.plot( f_val_list, '-r', label='Log Likelihood values')
plt.title("Log likelihood function values vs iterations")
plt.xlabel("Iteration")
plt.ylabel("Log Likelihood")
plt.legend(loc='lower right')
#plt.grid()
plt.show()

```

```python
#It would be useful to check the acutal y_i values against the predicted probabilities \hat{y}_i

predicted_probabilities = []
for i in range(n):
    x_i = float(data.iat[i,0]) #access element at Temperature column
    y_i = float(data.iat[i,1]) #access element at Erosion Indicator column
    p_i = 1.0/(1.0+np.exp(-beta_0 - beta_1*x_i)) #probability with the current beta_0, beta_1
    predicted_probabilities.append(p_i)

plt.scatter(data['Temperature'], data['Erosion Indicator'],label='actual')
plt.scatter(data['Temperature'], predicted_probabilities,label='predicted')

plt.title("Actual vs Predicted Probabilities")
plt.xlabel("Temperature")
plt.ylabel("Erosion Indicator")
plt.legend(loc='center right')
#plt.grid()
plt.show()
```

$\Large{\text{Exercise:}}$

Perform logistic regression with the following data sets:
1. Data: http://users.stat.ufl.edu/~winner/data/smkdth.dat
   Description: http://users.stat.ufl.edu/~winner/data/smkdth.txt

   Take the age and smoking status as predictor variables and survival status as response variable.

   For computing  prediction $\hat{y}(x)$ for an observation $X=x$, use the following rule:
   $
   \hat{y}(x) = \begin{cases}
          1 & \text { when } p(x) >= 0.5 \\
          0 &  \text { otherwise. }
          \end{cases}
   $

   Report the accuracy as the percent of correctly predicted samples using the logistic regression model.

2. Data: http://users.stat.ufl.edu/~winner/data/subscript.dat
   Description: http://users.stat.ufl.edu/~winner/data/subscript.txt

   Take the subscription category as predictor variable and renewal status as response variable.

Plot the predictions and actual values in a scatter plot. Report the accuracy as the percent of correctly predicted samples using the logistic regression model.
Explain your observations.

$\Large{\text{References:}}$

1.   Joseph M. Hilbe, $\textit{Practical Guide to Logistic Regression}$. CRC Press,  2015.
2.   David Hosmer, Stanley Lameshow and Rodney Sturdivant. $\textit{Applied Logistic Regression}$. John Wiley $\&$ Sons, Inc., Hoboken, New Jersey, 2013.

```python

```