# Constrained Shock Alignment for Multiblock Structured Grids

## Preamble
* Define "vec" command for LaTeX $\newcommand{vec}[1]{\boldsymbol{#1}}$

```python
# Configure python
%matplotlib notebook
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from scipy.linalg import solve_banded

# Defaults for plots
mpl.rc('lines', linewidth=0.8, marker='.')
```

## Motivation

To obtain accurate hypersonic heating estimates for blunt capsule geometries, current state of the art requires aligning the the computational grid with the bow shock. DPLR accomplishes this periodically reseting the location of the freestream boundary as follows:

  1. DPLR identifies the location of the bow shock using a jump detector based on Mach number.
  2. DPLR then constructs a smooth outer boundary slightly upstream of the bow shock position.
  3. DPLR then redistributes mesh points between the wall and the new outer boundary

There are a few shortcomings with this approach as it is currently implemented:

  1. Tailoring can only be performed for single layer topologies (all block must span from the wall to the free stream).
  2. With few exceptions, adaption will fail if the shock moves outwards and collides with the outer boundary. This is can occur when interpolating solutions to a finer grid as part of a mesh sequencing strategy. Therefore, tailoring must be done in stages where initially the outer boundary is significantly offset from the shock and then brought in over time.
  3. It is not possible to directly control the spacing of the grid at the shock. Instead, spacing at the shock is controlled by a combination of the spacing at the outer boundary and the margin from the boundary to the shock, which can be cumbersome.
  4. Similar to (3), the outer boundary margin for the $n$-th grid is computed based on the local grid spacing near the bow shock on the $n-1$st grid. Therefore, it typically takes several adaptions to reach a converged grid even if the bow shock is not moving appreciably.
  6. Since shock fitting is done periodically and is decoupled from the evolution of the flow field, each adaption disturbs the flow solution and introduces error that may takes many iterations to remove.
  7. The methods employed to smooth the outer boundary either push the outer boundary away from the shock in regions of high body curvature (ismooth=3) or have difficulty removing high-frequency tailoring errors (ismooth=1).

I propose replacing the current decoupled tailoring strategy with one that evolves the grid simultaneously with the solution variables in an attempt to fit a specifed plane in the mesh to the bow shock. It is believed that this method can address all the above shortcomings and greatly reduce the wall time required to obtain the tailored grid solution.

## Mathematical Definitions

Consider a structured grid consisting of a single body-fitted block. We will refer to this grid as our *background grid*. The only requirement on the background grid is that it be large enough to contain the bow shock; it need not be aligned with the shock. Let $\vec\zeta = [\zeta_1, \zeta_2, \zeta_3]$ be the curvilinear coordinates for the background grid.

The *computational grid* is constructed from the background grid by specifying a distribution of points along each constant-$\zeta_1,\zeta_2$ grid line. As the solution evolves, we wish to align the computational grid with the bow shock. Let $\xi = [\xi_1, \xi_2, \xi_2]$ represent the curvilinear coordinate for the computational grid.

Next, let $S(\vec\zeta)$ be the arc-length of the constant-$\zeta_1,\zeta_2$ geodesic connecting the point $\vec\zeta$ to the outer boundary of the background grid:

$$ S(\vec\zeta) = \left. \int_{\zeta_3}^{\zeta_3^{max}} \left\Vert \frac{\partial x}{\partial \zeta_3} \right\Vert d\zeta_3 \right\vert_{\zeta_1,\zeta_2=C} $$

Leveraging this definition, let $s_s(\zeta_1,\zeta_2)$ be a function that specifies the curvilinear distance to the bow shock for a given $\zeta_1,\zeta_2$. The location of the bow shock may be identified using any suitable jump detector.

$$ s_s(\zeta_1,\zeta_2) = S\left(\zeta_1,\zeta_2, \zeta_3^{shock}\right)$$

Additionally, let $s(\zeta_1,\zeta_2)$ be a function that specifies the location of the constant-$\xi_3$ isosurface in the computational grid that will be aligned with the bow shock.

$$ s(\zeta_1, \zeta_2) = S\left(\zeta_1, \zeta_1, \zeta_3^{\xi_3=C}\right) $$

While specifying the location of the shock and the $\xi_3$ isosurfaces in terms of curvilinear distance complicates the shock fitting process, it also provides several benefits:

  1. By locating these surfaces relative to the outer boundary of the background grid, it is possible to move the computational grid outwards if the shock begins to impinge on the outer boundary of the compuational grid.

  2. This parameterization specifies shock location independently of how the grid points are distributed in the wall-normal direction. This allows shock shapes to be to be easily and accurataely transferred a more refined grid.

  3. This approach guarantees that $s$ and $s_s$ are smooth, differentiable functions provided that: (1) the shock front itself is smooth, (2) the outer boundary of the background grid is smooth, and (3) a smooth interpolant is used to compute the curvilinear distance function. Note that it is not required that either the background or the computational grid to have a smooth interior. This is important because non-smooth volume grids are very common: when using algebraic grid generation techniques, edges or corners in the surface geometry will alway results in non-smooth volume grids.

## Grid Equation of Motion
In order to couple the grid tailoring process to the solution of the flow equations we must define an equation of motion that governs the evolution of the grid with respect to time. The approach adopted here will be to define a partial differential equation that governs the evolution of the constant-$\xi_3$ isosurface and use an algebraic grid distribution function to control the placement of grid points along constant-$\zeta_1,\zeta_2$ grid lines. By using a known, fixed algebraic distribution function the grid tailoring problem is reduced in dimensionality, greatly reducing the computational complexity of the problem.

The partial differential equation governing the constant-$\xi_3$ iso-surface is based on analogy with a spring-mass-damper network. For the moment, let's restrict ourselves to a 2D fitting problem on the domain $\zeta_1 \times \zeta_3$ and assume that  the true shock location, $s_s(\zeta_1)$ is known. We then distribute a  series of point masses along a chosen $\xi_3$-isocontour and denote the curilinear position of $i$-th mass as $s_i = s(\zeta_{1,i})$. The masses are then coupled as follows:

  1. Each mass is coupled to the shock with a spring-damper pair, where the force is proportional to the difference in curvilinear position:

$$
    f_i = -k  \left( s_i - s_s(\zeta_{1,i}) \right)
        - \mu \left( \dot{s}_i - \dot{s}_s(\zeta_{1,i}) \right)
$$

  2. Additionally, each mass is connected to its nearest neigbor with a spring-damper pair that resists relative displacement between masses:

$$
    f_i = -k^\prime \left( s_i - s_{i+1} \right)
        - \mu^\prime \left( \dot{s}_i - \dot{s}_{i+1} \right)
$$

The resulting force balance for each mass in the network is second order differential equation of the following form:

$$
    m_i \ddot{s}_i = -k          \left( s_i - s_{s,i} \right)
                     +k^\prime   \left( s_{i-1} - 2s_i + s_{i+1} \right)
                     -\mu        \left( \dot{s}_i - \dot{s}_{s,i} \right)
                     +\mu^\prime \left( \dot{s}_{i-1} - 2\dot{s}_i + \dot{s}_{i+1} \right)
$$

Dividing by mass and generalizing to the continuous limit, we arrive at the following four-parameter, linear partial differential equation for the evolution of the $\xi_3$ isosurface:

$$
    \ddot{s} + 2\left(\zeta\omega - \zeta^\prime \omega^\prime \nabla_\xi^2 \right)\dot{s}
             + \left(\omega^2 - \omega^{\prime 2}\nabla_\xi^2 \right)s
             = 2\zeta\omega\dot{s}_s + \omega^2 s_s
$$

A few notes about this equation:

  1. In this context the scalar $\zeta$ represents a damping ratio and is distinct from the vector quantity $\vec\zeta$.

  2. As written, the spatial gradient is taken with respect to the computational coordinates, $\vec\xi$. This implies that the steady state solution of the equation of motion is *not* grid independent. Rather, as the computational grid is refined the curvature in computional coordinates will be reduced and the $\xi_3$-isosurface will more accurately follow the shock.

## Steady State Solution to the Grid Equation
To gain insight into the chacteristics of grid motion equation, let us first consider the limit of steady state. Setting all time derivatives to zero yields:

$$
  s - \epsilon^2 \nabla_\xi^2 s = s_s,
$$

$$
  \epsilon = \omega^\prime/\omega
$$

Here we see that at steady state, the dynamic model will minimize the distance between the $\xi_3$ isosurface and the shock in a manner that balances fitting error against curvature of the isosurface. This suggests that the parameter $\epsilon$, which represents the ratio of the membrane stiffness to the stiffness of the force attacting the $\xi_3$-isosurface to the shock, should be small to maximize the accuracy of the fit.

However, in some cases where the shock structure itself is non-smooth, larger values may be necessary to ensure the smoothness of the grid. The figures below show the steady state solution in 1D for step and ramp inputs with various values of $\epsilon$. The Laplacian operator is approximated using the classical 3-point centered finite difference scheme with linear extrapolation at the boundaries ($s_{-1} = 2s_0 - s_1$). This discretization results in a tridiagonal system which is directly inverted onto reference input, $s_0$.

To my eye, an $\epsilon$ of ~2, looks to be a good number for non-smooth grids. Steps are smoothed over ~10 grid points to either side of the jump; ramps are smoothed using ~10 points total.

```python
def generate_smoothing_matrix(epsilon, N):
    ''' Return rank-N tri-diagonal smoothing matrix in band-packed format '''

    # Baseline smoothing kernel
    main  = 1+2*epsilon**2 * np.ones(N)
    upper =    -epsilon**2 * np.ones(N)
    lower =    -epsilon**2 * np.ones(N)

    # Impose linear extrapolation boundry conditions
    upper[ 0] = 0.0
    upper[ 1] = upper[ 1] +   epsilon**2
    main [ 0] = main [ 0] - 2*epsilon**2
    lower[-1] = 0.0
    lower[-2] = lower[-2] +   epsilon**2
    main [-1] = main [-1] - 2*epsilon**2

    return np.stack([upper, main, lower], axis=0)

def generate_function(name, N):
    ''' Returns an array with  specified function type'''
    x = np.linspace(-1.0, 1.0, N)
    if name == "step":
        y = np.ones(N)
        y[x <= 0.0] = -1.0
    elif name == "ramp":
        y = x.copy()
        y[x <= 0.0] = 0.0
    else:
        raise RuntimeError("Unknown function")
    return x,y

# Plot effect of smoothing
npoint = 40
epsilon = [1.0, 2.0, 4.0, 8.0]
for function_type in ["step", "ramp"]:
    ax = plt.figure().add_subplot(111)
    x,s0 = generate_function(function_type, npoint)
    for eps in epsilon:
        A = generate_smoothing_matrix(eps, npoint)
        s = solve_banded((1,1), A, s0)
        ax.plot(x,s)
    ax.plot(x, s0, linestyle='dashed', color='black', marker='None')
    ax.legend(['$\epsilon$ = %.1f' % e for e in epsilon] + ['$s_0$'], loc='upper left')
    ax.set_xlabel(r'Space Coordinate, $\xi$')
    ax.set_ylabel(r'Steady State Position, $s$')

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

## Dynamic Characteristics of the Grid Equation

To understand the dynamic behavior of our motion model, let us solve for the homogeneous dynamics of equation (?) in one dimension on an infinite interval. Since our partial differential equation is linear, solution via Fourier Transform is an expedient approach. Let,

$$
    s(t,\xi) = A_k(t)e^{ik\xi}
$$

Inserting this functional form into the governming equation yeilds a second order differential equation for the amplitude of the waveform:

$$
    \ddot{A}_k
    + 2\left(\zeta\omega + k^2\zeta^\prime\omega^\prime\right)\dot{A}_k
    + \left(\omega^2 + k^2\omega^{\prime 2}\right) A_k
    = 0
$$

This eqaution can be transformed into classical second order form by defining and effective damping ratio and angular rate, as shown below.

$$
    \tilde{\omega} = \sqrt{ \omega^2 + k^2\omega^{\prime 2} } = \omega \sqrt{1 + k^2\epsilon^2}
$$

$$
    \tilde{\zeta} = \frac{\zeta\omega + k^2\zeta^\prime\omega^\prime}
                         {\sqrt{\omega^2 + k^2\omega^{\prime 2}}}
                  = \zeta \frac{1}{\sqrt{1 + k^2\epsilon^2}}
                  + k\zeta^\prime \frac{k\epsilon}{\sqrt{1 + k^2\epsilon^2}}
$$

$$
    \ddot{A}_k
    + 2\tilde{\zeta}\tilde{\omega}\dot{A}_k
    + \tilde{\omega}^2 A_k
    = 0
$$

For $\tilde{\zeta} > 1$, the solution this equation is a linear sum of two exponential decay modes. Let

$$
    A_k(t) = A_{k0} e^{-t/\tau}
$$

Substituting into the differential equation for $A_k$, we arrive at a quadratic equation for $1/\tau$, which can be solved to yeild the chacateristice time constant for the two decay modes:

$$
    \frac{1}{\tau} = \tilde{\omega}\left(\tilde{\zeta} \pm \sqrt{\tilde\zeta^2 - 1}\right)
$$

This characteristic time constant is a function of four parameters: $\omega$, $\zeta$, $\zeta^\prime$, and $\epsilon$. Note that if we set $k = 0$ in the above equation, i.e. if the starting soltuion to the differential equation is a constant offset, the effective damping ratio and angular rate reduce to $\zeta$ and $\omega$ respectively. Therefore, these parameters govern how quickly the $\xi_3$-isosurface approaches the shock in an average sense.

The other two parameters, $\zeta^\prime$ and $\epsilon$, come into play for high spatial wavenumbers. If we consider the limit $k\rightarrow\infty$, it is possible to show that these two parameters set the decay constants for high-$k$ waveforms:

\begin{align}
    \lim_{k\rightarrow\infty} \frac{1}{\tau^+}
    &= \lim_{k\rightarrow\infty} \tilde{\omega} \left(\tilde{\zeta} + \sqrt{\tilde\zeta^2 - 1}\right) \\
    &= \lim_{k\rightarrow\infty} \epsilon k^2 \omega\zeta^\prime \left( 1 + \sqrt{1 - \left(k\zeta^\prime\right)^{-2}} \right) \\
    &= 2\epsilon k^2 \omega\zeta^\prime = 2k^2 \omega^\prime\zeta^\prime
\end{align}

\begin{align}
    \lim_{k\rightarrow\infty} \frac{1}{\tau^-}
    &= \lim_{k\rightarrow\infty} \tilde{\omega} \left(\tilde{\zeta} - \sqrt{\tilde\zeta^2 - 1}\right) \\
    &= \lim_{k\rightarrow\infty} \epsilon k^2 \omega \zeta^\prime \left( 1 + \sqrt{1 - \left( k\zeta^\prime \right)^{-2} } \right) \\
    &= \frac{\epsilon\omega}{2\zeta^\prime} = \frac{\omega^\prime}{2\zeta^\prime}
\end{align}

To demonstrate the effect of $\epsilon,\zeta^\prime$, let's compare how quickly disturbances damp relative to the "average" motion of the $\xi_3$-isosurface. To do this, we take the ratio of the time constants for the slow modes.

$$
\delta(k) = \frac{\tau^-\big|_{k=k}}{\tau^-\big|_{k=0}}
$$

With some algebra, this ratio can be written as:

$$
    \delta(k\epsilon) = \frac{\zeta - \sqrt{\zeta^2 -1}}{ b\zeta - \sqrt{b^2\zeta^2 - a^2}}
$$

\begin{align}
    a &= \sqrt{1 + k^2\epsilon^2}  \\
    b &= 1 + \alpha k^2 \epsilon^2 \\
    \alpha &= \frac{1}{\epsilon}\cdot\frac{\zeta^\prime}{\zeta}
\end{align}

Examining this equation, we see that the decay of higher order modes relative to the mean mode is a function of the scaled wavenumber, $k\epsilon$, with free parameters $\zeta$ and $\alpha$. Since $\alpha$ is proportional to $\zeta^\prime$ and $\epsilon$, the $\alpha$ parameter controls the decay time for high-frequency waves. Note in Figure 3 below how the curves converge to $\alpha = 0.6$ as the damping ratio for the mean mode is increased.

```python
def compute_decay_ratio(k, epsilon, zeta, zetap):
    alpha = zetap / (epsilon * zeta)
    ke2 = (k * epsilon)**2
    a = np.sqrt(1 + ke2)
    b = 1 + alpha * ke2
    return (zeta - np.sqrt(zeta**2 - 1))/(b*zeta - np.sqrt((b*zeta)**2 - a**2))

ke = np.logspace(-2,1)

# Plot influence of increasing zeta
ax = plt.figure().add_subplot(111)
ax.set_xlabel(r'Scaled Wavenumber, $k\epsilon$')
ax.set_ylabel(r'Decay Time Ratio, $\delta$')
alpha = 0.6
zeta = [1.2, 1.4, 1.8, 2.6, 4.2]
for z in zeta:
    ax.plot(ke, compute_decay_ratio(ke, 1.0, z, alpha*z))
ax.legend([r'$\alpha$ = %.1f, $\zeta$ = %.1f' % (alpha, z) for z in zeta])

# Plot influence of increasing alpha
ax = plt.figure().add_subplot(111)
ax.set_xlabel(r'Scaled Wavenumber, $k\epsilon$')
ax.set_ylabel(r'Decay Time Ratio, $\delta$')
alpha = [0.6, 0.8, 1.0, 1.2, 1.4]
zeta = 1.4
for a in alpha:
    ax.plot(ke, compute_decay_ratio(ke, 1.0, zeta, a*zeta))
ax.legend([r'$\alpha$ = %.1f, $\zeta$ = %.1f' % (a,zeta) for a in alpha])

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x7f9fe0285ac8>

Of particular interest, Figure 4 shows that for large $\alpha$ the decay ratio switches from monotonically decreasing to monotonically increasing. This is undesireable as it implies that the high frequency error modes will damp more slowly than the mean error. In the context of shock fitting, as the $\xi_3$-isosurface converges to the shock it will "overshoot" in some regions and then converge to the shock from the downstream side. We would prefer that the grid evolve to eliminate as much high-frequncy error as possible early in the fitting process and then converge to the shock from the upstream side in a uniform manner. This behavior can be obtained by selecting parameters that results in a monotonically decreasing decay ratio.

To ensure a monotonically decreasing decay ratio, $\delta$ must be less than unity for $k\rightarrow \infty$. Using the expression for the limits of $\tau^{-1}$ from above:

\begin{align}
    \lim_{k\rightarrow\infty}\delta
    &= \frac{2}{\epsilon}\zeta\zeta^\prime \left(1 - \sqrt{1 - \zeta^{-2}}\right) \\
    &= 2\alpha\zeta^2 \left(1 - \sqrt{1 - \zeta^{-2}}\right)
\end{align}

From which it follows that to prevent overshooting the shock during the fitting process, me must satisfy the following inequality:

$$
    2\alpha\zeta^2\left(1 - \sqrt{1 - \zeta^{-2}}\right) \lt 1
    \quad\Longleftrightarrow\quad
    \zeta - \sqrt{\zeta^2 - 1} \lt \frac{\epsilon}{2\zeta^\prime}
$$

Perhaps the most important aspect of this equation is the fact that the stiffness ratio $\epsilon$ appears in the numerator on the right hand side of the inequality. We stated previously that lower values of $\epsilon$ are beneficial for accurate smoothing; however, if this parameter is too low, we will not damp out high-frequency error before we converge to the shock, causing local overshoots. The plot below shows the variation of the left hand side of the inequality and may be used to select an appropriate value of $\zeta$.

```python
zeta = np.logspace(0,np.log10(4),201)
lhs = zeta - np.sqrt(zeta**2 - 1)
ax = plt.figure().add_subplot(111)
ax.set_xlabel(r'Low Frequency Damping, $\zeta$')
ax.set_ylabel(r'High Frequency Decay Parameter, $\epsilon/2\zeta^\prime$')
ax.plot(zeta, lhs, marker='None', linewidth=2.0, color='C3')
ax.fill_between(zeta, 0.0, lhs, color='C3', alpha=0.5, linewidth=0)
ax.text(1.2, 0.1, 'May Overshoot')
ax.text(2.5, 0.6, 'Will Not Overshoot')

```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f9fdc7db0f0>

## Effects of Discretization

The preceding dynamic analysis of the grid equation of motion assumed the exact Laplacian of the $\xi_3$-isosurface position and velocity are computable. In practice, these operators will be estimated using finite differences. In this section, we analyze how this change alters the convergence dynamics of the fitting process.

(Note: I am conciously choosing not to use finite element approach here. I don't want to have to deal with inverting a mass matrix to get my time derivatives at each update. I know I could factorize and store the matrix or use some kind of approximation to deal with the coupling, but given the low spatial accuracy required here I don't think it's worth the extra effort.)

Let's once again consider the 1D homogenous version of Equation (X). This time, we approximate the curvature operator with the classical second order accurate finite difference approximation. This results in a set of coupled differential equations, shown below. Note that since the Lapalacian is with respect to the computational coordinate, the finite difference stencil spacing, $\Delta\xi$, is equal to unity and has been omitted.

$$
    \ddot{s}_i
    + 2\zeta\omega\dot{s}_i
    - \zeta^\prime \omega^\prime \left(\dot{s}_{i-1} - 2\dot{s}_i + \dot{s}_{i+1}\right)
    + \omega^2s_i
    - \omega^{\prime 2}\left( s_{i-1} - 2s_i + s_{i+1} \right)
    = 0
$$

Taking the fourier transform with respect to the spatial coordinate then yields:

$$
    \ddot{A}_k
    + 2\left(\zeta\omega + 2\left(1-\cos{k}\right)\zeta^\prime\omega^\prime\right)\dot{A}_k
    + \left(\omega^2 + 2\left(1-\cos{k}\right)\omega^{\prime 2}\right) A_k
    = 0
$$

This result is identical in structure to its continuous counterpart. The main difference is that the wavespeed premultiplier, $k^2$, is now replaced by the function $2\left(1-\cos{k}\right)$. A comparison of these two functions is shown below; the premultiplier for the discrete form is much lower than the continuous case at higher wave numbers. Note that we have terminated the wavenumber plot at $k = \pi$ because discrete grids will alias frequency content above this limit.

```python
k = np.linspace(0,np.pi)
ax = plt.figure().add_subplot(111)
ax.plot(k, k**2, k, 2*(1- np.cos(k)))
ax.set_xlabel(r'Wavenumber, $k$')
ax.set_ylabel(r'Wavespeed Premulitplier')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f9fe01a8390>

Since the fundmental structure of the differential equation has not changed, all of our previous equations remain vaild. However, the new pre-multiplier acts as a "warping" of the wavenumber input: essentially, the plot above shows that a disturbance with $k=\pi$ will evolve in the discrete setting "as-if" it had a wave number of $k\sim2$.

The following figures shows the effect of this warping on the decay time ratio, $\delta$. Since high-wavenumber disturbances evolve like lower wave numbers in the discretized equation, the effect of the discretization is the stretch the the decay ratio curves to the right. Fortunately, the effect is small since the decay ratio falls off most rapidly while $k < 1$, which is the region where the discrete pre-multiplier closely tracks its continuous counterpart.

```python
def compute_modified_decay_ratio(k, epsilon, zeta, zetap):
    alpha = zetap / (epsilon * zeta)
    ke2 = 2 * (1-np.cos(k)) * epsilon**2
    a = np.sqrt(1 + ke2)
    b = 1 + alpha * ke2
    return (zeta - np.sqrt(zeta**2 - 1))/(b*zeta - np.sqrt((b*zeta)**2 - a**2))

ax = plt.figure().add_subplot(111)
ax.set_xlabel(r'Wave Number, $k$')
ax.set_ylabel(r'Decay Time Ratio, $\delta$')

k = np.linspace(0, np.pi)
epsilon = 1.6
zetap = 1.0
zeta = [1.2, 1.4, 1.8, 2.6, 4.2]
for z in zeta:
    ax.plot(
        k, compute_decay_ratio(k, epsilon, z, zetap),
        marker='None',
    )
ax.set_prop_cycle(None) # Reset colors
for z in zeta:
    ax.plot(
        k, compute_modified_decay_ratio(k, epsilon, z, zetap),
        linestyle='--',
        marker='None',
    )
ax.legend([r'$\zeta$ = %.1f' % z for z in zeta])
ax.set_title(
    'Continuous (solid) vs. Discrete (dashed)\n' +
    r'$\epsilon$ = %.1f, $\zeta^\prime$ = %.1f' % (epsilon, zetap)
)
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f9fe00e1748>

## Equation of Motion Summary

The tailoring of our grid will be accomplished by evolving a constant-$\xi_3$ plane of the grid according the the forced visco-elastic membrane equation below. The volume grid is then generated generated algebraically by distributing points from the isosurface to the wall along the constant-$\zeta_1,\zeta_2$ lines of the background grid.

$$
    \ddot{s} + 2\left(\zeta\omega - \zeta^\prime \omega^\prime \nabla_\xi^2 \right)\dot{s}
             + \left(\omega^2 - \omega^{\prime 2}\nabla_\xi^2 \right)s
             = 2\zeta\omega\dot{s}_s + \omega^2 s_s
$$

This equation requires specifying four parameters: $\omega, \zeta, \omega^\prime, \zeta^\prime$. The rationale for setting these parameters is as follows:

1. We begin by taking $\zeta^\prime$ to be 1.0; this will ensure we always minimize the time required to damp out high-order fitting errors, per Equation (x).

2. The analyst selects a value of $\epsilon$ that yields acceptable steady-state fitting accuracy. Values on the order of 0.2 - 2.0 appear promising, though higher values may be desired for problems with non-smooth shock structures.

3. A value for $\zeta$ is selected that satisfies the inequality in Equation (x). Satisfying this inequality guarantees that the resulting scheme will not exhibit local overshoots and will always converge to a stationary shock from the upstream side of the shock.

4. The analyst selects a time constant for how quickly the mesh should converge to the shock; with this time constant, $\omega$ can be computed using Equation (X). Note that if a large value of zeta was selected, care must be taken not to set the time constant too low as this  may result in very high grid velocities early in the simulation due to a rapid damping of high-frequency error modes.


# Hamming网

## 简介

Hamming网使用汉明距离度量两个向量的距离。

汉明距离衡量的是两个相同长度向量对应位不同的数量，因此Hamming网仅能处理有限个离散值输入向量的模式识别问题。

Hamming网通过递归竞争判定输入向量最接近哪一个记忆的向量，实现对输入向量的模式识别。

对于一个特定的输入，Hamming网络收敛后仅有一个神经元的输出为非0值，该输出表示输入与该神经元记忆的向量最为相似。

## 符号定义

|符号|含义|
|:-:|:-:|
|$\bm{X}$|输入向量|
|$n$|输入向量维度|
|$\bm{a}$|前馈层输出|
|$\bm{W}$|前馈层权矩阵|
|$\bm{b}$|前馈层偏置|
|$f$|前馈层激活函数|
|$c$|前馈层神经元数量(网络记忆的模式总数)|
|$\bm{V}$|递归层权矩阵|
|$\bm{\hat{y}}$|递归层输出|
|$\bm{y}$|真实输出|
|$\bm{r_i}$|网络记忆的第i个向量|

## Hamming网的正向计算

### 前馈层

$$
\begin{equation}
    \bm{X} = [x_1, x_2, \cdots, x_n]^T, \ x_i\in\{-1, 1\}, i=1, 2, 3, \cdots, n
\end{equation}
$$

$$
\begin{equation}
    \bm{a} = f(\bm{W}\bm{X}+\bm{b})
\end{equation}
$$

$$
\begin{equation}
    f(x) =
    \left\{
    \begin{array}{cc}
    x,&x\geq0 \\
    0,&x<0
    \end{array}
    \right.
\end{equation}
$$

### 递归层

$$
\begin{equation}
    \left\{
    \begin{array}{cc}
    \bm{\hat{y}}(0) = \bm{a} \\
    \bm{\hat{y}}(t+1) = f(\bm{V}\bm{\hat{y}}(t))
    \end{array}
    \right.
\end{equation}
$$

当递归层的输出$\bm{\hat{y}}(t+1)$仅有一个数为非0数时递归结束

## Hamming网的权重设计与学习

### 权重设计

* 前馈层权重
$$
\begin{equation}
    \bm{W} =
    \left[
        \begin{array}{cc}
            \bm{r_1}^T \\
            \bm{r_2}^T \\
            \vdots \\
            \bm{r_c}^T \\
        \end{array}
    \right]
\end{equation}
$$

* 前馈层偏置
$$
\begin{equation}
    \bm{b} = [n, n, \cdots, n]^T
\end{equation}
$$

* 递归层权重
$$
\begin{equation}
    \bm{V} =
    \left[
        \begin{array}{cc}
            1&-\epsilon&\cdots&-\epsilon \\
            -\epsilon&1&\cdots&-\epsilon \\
            \vdots&\ddots&\ddots&\vdots\\
            -\epsilon&-\epsilon&\cdots&1
        \end{array}
    \right] , \
    \epsilon \in (0, \frac{1}{c})
\end{equation}
$$

可以看到，上述的权重设计中，递归层对自身的反馈是正反馈，而对其他神经元的反馈是抑制的。这体现了Hamming网的思路：**竞争性**

### 权重学习

递归层的权重无需改变，仅需要改变前馈层的权重

对于待记忆的模式向量$\bm{r_i}$,其期望输出为$[0, 0, \cdots, 1, \cdots, 0]^T$，其中第i位为非零值

因此，对于前馈层的输出，应当第i位的输出尽可能大，其他位的输出尽可能小

由此可以得到对于待记忆的模式向量$\bm{r_i}$，有下式的权重更新方法
$$
\begin{equation}
    \bm{W} = \bm{W} + [0, 0, \cdots, 1, \cdots, 0]^T\bm{r_i}^T
\end{equation}
$$
通过上式可以得到如下的参数学习伪代码
```
# 定义迭代次数
def iteration
# 定义待记忆模式向量
def r_1
def r_2
...
def r_c
# 定义权重矩阵
def W
# 定义偏置
def b

# 迭代开始
# 遍历所有待记忆的向量
for r_i in [r_1, r_2, ..., r_c]
    while True
        if argmax(W@r_i+b) == i
            break
        else
            # 生成mask向量
            mask_vector = zeros((c, 1))
            mask_vector[i] = 1
            W = W + mask_vector @ r_i.T

        iter += 1
        if iter == iteration:
            break
```
权重学习没有太大的意义，对于Hamming网来说，设计得到的权重在归一化的情况下基本上可以认为是最优权重。

```python
import random
import numpy as np
import matplotlib.pyplot as plt
```

```python
class HammingNN(object):

    def __init__(self, input_dim, output_dim, max_iters):
        """
        input_dim: 输入维度
        output_dim: 输出维度，也是总记忆向量的数量
        max_iters: 递归层最大递归次数
        """
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.max_iters = max_iters

        self.forward_weight = np.zeros((output_dim, input_dim+1))
        self.recursive_weight = np.ones((output_dim, output_dim))

    def forward(self, input_vector):

        # 前馈层
        input_vector_expend = np.concatenate((input_vector, [[1]]), axis=0)
        self.forward_output = self.activate_func(np.matmul(self.forward_weight, input_vector_expend))
        # 递归层
        pred_vector = self.forward_output
        pred_vector_list = [pred_vector]
        for _ in range(self.max_iters):

            if np.sum(pred_vector > 0) == 1:
                break
            else:
                pred_vector = self.activate_func(np.matmul(self.recursive_weight, pred_vector))
                pred_vector_list.append(pred_vector)

        return pred_vector, np.array(pred_vector_list)

    def weight_design(self, standard_vectors):
        """
        standard_vectors : 待记忆向量列表
        """
        assert len(standard_vectors) == self.output_dim

        # 前馈层的权重设计
        for i, standard_vector in enumerate(standard_vectors):
            self.forward_weight[i] = np.append(np.array(standard_vector).reshape(-1), self.input_dim)
        # 递归层的权重设计
        self.recursive_weight = -np.ones((self.output_dim, self.output_dim)) / self.output_dim
        self.recursive_weight += np.diag(np.ones(self.output_dim)+ 1 / self.output_dim)

        print("weight design done...")

    def weight_learning(self, standard_vectors):
        pass

    @staticmethod
    def activate_func(input_vector):
        mask_ = np.array(input_vector) > 0
        return input_vector * mask_.astype(np.int32)

```

```python
# 测试

# 假设标准向量为[-1, -1], [1, 1]
standard_vectors = [[-1, -1], [1, 1]]
# 定义模型
hammingNN = HammingNN(input_dim=2, output_dim=2, max_iters=10)
# 权重设计
hammingNN.weight_design(standard_vectors)

# 测试输入
# 测试输入由[-1, 1]随机取的100个点组成
test_data = list()
trace_list = list()
random.seed(1024)
for i in range(200):
    test_data.append([random.random()*2-1, random.random()*2-1])
test_data.append([-1, -1])
test_data.append([1, 1])

plt.figure(figsize=(10, 10))
plt.title("test samples", fontsize=15)
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.xlim(0, 4)
plt.ylim(0, 4)
for test_sample in test_data:
    _, pred_list = hammingNN.forward(np.array(test_sample).reshape(-1, 1))
    if test_sample == [-1, -1] or test_sample == [1, 1]:
        plt.scatter(pred_list[0][0], pred_list[0][1], label="{}".format(test_sample), s=200)
        plt.plot(pred_list[:, 0], pred_list[:, 1])
    else:
        plt.scatter(pred_list[0][0], pred_list[0][1])
        plt.plot(pred_list[:, 0], pred_list[:, 1])

plt.legend(fontsize=15)
plt.show()
```

可以看到，所有的样本点均收敛到了[c, 0]或是[0, c]。收敛有明显的分界点，并且作为待记忆向量的输入经过前向计算后位于坐标轴上


# Markov Chains
## Intro
$$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\EE}[2][\,\!]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}$$Sometimes to assume that the successive values $X_i$ are all independent is just unjustified. Thus we define the ***stochastic process*** $\CB{X_n, n=0,1,2,\dots}$. If $X_n = i$, we say that the process is in state $i$ at time $n$.

We suppose that whenever the process is in state $i$, there is a fixed probability $P_{i j}$ that it will next be in state $j$. That is, we suppose that:

$$P\CB{X_{n+1} = j \mid X_n = i, X_{n-1} = i_{n-1},\dots, X_0 = i_0} = P_{ij}$$

for all states $i_0,i_1,\dots,i_{n-1},i,j$ and all $n \geq 0$.

- Markov Property马氏性: $P\CB{X_{n+1} = j\mid X_n =i,X_{n-1} = i_{n-1},\cdots, X_0 = i_0} = P\CB{X_{n+1}= j \mid X_n = i}$ for all states $i_0,i_1,\dots,i_{n-1},i,j$ and all $n \geq 0$.

$Remark$

>$$P\CB{X_{n+1} = j\mid X_n =i,X_{n-1} = i_{n-1},\cdots, X_0 = i_0} = P\CB{X_{n+1}= j \mid X_n = i}\\
\iff P\CB{X_{n+1} = j ,X_{n-1} = i_{n-1},\cdots, X_0 = i_0\mid X_n =i} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\\
\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;= P\CB{X_{n+1}= j \mid X_n = i} \cdot P\CB{X_{n-1} = i_{n-1},\cdots, X_0 = i_0\mid X_n =i}$$
>
>对未来判断只依赖今天$X_n$，与历史无关

- Time-homogenous时齐性: $P\CB{X_{n+1}=j\mid X_n = i} = P\CB{X_1 = j \mid X_0 = i}$ for all $n$, meaning that the transition is independent of $n$.

***

Then we focus on the one-step transition matrix. Let $P_{ij}$ be the probability that the process will, when in state $i$, next make a transition to state $j$. Then, $P_{ij}\geq 0$ for $i,j \geq 0$ and

$$\sum_{j=0}^{\infty} P_{ij} = 1, i = 0,1,2,\dots\\
\mathbf{P} = \begin{Vmatrix}
P_{00} & P_{01} & P_{02} & \cdots \\
P_{10} & P_{11} & P_{12} & \cdots \\
\vdots & \vdots & \vdots & \\
P_{i0} & P_{i1} & P_{i2} & \cdots \\
\vdots & \vdots & \vdots & \\
\end{Vmatrix}$$

**e.g.4**

Suppose that whether or not it rains today depends on previous weather conditions through the last two days. Specifically, suppose that if it has rained for the past two days, then it will rain tomorrow with probability $0.7$; if it rained today but not yesterday, then it will rain tomorrow with probability $0.5$; if it rained yesterday but not today, then it will rain tomorrow with
probability $0.4$; if it has not rained in the past two days, then it will rain tomorrow with probablity $0.2$.

>Since it's about the last TWO days, we can define $4$ states:

>1. if it rained both today and yesterday
>2. if it rained today but not yesterday
>3. if it rained yesterday but not today
>4. if it did not rain either yesterday or today
>
>$\bspace\begin{Vmatrix}
0.7 & 0 & 0.3 & 0 \\
0.5 & 0 & 0.5 & 0 \\
0 & 0.4 & 0 & 0.6 \\
0 & 0.2 & 0 & 0.8
\end{Vmatrix}$
>
>The $0$s represent those impossible transition.
***

**e.g.** Random Walk Model

A Markov chain whose state space is given by the integers $i = 0,\pm 1, \pm 2, \dots$ is said to be a ***random walk*** if, for some number $0 < p < 1$, we have

$$P_{i,i+1} = p = 1 - P_{i,i-1}$$

We can think this as a model for an individual walking on a straight line who at each point of time either takes one step to the right with probability $p$ or one step to the left with probability $1-p$
***

**e.g.** Gambling Model, more about the Random walk.

Start from random walk model. The man now stand on a cliff where only points from $0$ to $N$ are available otherwise he will fall. Suppose he starts from point $0$. Then the model will be

$\bspace P_{i,i+1} = p = 1 - P_{i,i-1},i = 1,2,\dots,N-1$ and $P_{00} = P_{NN} = 1$
***

## Chapman-Kolmogorov Equations

Now we define the $n$-step transition probabilities $P_{ij}^{\:\!n} = P\CB{X_{n+k} = j \mid X_{k} = i}, n\geq 0, i,j \geq0$. And to compute the probabilities, the ***Chapman-Kolmogorov Equations*** says that

$$\begin{align}
P_{ij}^{\:\!n+m} &= P\CB{X_{n+m} = j \mid X_0 = i}\\
&=\sum_{k=0}^{\infty} P\CB{X_{n+m} = j, X_n = k \mid X_0 = i} \\
&= \sum_{k=0}^{\infty} P\CB{X_{n+m} = j \mid X_n = k , X_0 = i} P\CB{X_n = k \mid X_0 = i} \\
&= \sum_{k=0}^{\infty} P_{ik}^{\:\!n}P_{kj}^{\:\!m}\end{align}$$

for all $n,m\geq 0$ and all $i,j$.

If we let $\mathbf{P}^{\P{n}}$ denote the matrix of $n$-step transition probabilities $P_{ij}^{\:\!n}$, then the last equation is to say $\mathbf{P}^{\P{n+m}} = \mathbf{P}^{\P{n}} \cdot \mathbf{P}^{\P{m}}$. And thus by induction we have $\mathbf{P}^{\P{n}} = \mathbf{P}^n$

$Remark$

>时齐性 combined with c-k equation, we have
>
>$$P\CB{X_{n+k} = j \mid X_n = i} = P\CB{X_k = j \mid X_0 = i}$$

**e.g.9** e.g.4 revisited

Given it rained on Monday and Tuesday, what's the probability that it will rain on Thursday?

> $\bspace\begin{align}
\mathbf{P}^{\P{2} } &= \mathbf{P}^2 \\
&= \bspace\begin{Vmatrix}
0.7 & 0 & 0.3 & 0 \\
0.5 & 0 & 0.5 & 0 \\
0 & 0.4 & 0 & 0.6 \\
0 & 0.2 & 0 & 0.8
\end{Vmatrix}\cdot \begin{Vmatrix}
0.7 & 0 & 0.3 & 0 \\
0.5 & 0 & 0.5 & 0 \\
0 & 0.4 & 0 & 0.6 \\
0 & 0.2 & 0 & 0.8
\end{Vmatrix} \\[0.7em]
&= \bspace\begin{Vmatrix}
0.49 & 0.12 & 0.21 & 0.18 \\
0.35 & 0.20 & 0.15 & 0.30 \\
0.20 & 0.12 & 0.20 & 0.48 \\
0.10 & 0.16 & 0.10 & 0.64
\end{Vmatrix}
\end{align}$
>
>And to rain on Thursday is equivalent to the process being in either state $1$ or $2$ (here the state number are from $1$ to $4$). Thus the answer is $P_{11}^{2} + p_{12}^2 = 0.49 + 0.12 = 0.61$.
***

**e.g.10**

An urn always contains $2$ balls, red or blue. At each stage a ball is randomly chosen and then replaced by a new ball, which with probability $0.8$ is the same color, and with probability $0.2$ is the opposite color, as the ball it replaces. If initially both balls are red, find the probability that the fifth ball selected is red.

>We use $X_n$, the number of red balls in the urn after $n$-th selection, to construct the Markov Chian: $X_n, n \geq 0$ with states $0,1,2$ and the transition matrix $\mathbf{P}=\begin{Vmatrix}
0.8 & 0.2 & 0 \\
0.1 & 0.8 &0.1\\
0 & 0.2 & 0.8
\end{Vmatrix}$. Then we can calculate the desired probability:
>
>$$\begin{align}
P\CB{\text{fifth selection is red}} &= P\CB{\text{fifth selection is red} \mid X_0 = 2}\\
&= \sum_{i=0}^{2} P\CB{\text{fifth selection is red} \mid X_4 = i} \cdot P\CB{X_4 = i \mid X_0 = 2}\\
&= 0 \times P_{2,0}^{4} + 0.5\times P_{2,1}^{4} + 1 \times P_{2,2}^{4}\\
&= \cdots = 0.7048
\end{align}$$
***

**e.g.11**

Suppose that balls are successively distributed among $8$ urns, with each ball being equally likely to be put in any of these urns. What is the probability that there will be exactly $3$ nonempty urns after $9$ balls have been distributed?

>Let $X_n$ be the $r.v.$ that is mentioned in the context, the nonempty urns after $n$-th distribution. Then we have the transition probabilities: $P_{i,i} = i/8 = 1-P_{i,i+1}$ for $i=0,1,\dots,8$. Everything is easy until we're going to find $\mathbf{P}^8$. It's complicated, so here's how to simplify the calculation.
>
>The probability we need is $P_{03}^{9}$, or even simpler, $P_{13}^{\:\!8}$ because it's destined to go from $0$ urns to $1$ urn in the first distribution. After that, the first and the last $4$ columns and rows in $\mathbf{P}$ are not gonna help with finding the answer, so we combine them together, and obtain
>
>$$\begin{array}{rrc}
&& X_{n+1} \\
&& \begin{array}{cccc}
\:1\: & \:\:2 & \:\:\,3\!\! & \:\,\geq4
\end{array}\\
X_n & \begin{array}{c}
1 \\
2 \\
3 \\
\geq 4
\end{array} & \begin{Vmatrix}
1/8 & 7/8 & 0 & 0 \\
0 & 2/8 & 6/8 & 0 \\
0 & 0 & 3/8 & 5/8 \\
0 & 0 & 0 & 1
\end{Vmatrix}
\end{array} \Rightarrow \mathbf{P}^4 = \begin{Vmatrix}
0.00012 & 0.0256 & 0.2563 & 0.7178 \\
0 & 0.0039 & 0.0952 & 0.9009 \\
0 & 0 & 0.0198 & 0.9802 \\
0 & 0 & 0 & 1
\end{Vmatrix}$$
>
>hence $P_{13}^{\:\!8} = 0.0002\times0.2563+0.0256\times0.0952+0.02563\times0.0198+0.7178\times0 = 0.00756$
***

Let $\mathscr{A}$ be a set of states. To determine $\beta=P\CB{X_k \in \mathscr{A} \text{ for some }k=1,\dots,m \mid X_0 = i}$ for $i\notin\mathscr{A}$ we need a new Markov Chain $\CB{W_n,n\geq 0}$ whose states are the states that are not in $\mathscr{A}$ plus an additional state, namely, $A$. This one is different from what we've learned before is evidenced by the fact that:

$\bspace$Once the $\CB{W_n}$ enters state $A$, it remains there *forever*.

Here's a more formal definition: Let $X_n$ denote the state at time $n$ of the Markov Chain with transition probabilities $P_{i,j}$ (not $P_{ij}$ any more), define $N = \min\CB{n:X_n \in \mathscr{A}}$, or $N = \infty$ if $X_n \notin \mathscr{A}$ for all $n$. In words, $N$ is the ***first time***, called the ***hitting time***, the Markov chain enters the set of states $\mathscr{A}$. Now, define:

$\bspace W_n = \begin{cases}
X_n, &\text{if }n < N \\
A,   &\text{if }n \leq N
\end{cases}$

Its transition probabilities are: $\begin{cases}
Q_{i,j} = P_{i,j}, &\text{if } i \notin \mathscr{A}, j \notin \mathscr{A} \\[0.7em]
Q_{i,A} = \d{\sum_{j \in \mathscr{A}} P_{i,j}}, &\text{if } i \notin \mathscr{A} \\
Q_{A,A} = 1
\end{cases}$

Also notice that the original Markov chain will have entered a state in $\mathscr{A}$ by time $m$ $iff$ the state at time $m$ of the new Markov chain is $\mathscr{A}$, we see that

$$P\CB{X_k \in \mathscr{A} \text{ for some } k =1,2,\dots,m\mid X_0 =i}\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ \\
=P\CB{W_m = A \mid X_0 = i} = P\CB{W_m = A \mid W_0 = i} = Q_{i,A}^{m}$$

Thus the desired probablity is equal to an $m$-step transition probability of the new chain.

**e.g.**

In a sequence of independent flips of a fair coin, let $N$ denote the number of flips until there is a run of $3$ consecutive heads. Find $P\CB{N \leq 8}$ and $P\CB{N = 8}$.

>The desired state is $3$ consecutive thus we need states $0,1,2,3$ in Markov China implying how many consecutive heads we've so far reached. Then the transition matrix:
>
>$$\mathbf{P} = \begin{Vmatrix}
0.5 & 0.5 & 0 & 0 \\
0.5 & 0 & 0.5 & 0 \\
0.5 & 0 & 0 & 0.5 \\
0 & 0 & 0 & 1
\end{Vmatrix}$$
>
>Then the desired probablity is $P\CB{N = 8} = P^{8}_{0,3} = 107/256$
>
>Then $P\CB{N \leq 8} = P\CB{N \leq 8} - P\CB{N \leq 7} = P^{8}_{0,3} - P^7_{0,3} = \cdots$
>***
>Or another way using the preceding discussion
>
>We now assign FOUR states. $i = 0,1,2$ keep unchanged while $i=3$ means that the consecutive $3$ heads has just occurred and $i=4$ means that it has happened in the past.
>
>$$\mathbf{Q} = \begin{Vmatrix}
0.5 & 0.5 & 0 & 0 & 0\\
0.5 & 0 & 0.5 & 0 & 0\\
0.5 & 0 & 0 & 0.5 & 0\\
0 & 0 & 0 & 0 & 1 \\
0 & 0 & 0 & 0 & 1
\end{Vmatrix}$$
>
>Here $\mathscr{A}$ is $\CB{\text{consecutive $3$ heads happend}}$. Then $P\CB{N = 8} = Q_{0,3}^8$

***

Suppose now that we want to compute the probability that the $\CB{X_n, n \geq 0}$ chain, starting from state $i$ to $j$ at time $m$ without ever entering any of the states in $\mathscr{A}$, where neither $i$ or $j$ is in $\mathscr{A}$. Consider $\alpha = P\CB{X_m = j, X_k \notin \mathscr{A}, k = 1,2,\dots,m-1\mid X_0 = i}$ and the $W_m$ we've defined before, we have

$$\alpha = P\CB{ W_m = j \mid X_0 = i} = Q_{i,j}^{m}$$

Refer **e.g.11** for a better understanding.
***
Now when $i \notin \mathscr{A}$ but $j$ is! Then what's $\CB{X_m = j,X_k \notin \mathscr{A}, k = 0,1,\dots,m-1 \mid X_0 = i}$ for $j \in \mathscr{A}$. We can easily get that

$$\begin{align}
\alpha &= \sum_{r \notin \mathscr{A}} P_{r,j} \cdot Q_{i,r}^{m-1} \\
&= \sum_{r \notin \mathscr{A}}P\CB{X_m = j, X_{m-1} = r, X_k \notin \mathscr{A}, k = 0,1,\dots,m-2 \mid X_0 = i}\\
&= \sum_{r \notin \mathscr{A}} \big( P\CB{X_m = j \mid X_{m-1} = r, X_k \notin \mathscr{A}, k = 0,1,\dots,m-2, X_0 = i}\\
&\bspace\bspace\bspace \times P\CB{X_{m-1} = r, X_k \notin \mathscr{A}, k = 0,1,\dots,m-2\mid X_0 = i}\big)
\end{align}$$
***
And when $i \in \mathscr{A}$, we have $\alpha = \d{\sum_{r\notin\mathscr{A}} P_{i,r} \cdot Q_{r,j}^{m-1}}$.
***
And the last one, for when given that the chain starts in state $i$ and has not entered any state in $\mathscr{A}$ by time $n$. Then for $i,j \notin \mathscr{A}$ we have:

$$P\CB{X_n = j \mid X_0 = i, X_k \notin \mathscr{A}, k = 1,2,\dots,k-1} = \ffrac{Q_{i,j}^n} {\d{\sum_{r\notin\mathscr{A}}} Q_{i,r}^{n}}$$

$Remark$

>$$P\CB{X_n = j} = \sum_{i=0}^{\infty} P\CB{X_n = j \mid X_0 = i} \cdot P\CB{X_0 = i} = \sum_{i=0}^{\infty} P_{ij}^{n} \cdot P\CB{X_0 = i}$$

## Classification of States

State $j$ is said to be ***accessible***  from state $i$ if $P_{ij}^{n}>0$ for some $n \geq 0 $. When two states are **accessible** to each other, they're said to ***communicate***, written as $i \leftrightarrow j$.

- State $i$ communicates with state $i$ itself for all $i \geq 0$.
- If state $i$ communicates with state $j$, then state $j$ communicates with state $i$.
- If state $i$ communicates with state $j$, and state $j$ communicates with state $k$, then state $i$ communicates with state $k$.

Two states that communicate are said to be in the same ***class***. An obvious conclusion is that

$\bspace$Any two classes of states are EITHER identical OR disjoint, NO other possible relations.

The Markov chain is said to be ***irreducible*** if there is only one class, that is, if all states communicate with each other.
***
For any state $i$ we let $f_i$ denote the probability that, starting in state $i$, the process will ever *reenter* state $i$. State $i$ is said to be ***recurrent*** if $f_i = 1$ and ***transient*** if $f_i < 1$. 不如就叫复发态和暂住态

If **recurrent**, then state $i$ will happen *infinitely often*. However, if **transient**, each time the process enters state $i$ there will be a positive probability $1-f_i$ that it will *never* again enter that state again. Then the number of time periods that the process will be in state $i$ has a geometric distribution with finite mean $\ffrac{1}{1-f_i}$.

And we claim that state $i$ is *recurrent* $iff$ starting in state $i$, the expected number of time periods that the process is in state $i$ is *infinite*. But letting, $I_n = \begin{cases}
1, & \text{if }X_n = i\\
0, & \text{if }X_n \neq i
\end{cases}$, we have

$$\begin{align}
\EE{\d{\sum_{n=0}^{N}I_n} \mid X_0 = i} &= \d{\sum_{n=0}^{N}\EE{I_n\mid X_0 = i}} \\
&=\sum_{n=0}^{\infty} P\CB{X_n = i \mid X_0 = i} \\
&= \sum_{n=0}^{\infty} P_{ii}^{n}
\end{align}$$

$Proposition.1$

$\bspace$State $i$ is **recurrent** if $\d{\sum_{n=1}^{\infty}}P_{ii}^{n} = \infty$ and **transient** if $\d{\sum_{n=1}^{\infty}}P_{ii}^{n} < \infty$

$Remark$

>This argument also shows that a transient state will only be visited a finite number of times. And the straightforward conslusion is that in a finite-state Markov chain not all states can be transient.

$Corollary.2$

If state $i$ is **recurrent**, and state $i$ communicates with state $j$, then state $j$ is **recurrent**.

$Proof$

>Since state $i$ communicates with state $j$, there exist integers $k$ and $m$ such that $P_{ij}^k > 0$ and $P_{ji}^m > 0$. Now, for any integer $n$: $P_{jj}^{m+n+k} \geq P_{ji}^{m} P_{ii}^n P_{ij}^k \Rightarrow \d{\sum_{n=1}^{\infty} P_{jj}^{m+n+k}} \geq P_{ji}^m P_{ij}^{k} \d{\sum_{n=1}^{\infty} P_{ii}^n} = \infty$
>
>Since $P_{ji}^{m} P_{ij}^{k} > 0$, and $\sum\limits_{n=1}^{\infty} P_{ii^{n}}$ is infinite since state $i$ is **recurrent**. Thus, by $Proposition.1$ we have state $j$ is also **recurrent**.

$Remark$

>The preceding corollary also implies that transience is a class property. For if state $i$ is **transient** and communicates with state $j$, then state $j$ must also be **transient**.
>
>Also, "not all states in a *finite Markov chain* can be transient" $\Rightarrow$ "all states of a *finite irreducible Markov chain* are recurrent".

**e.g.16**

Let the Markov chain consisting of the states $0, 1, 2, 3$ have the transition probability matrix

$\bspace\mathbf{P} = \begin{Vmatrix}
0 & 0 & 0.5 & 0.5 \\
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 1 & 0 & 0
\end{Vmatrix}$

Determine which states are transient and which are recurrent.

> It is a simple matter to check that all states communicate and, hence, since this is a finite chain, all states must be recurrent.
***

**e.g.17**

Consider the Markov chain having states $0, 1, 2, 3, 4$ and

$\bspace\mathbf{P} = \begin{Vmatrix}
0.5 & 0.5 & 0 & 0 & 0\\
0.5 & 0.5 & 0 & 0 & 0\\
0 & 0 & 0.5 & 0.5 & 0\\
0 & 0 & 0.5 & 0.5 & 0\\
0.25 & 0.25 & 0 & 0 & 0.5
\end{Vmatrix}$

Determine the recurrent state.

> This chain consists of the three classes: $\CB{0,1}, \CB{2,3},\CB{4}$. Easy to find that the first two classes are recurrent and the third transient.
***

**e.g.18** A Random Walk

Consider a Markov chain whose state space consists of the integers $i = 0, \pm1, \pm2, \dots$, and has transition probabilities given by $P_{i,i+1} = p = 1 - P_{i,i-1}$ for all $i$ where $1>p>0$. Since all states clearly communicate, so they all are either **transient**, or **recurrent**. So we are going to consider $\sum_{n=1}^{\infty} P^n_{00}$ whether it's finite or infinite.

> First thing to notice is that $n$ has to be even so that $P_{00}^{2n-1}=0$. Then for even time transition we have
>
>$$P_{00}^{2n}=\binom{2n}{n}p^n\P{1-p}^{n}=\ffrac{\P{2n}!}{n!n!}\P{p-p^2}^n$$
>
>By ***Stirling formula***: ($n! \sim n^{n+0.5} e^{-n} \sqrt{2\pi} $),
>
>$$\sum_{n=1}^{\infty}P_{00}^{2n} \sim\sum_{n=1}^{\infty}\ffrac{\P{4p-4p^2}^n} {\sqrt{\pi n}}, \begin{cases}
\text{recurrent},&\text{if the value is }\infty\\
\text{transient},&\text{if the value is }<\infty
\end{cases}$$
>
>Applying some knowledge from number serise, since $4p\P{1-p} < 1$ unless $p=0.5$, we assert that the chain is recurrent only when $p=0.5$.

$Remark$

>For this special case it's got a special name ***symmetric random walk***. And we could expand this to the second dimension where
>
> $$\mathbf{P}_{\P{i,j},\P{i+1,j}} = \mathbf{P}_{\P{i,j},\P{i-1,j}} = \mathbf{P}_{\P{i,j},\P{i,j+1}} = \mathbf{P}_{\P{i,j},\P{i,j+1}} = 0.25$$
>
>And this is also recurrent, using the same method and we can prove this. Find how in the textbook, wolalalalalalala!
>
>And that's the end, no higher dimensions have the same property. Oh my sad drunk man.

$Remark$

>For one-dimensional random walk as discussed in **e.g.18**, an direct argument can be made for establishing recurrence in the symmetric case, and for determining the probability that it ever returns to state $0$ in the nonsymmetric case. The drunk man starts at $0$, and we first let $\beta = P\CB{\text{ever return to }0}$. Then we write
>
>$$\beta = P\CB{\text{ever return to }0 \mid X_1 =1} p + P\CB{\text{ever return to }0 \mid X_1 =-1}\P{1- p}$$

>Now let $\alpha$ denote the probability that the Markov chain will ever return to $0$ given that it is currently in $1$. Condition on the next transition we obtain
>
>$$\begin{align}
\alpha &=P\CB{\text{ever return to }0 \mid X_1 = 1} \\
&= P\CB{\text{ever return to }0\mid X_1 = 1,X_2 = 0}\P{1-p} + P\CB{\text{ever return to }0\mid X_1 = 1,X_2 = 2}p\\
&= 1\times\P{1-p} + P\CB{\text{ever return to }0 \mid X_1 = 1} \cdot P\CB{\text{ever return to }1 \mid X_1 = 2}p\\
&= 1-p + p\alpha^2
\end{align}$$
>
>The roots are $1$ and $\P{1-p}/p$. When $p=0.5$ we have $\alpha = 1$. Then by symmetry, we have
>
>$$\beta = 1 \times p + 1 \times \P{1-p} = 1$$
>
>While when $p\neq 0.5$ we have $\alpha = \ffrac{1-p} {p}$, say $p>0.5$, then we first have $P\CB{\text{ever return to }0 \mid X=-1} = 1$ so that $\beta =\alpha p + 1-p = 2\P{1-p}$. And similarly we have when $p < 0.5$, $\beta = 2p$ so we conclude that
>
>$$\beta = P\CB{\text{ever return to }0} = 2\min\P{p,1-p}$$
***

**e.x.19** On the Ultimate Instability of the Aloha Protocol

>This problem is interesting. But time sucks! Skipped!

## Long-Run Proportions and Limiting Probabilities

For pairs of states $i \neq j$, let $f_{i,j}$ denote the probability that the Markov chain, starting in state $i$, will ever make a transition into state $j\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}$. That is,

$$f_{i,j} = P\CB{X_n = j \text{ for some }n>0 \mid X_0 = i}$$

$Proposition.3$

If $i$ is recurrent and $i$ communicates with $j$ , then $f_{i,j} = 1$.

$Proof$

>Since $i$ and $j$ communicate there's a non-negative $n$ $s.t.$ $P_{i,j}^{n} > 0$. Let $X_0 = i$ and let the first opportunity is a success if $X_n = j$, with probability $P_{i,j}^{n} > 0$. Since state $i$ is recurrent, thus every time the chain enters state $i$ we start to check whether $n$ steps later it stops at state $j$. And this success is with probability $P_{i,j}^{n} > 0$. Till it happens, we will stop. Then it's a geometric distribution! Then it follows that with probability $1$ a success will eventually occur and so, with probability $1$, state $j$ will eventually be entered.

$Def$

If state $j$ is recurrent, let $m_j$ denote the expected number of transitions that it takes the Markov chain when starting in state $j$ to return to that state. That is, with $N_j = \min\CB{n>0:X_n = j}$, this value equal to the number of transitions until the Markov chain makes a transition into state $j$, $m_j = \Exp\SB{N_j \mid X_0 = j}$. Then the recurrent state $j$ is ***positive recurrent*** if $m_j < \infty$ and ***null recurrent*** if $m_j = \infty$.

Letting $\pi_j$ denote the long-run proportion of time that the Markov chain is in state $j$, we have the following proposition.

$Proposition.4$

If the Markov chain is irreducible and recurrent, then for any initial state $\pi_j = \ffrac{1} {m_j}$.

$Proof$

>Suppose that the Markov chain starts in state $i$, and let $T_n$ for $n \geq 2$ is the number of transitions between the $\P{n-1}$th and the $n$th transition into state $j$, that is
>
>- $T_1$: the number of transitions until the chain enters state $j$
>- $T_2$: the additional number of transitions from time $T_1$ until the Markov chain next enters state $j$
>- $T_3$: the additional number of transitions from time $T_1 + T_2$ until the Markov chain next enters state $j$, and so on.
>
>By $Proposition.4$, $w.p.$ $1$ a transition into $j$ will eventually occur. And by the property of markov chain, $T_i$ are independent and identically distributed with mean $m_j$. After all these definitions we have
>
>$$\begin{align}
\pi_j &= \lim_{n \to \infty} \ffrac{n} {\d{\sum_{i=1}^{n} T_i}} \\
&= \lim_{n \to \infty} \ffrac{1} {\ffrac{T_1} {n} + \ffrac{T_2 + T_3+ \cdots + T_n} {n}}
\end{align}$$
>
>Notice that $\lim\limits_{n\to\infty}\ffrac{T_1} {n} = 0$ and by the strong law of large numbers, we have
>
>$$\lim_{n\to\infty} \ffrac{T_2 + \cdots + T_n} {n} = \lim_{n\to\infty} \ffrac{T_2 + \cdots + T_n} {n-1}\cdot\ffrac{n-1} {n} = m_j$$
>
>Thus $\pi_j = 1/m_j >0$

$Remark$

>It follows from the preceding that state $j$ is **positive recurrent** $iff$ $\pi_j > 0$ which is also equivalent to $m_j < \infty$.

$Proposition.5$

If $i$ is **positive recurrent** and $i \leftrightarrow j$ then $j$ is also **positive recurrent**.

$Proof$

>Let $n$ be such that $P_{i,j}^{n}>0$. Because $\pi_j$ is the long-run proportion of time that the chain is in state $i$, and $P_{i,j}^{n}$ is the long-run proportion of time when the Markov chain is in state $i$ that it will be in state $j$ after $n$ transitions. Then:

>$$\begin{align}
\pi_i \cdot P_{i,j}^{n} &= \text{long-run proportion of time the chain is in } i \\
&\bspace\text{and will be in $j$ after $n$ transitions}\\
&= \text{long-run proportion of time the chain is in } i \\
&\bspace\text{and will be in $j$ before $n$ transitions}\\
& \leq \text{long-run proportion of time the chain is in } i
\end{align}$$
>
>Hence, $\pi_j \geq \pi_j\cdot P_{i,j}^n > 0$, showing that j is positive recurrent.

$Remark$

> **positive recurrent** is a class property, well, so is the **null recurrent**. Since being recurrent and being positive recurrent are both class properties.
>
>Also, *an **irreducible finite** state Markov chain must be **positive recurrent***. Since we've already known that such a chain  must be recurrent; hence, all its states are either positive recurrent or null recurrent. If they were null recurrent then all the long run proportions would equal $0$, which is impossible when there are only a finite number of states.

Then from $\pi_i$ we move to $\pi_j$ by summing over all $i$: $\pi_j = \sum \pi_i P_{i,j}$

$Theorem.1$

Consider an irreducible Markov chain. If the chain is **positive recurrent** then the long-run proportions are the unique solution of the equations

$$\begin{cases}
\pi_j = \d{\sum_i \pi_i \cdot P_{i,j}},\bspace j \geq 1\\
\d{\sum_j \pi_j = 1}
\end{cases}$$

Moreover, if there is no solution of the preceding linear equations, then the Markov chain is either **transient** or **null recurrent** and all $\pi_j = 0$.

**e.g.20**

Assume that if it rains today, then it will rain tomorrow with probability $\alpha$; and if it does not rain today, then it will rain tomorrow with probability $\beta$. If we say that the state is $0$ when it rains and $1$ when it does not rain. What's $\pi_0$ and $\pi_1$?

>The equations are
>
>$$\begin{cases}
\pi_0 = \alpha \pi_0 + \beta \pi_1 \\[0.6em]
\pi_1 = \P{1-\alpha} \pi_0 + \P{1-\beta} \pi_1 \\[0.6em]
\pi_0 + \pi_1 = 1
\end{cases} \Rightarrow \begin{cases}
\pi_0 = \ffrac{\beta} {1+\beta-\alpha}\\
\pi_1 = \ffrac{1-\alpha} {1+\beta -\alpha}
\end{cases}$$
***

**e.g.23** The Hardy–Weinberg Law and a Markov Chain in Genetics

Consider a large population of individuals. Assume that the proportions of individuals whose gene pairs are $AA$, $aa$, or $Aa$ are, respectively, $p_0$, $q_0$, and $r_0$ where ($p_0 + q_0 + r_0 = 1$). We are interested in determining the proportions of individuals in the next generation whose genes are $AA$, $aa$, or $Aa$. Calling these proportions $p$, $q$, and $r$.

>First we can calculate the probability of a randomly chosen gene will be type $A$ in the next generation:
>
>$$P\CB{A} = P\CB{A\mid AA}p_0 + P\CB{A\mid aa}q_0+P\CB{A\mid Aa}r_0 = p_0 + \ffrac{r_0} {2}$$
>
>And similarly we have $P\CB{a} = q_0 + \ffrac{r_0}{2}$. Thus we have, under random mating,
>
>$$p=P\CB{A}P\CB{A}, q = P\CB{a}P\CB{a}, r = 2P\CB{A}P\CB{a}$$
>
>And an interesting fact is that the fraction of its genes that are $A$, will be unchanged from the previous generation. One way is by arguing that the total gene pool has not changed from generation to
generation or by the following simple algebra:
>
>$$\begin{align}
p + \ffrac{r}{2} &= \P{p_0 +r_0/2}^2 + \P{p_0 + r_0/2}\P{q_0 + r_0/2} \\
&= \P{p_0 +r_0/2}\SB{p_0 + r_0/2 + q_0 + r_0/2}\\[0.6em]
&\bspace\text{since }{p_0+q_0+r_0 = 1}\\[0.6em]
&= p_0 + r_0/2 = P\CB{A}
\end{align}$$
>
>And then for a given individual, let $X_n$ denote the genetic state of her descendant in the $n$th generation. The transition probability matrix of this Markov chain, namely,
>
>$$\begin{Vmatrix}
p + \ffrac{r}{2} & 0 & q+ \ffrac{r}{2}\\
0 & q + \ffrac{r}{2} & p + \ffrac{r}{2}\\
\ffrac{p} {2} + \ffrac{r} {4} & \ffrac{q} {2} + \ffrac{r} {4} & \ffrac{p} {2} + \ffrac{q} {2} + \ffrac{r} {2}
\end{Vmatrix}$$
>
>And if we want to find the limiting probabilities, here's the equations
>
>$$\begin{cases}
p = p\P{p+\ffrac{r} {2}} + r\P{\ffrac{p} {2}+\ffrac{r} {4}} = \P{p+\ffrac{r} {2}}^2\\
q = q\P{q+\ffrac{r} {2}} + r\P{\ffrac{q} {2}+\ffrac{r} {4}} = \P{q+\ffrac{r} {2}}^2\\
p+q+r = 1
\end{cases}$$
>
>And these equations are just the same with the discussion before for just the next generation.
***

**e.g.24**

Suppose that a production process changes states in accordance with an irreducible, positive recurrent Markov chain having transition probabilities $P_{ij}, i,j = 1,\dots, n$, and suppose that certain of the states are considered acceptable and the remaining unacceptable. Let $A$ denote the acceptable states and $A^c$ the unacceptable ones. If the production process is said to be "up" when in an acceptable state and "down" when in an unacceptable state, determine

1. the rate at which the production process goes from up to down
2. the average length of time the process remains down when it goes down
3. the average length of time the process remains up when it goes up

>Let $\pi_k$ denote the long-run proportions. Now for $i \in A$ and $j \in A^c$, the rate at which the process enters state $j$ from state $i$ is: $\pi_i P_{ij}$ and thus
>
>$\bspace\text{rate of entering }j \text{ from }A = \d{\sum_{i \in A} \pi_i P_{ij} \Rightarrow \text{rate of breakdowns} = \sum_{j \in A^c} \sum_{i \in A} \pi_i P_{ij}}$
>***
>Then let $\bar U$ and $\bar D$ denote the average time the process remains up when it goes up and down when it goes down. Because there is a single breakdown every $\bar U + \bar D$ time units on the average, it follows heuristically启发式地 that the rate at which breakdowns occur is $1/ \P{\bar U + \bar D}$.
>
>$$\sum_{j \in A^c} \sum_{i \in A} \pi_i P_{ij} = \ffrac{1} {\bar U + \bar D}$$
>
>And the second equation comes from the thoughts on the percentage of time the process is up, which should be $\sum\limits_{i\in A} \pi_i$. Then, by the definitine of $\bar U$ and $\bar D$, we have
>
>$$\sum_{i\in A} \pi_i = \ffrac{\bar U} {\bar U + \bar D}$$
>
>Combine the two equations and solve it, we have:
>
>$$\bar U = \ffrac{\sum_{i\in A} \pi_i} {\sum_{j \in A^c} \sum_{i \in A} \pi_i P_{ij}}, \bar D = \ffrac{1 - \sum_{i\in A} \pi_i} {\sum_{j \in A^c} \sum_{i \in A} \pi_i P_{ij}} = \ffrac{\sum_{i\in A^c} \pi_i} {\sum_{j \in A^c} \sum_{i \in A} \pi_i P_{ij}}$$
***

The long run proportions $\pi_j,j\geq 0$ are often called ***stationary probabilities***. The reason being that if the initial state is chosen according to the probabilities $\pi_j,j\geq 0$, then the probability of being in state $j$ at any time $n$ is also equal to $\pi_j$:

$\bspace P\CB{X_0 = j} = \pi_j,j\geq0 \Rightarrow P\CB{X_n = j} = \pi_j, j\geq 0, \forall n$

And this seems obvious by induction. If it's true when $n=0$ and suppose it true for $n-1$ then we can write:

$\bspace\begin{align}
P\CB{X_n =j}&= \sum_i P\CB{X_n = j \mid X_{n-1} = j }\cdot P\CB{X_{n-1} = i}\\
&= \sum_i P_{ij} \pi_i \\
&= \pi_j
\end{align}$

**e.g.25** A conprehensive example

Numbers of people check in the hotel on successive days  are independent Poisson $r.v.$s with mean $\lambda$. Number of days one stay in the hotel is a geometric $r.v.$ with parameter $P$, $0<p<1$. (Thus no matter how long he has stayed in the hotel, the probability that he left tomorrow is still $p$). If $X_n$ denotes the number of people that are checked in the hotel at the beginning of day $n$ then $\CB{X_n:n \geq 0}$ is a Markov chain.

1. Find the transition probabilities
2. Find $\Exp\SB{X_n \mid X_0 = i}$
3. Find the stationary probabilities

> Let $R_i$ be the number of perople that remain another day, and we can say that it is actually a binomial $r.v.$ with parameter $i$ and $1-P$. And let $N$ be the number of new people that check in that day, we see that
>
>$$\begin{align}
P_{i,j} &= P\CB{R_i + N = j}\\
&= \sum_{k=0}^{i} P\CB{R_i + N = j \mid R_i = k} \cdot\binom{i} {k} \P{1-p}^{k}p^{i-k}\\
&= \sum_{k=0}^{\min\P{i,j}} P\CB{N = j-k} \cdot\binom{i} {k} \P{1-p}^{k}p^{i-k}\\
&= \sum_{k=0}^{\min\P{i,j}} e^{-\lambda}\ffrac{\lambda^{j-k}} {\P{j-k}!} \cdot\binom{i} {k} \P{1-p}^{k}p^{i-k}\\
\end{align}
$$
>
> Notice that $\Exp\SB{X_n \mid X_{n-1} = i} = \Exp\SB{R_i + N} = iq + \lambda, q = 1-p$. Consequently, $\Exp\SB{X_n \mid X_{n-1}} = X_{n-1} q + \lambda$. And take the expectation again yielding that
>
>$$\Exp\SB{X_n} = \lambda + q\Exp\SB{X_{n-1}}$$
>
>Iterating the preceding gives $\Exp\SB{X_n} = \lambda\P{1+q+q^2+\cdots+q^{n-1}} +q^n\Exp\SB{X_0} $. Thus, we have $\Exp\SB{X_n \mid X_0 = i} = \ffrac{\lambda\P{1-q^n}} {p} + q^n i $
>***
>As for the stationary probability, the fact is that using the result in the first problem and solve that equation set, is just too complicated. Rather we will make use of the fact that the **stationary probability distribution** is the *only distribution on the initial state* that results in the *next state* having the *same distribution*.
>
>Consider the initial state $X_0$ and the next one, we need to assume a distribution for the initial number of people checked. Intuitively, it should be a Poisson $r.v.$ since the number of people check in the next day is a Poisson $r.v.$ with parameter $\lambda$.
>
>So now we assume the initial state $X_0$ has a Poisson distribution with mean $\alpha$, then the people left in the next day is also a poisson $r.v.$ with mean $\alpha \cdot q$. So that to find the $\pi_i$, we have
>
>$$\alpha = \lambda + \alpha \cdot q \Rightarrow \alpha = \ffrac{\lambda} {p}$$
>
>So that the distribution of all states are Poisson Distribution with parameter $\ffrac{\lambda} {p}$, which is exactly the desired stationary distribution. Then the stationary probabilities are
>
>$$\pi_i = \exp\CB{-\ffrac{\lambda} {p}} \P{\ffrac{\lambda} {p}}^{i} \ffrac{1} {i!},\bspace i \geq 0$$

$Remark$

>The generalization of this example and other examples are skipped... sad
***

$Proposition.6$

Let $\CB{X_n, n \geq 1}$ be an irreducible Markov chain with stationary probabilities $\pi_j$ for $j\geq0$, and let $r$ be a bounded function on the state space. Then, $w.p. 1$, we have

$$\lim_{N\to\infty} \ffrac{\sum\limits_{n=1}^{N} r\P{X_N}}{N} = \sum_{j=0}^{\infty} r\P{j} \pi_j$$

$Proof$

>If we let $a_{j}\P{N}$ be the amount of time the Markov chain spends in state $j$ during time periods $1,\dots,N$, then
>
>$$\sum_{n=1}^{N} r\P{X_N} = \sum_{j=0}^\infty a_j\P{N} r\P{j}$$
>
>Since $\ffrac{a_j\P{N}} {N} \to \pi_j$ the result follows from the preceding upon dividing by $N$ and then letting $N\to\infty$.

## Limiting Probabilities

A chain that can only return to a state in a multiple of $d>1$ steps is said to be ***periodic*** and does not have limiting probabilities. One example could be the chain where $P_{1,0} =P_{0,1} = 1$ so that

$$P_{0,0}^{\P{n}} = \begin{cases}
1,&\text{if $n$ is even}\\
0,&\text{if $n$ is odd}
\end{cases}$$

and it doesn't have a limiting probabilities as $n\to \infty$.

However, for an irreducible chain that is not **periodic**, and such chains are called ***aperiodic***, the limiting probabilities will always exist and will not depend on the initial state, with its value $\pi_j$ for state $j$, same with the long-run proportion of time the chain is in state $j$. We can find the result by first letting $\alpha_j = \d{\lim _{n\to\infty} P\CB{X_n = j} }$; then since $\d{\sum_{i=0}^{\infty} P\CB{X_n= i}=1}$ and

$$P\CB{X_{n+1} = j} = \sum_{i=0}^{\infty}P\CB{X_{n+1}= j \mid X_n = i} \cdot P\CB{X_n = j} = \sum_{i=0}^{\infty} P_{ij} P\CB{X_n = i}$$

letting $n\to\infty$ in the preceding two equations yields, upon assuming that we can bring the limit inside the summation, that

$$\alpha_j = \sum_{i=0}^{\infty} \alpha_i   P_{ij} ,\bspace 1 = \sum_{i=0}^{\infty} \alpha_j$$

And these're the SAME equations for $\pi_j$, showing that actually $\alpha_j = \pi_j, j \geq 0$. An **irreducible**, **positive recurrent**, **aperiodic** Markov chain is said to be ***ergodic***.

**<center>Summary</center>**

- $i \to j$: **accessible** $\iff$ $\exists n \geq 0$, $P_{ij}^n>0\\[0.6em]$
- $i \leftrightarrow j$: **communicate** $\iff$ $\exists n,m\geq 0$, $P_{ij}^n>0, P_{ji}^m>0\\[0.6em]$
- **Irreducible**, if there's only $1$ class, meaning that all states are **communicate** with each other.$\\[1em]$
- $f_{ij} = P\CB{\text{ever return }j \mid X_0 = i}$, and we have $f_{ii} = P\CB{\text{ever return }i \mid X_0 = i}\\[0.5em]$
    - $f_{ii} = 1$, state $i$ is **recurrent**
    - $f_{ii} < 1$, state $i$ is **transient**$\\[1em]$
- **Hitting time**终于说出口了没错就是他，*Stopping time*: $\tau_j = \inf\CB{n\geq 0:X_n=j}$, or a stronger version: $\tau^+=\inf\CB{n>0:X_n=j}$.
- The *expected* **hitting time**: $m_{ij} = \Exp\SB{\tau_j^+ \mid X_0 = i} = \sum\limits_{n=1}^{\infty} n\cdot P\CB{\tau_j^+ = n\mid X_0 = i} = \sum\limits_{n=1}^{\infty} n\cdot f^{\P{n}}_{ij}$
- Here the $f^{\P{n}}_{ij}$ is another probability defined as:
$$f^{\P{n}}_{ij} = P\CB{X_n = j,X_m\neq j,1\leq m < n} = P_i\CB{ \tau_j^+ = n }$$
- And then we have $f_{ij} = \sum\limits_{n=1}^{\infty} f_{ij}^{\P{n}} = P\CB{\exists\, n>0, \space s.t.\space X_n=j\mid X_0=i}$
- Also in $homework.6$ we've proved that $P_{ij}^{\P{n}} = \d{\sum_{m=1}^{n} f_{ij}^{\P{m}}\cdot P_{ij}^{\P{n-m}} }$. With this we can further infer that (or use the indicator method like in $Proposition4.1$)
$$\begin{align}
f_{ii} = 1 &\iff \sum\nolimits_{i=0}^{\infty} P_{ii}^{\P{n}} = \infty\\
f_{ii} < 1 &\iff \sum\nolimits_{i=0}^{\infty} P_{ii}^{\P{n}} = \ffrac{1} {1-f_{ii}} < \infty
\end{align}$$
- And finally we have
$$\begin{align}
f_{ii}<1, \text{transient} &\Rightarrow m_{ii} = \infty \\
f_{ii}=1, \text{recurrent} &\Rightarrow \begin{cases}
m_{ii} = \infty, &\text{null recurrent}\\[0.7em]
m_{ii} < \infty, &\text{positive recurrent}
\end{cases}
\end{align}\\[0.5em]$$

- The **long-run proportion of time**, or the **stationary probability**, is $\pi_j = 1/m_{jj}$. Meaning that at later time, $\forall \;n$, $P\CB{X_n=j} = \pi_j = P\CB{X_0 = j}, j\geq 0$.
- **stationary probability** may not be the **limiting probability**, which will exist only when the Markov Chain is **not periodic (aperiodic)**, and **irreducible**.
- **ergodic** means **irreducible**, **positive  recurrent**, and **aperiodic**.
- More on **periodic**. If **irreducible**, and for one state that $d_j=1$, we then have all states with period $1$. And then **aperiodic**.
- ***Limit Theorem***: An **irreducible**, **aperiodic** Markov Chain belongs to one of the following classes
    - Either the states are all **transient** or all **null recurrent**, where $p_{ij}^{\P{n}} \to 0$ as $n\to\infty$ for all $i,j$ and there exists no stationary distribution
    - Or else, all states are **positive recurrent**, that is when we have the stationary distribution

$$\lim_{n\to\infty} p_{ij}^{\P{n}} = \pi_j >0$$

## Some Applications
### The Gambler's Ruin Problem

Consider a gambler who at each play of the game has probability $p$ of winning one unit and probability $q = 1 − p$ of losing one unit. Assuming that successive plays of the game are independent, what is the probability that, starting with $i$ units, the gambler’s fortune will reach $N$ before reaching $0$?

Let $X_n$ denote the player's fortune at time $n$, then the process $\CB{X_n,n =0,1,2,\dots}$ is a Markov chain with transition probabilities $P_{00} = P_{NN} = 1$ and $P_{i,i+1} = p = 1-P_{i,i-1}$ for $i=1,2,\dots,N-1$.

There're $3$ classes and $\CB{N}$ and $\CB{0}$ are recurrent and $\CB{1,2,\dots, N-1}$ is transient. Since each transient state is visited only finitely often, it follows that, after some finite amount of time, the gambler
will either attain his goal of $N$ or go broke.

Let $P_i$, $i = 0, 1, \dots, N$, denote the probability that, starting with $i$, the gambler’s fortune will eventually reach $N$. Condition on the intial play, we obtain:

$$P_i = p\cdot P_{i+1} + q\cdot P_{i-1} \stackrel{p+1=1}{\Longrightarrow} P_{i+1} - P_i = \ffrac{q} {p}\P{P_i - P_{i-1}}, \bspace i=1,2,\dots,N-1$$

Solve this progression we have

$$P_i = P_1\SB{1+\P{\ffrac{q} {p}} + \P{\ffrac{q} {p}}^2 + \cdots + \P{\ffrac{q} {p}}^{i-1}} = \begin{cases}
i P_1, &\text{if }\ffrac{q} {p}=1\\
\ffrac{1-\P{\ffrac{q} {p}}^i} {1-\ffrac{q} {p}} P_1, &\text{if }\ffrac{q} {p}\neq1
\end{cases}$$

then by the fact that $P_N=1$, we have

$$P_1 = \begin{cases}
\ffrac{1}{N}, &\text{if }\ffrac{q} {p}=1\\
\ffrac{1-\ffrac{q} {p}}{1-\P{\ffrac{q} {p}}N}, &\text{if }\ffrac{q} {p}\neq1
\end{cases}\bspace P_i = \begin{cases}
\ffrac{i}{N}, &\text{if }\ffrac{q} {p}=1 \iff p=0.5\\
\ffrac{1-\P{\ffrac{q} {p}}^i} {1-\P{\ffrac{q} {p}}^N}, &\text{if }\ffrac{q} {p}\neq1 \iff p\neq 0.5
\end{cases}$$

As $N\to\infty$, we have $P_i = 0$ when $p\leq 0.5$, and $P_i = 1-\P{\ffrac{q} {p}}^i$ when $p>0.5$.

An application of this, the drug testing. Suppose that two new drugs have been developed for treating a certain disease. Drug $i$ has a cure rate $P_i$ for $i=1,2$, in the sense that each patient treated with drug $i$ will be cured with $P_i$, however two unknown rates. Our interest is to determine whether $P_1 >P_2$ or $P_2>P_1$. We'll test paris of patients where one for drug $1$ and one for the other. Let

$$X_j = \begin{cases}
1,&\text{if the patient in the $j$th pair to receive drug number $1$ is cured}\\
2,&\ow
\end{cases}\\
Y_j = \begin{cases}
1,&\text{if the patient in the $j$th pair to receive drug number $2$ is cured}\\
2,&\ow
\end{cases}$$

For a predetermined positive integer $M>0$ the test stops after pair $N$ where $N$ is the first value of $n$ such that

$$\sum_{i=1}^{n}X_i - \sum_{i=1}^{n}Y_i = \pm M$$

For positive right side we asset that $P_1>P_2$ and the inverse one otherwise. Then what's the probability that the test will incorrectly assert $P_1>P_2$ when actually $P_1<P_2$?

Note that after each pair is checked the cumulative difference of cures using drug $1$ versus drug $2$ will either go up by $1$ with probability $P_{1}\P{1-P_2}$ or go down by $1$ with probability $\P{1-P_1}P_2$, or remain the same otherwise. Hence, if we only consider those pairs in which the cumulative difference changes, then the difference will go up $1$ with probability $p$ and down $1$ with probability $1-p$ where

$$p=P\CB{\text{up }1 \mid \text{up $1$ or down $1$}} = \ffrac{P_1\P{1-P_2}} {P_1\P{1-P_2}+\P{1-P_1}P_2}$$

Hence, the probability that the test will assert that $P_2 > P_1$ is equal to the probability that a gambler who wins each bet for one unit with probability $P$ will go down $M$ before going up $M$. Thus, let $i=M$ and $N=2M$ showing that this probability is given by

$$P\CB{\text{test asserts that $P_2>P_1$}} =1-\ffrac{1-\P{\ffrac{q} {p}}^M} {1-\P{\ffrac{q} {p}}^{2M}}=\ffrac{1} {1+\P{\ffrac{q} {p}}^{M}}$$

## Mean Time Spent in Transient States

Consider now a finite state Markov chain and suppose that the states are numbered so that $T ={1, 2,\dots ,t}$ denotes the set of **transient states**. Let

$$\mathbf{P}_{T} = \begin{bmatrix}
P_{11} & P_{12} & \cdots & P_{1t} \\
\vdots & \vdots & \vdots & \vdots \\
P_{t1}& P_{t2} & \cdots & P_{tt}
\end{bmatrix}$$

Notice that this matrix only indicate the transition probabilities from transient states to transient states and thus the row sum could be less than $1$. For transient states $i$ and $j$, let $s_{ij}$ denote the expected number of time periods that the Markov chain is in state $j$, given that it starts in state $i$. Then let $\delta_{i,j}=1$ when $i=j$ and $0$ otherwise. Condition on the initial transition to obtain

$$\begin{align}
s_{ij} &= \delta_{i,j} + \sum_{k} P_{ik} s_{kj}\\
&= \delta_{i,j} + \sum_{k=1}^{t} P_{ik} s_{kj}
\end{align}$$

where the final equality follows since it is impossible to go from a recurrent to a transient state, implying that $s_{kj}=0$ when $k$ is a recurrent state. And we also define the matrix $\mathbf S$:

$$\mathbf S = \begin{bmatrix}
s_{11} & s_{12} & \cdots & s_{1t} \\
\vdots & \vdots & \vdots & \vdots \\
s_{t1}& s_{t2} & \cdots & s_{tt}
\end{bmatrix} = \mathbf{I}_t + \mathbf{P}_T \mathbf S \Longrightarrow \mathbf S = \P{\mathbf{I}_t - \mathbf{P}_T}^{-1}$$

**e.g.30**

Consider the gambler’s ruin problem with $p = 0.4$ and $N = 7$. Starting with $3$ units, determine the expected amount of time the gambler has $5$ units

>First we can write the matrix for $\mathbf{P}_T$, which specifies $P_{ij}$, $i,j\in \CB{1,2,3,4,5,6}$.
>
>$$\mathbf{P}_T = \begin{bmatrix}
0 & 0.4 & 0 & 0 & 0 & 0\\
0.6 & 0 & 0.4 & 0 & 0 & 0 \\
0 & 0.6 & 0 & 0.4 & 0 & 0 \\
0 & 0 & 0.6 & 0 & 0.4 & 0 \\
0 & 0 & 0 & 0.6 & 0 & 0.4 \\
0 & 0 & 0 & 0 & 0.6 & 0\\
\end{bmatrix}$$
>
>And then we invert $\mathbf{I}_6 - \mathbf{P}_T$ to find $S$:

>$$\mathbf{S} = \P{\mathbf{I}_6 - \mathbf{P}_T}^{-1} = \begin{bmatrix}
1.6149 & 1.0248 & 0.6314 & 0.3691 & 0.1943 & 0.0777\\
1.5372 & 2.5619 & 1.5784 & 0.9228 & 0.4857 & 0.1943\\
1.4206 & 2.3677 & 2.9990 & 1.7533 & \mathbf{0.9228} & 0.3691\\
1.2458 & 2.0763 & 2.6299 & 2.9990 & 1.5784 & 0.6314\\
0.9835 & 1.6391 & 2.0763 & 2.3677 & 2.5619 & 1.0248\\
0.5901 & 0.9835 & 1.2458 & 1.4206 & 1.5372 & 1.6149
\end{bmatrix}$$
>
>Hence, $s_{3,5} = 0.9228$
***

And now we will derive the formula combining $s_{ij}$ and $f_{ij}$, the probability that the Markov chain ever makes a transition into state $j$ given that it starts in state $i$.

$$\begin{align}
s_{ij} &= \Exp\SB{\text{time in $j$} \mid \text{start in $i$, ever transit to $j$}}\cdot f_{ij} + \Exp\SB{\text{time in $j$} \mid \text{start in $i$, never transit to $j$}}\cdot \P{1-f_{ij}}\\
&= \P{\delta_{i,j} +s_{jj}}\cdot f_{ij} + \delta_{i,j}\cdot \P{1-f_{ij}}\\
&= \delta_{i,j} + f_{ij} s_{jj}
\end{align}$$

And with this we have $f_{ij} = \ffrac{s_{ij} - \delta_{i,j}} {s_{jj}}$.

**e.g.31** extended **e.g.30**

What is the probability that the gambler ever has a fortune of $1$?

>$$f_{3,1} = \ffrac{s_{3,1} - \delta_{3,1}} {s_{1,1}} = \ffrac{1.4206} {1.6149} = 0.8797$$
>***
>And we can also check this answer with the conclusion obtained in the last section.
>
>For this gambler, $f_{3,1}$ is just the probability that a gambler starting with $3$ reaches $1$ before $7$. And that the same with, by the fact that MC is **Time-homogenous**, the probability that a gambler starting with $2$ will go down to $0$ before reaching $6$.
>
>$$f_{3,1} = 1 - \ffrac{1-\P{\ffrac{0.6} {0.4}}^2}{1-\P{\ffrac{0.6} {0.4}}^6}=0.8797$$
***

And the expected time until the Markov chain enters some sets of states $A$, can be obtained by letting all probabilities of states of $A$, $P_{ij}$, to be $1$ if $i=j$ and $0$ otherwise, meaning, an absorbing state. This process changes all states of $A$ into recurrent states, and changes all states outside of $A$ from which an eventual transition into $A$ is possible, into a transient state.

## Branching Processes

Consider a population consisting of individuals able to produce offspring of the same kind. Suppose that each individual will, by the end of its lifetime, have identically produced $j$ new offspring with probability $P_j <1$ with $j\geq0$, independent of the numbers produced by other individuals. The number of individuals initially is $X_0$, called the ***size of the zeroth generation***. And also let $X_n$ represent the size of the $n$th generation. It follows that $\CB{X_n,n=0,1,\dots}$ is a Markov chain.

Note that state $0$ is a recurrent state since clearly $P_{00}=1$ since clearly no individual makes no further generations. Also if $P_0 = P\CB{X_1 = 0 \mid X_0=1}>0$, all other states are transient, and this follows since $P_{i0} = P_0^i$. Moreover, since any finite set of transient states $\CB{1,2,\dots,n}$ will be visited only finitely often if $P_0>0$, the population will either *die out* or its size will *converge to infinity*.

$Remark$

>Compare this with random walk, their difference lies in the "jump-free" property where random walk can only move to the neighbour states, either left or right, however in branching process, the length of step to the right (more generations) is unlimited.
>
>Also $X_0=1$ is always supposed to be true.

Let $\mu = \d{\sum_{j=0}^{\infty} j\cdot P_j}$ denote the mean number of offspring of a single individual, and the variance is

$$\sigma^2 = \sum_{j=0}^{\infty} \P{j-\mu}^2\cdot P_j$$

Now, let $Z_i^{\P{n-1}}$ denote the number of offspring of the $i$th individual of the $(n − 1)$st generation. And with this we define

$$X_n = \sum_{i=1}^{X_{n-1}}Z_i^{\P{n-1}}$$

By conditioning on $X_{n-1}$ we have

$$\begin{align}
\Exp\SB{X_n} &= \Exp\SB{\Exp\SB{X_n\mid X_{n-1}}}\\
&= \Exp\SB{\Exp\SB{\sum_{i=1}^{X_{n-1}}Z_i^{\P{n-1}} \mid X_{n-1}}}\\
&= \Exp\SB{\mu\cdot X_{n-1}}\\
&= \mu\Exp\SB{X_{n-1}}\\
&\text{iteration}\\
\Longrightarrow \Exp\SB{X_n} &= \mu^{n}\Exp\SB{X_0}
\end{align}$$

Then by the fact that $\Exp\SB{X_0} = 1$, $\Exp\SB{X_n} =\mu^{n}$. And similarly we have the variance formula

$$\begin{align}
\Var{X_n} &= \Exp\SB{\Var{X_n\mid X_{n-1}}} + \Var{\Exp\SB{X_n\mid X_{n-1}}}\\
&= \Exp\SB{X_{n-1}\sigma^2} + \Var{X_{n-1}\mu}\\
&= \sigma^2 \mu^{n-1} + \mu^2 \Var{X_{n-1}}\\
&\text{iteration}\\
\Longrightarrow \Var{X_n} &= \begin{cases}
n\sigma^2, &\text{if }\mu = 1\\
\sigma^2\mu^{n-1} \P{\ffrac{1-\mu^n} {1-\mu}}, &\text{if }\mu \neq 1
\end{cases}
\end{align}$$

Let $\pi_0$ denote the probability that the population will eventually die out (under the assumption that $X_0 = 1$). More formally,

$$\pi_0 = \lim_{n\to \infty} P\CB{X_n = 0 \mid X_0 = 1}$$

If $\mu = \d{\sum_{j=0}^{\infty} j\cdot P_j} <1$, we assert that $\pi_0 = 1$, because

$$\begin{align}
\mu^n &= \Exp\SB{X_n}\\
&= \sum_{j=0}^{\infty} j\cdot P_j \\
&\geq \sum_{j=0}^{\infty} 1\cdot P_j \\
&= P\CB{X_n \geq 1}
\end{align}$$

And obviously $\mu^n \to 1$ when $n$ goes to infinity, we have $P\CB{X_n = 0} = 1 - P\CB{X_n \geq 1} \to 1$. And more interestly, $\pi_0 = 1$ even when $\mu = 1$. Only when $\mu >1$ can we get a $\pi_0$ less than $1$. To derive this we condition the probability of dying out on the number of offspring of the initial individual and obtain

$$\begin{align}
\pi_0 &= P\CB{\text{population dies out}} \\
&= \sum_{j=0}^{\infty} P\CB{\text{population dies out} \mid X_1 = j}\cdot P_j \\
&\bspace\text{each family is assumed to act independently}\\
&= \sum_{j=0}^{\infty} \pi_0^j \cdot P_j
\end{align}$$

And actually the smallest positive number satisfying this equation is $\pi_0$ given $\mu >1$.

**e.g.32**

$P_0 = 0.5$, $P_1 = 0.25$, $P_2 = 0.25$. Find $\pi_0$

>$$\mu = 0\times0.5 + 1 \times 0.25 + 2 \times 0.25 = 0.75 < 1$$
>
>Thus $\pi_0 = 1$

***

**e.g.33**

$P_0 = 0.25$, $P_1 = 0.25$, $P_2 = 0.5$. Find $\pi_0$

>We can write the equation $\pi_0 = 0.25 \times \pi_0^0 + 0.25 \times \pi_0^1 + 0.5 \times \pi_0^2$ and solve this we have $\pi_0 = 0.5$.

***

**e.g.34**

What is the probability that the population will die out if it initially consists of $n$ individuals?

>$\pi_0^n$, since the population will die out if and only if the families of each of the members of the initial generation die out.

***

$Remark$

>$\mu > 1$ is called the super critical; $\mu = 1$ is called the critical and $\mu<1$ is called the sub critial.
>
>Also, you will encounter some complex equation like $\pi_0 = \P{1-p}^2 + 2p\P{1-p}\pi_0 + p^2\pi_0$. Don't be afraid. First notice that $\pi_0 = 1$, then use ***Viere Theorem*** that $x_1x_2 = c/a$.

$Theorem$

Suppose that $p_0>0$ and $p_0 + p_1<1$, then $\pi_0$ is the **smallest positive** number satisfying

$$\pi_0 = \sum_{j=0}^{\infty} \pi_0^j p_j$$

And it is $\pi_0 = 1\iff \mu \leq 1$

$Proof$

>Let $\pi$ satisfy the equation. By induction we can show $\pi \geq P\CB{X_n = 0}$ for all $n$. Now
>
>$$\pi =\sum_{j=0}^{\infty} \pi^j p_j \geq \pi^0 p_0 = P\CB{X_1 = 0} $$
>
>Assuming $\pi \geq P\CB{X_n = 0}$, we have
>
>$$\begin{align}
P\CB{X_{n+1} =0} &= \sum_j P\CB{X_{n+1} = 0 \mid X_1 = j} \cdot p_j \\
&= \sum_j \P{ P\CB{X_n = 0} }^j \cdot p_j \\
&\leq \sum_{j=0}^{\infty} \pi^j p_j = \pi
\end{align}$$
>
>Letting $n\to\infty$ we have $\pi\geq \d{\lim_{n\to\infty} P\CB{X_n = 0}} = \pi_0$
>
>When $\mu\leq 1$, we first define the generating function $\phi\P{s} = \sum_{j=0}^{\infty} s^j \cdot p_j$. Since $p_0 + p_1 <1$, we have, for all $s \in \P{0,1}$
>
>$$\phi'\P{s} = \sum_{j=0}^{\infty} j \cdot s^{j-1} p_j > 0\\
\phi''\P{s} = \sum_{j=0}^{\infty} j\P{j-1} \cdot s^{j-2} p_j > 0$$
>
>Notice that $\phi\P{\pi_0} = \pi_0$, and $\phi'\P{1} = \mu$. Consider the intercept between $\phi\P{s}$ and $s$ from $\P{0,1}$. So that in order to intercept, it must satisfy that $\mu = \phi'\P{1} \leq 1$.

$Remark$

>Here $P\CB{X_{n+1} = 0 \mid X_1 = j} = \P{ P\CB{X_n = 0} }^j$ is easy to understand, one thing to notice is that if we define $\tau = \inf\CB{n,X_n = 0}$, then $P\CB{\tau = k \mid X_1 = j} \neq \P{ P\CB{\tau = k \mid X_1 = j} }^j$. However,
>
>$$P\CB{\tau \leq k \mid X_1 = j} = \P{ P\CB{\tau \leq k \mid X_1 = j} }^j$$
***

$Remark$

>Suppose that the population becomes extinct for the first time in $\tau$-th generation. Then given that $X_0=1$ we have $P\CB{\tau = n} = P\CB{\tau\leq n} - P\CB{\tau \leq n-1}$

## Time Reversible Markov Chains

Consider a **stationary ergodic** Markov chain having transition probabilities $P_{ij}$ and stationary probabilities $\pi_i$, and suppose that starting at some time we trace the sequence of states going *backward* in time. Or to say:

Starting at time $n$, consider the sequence of states $X_n, X_{n-1},X_{n-2},\dots$. It turns out that this sequence of states is itself a *Markov chain* with transition probabilities $Q_{ij}$ defined by

$$\begin{align}
Q_{ij} &= P\CB{X_m = j\mid X_{m+1} = i}\\
&= \ffrac{P\CB{X_m = j, X_{m+1} = i}} {P\CB{X_{m+1} = i}} \\
&= \ffrac{P\CB{X_m = j}\cdot P\CB{X_{m+1} = i \mid X_m = j}}{P\CB{X_{m+1} = i}}\\
&= \ffrac{\pi_j P_{ji}} {\pi_i}
\end{align}$$

We need to verify that $P\CB{X_m = j \mid X_{m+1} = i,X_{m+2},X_{m+3},\dots} = P\CB{X_m = j \mid X_{m+1} = i}$. To see this we suppose currently at time $m+1$. Since $X_0,X_1,\dots$ is a Markov Chain, it follows that the conditional distribution of the future $X_{m+2},X_{m+3},\dots$ given the present state $X_{m+1}$ is independent of the past state $X_m$. HOWEVER, *independence* is a **symmetric** relationship, meaning that given $X_{m+1}$, $X_m$ is independent of $X_{m+2},X_{m+3},\dots$. Done. And thus,

$$Q_{ij} = \ffrac{\pi_j P_{ji}} {\pi_i} $$

Besides, if $Q_{ij} = P_{ij} \iff \pi_i P_{ij} = \pi_j P_{ji}$ for all $i,j$, then the Markov chain is said to be ***time reversible***. And this can also be stated that, for all states $i$ and $j$, the rate at which the process goes from $i$ to $j$, $\pi_i P_{ij}$, is equal to the rate at which it goes from $j$ to $i$, $\pi_j P_{ji}$.

And an obvious conclusion is that,

- the rate at which the *forward* process makes a transition from $j$ to $i$
- the rate at which the *reverse* process makes a transition from $i$ to $j$
- if **time reversible**, then the *forward* process makes a transition from $i$ to $j$

Or, we can find the solution of:

$$\begin{cases}
x_iP_{ij} = x_jP_{ji},&\text{for all }i,j \\
\sum\limits_i x_i = 1
\end{cases}$$

Also if we summing over $j$ for the first one, it leads to

$$\begin{align}
\sum\limits_i x_iP_{ij} &= \sum\limits_i x_jP_{ji}\\
&= x_j\sum_i P_{ji} = x_j
\end{align}$$

Then it's obvious that $x_i = \pi_i$ for all $i$, is the *unique solution* of the preceding, and it's just the stationary probabilities, or the limiting probabilities.

$Remark$

>***weak symmetric (可配称)***: $x_i P_{ij} = x_jP_{ji}$, $\forall \;i,j$. And moreover, if $\sum\nolimits_i x_i < \infty$, meaning that the Markov Chain is **可合的** then it's **time-reversible**
>
>**Time-reversible**: either **ergodic**, or ***weak symmetric***.

**e.g.35**

Consider a random walk with states $0, 1,\dots, M$ and transition probabilities

$$\begin{cases}
P_{i,i+1} = \alpha_i = 1 - P_{i,i-1}, & i = 1,\dots,M-1\\
P_{0,1} = \alpha_0 = 1-P_{0,0}\\
P_{M,M} = \alpha_M = 1-P_{M,M-1}
\end{cases}$$

This is surely a Markov Chain, but it's also **time reversible**. Since any two transitions from $i$ to $i+1$ there must be one from $i+1$ to $i$, and true conversely. How can you move from state $i$ to $j$ *twice* without coming back?

Hence, it follows that the rate of transitions from $i$ to $i + 1$ equals the rate from $i + 1$ to $i$, and so the process is time reversible. Then the limiting probabiliies, by equating for each state, $0,1,\dots,M-1$, the rate at which the process goes from $i$ to $i + 1$ with the rate at which it goes from $i + 1$ to $i$.

$$\left\{\begin{align}
\pi_0\alpha_0 &= \pi_1\P{1- \alpha_1}\\
\pi_1\alpha_1 &= \pi_2\P{1- \alpha_2}\\
&\;\vdots\\
\pi_i\alpha_i &= \pi_{i+1}\P{1- \alpha_{i+1}}
\end{align}\right. \Longrightarrow \pi_i = \ffrac{\alpha_{i-1} \cdots \alpha_0}{\P{1-\alpha_i}\cdots\P{1-\alpha_1}}\pi_0, i = 1,2,\dots,M$$

Then since $\sum_{0}^{M} \pi_i = 1$, we obtain

$$\pi_0 = \SB{1+\sum_{j=1}^{M} \ffrac{\alpha_{j-1} \cdots \alpha_0}{\P{1-\alpha_j}\cdots\P{1-\alpha_1}}}^{-1}$$

And, in some special case, $\alpha \equiv \alpha$, then $\pi_0 = \ffrac{1-\beta}{1-\beta^{M+1}}$ where $\beta = \ffrac{\alpha}{1-\alpha}$. And $\pi_i = \ffrac{\beta^i \P{1-\beta}} {1-\beta^{M+1}}$
***

Another special case is for the urn model where $\alpha_i = \ffrac{M-i}{M}$ for $i = 0,1,\dots,M$. Hence

$$\pi_0 = \SB{1+\sum_{j=1}^{M}\ffrac{\P{M-j+1}\cdots\P{M-1}M}{j\P{j-1}\cdots 1}}^{-1} = \ffrac{1}{\d{\sum_{j=0}^{M}\binom{M}{j}}} = \P{\ffrac{1}{2}}^M\\
\pi_i = \binom{M}{i}\P{\ffrac{1}{2}}^M$$

And this result is quite intuitive that in the long run, the positions of each of the $M$ balls are independent and each one is equally likely to be in either urn.
***

Now we look back to the equation $x_i P_{ij} = x_j P_{ji}$. Could it turns out that no solution exists?

$$\begin{cases}
x_i P_{ij} = x_j P_{ji}\\
x_k P_{kj} = x_j P_{jk}\\
x_i P_{ik} = x_k P_{ki}
\end{cases}\Longrightarrow \ffrac{x_i}{x_k} = \ffrac{P_{ji}P_{kj}}{P_{ij}P_{jk}} = \ffrac{P_{ki}} {P_{ik}}, \bspace \text{if }P_{ij}P_{jk}>0$$

Thus a necessary condition for time reversibility is that $P_{ik}P_{kj}P_{ji} = P_{ij}P_{jk}P_{ki}$, for all $i,j,k$. And now we summrize this into a Theorem

$Theorem.2$

An ergodic Markov chain for which $P_{i j} = 0$ whenever $P_{ji} = 0$ is **time reversible** $iff$ starting in state $i$, any path back to $i$ has the same probability as the reversed path. That is, if

$$P_{i,i_1}P_{i_1,i_2} \cdots P_{i_k,i} = P_{i,i_k}P_{i_k,i_{{k-1}}} \cdots P_{i_1,i} $$

for all states $i,i_1,\dots,i_k$.

$Proof$

>We have already proven necessity. To prove sufficiency, fix states $i$ and $j$ in the series of states, and rewrite
>
>$$P_{i,i_1}P_{i_1,i_2} \cdots P_{i_k,j}P_{j,i} = P_{i,j}P_{j,i_k}P_{i_k,i_{{k-1}}} \cdots P_{i_1,i}$$
>
>Then sum both sides over all states $i_1,i_2,\dots,i_k$ yielding that in $k+1$ transitions,
>
>$$P_{ij}^{k+1} P_{ji} = P_{ij}P_{ji}^{k+1}$$
>
>Then letting $k\to\infty$ yields $\pi_j P_{ji} = P_{ij}\pi_i$. Time reversible!
***

**e..g.37**

Good example though, skipped.
***
$Proposition.9$

Consider an irreducible Markov chain with transition probabilities $P_{ij}$. If we can find positive numbers $\pi_i,i\geq 0$, summing to one, and a transition $\mathbf{Q} = \SB{Q_{ij}}$ such that

$\bspace\pi_iP_{ij} = \pi_j Q_{ji}$

then the $Q_{ij}$ are the transition probabilities of the reversed chain and the $\pi_i$ are the stationary probabilities both for the original and reversed chain.

**e.g.38**

When the light bulb in use fails, it is replaced by a new one at the beginning of the next day. Let $X_n$ equal $i$ if the bulb in use at the beginning of day $n$ is in its $i$th day of use (that is, if its present age is
$i$). For instance, if a bulb fails on day $n − 1$, then a new bulb will be put in use at the beginning of day $n$ and so $X_n = 1$. If we suppose that each bulb, independently, fails on its $i$th day of use with probability $p_i$, $i \geq 1$, then it is easy to see that ${ X_n , n \geq 1}$ is a Markov chain. Let $L$ be the $r.v.$ representing the bulb's life so that $P\CB{L=i} = p_i$, then the transition probabilities are as follows:

$$\begin{align}
P_{i,1} &= P\CB{\text{bulb which is on its $i$th day of use fails}}\\
&= P\CB{\text{life of bulb $= i$}\mid\text{life of bulb $\geq i$}}\\
&= \ffrac{P\CB{L = i}}{P\CB{L\geq i}}\\
P_{i,i+1}&=1-P_{i,1}
\end{align}$$

Suppose now that this chain has been in operation for a long (in theory, an infinite) time and consider the sequence of states going backward in time. The reverse chain will always decrease by $1$ until it reaches $1$
and then it will jump to a random value representing the lifetime of the (in real time) previous bulb. Then the transition probabilities for the reverse chain:

$\bspace\begin{align}
Q_{i,i-1} &= 1,&i>1\\
Q_{i,1} &=p_i,&i\geq1
\end{align}$

To check this and meantime, find the stationary probabilities, let's see whether positive $\pi_i$ exist, such that

$\bspace \pi_i P_{i,j} = \pi_j Q_{j,i}$

First let $j=1$ and there we have $\pi_i\ffrac{P\CB{L=i}}{P\CB{L\geq i}} = \pi_1 P\CB{L=i}$ or equivalently $\pi_i = \pi_1 P\CB{L \geq i}$. Summing over all $i$ yields

$\bspace 1=\d{\sum_{i=1}^{\infty} \pi_i = \pi_1 \sum_{i=1}^{\infty} P\CB{L\geq i}} = \pi_1\Exp\SB{L} \Longrightarrow \pi_i = \ffrac{P\CB{L\geq i}}{\Exp\SB{L}}, \bspace i\geq 1$

After this, we need to check $\pi_i P_{i,i+1} = \pi_{i+1} Q_{i+1,i}$, which is equivalent to

$\bspace\ffrac{P\CB{L\geq i}}{\Exp\SB{L}} \P{1-\ffrac{P\CB{L = i}}{P\CB{L\geq i}}} = \ffrac{P\CB{L \geq i+1}}{\Exp\SB{L}}\cdot1$

And this equation holds since $P\CB{L \geq i} - P\CB{L = i} = P\CB{L\geq i+1}$. Done!

***

Probability and Statistics/Applied Random Process/Chap_04.ipynb