```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from scipy import misc
```

### The Chain at a Fixed Time ###
Let $X_0, X_1, X_2, \ldots $ be a Markov Chain with state space $S$. We will start by setting up notation that will help us express our calculations compactly.

For $n \ge 0$, let $P_n$ be the distribution of $X_n$. That is,

$$
P_n(i) = P(X_n = i), ~~~~ i \in S
$$

Then the distribution of $X_0$ is $P_0$. This is called the *initial distribution* of the chain.

For $n \ge 0$ and $j \in S$,

\begin{align*}
P_{n+1}(j) &= P(X_{n+1} = j) \\
&= \sum_{i \in S} P(X_n = i, X_{n+1} = j) \\
&= \sum_{i \in S} P(X_n = i)P(X_{n+1} = j \mid X_n = i) \\
&= \sum_{i \in S} P_n(i)P(X_{n+1} = j \mid X_n = i)
\end{align*}

The conditional probability $P(X_{n+1} = j \mid X_n = i)$ is called a *one-step transition probability at time $n$*.

For many chains such as the random walk, these one-step transition probabilities depend only on the states $i$ and $j$, not on the time $n$. For example, for the random walk,

\begin{equation}
P(X_{n+1} = j \mid X_n = i) =
 \begin{cases}
      \frac{1}{2} & \text{if } j = i-1 \text{ or } j = i+1 \\
      0 & \text{ otherwise}
   \end{cases}
\end{equation}

for every $n$. When one-step transition probabilites don't depend on $n$, they are called *stationary* or *time-homogenous*. All the Markov Chains that we will study in this course have time-homogenous transition probabilities.

For such a chain, define the *one-step transition probability*

$$
P(i, j) = P(X_{n+1} = j \mid X_n = i)
$$

### The Probability of a Path ###
Given that the chain starts at $i$, what is the chance that the next three values are of the chain are $j, k$, and $l$, in that order?

We are looking for
$$
P(X_1 = j, X_2 = k, X_3 = l \mid X_0 = i)
$$

By repeated use of the multiplication rule and the Markov property, this is

$$
P(X_1 = j, X_2 = k, X_3 = l \mid X_0 = i) = P(i, j)P(j, k)P(k, l)
$$

In the same way, given that you know the starting point, you can find the probability of any path of finite length by multiplying one-step transition probabilities.

### The Distribution of $X_{n+1}$ ###
By our calculation at the start of this section,

\begin{align*}
P_{n+1}(j) &= P(X_{n+1} = j) \\
&= \sum_{i \in S} P_n(i)P(X_{n+1} = j \mid X_n = i) \\
&= \sum_{i \in S} P_n(i)P(i, j)
\end{align*}

The calculation is based on the straightforward observation that for the chain to be at state $j$ at time $n+1$, it had to be at some state $i$ at time $n$ and then get from $i$ to $j$ in one step.

Let's use all this in examples. You will quickly see that the distribution $P_n$ has interesting properties.

### Lazy Random Walk on a Circle ###
Let the state space be five points arranged on a circle. Suppose the process starts at Point 1, and at each step either stays in place with probability 0.5 (and thus is lazy), or moves to one of the two neighboring points with chance 0.25 each, regardless of the other moves.

This transition behavior can be summed up in a *transition diagram*:

At every step, the next move is determined by a random choice from among three options and by the chain's current location, not on how it got to that location. So the process is a Markov chain. Let's call it $X_0, X_1, X_2, \ldots $.

By our assumption, the initial distribution $P_0$ puts all the probability on Point 1. It is defined in the cell below. We will be using `prob140` Markov Chain methods based on [Pykov](https://github.com/riccardoscalco/Pykov) written by [Riccardo Scalco](http://riccardoscalco.github.io). Note the use of `states` instead of `values`. Please enter the states in ascending order, for technical reasons that we hope to overcome later in the term.

```python
s = np.arange(1, 6)
p = [1, 0, 0, 0, 0]
initial = Table().states(s).probability(p)
initial
```

            State	 Probability

            1    	 1

            2    	 0

            3    	 0

            4    	 0

            5    	 0

The transition probabilities are:
- For $2 \le i \le 4$, $P(i, i) = 0.5$ and $P(i, i-1) = 0.25 = P(i, i+1)$.
- $P(1, 1) = 0.5$ and $P(1, 5) = 0.25 = P(1, 2)$.
- $P(5, 5) = 0.5$ and $P(5, 4) = 0.25 = P(5, 1)$.

These probabilities are returned by the function `circle_walk_probs` that takes states $i$ and $j$ as its arguments.

```python
def circle_walk_probs(i, j):
        if i-j == 0:
            return 0.5
        elif abs(i-j) == 1:
            return 0.25
        elif abs(i-j) == 4:
            return 0.25
        else:
            return 0
```

All the transition probabilities can be captured in a table, in a process analogous to creating a joint distribution table.

```python
trans_tbl = Table().states(s).transition_function(circle_walk_probs)
```

```python
trans_tbl
```

            Source	 Target	 P(Target | Source)

            1     	 1     	 0.5

            1     	 2     	 0.25

            1     	 3     	 0

            1     	 4     	 0

            1     	 5     	 0.25

            2     	 1     	 0.25

            2     	 2     	 0.5

            2     	 3     	 0.25

            2     	 4     	 0

            2     	 5     	 0

... (15 rows omitted)</p

Just as when we were constructing joint distribution tables, we can better visualize this as a $5 \times 5$ table:

```python
circle_walk = trans_tbl.toMarkovChain()
circle_walk
```

      1
      2
      3
      4
      5

      1
      0.50
      0.25
      0.00
      0.00
      0.25

      2
      0.25
      0.50
      0.25
      0.00
      0.00

      3
      0.00
      0.25
      0.50
      0.25
      0.00

      4
      0.00
      0.00
      0.25
      0.50
      0.25

      5
      0.25
      0.00
      0.00
      0.25
      0.50

This is called the *transition matrix* of the chain.
- For each $i$ and $j$, the $(i, j)$ element of the transition matrix is the one-step transition probability $P(i, j)$.
- For each $i$, the $i$th row of the transition matrix consists of the conditional distribution of $X_{n+1}$ given $X_n = i$.

#### Probability of a Path ####
What's the probability of the path 1, 1, 2, 1, 2? That's the path $X_0 = 1, X_1 = 1, X_2 = 2, X_3 = 1, X_4 = 2$. We know that the chain is starting at 1, so the chance of the path is

$$
1 \cdot P(1, 1)P(1, 2)P(2, 1)P(1, 2) = 0.5 \times 0.25 \times 0.25 \times 0.25 = 0.0078125
$$

The method `prob_of_path` takes the initial distribution and path as its arguments, and returns the probability of the path:

```python
circle_walk.prob_of_path(initial, [1, 1, 2, 1, 2])
```

    0.0078125000000000035

#### Distribution of $X_n$ ####
Remember that the chain starts at 1. So $P_0$, the distribution of $X_0$ is:

```python
initial
```

            State	 Probability

            1    	 1

            2    	 0

            3    	 0

            4    	 0

            5    	 0

We know that $P_1$ must place probability 0.5 at Point 1 and 0.25 each the points 2 and 5. This is confirmed by the `distribution` method that applies to a MarkovChain object. Its first argument is the initial distribution, and its second is the number of steps $n$. It returns a distribution object that is the distribution of $X_n$.

```python
P_1 = circle_walk.distribution(initial, 1)
P_1
```

            State	 Probability

            1    	 0.5

            2    	 0.25

            3    	 0

            4    	 0

            5    	 0.25

What's the probability that the chain is has value 3 at time 2? That's $P_2(3)$ which we can calculate by conditioning on $X_1$:

$$
P_2(3) = \sum_{i=1}^5 P_1(i)P(i, 3)
$$

The distribution of $X_1$ is $P_1$, given above. Here are those probabilities in an array:

```python
P_1.column('Probability')
```

    array([ 0.5 ,  0.25,  0.  ,  0.  ,  0.25])

The `3` column of the transition matrix gives us, for each $i$, the chance of getting from $i$ to 3 in one step.

```python
circle_walk.column('3')
```

    array([ 0.  ,  0.25,  0.5 ,  0.25,  0.  ])

So the probability that the chain has the value 3 at time 2 is $P_2(3)$ which is equal to:

```python
sum(P_1.column('Probability')*circle_walk.column('3'))
```

    0.0625

Similarly, $P_2(2)$ is equal to:

```python
sum(P_1.column('Probability')*circle_walk.column('2'))
```

    0.25

And so on. The `distribution` method finds all these probabilities for us.

```python
P_2 = circle_walk.distribution(initial, 2)
P_2
```

            State	 Probability

            1    	 0.375

            2    	 0.25

            3    	 0.0625

            4    	 0.0625

            5    	 0.25

At time 3, the chain continues to be much more likely to be at 1, 2, or 5 compared to the other two states. That's because it started at Point 1 and is lazy.

```python
P_3 = circle_walk.distribution(initial, 3)
P_3
```

            State	 Probability

            1    	 0.3125

            2    	 0.234375

            3    	 0.109375

            4    	 0.109375

            5    	 0.234375

But by time 10, something interesting starts to emerge.

```python
P_10 = circle_walk.distribution(initial, 10)
P_10
```

            State	 Probability

            1    	 0.20577

            2    	 0.201783

            3    	 0.195332

            4    	 0.195332

            5    	 0.201783

The chain is almost equally likely to be at any of the five states. By time 50, it seems to have completely forgotten where it started, and is distributed uniformly on the state space.

```python
P_50 = circle_walk.distribution(initial, 50)
P_50
```

            State	 Probability

            1    	 0.2

            2    	 0.2

            3    	 0.2

            4    	 0.2

            5    	 0.2

As time passes, this chain gets "all mixed up", regardless of where it started. That is perhaps not surprising as the transition probabilities are symmetric over the five states. Let's see what happens when we cut the circle between Points 1 and 5 and lay it out in a line.

### Reflecting Random Walk ###
The state space and transition probabilities remain the same, except when the chain is at the two "edge" states.
- If the chain is at Point 1, then at the next step it either stays there or moves to Point 2 with equal probability: $P(1, 1) = 0.5 = P(1, 2)$.
- If the chain is at Point 5, then at the next step it either stays there or moves to Point 4 with equal probability: $P(5, 5) = 0.5 = P(5, 4)$.

We say that there is *reflection* at the boundaries 1 and 5.

```python
def ref_walk_probs(i, j):
    if i-j == 0:
        return 0.5
    elif 2 <= i <= 4:
        if abs(i-j) == 1:
            return 0.25
        else:
            return 0
    elif i == 1:
        if j == 2:
            return 0.5
        else:
            return 0
    elif i == 5:
        if j == 4:
            return 0.5
        else:
            return 0
```

```python
trans_tbl = Table().states(s).transition_function(ref_walk_probs)
refl_walk = trans_tbl.toMarkovChain()
print('Transition Matrix')
refl_walk
```

    Transition Matrix

      1
      2
      3
      4
      5

      1
      0.50
      0.50
      0.00
      0.00
      0.00

      2
      0.25
      0.50
      0.25
      0.00
      0.00

      3
      0.00
      0.25
      0.50
      0.25
      0.00

      4
      0.00
      0.00
      0.25
      0.50
      0.25

      5
      0.00
      0.00
      0.00
      0.50
      0.50

Let the chain start at Point 1 as it did in the last example. That initial distribution was defined as `initial`. At time 1, therefore, the chain is either at 1 or 2, and at times 2 and 3 it is likely to still be around 1.

```python
refl_walk.distribution(initial, 1)
```

            State	 Probability

            1    	 0.5

            2    	 0.5

            3    	 0

            4    	 0

            5    	 0

```python
refl_walk.distribution(initial, 3)
```

            State	 Probability

            1    	 0.3125

            2    	 0.46875

            3    	 0.1875

            4    	 0.03125

            5    	 0

But by time 20, the distribution is settling down:

```python
refl_walk.distribution(initial, 20)
```

            State	 Probability

            1    	 0.135533

            2    	 0.264896

            3    	 0.25

            4    	 0.235104

            5    	 0.114467

And by time 100 it has settled into what is called its *steady state*.

```python
refl_walk.distribution(initial, 100)
```

            State	 Probability

            1    	 0.125

            2    	 0.25

            3    	 0.25

            4    	 0.25

            5    	 0.125

This steady state distribution isn't uniform. But it is steady. If you increase the amount of time for which the chain has run, you get the same distribution for the value of the chain at that time.

That's quite remarkable. In the rest of this chapter, we will look more closely at what's going on.

```python

```


```python
%matplotlib inline
```

```python
%run proof_setup
```

First we derive the relevant derivatives of the true anomaly using the fact that

$$
M = E - e\,\sin E
$$

implies

$$
\mathrm{d}M = -\sin E\,\mathrm{d}e + (1 - e\,\cos E)\,\mathrm{d}E
$$

$$
\frac{\mathrm{d}E}{\mathrm{d}M} = \frac{1}{1 - e\,\cos E} \quad \mathrm{and} \quad
\frac{\mathrm{d}E}{\mathrm{d}e} = \frac{\sin E}{1 - e\,\cos E}
$$

```python
import sympy as sm

E, e, f = sm.symbols("E, e, f", real=True)

f0 = 2 * sm.atan(sm.sqrt(1 + e) / sm.sqrt(1 - e) * sm.tan(E / 2))
E0 = 2 * sm.atan(sm.sqrt(1 - e) / sm.sqrt(1 + e) * sm.tan(f / 2))

dEde = sm.sin(E) / (1 - e * sm.cos(E))
dEdM = 1 / (1 - e * sm.cos(E))

dfdE = sm.simplify(sm.diff(f0, E))
print("dfdE(E) =", dfdE)
print("dfdE(f) =", sm.simplify(dfdE.subs([(E, E0)])))

print()

dfdM = sm.simplify(sm.simplify(dfdE * dEdM).expand(trig=True))
print("dfdM(E) =", dfdM)
print("dfdM(f) =", sm.simplify(sm.simplify(dfdM.subs([(E, E0)])).expand(trig=True)))

print()

dfde = sm.simplify(dfdE * dEde + sm.diff(f0, e))
print("dfde(E) =", dfde)
print("dfde(f) =", sm.simplify(sm.simplify(dfde.subs([(E, E0)])).expand(trig=True)))

print()

print("dsinfdf =", sm.diff(sm.sin(f), f))
print("dcosfdf =", sm.diff(sm.cos(f), f))

print("\nduh.")
```

    dfdE(E) = sqrt(-e + 1)*sqrt(e + 1)/(-2*e*cos(E/2)**2 + e + 1)
    dfdE(f) = (2*e*cos(f/2)**2 - e + 1)/(sqrt(-e + 1)*sqrt(e + 1))

    dfdM(E) = sqrt(-e + 1)*sqrt(e + 1)/(e*cos(E) - 1)**2
    dfdM(f) = (-2*e*cos(f/2)**2 + e - 1)**2/((-e + 1)**(3/2)*(e + 1)**(3/2))

    dfde(E) = (-(-e + 1)**2*(e + 1)*(e - (e + 1)*tan(E/2)**2 - 1)*sin(E) + 2*(e - 1)*(e*cos(E) - 1)*(-2*e*cos(E/2)**2 + e + 1)*tan(E/2))/((-e + 1)**(3/2)*sqrt(e + 1)*(e*cos(E) - 1)*(e - (e + 1)*tan(E/2)**2 - 1)*(-2*e*cos(E/2)**2 + e + 1))
    dfde(f) = -(e*cos(f) + 2)*sin(f)/(e**2 - 1)

    dsinfdf = cos(f)
    dcosfdf = -sin(f)

    duh.

Now let's compare that to the numerical derivatives

```python
import theano
import theano.tensor as tt
import numpy as np
import exoplanet as xo
import matplotlib.pyplot as plt

def compare_grad(M_grad=True):
    M = tt.dvector()
    e = tt.dvector()

    op = xo.theano_ops.kepler.KeplerOp()
    sinf, cosf = op(M, e)

    if M_grad:
        M_val = np.linspace(-np.pi, np.pi, 1001)
        e_val = 0.8 + np.zeros_like(M_val)
        x = M_val
        x_obj = M
        label = "M"

    else:
        e_val = np.linspace(0, 0.99, 1000)
        M_val = 0.8 + np.zeros_like(e_val)
        x = e_val
        x_obj = e
        label = "e"

    res = theano.function([M, e], [sinf, cosf])(M_val, e_val)
    dsinf = theano.function([M, e], theano.grad(tt.sum(sinf), [x_obj])[0])(M_val, e_val)
    dcosf = theano.function([M, e], theano.grad(tt.sum(cosf), [x_obj])[0])(M_val, e_val)

    plt.figure()
    plt.plot(x, res[0], label="sin(f)")
    plt.plot(x, res[1], label="cos(f)")
    plt.xlabel(label)
    plt.legend()

    num_x = 0.5 * (x[:-1] + x[1:])
    num_y = [np.diff(res[0]) / np.diff(x), np.diff(res[1]) / np.diff(x)]

    plt.figure()
    plt.plot(num_x, num_y[0], label="numerical")
    plt.plot(x, dsinf, ":", label="analytic")
    plt.xlabel(label)
    plt.ylabel("d sin(f) / d {0}".format(label))
    plt.legend()

    plt.figure()
    plt.plot(num_x, num_y[1], label="numerical")
    plt.plot(x, dcosf, ":", label="analytic")
    plt.xlabel(label)
    plt.ylabel("d cos(f) / d {0}".format(label))
    plt.legend()
```

First, the derivatives of $\sin f$ and $\cos f$ with respect to $M$:

```python
compare_grad()
```

Then, with respect to $e$:

```python
compare_grad(False)
```

```python

```