# Reinforcement Learning as Probabilistic Inference

**Author**: Lisa Lee (lslee@cs.cmu.edu)

**Acknowledgements**: Thanks to Maruan Al-Shedivat, Ben Eysenbach, and Emilio Parisotto for providing helpful feedback.

## Introduction

This Colab notebook is based on Sergey Levine's tutorial ([Levine 2018](https://arxiv.org/abs/1805.00909)) that formalizes reinforcement learning (RL) as probabilistic inference.

We apply this framework to a simple chain environment. We implement the corresponding graphical model for our task, then implement a standard sum-product inference algorithm to infer an optimal policy. We compare the learned soft Q-function $Q(s_t,a_t) = \log \beta_t(s_t,a_t)$ and policy $p(a_t \mid s_t, O_{t:T}) = \frac{\beta_t(s_t, a_t)}{\beta_t(s_t)}$ with that of a classic model-free RL algorithm, by evaluating the policies in the environment.

## I. Markov Decision Process

### 1.1: Definition

Consider a simple chain environment consisting of $N$ states:

```
0 <-> 1 <-> ... <-> N-1
```

Suppose the agent always starts at state 0, and its task is to reach the goal state $N-1$. We formalize this task as a **Markov Decision Process** (MDP) consisting of a tuple $(\mathcal{S}, \mathcal{A}, P_0, \mathcal{T}, r)$, where:

* $\mathcal{S} = \{0,\ldots, N-1\}$ are the agent's possible **states**, indicating the agent's current location.
* $\mathcal{A} = \{\diamond, \leftarrow, \rightarrow\}$ are the agent's possible **actions**. The directional actions $\leftarrow, \rightarrow$ correspond to moving left or right one state, respectively. The action $\diamond$ corresponds to 'stay', i.e., staying in the same state.
* $P_0: \mathcal{S} \rightarrow [0, 1]$ is the agent's **initial state distribution**, where we assume the agent always starts in state 0:
$$
P_0(s) = \begin{cases}
1 & \text{if }s = 0\\
0 & \text{otherwise}
\end{cases}
$$
* $\mathcal{T}(s_{t+1} \mid s_t, a_t)$  is the **transition dynamics** probability that action $a_t$ in state $s_t$ will lead to the next state $s_{t+1}$. Assume there is a "action fail" probability $\epsilon \in [0, 1]$ such that, at each time step $t$, the agent stays in the same cell regardless of the action it takes. Note that the action $\diamond$ has deterministic transition dynamics: $\mathcal{T}(s' \mid s, \diamond)=\mathbb{1}(s' = s)$ for all $s, s' \in \mathcal{S}$.

* $r(s_t,a_t) : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is a **reward** function that provides a supervision signal for taking action $a_t$ in state $s_t$, where
$$
r(s_t,a_t) = \begin{cases}
0 & \text{if }s_t = N-1 \\
0 & \text{if }s_t = N-2\text{ and }a_t = \rightarrow \\
-1 & \text{otherwise}
\end{cases}
$$
In other words, the agent receives 0 reward for taking an action such that its next state $s_{t+1}$ might be the goal, and -1 otherwise.

Assume a finite-horizon MDP, and let $T$ be the episodic time horizon. Each episode in the MDP proceeds as follows:

1. The agent spawns in an initial state according to $s_1 \sim P_0(s)$.
2. At each time step $t$, the agent observes its current cell state $s_t \in \mathcal{S}$, and takes an action $a_t \sim \pi(a \mid s_t)$. The next state is determined by the transition dynamics $s_{t+1} \sim \mathcal{T}(s' \mid s_t, a_t)$. The agent also receives a reward $r_t = r(s_t, a_t)$.
4. The episode terminates once the agent reaches the goal, or if $t \geq T$. (If the episode terminates at time step $t < T$, you can assume that $r_{t'} = 0$, $s_{t'} = s_t$, and $a_{t'} = \diamond$ for $t' \in \{t+1, \ldots, T\}$.)

Here, $\pi(a \mid s):\mathcal{S} \times \mathcal{A} \rightarrow [0, 1]$ is the agent's **policy** which defines a probability distribution over actions $a \in \mathcal{A}$ given the current state $s$. A standard RL policy search problem aims to find an **optimal** policy that maximizes the expected cumulative reward:

$$
\pi^* = \underset{\pi}{\arg\max}  \mathbb{E}_{ \substack{
    s_1 \sim p_0(S) \\
    a_t \sim \pi(A \mid s_t) \\
    s_{t+1} \sim \mathcal{T}(S \mid s_t, a_t)
} }  \left[ \sum_{t=1}^{T} r(s_t, a_t) \right]
$$

where the expectation is taken under the policy's distribution over **trajectories** $\tau = \{s_1, a_t, \ldots, s_T, a_T \}$ given by

$$
p_\pi(\tau) = p_\pi(s_1, a_1, \ldots, s_T, a_T)
= P_0(s_1) \prod_{t=1}^{T} \pi(a_t \mid s_t) \mathcal{T}(s_{t+1} \mid s_t, a_t)
$$

### 1.2: Gym Environment

Below, we provide an implementation of the simple chain environment, following the [OpenAI Gym API](https://gym.openai.com/docs/). It uses the following default environment parameters, which you may assume for the remainder of the tutorial:
* Number of states is $N=|\mathcal{S}| = 5$.
* The "action fail" probability of the transition dynamics $\mathcal{T}(s_{t+1} \mid s_t, a_t)$ is $\epsilon = 0.2$.
* The maximum episode length is 50.

```
import numpy as np
import gym
from gym import spaces

class ChainEnv(gym.Env):
  def __init__(self, num_states=5, action_fail_prob=0.2, max_episode_length=50):
    self.action_space = spaces.Discrete(3)  # no-action, left, right
    self.state_space = spaces.Discrete(num_states)
    self.goal_state = num_states - 1

    self.max_episode_length = max_episode_length

    # self.transition_dynamics[s, a, S] = T(S|s,a).
    self.transition_dynamics = np.zeros(
        (self.state_space.n, self.action_space.n, self.state_space.n))
    for s in range(self.state_space.n):
      # Action 0: no-action
      self.transition_dynamics[s, 0, s] = 1

      # Action 1: left
      next_state = max(s - 1, 0)
      self.transition_dynamics[s, 1, next_state] += 1 - action_fail_prob
      self.transition_dynamics[s, 1, s] += action_fail_prob

      # Action 2: right
      next_state = min(s + 1, self.state_space.n - 1)
      self.transition_dynamics[s, 2, next_state] += 1 - action_fail_prob
      self.transition_dynamics[s, 2, s] += action_fail_prob

    # self.reward[s, a] = r(s, a)
    self.reward = - np.ones((self.state_space.n, self.action_space.n))
    self.reward[self.state_space.n - 2, 2] = 0  # r(N-2,->) = 0
    self.reward[self.state_space.n - 1] = 0     # r(N-1, a) = 0 for any action a

    # Reset MDP to the beginning of an episode.
    self.reset()

  def reset(self):
    self.t = 0      # Time step counter
    self.state = 0  # Agent always starts in state 0.
    return self.state

  def _sample_transition(self, state, action):
    next_state_prob = self.transition_dynamics[state, action]
    next_state = np.argmax(np.random.multinomial(1, next_state_prob))
    return next_state

  def step(self, action):
    self.t += 1
    rew = self.reward[self.state, action]
    self.state = self._sample_transition(self.state, action)
    done = self.state == self.goal_state or self.t >= self.max_episode_length

    # Add any useful diagnostic info here for debugging.
    info = {}

    return self.state, rew, done, info
```

### 1.3: Instantiation

We instantiate ChainEnv, and print some important instance variables:

* `env.state_space` is the state space $\mathcal{S}$.
* `env.action_space` is the action space $\mathcal{A}$.
* `env.transition_dynamics[s, a, S]` is the transition dynamics probability $\mathcal{T}(S \mid s, a)$ that taking action $a$ in state $s$ results in the next state $S$.
* `env.reward[s, a]` is the  reward matrix $r(s, a)$.

Note that we represent the reward function $r(s_t, a_t)$ as a $|\mathcal{S}| \times |\mathcal{A}|$ matrix, and the transition dynamics function $\mathcal{T}(s_{t+1} \mid s_t, a_t)$ as a $|\mathcal{S}| \times |\mathcal{A}| \times |\mathcal{S}|$ matrix. We can do this since the state and action spaces are finite.

```
env = ChainEnv()
print('env.state_space.n  = |S| =', env.state_space.n)
print('env.action_space.n = |A| =', env.action_space.n)

# Print the distribution over next states for a given state and action.
s = 0
a = 2
print('env.transition_dynamics[{0}, {1}, S] = T(S|s={0}, a={1}) = {2}'.format(
    s, a, env.transition_dynamics[s, a, :]))

# Print the state transition matrix T(s->S') for a given action.
a = 1
print('env.transition_dynamics[s, {0}, S] = T(S|s, a={0}) = \n{1}'.format(
    a, env.transition_dynamics[:, a, :]))

print('env.reward[s, a] = \n{}'.format(env.reward))
```

### 1.4: Evaluating a Policy

We represent a policy $\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0, 1]$ as a $\mathcal{S} \times \mathcal{A}$ matrix, where `policy[s, a]`$=\pi(a \mid s)$. Recall that a policy defines a probability distribution over actions, so each row of `policy` should sum to 1:
$$
1 = \sum_{a \in \mathcal{A}} \pi(a \mid s) \qquad \text{for any }s \in \mathcal{S}.
$$

Below, we provide a function which runs a given `policy` for 100 episodes, and prints the mean/std of the cumulative reward $\mathbb{E}\left[ \sum_{t=1}^T r(s_t, a_t) \right]$ and the state visitation frequencies.

```
from collections import Counter

def evaluate(env, policy, num_episodes=100):
  """
  Evaluates the given policy by running it for given num_episodes.

  Args:
    env (ChainEnv)    : The environment.
    num_episodes (int): Number of episodes to simulate
    policy (np.array) : |S| x |A| matrix where policy[s, a] = pi(a|s). If None,
                        runs a random policy.
  """
  print('Running the following policy[s, a] for {} episodes:\n{}'.format(
      num_episodes, policy))

  states_counter = Counter()
  returns = []
  for _ in range(num_episodes):
    state = env.reset()
    states_counter[state] += 1

    done = False
    total_rew = 0
    while not done:
      # Sample action from the policy for the current state.
      action_probs = policy[state]
      sample_onehot = np.random.multinomial(1, action_probs)
      action = np.argmax(sample_onehot)

      # Execute action, and observe the next state and reward.
      state, rew, done, info = env.step(action)

      total_rew += rew
      states_counter[state] += 1

    # Store cumulative reward at the end of each episode.
    returns.append(total_rew)

  # Compute statistics for episodic returns (cumulative rewards).
  avg_return = np.mean(returns)
  std_return = np.std(returns)

  # Compute normalized state visitation counts.
  state_visitation = np.zeros(env.state_space.n)
  total_count = sum(states_counter.values())
  for state, count in states_counter.items():
    state_visitation[state] = count / total_count

  print('Average Total Reward: {}, std: {}'.format(avg_return, std_return))
  print('State visitation frequency:', state_visitation)
  print()
```

#### 1.4.1: Uniform Policy

The code below evaluates a uniformly random policy $\pi(a \mid s) = \frac{1}{|\mathcal{A}|}$ which chooses each action uniformly at random.

```
uniform_policy = np.full((env.state_space.n, env.action_space.n),
                         1. / env.action_space.n)
evaluate(env, uniform_policy)
```

### 1.5: Reinforcement Learning

Below, we provide an implementation of **Q-Learning**, a classic model-free reinforcement learning algorithm that performs a simple value iteration update using the weighted average of the old value and the new information:

$$
Q(s_t,a_t) \leftarrow (1 - \alpha) Q(s_t, a_t) + \alpha (r_t + \gamma \max_a Q(s_{t+1}, a_t))
$$

Here, $\alpha \in (0, 1]$ is the learning rate and $\gamma \in [0, 1]$ is the discount factor, which we assume to be fixed hyperparameters.

Q-Learning uses an **epsilon-greedy strategy**: At each time step, it chooses a random action $a_t \in \mathcal{A}$ with probability $\epsilon$, and the max action $a_t = \arg\max_a Q(s_t, a)$ otherwise.

```
def train_qlearning(env, num_epochs=10):
  # Hyperparameters
  alpha = 1.0
  gamma = 0.9
  epsilon = 0.1

  q_table = np.zeros([env.state_space.n, env.action_space.n])

  # Train Q-Learning.
  for i in range(1, num_epochs):
    state = env.reset()
    done = False
    while not done:
      # Take the next action.
      if np.random.uniform(0, 1) < epsilon:
          action = env.action_space.sample()  # Explore action space
      else:
          action = np.argmax(q_table[state])  # Exploit learned values
      next_state, rew, done, info = env.step(action)

      old_value = q_table[state, action]
      next_max = np.max(q_table[next_state])
      new_value = (1 - alpha) * old_value + alpha * (rew + gamma * next_max)
      q_table[state, action] = new_value

      state = next_state

  # Compute policy pi(a|s) = argmax_a Q(s, a).
  argmax_actions = np.argmax(q_table, axis=1)
  policy = np.zeros([env.state_space.n, env.action_space.n])
  for state, action in enumerate(argmax_actions):
    policy[state, action] = 1

  return q_table, policy
```

#### Exercise 1.5.1: Q-Learning
Train Q-learning, then evaluate the learned greedy policy $\pi_\text{greedy}(a \mid s) := \arg\max_a Q(s, a)$ on the environment.

* Describe the policy $\pi_\text{greedy}$ in words -- how does it behave?
* How does $\pi_\text{greedy}$ compare to the uniform policy in Section 1.4.1 in terms of average total reward?

```
q_table, q_policy = train_qlearning(env)
print('Q(s,a):\n{}\n'.format(q_table))
evaluate(env, q_policy)
```

## II. Reinforcement Learning as Probabilistic Inference

In this section,  we formalize reinforcement learning as probabilistic inference, following the tutorial from [Levine 2018](https://arxiv.org/pdf/1805.00909.pdf).

### 2.1: Differences from the Original Tutorial

In all sections of this Colab tutorial (except Exercise 2.3.1), we do **not** restrict the action prior $p(a_t \mid s_t)$ to be uniform, so the joint optimality-action distribution
$$
\begin{aligned}
p(O_t = 1, a_t \mid s_t)
&= p(O_t = 1 \mid s_t, a_t) p(a_t \mid s_t)\\
&= \exp(r(s_t, a_t)) p(a_t \mid s_t)
\end{aligned}
$$
is not necessarily proportional to $p(O_t = 1 \mid s_t, a_t) := \exp\{ r(s_t, a_t) \} $.  We also define the backward state-action message as
$$
\beta_t(s_t, a_t) := p(O_{t:T}, a_t \mid s_t),
$$
whereas [Levine 2018](https://arxiv.org/abs/1805.00909) defines it as $\beta_t(s_t, a_t) = p(O_{t:T} \mid s_t, a_t)$. This results in slightly different derivations for the backward message update equations and the optimal policy $p(a_t \mid s_t, O_{t:T})$, which you will derive in Section 2.4. Thus, be careful when copying derivations from the original tutorial.

In Section 2.3, you will prove that any non-uniform action prior $p(a_t \mid s_t)$ can be incorporated into the reward function. We will later experiment with the Message-Passing algorithm using different action priors $p(a_t \mid s_t)$ for a fixed reward function $r(s_t, a_t)$, which is equivalent to using a uniform action prior with different reward functions.

### 2.2: Graphical Model for Control

We define a graphical model that allows us to embed control into the framework of PGMs.

A task can be defined by a **reward** function $r(s_t, a_t)$. However, a graphical model has no notion of rewards or costs, so we introduce a binary random variable for **optimality** where $O_t = 1$ indicates that time step $t$ is optimal.

We choose the distribution over $O_t$ to be

$$
\begin{aligned}
p(O_t = 1 \mid s_t, a_t) &:= \exp(r(s_t, a_t)) \\
p(O_t = 1, a_t \mid s_t)
&= p(O_t = 1 \mid s_t, a_t) p(a_t \mid s_t) \\
&= \exp(r(s_t, a_t)) p(a_t \mid s_t)
\end{aligned}
$$

where $p(a_t \mid s_t)$ is an **action prior**. This leads to a natural posterior distribution over actions when we condition on $O_t = 1$ for all $t \in [T]$:

$$
\begin{aligned}
p(\tau \mid o_{1:T})
\propto p(\tau, o_{1:T})
&= P_0(s_1) \prod_{t=1}^T p(O_t = 1, a_t \mid s_t) \mathcal{T}(s_{t+1} \mid s_t, a_t) \\
&= P_0(s_1) \prod_{t=1}^T \exp(r(s_t, a_t)) p(a_t \mid s_t) \mathcal{T}(s_{t+1} \mid s_t, a_t) \\
&=\underbrace{ \left[ P_0(s_1) \prod_{t=1}^T \mathcal{T}(s_{t+1} \mid s_t, a_t) p(a_t \mid s_t) \right] }_{\text{dynamics}} \exp  \underbrace{ \left( \sum_{t=1}^T r(s_t, a_t)  \right)  }_{\text{total reward}}
\end{aligned}
$$

Observe that, if we know the dynamics, then we can **infer a likely optimal trajectory** using
$$
\tau^* = \underset{\tau}{\arg\max} \; p(\tau, o_{1:T}).
$$

For the remainder of this tutorial, we use $O_{t:T}$ to denote $O_{t:T}=1$ for conciseness.

### 2.3: Reward vs. Action Prior

Note that the joint distribution  $p(\tau, o_{1:T})$ has an extra factor $p(a_t \mid s_t)$ compared to Eq. (4) in [Levine 2018](https://arxiv.org/abs/1805.00909), due to the fact that
$$
\begin{equation}
p(O_t = 1, a_t \mid s_t) = p(O_t = 1 \mid s_t, a_t) p(a_t \mid s_t)
\end{equation}.
$$
This is a typo in the original tutorial, but the math ends up being OK (by changing some equalities '$=$' to proportionalities '$\propto$') if we assume a **uniform action prior** $p(a_t \mid s_t) = \frac{1}{|\mathcal{A}|}$, because then $p(O_t = 1, a_t \mid s_t) \propto p(O_t = 1 \mid s_t, a_t)$.

[Levine 2018](https://arxiv.org/abs/1805.00909) assumes a uniform action prior, and argues that this assumption does not introduce any loss of generality. In the following exercise, you will show that any non-uniform action prior $p(a_t \mid s_t)$ can be incorporated into $p(O_t \mid s_t, a_t) := \exp\{ r_1(s_t, a_t) \}$ via the reward function $r_1(s_t, a_t)$.

#### Exercise 2.3.1: Non-Uniform Action Priors

Let $r(s_t, a_t)$ and $p(a_t \mid s_t)$ be any given reward function and action prior, respectively.  Show that there exists some reward function $r_1(s_t, a_t)$ such that the posterior distribution $p(\tau \mid o_{1:T})$ is equal for the following combinations of reward function and action prior:
1. The reward function $r(s_t, a_t)$ and action prior $p(a_t \mid s_t)$.
2. The reward function $r_1(s_t, a_t)$ and a uniform action prior.

Write down the expression for $r_1(s_t, a_t)$ in terms of $r(s_t, a_t)$ and $p(a_t \mid s_t)$.

### 2.4: Message-Passing Derivations

In this section, we will derive a standard sum-product inference algorithm to **infer the optimal policy** $p(a_t \mid s_t, O_{t:T})$. Define the following **backward messages** for $t \in \{T, \ldots, 1\}$:
$$
\begin{aligned}
\beta_t(s_t, a_t) &:= p(O_{t:T}, a_t \mid s_t) \\
\beta_t(s_t) &:= p(O_{t:T} \mid s_t)
\end{aligned}
$$

#### Exercise 2.4.1: Derivation of $\beta_t(s_t)$ Update

Show that the backward messages satisfy the following update equation for $\beta_t(s_t)$:

$$
\begin{equation}
\beta_t(s_t)
= \sum_{a_t \in \mathcal{A}} \beta_t(s_t, a_t)
\tag{1}
\end{equation}
$$

#### Exercise 2.4.2: Derivation of $\beta_t(s_t, a_t)$ Update

Show that the backward messages satisfy the following update equation for $\beta_t(s_t, a_t)$:
$$
\begin{equation}
\beta_t(s_t, a_t)
= \sum_{s_{t+1} \in \mathcal{S}} \beta_{t+1}(s_{t+1}) \mathcal{T}(s_{t+1} \mid s_t, a_t) p(O_t, a_t \mid s_t)
\tag{2}
\end{equation}
$$

#### Exercise 2.4.3: Derivation of the Optimal Policy

Show that the optimal policy $p(a_t \mid s_t, O_{t:T})$ satisfies
$$
\begin{equation}
p(a_t \mid s_t, O_{t:T})
= \frac{ \beta_t(s_t, a_t) }{ \beta_t(s_t) }
\tag{3}
\end{equation}
$$

### 2.5: Message-Passing Implementation

In this section, we provide code for running the message-passing algorithm.

#### 2.5.1: Message-Passing Algorithm

Below, we provide an implementation of the recursive message-passing algorithm for computing the backward messages $\beta_t(a_t, s_t)$ and $\beta_t(s_t)$. The algorithm starts from the last time step $t=T$,
$$
\beta_T(s_T, a_T) = p(O_T, a_T \mid s_T),
$$
and proceeds backwards through time to $t=1$, computing the following updates at each time step:

1. Update $\beta_{t}(s_{t}, a_{t})$ using Eq. (2).

2. Update $\beta_t(s_t)$ using Eq. (1).

Then the algorithm infers the optimal policy $p(a \mid s, O_{1:T})$
using Eq. (3).

```
def log_sum_exp(terms):
    """
    Uses log-sum-exp trick to compute
      log(\sum_i exp(terms[i])) = t* + log(\sum_i exp(terms[i] - t*))
    where t* = max(terms).
    """
    max_term = np.max(terms)
    diff = np.exp(terms - max_term)
    result = max_term + np.log(np.sum(diff))
    return result

class MessagePassing():
  def __init__(self, env, action_prior=None):
    """
    Args:
      env: GridworldEnv object
      action_prior: Action prior p(a|s), which is a matrix of shape |S| x |A|.
                    If None, we use a uniform action prior: p(s|a) = 1/|A|.
    """
    self.env = env

    # self.log_action_prior[s, a] = log p(a|s)
    if action_prior is None:
      # Uniform action prior: p(s|a) = 1/|A|.
      self.log_action_prior = np.full(
          (self.env.state_space.n, self.env.action_space.n),
          np.log(1. / self.env.action_space.n))
    else:
      assert (action_prior.shape[0] == self.env.state_space.n and
              action_prior.shape[1] == self.env.action_space.n)
      eps = 1e-10  # to prevent underflow
      self.log_action_prior = (np.log(action_prior + eps) -
                               np.log(1 + eps * self.env.action_space.n))

    # self.log_transition_dynamics[s, a, S] = log T(S|s,a)
    self.log_transition_dynamics = np.log(env.transition_dynamics)

    # self.log_optimality_dist[s, a] = log p(O|s,a) = r(s,a)
    self.log_optimality_dist = env.reward

    # self.log_optimality_action_dist[s, a] = log p(O,a|s)
    #                                       = log p(O|s,a) + log p(a|s).
    self.log_optimality_action_dist = (self.log_optimality_dist +
                                       self.log_action_prior)

    # self.log_state_action_message[s, a] = log beta(s, a).
    # It is initialized to the log optimality distribution, log p(O_T,a_T|s_T).
    self.log_state_action_message = (self.log_optimality_dist +
                                     self.log_action_prior)

    # self.state_message[s] = beta(s).
    self.log_state_message = self._compute_state_message_update()

    # self.policy[s, a] = p(a|s,O) = beta(s,a) / beta(s).
    self.policy = self.compute_policy()

  def _compute_state_message_update(self):
    """
    Computes the state-message update in Eq. (1) in log-space:
      log beta(s) = log(\sum_a exp(log beta(s,a))).
    """
    log_state_message = np.zeros(self.env.state_space.n)
    for s in range(self.env.state_space.n):
      log_state_message[s] = log_sum_exp(self.log_state_action_message[s, :])
    return log_state_message

  def _compute_state_action_message_update(self):
    """
    Computes the state-action-message update in Eq. (2) in log-space:
      log beta(s,a) = log(\sum_S exp(log beta(S) + log T(S|s,a) + log p(O,a|s)))
    """
    log_state_action_message = np.zeros(
        (self.env.state_space.n, self.env.action_space.n))
    for s in range(self.env.state_space.n):
      for a in range(self.env.action_space.n):
        terms = (self.log_state_message +
                 self.log_transition_dynamics[s, a] +
                 self.log_optimality_action_dist[s, a])
        log_state_action_message[s, a] = log_sum_exp(terms)
    return log_state_action_message

  def update_messages(self):
    """
    Performs a single step of the backward message-passing algorithm.
    """
    # beta(s,a) = \sum_S beta(S) * p(S|s,a) * p(O,a|s)
    self.log_state_action_message = self._compute_state_action_message_update()

    # beta(s) = \sum_a beta(s,a) * p(a|s)
    self.log_state_message = self._compute_state_message_update()

    return self.log_state_action_message, self.log_state_message

  def compute_policy(self):
    """
    Computes policy using the given backward messages:
      p(a|s,O) = beta(s,a) / beta(s).
    """
    log_policy = np.zeros((env.state_space.n, env.action_space.n))
    for s in range(env.state_space.n):
      log_policy[s] = (self.log_state_action_message[s] -
                       self.log_state_message[s])
    policy = np.exp(log_policy)
    return policy

  def get_log_messages(self):
    """
    Returns the soft value function and soft Q-function:
      V(s)   := log beta(s)
      Q(s,a) := log beta(s,a).
    """
    return self.log_state_message, self.log_state_action_message
```

#### 2.5.2: Script to run Message-Passing

We provide a function that runs message passing for $T$ steps using the given action prior $p(a_t \mid s_t)$. It prints the final policy, and the soft value function and soft Q-function:
$$
\begin{aligned}
V(s) &:= \log \beta_1(s) \\
Q(s,a) &:= \log \beta_1(s, a)
\end{aligned}
$$

Finally, it evaluates the policy on ChainEnv and prints the results.

```
def run_message_passing(env, action_prior, T=50):
  mp = MessagePassing(env, action_prior=action_prior)
  for _ in range(T):
    state_action_message, state_messages = mp.update_messages()

  # Print the soft value function V(s) and soft Q-function Q(s,a).
  V, Q = mp.get_log_messages()

  # Evaluate the learned policy for num_episodes.
  policy = mp.compute_policy()

  print('Action prior p(a|s):\n{}\n'.format(action_prior))
  print('V(s):\n{}\n'.format(V))
  print('Q(s,a):\n{}\n'.format(Q))
  evaluate(env, policy=policy)
```

### 2.6: Message-Passing Experiments

In this section, we will experiment running Message-Passing with different action priors $p(a_t \mid s_t)$.

#### Exercise 2.6.1: Uniform Action Prior

Run message-passing algorithm using a **uniform** action prior $p(a_t \mid s_t) = \frac{1}{|\mathcal{A}|}$.

* How does the learned policy compare to the Q-Learning policy $\pi_\text{greedy}$ from Exercise 1.5.1 in terms of behavior and average total reward?

```
# Run message-passing using a uniform action prior.
uniform_policy = np.full((env.state_space.n, env.action_space.n),
                         1. / env.action_space.n)
run_message_passing(env, action_prior=uniform_policy)
```

#### Exercise 2.6.2: Soft Action Prior

The following function `construct_policy()` returns a policy $\pi(a \mid s; \phi)$ such that, regardless of the state, it takes the 'right' action with probability $\phi$, and all other actions uniformly at random. That is:
$$
\pi(a \mid s;\phi) = \begin{cases}
\phi & \text{if }a=\rightarrow\\
\frac{1 - \phi}{|\mathcal{A}| - 1}& \text{otherwise}
\end{cases}
$$

Run message-passing algorithm using a "soft" action prior $\pi(a \mid s;\phi)$ for $\phi = 0.5$.

* How does the learned policy compare to the one from Exercise 2.6.1 (using uniform action prior) in terms of behavior and average total reward?
* (True or False) If $\phi > \frac{1}{|\mathcal{A}|}$, then using the action prior $\pi(a \mid s ; \phi)$ instead of a uniform action prior is equivalent to changing the reward function $r(s_t, a_t)$ such that the agent receives relatively greater reward for taking the action $a_t = \rightarrow$ in any state, and less reward otherwise.

```
def construct_policy(env, phi=0.5):
  """
  Returns a policy such that, regardless of the state, it takes the given action
  with probability phi, and all other actions uniformly at random.
  """
  right_action = 2
  policy = np.zeros((env.state_space.n, env.action_space.n))
  for s in range(env.state_space.n):
    policy[s, right_action] = phi
    for a in range(env.action_space.n):
      if a != right_action:
        policy[s, a] = (1. - phi) / (env.action_space.n - 1)
  return policy

# A "soft" policy that chooses 'right' with probability 0.5.
pi = construct_policy(env, phi=0.5)
run_message_passing(env, action_prior=pi)
```

#### Exercise 2.6.3: Hard Action Prior

Run message-passing algorithm using a "hard" action prior $\pi(s,a;\phi)$ for $\phi = 1.0$.

* How does the learned policy compare to the Q-Learning policy $\pi_\text{greedy}$ from Exercise 1.5.1 in terms of behavior and average total reward?

```
# A "hard" policy that chooses 'right' with probability 1.0.
pi = construct_policy(env, phi=1.0)
run_message_passing(env, action_prior=pi)
```

## III. Concluding Remarks

### 3.1: Q-Learning vs. Message-Passing

In this section, we will address how Q-Learning and Message-Passing are different, and when might one be preferred.

#### Exercise 3.1.1: Unknown Transition Dynamics

Suppose we don't know the transition dynamics $\mathcal{T}(s_{t+1} \mid s_t, a_t)$.

1. Can you learn the optimal policy via Q-learning?

2. Can you learn the optimal policy via Message-Passing?

#### Exercise 3.1.2: Equivalence

Is it the case that the optimal message-passing policy can be equivalent to the one discovered by Q-learning? If yes, under which conditions? If no, why not?


## Data Analytics

### Bootstrap Confidence Intervals in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Bootstrap Confidence Intervals in Python

Here's a simple workflow, demonstration of bootstrap for modeling workflows. This should help you get started with this important data analytics method to evaluate and integrate uncertainty in any sample statistics or model.

#### Bootstrap

The uncertainty in an estimated population parameter from a sample, represented as a range, lower and upper bound, based on a specified probability interval known as the **confidence level**.

* one source of uncertainty is the paucity of data.
* do 200 or even less sample data provide a precise (and accurate estimate) of the mean? standard deviation? skew? 13th percentile / P13? 3rd central moment? experimental variogram? mutual information? Shannon entropy? etc.

Would it be useful to know the uncertainty due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

* calculate and display the histogram or CDF to see the full distribution

* report the percentiles representing the lower and upper confidence intervals, e.g. for a confidence level of 95%, the 2.5 and 97.5 percentiles.

This is a very powerful method. Let's demonstrate a bunch of measures.

* when available, I compare to the distributions from the analytic expressions.

Let's demonstrate for a variety of parameters / statistics:

* mean / arithmetic average

* proportion

* interquartile range

* coefficient of variation

* correlation coefficient

#### Objective

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

We need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # plotting
import matplotlib                         # plotting
import seaborn as sns                     # plotting
from scipy import stats                   # summary statistics
import math                               # trig etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # random sampling
from scipy.stats import gaussian_kde      # for PDF calculation
from scipy.stats import t                 # Student's t distribution for analytical solution
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Set the Random Number Seed

For repeatability, set the random number seed.

* this ensures that the same random samples are drawn each run and everyone in the class will have the same results.

* change this seed number for a new set of random realizations

```python
seed = 73073
np.random.seed(seed = seed)
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv') # load our data table
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

```python
df = df.sample(frac = 0.02)                 # extract 50 random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 6 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head()                                  # display first 4 samples in the table as a preview
```

      X
      Y
      Facies
      Porosity
      Perm

      131
      320
      99
      1
      0.141749
      13.154686

      180
      750
      679
      0
      0.074619
      0.089169

      120
      900
      559
      1
      0.133474
      13.104697

      276
      730
      79
      1
      0.134691
      15.160833

      13
      300
      400
      1
      0.118448
      8.495566

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      6.0
      628.333333
      253.725574
      300.000000
      422.500000
      740.000000
      765.000000
      900.000000

      Y
      6.0
      467.500000
      351.097565
      79.000000
      174.250000
      479.500000
      649.000000
      989.000000

      Facies
      6.0
      0.666667
      0.516398
      0.000000
      0.250000
      1.000000
      1.000000
      1.000000

      Porosity
      6.0
      0.112892
      0.030695
      0.074373
      0.085576
      0.125961
      0.134387
      0.141749

      Perm
      6.0
      8.457717
      6.605513
      0.089169
      2.679906
      10.800132
      13.142189
      15.160833

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
permmin = 0.01; permmax = 10000
nx = 100; ny = 100; csize = 10.0
cmap = plt.cm.plasma                    # color map
cumul_prob = np.linspace(0.0,1.0,100)   # list of cumulative probabilities
```

Let's try out location maps, histograms and scatter plots.

```python
plt.subplot(231)
plt.scatter(df['X'],df['Y'],s = 20,c = df['Porosity'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = pormin,vmax = pormax)
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Porosity Location Map')

plt.subplot(234)
plt.hist(df['Porosity'],color = 'red',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)))
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')

plt.subplot(232)
plt.scatter(df['X'],df['Y'],s = 20,c = df['Perm'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = permmin,vmax = permmax,norm=matplotlib.colors.LogNorm())
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Permeability Location Map')

plt.subplot(235)
plt.hist(df['Perm'],color = 'red',alpha = 0.3,edgecolor='black',bins=np.logspace(np.log10(permmin),np.log10(permmax),int(len(df)/3)))
#sns.kdeplot(x=df['Perm'],color = 'black',alpha = 0.2,levels = 1,log_scale = True,bw_adjust = 1)
plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Permeability Histogram'); plt.xscale('log')

plt.subplot(233)
plt.scatter(df['Porosity'],df['Perm'],s = 20,color = 'red',alpha = 0.3,edgecolor='black')
#plt.contour(df['Porosity'],df['Perm'] Z, colors='black');
plt.ylabel('Permeability (mD)'); plt.xlabel('Porosity (fraction)'); plt.title('Permeability-Porosity Scatter Plot')
plt.yscale('log')
sns.kdeplot(x=df['Porosity'],y=df['Perm'],color = 'black',alpha = 0.2,levels = 4)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.2, wspace=0.2, hspace=0.2)
plt.show()
```

### Bootstrap Method in Python

If you are new to bootstrap and Python, here's the most simple code possible for bootstrap.

* specify the number of bootstrap realizations, $L$
* declare a list to store the bootstrap realizations of the statistic of interest
* loop over L bootstrap realizations
    * n MCS, random samples with replacement for a new realization of the data
    * calculate the realization of the statistic from the realization of the data
* summarize the resulting uncertainty model, histogram, summary statistics etc.

#### Bootstrap of the Sample Mean, Arithmetic Average

Let's demonstrate bootstrap for uncertainty in the arithmetic average with a simple workflow.

```python
por_avg_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    por_avg_real.append(np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset
```

We will compare the bootstrap uncertainty in the sample arithmetic average distribution with the analytical expression:

\begin{equation}
\overline{x} \pm t_{\frac{\alpha}{2},n-1}\sqrt{\frac{s^2}{n}}
\end{equation}

The remaining code in this block is just a super cool set of plots with the results.

```python
# Bootstrap method for uncertainty in the sample mean
L = 1000                               # set the number of bootstrap realizations
por_avg_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    por_avg_real.append(np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset

# Analytical solution for the uncertainty in the sample mean, Student's t distribution with the correct mean and standard deviation
analytical = t.pdf(np.linspace(pormin,pormax,100), len(df)-1,loc=np.average(df['Porosity']),scale=math.sqrt(np.var(df['Porosity'].values)/len(df)))

# Plot of the original data and bootstap and analytical results
fig = plt.subplot(131)
plt.hist(df['Porosity'],color = 'red',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)))
#plt.plot([np.average(df['Porosity']),np.average(df['Porosity'])],[0,100])
plt.axvline(x=np.average(df['Porosity']),linestyle="--",c='black')
plt.text(np.average(df['Porosity'])+0.005, 8.8, r'Average = ' + str(round(np.average(df['Porosity']),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 8.3, r'St.Dev. = ' + str(round(np.std(df['Porosity']),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 7.8, r'P10 = ' + str(round(np.percentile(df['Porosity'],10),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 7.3, r'P90 = ' + str(round(np.percentile(df['Porosity'],90),3)), fontsize=12)
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Original Porosity Histogram')

plt.subplot(132)
plt.hist(por_avg_real,color = 'red',alpha = 0.2,edgecolor = 'black',bins=np.linspace(pormin,pormax,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.plot(np.linspace(pormin,pormax,100),analytical,color = 'black',label = 'analytical',alpha=0.4)
plt.axvline(x=np.average(por_avg_real),linestyle="--",c='black')
plt.fill_between(np.linspace(pormin,pormax,100), 0, analytical, where = np.linspace(pormin,pormax,100) <= np.percentile(por_avg_real,10), facecolor='red', interpolate=True, alpha = 0.5)
plt.fill_between(np.linspace(pormin,pormax,100), 0, analytical, where = np.linspace(pormin,pormax,100) >= np.percentile(por_avg_real,90), facecolor='red', interpolate=True, alpha = 0.5)
plt.text(np.average(por_avg_real)+0.009, L*0.07, r'Average = ' + str(round(np.average(por_avg_real),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.066, r'St.Dev. = ' + str(round(np.std(por_avg_real),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.062, r'P90 = ' + str(round(np.percentile(por_avg_real,90),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.058, r'P10 = ' + str(round(np.percentile(por_avg_real,10),3)), fontsize=12)
plt.xlabel('Bootstrap Realizations and Analytical Uncertainty Distribution for Mean'); plt.ylabel('Frequency'); plt.title('Distribution of Average')
plt.legend()

fig = plt.subplot(133)
plt.hist(df['Porosity'],color = 'red',alpha = 0.2,edgecolor='grey',bins = np.linspace(pormin,pormax,int(len(df)/3)))
#plt.plot([np.average(df['Porosity']),np.average(df['Porosity'])],[0,100])
plt.axvline(x=np.average(df['Porosity']),c='black')
plt.axvline(x=np.percentile(por_avg_real,90),linestyle="--",c='black')
plt.axvline(x=np.percentile(por_avg_real,10),linestyle="--",c='black')
plt.text(np.percentile(por_avg_real,90)+0.009,8.8, r'Average = ' + str(round(np.average(por_avg_real),3)), fontsize=12)
plt.text(np.percentile(por_avg_real,90)+0.009,8.3, r'Average P90 = ' + str(round(np.percentile(por_avg_real,90),3)), fontsize=12)
plt.text(np.percentile(por_avg_real,90)+0.009,7.8, r'Average P10 = ' + str(round(np.percentile(por_avg_real,10),3)), fontsize=12)
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram with Uncertainty in Average')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Proportion

Let's demonstrate another case for which we have the analytical solution.

We will compare the bootstrap results with the analytical expression:

\begin{equation}
\hat{p} \pm t_{\frac{\alpha}{2},n-1}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}
\end{equation}

```python
#
L = 100000                               # set the number of bootstrap realizations
sand_prop_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
shale_prop_real = []
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Facies'].values, k=len(df)) # n Monte Carlo simulations
    sand_prop_real.append(samples.count(1)/len(df)) # calculate the statistic of interest from the new bootstrap dataset
    shale_prop_real.append(samples.count(0)/len(df)) # calculate the statistic of interest from the new bootstrap dataset
sand_prop = np.sum(df['Facies'] == 1)/len(df); shale_prop = np.sum(df['Facies'] == 0)/len(df)

analytical_shale = t.pdf(np.linspace(0.0,1.0,100), len(df)-1,loc=shale_prop,scale=math.sqrt(shale_prop*(1.0-shale_prop)/len(df)))

fig = plt.subplot(121)
barlist = plt.bar(x=['shale','sand'],height = [1-sand_prop,sand_prop],color = 'red',alpha = 0.3,edgecolor='black')
barlist[0].set_color('blue')
plt.text(-.3, shale_prop+0.02, r'Prop. Shale = ' + str(round(shale_prop,2)), fontsize=12)
plt.text(0.7, sand_prop+0.02, r'Prop. Sand = ' + str(round(sand_prop,2)), fontsize=12)
plt.xlabel('Rock Type / Facies'); plt.ylabel('Proportion'); plt.title('Facies Histogram')
plt.ylim([0,1]); plt.yticks(np.arange(0, 1.1, 0.1))

plt.subplot(122)
plt.hist(sand_prop_real,color = 'red',alpha = 0.2,edgecolor = 'black',bins=np.linspace(0.0,1.0,40),label = 'sand',density = True) # plot the distribution, could also calculate any summary statistics
analytical = t.pdf(np.linspace(0.0,1.0,100), len(df)-1,loc=sand_prop,scale=math.sqrt(sand_prop*(1.0-sand_prop)/len(df)))
plt.plot(np.linspace(0.0,1.0,100),analytical,color = 'black',label = 'analytical',alpha=0.4)
plt.axvline(x=sand_prop,linestyle="--",c='black')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical, where = np.linspace(0.0,1.0,100) <= np.percentile(sand_prop_real,10), facecolor='red', interpolate=True, alpha = 0.5)
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical, where = np.linspace(0.0,1.0,100) >= np.percentile(sand_prop_real,90), facecolor='red', interpolate=True, alpha = 0.5)
plt.text(np.average(sand_prop_real)+0.009, 7.0, r'Prop. = ' + str(round(sand_prop,2)), fontsize=12)
plt.text(np.average(sand_prop_real)+0.009, 6.4, r'Prop. P90 = ' + str(round(np.percentile(sand_prop_real,90),2)), fontsize=12)
plt.text(np.average(sand_prop_real)+0.009, 5.8, r'Prop. P10 = ' + str(round(np.percentile(sand_prop_real,10),2)), fontsize=12)

plt.hist(shale_prop_real,color = 'blue',alpha = 0.2,edgecolor = 'black',bins=np.linspace(0.0,1.0,40),label = 'shale',density = True) # plot the distribution, could also calculate any summary statistics
plt.plot(np.linspace(0.0,1.0,100),analytical_shale,color = 'black',alpha=0.4)
plt.axvline(x=shale_prop,linestyle="--",c='black')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical_shale, where = np.linspace(0.0,1.0,100) <= np.percentile(shale_prop_real,10), facecolor='blue', interpolate=True, alpha = 0.5)
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical_shale, where = np.linspace(0.0,1.0,100) >= np.percentile(shale_prop_real,90), facecolor='blue', interpolate=True, alpha = 0.5)
plt.text(np.average(shale_prop_real)+0.009, 7.0, r'Prop. = ' + str(round(shale_prop,2)), fontsize=12)
plt.text(np.average(shale_prop_real)+0.009, 6.4, r'Prop. P90 = ' + str(round(np.percentile(shale_prop_real,90),2)), fontsize=12)
plt.text(np.average(shale_prop_real)+0.009, 5.8, r'Prop. P10 = ' + str(round(np.percentile(shale_prop_real,10),2)), fontsize=12)

plt.xlabel('Bootstrap Realizations and Analytical Uncertainty Distributions for Proportion'); plt.ylabel('Frequency'); plt.title('Distribution of Bootstrap Proportions')
plt.legend(loc = 'upper left')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Interquartile Range

To prove that we can bootstrap any statistic let's select a more complicated measure of dispersion, the interquartile range.

```python
L = 1000                               # set the number of bootstrap realizations
iqr_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    iqr_real.append(np.percentile(samples,75) - np.percentile(samples,25)) # calculate the statistic of interest from the new bootstrap dataset

iqr = np.percentile(df['Porosity'],75) - np.percentile(df['Porosity'],25)

plt.subplot(111)
plt.hist(iqr_real,color = 'red',alpha = 0.2,edgecolor = 'black',bins=np.linspace(0.0,0.125,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=iqr,c='black')
plt.axvline(x=np.percentile(iqr_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(iqr_real,90),linestyle="--",c='black')
sns.kdeplot(x=iqr_real,color = 'grey',alpha = 0.1,levels = 1,bw_adjust = 1)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.07, r'Average = ' + str(round(np.average(iqr_real),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.066, r'St.Dev. = ' + str(round(np.std(iqr_real),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.062, r'P90 = ' + str(round(np.percentile(iqr_real,90),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.058, r'P10 = ' + str(round(np.percentile(iqr_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Interquartile Range'); plt.ylabel('Frequency'); plt.title('Distribution of Interquartile Range')
# plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Coefficient of Variation

Here's another statistic that requires multiple measures from the each bootstrap realization of the data.

* this reinforces that we bootstrap dataset realizations and then calculate the statistic on this dataset realization

For the coefficient of variation we will:

* calculate a bootstrap realization of the dataset with $n$ samples with replacement
* calculate the mean and standard deviation from this bootstrapped realization of the dataset
* calculate a boostrap realization of the coefficient of variation as the standard deviation divided by the mean

Repeat this $L$ times and then evaluate the resulting distribution.

```python
L = 1000                               # set the number of bootstrap realizations
cv_real = []                      # declare an empty list to store the bootstrap realizations of the statistic

for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    cv_real.append(np.std(samples)/np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset

cv = np.std(df['Porosity'])/np.average(df['Porosity'])

plt.subplot(111)
plt.hist(cv_real,color = 'red',alpha = 0.2,edgecolor = 'black',bins=np.linspace(0.1,0.4,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=cv,c='black')
plt.axvline(x=np.percentile(cv_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(cv_real,90),linestyle="--",c='black')
sns.kdeplot(x=cv_real,color = 'grey',alpha = 0.1,levels = 1,bw_adjust = 1)
plt.text(np.percentile(cv_real,90)+0.009, 16, r'Average = ' + str(round(np.average(cv_real),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 15, r'St.Dev. = ' + str(round(np.std(cv_real),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 14, r'P90 = ' + str(round(np.percentile(cv_real,90),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 13, r'P10 = ' + str(round(np.percentile(cv_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Coefficient of Variation'); plt.ylabel('Frequency'); plt.title('Distribution of Coefficient of Variation')
# plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Correlation Coefficient

Here's a statistic that requires us to work with multiple, paired features at once.
* this reinforces that we boostrap for a new realization of the dataset, a set of samples with all their features.

For the correlation coefficient we will:

* calculate a bootstrap realization of the dataset with $n$ samples with replacement as a new DataFrame with all features
* calculate the the correlation coefficient between 2 paired features

Repeat this $L$ times and then evaluate the resulting distribution.

```python
L = 1000                               # set the number of bootstrap realizations
corr_real = []                      # declare an empty list to store the bootstrap realizations of the statistic

for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = df.sample(n=len(df),replace=True,random_state = seed + k) # n random samples with replacement as a new DataFrame
    corr_real.append(samples[['Porosity','Perm']].corr()['Porosity'][1]) # calculate the statistic of interest from the new bootstrap dataset

corr = df[['Porosity','Perm']].corr()['Porosity'][1]

plt.subplot(111)
plt.hist(corr_real,color = 'red',alpha = 0.2,edgecolor = 'black',bins=np.linspace(0.5,1.0,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=corr,c='black')
plt.axvline(x=np.percentile(corr_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(corr_real,90),linestyle="--",c='black')
sns.kdeplot(x=corr_real,color = 'grey',alpha = 0.1,levels = 1,bw_adjust = 1)
# plt.text(np.percentile(corr_real,90)+0.009, 16, r'Average = ' + str(round(np.average(corr_real),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 15, r'St.Dev. = ' + str(round(np.std(corr_real),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 14, r'P90 = ' + str(round(np.percentile(corr_real,90),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 13, r'P10 = ' + str(round(np.percentile(corr_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Correlation Coefficient'); plt.ylabel('Frequency'); plt.title('Distribution of Correlation Coefficient')
# plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comments

This was a basic demonstration of bootstrap.

* you can use bootstrap to calculate uncertainty in any statistic!

* you are calculating a set of realizations of the statistic, representing uncertainty due to small sample size

* note the assumptions of bootstrap, including stationarity and representativity

* remember, get a dataset realization by bootstrap and then calculate the realization of the statistic from the dataset realization

* if your statistic has multiple inputs (e.g. P25 and P75), calculate each from the same bootstrap realization of the dataset.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```