Table of Contents

1  Multi-Armed Bandits

1.1  Differences Between A/B Testing and Bandit Testing

1.2  Bandit Algorithms

1.2.1  Algorithm 1 - Epsilon Greedy

1.2.2  Algorithm 2 - Boltzmann Exploration (Softmax)

1.2.3  Algorithm 3 - Upper Confidence Bounds (UCB)

1.3  Experimenting With Bandit Algorithms

2  Bayesian Bandits

2.1  Beta Distribution

2.2  Thompson Sampling

2.3  Notes On Bandit Testings

2.3.1  Short-term testing

2.3.2  Long-term testing

2.4  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', 'notebook_format'))

from formats import load_style
load_style(css_style='custom2.css', plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.stats import beta
from collections import namedtuple

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,scipy
```

    Ethen 2019-02-24 13:25:26

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.2
    pandas 0.23.4
    matplotlib 2.2.3
    scipy 1.0.1

# Multi-Armed Bandits

Imagine this scenario: we're in a casino. There are many different slot machines (known as "one-armed bandits", as they're known for robbing people), each with a lever (an arm, if you will). We think that some slot machines payout more frequently than others do, and our goal is to walk out of the casino with the most money.

The question is, how do we learn which slot machine rewards us with the most money in the shortest amount of time? We could try all the slot machines out to get a sense of the expected return from playing each machine. But remember, each time we play a poor performing machine, we lower our take that we walk out of the casino with that night. In order to maximize how much money we walk out of the casino with, we will have to be efficient with how we collect our data.

Rewriting the scenario above into a business language. Each time a shopper looks comes to a webpage, we show them one of the $K$ variations of the webpage. They either click on it or do not, and we log this information about the (binary) reward for each $K$ variations. Next, we proceed to the next shopper and have to choose one of $K$ webpage variations again.

## Differences Between A/B Testing and Bandit Testing

In both scenarios above, we would normally determine our "winner" (the slot machine that pays the most, or the best webpage variations that gets the most clicks) using the well-known **A/B testing** approach. The **A/B testing** approach consists of a period of pure exploration, where we're randomly assigning equal numbers of users to one of the $K$ variations and run the test until it's valid. After that, it jumps into pure exploitation, where you send 100% of your users to the more successful version of your site.

Two possible problems with the classical **A/B testing** approach is that:

- It jumps discretely from exploration to exploitation, when we might be able to transition more smoothly.
- During the exploratory phase (the test), it wastes resources exploring inferior options in order to gather as much data as possible.

Given the exploration - exploitation dilemma stated above, the **bandit testing** approach try to account for this. The following graph depicts the difference between the two types of testing method:

If we have three variations that we wish to test, the **A/B testing** approach we try out each of the three variations with equal proportions until we are done with our test at week 5, and then select the variation with the highest value.

As for **bandit testing**, it attempts to use what it knows about each variation from the very beginning, and it continuously updates the probabilities that it will select each variation throughout the optimization process. In the above chart we can see that with each new week, the **bandit testing** reduces how often it selects the lower performing options and increases how often if selects the highest performing option.

> We need to explore in order to figure out what works and what doesn't. On the other hand, if we exploit we take advantage of what we have learned. The **bandit testing** approach highlights the fact that collecting data also has its cost.

To be specific, **bandit testing** algorithms will try to minimize what's known as regret, which is the difference between our actual payoff and the payoff we would have collected had we played the optimal (best) options at every opportunity. There are tons of different bandit methods, in the next section we'll look at some of the more common ones.

## Bandit Algorithms

Before introducing the algorithms and trying them out through simulations, we'll denote some notations and terminologies to formally define the problem:

- **Arms** is simply the variations that we're testing (webpages that we're testing) and there will be $K$ of them in total.
- In a simulation of $t$ turns (how many samples in a simulation), we'll maintain **empirical means** of the reward for each arm (e.g. after trying out arm A for 10 turns, it got 3 clicks, the empirical means is simply 0.3) that are updated at every turn $t$.
- $u_i(t)$ is the empirical mean of arm $i$ after $t$ turns.
- $p_i(t)$ is the probability of picking arm $i$ at turn $t$.

Let's look at our simulated data before diving into each algorithms (hopefully the docstrings are self-explanatory).

```python
def generate_bernoulli_bandit_data(n_simulations, K):
    """
    Generate simluate data, that represents success / trial data

    Parameters
    ----------
    n_simulations : int
        the total number of turns in a simulation.

    K : int
        the total number of arms.

    Returns
    -------
    ctr : float 1d ndarray, shape[K,]
        the randomly generated empirical click through rate for each arm

    rewards : bool 2d ndarray, shape [n_simulations, K]
        given the empirical ctr, simulate in each turn of the simulation,
        whether the arm gets pulled will obtain the
        reward or not (whether the webpage gets clicked)
    """
    ctr = np.random.rand(K)
    rewards = np.random.rand(n_simulations, K) < np.tile(ctr, (n_simulations, 1))
    return ctr, rewards

K = 2
n_simulations = 5
ctr, rewards = generate_bernoulli_bandit_data(n_simulations, K)
print(ctr)
print(rewards)
```

    [0.02041346 0.55099322]
    [[False  True]
     [False  True]
     [False  True]
     [False  True]
     [False False]]

### Algorithm 1 - Epsilon Greedy

At each round $t = 1, 2, ...$ the **Epsilon Greedy** algorithm will:

- Choose a random arm with the probability of $\epsilon$.
- Choose the arm with the current best empirical mean with probability of $1-\epsilon$.

In mathematical notations:

\begin{align}
p_i(t+1)=
\begin{cases}
    1 - \epsilon + \epsilon \big/ K & \quad \text{if i = } argmax_{j = 1, ..., K} \ u_j(t) \\
    \epsilon \big/ K  & \quad otherwise
\end{cases}
\end{align}

Or more intuitively:

- When a new visitor comes to the site, the algorithm flips a coin that comes up tail with the probability of $\epsilon$. When it does in fact comes up tail, the algorithm is going to explore. The exploration phase is to randomly choose amongst any possible arm with equal (uniform) probability and showing it to the visitor.
- On the other hand, the algorithm will exploit the best known solution with the probability of $1- \epsilon$. To exploit, the algorithm looks up the current empirical means and shows the best one to the visitor.

The image below sums up the algorithm pretty well.

```python
def epsilon_greedy(counts, epsilon=0.5, decrease_const=1000):
    """
    Adaptive epsilon greedy

    Parameters
    ----------
    counts : int 2d-array, shape(K, 2), where K = the total number of arms
        success and failures for each arm where column 0 represents
        success, 1 represents failure

    epsilon : float
        the initial probability of choosing a random arm;
        1 - epsilon is the probability of choosing the current best arm

    decrease_const : int
        parameter for the adaptive (annealing) epsilon, where the epsilon
        parameter will decrease as time goes by.

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    n_arms = counts.shape[0]
    totals = counts.sum(axis=1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    epsilon /= (1 + total_counts / decrease_const)
    if np.random.rand() > epsilon:
        return np.argmax(empirical_means)
    else:
        return np.random.randint(0, n_arms)

# counts : stores the counts of success and failures for each arm
# where column 0 represents success, 1 represents failure.
# each arm's count is initialiated as 1 to ensure that each arm is
# played at least once, to prevent "cold start" problem and
# 0 division in the beginning
K = 2
counts = np.ones((K, 2))
print(counts)
epsilon_greedy(counts)
```

    [[1. 1.]
     [1. 1.]]

    0

The `decrease_const` parameter in the function above may look unfamiliar.

For the **Epsilon Greedy** algorithm, setting the $\epsilon$ can be a bit tricky. If it's too small, exploration will be slow at the beginning, and we will be slow to react to changes. If we happen to sample, say, the second-best arm the first few times, it may take a long time to discover that another arm is actually better. If $\epsilon$ is too big, we'll waste many trials pulling random arms without gaining much.

To accommodate for this situation, we will set the $\epsilon$ value at a higher value in the beginning and anneal (gradually lower) it over time. Intuitively, this simply means that after exploring around for a while, we become more certain about each arms' empirical means. After that, it's better to exploit.

In the function call above, the $\epsilon$ at turn $t$ will become:

\begin{align}
\epsilon(t) = \epsilon(0) \Big/ (1 + t/T)
\end{align}

Where $T$ is a new parameter that represents a decreasing constant.

Note that there are different ways of annealing a parameter, but the spirit is the same.

```python
# show adaptive learning rate
epsilon = 0.5
decrease_const = 1000

# the epsilon value after 10 turns
total_counts = 10
print(epsilon / (1 + total_counts / decrease_const))

# after 10000 turns
total_counts = 10000
print(epsilon / (1 + total_counts / decrease_const))
```

    0.49504950495049505
    0.045454545454545456

### Algorithm 2 - Boltzmann Exploration (Softmax)

The **Softmax** algorithm picks each arm with a probability that is proportional to its average reward.

\begin{align}
p_i(t+1)= \frac{ e^{u_i(t) / \tau} }{ \sum_{j=1}^K  e^{u_j(t) / \tau} }
\end{align}

Where $\tau$ is a temperature parameter, controlling the randomness of the choice. When $\tau$ = 0, the algorithm acts like pure greedy. As $\tau$ grows to infinity, the algorithm will pick arms uniformly at random.

```python
def softmax(counts):
    """
    adaptive softmax

    Parameters
    ----------
    counts : int 2d-array, shape( K, 2 ), where K = the total number of arms
        success and failures for each arm where column 0 represents
        success, 1 represents failure

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    totals = counts.sum(axis=1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    # annealing (adaptive learning rate)
    tau = 1 / np.log(total_counts + 0.000001)

    probs_n = np.exp(empirical_means / tau)
    probs_d = probs_n.sum()
    probs = probs_n / probs_d

    cum_prob = 0.
    z = np.random.rand()
    for idx, prob in enumerate(probs):
        cum_prob += prob
        if cum_prob > z:
            return idx

counts = np.ones((K, 2))
softmax(counts)
```

    1

### Algorithm 3 - Upper Confidence Bounds (UCB)

In the world of statistics, whenever we estimate some unknown parameter (such as the mean of a distribution) using random samples, there is a way to quantify the uncertainty inherent in our estimate.  For example, the true mean of a fair six-sided die is 3.5.  But if we only roll it once and get a 2, our best estimate of the mean is just 2.  Obviously that estimate is not very good, and we can quantify the confidence we have for our estimate.  There are confidence bounds which can be written, for example, as: "The mean of this die is 2, with a 95-th percentile lower bound of 1.4 and a 95-th percentile upper bound of 5.2."

The upper confidence bound (UCB) family of algorithms, as its name suggests, selects the arm with the largest upper confidence bound at each turn.  The intuition is this: the more times we roll the die, the tighter the confidence bounds, and if we roll the die an infinite number of times then the width of the confidence bound is zero. In short, as the number of rolls increases, the uncertainty decreases, and so does the width of the confidence bound.

Thus, unlike **Epsilon Greedy** and **Softmax** algorithm that only keeps track of the empirical means, the **UCB** algorithm also maintains the number of times that each arm has been played, denoted by $n_i(t)$. Initially, each arm is played once. Afterwards, at round t, the algorithm greedily picks the arm $j(t)$ as follows:

\begin{align}
j(t) = argmax_{i = 1, ..., K} \left( u_i + \sqrt{\frac{2 \cdot ln(t)}{n_i}} \right)
\end{align}

We can see that the **UCB** algorithm will try to learn about arms that we don't know enough about. The main advantages of these types of algorithms are:

- Take uncertainty of sample mean estimate into account in a smart way.
- No parameters (e.g. epsilon, annealing) to validate.

```python
def ucb(counts):
    """
    Upper Confidence Bounds

    Parameters
    ----------
    counts : int 2d ndarray, shape [K, 2], where K = the total number of arms
        success and failures for each arm where column 0 represents
        success, 1 represents failure

    Returns
    -------
    (int) the chosen arm
    """

    # calculate the empirical means and the total number of simulations that were ran
    totals = counts.sum(axis=1)
    successes = counts[:, 0]
    empirical_means = successes / totals
    total_counts = counts.sum()

    bonus = np.sqrt(2 * np.log(total_counts) / totals)
    return np.argmax(empirical_means + bonus)

counts = np.ones((K, 2))
softmax(counts)
```

    1

## Experimenting With Bandit Algorithms

In this section, we'll use our simulated data to experiment with our algorithms. To do this we'll also need a metric to calculate how well we are doing. Recall the absolute *best* we can do is to always pick the webpage (arm) with the largest click through rate (ctr). Denote this best arm's probability of $w_{opt}$. Our score should be relative to how well we would have done had we chosen the best arm from the beginning. This motivates the *total regret* of a strategy, defined as:

\begin{align}
R_T & = \sum_{t=1}^{T} \left( w_{opt} - w_{I(t)} \right) \nonumber \\
& = Tw_{opt} - \sum_{t=1}^{T} \;  w_{I(t)}
\end{align}

Where $T$ is the total number of samples in the experiment, $w_{I(t)}$ is the probability of obtaining the reward (getting clicked) of the chosen arm in the $t_{th}$ turn. A total regret of 0 means the strategy is attaining the best possible score. This is likely not possible, as initially our algorithm will often make the wrong choice. Ideally, a strategy's total regret should flatten as it learns the best bandit. (Mathematically, we achieve $w_{I(t)} = w_{opt}$ often)

We'll run the experiment and plot the cumulative regret of the three algorithms below:

```python
def run_bandit_algo(rewards, ctr, algo, **kwargs):
    """
    Run different types of bandit algorithms

    Parameters
    ----------
    rewards, ctr :
        Return value of the `generate_bernoulli_bandit_data` function

    algo : bandit function
        [epsilon_greedy, softmax, ucb]

    **kwargs :
        additional parameters to pass in to the algo

    Returns
    -------
    cum_regret : 1d ndarray, shape [n_simulations,]
        The total regret accumulated over the experiment, where the regret
        is measured by the maximum ctr - the chosen arm's ctr

    opt_arm_percentage : float
        The percentage of plays in which the optimal arm is pulled
    """
    n_simulations, K = rewards.shape

    # counts : success and failures for each arm where column 0 represents
    # success, 1 represents failure. Each arm's count is initialiated as 1
    # to ensure that each arm is played at least once, to prevent "cold start"
    # problem and 0 division in the beginning
    counts = np.ones((K, 2), dtype=np.int)
    regret = np.zeros(n_simulations)

    max_ctr_count = 0
    max_ctr = np.max(ctr)
    max_ctr_idx = np.argmax(ctr)

    for i in range(n_simulations):
        # 1. run the algorithm to obtain the arm that got pulled
        # 2. update the success / failure according to the generated rewards
        # 3. update the expected regret for each turn of the simulation
        # 4. if the arm that got pulled is the one with the opt ctr, increment this count
        arm = algo(counts, **kwargs)
        if rewards[i, arm] == 1:
            counts[arm, 0] += 1
        else:
            counts[arm, 1] += 1

        regret[i] = max_ctr - ctr[arm]
        if arm == max_ctr_idx:
            max_ctr_count += 1

    cum_regret = np.cumsum(regret)
    opt_arm_percentage = max_ctr_count / n_simulations
    return cum_regret, opt_arm_percentage
```

```python
def run_experiment(K, n_simulations, algorithms):
    """
    Run the bandit algorithm's simulation by the
    specified number of samples for simulation, the number of arms
    and the different version of algorithm

    Parameters
    ----------
    n_simulations : int
        the total number of turns in a simulation

    K : int
        the total number of arms

    algorithms : list of functions
        the list of bandit algorithms to simulate

    Returns
    -------
    ctr : float 1d-array, shape [K,]
        the randomly generated empirical click through rate for each arm

    algo_opt_arm_percentage : float list
        the percentage of simulations that chose the best arm

    algo_cum_regret : float 2d-array, shape [n_simulations, length of the algorithm]
        each column stores the cumulative regret for one algorithm

    fig : matplotlib figure
        the cumulative regret for each bandit algorithm
    """
    algo_opt_arm_percentage = []
    algo_cum_regret = np.zeros((n_simulations, len(algorithms)))

    fig = plt.figure(figsize=(10, 7))
    ctr, rewards = generate_bernoulli_bandit_data(n_simulations, K)

    for idx, algo in enumerate(algorithms):
        cum_regret, opt_arm_percentage = run_bandit_algo(rewards, ctr, algo=algo)
        algo_cum_regret[:, idx] = cum_regret
        algo_opt_arm_percentage.append(opt_arm_percentage)

        plt.semilogy(cum_regret, label=algo.__name__)

    plt.title('Simulated Bandit Performance for K = {}'.format(K))
    plt.ylabel('Cumulative Expected Regret')
    plt.xlabel('Round Index')
    plt.legend(loc='lower right')

    return ctr, algo_opt_arm_percentage, algo_cum_regret, fig
```

```python
# change default figure size and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

K = 5
n_simulations = 10000
algorithms = [epsilon_greedy, softmax, ucb]

np.random.seed(2345)
ctr, algo_opt_arm_percentage, algo_cum_regret, fig = run_experiment(K, n_simulations, algorithms)
plt.show()
print(ctr)
print(algo_opt_arm_percentage)
```

**Section Conclusion:** The cumulative expected regret plot from our experiment showed that all three different algorithms have converged (the cumulative expected regret gradually decreases to a steady level). And the **UCB** seems to be doing better than the other two algorithms in this limited horizon as the way to read the graph is the lower the better (the y-axis represents regrets).

# Bayesian Bandits

Next, we'll introduce a Bayesian method called **Thompson Sampling**. Recall that the problem we want to solve is the following. We came up with $K$ different variations of the webpage (e.g. different layout) and we wish to find the ones with the best click through rate (CTR), e.g. clicking to sign-up for the newsletter. Let's represent each CTR by $\theta_i$ - i.e., $\theta_i$ is the true probability that an individual user will click when they were shown with the $i_{th}$ webpage. It is important to note that we don't actually know what $\theta_i$ is - if we did, we could simply choose $i$ for which $\theta_i$ was largest and move on. We're simply pretending that we know in order to simulate the performance of the algorithm.

Using the Bayesian approach we will construct a prior probability distribution which represents our original belief about what the actual value of $\theta_i$, our ctr for the $i_{th}$ webpage is. The prior we'll use is the Beta distribution. Here's a quick recap of the distribution:

## Beta Distribution

The Beta distribution is very useful in Bayesian statistics. A random variable $X$ has a Beta distribution, with parameters $(\alpha, \beta)$, if its density function is:

\begin{align}
f_X(x | \; \alpha, \beta ) = \frac{ x^{(\alpha - 1)}(1-x)^{ (\beta - 1) } }{B(\alpha, \beta) }
\end{align}

where $B$ is the [Beta function](http://en.wikipedia.org/wiki/Beta_function) (hence the name). The random variable $X$ is only allowed in [0,1], making the Beta distribution a popular distribution for decimal values, probabilities and proportions. The values of $\alpha$ and $\beta$, both positive values, provide great flexibility in the shape of the distribution. Below we plot some Beta distributions with different $\alpha$ and $\beta$ values:

```python
plt.figure(figsize=(12, 5))

x = np.linspace(0.01, .99, 100)
params = [(2, 5), (1, 1), (5, 5), (20, 4)]

for a, b in params:
    y = beta.pdf(x, a, b)
    lines = plt.plot(x, y, label="(%.1f,%.1f)" % (a, b), lw=2)
    plt.fill_between(x, 0, y, alpha=0.2, color=lines[0].get_color())
    plt.autoscale(tight=True)

plt.legend(loc='upper left', title="(a,b)-parameters")
plt.show()
```

There are two important things to note about the Beta distribution:

The first is the presence of the flat distribution above, specified by parameters $(1,1)$. This is the Uniform distribution. Hence the Beta distribution is a generalization of the Uniform distribution.

The second is that there is an interesting connection between the Beta distribution and the Binomial distribution. Suppose we are interested in some unknown proportion or probability $p$. We assign a $\text{Beta}(\alpha, \beta)$ prior to $p$. We observe some data generated by a Binomial process, say $X \sim \text{Binomial}(N, p)$, with $p$ still unknown. Then our posterior *is again a Beta distribution*, i.e. $p | X \sim \text{Beta}( \alpha + X, \beta + N -X )$. Succinctly, one can relate the two by "a Beta prior with Binomial observations creates a Beta posterior".

In light of the above two paragraphs, if we start with a $\text{Beta}(1,1)$ prior on $p$ (which is a Uniform), observe data $X \sim \text{Binomial}(N, p)$, then our posterior is $\text{Beta}(1 + X, 1 + N - X)$.

## Thompson Sampling

So after assuming the priors on the probability of ctr for each webpage. To be explicit on the phrase "assuming the priors", we will assume that we're completely ignorant of these probabilities. So a very natural prior is the flat prior over 0 to 1, $\text{Beta}(\alpha=1,\beta=1)$. The algorithm then proceeds as follows:

For each turn:

1. Sample a random variable $X_i$ from the prior of arm $i$, for all $i$ ($K$ in total).
2. Select the arm with largest sample, i.e. select $i = \text{argmax}\; X_i$.
3. Observe the result of pulled arm $i$, and update your prior with that arm $i$.
4. Return to 1.

Like all the algorithms we've introduced before, **Thompson Sampling** suggests that we should not discard losers, but we should pick them at a decreasing rate as we gather confidence that there exist *better* webpages (arms). This follows because there is always a non-zero chance that a webpage with a lower ctr will get chosen, but the probability of this event decreases as we play more rounds.

```python
class BayesianBandit:
    """
    Thompson Sampling

    Parameters
    ----------
    K : int
        total number of arms

    prior_params : list of float length 2 tuple, default None, (optional)
        each element of the list is a tuple, where each tuple
        contains the alpha and beta parameter that represents the prior
        beta distribution for each arm. If not supplied
        it will assume that all arms's prior starts with an uniform distribution

    Attributes
    ----------
    trials, success : int 1d ndarray, shape [K,]
        stores the trials and success for each arm,
        e.g. trial = [ 1, 1 ] and success = [ 0, 1 ] means
        that both arm has been pulled once and arm 1 has generated
        the reward (clicked)
    """

    def __init__(self, K, prior_params=None):
        if prior_params:
            priors = namedtuple("priors", ["alpha", "beta"])
            prior = [priors(*p) for p in prior_params]
            self.alphas = np.array([p.alpha for p in prior])
            self.betas = np.array([p.beta  for p in prior])
        else:
            self.alphas = np.ones(K)
            self.betas = np.ones(K)

        self.trials = np.zeros(K, dtype=np.int)
        self.success = np.zeros(K, dtype=np.int)

    def get_recommendation(self):
        """
        for all arms, construct their beta distribution and
        draw a random sample from it, then return the arm
        with the maximum value random sample
        """
        theta = np.random.beta(self.alphas + self.success,
                               self.betas + self.trials - self.success)
        return np.argmax(theta)

    def update_result(self, arm, converted):
        """
        override the trials and success array, the success array
        will only be updated if it has generated a reward
        """
        self.trials[arm] += 1
        if converted:
            self.success[arm] += 1

        return self
```

```python
def experiment(T, ctr, prior_params=None):
    """
    run the experiment for Thompson Sampling,
    pass in ctr, the fixed ctr for each arm
    or K, the total number of arms to run the experiment,
    if K is supplied then it will be randomly generated

    Parameters
    ----------
    T : int
        number of simulation in an experiment

    ctr : float sequence, len = K (total number of arms)
        the empirical click through rate for each arm

    prior_params : list of float length 2 tuple, default None, (optional)
        each element of the list is a tuple, where each tuple
        contains the alpha and beta parameter that represents the prior
        beta distribution for each arm. If not supplied
        it will assume that all arms's prior starts with an uniform distribution

    Returns
    -------
    ctr : float sequence, len = K
        the supplied or the randomly generated ctr

    trials, success : int 2d ndarray, shape [T, K]
        trials and success recorded for each turn of the experiment

    alphas, betas : float 1d ndarray, shape [K,]
        the alpha and beta parameters for each arm
    """
    K = len(ctr)
    trials = np.zeros((T, K), dtype=np.int)
    success = np.zeros((T, K), dtype=np.int)

    bayes_bandit = BayesianBandit(K, prior_params)
    for t in range(T):
        arm = bayes_bandit.get_recommendation()
        converted = np.random.rand() < ctr[arm]
        bayes_bandit.update_result(arm, converted)
        trials[t] = bayes_bandit.trials
        success[t] = bayes_bandit.success

    return ctr, trials, success, bayes_bandit.alphas, bayes_bandit.betas
```

```python
def experiment_plot(ctr, trials, success):
    """
    Pass in the ctr, trials and success returned
    by the `experiment` function and plot
    the Cumulative Number of Turns For Each Arm and
    the CTR's Convergence Plot side by side
    """
    T, K = trials.shape
    n = np.arange(T) + 1
    fig = plt.figure(figsize=(14, 7))

    plt.subplot(121)
    for i in range(K):
        plt.loglog(n, trials[:, i], label="arm {}".format(i + 1))

    plt.legend(loc="upper left")
    plt.xlabel("Number of turns")
    plt.ylabel("Number of turns/arm")
    plt.title("Cumulative Number of Turns For Each Arm")

    plt.subplot(122)
    for i in range(K):
        plt.semilogx(n, np.zeros(T) + ctr[i], label="arm {}'s CTR".format(i + 1))

    plt.semilogx(n, (success[:, 0] + success[:, 1]) / n, label="CTR at turn t")

    plt.axis([1, T, 0, 1])
    plt.legend(loc="upper left")
    plt.xlabel("Number of turns")
    plt.ylabel("CTR")
    plt.title("CTR's Convergence Plot")
    return fig
```

```python
# number of simulation in an experiment
T = 10000

# the empirical click through rate for each arm
ctr = 0.25, 0.35

ctr, trials, success, alphas, betas = experiment(T=T, ctr=ctr)
trials
```

    array([[   1,    0],
           [   1,    1],
           [   1,    2],
           ...,
           [  44, 9954],
           [  44, 9955],
           [  44, 9956]])

```python
fig = experiment_plot(ctr, trials, success)
plt.show()
```

In our simulation, we gave the Bayesian bandit two webpages (arms) - one had a CTR of 0.25, the other had a CTR of 0.35. To start with, both webpages were displayed to the user with roughly equal probability. Over time, evidence accumulated that arm 2 was considerably better than arm 1. At this point the algorithm switched to displaying primarily webpage 1, and the overall CTR of the experiment converged to 0.35 (the optimal CTR).

We can also visualize our Beta distribution for each arms in different turns.

```python
def plot_beta_dist(ctr, trials, success, alphas, betas, turns):
    """
    Pass in the ctr, trials and success, alphas, betas returned
    by the `experiment` function and the number of turns
    and plot the beta distribution for all the arms in that turn
    """
    subplot_num = len(turns) / 2
    x = np.linspace(0.001, .999, 200)
    fig = plt.figure(figsize=(14, 7))

    for idx, turn in enumerate(turns):
        plt.subplot(subplot_num, 2, idx + 1)

        for i in range(len(ctr)):
            y = beta(alphas[i] + success[turn, i],
                     betas[i] + trials[turn, i] - success[turn, i]).pdf(x)
            line = plt.plot(x, y, lw=2, label="arm {}".format(i + 1))
            color = line[0].get_color()
            plt.fill_between(x, 0, y, alpha=0.2, color=color)
            plt.axvline(x=ctr[i], color=color, linestyle="--", lw=2)
            plt.title("Posteriors After {} turns".format(turn))
            plt.legend(loc="upper right")

    return fig
```

```python
turns = [1, 100, 1000, 9999]
posterior_fig = plot_beta_dist(ctr, trials, success, alphas, betas, turns)
plt.show()
```

As we can see, we started out with some prior Beta distributions that represents the our initial beliefs about possible ctr values for each arm. We then update these beliefs based on evidence by showing different webpages to other randomly chosen users and observing the ctr. After doing this for many number of turns, we incrementally narrow the width of each arm's Beta distribution. Meaning that as we gather more data, we become more confident about each arm's actual ctr.

Note that we don't really care how accurate we become about the inference of the hidden probabilities — for this problem we are more interested in becoming more confident in choosing the best arm. This is basically why at the end of experiment, arm 1's distribution is wider. The algorithm is ignorant about what that hidden probability might be, but we are reasonably confident that it is not the best, so the algorithm chooses to ignore it.

From the above, we can see that starting after 100 pulls, the majority of arm 2's distribution already leads the pack, hence the algorithm will almost always choose this arm. This is good, as this arm is indeed better.

## Notes On Bandit Testings

In world settings, we need to account for situations such as delayed batch update or delay feedback. In such cases, algorithms such as **Epsilon Greedy**, **Softmax**, **UCB** needs the distributions to be updated every single round to work properly. On the other hand, **Thompson Sampling** is more realistic as it relies on random samples, which will be different every time even if the distributions are each arm aren't updated for a while.

So, after gaining some knowledge on **bandit testings**. The question that comes into our head is that: "In general, when is it preferred over the classical **A/B testing**?"

### Short-term testing

> "Whenever you have a small amount of time for both exploration and exploitation, use a bandit testing."

**Bandit testing** are conducive for short tests for clear reasons – if you were to run a classic **A/B testing** you’d not even be able to enjoy the period of pure exploitation. Instead, **bandit testing** allow you to adjust in real time and send more traffic, more quickly, to the better variation. Here are some possible use cases:

- **Headlines:** News has a short life cycle. Why would you run **A/B testing** on a headline if by the time you learn which variation is best, the time where the answer is applicable is over?
- **Holiday Campaigns:** e.g. If you’re running tests on an ecommerce site for Black Friday, an A/B testing isn’t that practical – you might only be confident in the result at the end of the day. On the other hand, a **bandit testing** will drive more traffic to the better-performing variation – and that in turn can increase revenue.

### Long-term testing

Because **Bandit testing** automatically shift traffic to higher performing (at the time) variations, thus it is effective in long term (or ongoing) testing where you can set it and forget about it. e.g. Serving specific ads and content to user sets (targeting ads).

## Reference

- [Notes: Algorithms for the multi-armed bandit problem](http://www.cs.mcgill.ca/~vkules/bandits.pdf)
- [Blog: When to Run Bandit Tests Instead of A/B Tests](http://conversionxl.com/bandit-tests/)
- [Blog: Bayesian Bandits - optimizing click throughs with statistics](https://www.chrisstucchio.com/blog/2013/bayesian_bandit.html)
- [Blog: Balancing Earning with Learning: Bandits and Adaptive Optimization](http://conductrics.com/balancing-earning-with-learning-bandits-and-adaptive-optimization/)
- [Notebook: Bayesian Methods for Hackers Chapter 6](http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter6_Priorities/Ch6_Priors_PyMC2.ipynb)


# The Harmonic Oscillator Strikes Back

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week we continue our adventures with the harmonic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2} m \omega_0^2 {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

If the equilibrium value of the harmonic oscillator is not zero, then

$$ F=- \omega_0^2 m (x-x_{eq}), \text{ }  V(x) = {1 \over 2} m \omega_0^2 (x-x_{eq})^2$$

## 1. Harmonic oscillator from last time (with some better defined conditions)

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= -m \omega_0^2 (x-x_{eq}) $$

$$ a = - \omega_0^2 (x-x_{eq}) $$

$$ x(t)'' = - \omega_0^2 (x-x_{eq}) $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

This is already solved to remind you how we found these values

```python
import sympy as sym
sym.init_printing()
```

**Note** that this time we define some of the properties of the symbols. Namely, that the frequency is always positive and real and that the positions are always real

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
x(t),omega0
```

```python
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

```python
sol,sol.args[0],sol.args[1]
```

**Note** this time we define the initial positions and velocities as real

```python
x0,v0=sym.symbols("x_0,v_0",real=True)
ics=[sym.Eq(sol.args[1].subs(t, 0), x0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

### 1.1 Equation of motion for $x(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

### 1.2 Equation of motion for $p(t)$

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

## 2. Time average values for a harmonic oscillator

If we want to understand the average value of a time dependent observable, we need to solve the following integral

$${\left\langle {A(t)} \right\rangle}_t  = \begin{array}{*{20}{c}}
{\lim }\\
{\tau  \to 0}
\end{array}\frac{1}{\tau }\int\limits_0^\tau  {A(t)dt} $$

### 2.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
xfunc=full_sol.args[1]
xavet=(xfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
xavet
```

The computer does not always make the best choices the first time. If you treat each sum individually this is not a hard limit to do by hand. The computer is not smart. We can help it by inseting an `expand()` function in the statement

```python
xavet=(xfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
xavet
```

### 2.2 Excercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
p=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
pfunc=p.args[1]
pavet=(pfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
pavet
```

### 2.3 Exercise: Calculate the average kinetic energy of a harmonic oscillator

```python
#ke=p^2/2m
ke=pfunc**2/2*m
keavet=(ke.integrate((t,0,tau))/tau).limit(tau,sym.oo)
keavet
keavet=(ke.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
keavet
```

## 3.  Ensemble (Thermodynamic) Average values for a harmonic oscillator

If we want to understand the thermodynamics ensemble average value of an observable, we need to solve the following integral.

$${\left\langle {A(t)} \right\rangle}_{T}  = \frac{\int{A e^{-\beta H}dqdp}}{\int{e^{-\beta H}dqdp} } $$

You can think of this as a Temperature average instead of a time average.

Here $\beta=\frac{1}{k_B T}$ and the classical Hamiltonian, $H$ is

$$ H = \frac{p^2}{2 m} + V(q)$$

**Note** that the factors of $1/h$ found in the classical partition function cancel out when calculating average values

### 3.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

For a harmonic oscillator with equilibrium value $x_{eq}$, the Hamiltonian is
$$ H = \frac{p^2}{2 m} + \frac{1}{2} m \omega_0 (x-x_{eq})^2 $$

First we will calculate the partition function $\int{e^{-\beta H}dqdp}$

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z
```

Then we can calculate the numerator $\int{A e^{-\beta H}dqdp}$

```python
numx=sym.integrate(xT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numx
```

And now the average value

```python
xaveT=numx/z
xaveT
```

### 3.2 Exercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

After calculating the value, explain why you think you got this number

```python
anothanump=sym.integrate(pT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
anothanump

```

```python
paveT=anothanump/z
paveT
```

I think I got this number because the harmonic oscillator is symmetral, meaning it computes with a reflections, and all values for momenta will average out

### 3.3 Exercise: Calculate the average kinetic energy

The answer you get here is a well known result related to the energy equipartition theorem

```python
ken=pT**2/2*m
anothanumke=sym.integrate(ken*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
anothanumke

```

```python
keaveT=anothanumke/z
keaveT
```

# Back to the lecture

## 4. Exercise Verlet integrators

In this exercise we will write a routine to solve for the equations of motion for a hamonic oscillator.

Plot the positions and momenta (seprate plots) of the harmonic oscillator as a functions of time.

Calculate trajectories using the following methods:
1. Exact solution
2. Simple taylor series expansion
3. Predictor-corrector method
4. Verlet algorithm
5. Leapfrog algorithm
6. Velocity Verlet algorithm

```python
#Exact solution for position

fuller_sol = sym.simplify(full_sol.subs({x0:10, xeq:0 , v0:10, omega0:1}))
sym.plot(fuller_sol.rhs,(t,-10,10))
```

```python
#Exact solution for momentum

m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
Momenta=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
Momentaa = sym.simplify(Momenta.subs({x0:10, xeq:0 , v0:10, omega0:1, m:1}))
sym.plot(Momentaa.rhs,(t,-10,10))
```

```python
#Simple taylor series expansion

xt0=.1
t=.1
vt0=.1
xlist=[]
for i in range(0,1000):
    vt=vt0+(1/2)*t**2*(-(xt0))
    vt0=vt
    xt= xt0+(vt0)*(t)
    xt0=xt
    xlist.append(xt)
plt.plot(xlist)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Simple taylor series expansion')

```

*** I Skipped Number 3 ***

```python
# Verlet algorithm for position
#r(t+dt)=2r(t)-r(t-dt)+dt**2*a
#v(t)=(r(t+dt)-r(t-dt))/2dt

xt1=.1
xt2=.1
t=.1
a=.1
plist=[]
for i in range(0,100):
    vt=(xt2-xt1)/2*t
    xt1=xt2
    xt3=vt
    plist.append(vt)
plt.plot(xlist)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Verlet Algorithm for Position')
```

```python
# Verlet algorithm for momentum
#r(t+dt)=2r(t)-r(t-dt)+dt**2*a
#v(t)=(r(t+dt)-r(t-dt))/2dt

xt1=.1
xt2=.1
t=.1
a=.1
xlist=[]
for i in range(0,100):
    xt1=2*xt2-xt3+t**2*-xt2
    xt3=xt2
    xt2=xt1
    xlist.append(xt3)
plt.plot(xlist)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Verlet Algorithm for Position')
```

```python
#Leapfrog for position
#r(t+dt)=r(t)+dtv(t+1/2dt)
#v(t+1/2dt)=v(t-1/2dt)+dta(t)
#v(t)=1/2(v(t+1/2dt)+v(t-1/2dt))
xt1=1
t=.2
xtwohalf=.2
xlist=[]
for i in range(0,100):
    xtwhalf=xtwohalf+t*-xt1
    xtwohalf=xtwhalf
    xt2=xt1+t*xtwhalf
    xt1=xt2
    xlist.append(xt2)
plt.plot(xlist)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Leafrog Algorithm for Position')

```

```python
#Leapfrog for velocity
#r(t+dt)=r(t)+dtv(t+1/2dt)
#v(t+1/2dt)=v(t-1/2dt)+dta(t)
#v(t)=1/2(v(t+1/2dt)+v(t-1/2dt))
vt1=2
t=.2
vtwohalf=.2
vtwhalf=2
vlist=[]
for i in range(0,100):
    vt11=(1/2)*((vtwhalf)+(vtwohalf))
    vtwohalf=vt11
    vtwhalf=vtwohalf+t*-vt1
    vtwohalf=vtwhalf
    vt2=vt1+t*vtwhalf
    vt1=vt2
    vlist.append(vt11)
plt.plot(vlist)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Leafrog Algorithm for Velocity')
```

```python
#velocity verlet for position
#r(t+dt)=r(t)+dtv(t)+1/2dt^2a(t)
#v(t+dt)=v(t)+1/2dt(a(t)+a(t+dt))
#v(t+1/2dt)=v(t)+1/2dta(t)
#v(t+dt)=v(t+1/2dt)+1/2dta(t+dt)

dt=0.01
x1=0
v1=1
x=[]
for i in range(0,1000):
    x2=x1+v1*dt+1/2*-x1*dt**2
    x1=x2
    v2=v1+1/2*(-x2-x1)*dt
    v1=v2
    x.append(x2)
plt.plot(x)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Verlocity Verlet for Position')

```

```python
#velocity verlet for Velocity
#r(t+dt)=r(t)+dtv(t)+1/2dt^2a(t)
#v(t+dt)=v(t)+1/2dt(a(t)+a(t+dt))
#v(t+1/2dt)=v(t)+1/2dta(t)
#v(t+dt)=v(t+1/2dt)+1/2dta(t+dt)

xt1=1
dt=2
vt1=1
vell=[]
for i in range(0,10):
    vwhalf=vt1+1/2*dt*-xt1
    xt1=vwhalf
    vell.append(vwhalf)
plt.plot(vell)
plt.grid(True)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Verlocity Verlet for velocity')

```


# Welcome to the
```
   _  _____  ___  ___  _____________  ____  _  __
  / |/ / _ \/ _ \/ _ \/  _/_  __/ _ \/ __ \/ |/ /
 /    / , _/ // / // // /  / / / , _/ /_/ /    /
/_/|_/_/|_/____/____/___/ /_/ /_/|_|\____/_/|_/

```

<i>"Let's do some science."</i>

Wait, why's this a Jupyter notebook? Great question - I'm not sure.

Todo:
- Chamber assembly price
- distance travelled per period
- ICP skin depth
- [] Optimize number of turns to current
- [] enthalpy of vaporization

```python
%matplotlib nbagg

import matplotlib.pyplot as plt

from mendeleev import element
from thermo.chemical import Chemical
import sympy.physics.units as units
import scipy.constants as constants
amu = constants.physical_constants["atomic mass unit-kilogram relationship"][0]
import math
import sys
import numpy as np

sys.path.insert(0, '../../files/ionprinter/simulation/utilities/')
#import util

```

# Inputs

```python
#GLOBALS

print_species = element('Aluminum')

print_speed_g_h = 10.0 #g/hour

#convert to kg/s, kick CGS to the curb.
print_speed = print_speed_g_h/1000.0
print_speed /= 3600.0

number_of_bowties = 1024 #bowties/printer - an excellent unit

bowties_per_sector = 64

#hold or cut bowties

build_platform_distance = 0.03 #m

#BOWTIE

bowtie_nozzle_radius = 0.00033#m

# bowtie_hot_area = 10e-6 #m^2
bowtie_emissivity = 0.17
#graphite - ~0.95
#alumina - 0.4
#tungsten - ~0.04

bowtie_temperature = 1700.0 #Kelvin

#CHAMBER

ionization_chamber_radius = 0.003 #m
ionization_chamber_length = 0.01 #m

acceleration_gap = 0.00025 #m
acceleration_voltage = 100 #V
ionization_efficiency = 0.3

ICP_frequency = 1000000 #hz
ICP_peak_B = 0.05 #T

#IONIZATION E-BEAM

e_beam_energy = 100.0 #eV

#INJECTOR PLUMBING

plumbing_temperature = 1500

# import pandas as pd
# data = [[1, 2], [3, 4]]
# pd.DataFrame(data, columns=["Foo", "Bar"])

R_constant = constants.Boltzmann*constants.Avogadro

```

```python

```

```python

```

# Chamber vacuum level

PV=NRT to get chamber number density, then MFP.

```python
# vacuum_gas_atomic_radius = 65e-9 #nitrogen
# vacuum_gas_cross_section = math.pi*(chamber_gas_atomic_radius**2.0)
# vacuum_gas_pressure = 1e-4 #Pa

# chamber_number_density (vacuum_gas_pressure/(273.15))=N

# ionization_electron_mean_free_path = 1.0/(nozzle_number_density*)

# ionization_electrons_per_neutral = bowtie_nozzle_radius/ionization_electron_mean_free_path
```

# Neutral stuff.

```python
#Total n of atoms that must be emitted to achieve desired mass deposition per hour
atoms_per_second = print_speed/(print_species.mass*amu)

moles_per_second = atoms_per_second/constants.Avogadro

#Volume deposited per hour.
hourly_print_volume = ((print_speed*3600.0)/(print_species.density*1000.0))/1.0e-6

total_beam_current = atoms_per_second*constants.elementary_charge
per_chamber_beam_current = total_beam_current/number_of_bowties

print("Atoms per second: \nper emitter %0.5g atoms/s \ntotal %0.5g atoms/s\n" % ((atoms_per_second/number_of_bowties),
                                           atoms_per_second))
print("Total moles per second: %0.5g M/s" % moles_per_second)
print("Deposited volume: %0.5g cm^3/h" % hourly_print_volume)
print("Total beam current: %0.5g Amps" % total_beam_current)
print("Per-emitter current: %0.5g Amps" % per_chamber_beam_current)

#I find it infinitely amusing that "40 amps of aluminum" is "40 grams per second of aluminum" within 0.6%
```

    Atoms per second:
    per emitter 6.0545e+16 atoms/s
    total 6.1999e+19 atoms/s

    Total moles per second: 0.00010295 M/s
    Deposited volume: 3.7052 cm^3/h
    Total beam current: 9.9333 Amps
    Per-emitter current: 0.0097005 Amps

## RMS neutral velocity

Determine RMS neutral gas velocity at a certain temperature:

$$\sqrt{\frac{3RT}{m}}$$

```python
bowtie_rms_neutral_velocity = math.sqrt((3*constants.Boltzmann*bowtie_temperature)/(print_species.mass*amu))
bowtie_rms_energy = 0.5*print_species.mass*amu*(bowtie_rms_neutral_velocity**2.0)
print("RMS particle velocity: %0.5g m/s" % bowtie_rms_neutral_velocity)
print("RMS particle energy: %0.5g eV" % (bowtie_rms_energy/constants.electron_volt))
```

    RMS particle velocity: 1253.6 m/s
    RMS particle energy: 0.21974 eV

## Bowtie nozzle gas pressure

Determine approximate neutral gas pressure in a virtual volume "1 thermal velocity" long via a simple

$$\text{PV}=\text{nRT}$$

(I'm not totally sure that you can do this.)

Also approximately determine mean free path with the neutral Van Der Waals radius.

```python

#volume nozzle area times 1 thermal neutral injection velocity long
nozzle_virtual_volume = (math.pi*(bowtie_nozzle_radius**2))*bowtie_rms_neutral_velocity

bowtie_nozzle_pressure = ((moles_per_second/number_of_bowties)*R_constant*bowtie_temperature)/(nozzle_virtual_volume)

mean_free_path = 1.0/(((atoms_per_second/number_of_bowties)/nozzle_virtual_volume)*math.pi*((print_species.vdw_radius*1e-12)**2))

nozzle_number_density = (atoms_per_second/nozzle_virtual_volume)

print("Bowtie nozzle pressure: %0.5g Pa" % bowtie_nozzle_pressure)
print("Approximate nozzle mean free path: %0.5g m" % mean_free_path)

#number density of atoms within that virtual volume - useful for DSMC stuff
print("nrho: %0.5g" % (nozzle_number_density))

print("Knudsen number: local %0.5g, global %0.5g" % ((mean_free_path/bowtie_nozzle_radius),
                                                     (mean_free_path/build_platform_distance)))

print("Worst-case space charge density: %0.5g" % ((atoms_per_second/number_of_bowties)/nozzle_virtual_volume))
```

    Bowtie nozzle pressure: 3.3133 Pa
    Approximate nozzle mean free path: 0.066601 m
    nrho: 1.4456e+23
    Knudsen number: local 201.82, global 2.22
    Worst-case space charge density: 1.4117e+20

## Actual vapor pressure

Now a rough approximation of the actual print species vapor pressure one can expect to find at that temperature:

(likely off by at least 50% - https://www.iap.tuwien.ac.at/www/surface/vapor_pressure is a more accurate source)

```python
vapor_pressure = Chemical(print_species.name).VaporPressure.calculate(bowtie_temperature,"BOILING_CRITICAL") # takes temp in K, returns pressure in P
print("Approx. species vapor pressure: %0.5g Pa" % vapor_pressure)
```

    Approx. species vapor pressure: 36.541 Pa

#### Bowtie hot area

```python
bowtie_radius = 0.0005
bowtie_hot_area = 0.01 * math.pi * (2.0*bowtie_radius)
```

```python
plumbing_radius = 0.0005
plumbing_hot_area = 0.0 * math.pi * (2.0*bowtie_radius)
plumbing_temperature = 1500
```

## Bowtie radiation power loss

Bowtie IR/Vis radiation power by the Stefan-Boltzmann law:

```python
per_bowtie_power = (bowtie_hot_area * bowtie_emissivity * constants.Stefan_Boltzmann * (bowtie_temperature**4.0))
total_bowtie_power = per_bowtie_power*number_of_bowties

print("Bowtie power: per %0.5g W, total %0.5g W" % (per_bowtie_power,total_bowtie_power))

per_plumbing_power = (plumbing_hot_area * bowtie_emissivity * constants.Stefan_Boltzmann * (plumbing_temperature**4.0))
total_plumbing_power = per_plumbing_power*number_of_bowties

print("Plumbing power: per %0.5g W, total %0.5g W" % (per_plumbing_power,total_plumbing_power))

```

    Bowtie power: per 2.5293 W, total 2590 W
    Plumbing power: per 0 W, total 0 W

```python

```

#### Multi-layer insulation

From Wikipedia (https://en.wikipedia.org/wiki/Multi-layer_insulation):

Simplified two-flux method:

$ Q=UA\Delta T$

$ U = 4 \sigma T_{average}^3 \frac{1}{N(2/\epsilon-1)+1}$

Where $\sigma$ is the Stefan-Boltzmann constant, Delta T is the temperature difference between the interior and exterior layers, and T_average is the average temperature of all the layers. This breaks if the layer temperature difference is enormous.

https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20080013560.pdf

For comparison, alumina silicate insulation is typically 0.12 W/(m K).

```python

```

```python
Thot = 1600.0+273.15 #Kelvin
Tcold = 400.0 #Kelvin
Tave = (Thot + Tcold) / 2.0 #Not accurate
area = 0.01*0.1*6 #m^2
area = (math.pi*(0.03**2.0))-(math.pi*(0.029**2.0))*0.1 #m^2
N = 3
emissivity = 0.5

Q = area*(Thot - Tcold)*(Tave**3.0)*4.0*constants.Stefan_Boltzmann*(1.0/(N*((2.0/emissivity) - 1.0)+1.0))
Q
#area*(Thot - Tcold)*(0.12/0.02)
```

    125.74746335260791

```python

```

```python

```

# Acceleration

```python
acceleration_velocity = math.sqrt((2.0*acceleration_voltage*constants.electron_volt)/(print_species.mass*amu))

print("Accel. vel: %0.5g m/s" % acceleration_velocity)
print("Accel. power per chamber: %0.5g W, total %0.5g W " % (acceleration_voltage*per_chamber_beam_current
                                                        ,acceleration_voltage*total_beam_current))

```

    Accel. vel: 26743 m/s
    Accel. power per chamber: 0.97005 W, total 993.33 W

# Ionization

### Power
Simple first-ionization power consumption estimate (ionization energy of species * 100):

```python
per_chamber_ionization_power = ((atoms_per_second/number_of_bowties)*print_species.ionenergies[1]*constants.electron_volt)

total_ionization_power = per_chamber_ionization_power*number_of_bowties

ionization_chamber_area = math.pi*(ionization_chamber_radius**2.0)

print("Ionization power: per ~%0.5g W, total %0.5g W" % (per_chamber_ionization_power,total_ionization_power))
```

    Ionization power: per ~0.058065 W, total 59.458 W

### E-beam impact

An equation that accurately describes ionization cross section in relation to electron energy does not appear to have been developed.

Here's a paper from antiquity that describes the situation quite well.

Lotz, Wolfgang. "Electron-impact ionization cross-sections for atoms up to Z= 108." Zeitschrift für Physik A Hadrons and nuclei 232.2 (1970): 101-107.

https://link.springer.com/content/pdf/10.1007/BF01393132.pdf

These values can be assumed to be half an order of magnitude off.

If we assume a transverse beam of electrons,

$$ \lambda=\frac{1}{\rho\sigma}$$

where $\sigma$ is the collision cross section, $\rho$ is the local number density (neutral atoms/m^3), and $\lambda$ is the electron MFP.

We can multiply this MFP by the beam diameter (simplifying to a square beam, of course) to get the odds that a certain electron will collide with a particle in the beam.

This is vastly oversimplified, of course - we're ignoring the townsend secondaries entirely, second- and third- ionizations are likely, and there will probably be some weird space charge compensation effects from the nearby ionized particles - but we've got E-beam current to spare.

A measurement of the ionization cross-section
of helium ions by electron impact

THE IONIZATION OF HELIUM, NEON,
AND ARGON BY ELECTRON IMPACT

Todo: calculate values per-element based on electron configuration

An even better reference: @kim2005electron, a nist database

### First ionization

```python
impact_cross_section = 3.0e-20 #m^2
#value for aluminum approx 3.0e-20

ionization_electron_mean_free_path = 1.0/(nozzle_number_density*impact_cross_section)

ionization_electrons_per_neutral = ionization_electron_mean_free_path/bowtie_nozzle_radius

print("Ionization mean-free-path ~%0.5gm" % (ionization_electron_mean_free_path))
print("Required e-beam to heavy-beam factor for full ionization: ~%0.5gx " % (ionization_electrons_per_neutral))
```

    Ionization mean-free-path ~0.00023059m
    Required e-beam to heavy-beam factor for full ionization: ~0.69876x

Clearly the e-beam to heavy-beam ratio can't be < 1 unless secondaries are included, so this is an underestimate.

### Second ionization

"Electron-Impact Ionization Cross-Sections and Ionization Rate Coefficients for Atoms and Ions" http://adsbit.harvard.edu//full/1967ApJS...14..207L/0000211.000.html

The second ionization cross sections seem to

```python
# second_impact_cross_section = 3.0e-20 #m^2

# ionization_electron_mean_free_path = 1.0/(nozzle_number_density*impact_cross_section)

# ionization_electrons_per_neutral = bowtie_nozzle_radius/ionization_electron_mean_free_path

# print("Ionization mean-free-path ~%0.5gm" % (ionization_electron_mean_free_path))
# print("Required e-beam to heavy-beam factor: ~%0.5gx " % (ionization_electrons_per_neutral))
```

(old, ignore) ICP acceleration field, (line integral of Maxwell's second equation, eq. 4.5-3 of "Fundamentals"):

```python
# (-(1.0j*2.0*math.pi*ICP_ionization_frequency)/2.0)*
```

### Required hot cathode filament area

```python

# CATHODE

cathode_power_consumption = 17.5 #"efficiency", watts per amp!
cathode_wire_diameter = 0.0001 #m

specific_emission = 4 #amps/cm^2
specific_emission /= 0.0001 # convert to A/mm^2

required_area = per_chamber_beam_current/specific_emission
required_area = required_area * 1000000.0 #mm^2
required_area
print("The ionization e-beam will require a cathode area of {}")

```

    The ionization e-beam will require a cathode area of {}

# Ion beam deflection / scattering by e-beam collision:

Basic momentum transfer upper bound. Ignoring the energy of ejected secondary electron and all other important factors

```python
heavy_beam_collision_delta_v = ((math.sqrt((2.0*e_beam_energy*constants.electron_volt)/constants.electron_mass))*constants.electron_mass)/(print_species.mass*amu)
heavy_beam_deflection_angle = math.degrees(math.atan(heavy_beam_collision_delta_v/bowtie_rms_neutral_velocity))

print("Heavy beam will have up to <%0.5g m/s added" % (heavy_beam_collision_delta_v))
print("Heavy beam will be deflected by <%0.5g degrees" % (heavy_beam_deflection_angle))
```

    Heavy beam will have up to <120.59 m/s added
    Heavy beam will be deflected by <5.4944 degrees

```python

```

 ## Assembly

 Charge assembly energy, circular approximation (eq. 8.7 FLP Vol II):

 TODO: add cylindrical eq. - this is horribly inaccurate, but at least it's an upper bound.

 FIXME:

 Total charge:
 $$ Q_T = \text{pL}\pi\text{R}^2$$

```python
ionization_chamber_volume = (math.pi*(ionization_chamber_radius**2))*bowtie_rms_neutral_velocity

per_chamber_assembled_energy = (3.0/5.0)*(((atoms_per_second/number_of_bowties)* \
                                 (ionization_chamber_radius/bowtie_rms_neutral_velocity)*constants.e)**2.0) \
                                /(4.0*math.pi*constants.epsilon_0*ionization_chamber_radius)*(bowtie_rms_neutral_velocity/ionization_chamber_radius)

print("Charge assembly power: per ~%0.5g W, total %0.5g W" % (per_chamber_assembled_energy, per_chamber_assembled_energy*number_of_bowties))
```

    Charge assembly power: per ~404.77 W, total 4.1448e+05 W

Energy to form one sphere of charge:

$$ U=(3/5)\left(\frac{\left(\frac{\text{I C}}{\frac{v}{2R}}\right)^2}{4\pi\epsilon_0 R}\right) $$

simplifies to

$$ (3/5)(Q^2 R)/(π v^2 ϵ) $$

Again from http://www.feynmanlectures.caltech.edu/II_08.html

```
I = beam current, amps
C = Coulomb's constant
v = particle velocity, post-acceleration
R = final beam radius
```

```python
#Take two:
#that can't be right, it's independent of beam radius
#which is absurd
minimum_beam_radius = 0.001
per_chamber_assembled_energy = (3.0/5.0)*(((constants.e*(atoms_per_second/number_of_bowties))**2.0)*minimum_beam_radius)/(math.pi*(bowtie_rms_neutral_velocity**2.0)*constants.epsilon_0)*(bowtie_rms_neutral_velocity/minimum_beam_radius)
assembled_beam_energy_per_particle = ((per_chamber_assembled_energy/(atoms_per_second/number_of_bowties))/constants.electron_volt)
per_chamber_assembled_energy
assembled_beam_energy_per_particle
```

    166907.56692544578

Take 3:
Dividing the minimum beam volume by each sphere volume, then multiplying by the beam charge per second:

$$(((electron charge * (3v/4r)*(total beam charge))^2)/R) * (1/(4pi \epsilon_0)) * (3/5) $$

```python
per_chamber_assembled_energy = (3.0/5.0)*(((((constants.e*(atoms_per_second/number_of_bowties))*((3.0*bowtie_rms_neutral_velocity)/(4.0*minimum_beam_radius)))**2.0)/minimum_beam_radius)/(4.0*math.pi*constants.epsilon_0))*(bowtie_rms_neutral_velocity/minimum_beam_radius)
per_chamber_assembled_energy

```

    5.623487180129447e+26

Moayedi, Seyed Kamran, Mansoureh Shafabakhsh, and Farkhondeh Fathi. "Analytical calculation of stored electrostatic energy per unit length for an infinite charged line and an infinitely long cylinder in the framework of Born-Infeld electrostatics." Advances in High Energy Physics 2015 (2015).

Another cylindrical approximation, from the last equation on

https://nptel.ac.in/courses/115101005/downloads/lectures-doc/Lecture-10.pdf

```python
linear_charge_density = ((atoms_per_second/number_of_bowties)*constants.e)/acceleration_velocity
per_chamber_assembled_energy = ((linear_charge_density**2.0)/(16.0*math.pi*constants.epsilon_0))*acceleration_velocity
per_chamber_assembled_energy*number_of_bowties
assembled_beam_energy_per_particle = ((per_chamber_assembled_energy/(atoms_per_second/number_of_bowties))/constants.electron_volt)
assembled_beam_energy_per_particle
```

    815.0068412265659

Electron-heavy space charge compensation test:

Complete compensation requires that the two beams have equal linear charge densities.

$$ \frac{I_e e^-}{\sqrt{2E_e/m_e}} = \frac{I_b q_b}{\sqrt{2E_b/m_b}} $$

The heavy beam current and energy are quite fixed; the only free variables are the E-beam current and energy.

```python

```

# Child-Langmuir Ion Current Limit

$$ \text{I} = \frac{4}{9}\epsilon_0 \left(\frac{-2q}{m} \right) ^{0.5} \frac{\text{V}^{1.5}}{ \text{D}^2} \text{A} $$

```
I = beam current, amps
V = acceleration voltage, volts
q = ion charge, C
m = ion mass, kg
D = gap between acceleration plates, m
A = area

```

This value really isn't particularly helpful.

Three-dimensional Child–Langmuir law for uniform hot electron emission

Suggested reading:

Fundamentals of Electric Propulsion, "Basic plasma physics". ../references/Fundamentals.pdf

The Child-Langmuir Law, Stanford EDU, Lucas et al [1]

[2]

[1]: https://web.stanford.edu/~ajlucas/The%20Child-Langmuir%20Law.pdf

[2]: http://www.physics.csbsju.edu/370/thermionic.pdf

"A simple physical derivation of Child–Langmuir space-charge-limited
emission using vacuum capacitance"

https://arxiv.org/pdf/1401.3951.pdf

Three-dimensional Child–Langmuir law for uniform hot electron emission

```python
def child_langmuir_current(voltage, charge, mass, gap, area):
    return (4.0/9.0)*constants.epsilon_0*(((2.0*charge/(mass))**0.5)*((voltage**1.5)/(gap**2.0))) * area

per_emitter_CL_ion_current = child_langmuir_current(acceleration_voltage,constants.e,(print_species.mass*amu),acceleration_gap,ionization_chamber_area)

print("Child-langmuir ion current: %0.5g A/chamber" % per_emitter_CL_ion_current)
print("Total CL current: %0.5g A" % (per_emitter_CL_ion_current*number_of_bowties))

```

    Child-langmuir ion current: 0.0047609 A/chamber
    Total CL current: 4.8752 A

2d ion

Lau, Y. Y. "Simple theory for the two-dimensional Child-Langmuir law." Physical review letters 87.27 (2001): 278301.

# Space charge

```python
beam_exit_velocity = bowtie_rms_neutral_velocity+acceleration_velocity

focus_field = util.scharge_efield(per_chamber_beam_current,beam_exit_velocity,ionization_chamber_radius)
print("Required focus field: %0.5g V/m" % focus_field)

# #sine focus voltage equation from that Klystron paper would be cool
# print("Required focus voltage: ~%0.5g V" % (focus_field*(acceleration_gap)))

```

```python
#todo: 2d scharge from
```

# Warp acceleration gap sim

Removed.

This required a modification to warp - line 28 of warp.py needs to have
```
try:
    __IPYTHON__
    pass
except NameError:
    warpoptions.parse_args()
```

# Recombination

## Child-Langmuir recombination current

```python
per_chamber_CL_electron_current = child_langmuir_current(5,constants.e,constants.electron_mass,acceleration_gap,ionization_chamber_area)

print("Child-Langmuir electron current: %0.5g A/bowtie" % per_chamber_CL_electron_current )
print("Total CL e current: %0.5g A" % (per_chamber_CL_electron_current*number_of_bowties))

```

    Child-Langmuir electron current: 0.011805 A/bowtie
    Total CL e current: 12.088 A

https://casper.ssl.berkeley.edu/astrobaki/index.php/Recombination_Coefficients

$${\textrm{# of recombinations}\over \textrm{volume} \cdot \textrm{time} }= n_+n_e{\sigma v}$$

Where `v` is relative velocity of the two particles, and n+ and ne are the number of electrons and

The capture cross section for thermal electrons (cm^2, as given by @ biondi1949measurement, page 5) is as follows:

$$\sigma = 8.64\left(\frac{Z_{\text{effective}}^{8/5}e^4}{\text{m}_e^2 c^{(6/5)}v^{(14/5)}}\right)$$

> where a- is the capture cross section; Ze is the effective charge of the ion; e and m are the electron charge and mass, respectively, and c is the velocity of light.

> Using the quantum mechanical model of the helium ion and the
Thomas-Fermi model for the neon and argon ions, we may calculate the effective ion
charge Ze at the distance of closest approach of the electron to the ion. For thermal
electrons, the value of Ze for helium is approximately 2; for neon, it lies between 8
and 9; and for argon, between 11 and 13. Since most of the energy is radiated while the
electron is close to the ion, the use of these values of Ze in Eq. (1) introduces only a
small error in the calculation.

Even the best models of recombination seem to be quite inaccurate: theory often differs with reality by between 10 to 1000x. This will be a lower bound.

I'm not sure whether the velocities are supposed to be m/s or cm/s.

```python
Z_e = 2.0 #depends on print species
# differential_energy = 0.00001
differential_velocity = 1
# differential_velocity = math.sqrt((2.0*differential_energy*constants.electron_volt)/(print_species.mass*amu))
# v matching of 1 part in 1000 seems achievable - just a guess though
# matching will be limited by ion and electron thermal velocity

#cm^2
cross_section = 8.64*(((Z_e**(8.0/5.0))*(constants.e**4.0))/((constants.electron_mass**2.0)*(constants.c**(6.0/5.0))*(differential_velocity**(14.0/5.0))))
# cross_section *= 0.0001 #convert to m^2
cross_section

recombination_electron_mean_free_path = 1.0/(nozzle_number_density*cross_section)

recombination_region_length = 0.001 #mm

recombination_electrons_per_neutral = recombination_electron_mean_free_path/recombination_region_length

print("Ionization mean-free-path ~%0.5gm" % (recombination_electron_mean_free_path))
print("Required e-beam to heavy-beam factor: ~%0.5gx" % (recombination_electrons_per_neutral))
```

    Ionization mean-free-path ~4.9445m
    Required e-beam to heavy-beam factor: ~4944.5x

```python
Z_e = 2.0 #depends on print species
# differential_energy = 0.00001
differential_velocity = 0.1
# differential_velocity = math.sqrt((2.0*differential_energy*constants.electron_volt)/(print_species.mass*amu))
# v matching of 1 part in 1000 seems achievable - just a guess though
# matching will be limited by ion and electron thermal velocity

#cm^2
cross_section = 8.64*(((Z_e**(8.0/5.0))*(constants.e**4.0))/((constants.electron_mass**2.0)*(constants.c**(6.0/5.0))*(differential_velocity**(14.0/5.0))))
# cross_section *= 0.0001 #convert to m^2
cross_section

recombination_electron_mean_free_path = 1.0/(nozzle_number_density*cross_section)

recombination_electrons_per_neutral = recombination_electron_mean_free_path/bowtie_nozzle_radius

print("Ionization mean-free-path ~%0.5gm" % (recombination_electron_mean_free_path))
print("Required e-beam to heavy-beam factor: ~%0.5gx" % (recombination_electrons_per_neutral))
```

    Ionization mean-free-path ~0.0078365m
    Required e-beam to heavy-beam factor: ~23.747x

## Hot cathode

```python

```

# Part heating

The beam heat flux will cause a temperature gradient from the chilled build platform to the extreme end of the printed part. Assuming a constant cross section, the peak temperature will be

$$ \frac{\text{(P/A)}}{k}L = \Delta \text{T} \\ \Delta \text{T} + \text{T}_i = T $$

```
P = Power applied at surface, W
A = Area of the part, m^2
k = Thermal conductivity of material, W/m-K
L = Length, M
T = Change in temperature across material, K
Ti = Build platform temperature
```

Ionization power is likely emitted in the form of photons at recombination, and so should be neglected.

http://web.mit.edu/16.unified/www/SPRING/propulsion/notes/node116.html

TODO:
- Add radiation here. A bit tricky, since the temperature's not constant.
- Add some kind of duty cycle approximation?
- cool graph

```python
print_area = 0.01*0.1 #m^2
printed_object_height = 0.03 #m
build_platform_temperature = 200.0 #Kelvin - likely liquid-cooled

printed_emissivity = 0.05
beam_power = (acceleration_voltage*total_beam_current)

part_peak_temperature = (((beam_power/print_area)/print_species.thermal_conductivity)*printed_object_height)+build_platform_temperature
print("Part peak temperature: %0.5g K" % (part_peak_temperature))

# per_bowtie_power = (bowtie_hot_area * printed_emissivity * constants.Stefan_Boltzmann * (bowtie_temperature**4.0))

```

    Part peak temperature: 325.74 K

Modulated beam capacitive coupling - equation 6.4 @ shafer1992beam

```python
plate_width = 0.002
plate_length = 0.002
plate_capacitance = 10e-12
beam_modulation = 0.0001 / (1/100.0)
beam_velocity = 40000

voltage_coupled = ((plate_width*plate_length)/(2.0*math.pi*plate_capacitance)) * (beam_modulation/beam_velocity)
voltage_coupled
```

    0.015915494309189534

# A final tally

```python
total_power_consumption = total_bowtie_power + (acceleration_voltage*total_beam_current) + total_ionization_power
print("Total power consumption: %0.5g W" % (total_power_consumption))

single_chamber_assembly_area = math.pi * (0.005**2.0)

print("Print head area: %0.5g m^2 (a square %0.5g m to the side)" % ((single_chamber_assembly_area*number_of_bowties),((single_chamber_assembly_area*number_of_bowties)**0.5)))

print("Allowable power consumption per bowtie")
```

    Total power consumption: 3642.8 W
    Print head area: 0.080425 m^2 (a square 0.28359 m to the side)
    Allowable power consumption per bowtie

```python

```

```python
#economics!

cost_per_watt = 0.2/1000.0 #20 cents per kilowatt-hour

consumable_cost = 200.0/1000.0 #dollars per 1000 hours

material_cost = 20.0/1000.0 #dollars per gram

joules_per_gram = total_power_consumption/print_speed_g_h

print("Print cost: %0.5g J/g, $%0.5g/100g" % ((joules_per_gram),(cost_per_watt*joules_per_gram*100.0+material_cost+consumable_cost)))

```

    Print cost: 364.28 J/g, $7.5056/100g

```python

```


# Wealth Distribution Dynamics

## Contents

- [Wealth Distribution Dynamics](#Wealth-Distribution-Dynamics)
  - [Overview](#Overview)
  - [Lorenz Curves and the Gini Coefficient](#Lorenz-Curves-and-the-Gini-Coefficient)
  - [A Model of Wealth Dynamics](#A-Model-of-Wealth-Dynamics)
  - [Implementation](#Implementation)
  - [Applications](#Applications)
  - [Exercises](#Exercises)

In addition to what’s in Anaconda, this lecture will need the following libraries:

```python
!pip install --upgrade quantecon
```

    Requirement already up-to-date: quantecon in c:\users\harve\appdata\roaming\python\python36\site-packages (0.4.6)
    Requirement already satisfied, skipping upgrade: requests in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from quantecon) (2.22.0)
    Requirement already satisfied, skipping upgrade: numpy in c:\users\harve\appdata\roaming\python\python36\site-packages (from quantecon) (1.17.4)
    Requirement already satisfied, skipping upgrade: scipy>=1.0.0 in c:\users\harve\appdata\roaming\python\python36\site-packages (from quantecon) (1.2.0)
    Requirement already satisfied, skipping upgrade: numba>=0.38 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from quantecon) (0.38.0)
    Requirement already satisfied, skipping upgrade: sympy in c:\users\harve\appdata\roaming\python\python36\site-packages (from quantecon) (1.3)
    Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from requests->quantecon) (2019.9.11)
    Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from requests->quantecon) (1.24.2)
    Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from requests->quantecon) (2.8)
    Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from requests->quantecon) (3.0.4)
    Requirement already satisfied, skipping upgrade: llvmlite>=0.23.0dev0 in c:\program files (x86)\microsoft visual studio\shared\anaconda3_64\lib\site-packages (from numba>=0.38->quantecon) (0.23.2+1.g67a89c7)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in c:\users\harve\appdata\roaming\python\python36\site-packages (from sympy->quantecon) (1.1.0)

## Overview

This notebook gives an introduction to wealth distribution dynamics, with a
focus on

- modeling and computing the wealth distribution via simulation,
- measures of inequality such as the Lorenz curve and Gini coefficient, and
- how inequality is affected by the properties of wage income and returns on assets.

One interesting property of the wealth distribution we discuss is Pareto
tails.

The wealth distribution in many countries exhibits a Pareto tail

- See [this lecture](heavy_tails.ipynb) for a definition.
- For a review of the empirical evidence, see, for example, [[BB18]](zreferences.ipynb#benhabib2018skewed).

This is consistent with high concentration of wealth amongst the richest households.

It also gives us a way to quantify such concentration, in terms of the tail index.

One question of interest is whether or not we can replicate Pareto tails from a relatively simple model.

### A Note on Assumptions

The evolution of wealth for any given household depends on their
savings behavior.

Modeling such behavior will form an important part of this lecture series.

However, in this particular lecture, we will be content with rather ad hoc (but plausible) savings rules.

We do this to more easily explore the implications of different specifications of income dynamics and investment returns.

At the same time, all of the techniques discussed here can be plugged into models that use optimization to obtain savings rules.

We will use the following imports.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import quantecon as qe
from numba import njit, jitclass, float64, prange
```

## Lorenz Curves and the Gini Coefficient

Before we investigate wealth dynamics, we briefly review some measures of
inequality.

### Lorenz Curves

One popular graphical measure of inequality is the [Lorenz curve](https://en.wikipedia.org/wiki/Lorenz_curve).

The package [QuantEcon.py](https://github.com/QuantEcon/QuantEcon.py), already imported above, contains a function to compute Lorenz curves.

To illustrate, suppose that

```python
n = 10_000                      # size of sample
w = np.exp(np.random.randn(n))  # lognormal draws
```

is data representing the wealth of 10,000 households.

We can compute and plot the Lorenz curve as follows:

```python
f_vals, l_vals = qe.lorenz_curve(w)

fig, ax = plt.subplots()
ax.plot(f_vals, l_vals, label='Lorenz curve, lognormal sample')
ax.plot(f_vals, f_vals, label='Lorenz curve, equality')
ax.legend()
plt.show()
```

This curve can be understood as follows: if point $ (x,y) $ lies on the curve, it means that, collectively, the bottom $ (100x)\% $ of the population holds $ (100y)\% $ of the wealth.

The “equality” line is the 45 degree line (which might not be exactly 45
degrees in the figure, depending on the aspect ratio).

A sample that produces this line exhibits perfect equality.

The other line in the figure is the Lorenz curve for the lognormal sample, which deviates significantly from perfect equality.

For example, the bottom 80% of the population holds around 40% of total wealth.

Here is another example, which shows how the Lorenz curve shifts as the
underlying distribution changes.

We generate 10,000 observations using the Pareto distribution with a range of
parameters, and then compute the Lorenz curve corresponding to each set of
observations.

```python
a_vals = (1, 2, 5)              # Pareto tail index
n = 10_000                      # size of each sample
fig, ax = plt.subplots()
for a in a_vals:
    u = np.random.uniform(size=n)
    y = u**(-1/a)               # distributed as Pareto with tail index a
    f_vals, l_vals = qe.lorenz_curve(y)
    ax.plot(f_vals, l_vals, label=f'$a = {a}$')
ax.plot(f_vals, f_vals, label='equality')
ax.legend()
plt.show()
```

You can see that, as the tail parameter of the Pareto distribution increases, inequality decreases.

This is to be expected, because a higher tail index implies less weight in the tail of the Pareto distribution.

### The Gini Coefficient

The definition and interpretation of the Gini coefficient can be found on the corresponding [Wikipedia page](https://en.wikipedia.org/wiki/Gini_coefficient).

A value of 0 indicates perfect equality (corresponding the case where the
Lorenz curve matches the 45 degree line) and a value of 1 indicates complete
inequality (all wealth held by the richest household).

The [QuantEcon.py](https://github.com/QuantEcon/QuantEcon.py) library contains a function to calculate the Gini coefficient.

We can test it on the Weibull distribution with parameter $ a $, where the Gini coefficient is known to be

$$
G = 1 - 2^{-1/a}
$$

Let’s see if the Gini coefficient computed from a simulated sample matches
this at each fixed value of $ a $.

```python
a_vals = range(1, 20)
ginis = []
ginis_theoretical = []
n = 100

fig, ax = plt.subplots()
for a in a_vals:
    y = np.random.weibull(a, size=n)
    ginis.append(qe.gini_coefficient(y))
    ginis_theoretical.append(1 - 2**(-1/a))
ax.plot(a_vals, ginis, label='estimated gini coefficient')
ax.plot(a_vals, ginis_theoretical, label='theoretical gini coefficient')
ax.legend()
ax.set_xlabel("Weibull parameter $a$")
ax.set_ylabel("Gini coefficient")
plt.show()
```

The simulation shows that the fit is good.

## A Model of Wealth Dynamics

Having discussed inequality measures, let us now turn to wealth dynamics.

The model we will study is

$$
w_{t+1} = (1 + r_{t+1}) s(w_t) + y_{t+1} \tag{1}
$$

where

- $ w_t $ is wealth at time $ t $ for a given household,
- $ r_t $ is the rate of return of financial assets,
- $ y_t $ is current non-financial (e.g., labor) income and
- $ s(w_t) $ is current wealth net of consumption

Letting $ \{z_t\} $ be a correlated state process of the form

$$
z_{t+1} = a z_t + b + \sigma_z \epsilon_{t+1}
$$

we’ll assume that

$$
R_t := 1 + r_t = c_r \exp(z_t) + \exp(\mu_r + \sigma_r \xi_t)
$$

and

$$
y_t = c_y \exp(z_t) + \exp(\mu_y + \sigma_y \zeta_t)
$$

Here $ \{ (\epsilon_t, \xi_t, \zeta_t) \} $ is IID and standard
normal in $ \mathbb R^3 $.

The value of $ c_r $ should be close to zero, since rates of return
on assets do not exhibit large trends.

When we simulate a population of households, we will assume all shocks are idiosyncratic (i.e.,  specific to individual households and independent across them).

Regarding the savings function $ s $, our default model will be

$$
s(w) = s_0 w \cdot \mathbb 1\{w \geq \hat w\} \tag{2}
$$

where $ s_0 $ is a positive constant.

Thus, for $ w < \hat w $, the household saves nothing. For
$ w \geq \bar w $, the household saves a fraction $ s_0 $ of
their wealth.

We are using something akin to a fixed savings rate model, while
acknowledging that low wealth households tend to save very little.

## Implementation

Here’s some type information to help Numba.

```python
wealth_dynamics_data = [
    ('w_hat',  float64),    # savings parameter
    ('s_0',    float64),    # savings parameter
    ('c_y',    float64),    # labor income parameter
    ('μ_y',    float64),    # labor income paraemter
    ('σ_y',    float64),    # labor income parameter
    ('c_r',    float64),    # rate of return parameter
    ('μ_r',    float64),    # rate of return parameter
    ('σ_r',    float64),    # rate of return parameter
    ('a',      float64),    # aggregate shock parameter
    ('b',      float64),    # aggregate shock parameter
    ('σ_z',    float64),    # aggregate shock parameter
    ('z_mean', float64),    # mean of z process
    ('z_var', float64),     # variance of z process
    ('y_mean', float64),    # mean of y process
    ('R_mean', float64)     # mean of R process
]
```

Here’s a class that stores instance data and implements methods that update
the aggregate state and household wealth.

```python
@jitclass(wealth_dynamics_data)
class WealthDynamics:

    def __init__(self,
                 w_hat=1.0,
                 s_0=0.75,
                 c_y=1.0,
                 μ_y=1.0,
                 σ_y=0.2,
                 c_r=0.05,
                 μ_r=0.1,
                 σ_r=0.5,
                 a=0.5,
                 b=0.0,
                 σ_z=0.1):

        self.w_hat, self.s_0 = w_hat, s_0
        self.c_y, self.μ_y, self.σ_y = c_y, μ_y, σ_y
        self.c_r, self.μ_r, self.σ_r = c_r, μ_r, σ_r
        self.a, self.b, self.σ_z = a, b, σ_z

        # Record stationary moments
        self.z_mean = b / (1 - a)
        self.z_var = σ_z**2 / (1 - a**2)
        exp_z_mean = np.exp(self.z_mean + self.z_var / 2)
        self.R_mean = c_r * exp_z_mean + np.exp(μ_r + σ_r**2 / 2)
        self.y_mean = c_y * exp_z_mean + np.exp(μ_y + σ_y**2 / 2)

        # Test a stability condition that ensures wealth does not diverge
        # to infinity.
        α = self.R_mean * self.s_0
        if α >= 1:
            raise ValueError("Stability condition failed.")

    def parameters(self):
        """
        Collect and return parameters.
        """
        parameters = (self.w_hat, self.s_0,
                      self.c_y, self.μ_y, self.σ_y,
                      self.c_r, self.μ_r, self.σ_r,
                      self.a, self.b, self.σ_z)
        return parameters

    def update_states(self, w, z):
        """
        Update one period, given current wealth w and persistent
        state z.
        """

        # Simplify names
        params = self.parameters()
        w_hat, s_0, c_y, μ_y, σ_y, c_r, μ_r, σ_r, a, b, σ_z = params
        zp = a * z + b + σ_z * np.random.randn()

        # Update wealth
        y = c_y * np.exp(zp) + np.exp(μ_y + σ_y * np.random.randn())
        wp = y
        if w >= w_hat:
            R = c_r * np.exp(zp) + np.exp(μ_r + σ_r * np.random.randn())
            wp += R * s_0 * w
        return wp, zp
```

Here’s function to simulate the time series of wealth for in individual households.

```python
@njit
def wealth_time_series(wdy, w_0, n):
    """
    Generate a single time series of length n for wealth given
    initial value w_0.

    The initial persistent state z_0 for each household is drawn from
    the stationary distribution of the AR(1) process.

        * wdy: an instance of WealthDynamics
        * w_0: scalar
        * n: int

    """
    z = wdy.z_mean + np.sqrt(wdy.z_var) * np.random.randn()
    w = np.empty(n)
    w[0] = w_0
    for t in range(n-1):
        w[t+1], z = wdy.update_states(w[t], z)
    return w
```

Now here’s function to simulate a cross section of households forward in time.

Note the use of parallelization to speed up computation.

```python
@njit(parallel=True)
def update_cross_section(wdy, w_distribution, shift_length=500):
    """
    Shifts a cross-section of household forward in time

    * wdy: an instance of WealthDynamics
    * w_distribution: array_like, represents current cross-section

    Takes a current distribution of wealth values as w_distribution
    and updates each w_t in w_distribution to w_{t+j}, where
    j = shift_length.

    Returns the new distribution.

    """
    new_distribution = np.empty_like(w_distribution)

    # Update each household
    for i in prange(len(new_distribution)):
        z = wdy.z_mean + np.sqrt(wdy.z_var) * np.random.randn()
        w = w_distribution[i]
        for t in range(shift_length-1):
            w, z = wdy.update_states(w, z)
        new_distribution[i] = w
    return new_distribution
```

Parallelization is very effective in the function above because the time path
of each household can be calculated independently once the path for the
aggregate state is known.

## Applications

Let’s try simulating the model at different parameter values and investigate
the implications for the wealth distribution.

### Time Series

Let’s look at the wealth dynamics of an individual household.

```python
wdy = WealthDynamics( w_hat=1.0,
                 s_0=0.75,
                 c_y=1.0,
                 μ_y=1.0,
                 σ_y=0.2,
                 c_r=0.05,
                 μ_r=0.1,
                 σ_r=0.5,
                 a=0.5,
                 b=0.0,
                 σ_z=0.1)

ts_length = 200
w = wealth_time_series(wdy, wdy.y_mean, ts_length)

fig, ax = plt.subplots()
ax.plot(w)
plt.show()
```

Notice the large spikes in wealth over time.

Such spikes are similar to what we observed in time series when [we studied Kesten processes](kesten_processes.ipynb).

### Inequality Measures

Let’s look at how inequality varies with returns on financial assets.

The next function generates a cross section and then computes the Lorenz
curve and Gini coefficient.

```python
def generate_lorenz_and_gini(wdy, num_households=100_000, T=500):
    """
    Generate the Lorenz curve data and gini coefficient corresponding to a
    WealthDynamics mode by simulating num_households forward to time T.
    """
    ψ_0 = np.ones(num_households) * wdy.y_mean
    z_0 = wdy.z_mean

    ψ_star = update_cross_section(wdy, ψ_0, shift_length=T)
    return qe.gini_coefficient(ψ_star), qe.lorenz_curve(ψ_star)
```

Now we investigate how the Lorenz curves associated with the wealth distribution change as return to savings varies.

The code below plots Lorenz curves for three different values of $ \mu_r $.

If you are running this yourself, note that it will take one or two minutes to execute.

This is unavoidable because we are executing a CPU intensive task.

In fact the code, which is JIT compiled and parallelized, runs extremely fast relative to the number of computations.

```python
fig, ax = plt.subplots()
μ_r_vals = (0.0, 0.025, 0.05)
gini_vals = []

for μ_r in μ_r_vals:
    wdy = WealthDynamics(w_hat=1.0,
                 s_0=0.75,
                 c_y=1.0,
                 μ_y=1.0,
                 σ_y=0.2,
                 c_r=0.05,
                 μ_r=μ_r,
                 σ_r=0.5,
                 a=0.5,
                 b=0.0,
                 σ_z=0.1)
    gv, (f_vals, l_vals) = generate_lorenz_and_gini(wdy)
    ax.plot(f_vals, l_vals, label=f'$\psi^*$ at $\mu_r = {μ_r:0.2}$')
    gini_vals.append(gv)

ax.plot(f_vals, f_vals, label='equality')
ax.legend(loc="upper left")
plt.show()
```

The Lorenz curve shifts downwards as returns on financial income rise, indicating a rise in inequality.

We will look at this again via the Gini coefficient immediately below, but
first consider the following image of our system resources when the code above
is executing:

Notice how effectively Numba has implemented multithreading for this routine:
all 8 CPUs on our workstation are running at maximum capacity (even though
four of them are virtual).

Since the code is both efficiently JIT compiled and fully parallelized, it’s
close to impossible to make this sequence of tasks run faster without changing
hardware.

Now let’s check the Gini coefficient.

```python
fig, ax = plt.subplots()
ax.plot(μ_r_vals, gini_vals, label='gini coefficient')
ax.set_xlabel("$\mu_r$")
ax.legend()
plt.show()
```

Once again, we see that inequality increases as returns on financial income
rise.

Let’s finish this section by investigating what happens when we change the
volatility term $ \sigma_r $ in financial returns.

```python
fig, ax = plt.subplots()
σ_r_vals = (0.35, 0.45, 0.52)
gini_vals = []

for σ_r in σ_r_vals:
    wdy = WealthDynamics(w_hat=1.0,
                 s_0=0.75,
                 c_y=1.0,
                 μ_y=1.0,
                 σ_y=0.2,
                 c_r=0.05,
                 μ_r=0.1,
                 σ_r=σ_r,
                 a=0.5,
                 b=0.0,
                 σ_z=0.1)
    gv, (f_vals, l_vals) = generate_lorenz_and_gini(wdy)
    ax.plot(f_vals, l_vals, label=f'$\psi^*$ at $\mu_r = {σ_r:0.2}$')
    gini_vals.append(gv)

ax.plot(f_vals, f_vals, label='equality')
ax.legend(loc="upper left")
plt.show()
```

We see that greater volatility has the effect of increasing inequality in this model.

## Exercises

### Exercise 1

For a wealth or income distribution with Pareto tail, a higher tail index suggests lower inequality.

Indeed, it is possible to prove that the Gini coefficient of the Pareto
distribution with tail index $ a $ is $ 1/(2a - 1) $.

To the extent that you can, confirm this by simulation.

In particular, generate a plot of the Gini coefficient against the tail index
using both the theoretical value just given and the value computed from a sample via `qe.gini_coefficient`.

For the values of the tail index, use `a_vals = np.linspace(1, 10, 25)`.

Use sample of size 1,000 for each $ a $ and the sampling method for generating Pareto draws employed in the discussion of Lorenz curves for the Pareto distribution.

To the extend that you can, interpret the monotone relationship between the
Gini index and $ a $.

```python
a_vals = np.linspace(1, 10, 25)
ginis = np.empty(len(a_vals))

samp_size= 1000

fig, ax = plt.subplots()

for i,a in enumerate(a_vals):
    y = np.random.uniform(size=samp_size)**(-1/a)
    ginis[i]=qe.gini_coefficient(y)

ax.plot(a_vals, ginis)
ax.plot(a_vals, 1/(2*a_vals - 1))
plt.show()
```

```python

```

### Exercise 2

The wealth process [(1)](#equation-wealth-dynam-ah) is similar to a [Kesten process](kesten_processes.ipynb).

This is because, according to [(2)](#equation-sav-ah), savings is constant for all wealth levels above $ \hat w $.

When savings is constant, the wealth process has the same quasi-linear
structure as a Kesten process, with multiplicative and additive shocks.

The Kesten–Goldie theorem tells us that Kesten processes have Pareto tails under a range of parameterizations.

The theorem does not directly apply here, since savings is not always constant and since the multiplicative and additive terms in [(1)](#equation-wealth-dynam-ah) are not IID.

At the same time, given the similarities, perhaps Pareto tails will arise.

To test this, run a simulation that generates a cross-section of wealth and
generate a rank-size plot.

In viewing the plot, remember that Pareto tails generate a straight line.  Is
this what you see?

For sample size and initial conditions, use

```python
@jitclass(wealth_dynamics_data)
class WealthDynamics:

    def __init__(self,
                 w_hat=1.0,
                 s_0=0.75,
                 c_y=1.0,
                 μ_y=1.0,
                 σ_y=0.2,
                 c_r=0.05,
                 μ_r=0.1,
                 σ_r=0.5,
                 a=0.5,
                 b=0.0,
                 σ_z=0.1):

        self.w_hat, self.s_0 = w_hat, s_0
        self.c_y, self.μ_y, self.σ_y = c_y, μ_y, σ_y
        self.c_r, self.μ_r, self.σ_r = c_r, μ_r, σ_r
        self.a, self.b, self.σ_z = a, b, σ_z

        # Record stationary moments
        self.z_mean = b / (1 - a)
        self.z_var = σ_z**2 / (1 - a**2)
        exp_z_mean = np.exp(self.z_mean + self.z_var / 2)
        self.R_mean = c_r * exp_z_mean + np.exp(μ_r + σ_r**2 / 2)
        self.y_mean = c_y * exp_z_mean + np.exp(μ_y + σ_y**2 / 2)

        # Test a stability condition that ensures wealth does not diverge
        # to infinity.
        α = self.R_mean * self.s_0
        if α >= 1:
            raise ValueError("Stability condition failed.")

    def parameters(self):
        """
        Collect and return parameters.
        """
        parameters = (self.w_hat, self.s_0,
                      self.c_y, self.μ_y, self.σ_y,
                      self.c_r, self.μ_r, self.σ_r,
                      self.a, self.b, self.σ_z)
        return parameters

    def update_states(self, w, z):
        """
        Update one period, given current wealth w and persistent
        state z.
        """

        # Simplify names
        params = self.parameters()
        w_hat, s_0, c_y, μ_y, σ_y, c_r, μ_r, σ_r, a, b, σ_z = params
        zp = a * z + b + σ_z * np.random.randn()

        # Update wealth
        y = c_y * np.exp(zp) + np.exp(μ_y + σ_y * np.random.randn())
        wp = y
        if w >= w_hat:
            R = c_r * np.exp(zp) + np.exp(μ_r + σ_r * np.random.randn())
            wp += R * s_0 * w
        return wp, zp
@njit
def wealth_time_series(wdy, w_0, n):
    """
    Generate a single time series of length n for wealth given
    initial value w_0.

    The initial persistent state z_0 for each household is drawn from
    the stationary distribution of the AR(1) process.

        * wdy: an instance of WealthDynamics
        * w_0: scalar
        * n: int

    """
    z = wdy.z_mean + np.sqrt(wdy.z_var) * np.random.randn()
    w = np.empty(n)
    w[0] = w_0
    for t in range(n-1):
        w[t+1], z = wdy.update_states(w[t], z)
    return w
```

```python
num_households = 250_000
T = 500                                      # shift forward T periods
ψ_0 = np.ones(num_households) * wdy.y_mean   # initial distribution
z_0 = wdy.z_mean

#@njit
def wealth_time_series(wdy, w_0, n):
    """
    Generate a single time series of length n for wealth given
    initial value w_0.

    The initial persistent state z_0 for each household is drawn from
    the stationary distribution of the AR(1) process.

        * wdy: an instance of WealthDynamics
        * w_0: scalar
        * n: int

    """
    z = wdy.z_mean + np.sqrt(wdy.z_var) * np.random.randn()
    w = np.empty(n)
    w[0] = w_0
    for t in range(n-1):
        w[t+1], z = wdy.update_states(w[t], z)
    return w

household=wealth_time_series(wdy, ψ_0[0], T)
ψ_star = update_cross_section(wdy, ψ_0, shift_length=T)
```

```python
from scipy.stats import pareto, norm
import matplotlib.pyplot as plt
import numpy as np

log_size=np.sort(np.log(household))
log_size_rank=np.log(log_size.argsort()[::-1]+1)
fig, ax = plt.subplots()

ax.scatter(y=log_size,x=log_size_rank, marker='o', alpha=0.5)
ax.set_xlabel("log rank")
ax.set_ylabel("log size")

plt.show()

```