```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator
from scipy import stats

from compton import setup_rc_params

setup_rc_params()
```

In GMP 2016 they use
\begin{align}
    \xi^{(s)} & = c_0^{(s)} + c_2^{(s)} \delta^2 + \Delta_2^{(s)} \\
    \xi^{(v)} & = c_0^{(v)} + \Delta_0^{(v)}
\end{align}
and find a posterior for $\Delta$ given the convergence pattern as in Furnstahl (2015).
So the first omitted term for the scalar pieces goes as $\delta^3$ and the first omitted term for the vector piece goes as $\delta^1$.

```python
# e^2 delta^2, e^2 delta^3, e^2 delta^4-fitted
gamma_e1e1_proton = np.array([-5.68562, -5.45127, -1.1])
gamma_e1e1_neutron = np.array([-5.68562, -5.45127, -4.0227])

# gamma_m1m1_proton = np.array([-1.13712, 1.95882, 2.2])
# gamma_m1m1_neutron = np.array([-1.13712, 1.95882, 1.26474])

# Use a fitted value for NLO...?
gamma_m1m1_proton = np.array([-1.13712, 1.63, 2.2])
gamma_m1m1_neutron = np.array([-1.13712, 1.63, 1.26474])

gamma_e1m2_proton = np.array([1.13712, 0.55244, -0.4])
gamma_e1m2_neutron = np.array([1.13712, 0.55244, -0.1343])

gamma_m1e2_proton = np.array([1.13712, 1.2701, 1.9])
gamma_m1e2_neutron = np.array([1.13712, 1.2701, 2.36763])

gamma_0_proton = - (gamma_e1e1_proton + gamma_m1m1_proton + gamma_e1m2_proton + gamma_m1e2_proton)
gamma_pi_proton = - gamma_e1e1_proton + gamma_m1m1_proton - gamma_e1m2_proton + gamma_m1e2_proton

gamma_0_neutron = - (gamma_e1e1_neutron + gamma_m1m1_neutron + gamma_e1m2_neutron + gamma_m1e2_neutron)
gamma_pi_neutron = - gamma_e1e1_neutron + gamma_m1m1_neutron - gamma_e1m2_neutron + gamma_m1e2_neutron
```

```python
gamma_0_proton
```

    array([ 4.5485 ,  1.99873, -2.6    ])

```python
gamma_pi_proton
```

    array([4.5485 , 7.79893, 5.6    ])

```python
gamma_0_neutron
```

    array([4.5485 , 1.99873, 0.52463])

```python
gamma_pi_neutron
```

    array([4.5485 , 7.79893, 7.78937])

```python
# Table 2
# gamma_e1e1_scalar = [-5.7, -2.6]
# gamma_e1e1_vector = [+1.5]

# gamma_m1m1_scalar = [-1.1, +1.8]
# gamma_m1m1_vector = [+0.5]

# gamma_e1m2_scalar = [+1.1, -0.3]
# gamma_e1m2_vector = [-0.1]

# gamma_m1e2_scalar = [+1.1, +2.2]
# gamma_m1e2_vector = [-0.2]

scalar_idxs = [0, -1]
scalar_idxs_m1m1 = [0, 1, 2]
vector_idxs = [-1]

gamma_e1e1_scalar = (gamma_e1e1_proton + gamma_e1e1_neutron)[scalar_idxs] / 2
gamma_e1e1_vector = (gamma_e1e1_proton - gamma_e1e1_neutron)[vector_idxs] / 2

gamma_m1m1_scalar = (gamma_m1m1_proton + gamma_m1m1_neutron)[scalar_idxs_m1m1] / 2
gamma_m1m1_vector = (gamma_m1m1_proton - gamma_m1m1_neutron)[vector_idxs] / 2

gamma_e1m2_scalar = (gamma_e1m2_proton + gamma_e1m2_neutron)[scalar_idxs] / 2
gamma_e1m2_vector = (gamma_e1m2_proton - gamma_e1m2_neutron)[vector_idxs] / 2

gamma_m1e2_scalar = (gamma_m1e2_proton + gamma_m1e2_neutron)[scalar_idxs] / 2
gamma_m1e2_vector = (gamma_m1e2_proton - gamma_m1e2_neutron)[vector_idxs] / 2

gamma_0_scalar = (gamma_0_proton + gamma_0_neutron)[scalar_idxs_m1m1] / 2
gamma_0_vector = (gamma_0_proton - gamma_0_neutron)[vector_idxs] / 2

gamma_pi_scalar = (gamma_pi_proton + gamma_pi_neutron)[scalar_idxs_m1m1] / 2
gamma_pi_vector = (gamma_pi_proton - gamma_pi_neutron)[vector_idxs] / 2
```

```python
print('gamma E1E1 Scalar:', gamma_e1e1_scalar)
print('gamma E1E1 Vector:', gamma_e1e1_vector)

print('gamma M1M1 Scalar:', gamma_m1m1_scalar)
print('gamma M1M1 Vector:', gamma_m1m1_vector)

print('gamma E1M2 Scalar:', gamma_e1m2_scalar)
print('gamma E1M2 Vector:', gamma_e1m2_vector)

print('gamma M1E2 Scalar:', gamma_m1e2_scalar)
print('gamma M1E2 Vector:', gamma_m1e2_vector)

print('gamma 0 Scalar:', gamma_0_scalar)
print('gamma 0 Vector:', gamma_0_vector)

print('gamma pi Scalar:', gamma_pi_scalar)
print('gamma pi Vector:', gamma_pi_vector)
```

    gamma E1E1 Scalar: [-5.68562 -2.56135]
    gamma E1E1 Vector: [1.46135]
    gamma M1M1 Scalar: [-1.13712  1.63     1.73237]
    gamma M1M1 Vector: [0.46763]
    gamma E1M2 Scalar: [ 1.13712 -0.26715]
    gamma E1M2 Vector: [-0.13285]
    gamma M1E2 Scalar: [1.13712  2.133815]
    gamma M1E2 Vector: [-0.233815]
    gamma 0 Scalar: [ 4.5485    1.99873  -1.037685]
    gamma 0 Vector: [-1.562315]
    gamma pi Scalar: [4.5485   7.79893  6.694685]
    gamma pi Vector: [-1.094685]

```python
def compute_coefficients(y, Q, orders=None):
    if orders is None:
        orders = np.arange(len(y))
    y = np.atleast_1d(y)
    diffs = np.diff(y)
    diffs = np.insert(diffs, 0, y[0])
    return diffs / Q ** orders
```

```python
delta = 0.4  # Eq. (2)
```

Extract coefficients

```python
orders_isoscl = np.array([0, 2])
orders_isoscl_m1m1 = np.array([0, 1, 2])
orders_isovec = np.array([0])

coefs_e1e1_scalar = compute_coefficients(gamma_e1e1_scalar, delta, orders=orders_isoscl)
coefs_e1e1_vector = compute_coefficients(gamma_e1e1_vector, delta, orders=orders_isovec)

coefs_m1m1_scalar = compute_coefficients(gamma_m1m1_scalar, delta, orders=orders_isoscl_m1m1)
coefs_m1m1_vector = compute_coefficients(gamma_m1m1_vector, delta, orders=orders_isovec)

coefs_e1m2_scalar = compute_coefficients(gamma_e1m2_scalar, delta, orders=orders_isoscl)
coefs_e1m2_vector = compute_coefficients(gamma_e1m2_vector, delta, orders=orders_isovec)

coefs_m1e2_scalar = compute_coefficients(gamma_m1e2_scalar, delta, orders=orders_isoscl)
coefs_m1e2_vector = compute_coefficients(gamma_m1e2_vector, delta, orders=orders_isovec)

coefs_0_scalar = compute_coefficients(gamma_0_scalar, delta, orders=orders_isoscl_m1m1)
coefs_0_vector = compute_coefficients(gamma_0_vector, delta, orders=orders_isovec)

coefs_pi_scalar = compute_coefficients(gamma_pi_scalar, delta, orders=orders_isoscl_m1m1)
coefs_pi_vector = compute_coefficients(gamma_pi_vector, delta, orders=orders_isovec)
```

```python
print('Scalar E1E1 c_n:', coefs_e1e1_scalar)
print('Vector E1E1 c_n:', coefs_e1e1_vector)
```

    Scalar E1E1 c_n: [-5.68562   19.5266875]
    Vector E1E1 c_n: [1.46135]

```python
print('Scalar M1M1 c_n:', coefs_m1m1_scalar)
print('Vector M1M1 c_n:', coefs_m1m1_vector)
```

    Scalar M1M1 c_n: [-1.13712    6.9178     0.6398125]
    Vector M1M1 c_n: [0.46763]

```python
print('Scalar E1M2 c_n:', coefs_e1m2_scalar)
print('Vector E1M2 c_n:', coefs_e1m2_vector)
```

    Scalar E1M2 c_n: [ 1.13712   -8.7766875]
    Vector E1M2 c_n: [-0.13285]

```python
print('Scalar M1E2 c_n:', coefs_m1e2_scalar)
print('Vector M1E2 c_n:', coefs_m1e2_vector)
```

    Scalar M1E2 c_n: [1.13712    6.22934375]
    Vector M1E2 c_n: [-0.233815]

```python
print('Scalar 0 c_n:', coefs_0_scalar)
print('Vector 0 c_n:', coefs_0_vector)
```

    Scalar 0 c_n: [  4.5485      -6.374425   -18.97759375]
    Vector 0 c_n: [-1.562315]

```python
print('Scalar pi c_n:', coefs_pi_scalar)
print('Vector pi c_n:', coefs_pi_vector)
```

    Scalar pi c_n: [ 4.5485      8.126075   -6.90153125]
    Vector pi c_n: [-1.094685]

## 2016 Analysis (Set A Epsilon Priors)

```python
def compute_A_eps1_posterior(x, c, Q, first_omitted_order=None, verbose=False):
    if first_omitted_order is None:
        first_omitted_order = len(c)
    n_c = len(c)
#     n_c = first_omitted_order
    max_c = np.max(np.abs(c))
    R = max_c * Q**first_omitted_order
    if verbose:
        print('R', R)
    factor = n_c / (n_c + 1.) / (2. * R)
    with np.errstate(divide='ignore'):
        return factor * np.where(np.abs(x) <= R, 1., (R / np.abs(x)) ** (n_c + 1))
```

```python
def plot_scalar_and_vector(x, c_s, c_v, Q, first_omitted_order_s, first_omitted_order_v, num_x_minor=5, num_y_minor=5, ax=None):
    """A convenience function to make the same plot easily"""
    from matplotlib.ticker import MultipleLocator, AutoMinorLocator

    if ax is None:
        fig, ax = plt.subplots()
    ax.plot(x, compute_A_eps1_posterior(x, c_s, Q, first_omitted_order=first_omitted_order_s, verbose=True), c='b', ls='--')
    ax.plot(x, compute_A_eps1_posterior(x, c_v, Q, first_omitted_order=first_omitted_order_v, verbose=True), c='g', ls=':')

    # Grid that work well enough for these densities
    dense_x = np.linspace(-50, 50, 5000)
    post_s = compute_A_eps1_posterior(dense_x, c_s, Q, first_omitted_order=first_omitted_order_s)
    post_v = compute_A_eps1_posterior(dense_x, c_v, Q, first_omitted_order=first_omitted_order_v)
    conv = np.convolve(post_s, post_v, mode='same')
    conv /= np.trapz(conv, dense_x)
    conv_interp = np.interp(x, dense_x, conv)
    ax.plot(x, conv_interp, c='r')
    ax.margins(x=0)
    ax.xaxis.set_minor_locator(AutoMinorLocator(num_x_minor))
    ax.yaxis.set_minor_locator(AutoMinorLocator(num_y_minor))
    ax.tick_params(which='both', top=True, right=True)
    ax.set_ylim(0, None)
    return ax
```

```python
N = 500
d_e1e1 = np.linspace(0, 10, N)
d_m1m1 = np.linspace(0, 3, N)
d_e1m2 = np.linspace(0, 3, N)
d_m1e2 = np.linspace(0, 3, N)
d_0 = np.linspace(0, 8, N)
d_pi = np.linspace(0, 8, N)

first_omitted_s = 3
first_omitted_v = 1
```

```python
fig, axes = plt.subplots(3, 2, figsize=(7, 5))

print('E1E1')
ax_e1e1 = plot_scalar_and_vector(
    d_e1e1, coefs_e1e1_scalar, coefs_e1e1_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    ax=axes[0, 0]
)
ax_e1e1.yaxis.set_major_locator(MultipleLocator(0.1))
ax_e1e1.set_ylabel(r'pr$_{\gamma_{E1E1}}(\Delta)$')

print('M1M1')
ax_m1m1 = plot_scalar_and_vector(
    d_m1m1, coefs_m1m1_scalar, coefs_m1m1_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    num_y_minor=4, ax=axes[0, 1]
)
ax_m1m1.yaxis.set_major_locator(MultipleLocator(0.2))
ax_m1m1.set_ylabel(r'pr$_{\gamma_{M1M1}}(\Delta)$')

print('E1M2')
ax_e1m2 = plot_scalar_and_vector(
    d_e1m2, coefs_e1m2_scalar, coefs_e1m2_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    ax=axes[1, 0]
)
ax_e1m2.set_ylabel(r'pr$_{\gamma_{E1M2}}(\Delta)$')

print('M1E2')
ax_m1e2 = plot_scalar_and_vector(
    d_m1e2, coefs_m1e2_scalar, coefs_m1e2_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    ax=axes[1, 1]
)
ax_m1e2.set_ylabel(r'pr$_{\gamma_{M1E2}}(\Delta)$')

print('0')
ax_0 = plot_scalar_and_vector(
    d_0, coefs_0_scalar, coefs_0_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    ax=axes[2, 0]
)
ax_0.yaxis.set_major_locator(MultipleLocator(0.1))
ax_0.set_ylabel(r'pr$_{\gamma_0}(\Delta)$')

print('pi')
ax_pi = plot_scalar_and_vector(
    d_pi, coefs_pi_scalar, coefs_pi_vector, delta,
    first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
    ax=axes[2, 1]
)
ax_pi.yaxis.set_major_locator(MultipleLocator(0.1))
ax_pi.set_ylabel(r'pr$_{\gamma_{\pi}}(\Delta)$')

fig.tight_layout()
fig.savefig('pol_priors_set_Aeps')
```

Not bad! The $\gamma_0$ and $\gamma_\pi$ are still not exactly right. I'm not sure if I got the NLO corrections right.

## Updated Analysis

Here $\nu_0 = 0$ means that the prior is completely uninformative. To mirror the previous results, use that.

```python
def compute_nu_and_tau(c, nu0=0, tau0=1):
    c = np.atleast_1d(c)
    nu = nu0 + len(c)
    tau_sq = (nu0 * tau0**2 + c @ c) / nu
    return nu, np.sqrt(tau_sq)

def compute_cbar_estimate(c, nu0=0, tau0=1):
    nu, tau = compute_nu_and_tau(c, nu0=nu0, tau0=tau0)
    # Either get the MAP value or the mean value...
#     return tau * np.sqrt(nu / (nu - 2))
    return tau * np.sqrt(nu / (nu + 2))
```

```python
nu_0 = 0
tau_0 = 1
```

```python
def plot_scalar_and_vector_gaussian(x, c_s, c_v, Q, first_omitted_order_s, first_omitted_order_v, num_x_minor=5, num_y_minor=5, nu0=0, tau0=1, ax=None, full_sum=True):
    """A convenience function to make the same plot easily"""
    from matplotlib.ticker import MultipleLocator, AutoMinorLocator

#     nu_s, tau_s = compute_nu_and_tau(c_s, nu0=nu0, tau0=tau0)
#     nu_v, tau_v = compute_nu_and_tau(c_v, nu0=nu0, tau0=tau0)
    cbar_s = compute_cbar_estimate(c_s, nu0=nu0, tau0=tau0)
    cbar_v = compute_cbar_estimate(c_v, nu0=nu0, tau0=tau0)
#     print('cbar_s:', cbar_s)
#     print('cbar_v:', cbar_v)

    if full_sum:
        Q_sum_s = Q**first_omitted_order_s / np.sqrt(1 - Q**2)
        Q_sum_v = Q**first_omitted_order_v / np.sqrt(1 - Q**2)
    else:
        Q_sum_s = Q**first_omitted_order_s
        Q_sum_v = Q**first_omitted_order_v

    std_s = cbar_s * Q_sum_s
    std_v = cbar_v * Q_sum_v

    if ax is None:
        fig, ax = plt.subplots()

    norm_s = stats.norm(scale=std_s)
    norm_v = stats.norm(scale=std_v)
    ax.plot(x, norm_s.pdf(x), c='b', ls='--')
    ax.plot(x, norm_v.pdf(x), c='g', ls=':')

    norm_tot = stats.norm(scale=np.sqrt(std_s**2 + std_v**2))
    ax.plot(x, norm_tot.pdf(x), c='r')
    ax.margins(x=0)
    ax.xaxis.set_minor_locator(AutoMinorLocator(num_x_minor))
    ax.yaxis.set_minor_locator(AutoMinorLocator(num_y_minor))
    ax.tick_params(which='both', top=True, right=True)
    ax.set_ylim(0, None)
    return ax

def plot_scalar_and_vector_student(x, c_s, c_v, Q, first_omitted_order_s, first_omitted_order_v, num_x_minor=5, num_y_minor=5, nu0=0, tau0=1, ax=None, full_sum=True):
    """A convenience function to make the same plot easily"""
    from matplotlib.ticker import MultipleLocator, AutoMinorLocator

    nu_s, tau_s = compute_nu_and_tau(c_s, nu0=nu0, tau0=tau0)
    nu_v, tau_v = compute_nu_and_tau(c_v, nu0=nu0, tau0=tau0)

    if full_sum:
        Q_sum_s = Q**first_omitted_order_s / np.sqrt(1 - Q**2)
        Q_sum_v = Q**first_omitted_order_v / np.sqrt(1 - Q**2)
    else:
        Q_sum_s = Q**first_omitted_order_s
        Q_sum_v = Q**first_omitted_order_v

    scale_s = tau_s * Q_sum_s
    scale_v = tau_v * Q_sum_v

    if ax is None:
        fig, ax = plt.subplots()

    t_s = stats.t(df=nu_s, scale=scale_s)
    t_v = stats.t(df=nu_v, scale=scale_v)

    ax.plot(x, t_s.pdf(x), c='b', ls='--')
    ax.plot(x, t_v.pdf(x), c='g', ls=':')

    # Grid that work well enough for these densities
    dense_x = np.linspace(-50, 50, 5000)
    post_s = t_s.pdf(dense_x)
    post_v = t_v.pdf(dense_x)
    conv = np.convolve(post_s, post_v, mode='same')
    conv /= np.trapz(conv, dense_x)
    conv_interp = np.interp(x, dense_x, conv)
    ax.plot(x, conv_interp, c='r')
    ax.margins(x=0)
    ax.xaxis.set_minor_locator(AutoMinorLocator(num_x_minor))
    ax.yaxis.set_minor_locator(AutoMinorLocator(num_y_minor))
    ax.tick_params(which='both', top=True, right=True)
    ax.set_ylim(0, None)
    return ax
```

The Gaussian plots below show the problem with using MAP values when you actually have quite long tails! Point estimates are not always representative of the full distribution.

```python
for sum_to_inf in [False, True]:
    fig, axes = plt.subplots(3, 2, figsize=(7, 5))
    ax_e1e1 = plot_scalar_and_vector_gaussian(
        d_e1e1, coefs_e1e1_scalar, coefs_e1e1_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[0, 0], full_sum=sum_to_inf
    )
    ax_e1e1.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_e1e1.set_ylabel(r'pr$_{\gamma_{E1E1}}(\Delta)$')

    ax_m1m1 = plot_scalar_and_vector_gaussian(
        d_m1m1, coefs_m1m1_scalar, coefs_m1m1_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        num_y_minor=4, nu0=nu_0, tau0=tau_0, ax=axes[0, 1], full_sum=sum_to_inf
    )
    ax_m1m1.yaxis.set_major_locator(MultipleLocator(0.2))
    ax_m1m1.set_ylabel(r'pr$_{\gamma_{M1M1}}(\Delta)$')

    ax_e1m2 = plot_scalar_and_vector_gaussian(
        d_e1m2, coefs_e1m2_scalar, coefs_e1m2_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[1, 0], full_sum=sum_to_inf
    )
    ax_e1m2.set_ylabel(r'pr$_{\gamma_{E1M2}}(\Delta)$')

    ax_m1e2 = plot_scalar_and_vector_gaussian(
        d_m1e2, coefs_m1e2_scalar, coefs_m1e2_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[1, 1], full_sum=sum_to_inf
    )
    ax_m1e2.set_ylabel(r'pr$_{\gamma_{M1E2}}(\Delta)$')

    ax_0 = plot_scalar_and_vector_gaussian(
        d_0, coefs_0_scalar, coefs_0_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[2, 0], full_sum=sum_to_inf
    )
    ax_0.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_0.set_ylabel(r'pr$_{\gamma_0}(\Delta)$')

    ax_pi = plot_scalar_and_vector_gaussian(
        d_pi, coefs_pi_scalar, coefs_pi_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[2, 1], full_sum=sum_to_inf
    )
    ax_pi.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_pi.set_ylabel(r'pr$_{\gamma_{\pi}}(\Delta)$')

    if sum_to_inf:
        fig.suptitle('Gaussian Distribution, Sum to $Q^\infty$', y=1)
    else:
        fig.suptitle('Gaussian Distribution, First-Omitted Term Approximation', y=1)

    fig.tight_layout()
    plt.show()
    fig.savefig(f'pol_priors_conjugate_gaussian_sum-inf-{sum_to_inf}')
```

The integrated distributions below look much better.

```python
for sum_to_inf in [False, True]:
    fig, axes = plt.subplots(3, 2, figsize=(7, 5))

    ax_e1e1 = plot_scalar_and_vector_student(
        d_e1e1, coefs_e1e1_scalar, coefs_e1e1_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[0, 0], full_sum=sum_to_inf
    )
    ax_e1e1.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_e1e1.set_ylabel(r'pr$_{\gamma_{E1E1}}(\Delta)$')

    ax_m1m1 = plot_scalar_and_vector_student(
        d_m1m1, coefs_m1m1_scalar, coefs_m1m1_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        num_y_minor=4, nu0=nu_0, tau0=tau_0, ax=axes[0, 1], full_sum=sum_to_inf
    )
    ax_m1m1.yaxis.set_major_locator(MultipleLocator(0.2))
    ax_m1m1.set_ylabel(r'pr$_{\gamma_{M1M1}}(\Delta)$')

    ax_e1m2 = plot_scalar_and_vector_student(
        d_e1m2, coefs_e1m2_scalar, coefs_e1m2_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[1, 0], full_sum=sum_to_inf
    )
    ax_e1m2.set_ylabel(r'pr$_{\gamma_{E1M2}}(\Delta)$')

    ax_m1e2 = plot_scalar_and_vector_student(
        d_m1e2, coefs_m1e2_scalar, coefs_m1e2_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[1, 1], full_sum=sum_to_inf
    )
    ax_m1e2.set_ylabel(r'pr$_{\gamma_{M1E2}}(\Delta)$')

    ax_0 = plot_scalar_and_vector_student(
        d_0, coefs_0_scalar, coefs_0_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[2, 0], full_sum=sum_to_inf
    )
    ax_0.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_0.set_ylabel(r'pr$_{\gamma_0}(\Delta)$')

    ax_pi = plot_scalar_and_vector_student(
        d_pi, coefs_pi_scalar, coefs_pi_vector, delta,
        first_omitted_order_s=first_omitted_s, first_omitted_order_v=first_omitted_v,
        nu0=nu_0, tau0=tau_0, ax=axes[2, 1], full_sum=sum_to_inf
    )
    ax_pi.yaxis.set_major_locator(MultipleLocator(0.1))
    ax_pi.set_ylabel(r'pr$_{\gamma_{\pi}}(\Delta)$')

    if sum_to_inf:
        fig.suptitle('Student $t$ Distribution, Sum to $Q^\infty$', y=1)
    else:
        fig.suptitle('Student $t$ Distribution, First-Omitted Term Approximation', y=1)
    fig.tight_layout()
    plt.show()
    fig.savefig(f'pol_priors_conjugate_student_sum-inf-{sum_to_inf}')
```

```python

```


```
import bluefog.torch as bf
import matplotlib.pyplot as plt
import numpy as np
import networkx as nx
import torch

%matplotlib inline
```

```
import ipyparallel as ipp

rc = ipp.Client(profile="bluefog")
rc.ids
```

    [0, 1, 2, 3, 4, 5, 6, 7]

```
%%px
import torch
import bluefog.torch as bf
from bluefog.common import topology_util

bf.init()

print(f"Rank: {bf.rank()}, Size: {bf.size()}")
```

    [stdout:0] Rank: 0, Size: 8
    [stdout:1] Rank: 5, Size: 8
    [stdout:2] Rank: 7, Size: 8
    [stdout:3] Rank: 6, Size: 8
    [stdout:4] Rank: 4, Size: 8
    [stdout:5] Rank: 1, Size: 8
    [stdout:6] Rank: 3, Size: 8
    [stdout:7] Rank: 2, Size: 8

#  Average Consensus Over General Dynamic Topology

In previous example, dynamic one peer strategy over exponential 2 graph has several nice properties. Every iteration, the combination matrix $W$ is always doubly stochastic. This guarantees the unbiased result of running average consensus algorithm. However, this property no longer holds for general case. Let's use star graph to illustrate that.

```
nx.draw_spring(bf.StarGraph(8))
```

```
generator = bf.GetDynamicOnePeerSendRecvRanks(bf.StarGraph(8), self_rank=0)
for _ in range(10):
    print(next(generator))
```

    ([1], [1, 2, 3, 4, 5, 6, 7])
    ([2], [1, 2, 3, 4, 5, 6, 7])
    ([3], [1, 2, 3, 4, 5, 6, 7])
    ([4], [1, 2, 3, 4, 5, 6, 7])
    ([5], [1, 2, 3, 4, 5, 6, 7])
    ([6], [1, 2, 3, 4, 5, 6, 7])
    ([7], [1, 2, 3, 4, 5, 6, 7])
    ([1], [1, 2, 3, 4, 5, 6, 7])
    ([2], [1, 2, 3, 4, 5, 6, 7])
    ([3], [1, 2, 3, 4, 5, 6, 7])

```
%%px
data_size = 1
seed = 1234
max_iters = 100
torch.random.manual_seed(seed * bf.rank())

x = torch.randn(data_size, dtype=torch.double)
x_bar = bf.allreduce(x, average=True)
```

```
%%px
bf.set_topology(topology_util.StarGraph(bf.size()))
dynamic_neighbor_allreduce_gen = topology_util.GetDynamicOnePeerSendRecvRanks(
    bf.load_topology(), bf.rank()
)
x_list = []
for ite in range(max_iters):
    send_neighbors, recv_neighbors = next(dynamic_neighbor_allreduce_gen)
    uniform_weight = 1 / (len(recv_neighbors) + 1)
    neighbor_weights = {r: uniform_weight for r in recv_neighbors}
    self_weight = uniform_weight

    x = bf.neighbor_allreduce(
        x,
        name="x",
        self_weight=self_weight,
        src_weights=neighbor_weights,
        dst_weights=send_neighbors,
        enable_topo_check=True,
    )
    x_list.append(x)
```

```
x_all_ranks = rc[:].pull("x_list", block=True)
x_bar = rc[0].pull("x_bar", block=True)
```

```
for i in rc.ids:
    plt.plot(x_all_ranks[i], label=f"agent {i}")
plt.axhline(x_bar, linestyle="--", color="k")
plt.legend()
plt.xlabel("relative time")
plt.ylabel("value")
```

```
for i in rc.ids:
    plt.plot(x_all_ranks[i], label=f"agent {i}")
plt.axhline(x_bar, linestyle="--", color="k")
plt.legend()

plt.axis([-1, len(x_all_ranks[0]), x_bar + 0.1, x_bar - 0.1])
plt.xlabel("relative time")
plt.ylabel("value")
```

We can see that although all agents converge to one consensus value, it is different from the true mean as expected.

# Push-Sum Consensus Algorithm
The Push-Sum consensus algorithm is proposed to correct these biased problem.
The algorithm is to introduce another weights $p$ and the final average concensus value is the original vector divided by the weights.
For each agent $i$, run it in parallel:
$$
    x_{i, k+1} = \sum_{j\in \mathcal{N}_i} w_{ij}^{(k)} x_{j,k}\\
    p_{i, k+1} = \sum_{j\in \mathcal{N}_i} w_{ij}^{(k)} p_{j,k}\\
    y_{i, k+1} = x_{i, k+1} / p_{i, k+1}
$$
where $p_{i, 0}$ is initialize as 1.
Here is one restriction here the combination matrix $[w_{ij}]$ has to be column stochastic matrix, i.e.
$$
    \sum_{j} w_{ij} = 1.
$$
instead of row stochastic matrix.
Under this property and more connective assumption, it can be shown that
\begin{align}
    \mathbb{1}^T p_k &\;\equiv \mathbb{1}, \\
    \mathbb{1}^T x_{k} &\; = \mathbb{1}^T x_{k-1} = \cdots = \mathbb{1}^T x_{0},\\
    x_{k}/p_{k} &\;\to \frac{1}{N} \mathbb{1}^T x_{0} \mathbb{1}
\end{align}

However, there is one problem that if we normalize the recieve weights through the recieving rank list, we will generate row stochastic matrix instead of column one.
In this case, we should use `dst_weights` argument instead of `src_weights`.

```
%%px
x = torch.randn(data_size, dtype=torch.double)
p = torch.ones(1)  # the correction weights
x_bar = bf.allreduce(x, average=True)
```

```
%%px
bf.set_topology(topology_util.MeshGrid2DGraph(bf.size()))
dynamic_neighbor_gen = topology_util.GetDynamicOnePeerSendRecvRanks(
    bf.load_topology(), bf.rank()
)
x_list = []
y_list = []
for ite in range(max_iters):
    dst_neighbors, src_neighbors = next(dynamic_neighbor_gen)
    src_weights = {r: 1 for r in src_neighbors}
    self_weight = 1 / (len(dst_neighbors) + 1)
    dst_weights = {r: self_weight for r in dst_neighbors}
    x = bf.neighbor_allreduce(
        x,
        name="x",
        self_weight=self_weight,
        src_weights=src_weights,
        dst_weights=dst_weights,
        enable_topo_check=True,
    )
    p = bf.neighbor_allreduce(
        p,
        name="p",
        self_weight=self_weight,
        src_weights=src_weights,
        dst_weights=dst_weights,
        enable_topo_check=True,
    )
    x_list.append(x)
    y_list.append(x / p)
```

```
x_bar = rc[0].pull("x_bar", block=True)
x_all_ranks = rc[:].pull("x_list", block=True)
y_all_ranks = rc[:].pull("y_list", block=True)
for i in rc.ids:
    plt.plot(y_all_ranks[i], label=f"agent {i} (y)")
plt.axhline(x_bar, linestyle="--", color="k")
plt.legend()
plt.xlabel("relative time")
plt.ylabel("value")
```


# Problem set 1: Financial Frictions, Liquidity and the Business Cycle

```python
import numpy as np
import math
from scipy import optimize
import scipy.stats as stats
import PS1_1functions as func

import scipy.stats as stats

# For plots:
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
```

# Exercise 3.5 in JT (The Theory of Corporate Finance)

Consider the continuous the continuous investment model with decreasing returns to scale outlined in chapter 3 of JT. The core of the model is as follows:
- Let $I\in[0, \infty)$ be the level of investment in a given project.
- Entrepreneur proposes a project. If successful the project generates income $R(I)$; if not the project generates $0$.
- The probability of success depends on the behavior of the entrepreneur. If E behaves $(b)$ then probability of success is $p_H\in(0,1)$. If E does not behave $(nb)$ then the probability of success is $p_L$ where $0\leq p_L<p_H$.
- The entrepreneur has an incentive to not behave $(nb)$, as he receives $BI$ in private benefits in this case.
- The entrepreneur is endowed with A in assets.
- No investment project is profitable, if entrepreneur chooses not to behave.
- The entrepreneur's technology obeys the following conditions:
    1. Positive, but decreasing returns to investments: $R'(I)>0, R''(I)<0$.
    2. *Regularity condition 1:* Under perfect information a positive investment level would be optimal, i.e. $R'(0)>1/p_H$.
    3. *Regularity condition 2:* Under perfect information a finite level of investment is optiaml, i.e. $ lim_{I\rightarrow \infty} R'(I) < 1/p_H$.
- Assume perfect competition between lenders.

#### The Entrepreneurial problem, part I

The first part of the entrepreneur's problem is the discrete choice: $b$ or $nb$. The utility from each choice is given by:

$$\begin{align}
u_b &= p_HR_b(I)-A \\
u_{nb}&= p_LR_b(I)+BI-A.
\end{align}$$

The **incentive compatability constraint (IC)** is the threshold when option $b$ is preferred. This is defined as:

$$\begin{align}
u_b-u_{nb}\geq 0 && \Rightarrow && \Delta p R_b(I) \geq BI, \tag{IC}
\end{align}$$

where $\Delta p \equiv p_H-p_L$.

#### The lender's problem:

The lender faces two problems here: (1) Risk over outcomes and (2) Moral hazard. The moral hazard comes from the entrepreneur who has an incentive to not behave, due to the potential private benfits BI. Note two observations before we set up the lender's problem:

1. The project is never profitable if the entrepreneur chooses not to behave. This implies that the **(IC)** constraint must always hold. This means that the contract $R_b(I)$ must at least secure:

$$\begin{align}
    R_b^* = \dfrac{BI}{\Delta p}.
\end{align}$$

2. We assume that lenders operate under perfect competition. Thus in equilibrium the expected profits of a lender is driven to zero.

With this in mind define the expected profits from the lender $\pi_l^e$ as:

$$\begin{align}
\pi_l^e = p_H \left[R(I)-R_b\right]-\left(I-A\right)\geq0 \tag{IR}
\end{align}$$

Here we have used: (1) The lender will always make sure that the entrepreneur chooses $b$. Thus we only use $p_H$ as probability of success. (2) The lender offers $I-A$ for the project, as the entrepreneur puts up $A$ himself. The lender thus only enters into any contract, if his **rationality constraint (IR)** holds, i.e. if $\pi_l^e\geq0$.

#### The Entrepreneurial problem, part II

The entrepreneur knows that to land a contract, lender will always offer a contract ensuring that the **(IC)** constraint holds. Thus he only considers the outcomes where he behaves. Using that there are perfect competition for lenders, the entrepreneur chooses an $I$, such that the **(IR)** constraint holds with equality. The utility in this scenario for the entrepreneur is then given by:

$$\begin{align}
    u_b &= p_H R_b(I) -A \\
        &= p_H R(I)-I,
\end{align}$$
where $p_HR_b(I)$ is substituted for using the **(IR)** constraint. The problem of the entrepreneur that recognizes all of the above is then given by:

$$ \begin{align}
    \max_{I} p_HR(I)-I, && \text{s.t. }\overbrace{p_H\left[R(I)-\dfrac{BI}{\Delta p}\right]}^{\text{IR constraint s.t. IC constraint}}\geq I-A
\end{align}$$

The constraint states that the entrepreneur recognizes the IR constraint of the lender **and** that the lender offers an **IC** binding contract. Denote this the **(IR-IC)** constraint.

The first order condition characterizing the solution is then given by:

$$ \begin{align}
    p_HR'(I) &= 1+\dfrac{\mu}{1+\mu} \dfrac{p_HB}{\Delta p}\geq 1,  \tag{FOC 1}
\end{align} $$

along with the Karush-Kuhn-Tucker conditions

$$ \begin{align}
    \mu\left[p_H\left(R(I)-\dfrac{BI}{\Delta p}\right)-(I-A)\right]=0, && \mu\geq 0, &&  p_H\left(R(I)-\dfrac{BI}{\Delta p}\right)-(I-A)\geq 0 \tag{FOC 2 / KKT}
\end{align} $$

.

## Q1: How does investments vary with assets A?

Let us start with some comments on the solution:
* Note that if the **(IR-IC)** constraint is **not** binding, the shadow-variable $\mu=0$ and the optimum is given by $p_HR'(I) =1$.
* If the **(IR-IC)** constraint **is** binding, the level of $I$ is determined entirely from the **(IR-IC)** constraint.

To illustrate the solution, choose a type of technology function $R(I)$ and parameter values. Note the following conventions:
* Rtype can so far be of types 'sqroot' (sq), 'ln' (ln) or 'FlexiblePower' (fp). The three versions imply the following technologies:

$$\begin{align}
    &\text{For }(sq): && R(I)=\sqrt{I} \\
    &\text{For }(ln): && R(I)=ln(I) \\
    &\text{For }(fp): && R(I)=I^p
\end{align}$$

In the following $p$ denotes the power used in the (fp) case.
* We require $1>p_H>p_L>0$ and $p\in(0,1)$.

**1: Define variables**

```python
Rtype = 'FlexiblePower'
pH = 0.8
pL = 0.5
B = 2
p = 0.3
```

**2: Create a grid of assets and define the level of investments (I) implied by the IC constraint**

```python
Amin = 0 #minimum level of A to test
Amax = 2 # maximum level of A to test
N_A = 100 # How many levels of A between Amin and Amax to test
A = np.linspace(Amin,Amax,N_A) # creates a linearly spaced grid from Amin to Amax with N_A steps.
sol_IC = func.IR_constraint(Rtype,pH,pL,B,A,p) # User-defined function that solves for the investment level I, for each value of A from the array defined above.
```

**3: Define the level of investments that are optimal if the IC constraint is not binding**

This is given by (FOC 1) where the shadow-variable $\mu=0$. Note that this is independent of A

```python
sol_unconstrained = func.Istar(Rtype, pH, p)*np.ones(A.size) # User-defined function that solves for I* in unconstrained case. Re-defined as a vector of same size as A, to plot below.
```

**4: Plot the two functions**

Note that the IC constraint is violated **if** A is relatively small. Thus the *equilbrium investment behavior* is defined as the minimum of the two functions.

```python
# OPTIMAL FUNCTION:
sol_equilibrium = np.minimum(sol_IC, sol_unconstrained)

fig = plt.figure(frameon=False, figsize=(8, 6), dpi=100)
ax = fig.add_subplot(1, 1, 1)

ax.plot(A, sol_IC, linestyle='--')
ax.plot(A, sol_unconstrained, linestyle='--')
ax.plot(A, sol_equilibrium, linewidth=3)

ax.set_xlim([Amin, math.ceil(Amax)])

# This little code makes sure the scale of the y-axis is consistent with level of variables:
y_level = len(str(abs(int(max(sol_IC))))) # counts the number of digits (before .)
if max(sol_IC)<1:
    ax.set_ylim([0, (math.ceil(10*max(sol_IC))/10)])
else:
    ax.set_ylim([0, (math.ceil(10**(y_level-1)*max(sol_IC))/(10**(y_level-1)))])

# Labels:
ax.set_xlabel('Assets')
ax.set_ylabel('Investments')

# Legend:
plt.legend(('Binding IC constraint', 'Unconstrained solution', 'Equilibrium'),
           loc='upper right')

# Add a cool layout
fig.tight_layout()

#def update(pL=0.5,B=2):
#    sol_IC = func.IR_constraint(Rtype,pH,pL,B,A,p)
#    ax.plot(A,sol_IC)

#widgets.interact(update, pL = widgets.FloatSlider(
#                            description="$p_L$", min=0.1, max=0.5, step=0.05, value=0.5),
#                         B = widgets.FloatSlider(
#                            description="$B$", min=1, max=5, step=1, value=2),
#                );

```

**Results/comments:**

* If $A$ is sufficiently large $\Rightarrow \mu=0$ and $I$ is efficiently chosen.
* If not $\mu>0$ and investments are suboptimally low. The reason for this is the **moral hazard** of the scheme:
    * The entrepreneur receives private benefits of $BI$ from not behaving.
    * If $A$ is low entrepreneurs have to borrow a lot.
    * The private benefits from not behaving increases linearly with I; thus the lender has to offer a contract with linearly increasing benefits $R_b$ to avoid the bad behavior.
    * For the lender to avoid negative profits (IR constraint), the price of borrowing is increased.
    * The marginal benefits of the loan decreases with the size $(R''(I)<0)$. Thus if the loan is too large the optimal level $I^*$ will not be financed.

If you are further interested in how the optimal solution looks like, you can compare the baseline case, with a new instance below:

```python
# New parameter values:
Rtype = 'FlexiblePower'
pH2 = 0.9
pL2 = 0.8
B2 = 2
p2 = 0.3
```

```python
sol_IC2 = func.IR_constraint(Rtype,pH2,pL2,B2,A,p2)
sol_unconstrained2 = func.Istar(Rtype, pH2, p2)*np.ones(A.size)
sol_equilibrium2 = np.minimum(sol_IC2, sol_unconstrained2)

fig = plt.figure(frameon=False, figsize=(8,6), dpi=100)
ax = fig.add_subplot(1, 1, 1)
ax.plot(A, sol_equilibrium)
ax.set_xlim([Amin, math.ceil(Amax)])
ax.plot(A, sol_equilibrium2, linestyle='--')

# This little code makes sure the scale of the y-axis is consistent with level of variables:
y_level = len(str(abs(int(max(np.concatenate((sol_equilibrium, sol_equilibrium2))))))) # counts the number of digits (before .)
if max(sol_IC)<1:
    ax.set_ylim([0, (math.ceil(10*max(np.concatenate((sol_equilibrium, sol_equilibrium2))))/10)])
else:
    ax.set_ylim([0, (math.ceil(10**(y_level-1)*max(np.concatenate((sol_equilibrium, sol_equilibrium2))))/(10**(y_level-1)))])

# Labels:
ax.set_xlabel('Assets')
ax.set_ylabel('Investment')

# Legend:
plt.legend(('Original equilibrium', 'New equilibrium'),
           loc='upper right')
fig.tight_layout()

```

## Q2: How does the shadow-value of assets vary with A?

Let us start by defining how the optimal choice of investments $(I)$ change with A:

* If the IC constraint is not binding (when A is large), I was independent of A. Define the threshold $\bar{A}$ as the level where the IC constraint is no longer binding.
* If the IC constraint is binding, A increases $I$. Define the implicit function $I(A)$. By total differentiation of IC we then have:

$$ \begin{align}
    p_H\left[R'(I) -\dfrac{B}{\Delta p}\right]\dfrac{dI}{dA} = \dfrac{dI}{dA}-1, && \text{for }A<\bar{A},
\end{align} $$

and otherwise 0. Isolating for $dI/dA$ we then have:

$$\begin{align}
    \dfrac{dI}{dA} = \dfrac{1}{1-p_HR'(I)+\frac{p_HB}{\Delta p}}> 0, && \text{for }A<\bar{A}, \tag{MMIE}
\end{align} $$

where the denominator is positive. To see this substitute for (FOC 1) to get

$$ \begin{align}
1-p_HR'(I) + \frac{pH B}{\Delta p} = \underbrace{-\dfrac{\mu}{1+\mu}\dfrac{p_HB}{\Delta p}}_{\text{from FOC 1}}+\dfrac{p_HB}{\Delta p}  = \dfrac{p_HB}{\Delta p}\dfrac{1}{1+\mu}>0
\end{align} $$

Finally, what is then the shadow value of A? Differentiate the utility function:

$$ \begin{align}
    \dfrac{du_b}{dA} &= \left[p_HR'(I)-1\right]\dfrac{dI}{dA} \\
                     &= \underbrace{\dfrac{\mu}{1+\mu}\dfrac{p_H B}{\Delta p}}_{\text{From FOC 1}}\dfrac{dI}{dA}. \tag{SVE}
\end{align} $$

**Comments for the results:**

* When $A>\bar{A}: du/dA=0$. When A is large enough the optimal level of investments is ensured; sufficient access to credit.
* When $A<\bar{A}: du/dA>0$. When the entrepreneur is constrained in his investment, an increase in A relaxes credit constraint.
* The term $dI/dA$ in $(MMIE)$ is the Marginal Multiplier of Initial Equity.
* The term $du/dA$ in $(SVE)$ is the Shadow Value of Equity. In the face of imperfect information and moral hazard, the famous Modigliani and Miller (1963) result that the source of financing does not matter, breaks down.

# Exercise 6.1 in JT: Privately known private benefit and market breakdown

Let us start with a brief outline of the setup (from section 6.2), compared to exercise 3.5 (a lot of is the same, will not be repeated here):

* Two types of entrepreneurs: Good and bad types with private benefits of not behaving $B_H>B_L$. (good type has $B_L$)
* No equity (A=0),
* Investment is not continuous, but either 0 or I.
* Investment is either successfull (return R) or not (return 0).
* Capital markets put probability $\alpha\in(0,1)$ on type 'good' and $1-\alpha$ on type 'bad'.
* Regulary conditions:

$$ \begin{align}
    p_H\left(R-\dfrac{B_H}{\Delta p}\right)<I<p_H\left(R-\dfrac{B_L}{\Delta p}\right), && \text{and} && p_LR<I.
\end{align} $$

Recall that the IC condition for an entrpreneur was **behave if**: $\Delta p R_b \geq B$.

The regularity conditions thus state that:
1. It is not profitable for lenders to invest in project with 'bad' entrepreneurs **and** making sure that they behave. (first inequality)
2. It is profitable to invest in 'good type' entrepreneur **and** making sure that he behaves (second inequality).
3. It is not profitable to invest in **any** project, where the entrepreneur mis-behaves (third inequality).

## Q1: How does the level of rewards $R_b$ in case of success, change the equilibrium in various ranges?

Generally, there are three possible outcomes:

1. If $R_b<B_L/\Delta p$: Both IC constraints are violated $\Rightarrow$ all entrepreneurs misbehave $\Rightarrow$ no loan contracts are offered (third regularity condition)
2. If $R_b>B_H/\Delta p$: If the lender offers $R_b$ both types of entrepreneurs will state that they are of type 'bad' and receive the higher income. This is not profitable (first regularity condition). No loan contracts are offered.
3. If $B_L/\Delta p < R_b < B_H/\Delta p$: A range of values that induces 'good types' to work and may be profitable for the lender. Bad types identify themselves as 'good' types, but do not behave (as the IC constraint for them is not satisfied).

Thus only pooling equilibria in the 3. range are possible.

## Q2: Show that there exists a threshold $\alpha^*$, of good types, where financing occurs only if $\alpha>\alpha^*$

In Q1 we conluded that only pooling equilibria are feasible. Now we add the IR constraint for lender (no negative profits), given we know that only the pooling equilibria are feasible:

$$\begin{align}
    \underbrace{\left[\alpha p_H + (1-\alpha) p_L\right]}_{\text{probability of success}} \underbrace{\left(R-R_b\right)}_{\text{Revenue if success}} \geq I.
\end{align} $$

To find the threshold of $\alpha$ where this holds note:

* The left-hand side is increasing in $\alpha$,
* The left-hand side is decreasing in $R_b$; i.e. the contract (that is still feasible according to Q1) that offers the highest profits are $R_b=B_L/\Delta p$.

The threshold $\alpha^*$ is then defined by:

$$\begin{align}
    \left(\alpha^*p_H+(1-\alpha^*)p_L\right)\left(R-\dfrac{B_L}{\Delta p}\right) = I.
\end{align}$$

## Q3: Describe the 'cross-subsidies' between types that occur when borrowing is feasible.

Repetition of JT textbook terms on bullet form:

* **Adverse selection** ( pooling equilibrium $(\alpha\geq \alpha^*)$),
* **Over investment:** All entrepreneurs borrow (only 'good type' was optimal under perfect information)
* the quality of average lender drops,
* **Cross-subsidization:** 'Good type' subsidizes the 'bad type'.
* **Credit-rationing:** If $\alpha<\alpha^*$ the only pooling equilibrium is a collapse of the market; if the degree of adverse selection is too strong.

Simplest extensions to this setup, see: Signalling (Leland and Pyle (1977)), more general equilibrium descriptions (JT 13.1).

# Exercise 6.2 in JT: More on pooling in credit markets.

Compared to before, alter the setup as follows:

* Continuum of types instead of good/bad. For entrepreneur $i$ the value $B_i$ is distributed according to the CDF function $H(B)$, with support on $[0, \bar{B}]$
* Monopoly lender offers credit to borrowers (entrepreneurs).

The lender offers $R_b$ for a successfull investment, otherwise 0. Borrower $i$ then behaves **if** the IC constraint holds: $B_i\leq \Delta p R_b$. The expected profits from offering the contract is then defined by:

$$\begin{align}
    \pi(R_b) = H\big(\Delta p R_b\big)p_H(R-R_b)+\left[1-H\big(\Delta p R_b\big)\right] p_L(R-R_b)-I, \tag{Profits}
\end{align} $$

where $H(\Delta p R_b)$ measures the share of borrowers that behave, i.e. with $B_i<\Delta R_b$. From this note:

* The share of high-quality borrowers increase with $R_b$ (bad types start to behave).
* Same dynamics as before: Adverse selection reduces quality of lending, induces cross-subsidies between types.

Before proceeding to the questions, we illustrate the case where $H(B)$ is uniformly distributed.

```python
LowerBound = 0
UpperBound = 10
N = 100
x = np.linspace(LowerBound, UpperBound, N)
pH = 0.9
pL = 0.5
R = 10
I = 2
rv = stats.uniform(loc=LowerBound, scale=UpperBound-LowerBound)
Exp_profits = rv.cdf(x)*pH*(R-x)+(1-rv.cdf(x))*pL*(R-x)-I
def prof_func(y):
    return rv.cdf(y)*pH*(R-y)+(1-rv.cdf(y))*pL*(R-y)-I
# For what value of B does the profits cross zero?
Zero_profits = optimize.newton(prof_func, 0)
print(Zero_profits)
```

    7.500000000000001

```python
fig = plt.figure(frameon=False, figsize=(8, 6), dpi=100)
ax = fig.add_subplot(1, 1, 1)

ax.plot(x, rv.cdf(x))
ax.plot(x, rv.pdf(x))
ax.set_xlim([LowerBound, UpperBound])
ax.set_ylim([0, 1])

# Labels:
ax.set_xlabel('$B$')
#ax.set_ylabel('Share of entrepreneurs with $B_i\leq B$')
plt.legend(('Share of entrepreneurs with $B_i\leq B$', '$dH/dB_i$'),
           loc='upper left')

# Add a cool layout
fig.tight_layout()

fig2 = plt.figure(frameon=False, figsize=(8, 6), dpi=100)
ax2 = fig2.add_subplot(1, 1, 1)
ax2.plot(x,Exp_profits)
ax2.set_xlim([LowerBound, UpperBound])
ax2.set_ylim([math.floor(min(Exp_profits)), math.ceil(max(Exp_profits))])
ax2.set_xlabel('$B$')
ax2.set_ylabel('Expected profits')
plt.axvline(x=Zero_profits, color='k', linestyle='--')
plt.axhline(y=0, color='k')

plt.legend(('Expected profits' , 'Level of B implying zero profits' ),
          loc='lower left')
plt.text(Zero_profits+0.25, (math.ceil(max(Exp_profits))-math.floor(min(Exp_profits)))/2, 'Zero profits at B='+str(Zero_profits), bbox=dict(facecolor='darkblue', alpha=0.5))

fig2.tight_layout()
```

## Q: Show a number of results:
1. Proportion of high-quality borrowers is endogenous and depends on $R_b$.
2. Adverse selection reduces quality of lending
3. externality among types: Presence of low-quality borrower forces lender to charge interest rate, generating strictly positive profits on high-quality types.
4. Show an example of 'bad lemons' breaking down the market.

The first 3 are answered straightforwardly:

* A high quality borrower, is one that behaves. The share of high quality borrowers is defined as $H(\Delta p R_b)$. If $R_b$ increases, so does this share.
* Adverse selection means that the lender cannot e.g. **only** offer the contract $R_b$ to certain types / offer individual contracts based on their types $R_b^i(B_i)$.

Lastly, we give an example where the market breaks down. All we need for the market to break down, is to find a scenario where profits of the lender is negative:

* Assume that $p_L=0$, and that borrowers are **uniformly distributed** that is $H(B) = B/\bar{B}$, for $B\in[0,\bar{B}]$.
* In this case the expected profits are given by:

$$\begin{align}
    \pi(R_b) &= \overbrace{\dfrac{\Delta p R_b}{\bar{B}}}^{\equiv H(\Delta p R_b)} p_H (R-R_b)-I  \\
             &= \dfrac{p_H R_b}{\bar{B}}p_H(R-R_b)-I.
\end{align} $$

The profit maximizing level of $R_b$ is then given by: $R_b^* = R/2$, which yields profits of

$$\begin{align}
    \pi(R_b^*) &= \dfrac{p_H R/2}{\bar{B}}p_H\left(R-\dfrac{R}{2}\right)-I \\
               &= \dfrac{p_H^2R^2}{4\bar{B}}-I.
\end{align} $$

Thus as long as $I>p_H^2R^2/(4\bar{B})$ the market breaks down.

# Exercise 13.1 JT: Improved governance

The setup in this exercise:

* Two period model, $t=0,1$.
* Continuum of entrepreneurs with initial wealth $(A)$ distributed with the CDF $G(A)$ with support on $[0,I]$.
* At time $t=0$: Entrepreneurs can invest $I$ and borrow $(I-A)$ or invest in other firms.
* Entrepreneurs are risk neutral and thus have utility $u_e=c_0+c_1$.
* As before: Receive $R$ is investment is successful, with probability $p_H$ if entrepreneur behaves or $p_L$ if he shirks (misbehaves). Shirking yields private benefits of $B$.
* Now add consumers: Behavior results in an increasing savings function $S(r)$, with $S(r)=0$ for $r<0$.

## Q1: When are projects financed?

Consider the entrepreneur again (similar to the exercises above):

* He behaves if (IC) holds:

$$\begin{align}
    (\Delta p)R_b\geq B. \tag{IC}
\end{align}$$

* If (IC) holds profits are given by:

$$\begin{align}
    p_H\left(R-\dfrac{B}{\Delta p}\right). \tag{IC, $\pi$}
\end{align}$$

* Lenders will finance project as long as profits are larger than alternative cost:

$$\begin{align}
    (1+r)(I-A)\leq p_H\left(R-\dfrac{B}{\Delta p}\right), \tag{IR}
\end{align}$$

where the LHS is the profit from saving (alternative cost w. rent r) and the RHS is the profit from investing with a loan contract such that the (IC) constraint holds with equality.

* As we saw earlier the solution depends on the level $A$. All $A>A^*$ such that (IR) holds are financed.

## Q2: Give the expression determining the interest rate r. How does it change if investor protection improves (interpreted as fall in B)?

The **supply** side of the financial market consists of consumers with savings function $S(r)$ and entrepreneurs who do not get extended credit to carry out the project, i.e. all $A<A^*$. Thus:

$$\begin{align}
    \tilde{S}(r) = S(r)+\int_0^{A^*(r)}A\mbox{ }dG(A)
\end{align}$$

The **demand** side of the financial market consists of entrepreneurs with $A\geq A^*$:

$$\begin{align}
    D(r) = \int_{A^*(r)}^I (I-A)\mbox{ }dG(A)
\end{align}$$

Equilibrium balances demand and supply, i.e. $D(r)=\tilde{S}(r)$.

**Investor protection improves:** ($B$ falls)

* Motive for not behaving drops for all entrepreneurs. Thus entrepreneurs that were before marginally unprofitable to invest in, now increase the *pledgeable income* they can promise investors. In other words; the IC constraint is easier to satisfy, $R_b$ is lower.
* This lowers the threshold $A^*$, i.e. more entrepreneurs are extended credit. Demand increases.
* For supply to increase and adjust to the higher demand, the interest rate increaes.

**An extra comment on the model, The balance-sheet channel:**
Note that in this model, a marginal change in r has the effect on supply:

$$\begin{align}
    S'(r)+A^*(r)g(A^*(r))\dfrac{\partial A^*(r)}{\partial r}>0.
\end{align}$$
If $r$ increaess, less investment projects are profitable $\Rightarrow$ supply increases. The effet on demand is given by

$$\begin{align}
    -(I-A^*(r))dg(A^*(r))\leq 0
\end{align}$$

The implication is that *shocks that shift demand upwards (downwards) and supply downwards (upwards) can lead to large fluctuations in investment, without significant changes in the interest rate (r)*. Large fluctuations in investment, but small changes in r, is a stylized fact; something that we observe.

# Exercise from PS: Adverse selection with heterogenous ability

## The setup:

* Continuum of risk-neutral entrepreneurs (same initial $A$)
* Investment cost of cost $I$ (where $I>A$) yields $R$ in return w. probability $p$. Otherwise $0$ return.
* Agents differ in the probability of success $p$. Is distributed on $[0,1]$ according to the pdf function $f(p_i)$. (As usual $F(p)$ is the share of agents w. $p_i\leq p$. $f(p)$ is the marginal change in that probability around $p_i=p$.)
* Total supply of loanable funds is simply presented by the increasing function $S(r)$.
* This $S(r)$ is supplied by a competetive market (e.g. banks).
* Assume perfect information.
* Compared to before: No assumption of private benefits from not behaving.

## Q1: Individual rationality constraint, given that the agent has the choice between investment $I$ or depositing $A$.

This is straightforwardly given by:

$$\begin{align}
    p(R-R_b(p)) \geq (1+r)A \tag{IR}
\end{align}$$

Note that we assume that the payment $R_b$ may depend on $p$, as we assume **perfect information** for now. Note here:

* The LHS denotes profits from the investments after paying $R_b(p)$ on loan if successful. Otherwise zero.
* The RHS denotes the value of depositing $A$ at interest rate $r$.

## Q2: Write equations determining the riskless rate $r$, the identity of the marginal investor, and the zero (marginal) profit for the bank.

Let us start by imposing **zero profits** for the banks:

$$ \begin{align}
pR_b(p) = (1+r)(I-A), && \forall p \tag{ZP}
\end{align}$$

where $pR_b(p)$ is the payment if the project is successful, and the cost is the alternative cost of the loan: $(1+r)(I-A)$. Denote for now $B\equiv I-A$ the loan given by the bank. Note once again that the perfect information assumption implies that the zero profits hold for all $p$.

The **marginal investor** is given by the $\bar{p}$, where the IR constraint exactly holds. Imposing the zero profits condition we then have:

$$\begin{align}
  \bar{p}R-\underbrace{(1+r)(I-A)}_{\equiv \bar{p}R_b(\bar{p})\text{ from (ZP)}}=(1+r)A && \Rightarrow && \bar{p}(r) = \dfrac{1+r}{R}I
\end{align}$$

Finally the **riskless rate (r)** (it is riskless, due to the perfect information assumption) is determined by the equilibrium condition; i.e. that demand equals supply. With the threshold $\bar{p}$ all entrepreneurs who have a larger probability of success than the level $\bar{p}$, where profits are zero and (IR) holds with equality.

Recall that $f(p)$ denoted the distribution of the entrepreneurs. Thus demand for credit is extended to all entrepreneurs with $p\geq \bar{p}$, or:

$$\begin{align}
    D(r) = \int_{\bar{p}(r)}^1(I-A)f(p)\mbox{ }dp
\end{align}$$

Supply is given by the *aggregate supply function* $S(r)$, as well as the initial wealth $(A)$ from the entrepreneurs who do not obtain credit to carry out their investment. Define this new *total* supply function $\tilde{S}(r)$:

$$\begin{align}
    \tilde{S}(r) = S(r) + \int_0^{\bar{p}(r)} A f(p)\mbox{ }dp
\end{align}$$

## Q3: Confirm that only projects with positive net present value are carried out.

This is given, as banks have a zero profits condition, and entrepreneurs have the IR condition.

## Q4: Assume p is not observable. Compute equilibrium lending contract, the idendity of the marginal investor and show that there is overinvestment.

Now that the banks cannot observe $p$, they cannot make the contracts contingent on $p$. Thus they can only propose one contract, offering $R_b^*$ if the project is successful. Let $p^*$ denote the marginal investor now. From the IR constraint (with equality) we get:

$$\begin{align}
    p^*(R-R^*_b)=(1+r)A \tag{IR*}
\end{align}$$

Zero profits for the bank:

$$\begin{align}
    \int_{p^*}^1 p R_b^*f(p)\mbox{ }dp = \left(1- F(p^*)\right)(1+r)(I-A).
\end{align}$$

The LHS gives the payout to successful projects. The RHS is the alternative cost of the loan contracts. Note that $1-F(p^*)$ is the share of entrepreneurs with $p\geq p^*$.

Note that we can define this as the **conditional expected value**:

$$\begin{align}
    \dfrac{\int_{p^*}^1 p R_b^*f(p)\mbox{ }dp}{1- F(p^*)} \equiv R_b^* \mathbb{E}\left[p|p\geq p^*\right] = (1+r)(I-A).  \tag{ZP*}
\end{align}$$

Combining the zero profits, and the marginal investor identity we then have:

$$\begin{align}
        (1+r)A&= p^*R-p^*R_b^* \\
              &= p^*R-R_b^* \mathbb{E}\left[p|p\geq p^*\right] \dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]} \\
              &= p^*R-(1+r)*(I-A)*\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}
\end{align}$$

where the last line uses $(IR^*)$.

Rearranging this expression we get:

$$\begin{align}
    p^*R = I (1+r)\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}+A(1+r)\left[1-\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}\right]\leq (1+r)I \tag{Marginal*}
\end{align}$$

The RHS is a linear combination of $(1+r)I$ and $(1+r)A$. Compared to this the marginal investor was defined by $\bar{p}R=(1+r)I$ previously. As $I>A$ this establishes the inequality in (Marginal$^*$). Finally this entails directly that $p^*\leq \bar{p}$.

This inequality entails **overinvestment**:

* When $p$ is observed by the bank, the marginal investor is identified, such that *marginal* profits are zero.
* When $p$ is not observed by the bank, the marginal investor is identified, such that *average* profits are zero.
* Thus credit is extended to types that are unprofitable, $p^*\leq \bar{p}$.


# Design an A/B Test

[Cédric Campguilhem](https://github.com/ccampguilhem/Udacity-DataAnalyst), March 2018

## Table of contents

- [Introduction](#Introduction)
- [Experiment design](#Design)
    - [Metric choice](#Metric)
    - [Measuring standard deviation](#Standard deviation)
    - [Sizing](#Sizing)
- [Experiment analysis](#Analysis)
    - [Sanity checks](#Sanity)
    - [Result analysis](#Result)
    - [Recommendations](#Recommendations)
- [Follow-up experiment](#Followup)
- [Appendix](#Appendix)

## Introduction [*top*](#Top)

This project is related to A/B testing course for Udacity Data Analyst Nanodegree program. The purpose of this project is to analyse an experiment made at Udacity.

The experiment is related to a change when student clicks "start free trial" button. A message asks them how much time they would dedicate to the course. If the student indicated 5 or more hours per week, they would be taken through the checkout process as usual. If they indicated fewer than 5 hours per week, a message would appear indicating that Udacity courses usually require a greater time commitment for successful completion, and suggesting that the student might like to access the course materials for free. At this point, the student would have the option to continue enrolling in the free trial, or access the course materials for free instead.

The hypothesis was that this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time—without significantly reducing the number of students to continue past the free trial and eventually complete the course. If this hypothesis held true, Udacity could improve the overall student experience and improve coaches' capacity to support students who are likely to complete the course.

## Experiment design [*top*](#Top)

### Metric choice [*Experiment design*](#Design)

The following parameters have been selected as invariants of the analysis (i.e. parameters which should not be affected by the change being analyzed).

Invariant                    | Description
:----------------------------|:---------------------------------
Number of cookies            | Number of unique cookies to view the course overview page.
Number of clicks             | Number of unique cookies to click on "Start free trial" button
Click-through-probability    | Number of unique cookies to click on "Start free trial" button divided by the number of unique cookies to view the course page overview.

The following parameters have been selected as evaluation metrics because it collects information downstream to change and are related to the objectives of this A/B test which are:

- minimizing the proportion of enrolled students quiting during the trial
- keeping the same proportion of students clicking the start free trial and continuing the course afterwards

Evaluation metrics  | Description | Practical significance boundary
:-------------------|:------------|:-------------------------------
Gross conversion    | Number of user-ids to enroll in the free trial divided by the number of unique cookies to click on the "Start free trial" button. | ${d}_{min} = 0.01$
Retention           | Number of user-ids to remain enrolled after the trial divided by the number of user-ids enrolled during the trial. | ${d}_{min} = 0.01$
Net conversion      | Number of user-ids to remain enrolled after the trial divided by the number of unique cookies to click the "Start free trial" button. | ${d}_{min} = 0.0075$

Reasons of choice for evaluation and invariant metrics:

- **Number of cookies**: Is a good invariant metric because it's being directly randomized between experiment and control group. Is a bad evaluation metric because it won't be different between experiment and control.
- **Number of user-ids**: Is a bad invariant metric because the experiment may change the number of users involved in the free trial period. It's only at enrollment that a user checkout is made. Could have been used as an evaluation metric but is redundant with gross conversion. The later one has the advantage of being normalized so that comparisons between control and experiment group are easier.
- **Number of clicks**: Is a good invariant as it is recorded before user has a chance to see the change brought by the experiment. Is a bad evaluation metric for the same reason: it's recorded upstream to the change.
- **Click-through-probability**: For the same reason that the number of clicks, the click-through-probability is recorded upstream to the change. It is then a good invariant and a bad evaluation metric.
- **Gross conversion**: Is a bad invariant because it's recorded downstream to change and may be affected by it. Is a good metric because it brings information related to the hypothesis being tested by capturing the proportion of students changing their mind after the time commitment warning.
- **Retention**: Is a bad invariant because it's recorded downstream to change and may be affected by it. Is a good evaluation metric because it brings information related to the hypothesis being tested by assessing the first objective: minimize number of students quitting during the trial.
- **Net conversion**: Is a bad invariant because it's recorded downstream to change and may be affected by it. Is a good evaluation metric because it is relevant to the second objective being tested: keep the same proportion of students enrolled in the long term.

To launch this experiment we expect the gross conversion and retention to decrease with practical significance and the net conversion not to decrease with practical significance.

### Measuring standard error [*Experiment design*](#Design)

The standard error for each evaluation metrics will be calculated with the following data:

Parameter | Value
:---------|:------
Unique cookies to view course overview page per day |	40000
Unique cookies to click "Start free trial" per day | 3200
Enrollments per day |	660
Click-through-probability on "Start free trial" |	0.08
Probability of enrolling, given click |	0.20625
Probability of payment, given enroll |	0.53
Probability of payment, given click	| 0.1093125

The *probability of enrolling, given click* is linked to the *gross conversion* metric. *Probability of payment, given enroll* is in relation with the *retention* metric. Finally, *probability of payment, given click* is related to *net conversion* metric. As we are dealing with probabilities, we will assume to have a binomial distribution. We can then estimate the standard error for each metric using the binomial standard deviation:

\begin{align}
SE = \sqrt{\frac{p(1-p)}{n}}
\end{align}

Where:
- p is the probability of event
- n is the number of repetitions of the event

```python
nb_cookies = 5000.
nb_clicks = nb_cookies * 3200. / 40000.
nb_enrollments = nb_clicks * 660. / 3200.
print nb_clicks, nb_enrollments
```

    400.0 82.5

```python
import math
stddev_gross = math.sqrt(0.20625 * (1 - 0.20625) / nb_clicks)
stddev_retention = math.sqrt(0.53 * (1 - 0.53) / nb_enrollments)
stddev_conversion = math.sqrt(0.1093125 * (1 - 0.1093125) / nb_clicks)
print stddev_gross, stddev_retention, stddev_conversion
```

    0.020230604137 0.0549490121785 0.0156015445825

The sample size is 5000 cookies. We can then assume that we will have 400 clicks on the "Start free trial button" and 82.5 enrollments. The standard deviations are reported in the table below:

Evaluation metrics | Units of analysis (n) | Estimated standard deviation
:------------------|:----------------------|:----------------------------
Gross conversion   | cookie (400)          | 0.0202
Retention          | user-id (82)          | 0.0549
Net conversion     | cookie (400)          | 0.0156

Gross conversion and net conversion use cookie as unit of analysis and unit of diversion, so the analytical standard error calculated here shall be quite close from empirical values. This is not the case for retention metrics as it uses user-id and we could have differences between empirical variability and the one estimated above.

### Sizing [*Experiment design*](#Design)

#### Number of samples vs power

We have multiple hypothesis tested in this experiment. Testing multiple hypothesis increases the risk of making type I error (incorrect rejection of null hypothesis). The Bonferroni correction may be used in a context where type I error shall be avoided but is discouraged where type II errors shall be avoided (see this [paper](https://www.onlinelibrary.wiley.com/doi/pdf/10.1111/opo.12131) for reference). I will not use Bonferroni correction to avoid type II error (incorrect retaining of null hypothesis).

I have used the online calculator provide by [Evan Miller](http://www.evanmiller.org/ab-testing/sample-size.html) to estimate sample size for A/B test. The results are provided in the table below:

Parameter            | Base conversion rate | Practical significance | $\alpha$ | $1 - \beta$ | Sample size per variation
:--------------------|:---------------------|:-----------------------|:---------|:------------|:-------------------------
Gross conversion     | 20.625 %             | 1.0 %                  | 5.0 %    | 80.0 %      | 25835
Retention            | 53.0 %               | 1.0 %                  | 5.0 %    | 80.0 %      | 39115
Net conversion       | 10.93125 %           | 0.75 %                 | 5.0 %    | 80.0 %      | 27413

The retention metrics is the one requiring the most samples per variation. But as this metrics is also using user-id as units of analysis, it also need to be converted to clicks, increasing again the number of page views (only 8% of view lead to clicks):

\begin{equation}
{pageviews} = \frac{39115 * 2}{0.08 * 0.20625}
\end{equation}

The equation above assumes that both control and test groups are seeing the same number of pages and leads to 4741212 page views.

```python
print 1 - (0.95 * 0.95)
print 39115. / (0.08 * 0.20625) * 2.
```

    0.0975
    4741212.12121

#### Duration vs exposure

We have 40000 unique cookies to view course overview per day. If we redirect half of the traffic, the duration would be:

\begin{equation}
duration = \frac{4741212}{40000 * 0.5}
\end{equation}

The equation above leads to 238 days ! That's a long experiment and Udacity does not want to spend that long. We need to rework some of the previous decisions we have made.

The retention metric is really demanding in terms of page views. If we drop this metric, the dimensionning metric is net conversion which now requires 685325 page views. If we increase the redirection factor to two-third of the traffic, this lead to a duration of 26 days which is much more manageable.

```python
print 4741212 / (40000 * 0.5)
```

    237.0606

```python
print 27413 / 0.08 * 2.
print 685325.0 / (40000 * 0.66)
```

    685325.0
    25.959280303

This desing exposes one third of students to a new feature during less than one month. The nature of feature is to minimize students starting the free trial without willing to dedicate more than 5 hours a week to follow the course. This feature shall not change the mind of students wanting to take the course and agreeing to dedicate a long time to it. Additionally no personal data is required and so there is no risk in terms of ethics. Also, the nature of change is harmless to the users. Overall, running this test is probably an acceptable risk.

## Experiment analysis [*top*](#Top)

### Sanity checks [*Experiment analysis*](#Analysis)

With a alpha of 0.05, the critical $z^*$ value for a two tailed test is 1.96.

The total number of cookies in each group is:

- experiment group: $n_1$
- control group: $n_2$
- probability to be in experiment group (by design): $p=0.5$
- observed probability to be in experiment group: ${p}_{obs}=\frac{n_2}{n_1+n_2}$

The pooled standard error is:

\begin{equation}
SE = \sqrt{\frac{p(1-p)}{n_1+n_2}}
\end{equation}

The margin of error is:

\begin{equation}
margin = SE * z^*
\end{equation}

The confidence interval is:

\begin{equation}
CI = [0.5 - margin, 0.5 + margin]
\end{equation}

The sanity check is passed if observed probability $p_{obs}$ is within confidence interval.

```python
import scipy.stats

#Probability
alpha = 0.05
z_star = -scipy.stats.norm.ppf(alpha / 2.) #two-tailed tests
n1 = 344660.
n2 = 345543.
p = 0.5
p_obs = n2 / (n1 + n2)

#Pooled Standard error
SE = math.sqrt(p * (1 - p) / (n1 + n2))

#Confidence interval
me = SE * z_star
ci = p - me, p + me

#Measure
print z_star, SE, ci, p_obs, ci[0] <= p_obs <= ci[1]
```

    1.95996398454 0.000601840740294 (0.49882041382459419, 0.50117958617540581) 0.500639666881 True

For number of cookies:
- confidence interval: [0.4988, 0.5012]
- observed probability: 0.4994
- sanity check: **passed**

```python
#Probability
alpha = 0.05
z_star = -scipy.stats.norm.ppf(alpha / 2.) #two-tailed test
n1 = 28325.
n2 = 28378.
p = 0.5
p_obs = n2 / (n1 + n2)

#Pooled Standard error
SE = math.sqrt(p * (1 - p) / (n1 + n2))

#Confidence interval
me = SE * z_star
ci = p - me, p + me

#Measure
print z_star, SE, ci, p_obs, ci[0] <= p_obs <= ci[1]
```

    1.95996398454 0.0020997470797 (0.49588457134714631, 0.50411542865285364) 0.500467347407 True

For number of clicks:
- confidence interval: [0.4959, 0.5041]
- observed probability: 0.5005
- sanity check: **passed**

```python
#Probability
alpha = 0.05
z_star = -scipy.stats.norm.ppf(alpha / 2.) #two-tailed test
n1 = 344660.
c1 = 28325.
n2 = 345543.
c2 = 28378.
p = c2 / n2
p_obs = c1 / n1

#Standard error
SE = math.sqrt(p * (1 - p) / n2)

#Confidence interval
me = SE * z_star
ci = p - me, p + me

#Measure
print z_star, SE, ci, p, p_obs, ci[0] <= p_obs <= ci[1]
```

    1.95996398454 0.000467068276555 (0.081210376574208529, 0.083041250574945116) 0.0821258135746 0.0821824406662 True

For the click-through-probability we use a slightly different approach. The average click-through-probability is 0.0821 based on control group. We are no longer using a pooled standard error. We then need to check that click-through-probability for experiment group lies within the confidence interval:

- confidence interval: [0.0812, 0.0830]
- observed probability: 0.0822
- sanity check: **passed**

### Result analysis [*Experiment analysis*](#Analysis)

#### Effect size tests

The dataset records 23 days of experiment in terms of unique cookies, clicks, enrollments and payments. This duration is lower than the duration we have designed. All other things being equal, this means that we are losing statistical power $1 - \beta$ (percent of the time the minimum effect size will be detected, assuming it exists). A reduction of samples is related to an increase of $\beta$.

That being said, we can calculate confidence interval and state whether each evaulation metric is statistically significant and practically significant).

The method used to calculate confidence interval changes:

\begin{equation}
p = \frac{events_{control} + events_{experiment}}{clicks_{control} + clicks_{experiment}} \\
SE = \sqrt{p(1-p)*\Bigl(\frac{1}{clicks_{control}}+\frac{1}{clicks_{experiment}}\Bigr)} \\
margin = SE * z^* \\
d = \frac{events_{control}}{clicks_{control}} - \frac{events_{experiment}}{clicks_{experiment}} \\
CI = [d - margin, d + margin]
\end{equation}

The $z^*$ value is calculated from alpha assuming a two-tailed test:

\begin{equation}
z^* = 1.96
\end{equation}

A metric is statistically significant if 0 is not included in the confidence interval (there is high chance that there is a difference between experiment and control). Additionaly, it becomes practically significant if the practical difference $d_{min}$ is not in the confidence interval: there is high chance that business sees a difference.

```python
z_star = -scipy.stats.norm.ppf(0.05 / 2.) #two-tailed test + Bonferroni correction
#z_star = -scipy.stats.norm.ppf(0.025 / 2.) #two-tailed test + Bonferroni correction
print z_star
```

    1.95996398454

```python
#Gross conversion
events_control = 3785
clicks_control = 17293.
events_experiment = 3423
clicks_experiment = 17260.
d_min = 0.01
p = (events_control + events_experiment) / (clicks_control + clicks_experiment)
SE = math.sqrt(p*(1-p)*(1./clicks_control + 1./clicks_experiment))
margin = SE * z_star
d = events_experiment / clicks_experiment - events_control / clicks_control
CI = d - margin, d + margin
print p, SE, margin, d, CI, not(CI[0] <= 0. <= CI[1]), not(CI[0] <= d_practical <= CI[1])
```

    0.208607067404 0.00437167538523 0.00856832630714 -0.0205548745804 (-0.029123200887504669, -0.011986548273218461) True True

```python
#Net conversion
events_control = 2033
clicks_control = 17293.
events_experiment = 1945
clicks_experiment = 17260.
d_min = 0.0075
p = (events_control + events_experiment) / (clicks_control + clicks_experiment)
SE = math.sqrt(p*(1-p)*(1./clicks_control + 1./clicks_experiment))
margin = SE * z_star
d = events_experiment / clicks_experiment - events_control / clicks_control
CI = d - margin, d + margin
print p, SE, margin, d, CI, not(CI[0] <= 0. <= CI[1]), not(CI[0] <= d_practical <= CI[1])
```

    0.115127485312 0.00343413351293 0.00673077800345 -0.00487372267454 (-0.011604500677993734, 0.0018570553289054001) False True

The results are reported in the table below:

Evaluation metric | Lower bound | Upper bound | Statistical significance | Practical significance
:-----------------|:------------|:------------|:-------------------------|:----------------------
gross conversion  | -0.0291     | -0.0120     | Yes                      | Yes
net conversion    | -0.0116     |  0.0019     | No                       | No

#### Sign tests

We have records of 23 days of experiment. The number of enrollments is higher in the experiment group in 4 days (in a row). The number of payments is higher in the experiment group in 10 different days.

With the use of [GraphPad](https://www.graphpad.com/quickcalcs/binomial1.cfm), setting the number of "successes" to 4 and 10 respectively and a probability of 0.5 we get the following numbers:

Gross conversion:
- Number of successes: 4
- Probability: 0.5
- Two-tail p-value: 0.0026
- Alpha value: 0.025 (0.05 divided by 2 for a two-tailes test)
- The result is statistically significant.

Net conversion:
- Number of successes: 10
- Probability: 0.5
- Two-tail p-value: 0.6776
- Alpha value: 0.025 (0.05 divided by 2 for a two-tailes test)
- The result is statistically unsignificant.

#### Summary

As introduced in section [Number of samples vs power](#Sizing), I have not used Bonferroni correction. Bonferroni correction shall not be used where type II errors have to be avoided.

In this experiment, the null hypothesis could be written like this:

- $H_{0,1}$: the proportion of students quitting during the trial period remains the same.
- $H_{0,2}$: the proportion of students clicking the "start trial" button and continuing after the trial period remains the same.

The second null hypothesis is critical: if the proportion decreases, it may affect the Udacity revenues.

A type II error in this contect would be not to find a change in the proportion of students continuing after the trial period while there is actually a decrease in the proportion. This is probably not an acceptable risk for the business.

### Recommendations [*Experiment analysis*](#Analysis)

The experiments shows that the impact on gross conversion is both statistically and practically significant. The change in net conversion is unsignificant.

The initial objective of the change is:

- minimizing the proportion of enrolled students quiting during the trial (retention)
- keeping the same proportion of students clicking the start free trial and continuing the course afterwards (net conversion)

As a reminder, the null hypothesis are:

- $H_{0,1}$: the proportion of students quitting during the trial period remains the same.
- $H_{0,2}$: the proportion of students clicking the "start trial" button and continuing after the trial period remains the same.

The significant (statistical and practical) decrease of gross conversion shows that number of students enrolled in the free trial has decreased. We are sure about that because both lower and upper bounds of confidence interval are negative numbers.

Yet, the experiment failed to reject the second null hypothesis. Failing to reject the null hypothesis is not a proof that null hypothesis is valid. The confidence interval for net conversion ranges from negative values to positive values, meaning that there is a risk that proportion of students remaining enrolled after the free trial decreases.

This is a risk of revenue loss for Udacity and I would not recommend implementing the change.

## Follow-up experiment [*top*](#Top)

I really wanted to follow the Data Analyst course, to me quitting before the end of free trial was not an option. I already knew that I had to dedicate time to the course, my feeling is that you cannot learn something if you are not willing to spend some time with it !

However, as a French, I must say that the way the Udacity courses work is pretty different than my experience with French colleges and high schools. And I would have advised any of my colleagues interested in the course to have a taste of that way before taking commitment. The balance between theory and practical examples is completly in opposition. Udacity course offers much more practical examples than theory. I like it, but it's pretty different to what I had experienced so far.

I am not sure how other countries balance theory and practice in their education. But different cultures may lead to different kind of frustrations.

One experiment idea would be to suggest a specific lesson explaining educational choices made by Udacity to give a taste to students before enrollment. The idea would be similar to the experiment conducted here: reduce frustrations of students in the free trial period while focusing Udacity coaches on long-term enrolled students without decreasing significantly long-term enrollments. The specific lesson may not have any coaching at all and students willing to start a new course would be redirected to such lesson first (whatever the course they have chosen).

Due to the very close nature of the experiment with the one we have conducted here, I would make almost the same design choices:

- Evaluation metrics: net conversion, gross conversion, retention.
- Invariants: number of unique cookies, number of clicks on "start free trial" button, click-through-probability for "start free trial button".

The risk of taking such an experiment is the entry cost with no guarantee of break even:

- Modification of the user interface to redirect free trials to that specific lesson
- Creation of the lesson itself requires some time

I would keep the same unit of diversion (cookies) as this specific lesson would not require user checkout. Without changing the practical significane, the duration of the experiment would be the same than this one: 26 days if we consider a redirection of two-third of traffic.

The Bonferroni correction would not be used either to prevent the risk of not detecting decrease in net conversion while there is.

The test may be conducted at the same time than other tests as long as they do not overlap (other test related to free trials and enrollments) with this experiment. This would increase the risk of false positives (type I error) just like if we were considering more null hypotheses. Preventing such increase of type I error would be possible by using correction like Bonferroni but at the risk of increasing type II errors. As this is something we want to avoid I would discourage any other overlapping experiment.

If at the end of this experiment gross conversion decreases practically and statistically significantly and net conversion does not decrease practically and statistically significantly then this new feature may be launched.

## Appendix [*top*](#Top)

```python
#Convert notebook to html
!jupyter nbconvert --to html --template html_minimal.tpl Design_an_AB_test.ipynb
```

    [NbConvertApp] Converting notebook Design_an_AB_test.ipynb to html
    [NbConvertApp] Writing 285196 bytes to Design_an_AB_test.html

[Evan Miller](http://www.evanmiller.org/ab-testing/sample-size.html) online calculator for A/B tests sizing.

[Discussion](https://www.widerfunnel.com/3-mistakes-invalidate-ab-test-results/) on whether or not to use Bonferroni correction. [Paper](https://www.onlinelibrary.wiley.com/doi/pdf/10.1111/opo.12131) on the same theme.

[Bonferroni](https://en.wikipedia.org/wiki/Bonferroni_correction) correction on Wikipedia.

Multiple A/B tests at the [same time](https://conversionxl.com/blog/can-you-run-multiple-ab-tests-at-the-same-time/) ?