    Total cumulative mutual information

 created by:
 Benjamin Regler1,
 Matthias Scheffler1,
 and Luca M. Ghiringhelli 1

1 Fritz Haber Institute of the Max Planck Society, Faradayweg 4-6, D-14195 Berlin, Germany

[Last updated: February 28, 2022]

This interactive notebook introduces the concepts and the original implementation of total cumulative mutual information (TCMI) to reproduce the main results presented in the publication:

B. Regler, M. Scheffler, and L. M. Ghiringhelli: "TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions" [arxiv:2001.11212] [pdf]

TCMI is a measure of the relevance of mutual dependencies based on cumulative probability distributions. TCMI can be estimated directly from sample data and is a non-parametric, robust and deterministic measure that facilitates comparisons and rankings between feature sets with different cardinality. The ranking induced by TCMI allows for feature selection, i.e. the identification of the set of relevant features that are statistical related to the process or the property of a system, while taking into account the number of data samples as well as the cardinality of the feature subsets.

It is compared to [Cumulative mutual information (CMI)](https://dx.doi.org/10.1137/1.9781611972832.22), [Multivariate maximal correlation analysis (MAC)](http://proceedings.mlr.press/v32/nguyenc14.html), [Universal dependency analysis (UDS)](https://dx.doi.org/10.1137/1.9781611974348.89), and [Monte Carlo dependency estimation (MCDE)](https://dx.doi.org/10.1145/3335783.3335795).

This repository (notebook and code) is released under the [Apache License, Version 2.0](http://www.apache.org/licenses/). Please see the [LICENSE](LICENSE) file.

---

**Important notes:**

All comparisons have been computed with the Java package MCDE v1.0 written in Scala, which is not part of the repository due to licensing issues. To download the package, please visit https://github.com/edouardfouche/MCDE-experiments. To build the package, use the sbt build command (sbt compile, sbt package, sbt assembly). Then, copy the resulting java package into the assets folder, rename it to mcde.jar, and run all examples with 50,000 iterations.

For the sake of simplicity, all results have been cached. However, results can be recalculated after adjusting the respective test sections. Depending on the test, the calculation time ranges from minutes to days.

---

```python
# Toggle caching
use_cache = True
```

## Import modules

```python
import os
import re
import shutil
import joblib
import warnings
import functools
import itertools
import multiprocessing

import numpy as np
import scipy as sp
import pandas as pd
import lightgbm as lgbm
import matplotlib as mpl
import matplotlib.pyplot as plt

# Our package
import tcmi
from tcmi import utils
from tcmi import entropy
from tcmi.cache import Cache
from tcmi.subspace_search import get_subspaces
from tcmi.estimators import DependenceEstimator
from tcmi.model_selection import RepeatedSortedStratifiedKFold, get_statistics

def get_storage_data(key, default=None, overwrite=False, force=False):
    """Wrapper for storage access. Uses use_cache.
    """
    return storage.get(key, default) if not force and (use_cache or overwrite) else default

# Main loop
if __name__ == '__main__':
    # Provide cache
    storage = Cache('data')

    # Configure plot environment
    mpl.rc('font', family='sans', size=14)
    mpl.rcParams.update({
        'figure.facecolor': (0, 0, 0, 0),
        'axes.facecolor': (0, 0, 0, 0),
        'xtick.labelsize': 14,
        'ytick.labelsize': 14
    })

    # Define colors
    cmap1 = plt.get_cmap('cividis')
    cmap2 = plt.get_cmap('RdBu_r')
    cmap_neutral = plt.get_cmap('binary')

    neutral_color1 = '#666666'
    neutral_color2 = '#999999'
    neutral_color3 = '#cccccc'

    # General settings (please do not touch)
    kwargs = dict(n_jobs=-1, return_scores=True)
    seed = 2019
```

## 1. Basic tests

This section studies some of the properties of total cumulative mutual information. In particular, we check that the

- score is a monotonous function in the order of the conditionals
- score attains it's maximum and minimun theoretical values (linear and zero case)
- correction vanishes with increasing number of data samples
- adjusted version of the score is (almost) constant with respect to subset dimensionality and sample size

```python
# Test case 1
methods = ['cmi', 'mac', 'uds', 'mcde']
size = 200

# Test case 2
sizes = [10, 50, 100, 500]
n_repeats = 50
dimensions = 4
```

### 1.1. Monotonicity check and ranking of monotonous functions

**Test**: Monotonicity check of score

**Expected**: linear must be first, followed by step functions, zero must be last

```python
rng = np.random.RandomState(seed=seed)
tests = {
    # Common operations
    'linear': np.arange(size),
    'exponential': np.exp(np.linspace(0, 1, num=size)),

    # Adding copies of values
    'step_2': np.repeat(np.arange(size // 2), 2),
    'step_4': np.repeat(np.arange(size // 4), 4),
    'step_8': np.repeat(np.arange(size // 8), 8),

    # Interleave copies
    'sawtooth_2': np.stack(tuple(zip(*[np.arange(2) for i in range(size // 2)])),
                     axis=1).flatten(),
    'sawtooth_4': np.stack(tuple(zip(*[np.arange(4) for i in range(size // 4)])),
                     axis=1).flatten(),
    'sawtooth_8': np.stack(tuple(zip(*[np.arange(8) for i in range(size // 8)])),
                     axis=1).flatten(),

    'random': rng.random_sample(size),
    'zero': np.zeros(size)
}

ranks = {}
output = np.arange(size) + 1

for name, value in tests.items():
    # Compute scores from other dependency measures for comparison
    key = 'monotonicity-check-' + name
    scores = get_storage_data(key, [], overwrite=True)

    if not scores:
        score, scores = entropy.cumulative_mutual_information(output, (value, ), **kwargs)
        scores = list(s.mean() for s in scores)

        # Compute scores from other dependency measures for comparison
        for method in methods:
            estimator = DependenceEstimator(method=method, n_jobs=-1)
            score = estimator.score(value, output)
            scores.append(score)

    ranks[name] = [np.around(s, decimals=4) for s in scores]

    # Compute correlation
    correlation = 0
    if np.unique(value).size > 1:
        correlation = sp.stats.spearmanr(output, value)[0]
    ranks[name].insert(0, np.around(correlation**2, decimals=4))

# Show ranking
ranks = sorted(ranks.items(), key=lambda x: x[1], reverse=True)
index, data = tuple(zip(*ranks))
data = np.array([(rho, total, s, np.sum([s01, s02])) + tuple(rest)
                 for rho, total, s, s01, s02, *rest in data])

columns = ['rho2', 'adjusted_score', 'score', 'score0']
columns.extend(methods)

ranks = pd.DataFrame(data, index=index, columns=columns)
ranks
```

### 1.2. Dependence of the correction term

**Test**: Baseline correction dependence with respect to number of data samples

**Expected**: Baseline correction monotonically decreases in the number of data samples

```python
key = 'correction-term'
steps = np.concatenate((np.linspace(1, 9, num=9), np.linspace(10, 100, num=10)))

values = get_storage_data(key, [])
if len(values) == 0:
    values = np.zeros(len(steps), dtype=np.float_)
    for i, size in enumerate(steps):
        output = np.arange(size) + 1
        if size == 1:
            values[i] = 1
            continue

        ce = entropy.cumulative_entropy(output)
        hce0 = entropy.cumulative_baseline_correction(output, output, n_jobs=-1)
        score0 = 1 - hce0 / ce

        # Save average score
        values[i] = score0.mean()

##
# Plot dependence
##
offset = 19
fig, ax = plt.subplots(figsize=plt.figaspect(1), dpi=100)

# Plot baseline correction
ax.plot(steps[:offset], values[:offset], '-o', color=cmap1(0.1),
        clip_on=False, linewidth=2, label=r'$\langle \hat{\mathcal{D}}^{(\prime)}_0(Y; X) \rangle$')
ax.plot(steps[:offset], steps[:offset]**(-2/3), color=cmap2(0.9),
        linestyle='--', label='Fit')

# Show approximate relationship
ax.annotate(r'$\langle \hat{\mathcal{D}}^{(\prime)}_0(Y; X) \rangle \sim n^{-2/3}$',
            (20, 0.4), color=cmap2(0.9),fontsize=15)

# Plot styles
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

ax.spines['left'].set_position(('outward', 10))
ax.spines['bottom'].set_position(('outward', 10))

# Set axis limits
ax.set_xlim(0, steps[offset - 1])
ax.set_ylim(0, 1)

ax.set_xlabel('Sample size $n$')
ax.set_ylabel(r'$\langle \hat{\mathcal{D}}_0(Y; X) \rangle\ /\ \langle \hat{\mathcal{D}}_0^\prime(Y; X) \rangle$')

ax.legend(loc='upper right', facecolor='w', frameon=False, bbox_to_anchor=(1.1, 1.05))
plt.tight_layout()
plt.show()

# Show table
pd.DataFrame(np.atleast_2d(values), columns=steps)
```

### 1.3. Baseline correction

**Test**: Baseline correction of measure with respect to number of samples and dimension

**Expected**: Constant scores for different dimensionalities and number of samples

```python
results = {}
for size in sizes:
    output = np.arange(size) + 1
    print('\n:: Size: {:d}\n'.format(size))

    dtypes = [('dimension', np.int_), ('score', np.float_)]
    cache_key = 'baseline_correction.method={:s}_size={:d}_dimension={:d}_repeats={:d}'

    # Compute scores from other dependency measures for comparison
    for method in methods + ["tcmi"]:
        print("/**\n"
              " * Method: {:s}\n"
              " */\n".format(method))

        if method == "tcmi":
            dtypes = [('total_score', np.float_), ('score', np.float_),
                      ('score_corr', np.float_), ('score0', np.float_)]
        else:
            dtypes = np.float_

        scores_dimensions = np.arange(dimensions + 1, dtype=np.int_)
        scores_mean = np.zeros(scores_dimensions.size, dtype=dtypes)
        scores_std = np.zeros(scores_dimensions.size, dtype=dtypes)
        bucket = results.setdefault(method, {})

        for i, dimension in enumerate(scores_dimensions):
            print('- Dimension: {:d} '.format(dimension), end='   ')

            key = cache_key.format(method, size, dimension, n_repeats)
            scores = get_storage_data(key, None)

            if scores is None:
                # Make sure results are reproducible
                rng = np.random.RandomState(seed=seed)
                scores = []

                for _ in range(n_repeats):
                    if dimension < 1:
                        # Add predictions on special first dimensional case
                        z = [rng.random_sample(size)]

                        noise = 2 * rng.random_sample(size) - 1
                        z.append(z[0] + 0.1 * noise)
                    else:
                        z = [rng.random_sample(size) for _ in range(dimension)]

                    estimator = DependenceEstimator(method=method, n_jobs=-1)
                    score = estimator.score(np.column_stack(z), output)
                    scores.append(score)

                    # Show update
                    print('.', end='')

                # Cache results
                scores = np.array(scores)

            # Collect statistics
            if method == "tcmi":
                scores_mean[i] = tuple(np.mean(score) for score in zip(*scores))
                scores_std[i] = tuple(np.std(score) for score in zip(*scores))
            else:
                scores_mean[i] = np.mean(scores)
                scores_std[i] = np.std(scores)
            print()

        bucket[size] = (scores_dimensions, scores_mean, scores_std)
        print()
```

```python
##
# Plot data
##

num_rows = len(results[next(iter(results.keys()), None)])
ratio = (len(methods) + 1) / (num_rows * (2 / 1.5) + 0.1)

width_ratios = np.ones(num_rows, dtype=np.int_)
height_ratios = np.full(len(methods) + 1, 1.5, dtype=np.float_)
height_ratios[0] = 2

##
# Figure setup
##
fig = plt.figure(figsize=(12, 12 / ratio))
gs = fig.add_gridspec(len(methods) + 1, num_rows, width_ratios=width_ratios,
                      height_ratios=height_ratios,
                      wspace=.1, hspace=.3, left=0.1, right=0.98, top=0.98, bottom=0.08)

for i, method in enumerate(["tcmi"] + methods):
    axs = [fig.add_subplot(gs[i, j]) for j in range(num_rows)]

    scores = results[method]
    last = method == methods[-1]

    x = np.arange(1, dimensions + 1)
    y = np.linspace(-1, 1, num=5)
    margin = 0.5

    for i, (ax, key) in enumerate(zip(axs, sorted(scores))):
        scores_dimension, scores_mean, scores_std = scores[key]

        if method == "tcmi":
            # Add up all corrections to the score
            scores_zero = (scores_mean['total_score'][0], scores_std['total_score'][0])

            # Compute mean score from repeated runs
            scores_mean = [(adjusted_score, score, np.sum(score0))
                           for adjusted_score, score, *score0 in scores_mean[1:]]
            adjusted_score, score, score0 = [np.array(values) for values in zip(*scores_mean)]

            # Compute standard deviation of score from repeated runs
            scores_std = [(adjusted_score, score, np.sum(score0))
                          for adjusted_score, score, *score0 in scores_std[1:]]
            adjusted_score_std, score_std, score0_std = [np.array(values) for values in zip(*scores_std)]

            # Compute special point (shuffled version)
            special_points = [(1, adjusted_score[1])]

            corrected_score = adjusted_score.copy()
            corrected_score[0] = scores_zero[0]
            constant_line = corrected_score
        else:
            scores_zero = (scores_mean[0], scores_std[0])
            scores_mean, scores_std = scores_mean[1:], scores_std[1:]

            special_points = [(1, scores_zero[0])]
            constant_line = scores_mean

        # Show reference lines
        for value in np.linspace(-1, 1, num=9):
            ax.axhline(value, color=neutral_color3, linewidth=1, linestyle=':', zorder=-1)
        ax.axhline(0, color=neutral_color2, linewidth=1)

        # Plot special point
        xx, yy = tuple(zip(*special_points))
        ax.errorbar(xx, yy, fmt='x', color=cmap2(0.1), markersize=15, capsize=5, linewidth=5)

        # Show trend line
        constant = np.ones_like(constant_line)
        model = np.column_stack((constant, constant))

        coeff = np.linalg.lstsq(model, constant_line, rcond=-1)[0]
        ax.axhline(np.sum(coeff), color=cmap2(0.1), linewidth=2, linestyle=':',
                   label='Average')

        # Plot score contributions
        if method == "tcmi":
            ax.bar(x, score, color=cmap1(0.9), width=0.6, alpha=0.9,
                   label="Unadjusted score")
            errors = ax.errorbar(x, score, yerr=score_std, color=neutral_color2, clip_on=False,
                                 fmt='D', markersize=6, capsize=3, linewidth=1, linestyle=None)

            ax.bar(x, -score0, bottom=0, color=cmap1(0.1), width=0.6, alpha=0.9,
                   label="Baseline correction")
            ax.errorbar(x, -score0, yerr=score0_std, color=neutral_color2, clip_on=False,
                        fmt='D', markersize=6, capsize=3, linewidth=1, linestyle=None)

            ax.errorbar(x, adjusted_score, yerr=adjusted_score_std, color=cmap2(0.9),
                        fmt='o', label="Dependence strength", #  r'$\langle \hat{\mathcal{D}}_{TCMI}^*(Y; X) \rangle$',
                        markersize=8, capsize=5, linewidth=2)
        else:
            ax.bar(x, scores_mean, color=cmap1(0.9), width=0.6, clip_on=False,
                   label=r'$\hat{{\mathcal{{D}}}}_{{{:s}}}(Y; X)$'.format(method), alpha=0.9)
            errors = ax.errorbar(x, scores_mean, yerr=scores_std, color=cmap2(0.9), clip_on=False,
                                 fmt='o', label=r'$\hat{{\mathcal{{D}}}}_{{{:s}}}(Y; X)$'.format(method),
                                 markersize=8, capsize=5, linewidth=2)
        for b in errors[1]:
            b.set_clip_on(False)

        if method == "tcmi":
            ax.text(x[0] - margin, y[0], 'Sample size: {:d}'.format(key), ha='left',
                    va='bottom', fontweight='bold', color=neutral_color1, fontsize='small')

        # Plot styles
        ax.spines['top'].set_visible(False)
        ax.spines['right'].set_visible(False)

        ax.spines['left'].set_position(('outward', 10))
        ax.spines['bottom'].set_position(('outward', 10))

        ax.set_xticks(x)
        if last:
            ax.set_xticklabels(x)
        else:
            ax.set_xticklabels([])
        ax.set_xlim(x[0] - margin, x[-1] + margin)

        # Show axis only for first subfigure
        if i > 0:
            ax.set_yticklabels([])
            ax.spines['left'].set_visible(False)
            ax.set_yticks([])
        else:
            ay = y if method == "tcmi" else y[y.size // 2:]
            ax.set_yticks(ay)
            ax.set_yticklabels(['{:.1f}'.format(v) for v in np.abs(ay)])

            if method == "tcmi":
                label = ''.join([
                    'TCMI\n',
                    r'$\leftarrow \langle \hat{\mathcal{D}}_0(Y; X) \rangle$ | '
                    r'$\langle \hat{\mathcal{D}}(Y; X) \rangle \rightarrow$'
                ])
            else:
                label = "{name:s}\n$\hat{{D}}_{{{name:s}}}(Y; X) \\rightarrow$"
                label = label.format(name=method.upper())
            ax.set_ylabel(label)
        ax.set_ylim(ay[0], ay[-1])

        #aspect = 1.5 if method == "tcmi" else 2
        #ax.set_aspect(aspect, adjustable="datalim", anchor="S")

        # Show legend
        if method == "tcmi" and ax is axs[-1]:
            handles, labels = ax.get_legend_handles_labels()
            handles.append(handles.pop(0))
            labels.append(labels.pop(0))
            cax = fig.add_axes([0.91, 1, .5, 1])

            # HACK: Create legend in different context (0.18, -0.825)
            legend = plt.legend(handles, labels, loc='lower right', facecolor='w',
                                ncol=4, handletextpad=0.4, handlelength=1.5,
                                columnspacing=1, bbox_to_anchor=(0.18, -1.006), frameon=False)
            cax.remove()
            fig.add_artist(legend)

            plt.text(0.2, 0.01, 'Legend:', transform=fig.transFigure)
            plt.text(0.08, 0.035, r'Subset dimension $|X| \longrightarrow$',
                     transform=fig.transFigure)

plt.show()
```

### 1.4. Invariance against scaling

**Test**: Invariance of TCMI against invertible transformations

**Expected**: Same score (here: showcases some very simple examples)

```python
def get_groups(data):
    # Compare distributions by using cumulative probability
    fingerprints = {}
    for key, value in data.items():
        vs = np.sort(value)
        unique_vector = np.searchsorted(vs, value, side='right')
        fingerprints[key] = unique_vector.tobytes()

    keys = list(data)
    duplicates = set()

    groups = []
    for i, a_key in enumerate(keys, 1):
        a_value = fingerprints[a_key]
        if a_key in duplicates:
            continue

        groups.append([a_key])
        duplicates.add(a_key)

        for b_key in keys[i:]:
            if b_key in duplicates:
                continue

            b_value = fingerprints[b_key]
            if a_value == b_value:
                groups[-1].append(b_key)
                duplicates.add(b_key)
    return groups

for size in sizes:
    print(":: Size {:d}".format(size))

    # Reinitialize random number generator for each size
    rng = np.random.RandomState(seed=seed)

    xx = 2 * rng.random_sample(size) - 1
    yy = np.linspace(0, 1, num=size)

    # Noise term
    sigma = 0.1
    noise = rng.uniform(low=-1, high=1, size=size)
    rr = yy + sigma*noise

    # Generate some data and dependences
    data = {
        'yy': yy,                                         # Target
        'constant': np.ones_like(yy),                     # Constant
        'x1': xx, 'x2': np.exp(xx), 'x3': xx**3 + xx,     # Random variables
        'r1': rr, 'r2': np.exp(rr), 'r3': rr**3 + rr,     # Target plus noise
    }
    rands = [rng.randint(-10, 10, 3) for _ in range(5)]
    data.update({
        "{:d}x**({:d}){:+d}".format(a, k, b): a * rr**k + b
        for a, k, b in rands
    })

    # Groups
    print("\n/**\n"
          " * Groups that must have identical score\n"
          " */\n")

    groups = get_groups(data)
    max_length = max(len(k) for k in data)
    for i, group in enumerate(groups, 1):
        print("  {:d}.  {:s}".format(i, " == ".join(g.ljust(max_length) for g in group)))

    # Scale invariance
    print("\n/**\n"
          " * Check scale invariance\n"
          " */\n")

    for method in methods + ["tcmi"]:
        print("{:>4s}:".format(method), end=" ")
        estimator = DependenceEstimator(method=method, cache=False, n_jobs=-1)
        atol = 1e-2 if method == "mcde" else 5e-3

        # Last round was not computed with TCMI
        if method == "tcmi" and size == sizes[-1]:
            print("n/a")
            continue

        failed = False
        for group in groups:
            results = {}
            for feature in sorted(group):
                key = "scale_invariance_{:d}_{:s}_{:s}".format(size, method, feature)
                actual_score = get_storage_data(key, None)
                if actual_score is None:
                    actual_score = estimator.score(np.column_stack((data[feature], )), yy)

                # All variables in a group need to have the same score
                prev_score = results.setdefault("score", actual_score)
                if not np.isclose(prev_score, actual_score, atol=atol):
                    delta = np.abs(prev_score - actual_score)
                    failed = True

                    print("\u2717 (delta={:.1g})".format(delta))
                    break

            if failed:
                break
        else:
            print("(\u2713)" if method == "mcde" else "\u2713")

    # Permutation invariance
    print("\n/**\n"
          " * Check permutation invariance\n"
          " */\n")

    for method in methods + ["tcmi"]:
        print("{:>4s}:".format(method), end=" ")
        estimator = DependenceEstimator(method=method, cache=False, n_jobs=-1)
        atol = 1e-2 if method == "mcde" else 5e-3

        # Last round was not computed with TCMI
        if method == "tcmi" and size == sizes[-1]:
            print("n/a")
            continue

        failed = False
        for a_keys in groups:
            b_keys = sorted(set(data).difference(a_keys))

            results = {}
            for k1, k2 in itertools.product(sorted(a_keys), b_keys):
                key = "permuation_invariance_{:d}_{:s}_{:s}_{:s}".format(size, method, k1, k2)
                actual_score = get_storage_data(key, None)
                if actual_score is None:
                    actual_score = estimator.score(np.column_stack((data[k1], data[k2])), yy)

                prev_score = results.setdefault(k2, actual_score)
                if not np.isclose(prev_score, actual_score, atol=atol):
                    delta = np.abs(prev_score - actual_score)
                    failed = True

                    print("\u2717 (delta={:.1g})".format(delta))
                    break

            if failed:
                break
        else:
            print("(\u2713)" if method == "mcde" else "\u2713")
    print()
```

## 2. Bivariate Gaussian distribution

In this section, we examine a simple feature selection task with a known distribution and nonlinear dependencies between features and the output variable. Essentially, we consider bivariate Gaussian distributions with different sample sizes, add noisy features, and test dependency estimators to find the optimal subset of features. Since the ground truth is known and the problem is two-dimensional, we expect only two traits to be selected by all dependency estimators.

**Settings:**

```python
methods = ['tcmi', 'cmi', 'mac', 'uds', 'mcde']
sizes = [50, 100, 200, 500]

seed = 2019
num_splits = 10
num_repeats = 500

# Test case
noise_levels = 5
num_samples = 500
confidence = 0.95
```

**The dataset:**

```python
target = 'Gaussian'
data = pd.read_csv('data/2d_gaussian.csv', low_memory=False)
#data = utils.prepare_data(data, target)

# Get data
x, z = data.drop(labels=target, axis=1), data[target]

# Setup noise levels
noises = np.linspace(0, 1, num=noise_levels + 1)

# Generate indices for sub-sampling
rng = np.random.RandomState(seed=seed)
indices = np.arange(data.shape[0])
rng.shuffle(indices)

# Plot empirical mean and covariance
xy = np.vstack((data['x'], data['y']))
pretty_print = lambda x: re.sub('\s+', ' ', repr(x))
print('Empirical mean       = {}\nEmpirical covariance = {}'
      .format(pretty_print(np.mean(xy, axis=-1)), pretty_print(np.cov(xy))))
```

### 2.1. Check score of optimal feature subset

**Test**: List TCMI scores for optimal two-dimensional subset `{x,y}`

**Expected**: TCMI scores of `{x,y}`

```python
estimator = DependenceEstimator(method='tcmi', n_jobs=-1)
key = 'gaussian_optimal_subspace'

gaussian = data.iloc[indices[:sizes[-1]]]

scores = get_storage_data(key, None)
if scores is None:
    scores = []
    for k1, k2 in itertools.product(['x', '-x', '|x|', '-|x|'], ['y', '-y', '|y|', '-|y|']):
        temp_xy, temp_z = gaussian[[k1, k2]], gaussian[target]
        score = estimator.score(temp_xy, temp_z)
        scores.append((k1, k2, score))

    scores.sort(key=lambda x: -x[-1])

print('\n'.join('{{{:s},{:s}}} = {:.3f}'.format(*s) for s in scores))
```

### 2.2. Compute subset score

**Test**: Compute subset score `{x, y}` -> z with respect to increasing number of data points

**Expected**: Monotonic increase for information-theoretic measures

```python
results = {}
for size in sizes:
    gaussian = np.take(xy, indices[:size], axis=-1)
    output = np.take(z, indices[:size], axis=0)
    gaussian = gaussian.T

    results[size] = {}
    for method in methods:
        estimator = DependenceEstimator(method=method, n_jobs=-1)
        key = 'gaussian_xy_{:s}_{:d}'.format(method, size)

        score = get_storage_data(key, default=None, overwrite=bool(method != 'tcmi'))
        if score is None:
            score = estimator.score(gaussian, output)
            storage.set(key, score)
        results[size].setdefault(method, score)

df = np.array([[results[size][method] for method in methods] for size in sizes])
pd.DataFrame(df, index=sizes, columns=methods)
```

### 2.3. Find optimal subset of features

**Test**: Find optimal subset of features

**Expected**: Optimal subset must be `{x, y}`, prefactors allowed, feature "normal" is similar to "x" or "y"

```python
results = {}
for size in sizes:
    gaussian = data.iloc[indices[:size]]

    print('\n/**'
          '\n * Data points = {:d}'
          '\n */'.format(size))

    results[size] = {}
    for method in methods:
        estimator = DependenceEstimator(method=method, n_jobs=-1)
        key = 'gaussian_subspace_{:s}_{:d}'.format(method, size)

        subsets = get_storage_data(key, default=None, overwrite=bool(method != 'tcmi'))
        if subsets is None:
            subsets = tcmi.get_subspaces(gaussian, target, estimator, cv=None,
                                         depth=2, scoring='mutual_information_score',
                                         fit_params=None, verbose=1, n_jobs=-1)

        subsets = utils.filter_subsets(subsets, remove_duplicates=True)

        output = []
        cursor = 3
        cursor = -1
        threshold = 1

        has_xy = False
        for i, subset in enumerate(subsets):
            subspace = subset['subspace']
            score = subset['stats']['mutual_information_score_mean']
            depth = len(subspace)

            if i == 0 or cursor == -1 or cursor > depth:
                line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                    len(subspace), ','.join(subspace), score)
                output.append(line)

                if i > 0 or cursor == -1:
                    threshold = 0.95 * score
                    cursor = depth

            elif score > threshold:
                line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                    len(subspace), ','.join(subspace), score)
                output.append(line)

        results[size].setdefault(method, subsets)
        print('\nMethod: {:s}\n  {:s}'.format(method, '\n  '.join(output)))
    print('')
```

### 2.4. Statistical power analysis (95% confidence)

To perform statistical power analysis we see, that all of the above dependency measures converge to the optimal feature subsets ${x,y}$ for at least 500 data samples, which will be chosen as the sample size in the following.

**Test**: Statistical power analysis (95% confidence)

**Expected**: High statistical power as well as high contrast between the actual score and independence

```python
def generate_dataset(data, target, noise, n_splits=10, n_repeats=100, seed=None):
    """Generate data set.
    """
    size = data.shape[0]

    # Convert cross-validation into number of data samples
    cv = np.floor(size * (n_splits if n_splits < 1 else (1 - 1 / n_splits))).astype(np.int_)

    # Initialize generator
    rng = np.random.RandomState(seed)
    indices = np.arange(size)

    # Evaluate dataset
    noise /= 2
    for _ in range(n_repeats):
        # Split dataset and make sure to make vectors immutable
        index = indices[:cv].copy()
        dataset = data.iloc[index].copy()

        # Add (centered) noise
        for key in dataset:
            if key == target:
                continue

            # Center noise around actual value
            dataset[key] += rng.uniform(-noise, noise, size=cv)

        # Return generated dataset
        yield dataset, index

        # Prepare for next iteration
        rng.shuffle(indices)

def compute_score(method, noise, dataset, indices):
    """Compute score for dataset.
    """
    # Setup
    test_xy, test_z = dataset.drop(labels=target, axis=1), dataset[target]
    estimator = DependenceEstimator(method=method, n_jobs=1, cache=None)
    h = cache.compute_hash((dataset, indices))

    # Compute measure with respect to output
    key = 'score_{:s}_{:.2f}_{:s}'.format(method, noise, h)
    score = storage.get(key, None)
    if score is None:
        score = estimator.score(test_xy, test_z)
        storage.set(key, score, retry=True)

    # Compute measure with respect to independent variables
    key = 'score0_{:s}_{:.2f}_{:s}'.format(method, noise, h)
    score0 = storage.get(key, None)
    if score0 is None:
        score0 = estimator.score(test_independent[indices], test_z)
        storage.set(key, score0, retry=True)

    # Return scores
    return score, score0

datasets = [
    ('Bivariate normal distribution: $\{x,y\}$', 'gaussian', 'data/2d_gaussian.csv',
     'Gaussian', ('-|x|', '-|y|'))
]

collections = []
for label, name, file, target, subset in datasets:
    data = pd.read_csv(file, low_memory=False)
    if target == 'Delta E':
        del data['Combination']

    print('\n/**'
          '\n * Data set: {:s}'
          '\n */\n'.format(name))

    data = utils.prepare_data(data, target)
    size = min(len(data), num_samples)
    subset += (target, )

    # Setup noise levels
    noises = np.linspace(0, 1, num=noise_levels + 1)

    # Generate indices for test set
    rng = np.random.RandomState(seed=seed)
    indices = np.arange(len(data))
    rng.shuffle(indices)

    # Consider subset of data samples
    data = data[list(subset)].iloc[indices[:size]]
    test_independent = rng.uniform(0, 1, (size, 2))

    dtypes = [('noise', np.float_), ('power', np.float_),
              ('score', np.float_), ('score0', np.float_)]
    print('Data points = {:d}\nNoise levels = {:d}'
          .format(size, len(noises)))

    # Initialize joblib
    processor = joblib.Parallel(n_jobs=-1)
    callback = joblib.delayed(compute_score)

    # Compute statistical power
    collection = {}
    for method in methods:
        print('\nMethod: {:s}\n  '.format(method), end='')

        statistics = np.zeros(noises.size, dtype=dtypes)
        for i, noise in enumerate(noises):
            print('{:.2f}'.format(noise), end=', ')

            key = '{:s}_statistical_power_{:s}_{:d}_noise_{:.2f}'
            key = key.format(name, method, size, noise)
            values = get_storage_data(key, default=None, overwrite=bool(method != 'tcmi'))

            if values is None:
                # Perform calculation in parallel
                iterator = generate_dataset(data, target, noise, n_splits=num_splits,
                                            n_repeats=num_repeats, seed=seed)
                values = processor(callback(name, method, noise, dataset, indices)
                                   for dataset, indices in iterator)
                scores, scores0 = tuple(zip(*values))

                values = tuple(np.array(value) for value in zip(*values))

            # Compute statistical power
            scores, scores0 = values
            cutoff = np.quantile(scores0, confidence)

            power = np.mean(scores > cutoff)
            score_avg = scores.mean()
            score0_avg = scores0.mean()

            # Save statistical power statistics
            statistics[i] = (noise, power, score_avg, score0_avg)

        print('\n')
        for values in statistics:
            print('Noise = {:.2f}, Power = {:.2f}, Score = {:.2f}, Score0 = {:.2f}'.format(*values))
        collection[method] = statistics

    collections.append((label, collection))
```

```python
##
# Plot
##

from matplotlib.legend_handler import HandlerTuple
import matplotlib.patheffects as path_effects

# Plot
margin = 0.1
ncols = len(methods)
nrows = len(collections)
idx = np.where(np.arange(noise_levels + 1) % 2 == 0)[0]

fig, axs = plt.subplots(figsize=(16, 3 * nrows), ncols=ncols, nrows=nrows)
if len(collections) == 1:
    axs = [axs]

handlers = []
for i, (caxs, item) in enumerate(zip(axs, collections)):
    name, collection = item

    for j, (ax, method) in enumerate(zip(caxs, methods)):
        # Show reference lines
        for y in np.linspace(0, 1, num=5):
            ax.axhline(y, color=neutral_color3, linewidth=1, linestyle=':', zorder=-1)

        # Get data
        data = collection[method]

        # Initialize coordinates
        x = data['noise']
        y = np.linspace(0, 1, num=5)
        color = 0.8 * (j / len(methods)) + 0.1

        # Show contrast of score
        score = np.maximum(data['score'] - data['score0'], 1e-2)
        bars = ax.bar(x, data['score'], data['score0'])
        handler = ax.fill_between(x, data['score'], data['score0'], color=cmap1(color),
                                  label='Dependence scores')
        handlers.append(handler)

        if j == 0:
            ax.text(x.min() - 0.1, y.max() + 0.17, name, va='bottom', ha='left')

        ax.plot(x, data['power'], color=cmap2(0.9), marker='s', clip_on=False,
                label='Statistical power ($\gamma = 0.95$)')

        ax.text(x[0] - margin, y[-1] + 0.05, method.upper(), ha='left',
                va='bottom', fontweight='bold', color=neutral_color1, fontsize='small')

        # Show statistical power
        for bar, value, value0, score0 in zip(bars, data['power'], score, data['score0']):
            bar_x = bar.get_x() + bar.get_width() / 2 # - 0.01 * bar.get_width()
            bar_y = bar.get_y() + bar.get_height() + 0.05
            text = '{:.2f}'.format(value)

            va = 'bottom'
            if value > 0.5:
                value -= 0.04
                va = 'top'

            if value < 0.5: # or value > 0.9:
                value += 0.08
                va = 'bottom'

            text = ax.text(bar_x, value - 0.02, text, ha='center', va=va,
                           fontsize='small', color=cmap2(0.9), rotation=90)
            text.set_path_effects(
                [path_effects.withStroke(linewidth=5, foreground='w')])

        bars.remove()

        # Plot styles
        ax.spines['top'].set_visible(False)
        ax.spines['right'].set_visible(False)

        ax.spines['left'].set_position(('outward', 10))
        ax.spines['bottom'].set_position(('outward', 10))

        ax.set_xticks(x)
        ax.set_xlim(x[0] - margin, x[-1] + margin)
        #ax.set_xlabel('Noise levels')

        # Show axis only for first subfigure
        if j > 0:
            ax.set_yticklabels([])
            ax.spines['left'].set_visible(False)
            ax.set_yticks([])
        else:
            ax.set_yticks(y)
            ax.set_yticklabels(['{:.2f}'.format(value) for value in y])
            ax.set_ylabel('Score/Power')
        ax.set_ylim(y[0], y[-1])
        ax.set_xlim(x[0], x[-1])

    # Show legend
    if caxs is axs[-1]:
        handles, labels = ax.get_legend_handles_labels()
        handles[-1] = tuple(handlers)

        cax = fig.add_axes([1, 1, 0, 1])

        # HACK: Create legend in different context
        # https://matplotlib.org/3.1.1/gallery/text_labels_and_annotations/legend_demo.html
        legend = plt.legend(handles[::-1], labels[::-1], loc='upper right', facecolor='w',
                            ncol=3, handletextpad=0.3, handlelength=2, numpoints=1,
                            columnspacing=0.75, bbox_to_anchor=(0, -1), frameon=False,
                            handler_map={tuple: HandlerTuple(ndivide=None,pad=0)})

        # Get the bounding box of the original legend
        bb = legend.get_bbox_to_anchor().inverse_transformed(ax.transAxes)

        # Change to location of the legend.
        xOffset = -0.6
        bb.x0 += xOffset
        bb.x1 += xOffset
        legend.set_bbox_to_anchor(bb, transform = ax.transAxes)

        cax.remove()
        fig.add_artist(legend)

        ax.text(0.48, -0.12, 'Legend:', transform=fig.transFigure)
        ax.text(0.07, -0.12, r'Noise levels $\longrightarrow$', transform=fig.transFigure)

plt.show()
```

## 3. UCI Regression data sets

1. Friedman - https://sci2s.ugr.es/keel/dataset.php?cod=81 . It has been obtained from the LIACC repository. The original page where the data set can be found is: http://www.liaad.up.pt/~ltorgo/Regression/DataSets.html.
2. Concrete Compressive Strength - https://archive.ics.uci.edu/ml/datasets/Concrete+Compressive+Strength
3. Forest Fires Data Set - https://archive.ics.uci.edu/ml/datasets/Forest+Fires

```python
methods = ['tcmi', 'cmi', 'mac', 'uds', 'mcde']
```

### 3.1. Generate Friedman data set

```python
# https://blog.datadive.net/selecting-good-features-part-iv-stability-selection-rfe-and-everything-side-by-side/
# Friedman #1 regression dataset

np.random.seed(0)

size = 500
target = 'y'

# "Friedamn #1” regression problem
X = np.random.uniform(0, 1, (size, 14))
Y = (10 * np.sin(np.pi*X[:,0]*X[:,1]) + 20*(X[:,2] - .5)**2 +
     10*X[:,3] + 5*X[:,4] + np.random.normal(0,1))
# Add 3 additional correlated variables (correlated with X1-X3)
X[:,10:] = X[:,:4] + np.random.normal(0, .025, (size,4))

names = ["x%s" % (i + 1) for i in range(X.shape[-1])]
data = pd.DataFrame(data=X, columns=names)
data[target] = Y

# data.to_csv('data/friedman.csv', index=False)

from scipy import stats

# Show which features have strong correlations
for key in data:
    if key == target:
        continue

    r2 = np.corrcoef(data[key], data[target])[0, 1]**2
    rho, pval = stats.spearmanr(data[key], data[target])
    print('{:<3s}: r2={:<4.2f}  rho={:>5.2f}  pval={:<.2g}'
          .format(key, r2, rho, pval))

x, z = data.drop(labels=target, axis=1), data[target]
```

### 3.2. Optimal subspace search

**Test**: Find optimal subsets for several available datasets

**Expected**: Convergence and stability of all dependence measures to the same subset

```python
# Load datasets
datasets = [
    ('friedman', 'data/friedman.csv', 'y'),
    ('concrete', 'data/concrete.csv', 'compressive_strength'),
    ('forest_fires', 'data/forestfires.csv', 'area')
]

for name, filename, target in datasets:
    data = pd.read_csv(filename, low_memory=False)
    data = utils.prepare_data(data, target)

    print('\n/**'
          '\n * Model: {:s}'
          '\n */\n'
          '\nKeys: {{{:s}}}'
          '\nSize: {:d}'
          .format(name, ','.join(data), data.shape[-1] - 1))

    for method in methods:
        estimator = DependenceEstimator(method=method, n_jobs=1)
        key = 'subspace_search_{:s}_{:s}'.format(target, method)
        print('\nMethod: {:s}'.format(method))

        subsets = get_storage_data(key, default=None, overwrite=bool(method != 'tcmi'))
        if subsets is None:
            subsets = tcmi.get_subspaces(data, target, estimator, cv=None,
                                         depth=-1, scoring='mutual_information_score',
                                         fit_params=None, verbose=1, n_jobs=-1)

        output = []
        if len(subsets) > 1000:
            output.append('More than {:d} subsets.'.format(len(subsets)))
        else:
            cursor = -1
            threshold = 1
            for i, subset in enumerate(subsets):
                subspace = subset['subspace']
                score = subset['stats']['mutual_information_score_mean']
                depth = len(subspace)

                if i == 0 or cursor == -1 or cursor > depth:
                    line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                        len(subspace), ','.join(subspace), score)
                    output.append(line)

                    if i > 0 or cursor == -1:
                        threshold = 0.95 * score
                        cursor = depth

                elif score > threshold:
                    line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                        len(subspace), ','.join(subspace), score)
                    output.append(line)
        print('  {:s}'.format('\n  '.join(output)))
```

## 4. Octet-binary compound semiconductors data set

Octet-binary compound semiconductors are materials consisting of two elements formed by groups of I/VII, II/VI, III/V, or IV/IV elements leading to a full valence shell. They crystallize in rock salt (RS) or zinc blende (ZB) structures.

The data set is composed of 82 materials with two atomic species in the unit cell. The objective is to accurately predict the energy difference $\Delta E$ between RS and ZB structures based on 8 electro-chemical atomic properties for each atomic species $A/B$ (in total 16) such as atomic ionization potential $\text{IP}$, electron affinity $\text{EA}$, the energies of the highest-occupied and lowest-unoccupied Kohn-Sham levels, $\text{H}$ and $\text{L}$, and the expectation value of the radial probability densities of the valence $s$-, $p$-, and $d$-orbitals, $r_s$, $r_p$, and $r_d$, respectively.

    L. M. Ghiringhelli, J. Vybiral, S. V. Levchenko, C. Draxl, C. & M. Scheffler: Big Data of Materials Science: Critical Role of the Descriptor. Physical Review Letters 114, 105503 (2015). DOI: 10.1103/PhysRevLett.114.105503

The additional features from the reference are:

$$
\begin{equation}
    D1 = \frac{\text{IP}(B) - \text{EA}(B)}{r_p(A)^2}\ ,\quad \ D2 = \frac{|r_s(A) - r_p(B)|}{\exp[r_s(A)]} \ .
\end{equation}
$$

### 4.1 Subspace search

**Test**: Optimal feature subset search

**Expected**: Find features to best predict $\Delta E$. Potential candidates are listed in the reference above

```python
include_extra_features = False
```

```python
methods = ['tcmi', 'cmi', 'mac', 'uds', 'mcde']
data = pd.read_csv('data/octet-binary-compound-semiconductors.csv', low_memory=False)
materials = data.drop(columns='Combination', inplace=True)
target = 'Delta E'

# Add extra features from PRL?
if include_extra_features:
    extra = {
        'D1': (data['EA(B)'] - data['IP(B)']) / data['rp(A)']**2,
        'D2': np.abs(data['rs(A)'] - data['rp(B)']) / np.exp(data['rs(A)'])
    }

    for k, v in extra.items():
        data[k] = v

data = utils.prepare_data(data, target, copy=True)

print('\n/**'
      '\n * Model: Octet-binary compound semiconductors data set'
      '\n */\n'
      '\nKeys: {{{:s}}}'
      '\nSize: {:d}'
      .format(','.join(data), data.shape[-1] - 1))

results = {}
for method in ["mcde", "uds"] + methods:
    print('\nMethod: {:s}'.format(method))
    estimator = DependenceEstimator(method=method, cache=True)

    key = 'binary_octets_{:s}'.format(method)
    subsets = get_storage_data(key, None, force=include_extra_features)
    if subsets is None:
        subsets = get_subspaces(data, target, estimator, cv=None, depth=-1,
                                scoring='mutual_information_score',
                                fit_params=None, verbose=1, n_jobs=-1)
        storage.set(key, subsets)

    threshold = 1
    output = []
    cursor = -1
    subsets = utils.filter_subsets(subsets)

    candidates = []
    for i, subset in enumerate(subsets):
        subspace = subset['subspace']
        score = subset['stats']['mutual_information_score_mean']
        depth = len(subspace)

        if i == 0 or cursor == -1 or cursor > depth:
            line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                len(subspace), ','.join(subspace), score)
            candidates.append((subspace, score))
            output.append(line)

            if i > 0 or cursor == -1:
                threshold = 0.95 * score
                cursor = depth

        elif score > threshold:
            line = '[{:>3d}] {{{:s}}} = {:.2f}'.format(
                len(subspace), ','.join(subspace), score)
            candidates.append((subspace, score))
            output.append(line)

    print('  {:s}'.format('\n  '.join(output)))
    keys = sorted([k for k in data.columns if k != target and '|' not in k and '-' not in k],
                  key=lambda x: (x[-2], x))
    candidates.insert(0, (keys, 1))
    results[method] = candidates
```

### 4.2 Model construction

```python
# Helper function to get optimal number of estimators
def get_estimators(**params):
    manager = multiprocessing.Manager()
    iterations = manager.list()
    lock = manager.Lock()
    model = [None, 0]

    def stop_model(env):
        # Make sure to ignore this model
        scores = []
        for evaluation in env.evaluation_result_list:
            evaluation = list(evaluation)
            evaluation[2] = (float('-inf') if evaluation[3] else float('inf'))
            scores.append(tuple(evaluation))

        # Stop early
        raise lgbm.callback.EarlyStopException(env.iteration, scores)

    def _start_callback(env):
        if model[0] is not env.model:
            model[0] = env.model

            # Process model
            with lock:
                model[1] = len(iterations)
                iterations.append(0)

    def _end_callback(env=None, **kwargs):
        result = None
        if env:
            index = model[1]
            iterations[index] = env.iteration

            # Check for valid parameters
            settings = env.model.params
            if settings['max_depth'] > -1 and settings['num_leaves'] > 2**settings['max_depth']:
                stop_model(env)

            for _, name, score, _ in env.evaluation_result_list:
                if name in ['l1', 'l2'] and score < params.get('threshold', 0):
                    iterations[index] =np.nan
                    stop_model(env)
        else:
            # Adjust number of estimators
            result = np.nanmax(iterations)
            divisor = kwargs.get('divisor', 0)
            if divisor:
                result = (result // divisor + 1) * divisor

        return result

    # Internal attributes used inside LightGBM
    _start_callback.before_iteration = True
    _start_callback.order = 0
    _end_callback.order = 1000
    return _start_callback, _end_callback

num_splits = 10
num_repeats = 5

# Setup estimator
params = {}
params['n_estimators'] = 2000
params['min_child_samples'] = int(0.01 * data.shape[0])

estimator = lgbm.LGBMRegressor(n_jobs=-1, random_state=seed, **params)
cv = RepeatedSortedStratifiedKFold(n_splits=num_splits, n_repeats=num_repeats,
                                   random_state=seed, side='uniform', test_size=0.1)

# Fit params
fit_params = dict(eval_metric=['l1', 'l2_root'], early_stopping_rounds=50, verbose=None)
x, y = data.drop(labels=target, axis=1), data[target]

# Prepare dataset
if cv:
    preprocess, postprocess = get_estimators(threshold=1e-3)
    params = fit_params.copy()
    params.update({
        'eval_set': [(x.iloc[index], y .iloc[index]) for _, index in cv.split(x, y)],
        'callbacks': [preprocess, postprocess]
    })
else:
    params = {}

scores_mapping = {
    'root_mean_squared_error_mean': 'rmse',
    'mean_absolute_error_mean': 'mae',
    'max_absolute_error_mean': 'maxae',
    'r2_mean': 'r2',
    'root_mean_squared_error_std': 'rmse_std',
    'mean_absolute_error_std': 'mae_std',
    'max_absolute_error_std': 'maxae_std',
    'r2_std': 'r2_std'
}

scorings = ['rmse', 'mae', 'maxae', 'r2']
line = '[{size:>3d}] {{{subspace:s}}} = {score:.2f}\n      '
line += '  '.join(scoring + '={' + scoring + ':.3f} +- {' + scoring + '_std:.3f}' for scoring in scorings)
```

```python
for method in sorted(results):
    print("/**\n"
          " * Method: {:s}\n"
          " */\n".format(method))

    print(":: Found {:d} subsets\n".format(len(results[method]) - 1))

    candidates = results[method]
    for index, (candidate, score) in enumerate(candidates):
        features = list(candidate)
        key = "subset_{:s}".format('_'.join(features))

        stats = get_storage_data(key, default=None)
        if stats is None:
            # Prepare dataset
            preprocess, postprocess = get_estimators(threshold=1e-3)
            params = fit_params.copy()
            xx = x[features]

            params.update({
                'eval_set': [(xx.iloc[index], y.iloc[index]) for _, index in cv.split(x, y)],
                'callbacks': [preprocess, postprocess]
            })

            stats = get_statistics(estimator, xx, y, cv=cv, fit_params=params,
                                   verbose=0, n_jobs=-1)
            for old, new in scores_mapping.items():
                stats[new] = stats.pop(old)
            storage.set(key, stats)

        features = sorted(features, key=lambda x: (x[-2], x))
        print(line.format(size=len(features), score=score, subspace=','.join(features), **stats), flush=True)
    print()
```

```python

```


# Fixed Coefficients Random Utility (Demand) Estimation

This notebook reviews the estimation and inference of a **linear** random utility model when the agent is facing a finite number of alternatives.

## Introduction
Consider a set of $J+1$ alternatives $\{0,1,2,...,J\}$. The utility that decision maker (DM) $i$ receives from buying projusct $j$ is
$$ u_{ij} = x_{ij}' \beta -\alpha p_j + \xi_j+ \epsilon_{ij}.$$
The DM maximizes her utility
$$y_i =\arg \min_{j} u_{ij}.$$

We now assume that $\epsilon_{ij}$ are $i.i.d.$ across DMs and across alternatives. In addition, we assume that $\epsilon_{ij}$ are distributed (standard) T1EV. We can write the following Conditional Choice Probabilities (CCP):
$$ Pr(y_i = j) = \frac{e^{x_{ij}'\beta}}{\sum_{k=0}^{J}e^{x_{ik}'\beta}}.$$

**Aggregate, market-level data** In Berry, Levinson, and Pakes (1995) and in many other empirical work following BLP the researcher observes only market-level data. That means that the characteristics vector of the alternatives is not indexed by $i$. The variation in product characteristics are unobserved and get absorbed by the error term $\epsilon$. The choice probabilities become
$$ Pr(y = j|x_j, \xi_j; \beta) \ \text{for} \ j=0,,,J = \frac{e^{x_{j}'\beta}}{\sum_{k=0}^{J}e^{x_{k}'\beta}}.$$
The left-hand side is simply the market share of product/alternative $j$. We will denote these market shares as $s_0,s_1,...,s_{J-1}$. The CCP above all have the same denominator. Moreover, for identification reasons, we normalize $x_0 = 0$. Therefore,
$$ \frac{s_j}{s_0} = e^{x_j'\beta}.$$

Using Berry's Inversion (1994), and take log for both sides gives us:

$$ \text{ln}(s_j) - \text{ln} (s_0) \ = \delta_j \equiv x_j' \beta - \alpha p_j + \xi_j \ \ \ \ \ \text{(eq} A) $$

where
- Mean utility level $\delta_j$ contains the product characteristic $x_j$, $price_j$ and the aggregate error $\xi_j$.
- Econometricians observe aggregate market share for good $j$, outside good, product characteristics $x_j$, price $p_j$.
- $\delta_j$ is uniquely identified directly from a simple algebraic calculation involving market share.
- This is a OLS. (We need instruments for the price!)

# For the rest of this notebook, we will introduce two empirical examples:

## A. Estimate logit-demand using, BLP(1995)'s aggregate market level data.

- A1. Introduction of car data from BLP(1995)
- A2. Data cleaning
- A3. Run linear regression using eq(A)
- A4. Run 2sls using instruments

## B. Monte Carlo Example: estimate logit-demand after solving Nash-Bertrand game

- B1. Data Generating Process
- B2. Obtain (numerically) equilibrium price and market shares
- B3. Regress using OLS / IV (cost shifters, and competitors' product charactersitics as instruments for price. (Table 1, Berry (1994)

### A1.Introduction of car data from BLP(1995)

#### As an empirical study, we will replicate Table 3, as in BLP (1995).

- We obtain product characteristics from the annual issues of the  Automotive News Market Data Book and you can find BLP.csv.

- Data includes the number of cylinders, number of doors, weight, engine displacement, horsepower, length, width, wheelbase, EPA miles per gallon rating  (MPG), and dummy variables for whether the car has front-wheel drive,  automatic transmission, power steering, and air conditioning.
- The data set includes this information on observed products from 1971-1990.
- The price variable is list retail price (in \$1000's). Prices are in 1983 dollars. (We used the Consumer Price Index to  deflate.)
- The sales variable corresponds to U.S. sales (in 1000's) by nameplate.
- The product characteristics correspond to the characteristics of the base model for the given nameplate.
- To capture the cost of driving, we include milers per dollar (MP\$), calculated as MPG divided by price per gallon. (Notice that MPG and pricer per gallon is provided.)

- In terms of potential market size, there is no formal definition. we used the yearly number of households in the U.S. from Statistical Abstract of the U.S.
- We assume that each model comprises a single firm to avoid a multi-product pricing problem.

```julia
# Query / DataFramesMeta is used for cleaning dataset
# FixedEffectModels is used for running regression
# Distributions, Random, NLsolve are used for Monte Carlo study
using CSV, DataFrames, Query, DataFramesMeta, FixedEffectModels, Distributions, Random, NLsolve
```

```julia
ENV["COLUMNS"],ENV["LINES"] = 350,50  #This is not specific to Julia, it's a Jupyter notebook environment variable

#dataset = CSV.read("c:\\data\\BLP.csv");  # <---- change this
dataset = CSV.read("/Users/jinkim/Dropbox/2020 Summer/dropbox_RA_work/Berry/BLP.csv")
first(dataset,10)
```

    ┌ Warning: `CSV.read(input; kw...)` is deprecated in favor of `using DataFrames; CSV.read(input, DataFrame; kw...)
    │   caller = read(::String) at CSV.jl:40
    └ @ CSV /Users/jinkim/.julia/packages/CSV/MKemC/src/CSV.jl:40

	name	id	year	cy	dr	at	ps	air	drv	p	wt	dom	disp	hp	lng	wdt	wb	mpg	q	firmids	euro	reli	dfi	hp2wt	size	japan	cpi	gasprice	nb_hh	cat	door2	door3	door4	door5	sampleweight	mpgd	dpm	model

	String	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Float64	Int64	Int64	Float64	Int64	Float64	Float64	Float64	Float64	Float64	Int64	Int64	Int64	Int64	Float64	Float64	Int64	Float64	Float64	Int64	String	Int64	Int64	Int64	Int64	Int64	Float64	Float64	String

10 rows × 38 columns

1	ACINTE	3735	1986	4	3	0	1	0	1	8.48358	2249	0	97.0	113	168.5	65.6	96.5	27.0	27.807	3	0	5	0	0.502446	1.10536	1	109.6	0.826794	88458	compact	0	1	0	0	27807	32.6562	0.030622	ACINTE1986

2	ACINTE	4030	1987	4	3	0	1	0	1	8.6787	2326	0	97.0	113	168.5	65.6	96.5	26.0	54.757	3	0	5	0	0.485813	1.10536	1	113.6	0.818662	89479	compact	0	1	0	0	54757	31.7591	0.031487	ACINTE1987

3	ACINTE	4327	1988	4	5	0	1	0	1	9.55199	2390	0	97.0	118	171.5	65.6	99.2	17.0	57.468	3	0	5	0	0.493724	1.12504	1	118.3	0.785362	91066	compact	0	0	0	1	57468	21.6461	0.0461978	ACINTE1988

4	ACINTE	4421	1989	4	2	0	1	0	1	10.5403	2313	0	97.0	118	168.7	65.6	96.5	26.0	77.4	3	0	5	0	0.51016	1.10667	1	124.0	0.810081	92830	compact	1	0	0	0	77400	32.0956	0.0311569	ACINTE1989

5	ACINTE	5421	1990	4	2	0	1	0	1	9.14308	2549	0	1.8	130	172.9	67.4	100.4	21.0	83.599	3	0	5	0	0.510004	1.16535	1	130.7	0.875542	93347	compact	1	0	0	0	83599	23.9851	0.0416925	ACINTE1990

6	ACLEGE	3736	1986	6	4	0	1	1	1	17.6077	2970	0	152.0	151	189.4	68.3	108.6	20.0	25.062	3	0	5	0	0.508417	1.2936	1	109.6	0.826794	88458	midsize	0	0	1	0	25062	24.1898	0.0413397	ACLEGE1986

7	ACLEGE	4031	1987	6	4	0	1	1	1	17.8327	3078	0	152.0	151	189.4	68.3	108.7	20.0	54.713	3	0	5	0	0.490578	1.2936	1	113.6	0.818662	89479	midsize	0	0	1	0	54713	24.4301	0.0409331	ACLEGE1987

8	ACLEGE	4328	1988	6	4	0	1	1	1	17.7599	3067	0	163.2	161	189.4	68.3	108.7	19.0	70.77	3	0	5	0	0.524943	1.2936	1	118.3	0.785362	91066	midsize	0	0	1	0	70770	24.1927	0.0413348	ACLEGE1988

9	ACLEGE	4422	1989	6	4	0	1	1	1	18.2258	3170	0	163.0	160	190.6	68.9	108.7	19.0	64.6	3	0	5	0	0.504732	1.31323	1	124.0	0.810081	92830	midsize	0	0	1	0	64600	23.4545	0.0426358	ACLEGE1989

10	ACLEGE	5422	1990	6	2	0	1	1	1	18.9441	3139	0	2.7	160	188.0	68.7	106.5	19.0	53.666	3	0	5	0	0.509716	1.29156	1	130.7	0.875542	93347	midsize	1	0	0	0	53666	21.7008	0.0460811	ACLEGE1990

## Variable name/short description
#### For detailed description, please see BLP(1995) section 7.1. (Data section)

| Variable name    | Description                 |
|------------------|-----------------------------|
| name             | Car                         |
| id               | Car ID                      |
| ye               | Year                        |
| cy               | Cylinder                    |
| dr               | Number of Doors             |
| at               | Automatic Transmission      |
| ps               | Power Steering              |
| air              | Air Conditioning            |
| drv              | Front Wheel Drive           |
| p                | Price (in \$ 1000's)        |
| wt               | Weight                      |
| dom              | Domestic                    |
| disp             | Engine Displacement         |
| hp               | Horse Power                 |
| lng              | Length                      |
| wdt              | Width                       |
| wb               | Wheelbase                   |
| mpg              | Miles per Gallon            |
| q                | Quantities                  |
| firmids          | Firm ID                     |
| euro             | Indicator for EURO car      |
| reli             | Rating                      |
| dfi              | Indicator for Digital Fuel Injection       |
| hp2wt            | HP to Weight (ratio)        |
| size             | Length X Width (/1000)      |
| japan            | Japan                       |
| cpi              | CPI                         |
| gasprice         | Gas Price per gallon        |
| nb_hh            | Size of Household (Potential Market Size)          |
| cat              | Size Cateogry               |
| cat              | (Using for nested logit)    |
| door2            | I(door=2)                   |
| door3            | I(door=3)                   |
| door4            | I(door=4)                   |
| door5            | I(door=5)                   |
| sampleweight     | Weights                     |
| mpgd             | Miles per gallon  (imputed from gas prices) |
| dpm              | Dollars per miles (imputed from gas prices) |
| modelid          | Car name                                    |

```julia
#summary statistics
describe(dataset,:all)
```

	variable	mean	std	min	q25	median	q75	max	nunique	nmissing	first	last	eltype

	Symbol	Union…	Union…	Any	Union…	Union…	Union…	Any	Union…	Nothing	Any	Any	DataType

38 rows × 13 columns

1	name			ACINTE				YUYUGO	542		ACINTE	YUYUGO	String

2	id	2560.89	1517.55	129	1309.0	2325.0	3927.0	5592			3735	4506	Int64

3	year	1981.54	5.74082	1971	1977.0	1982.0	1987.0	1990			1986	1989	Int64

4	cy	5.3207	1.55712	0	4.0	4.0	6.0	12			4	4	Int64

5	dr	3.29409	0.965154	2	2.0	4.0	4.0	5			3	2	Int64

6	at	0.326116	0.468896	0	0.0	0.0	1.0	1			0	0	Int64

7	ps	0.533153	0.499012	0	0.0	1.0	1.0	1			1	0	Int64

8	air	0.241768	0.428251	0	0.0	0.0	0.0	1			0	0	Int64

9	drv	0.354984	0.478616	0	0.0	0.0	1.0	1			1	1	Int64

10	p	11.7614	8.64378	3.39327	6.71375	8.72865	13.0741	68.5968			8.48358	3.50726	Float64

11	wt	2930.47	722.366	1445	2375.0	2861.0	3383.0	5362			2249	1832	Int64

12	dom	0.589986	0.491947	0	0.0	1.0	1.0	1			0	0	Int64

13	disp	177.746	102.032	1.0	109.0	151.0	231.0	500.0			97.0	68.0	Float64

14	hp	117.005	46.6881	39	88.0	105.0	140.0	365			113	52	Int64

15	lng	186.705	20.0657	139.0	172.2	185.0	200.0	236.0			168.5	139.0	Float64

16	wdt	69.6557	5.29795	53.0	65.9	69.0	73.0	81.0			65.6	60.7	Float64

17	wb	104.616	9.817	14.3	97.0	103.4	110.8	212.1			96.5	84.6	Float64

18	mpg	20.9964	5.8107	9.13	17.0	20.0	25.0	53.0			27.0	28.0	Float64

19	q	78.804	89.0799	0.049	15.603	47.35	109.002	646.526			27.807	10.5	Float64

20	firmids	13.7438	6.25909	1	8.0	16.0	19.0	26			3	23	Int64

21	euro	0.23816	0.426053	0	0.0	0.0	0.0	1			0	1	Int64

22	reli	3.0433	1.29108	1	2.0	3.0	4.0	5			5	1	Int64

23	dfi	0.0135318	0.115563	0	0.0	0.0	0.0	1			0	0	Int64

24	hp2wt	0.394375	0.0966429	0.170455	0.336585	0.375049	0.427509	0.947581			0.502446	0.283843	Float64

25	size	1.31016	0.237637	0.756	1.13128	1.26983	1.4527	1.888			1.10536	0.84373	Float64

26	japan	0.171854	0.377338	0	0.0	0.0	0.0	1			1	0	Int64

27	cpi	88.3488	28.8772	40.5	60.6	96.5	113.6	130.7			109.6	124.0	Float64

28	gasprice	1.02727	0.206509	0.785362	0.826794	1.01788	1.13523	1.47121			0.826794	0.810081	Float64

29	nb_hh	81539.0	8770.71	64778	74142.0	83527.0	89479.0	93347			88458	92830	Int64

30	cat			compact				midsize	3		compact	compact	String

31	door2	0.341903	0.474454	0	0.0	0.0	1.0	1			0	1	Int64

32	door3	0.0419486	0.200517	0	0.0	0.0	0.0	1			1	0	Int64

33	door4	0.596301	0.490749	0	0.0	1.0	1.0	1			0	0	Int64

34	door5	0.0198466	0.139505	0	0.0	0.0	0.0	1			0	0	Int64

35	sampleweight	78804.0	89079.9	49	15603.0	47350.0	109002.0	646526			27807	10500	Int64

36	mpgd	21.1248	6.94301	8.61178	15.9838	20.5059	24.8007	65.4256			32.6562	34.5645	Float64

37	dpm	0.052389	0.0167817	0.0152845	0.0403215	0.0487664	0.0625634	0.11612			0.030622	0.0289315	Float64

38	model			ACINTE1986				YUYUGO1989	2172		ACINTE1986	YUYUGO1989	String

```julia
### Replicate Table 1: Summary Stats
using Statistics

#Add or substrct column names as needed:
cnames = [:year,:cy,:dr,:at,:ps,:air,:drv,:p,:wt,:dom,:disp,:hp,:lng,:wdt,:wb,:mpg]
aggregate(dataset[!,cnames], [:year], mean)
```

    ┌ Warning: `aggregate(d, cols, f, sort=false, skipmissing=false)` is deprecated. Instead use combine(groupby(d, cols, sort=false, skipmissing=false), names(d, Not(cols)) .=> f)`
    │   caller = top-level scope at In[4]:6
    └ @ Core In[4]:6

	year	cy_mean	dr_mean	at_mean	ps_mean	air_mean	drv_mean	p_mean	wt_mean	dom_mean	disp_mean	hp_mean	lng_mean	wdt_mean	wb_mean	mpg_mean

	Int64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Float64

20 rows × 16 columns

1	1986	4.94615	3.43846	0.330769	0.684615	0.323077	0.576923	11.7826	2733.55	0.569231	160.212	110.192	180.583	67.9892	101.829	23.5546

2	1987	5.0	3.43357	0.321678	0.72028	0.391608	0.615385	13.4363	2785.15	0.538462	163.357	117.028	180.416	67.9965	101.909	22.6503

3	1988	5.09333	3.44	0.4	0.766667	0.446667	0.613333	14.8857	2847.8	0.533333	163.836	125.053	181.756	68.2853	102.443	21.88

4	1989	4.9932	2.9932	0.37415	0.795918	0.503401	0.680272	16.6905	2895.97	0.496599	163.503	133.837	181.348	68.5537	103.037	21.7891

5	1990	5.0687	2.85496	0.40458	0.816794	0.458015	0.717557	14.0384	2923.47	0.51145	2.7084	133.588	182.156	68.655	101.843	21.8168

6	1971	6.04348	3.36957	0.152174	0.163043	0.0	0.0	8.85633	3176.08	0.684783	245.548	171.891	195.446	72.9891	110.787	17.1995

7	1975	5.86022	3.26882	0.311828	0.387097	0.107527	0.0752688	9.64184	3328.6	0.645161	238.5	119.473	195.312	71.7419	108.129	16.2688

8	1976	5.62626	3.28283	0.282828	0.343434	0.10101	0.0909091	9.49007	3159.45	0.616162	218.66	112.889	192.515	70.8889	106.833	18.5556

9	1977	5.61053	3.24211	0.294737	0.378947	0.0842105	0.136842	9.86429	3070.06	0.610526	204.085	109.463	190.747	70.4947	106.038	19.7789

10	1980	5.23301	3.35922	0.252427	0.359223	0.174757	0.252427	10.7269	2776.99	0.601942	179.446	99.699	185.398	69.4854	103.684	21.6699

11	1981	5.16379	3.21552	0.336207	0.491379	0.241379	0.232759	13.0352	2819.37	0.517241	171.325	102.017	186.56	69.3276	103.929	22.5

12	1982	4.98182	3.32727	0.336364	0.490909	0.236364	0.381818	11.5913	2747.25	0.6	158.536	98.3	185.273	68.8636	103.151	23.7727

13	1983	4.88696	3.30435	0.33913	0.530435	0.2	0.434783	11.1408	2736.82	0.617391	156.169	97.8522	184.087	68.7913	102.857	25.1478

14	1984	4.92035	3.35398	0.345133	0.663717	0.283186	0.477876	11.6477	2765.41	0.59292	166.069	107.265	183.822	68.4637	102.735	24.1504

15	1985	4.96324	3.31618	0.345588	0.669118	0.330882	0.522059	12.4764	2759.99	0.566176	163.106	108.081	182.096	68.1625	102.349	22.3868

16	1978	5.57895	3.33684	0.263158	0.347368	0.0947368	0.178947	10.6021	2964.8	0.642105	199.494	107.695	189.347	70.3579	105.337	19.9263

17	1979	5.33333	3.28431	0.235294	0.313725	0.0882353	0.215686	10.4513	2840.49	0.598039	184.635	104.108	186.559	69.5	103.849	20.1078

18	1972	6.20225	3.44944	0.337079	0.348315	0.0449438	0.0	9.04282	3253.78	0.696629	256.947	134.348	196.775	73.3146	111.563	16.3317

19	1973	6.37209	3.44186	0.395349	0.372093	0.0697674	0.0	9.0452	3337.42	0.709302	261.949	131.256	198.384	73.0233	111.215	16.2506

20	1974	6.0	3.30556	0.375	0.375	0.125	0.0	9.25473	3268.29	0.652778	239.179	122.556	196.194	71.9306	108.874	16.3329

### A2. Data cleaning

#### Step 1. Obtain market share for each good $j$: $s_{jt}$ = $\frac{q_{jt}}{nb\_hh_{t}}$

For notation, let denote total market size $nb\_hh_t = M_t$

```julia
dataset = @linq dataset |>
groupby([:year]) |>
transform(Total_Q = sum(:q))

dataset.s_j = dataset.q./dataset.nb_hh;
```

#### Step 2. Obtain market share for outside good 0: $s_{0t}$ = $\frac{ \Big(nb\_hh_t - \sum_{k=1}^J(q_{kt}) \Big)}{nb\_hh_t}$

```julia
dataset.s_0 = (dataset.nb_hh-dataset.Total_Q)./dataset.nb_hh;
```

#### Step 3. Construct dependent variable: $ \text{ln}(s_{jt}) - \text{ln}(s_{0t}) $

```julia
dataset.log_s_j_0 = log.(dataset.s_j) - log.(dataset.s_0);
```

### A3. Run linear regression using eq(A)

#### Step 4. we use hp2wt, air, mpgd, size as product characteristics:
$$ \text{ln}(s_j) - \text{ln} (s_0) \ = \delta_j \equiv x_j' \beta - \alpha p_j + \xi_j $$

```julia
result = reg(dataset, @formula(log_s_j_0 ~ p+hp2wt+air+mpgd+size ),  save = true,);
print(result)
```

                                   Linear Model
    ===========================================================================
    Number of obs:                  2217   Degrees of freedom:                6
    R2:                            0.390   R2 Adjusted:                   0.388
    F Statistic:                  282.47   p-value:                       0.000
    ===========================================================================
                   Estimate  Std.Error   t value Pr(>|t|)  Lower 95%  Upper 95%
    ---------------------------------------------------------------------------
    p            -0.0886398 0.00401348  -22.0855    0.000 -0.0965104 -0.0807692
    hp2wt        -0.0731735   0.276701  -0.26445    0.791  -0.615794   0.469447
    air          -0.0380115  0.0726455 -0.523246    0.601  -0.180472   0.104449
    mpgd          0.0288388 0.00439518   6.56147    0.000  0.0202197  0.0374579
    size            2.40052   0.126801   18.9315    0.000    2.15186    2.64919
    (Intercept)    -10.2035    0.26002  -39.2412    0.000   -10.7134   -9.69356
    ===========================================================================

####  Step 5. Obtain Price elasticities:
Note that own price elasticities $(\eta_j$) is given by:

\begin{align}
\eta_j & = \frac{\partial Pr(j)}{\partial price_j} \underbrace{\frac{price_j}{Pr(j)}}_{\frac{price_j}{s_j \times M}} \\
& \text{Note that} \ \frac{\partial Pr(j)}{\partial price_j} = \frac{\partial s_j}{\partial price_j} \times M  \ \text{where} \ s_j = \frac{e^{\delta_j}}{\sum_k^J e^{\delta_j}}  \\
& \text{Appealing to chain rule}: \frac{\partial s_j}{\partial price_j} \ M = \Bigg[ \alpha \frac{e^{\delta_j}}{\sum_k^J e^{\delta_j}} - \alpha \Big(  \frac{e^{\delta_j}}{\sum_k^J e^{\delta_j}}\Big)^2   \Bigg] = M \alpha [s_j - s_j^2]  = M \alpha s_j[1- s_j]\\
& \text{Rearranging these terms gives us:} \\
& \eta_j = \underbrace{\frac{\partial Pr(j)}{\partial price_j}}_{M \alpha s_j[1- s_j]} \underbrace{\frac{price_j}{Pr(j)}}_{\frac{price_j}{s_j \times M}} = M \alpha s_j[1- s_j] \times \frac{price_j}{s_j} \frac{1}{M} = \underbrace{\alpha \times (1-s_j) \times price_j}_\text{price elasticities for good j} \\
& = \alpha \times (1-s_j) \times price_j
\end{align}

```julia
# Following price elasticities, I can derive price elasticities for each good j, using price coefficients alpha.

price_coef = coef(result)[2];
dataset.e = price_coef * (ones(nrow(dataset))-dataset.s_j) .* dataset.p;

q1 = @from i in dataset begin;
            @where i.e >-1
            @select {elasticity=i.e}
            @collect DataFrame
    end;
nrow(q1)

```

    1502

#### Replication: BLP Table 3, IV Logit Demand Column (Row: No. Inelastic De) in page 873.
#### I derive the number of inelastic car model. My estimates are 1,502. BLP's estimates were 1,494, which is pretty close.

### A4. Run 2sls using instruments

Following BLP, I use the following instruments for price.

#### 1. the sum of size at market $t$. (Note that you need to drop product $j$'s own size.)
#### 2. the sum of size across rival firm products at market $t$.

```julia
# IV 1

dataset = @linq dataset |>
groupby([:year]) |>
transform(Total_size = sum(:size));
dataset.iv_size1 = dataset.Total_size - dataset.size;
```

```julia
# IV 2

dataset = @linq dataset |>
groupby([:year , :firmids]) |>
transform(sum_size = sum(:size));
dataset.iv_size2 = dataset.Total_size - dataset.sum_size;
```

```julia
# 2SLS Regression for demand estimation
# First stage: regress price on Z and X

first_stage_result = reg(dataset, @formula(p ~ iv_size1+ iv_size2+hp2wt + air+mpgd+size), save = true, );
print(first_stage_result)
```

                                  Linear Model
    =========================================================================
    Number of obs:                 2217   Degrees of freedom:               7
    R2:                           0.592   R2 Adjusted:                  0.591
    F Statistic:                534.872   p-value:                      0.000
    =========================================================================
                  Estimate  Std.Error  t value Pr(>|t|)  Lower 95%  Upper 95%
    -------------------------------------------------------------------------
    iv_size1     -0.030544 0.00999712 -3.05528    0.002 -0.0501487 -0.0109393
    iv_size2     0.0919841 0.00809711  11.3601    0.000  0.0761054   0.107863
    hp2wt          25.8362    1.31783  19.6051    0.000    23.2518    28.4205
    air            9.57023   0.327809  29.1945    0.000    8.92738    10.2131
    mpgd         -0.272477  0.0270151 -10.0861    0.000  -0.325455  -0.219499
    size           2.25533   0.727117  3.10174    0.002   0.829424    3.68123
    (Intercept)   -5.21123    1.50736  -3.4572    0.001   -8.16721   -2.25525
    =========================================================================

```julia
# Second Stage: regress log(s_j)-log(s_0) on xhat

xhat = predict(first_stage_result, dataset);
dataset.p_iv = xhat;
second_stage_result = reg(dataset, @formula(log_s_j_0 ~ p_iv+hp2wt + air+mpgd+size), save = true);
print(second_stage_result)
```

                                    Linear Model
    ============================================================================
    Number of obs:                   2217  Degrees of freedom:                 6
    R2:                             0.355  R2 Adjusted:                    0.354
    F Statistic:                  243.848  p-value:                        0.000
    ============================================================================
                    Estimate  Std.Error  t value Pr(>|t|)  Lower 95%   Upper 95%
    ----------------------------------------------------------------------------
    p_iv           -0.289427  0.0156063 -18.5455    0.000  -0.320032   -0.258823
    hp2wt            5.63223   0.513603  10.9661    0.000    4.62503     6.63942
    air              2.18099   0.182327  11.9619    0.000    1.82344     2.53854
    mpgd         -0.00984143 0.00536771 -1.83345    0.067 -0.0203677 0.000684853
    size             2.19676   0.131213  16.7419    0.000    1.93944     2.45407
    (Intercept)      -9.5444   0.271767 -35.1198    0.000   -10.0773    -9.01145
    ============================================================================

```julia
price_coef = coef(second_stage_result)[2];
dataset.e_iv = price_coef * (ones(nrow(dataset))-dataset.s_j) .* dataset.p;
```

```julia
q1 = @from i in dataset begin;
            @where i.e_iv >-1
            @select {number_of_children=i.e_iv}
            @collect DataFrame
    end;
```

### Comparision with BLP Table 3, IV Logit Demand Column (Row: No. Inelastic De) in page 873.

#### Note that I have slightly different price coefficients, I observe number of inelastic demand good is 2. BLP estimates were 22.

```julia
nrow(q1)
```

    2

### Step 6. Discussion : IV regressions

#### Reported price coefficient ($\alpha$) is -0.0886 in OLS.
#### Now we have -0.2894  ($\alpha$) in IV regression. Prices are upward biased in OLS.

```julia
# OLS Results
print(result)
```

                                   Linear Model
    ===========================================================================
    Number of obs:                  2217   Degrees of freedom:                6
    R2:                            0.390   R2 Adjusted:                   0.388
    F Statistic:                  282.47   p-value:                       0.000
    ===========================================================================
                   Estimate  Std.Error   t value Pr(>|t|)  Lower 95%  Upper 95%
    ---------------------------------------------------------------------------
    p            -0.0886398 0.00401348  -22.0855    0.000 -0.0965104 -0.0807692
    hp2wt        -0.0731735   0.276701  -0.26445    0.791  -0.615794   0.469447
    air          -0.0380115  0.0726455 -0.523246    0.601  -0.180472   0.104449
    mpgd          0.0288388 0.00439518   6.56147    0.000  0.0202197  0.0374579
    size            2.40052   0.126801   18.9315    0.000    2.15186    2.64919
    (Intercept)    -10.2035    0.26002  -39.2412    0.000   -10.7134   -9.69356
    ===========================================================================

```julia
# 2SLS Results
print(second_stage_result)
```

                                    Linear Model
    ============================================================================
    Number of obs:                   2217  Degrees of freedom:                 6
    R2:                             0.355  R2 Adjusted:                    0.354
    F Statistic:                  243.848  p-value:                        0.000
    ============================================================================
                    Estimate  Std.Error  t value Pr(>|t|)  Lower 95%   Upper 95%
    ----------------------------------------------------------------------------
    p_iv           -0.289427  0.0156063 -18.5455    0.000  -0.320032   -0.258823
    hp2wt            5.63223   0.513603  10.9661    0.000    4.62503     6.63942
    air              2.18099   0.182327  11.9619    0.000    1.82344     2.53854
    mpgd         -0.00984143 0.00536771 -1.83345    0.067 -0.0203677 0.000684853
    size             2.19676   0.131213  16.7419    0.000    1.93944     2.45407
    (Intercept)      -9.5444   0.271767 -35.1198    0.000   -10.0773    -9.01145
    ============================================================================

## B. Monte Carlo Example: estimate logit-demand after solving Nash-Bertrand game
- B1. Data Generating Process
- B2. Obtain (numerically) equilibrium price and market shares
- B3. Regress using OLS / IV

### B1. Data Generating Process

Market is characterized by dupoly firms which procuce single good, with aggregate market shares, and price for each good. We assume that duopoly firms compete in 500 "independent" (isolated) markets.

In D.G.P., we solve Nash-Bertrand game so that we derive dupoly firm's price, and market shares. We use cost shifters and product characteritics to numerically solve this game. Since it is D.G.P. we use true parameters to obtain price, and market shares.

As an econometrician, we observe dupoly firm's market share, price, costs, and product characteristics.

The utility of each consumer $i$ in each market is given by:

\begin{equation}
    u_{ij} = \beta_0 + \beta_1 x_j + \sigma_d \xi_j - \alpha p_j + \epsilon_{ij}
\end{equation}
Marginal cost is constrained to be postitive and given by:

\begin{equation}
    c_j = e^{\gamma_0 + \gamma_x x_j + \sigma_c \xi_j + \gamma_w w_j + \sigma_\omega \omega_j}
\end{equation}

The exogenous data $x_j, \xi_j, w_j, $ and $\omega_j$ are all created standard normal random variables.

True parameter is given by:

| Parameter        | True Value | Description |
|------------------|------------|-------------|
| $ \beta_0$       | 5          |  Intercept (demand)          |
| $ \beta_x$       | 2          |  Utility from good $x$       |
| $ \sigma_d$      | 1 / 3 (second monte carlo)      |  Covariance                  |
| $\alpha $        | 1          |  Price coefficients          |
| $ \gamma_0$      | 1          |  Intercept (Supply)          |
| $\gamma_x $      | 5          |  Cost from good $x$          |
| $ \sigma_c$      | 0.25       |  Covariance  $(\xi_j)$                |
| $ \gamma_w$      | 0.25       |   Parameters for input costs |
| $ \sigma_\omega$ | 0.25       |   Covariance $(\omega_j)$                |

### B2. Obtain (numerically) equilibrium price and market shares (still D.G.P)
I solve nonlinear equation where under j=1,2. (Argument: $p_1, p_2, s_1(p_1, p_2), s_2(p_1,p_2)$)  $X$ is a vector dupoly firm's product charactersitics $ X = \{ x_1, x_2 \}$
Note that $s_0 = 1-s_1-s_2$

\begin{align}
p_1 & = c_1 - \frac{1}{\alpha (1-s_1)} \\
p_2 & = c_2 - \frac{1}{\alpha (1-s_2)} \\
& \text{Note that $s_1$, and $s_2$ is given by} \\
s_1(X,p_1, p_2) & = \frac{exp^{\beta_0 + \beta_1 x_1 + \sigma_d \xi_1 - \alpha p_1}}{1+exp^{\beta_0 + \beta_1 x_1 + \sigma_d \xi_1 - \alpha p_1} +exp^{\beta_0 + \beta_1 x_2 + \sigma_d \xi_2 - \alpha p_2} } \\
s_2(X,p_1, p_2) & = \frac{exp^{\beta_0 + \beta_1 x_2 + \sigma_d \xi_2 - \alpha p_2}}{1+exp^{\beta_0 + \beta_1 x_1 + \sigma_d \xi_1 - \alpha p_1} + exp^{\beta_0 + \beta_1 x_2 + \sigma_d \xi_2 - \alpha p_2}}
\end{align}

Using Nonlinear solver, we can obtain equilibrium outcome: $p_1, p_2, s_1, s_2 (s_0= 1-s_1-s_2).$ One might concern about multiple equilibria for this game. Since we solve single product under duopoly (which is simple market), we observe unique solution for this Monte Carlo Study. Please see Nalebuff (1991) for multi-product firm problem, or uniquness of this game.

### B3. Regress using OLS / IV
Following Berry's inversion an econometrician run following OLS/IV regression. An econometrician observes price, product characteristics, and cost shifters for 500 independent dupoly markets.

\begin{align}
\text{ln} (s_j) - \text{ln} (s_0) \ & =  \delta_j \\
& = \beta_0 + \beta_1 x_j - \alpha p_j + \sigma_d \xi_j
\end{align}

For IV, I use cost shifters, and competitors' product characteristics, as in Berry (1994)

Note that in OLS, since an econometrician cannot observe $\xi_j$ term, price coefficients $\alpha$ is upward biased. In the IV regression, the observed cost factors, $w_j$, and the product characteristic of the rival firm are used as instruments for price.

#### For each simulation, we independently draw these for 500 markets, as in Berry(1994).
#### Repeat 100 times, and report Monte Racrlo results

```julia
# Define parameters, as in Berry's 1994 Monte Carlo

beta_0 = 5.0
beta_x = 2.0
sigma_d = 1.0
alpha = -1.0

gamma_0 = 1.0
gamma_x = 0.5

sigma_c = 0.25
gamma_w = 0.25
sigma_omega = 0.25

T = 500
S = 100
d = Normal()
```

    Normal{Float64}(μ=0.0, σ=1.0)

```julia
# Define Non-linear solver

function f!(F, x)

    # D.G.P for true costs
    cost_1 = exp(gamma_0 + gamma_x * data_temp1[:x] + sigma_c * data_temp1[:xi] + gamma_w * data_temp1[:w] + sigma_omega * data_temp1[:omega])
    cost_2 = exp(gamma_0 + gamma_x * data_temp2[:x] + sigma_c * data_temp2[:xi] + gamma_w * data_temp2[:w] + sigma_omega * data_temp2[:omega])

    # Derive equillibrium price / quantity(markset shares)
    price_1 = cost_1 - 1/(alpha*(1-x[1]))
    price_2 = cost_2 - 1/(alpha*(1-x[2]))
    w
    #x[1]: market share of good 1
    #x[2]: market share of good 2

    denom = 1 + exp(beta_0 + beta_x*data_temp1[:x] + sigma_d*data_temp1[:xi] + alpha*price_1)  + exp(beta_0 + beta_x*data_temp2[:x] + sigma_d*data_temp2[:xi] + alpha*price_2)
    F[1] = x[1] - exp(beta_0 + beta_x*data_temp1[:x] + sigma_d*data_temp1[:xi] + alpha*price_1)/denom
    F[2] = x[2] - exp(beta_0 + beta_x*data_temp2[:x] + sigma_d*data_temp2[:xi] + alpha*price_2)/denom
end
```

### Replicate Table 1 in Berry(1994), column (1) and (2), where $\sigma_d=1$

```julia
### It takes 5 seconds, at the current workflow
sigma_d = 1

for s = 1:S
    ### Step B1. D.G.P.

    # If you want to have same results, you need to assign random seed
    Random.seed!(s*100+T+1)
    x_1 = rand(d, T);
    Random.seed!(s*100+T+2)
    xi_1 = rand(d, T);
    Random.seed!(s*100+T+3)
    w_1 = rand(d, T);
    Random.seed!(s*100+T+4)
    omega_1 = rand(d, T);

    Random.seed!(s*100+T+5)
    x_2 = rand(d, T);
    Random.seed!(s*100+T+6)
    xi_2 = rand(d, T);
    Random.seed!(s*100+T+7)
    w_2 = rand(d, T);
    Random.seed!(s*100+T+8)
    omega_2 = rand(d, T);

    data_1 = DataFrame(x = x_1[1:T], xi = xi_1[1:T], w = w_1[1:T], omega = omega_1[1:T], iv=x_2[1:T]);
    data_2 = DataFrame(x = x_2[1:T], xi = xi_2[1:T], w = w_2[1:T], omega = omega_2[1:T], iv=x_1[1:T]);

    # For the first periods

    data_temp1 = data_1[1,:]
    data_temp2 = data_2[1,:]
    global data_temp1, data_temp2

    ### Step B2. Solve Equilibrium price and market shares using nonlinear-solver

    a= nlsolve(f!, [0.1; 0.1])

    vector_s1 = [a.zero[1]]
    vector_s2 = [a.zero[2]]
    vector_s0 = [1-a.zero[1]-a.zero[2]]

    cost_1 = exp(gamma_0 + gamma_x * data_temp1[:x] + sigma_c * data_temp1[:xi] + gamma_w * data_temp1[:w] + sigma_omega * data_temp1[:omega])
    cost_2 = exp(gamma_0 + gamma_x * data_temp2[:x] + sigma_c * data_temp2[:xi] + gamma_w * data_temp2[:w] + sigma_omega * data_temp2[:omega])

    vector_p1 = [cost_1 - 1/(alpha*(1-a.zero[1]))]
    vector_p2 = [cost_2 - 1/(alpha*(1-a.zero[2]))]

    vector_delta_1 = [beta_0 + beta_x * data_temp1[:x] + alpha*(cost_1 - 1/(alpha*(1-a.zero[1])) )]
    vector_delta_2 = [beta_0 + beta_x * data_temp2[:x] + alpha*(cost_2 - 1/(alpha*(1-a.zero[2])) )]

    # From the second market to T markets.
    t=2
    for t = 2:T

        data_temp1 = data_1[t,:]
        data_temp2 = data_2[t,:]

        # Step 1. Solve Equilibrium price / market shares

        a= nlsolve(f!, [0.0; 0.0])

        append!(vector_s1, [a.zero[1]]);
        append!(vector_s2, [a.zero[2]]);
        append!(vector_s0, [1-a.zero[1]-a.zero[2]]);
        cost_1 = exp(gamma_0 + gamma_x * data_temp1[:x] + sigma_c * data_temp1[:xi] + gamma_w * data_temp1[:w] + sigma_omega * data_temp1[:omega])
        cost_2 = exp(gamma_0 + gamma_x * data_temp2[:x] + sigma_c * data_temp2[:xi] + gamma_w * data_temp2[:w] + sigma_omega * data_temp2[:omega])

        append!(vector_p1, [cost_1 - 1/(alpha*(1-a.zero[1]))]);
        append!(vector_p2, [cost_2 - 1/(alpha*(1-a.zero[2]))]);

        append!(vector_delta_1, [beta_0 + beta_x * data_temp1[:x] + alpha*(cost_1 - 1/(alpha*(1-a.zero[1])) )]);
        append!(vector_delta_2, [beta_0 + beta_x * data_temp2[:x] + alpha*(cost_2 - 1/(alpha*(1-a.zero[2])) )]);

    end

    data_1.price = vector_p1;
    data_2.price = vector_p2;

    data_1.s = vector_s1;
    data_2.s = vector_s2;

    data_1.delta = vector_delta_1;
    data_2.delta = vector_delta_2;

    data_1.log_sj_s0 = log.(vector_s1) - log.(vector_s0);
    data_2.log_sj_s0 = log.(vector_s2) - log.(vector_s0);

    # Merge into dataset
    data_merged =  append!(data_1, data_2);

    ### B3. Regress using OLS / IV

    ## OLS Regression

    ols_result = reg(data_merged, @formula(log_sj_s0 ~ x + price), save = true, Vcov.robust());
    ols_cons = coef(ols_result)[1];
    ols_x = coef(ols_result)[2];
    ols_p = coef(ols_result)[3];

    ## IV Regression
    first_stage_result = reg(data_merged, @formula(price ~  iv + w +x), save = true, Vcov.robust());

    xhat = predict(first_stage_result, data_merged);
    data_merged.xhat = xhat;

    iv_result = reg(data_merged, @formula(log_sj_s0 ~  x + xhat), save = true, Vcov.robust());
    iv_cons = coef(iv_result)[1];
    iv_x = coef(iv_result)[2];
    iv_p = coef(iv_result)[3];

    if s == 1
        vector_ols_cons = [ols_cons]
        vector_ols_x = [ols_x]
        vector_ols_p = [ols_p]

        vector_iv_cons = [iv_cons]
        vector_iv_x = [iv_x]
        vector_iv_p = [iv_p]

        # Store Monte Carlo Results
        global vector_ols_cons, vector_ols_x, vector_ols_p, vector_iv_cons, vector_iv_x, vector_iv_p
    else

        append!(vector_ols_cons, [ols_cons])
        append!(vector_ols_x, [ols_x])
        append!(vector_ols_p, [ols_p])

        append!(vector_iv_cons, [iv_cons])
        append!(vector_iv_x, [iv_x])
        append!(vector_iv_p, [iv_p])

    end
end

print("Monte Carlo Parameter Estimates 100 Random Samples of 500 Duopoly Markets  Logit Utility (sigma_d  = 1)")

result_summary =DataFrame( True_parameter = [beta_0, beta_x, alpha], OLS_mean = [mean(vector_ols_cons),mean(vector_ols_x),mean(vector_ols_p)], OLS_se = [std(vector_ols_cons),std(vector_ols_x),std(vector_ols_p)],
    IV_mean = [mean(vector_iv_cons),mean(vector_iv_x),mean(vector_iv_p)], IV_se =[std(vector_iv_cons),std(vector_iv_x),std(vector_iv_p)]);
print("Result Summary")
print(result_summary)
```

    Monte Carlo Parameter Estimates 100 Random Samples of 500 Duopoly Markets  Logit Utility (sigma_d  = 1)Result Summary3×5 DataFrame
    │ Row │ True_parameter │ OLS_mean │ OLS_se    │ IV_mean  │ IV_se     │
    │     │ Float64        │ Float64  │ Float64   │ Float64  │ Float64   │
    ├─────┼────────────────┼──────────┼───────────┼──────────┼───────────┤
    │ 1   │ 5.0            │ 3.1872   │ 0.235697  │ 5.01814  │ 0.266852  │
    │ 2   │ 2.0            │ 1.33611  │ 0.0742151 │ 2.01013  │ 0.0994837 │
    │ 3   │ -1.0           │ -0.63979 │ 0.0482097 │ -1.00436 │ 0.0513297 │

### Replicate Table 1 in Berry(1994), column (1) and (2), where $\sigma_d=3$

```julia
sigma_d = 3.0

for s = 1:S
    ### Step B1. D.G.P.

    # If you want to have same results, you need to assign random seed
    Random.seed!(s*100+T+1)
    x_1 = rand(d, T);
    Random.seed!(s*100+T+2)
    xi_1 = rand(d, T);
    Random.seed!(s*100+T+3)
    w_1 = rand(d, T);
    Random.seed!(s*100+T+4)
    omega_1 = rand(d, T);

    Random.seed!(s*100+T+5)
    x_2 = rand(d, T);
    Random.seed!(s*100+T+6)
    xi_2 = rand(d, T);
    Random.seed!(s*100+T+7)
    w_2 = rand(d, T);
    Random.seed!(s*100+T+8)
    omega_2 = rand(d, T);

    data_1 = DataFrame(x = x_1[1:T], xi = xi_1[1:T], w = w_1[1:T], omega = omega_1[1:T], iv=x_2[1:T]);
    data_2 = DataFrame(x = x_2[1:T], xi = xi_2[1:T], w = w_2[1:T], omega = omega_2[1:T], iv=x_1[1:T]);

    # For the first periods

    data_temp1 = data_1[1,:]
    data_temp2 = data_2[1,:]
    global data_temp1, data_temp2

    ### Step B2. Solve Equilibrium price and market shares using nonlinear-solver

    a= nlsolve(f!, [0.1; 0.1])

    vector_s1 = [a.zero[1]]
    vector_s2 = [a.zero[2]]
    vector_s0 = [1-a.zero[1]-a.zero[2]]

    cost_1 = exp(gamma_0 + gamma_x * data_temp1[:x] + sigma_c * data_temp1[:xi] + gamma_w * data_temp1[:w] + sigma_omega * data_temp1[:omega])
    cost_2 = exp(gamma_0 + gamma_x * data_temp2[:x] + sigma_c * data_temp2[:xi] + gamma_w * data_temp2[:w] + sigma_omega * data_temp2[:omega])

    vector_p1 = [cost_1 - 1/(alpha*(1-a.zero[1]))]
    vector_p2 = [cost_2 - 1/(alpha*(1-a.zero[2]))]

    vector_delta_1 = [beta_0 + beta_x * data_temp1[:x] + alpha*(cost_1 - 1/(alpha*(1-a.zero[1])) )]
    vector_delta_2 = [beta_0 + beta_x * data_temp2[:x] + alpha*(cost_2 - 1/(alpha*(1-a.zero[2])) )]

    # From the second market to T markets.
    t=2
    for t = 2:T

        data_temp1 = data_1[t,:]
        data_temp2 = data_2[t,:]

        # Step 1. Solve Equilibrium price / market shares

        a= nlsolve(f!, [0.0; 0.0])

        append!(vector_s1, [a.zero[1]]);
        append!(vector_s2, [a.zero[2]]);
        append!(vector_s0, [1-a.zero[1]-a.zero[2]]);
        cost_1 = exp(gamma_0 + gamma_x * data_temp1[:x] + sigma_c * data_temp1[:xi] + gamma_w * data_temp1[:w] + sigma_omega * data_temp1[:omega])
        cost_2 = exp(gamma_0 + gamma_x * data_temp2[:x] + sigma_c * data_temp2[:xi] + gamma_w * data_temp2[:w] + sigma_omega * data_temp2[:omega])

        append!(vector_p1, [cost_1 - 1/(alpha*(1-a.zero[1]))]);
        append!(vector_p2, [cost_2 - 1/(alpha*(1-a.zero[2]))]);

        append!(vector_delta_1, [beta_0 + beta_x * data_temp1[:x] + alpha*(cost_1 - 1/(alpha*(1-a.zero[1])) )]);
        append!(vector_delta_2, [beta_0 + beta_x * data_temp2[:x] + alpha*(cost_2 - 1/(alpha*(1-a.zero[2])) )]);

    end

    data_1.price = vector_p1;
    data_2.price = vector_p2;

    data_1.s = vector_s1;
    data_2.s = vector_s2;

    data_1.delta = vector_delta_1;
    data_2.delta = vector_delta_2;

    data_1.log_sj_s0 = log.(vector_s1) - log.(vector_s0);
    data_2.log_sj_s0 = log.(vector_s2) - log.(vector_s0);

    # Merge into dataset
    data_merged =  append!(data_1, data_2);

    ### B3. Regress using OLS / IV

    ## OLS Regression

    ols_result = reg(data_merged, @formula(log_sj_s0 ~ x + price), save = true, Vcov.robust());
    ols_cons = coef(ols_result)[1];
    ols_x = coef(ols_result)[2];
    ols_p = coef(ols_result)[3];

    ## IV Regression
    first_stage_result = reg(data_merged, @formula(price ~  iv + w +x), save = true, Vcov.robust());

    xhat = predict(first_stage_result, data_merged);
    data_merged.xhat = xhat;

    iv_result = reg(data_merged, @formula(log_sj_s0 ~  x + xhat), save = true, Vcov.robust());
    iv_cons = coef(iv_result)[1];
    iv_x = coef(iv_result)[2];
    iv_p = coef(iv_result)[3];

    if s == 1
        vector_ols_cons = [ols_cons]
        vector_ols_x = [ols_x]
        vector_ols_p = [ols_p]

        vector_iv_cons = [iv_cons]
        vector_iv_x = [iv_x]
        vector_iv_p = [iv_p]

        # Store Monte Carlo Results
        global vector_ols_cons, vector_ols_x, vector_ols_p, vector_iv_cons, vector_iv_x, vector_iv_p
    else

        append!(vector_ols_cons, [ols_cons])
        append!(vector_ols_x, [ols_x])
        append!(vector_ols_p, [ols_p])

        append!(vector_iv_cons, [iv_cons])
        append!(vector_iv_x, [iv_x])
        append!(vector_iv_p, [iv_p])

    end
end

print("Monte Carlo Parameter Estimates 100 Random Samples of 500 Duopoly Markets  Logit Utility (sigma_d  = 1)")

result_summary =DataFrame( True_parameter = [beta_0, beta_x, alpha], OLS_mean = [mean(vector_ols_cons),mean(vector_ols_x),mean(vector_ols_p)], OLS_se = [std(vector_ols_cons),std(vector_ols_x),std(vector_ols_p)],
    IV_mean = [mean(vector_iv_cons),mean(vector_iv_x),mean(vector_iv_p)], IV_se =[std(vector_iv_cons),std(vector_iv_x),std(vector_iv_p)]);
print("Result Summary")
print(result_summary)
```

    Monte Carlo Parameter Estimates 100 Random Samples of 500 Duopoly Markets  Logit Utility (sigma_d  = 3)3×5 DataFrame
    │ Row │ True_parameter │ OLS_mean  │ OLS_se    │ IV_mean  │ IV_se    │
    │     │ Float64        │ Float64   │ Float64   │ Float64  │ Float64  │
    ├─────┼────────────────┼───────────┼───────────┼──────────┼──────────┤
    │ 1   │ 5.0            │ -0.762803 │ 0.418897  │ 5.03208  │ 0.847055 │
    │ 2   │ 2.0            │ 0.0195958 │ 0.115257  │ 1.99649  │ 0.301829 │
    │ 3   │ -1.0           │ 0.105563  │ 0.0831606 │ -1.00709 │ 0.166926 │

```julia

```

# References

Berry, Steven T. "Estimating discrete-choice models of product differentiation." The RAND Journal of Economics (1994): 242-262.

Berry, Steven, James Levinsohn, and Ariel Pakes. "Automobile prices in market equilibrium." Econometrica: Journal of the Econometric Society (1995): 841-890.


# $\LaTeX$ Math
This is just markdown that can include latex math.

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

# System Info

```python
import IPython
print(IPython.sys_info())
```

    {'commit_hash': '7f10f7bb3',
     'commit_source': 'installation',
     'default_encoding': 'UTF-8',
     'ipython_path': '/Users/sk/anaconda3/envs/class/lib/python3.6/site-packages/IPython',
     'ipython_version': '6.4.0',
     'os_name': 'posix',
     'platform': 'Darwin-17.2.0-x86_64-i386-64bit',
     'sys_executable': '/Users/sk/anaconda3/envs/class/bin/python',
     'sys_platform': 'darwin',
     'sys_version': '3.6.5 |Anaconda custom (64-bit)| (default, Apr 26 2018, '
                    '08:42:37) \n'
                    '[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]'}

# Data

```python
import pandas as pd
import numpy as np
curve = pd.DataFrame(np.arange(0.0, 2.0, 0.01), columns=list('t'))
curve['x'] = 1 + np.sin(2 * np.pi * curve['t'])
curve
```

      t
      x

      0
      0.00
      1.000000

      1
      0.01
      1.062791

      2
      0.02
      1.125333

      3
      0.03
      1.187381

      4
      0.04
      1.248690

      5
      0.05
      1.309017

      6
      0.06
      1.368125

      7
      0.07
      1.425779

      8
      0.08
      1.481754

      9
      0.09
      1.535827

      10
      0.10
      1.587785

      11
      0.11
      1.637424

      12
      0.12
      1.684547

      13
      0.13
      1.728969

      14
      0.14
      1.770513

      15
      0.15
      1.809017

      16
      0.16
      1.844328

      17
      0.17
      1.876307

      18
      0.18
      1.904827

      19
      0.19
      1.929776

      20
      0.20
      1.951057

      21
      0.21
      1.968583

      22
      0.22
      1.982287

      23
      0.23
      1.992115

      24
      0.24
      1.998027

      25
      0.25
      2.000000

      26
      0.26
      1.998027

      27
      0.27
      1.992115

      28
      0.28
      1.982287

      29
      0.29
      1.968583

      ...
      ...
      ...

      170
      1.70
      0.048943

      171
      1.71
      0.031417

      172
      1.72
      0.017713

      173
      1.73
      0.007885

      174
      1.74
      0.001973

      175
      1.75
      0.000000

      176
      1.76
      0.001973

      177
      1.77
      0.007885

      178
      1.78
      0.017713

      179
      1.79
      0.031417

      180
      1.80
      0.048943

      181
      1.81
      0.070224

      182
      1.82
      0.095173

      183
      1.83
      0.123693

      184
      1.84
      0.155672

      185
      1.85
      0.190983

      186
      1.86
      0.229487

      187
      1.87
      0.271031

      188
      1.88
      0.315453

      189
      1.89
      0.362576

      190
      1.90
      0.412215

      191
      1.91
      0.464173

      192
      1.92
      0.518246

      193
      1.93
      0.574221

      194
      1.94
      0.631875

      195
      1.95
      0.690983

      196
      1.96
      0.751310

      197
      1.97
      0.812619

      198
      1.98
      0.874667

      199
      1.99
      0.937209

200 rows × 2 columns

# Plot

```python
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [8, 4]
curve.plot.line(x = 't', y = 'x')
```