# The Deconfounder in Action

In this notebook, we are going to see **the deconfounder in action**.

We will perform **causal inference** with the deconfounder on a **breast cancer** dataset.

**Goal:**
To convince all of us that the deconfounder is **easy** to use!

The **deconfounder** operates in three steps:

1.   **Fit** a factor model to the assigned causes; it leads to a candidate substitute confounder.
2.   **Check** the factor model with a predictive check.
3.   **Correct** for the substitute confounder in a causal inference.

Let's get started!

# Getting ready to work!

```python
!pip install tensorflow_probability
```

    Requirement already satisfied: tensorflow_probability in /usr/local/lib/python3.6/dist-packages (0.3.0)
    Requirement already satisfied: tensorflow>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.11.0rc2)
    Requirement already satisfied: numpy<=1.14.5,>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.14.5)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.11.0)
    Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (0.2.0)
    Requirement already satisfied: tensorboard<1.11.0,>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (1.10.0)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (0.31.1)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (1.15.0)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (1.1.0)
    Requirement already satisfied: setuptools<=39.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (39.1.0)
    Requirement already satisfied: keras-preprocessing>=1.0.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (1.0.3)
    Requirement already satisfied: keras-applications>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (1.0.5)
    Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (3.6.1)
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (0.7.1)
    Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.10.0->tensorflow_probability) (0.5.0)
    Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow>=1.10.0->tensorflow_probability) (0.14.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow>=1.10.0->tensorflow_probability) (3.0)
    Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras-preprocessing>=1.0.3->tensorflow>=1.10.0->tensorflow_probability) (0.19.1)
    Requirement already satisfied: keras>=2.1.6 in /usr/local/lib/python3.6/dist-packages (from keras-preprocessing>=1.0.3->tensorflow>=1.10.0->tensorflow_probability) (2.1.6)
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.5->tensorflow>=1.10.0->tensorflow_probability) (2.8.0)
    Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras>=2.1.6->keras-preprocessing>=1.0.3->tensorflow>=1.10.0->tensorflow_probability) (3.13)

```python
import tensorflow as tf
import numpy as np
import numpy.random as npr
import pandas as pd
import tensorflow as tf
import tensorflow_probability as tfp
import statsmodels.api as sm

from tensorflow_probability import edward2 as ed
from sklearn.datasets import load_breast_cancer
from pandas.tools.plotting import scatter_matrix
from scipy import sparse, stats
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve

import matplotlib
matplotlib.rcParams.update({'font.sans-serif' : 'Helvetica',
                            'axes.labelsize': 10,
                            'xtick.labelsize' : 6,
                            'ytick.labelsize' : 6,
                            'axes.titlesize' : 10})
import matplotlib.pyplot as plt

import seaborn as sns
color_names = ["windows blue",
               "amber",
               "crimson",
               "faded green",
               "dusty purple",
               "greyish"]
colors = sns.xkcd_palette(color_names)
sns.set(style="white", palette=sns.xkcd_palette(color_names))
```

    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/local/lib/python3.6/dist-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/local/lib/python3.6/dist-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)
    /usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
      return f(*args, **kwds)

```python
!pip show tensorflow
```

    Name: tensorflow
    Version: 1.11.0rc2
    Summary: TensorFlow is an open source machine learning framework for everyone.
    Home-page: https://www.tensorflow.org/
    Author: Google Inc.
    Author-email: opensource@google.com
    License: Apache 2.0
    Location: /usr/local/lib/python3.6/dist-packages
    Requires: keras-preprocessing, numpy, tensorboard, setuptools, absl-py, grpcio, gast, protobuf, wheel, astor, six, keras-applications, termcolor
    Required-by: tensorflow-probability

```python
!pip show tensorflow_probability
```

    Name: tensorflow-probability
    Version: 0.3.0
    Summary: Probabilistic modeling and statistical inference in TensorFlow
    Home-page: http://github.com/tensorflow/probability
    Author: Google LLC
    Author-email: no-reply@google.com
    License: Apache 2.0
    Location: /usr/local/lib/python3.6/dist-packages
    Requires: six, tensorflow, numpy
    Required-by:

```python
# set random seed so everyone gets the same number
import random
randseed = 123
print("random seed: ", randseed)
random.seed(randseed)
np.random.seed(randseed)
tf.set_random_seed(randseed)
```

    random seed:  123

## The scikit-learn breast cancer dataset

*   It is a data set about **breast cancer**.
*   We are interested in how tumor properties **affect** cancer diagnosis.
*   The **(multiple) causes** are tumor properties, e.g. sizes, compactness, symmetry, texture.
*   The **outcome** is tumor diagnosis, whether the breast cancer is diagnosed as malignant or benign.

```python
data = load_breast_cancer()
```

```python
print(data['DESCR'])
```

    Breast Cancer Wisconsin (Diagnostic) Database
    =============================================

    Notes
    -----
    Data Set Characteristics:
        :Number of Instances: 569

        :Number of Attributes: 30 numeric, predictive attributes and the class

        :Attribute Information:
            - radius (mean of distances from center to points on the perimeter)
            - texture (standard deviation of gray-scale values)
            - perimeter
            - area
            - smoothness (local variation in radius lengths)
            - compactness (perimeter^2 / area - 1.0)
            - concavity (severity of concave portions of the contour)
            - concave points (number of concave portions of the contour)
            - symmetry
            - fractal dimension ("coastline approximation" - 1)

            The mean, standard error, and "worst" or largest (mean of the three
            largest values) of these features were computed for each image,
            resulting in 30 features.  For instance, field 3 is Mean Radius, field
            13 is Radius SE, field 23 is Worst Radius.

            - class:
                    - WDBC-Malignant
                    - WDBC-Benign

        :Summary Statistics:

        ===================================== ====== ======
                                               Min    Max
        ===================================== ====== ======
        radius (mean):                        6.981  28.11
        texture (mean):                       9.71   39.28
        perimeter (mean):                     43.79  188.5
        area (mean):                          143.5  2501.0
        smoothness (mean):                    0.053  0.163
        compactness (mean):                   0.019  0.345
        concavity (mean):                     0.0    0.427
        concave points (mean):                0.0    0.201
        symmetry (mean):                      0.106  0.304
        fractal dimension (mean):             0.05   0.097
        radius (standard error):              0.112  2.873
        texture (standard error):             0.36   4.885
        perimeter (standard error):           0.757  21.98
        area (standard error):                6.802  542.2
        smoothness (standard error):          0.002  0.031
        compactness (standard error):         0.002  0.135
        concavity (standard error):           0.0    0.396
        concave points (standard error):      0.0    0.053
        symmetry (standard error):            0.008  0.079
        fractal dimension (standard error):   0.001  0.03
        radius (worst):                       7.93   36.04
        texture (worst):                      12.02  49.54
        perimeter (worst):                    50.41  251.2
        area (worst):                         185.2  4254.0
        smoothness (worst):                   0.071  0.223
        compactness (worst):                  0.027  1.058
        concavity (worst):                    0.0    1.252
        concave points (worst):               0.0    0.291
        symmetry (worst):                     0.156  0.664
        fractal dimension (worst):            0.055  0.208
        ===================================== ====== ======

        :Missing Attribute Values: None

        :Class Distribution: 212 - Malignant, 357 - Benign

        :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian

        :Donor: Nick Street

        :Date: November, 1995

    This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
    https://goo.gl/U2Uwz2

    Features are computed from a digitized image of a fine needle
    aspirate (FNA) of a breast mass.  They describe
    characteristics of the cell nuclei present in the image.

    Separating plane described above was obtained using
    Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree
    Construction Via Linear Programming." Proceedings of the 4th
    Midwest Artificial Intelligence and Cognitive Science Society,
    pp. 97-101, 1992], a classification method which uses linear
    programming to construct a decision tree.  Relevant features
    were selected using an exhaustive search in the space of 1-4
    features and 1-3 separating planes.

    The actual linear program used to obtain the separating plane
    in the 3-dimensional space is that described in:
    [K. P. Bennett and O. L. Mangasarian: "Robust Linear
    Programming Discrimination of Two Linearly Inseparable Sets",
    Optimization Methods and Software 1, 1992, 23-34].

    This database is also available through the UW CS ftp server:

    ftp ftp.cs.wisc.edu
    cd math-prog/cpo-dataset/machine-learn/WDBC/

    References
    ----------
       - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction
         for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on
         Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
         San Jose, CA, 1993.
       - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and
         prognosis via linear programming. Operations Research, 43(4), pages 570-577,
         July-August 1995.
       - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques
         to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994)
         163-171.

***For simplicity, we will work with the first 10 features, i.e. the mean radius/texture/perimeter***/....

```python
num_fea = 10
df = pd.DataFrame(data["data"][:,:num_fea], columns=data["feature_names"][:num_fea])
```

```python
df.shape
```

    (569, 10)

```python
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean radius</th>
      <th>mean texture</th>
      <th>mean perimeter</th>
      <th>mean area</th>
      <th>mean smoothness</th>
      <th>mean compactness</th>
      <th>mean concavity</th>
      <th>mean concave points</th>
      <th>mean symmetry</th>
      <th>mean fractal dimension</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>17.99</td>
      <td>10.38</td>
      <td>122.80</td>
      <td>1001.0</td>
      <td>0.11840</td>
      <td>0.27760</td>
      <td>0.3001</td>
      <td>0.14710</td>
      <td>0.2419</td>
      <td>0.07871</td>
    </tr>
    <tr>
      <th>1</th>
      <td>20.57</td>
      <td>17.77</td>
      <td>132.90</td>
      <td>1326.0</td>
      <td>0.08474</td>
      <td>0.07864</td>
      <td>0.0869</td>
      <td>0.07017</td>
      <td>0.1812</td>
      <td>0.05667</td>
    </tr>
    <tr>
      <th>2</th>
      <td>19.69</td>
      <td>21.25</td>
      <td>130.00</td>
      <td>1203.0</td>
      <td>0.10960</td>
      <td>0.15990</td>
      <td>0.1974</td>
      <td>0.12790</td>
      <td>0.2069</td>
      <td>0.05999</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11.42</td>
      <td>20.38</td>
      <td>77.58</td>
      <td>386.1</td>
      <td>0.14250</td>
      <td>0.28390</td>
      <td>0.2414</td>
      <td>0.10520</td>
      <td>0.2597</td>
      <td>0.09744</td>
    </tr>
    <tr>
      <th>4</th>
      <td>20.29</td>
      <td>14.34</td>
      <td>135.10</td>
      <td>1297.0</td>
      <td>0.10030</td>
      <td>0.13280</td>
      <td>0.1980</td>
      <td>0.10430</td>
      <td>0.1809</td>
      <td>0.05883</td>
    </tr>
  </tbody>
</table>
</div>

```python
dfy = data["target"]
```

```python
dfy.shape, dfy[:100] # binary outcomes
```

    ((569,),
     array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
            0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0,
            1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0,
            1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0]))

## Preparing the dataset for the deconfounder

### Only one step of preprocessing needed!

### We need to get rid of the highly correlated causes

**Why** do we need to get rid of highly correlated causes?

If two causes are **highly correlated**, a valid substitute confounder will largely **inflate the variance** of causal estimates downstream.

This phenomenon is **closely related to** the variance inflation phenomenon in linear regression.

***A more technical explanation (ignorable)***

Think of the extreme case where two causes are perfectly collinear $A_1 = 5A_2$. The only random variable Z that
$$A_1 \perp A_2 | Z,$$

$(A_1, A_2)$ **must** be a **deterministic function** of Z. For example, $Z = A_1$ or $Z = A_2$.

Such a substitute confounder Z **breaks one of the conditions** the deconfounder requires. See "***A note on overlap***" in the theory section of the paper.

**How** do we get rid of highly correlated causes?

*   We first make a **scatter plot** of **all pairs** of the causes.

*   It reveals which causes are **highly correlated**.

*   We will **exclude** these highly correlated causes by hand.

```python
sns.pairplot(df, size=1.5)
```

```python
# perimeter and area are highly correlated with radius
fea_cols = df.columns[[(not df.columns[i].endswith("perimeter")) \
                     and (not df.columns[i].endswith("area")) \
                     for i in range(df.shape[1])]]
```

```python
dfX = pd.DataFrame(df[fea_cols])

print(dfX.shape, dfy.shape)
```

    (569, 8) (569,)

### How does the dataset look like after preprocessing?

```python
# The causes
dfX.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean radius</th>
      <th>mean texture</th>
      <th>mean smoothness</th>
      <th>mean compactness</th>
      <th>mean concavity</th>
      <th>mean concave points</th>
      <th>mean symmetry</th>
      <th>mean fractal dimension</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>17.99</td>
      <td>10.38</td>
      <td>0.11840</td>
      <td>0.27760</td>
      <td>0.3001</td>
      <td>0.14710</td>
      <td>0.2419</td>
      <td>0.07871</td>
    </tr>
    <tr>
      <th>1</th>
      <td>20.57</td>
      <td>17.77</td>
      <td>0.08474</td>
      <td>0.07864</td>
      <td>0.0869</td>
      <td>0.07017</td>
      <td>0.1812</td>
      <td>0.05667</td>
    </tr>
    <tr>
      <th>2</th>
      <td>19.69</td>
      <td>21.25</td>
      <td>0.10960</td>
      <td>0.15990</td>
      <td>0.1974</td>
      <td>0.12790</td>
      <td>0.2069</td>
      <td>0.05999</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11.42</td>
      <td>20.38</td>
      <td>0.14250</td>
      <td>0.28390</td>
      <td>0.2414</td>
      <td>0.10520</td>
      <td>0.2597</td>
      <td>0.09744</td>
    </tr>
    <tr>
      <th>4</th>
      <td>20.29</td>
      <td>14.34</td>
      <td>0.10030</td>
      <td>0.13280</td>
      <td>0.1980</td>
      <td>0.10430</td>
      <td>0.1809</td>
      <td>0.05883</td>
    </tr>
  </tbody>
</table>
</div>

```python
# The outcome
dfy[:25]
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
           0, 0, 0])

# The dataset is ready. Let's do causal inference with the deconfounder!

## Step 1: Fit a factor model to the assigned causes; it leads to a substitute confounder.

### We start with trying out a random factor model. How about a probabilistic PCA model?

The matrix of assigned causes $X$

*   It has N=569 rows and D=8 columns.
*   N is the number of subjects/data points.
*   D is the number of causes/data dimension.

### Step 1.1: Some chores first...

#### Standardize the data
This step is optional to the deconfounder.

It only makes finding a good probabilistic PCA model easier.

```python
# dfX.std()
```

```python
# standardize the data for PPCA
X = np.array((dfX - dfX.mean())/dfX.std())
```

#### Then holdout some data!

We will later need to check the factor model with some heldout data.
So let's holdout some now.

```python
# randomly holdout some entries of X
num_datapoints, data_dim = X.shape

holdout_portion = 0.2
n_holdout = int(holdout_portion * num_datapoints * data_dim)

holdout_row = np.random.randint(num_datapoints, size=n_holdout)
holdout_col = np.random.randint(data_dim, size=n_holdout)
holdout_mask = (sparse.coo_matrix((np.ones(n_holdout), \
                            (holdout_row, holdout_col)), \
                            shape = X.shape)).toarray()

holdout_subjects = np.unique(holdout_row)

x_train = np.multiply(1-holdout_mask, X)
x_vad = np.multiply(holdout_mask, X)
```

### Step 1.2: We are ready to fit a probabilistic PCA model to x_train.

This step of "**fitting** a factor model" involves **inferring latent variables** in probability models.

We will rely on **Tensorflow Probability**, a library for probabilistic reasoning and statistical analysis in TensorFlow.

There are many **other probabilistic programming toolboxes** for fitting factor models, e.g. Pyro, Stan.

Some of the latent variable models can also be fit with **scikit-learn**.

We are free to use any of these with the deconfounder!

**What does a probabilistic PCA model look like?**

*  Probabilistic PCA is a dimensionality reduction technique. It models data with a lower dimensional latent space.

*  We consider the assigned causes of the $n$th subject. We write it as $\mathbf{x}_n$, which is a $D=8$ dimensional vector.

*  The probabilistic PCA assumes the following data generating process for each $\mathbf{x}_n$, $n = 1, ..., N$:

\begin{equation*}
\mathbf{z}_{n} \stackrel{iid}{\sim} N(\mathbf{0}, \mathbf{I}_K),
\end{equation*}

\begin{equation*}
\mathbf{x}_n \mid \mathbf{z}_n
\sim N(\mathbf{z}_n\mathbf{W}, \sigma^2\mathbf{I}_D).
\end{equation*}

*  We construct a $K$-dimensional substitute confounder $\mathbf{z}_{n}$ for each subject $n$, $n = 1, ..., N$.
*  Each $\mathbf{z}_{n}$ is a $K$-dimensional latent vector, $n = 1, ..., N$.

```python
def ppca_model(data_dim, latent_dim, num_datapoints, stddv_datapoints):
    w = ed.Normal(loc=tf.zeros([latent_dim, data_dim]),
                scale=tf.ones([latent_dim, data_dim]),
                name="w")  # parameter
    z = ed.Normal(loc=tf.zeros([num_datapoints, latent_dim]),
                scale=tf.ones([num_datapoints, latent_dim]),
                name="z")  # local latent variable / substitute confounder
    x = ed.Normal(loc=tf.multiply(tf.matmul(z, w), 1-holdout_mask),
                scale=stddv_datapoints * tf.ones([num_datapoints, data_dim]),
                name="x")  # (modeled) data
    return x, (w, z)

log_joint = ed.make_log_joint_fn(ppca_model)
```

**Let's fit a probabilistic PCA model with K = 2 latent dimensions.**

```python
latent_dim = 2
stddv_datapoints = 0.1

model = ppca_model(data_dim=data_dim,
                   latent_dim=latent_dim,
                   num_datapoints=num_datapoints,
                   stddv_datapoints=stddv_datapoints)
```

The cell below implements **variational inference** for probabilistic PCA  in tensorflow probability.

You are free to fit the probabilistic PCA in your favourite ways with your favourite package.

Note: approximate inference is perfectly fine!

It is orthogonal to our discussion around the deconfounder.

Let's **ignore** that for now (and forever).

```python
def variational_model(qw_mean, qw_stddv, qz_mean, qz_stddv):
    qw = ed.Normal(loc=qw_mean, scale=qw_stddv, name="qw")
    qz = ed.Normal(loc=qz_mean, scale=qz_stddv, name="qz")
    return qw, qz

log_q = ed.make_log_joint_fn(variational_model)

def target(w, z):
    """Unnormalized target density as a function of the parameters."""
    return log_joint(data_dim=data_dim,
                   latent_dim=latent_dim,
                   num_datapoints=num_datapoints,
                   stddv_datapoints=stddv_datapoints,
                   w=w, z=z, x=x_train)

def target_q(qw, qz):
    return log_q(qw_mean=qw_mean, qw_stddv=qw_stddv,
               qz_mean=qz_mean, qz_stddv=qz_stddv,
               qw=qw, qz=qz)

qw_mean = tf.Variable(np.ones([latent_dim, data_dim]), dtype=tf.float32)
qz_mean = tf.Variable(np.ones([num_datapoints, latent_dim]), dtype=tf.float32)
qw_stddv = tf.nn.softplus(tf.Variable(-4 * np.ones([latent_dim, data_dim]), dtype=tf.float32))
qz_stddv = tf.nn.softplus(tf.Variable(-4 * np.ones([num_datapoints, latent_dim]), dtype=tf.float32))

qw, qz = variational_model(qw_mean=qw_mean, qw_stddv=qw_stddv,
                           qz_mean=qz_mean, qz_stddv=qz_stddv)

energy = target(qw, qz)
entropy = -target_q(qw, qz)

elbo = energy + entropy

optimizer = tf.train.AdamOptimizer(learning_rate = 0.05)
train = optimizer.minimize(-elbo)

init = tf.global_variables_initializer()

t = []

num_epochs = 500

with tf.Session() as sess:
    sess.run(init)

    for i in range(num_epochs):
        sess.run(train)
        if i % 5 == 0:
            t.append(sess.run([elbo]))

        w_mean_inferred = sess.run(qw_mean)
        w_stddv_inferred = sess.run(qw_stddv)
        z_mean_inferred = sess.run(qz_mean)
        z_stddv_inferred = sess.run(qz_stddv)

print("Inferred axes:")
print(w_mean_inferred)
print("Standard Deviation:")
print(w_stddv_inferred)

plt.plot(range(1, num_epochs, 5), t)
plt.show()

def replace_latents(w, z):

    def interceptor(rv_constructor, *rv_args, **rv_kwargs):
        """Replaces the priors with actual values to generate samples from."""
        name = rv_kwargs.pop("name")
        if name == "w":
            rv_kwargs["value"] = w
        elif name == "z":
            rv_kwargs["value"] = z
        return rv_constructor(*rv_args, **rv_kwargs)

    return interceptor
```

So we just played some **magic** to **fit the probabilistic PCA** to the matrix of assigned causes $\mathbf{X}$.

**The only important thing here is: **

We have **inferred** the latent variables $\mathbf{z}_n, n=1, ..., N$ and the parameters $\mathbf{W}$.

Specifically, we have obtained from this step

```
w_mean_inferred,
w_stddv_inferred,
z_mean_inferred,
z_stddv_inferred.
```

## Step 2: Check the factor model with a predictive check.

Now we are ready to **check** the probabilistic PCA model.

The checking step is **very important** to the deconfounder.

Pleeeeeze **always** check the factor model!

**How** do we perform the predictive check?

1.  We will **generate** some replicated datasets for the heldout entries.
2.  And then **compare** the replicated datasets with the original dataset on the heldout entries.
3.  If they **look similar**, then we are good to go.

#### Step 2.1: We generate some replicated datasets first.

*  We will start with generating some **replicated datasets** from the predictive distribution of the assigned causes $X$:
\begin{align}
  p(\mathbf{X^{rep}_{n,heldout}} \,|\, \mathbf{X_{n, obs}}) =
  \int p(\mathbf{X_{n, heldout}} \,|\, \mathbf{z}_n) p(\mathbf{z_n} \,|\, \mathbf{X}_{n, obs}) \mathrm{d} \mathbf{z_n}.
\end{align}

*  That is, we generate these datasets from a probabilistic PCA model given the **inferred** latent variables $\hat{p}(\mathbf{z}_n)$ and $\hat{p}(\mathbf{W})$:

\begin{equation*}
\mathbf{z}_{n} \sim \hat{p}(\mathbf{z}_n),
\end{equation*}

\begin{equation*}
\mathbf{W} \sim \hat{p}(\mathbf{W}),
\end{equation*}

\begin{equation*}
\mathbf{x}_n \mid \mathbf{z}_n
\sim N(\mathbf{z}_n\mathbf{W}, \sigma^2\mathbf{I}_D).
\end{equation*}

*  These replicated datasets tell us what the assigned causes $X$ **should look like** if it is indeed generated by the fitted probabilistic PCA model.

```python
n_rep = 100 # number of replicated datasets we generate
holdout_gen = np.zeros((n_rep,*(x_train.shape)))

for i in range(n_rep):
    w_sample = npr.normal(w_mean_inferred, w_stddv_inferred)
    z_sample = npr.normal(z_mean_inferred, z_stddv_inferred)

    with ed.interception(replace_latents(w_sample, z_sample)):
        generate = ppca_model(
            data_dim=data_dim, latent_dim=latent_dim,
            num_datapoints=num_datapoints, stddv_datapoints=stddv_datapoints)

    with tf.Session() as sess:
        x_generated, _ = sess.run(generate)

    # look only at the heldout entries
    holdout_gen[i] = np.multiply(x_generated, holdout_mask)
```

#### Step 2.2: Then we compute the test statistic on both the original and the replicated dataset.

*   We use the **test statistic** of **expected heldout log likelihood**:
\begin{align}
  t(\mathbf{X_{n,heldout}}) = \mathbb{E}_{\mathbf{Z}, \mathbf{W}}[{\log p(\mathbf{X_{n,heldout}} \,|\, \mathbf{Z}, \mathbf{W}) \,|\,
  \mathbf{X_{n,obs}}}].
\end{align}

*   We calculate this test statistic **for each $n$** and for **both** the **original** dataset $\mathbf{X_{n,heldout}}$ and the **replicated** dataset $\mathbf{X^{rep}_{n,heldout}}$.

```python
n_eval = 100 # we draw samples from the inferred Z and W
obs_ll = []
rep_ll = []
for j in range(n_eval):
    w_sample = npr.normal(w_mean_inferred, w_stddv_inferred)
    z_sample = npr.normal(z_mean_inferred, z_stddv_inferred)

    holdoutmean_sample = np.multiply(z_sample.dot(w_sample), holdout_mask)
    obs_ll.append(np.mean(stats.norm(holdoutmean_sample, \
                        stddv_datapoints).logpdf(x_vad), axis=1))

    rep_ll.append(np.mean(stats.norm(holdoutmean_sample, \
                        stddv_datapoints).logpdf(holdout_gen),axis=2))

obs_ll_per_zi, rep_ll_per_zi = np.mean(np.array(obs_ll), axis=0), np.mean(np.array(rep_ll), axis=0)
```

#### Step 2.3: Finally we compare the test statistic of the original and the replicated dataset.

*   We compare the test statistics via the $p$-values.
\begin{equation*}
  \text{$p$-value} = p\left(t(\mathbf{X_{n,heldout}^{rep}}) < t(\mathbf{X_{n, heldout}})\right).
\end{equation*}

*   The **smaller** the $p$-value is, the **more different** the original dataset is from the replicated dataset.

*   We **fail** the check if the $p$-value is **small**.

*   Note this goes in the opposite direction to the conventional usage of $p$-values.

We compute a $p$-value for each $n$ and output the average $p$-values.

```python
pvals = np.array([np.mean(rep_ll_per_zi[:,i] < obs_ll_per_zi[i]) for i in range(num_datapoints)])
holdout_subjects = np.unique(holdout_row)
overall_pval = np.mean(pvals[holdout_subjects])
print("Predictive check p-values", overall_pval)
```

    Predictive check p-values 0.6842410714285714

**We passed the check!**

The substitute confounder $\mathbf{z}_n$ constructed in Step 1 is valid. We are ready to move on!

#### An optional step

We can also peak at **the predictive check of individual subjects**.

This step is just for fun. It is how we generate Figure 2 of the paper.

*   We randomly choose a subject.
*   Plot the kernel density estimate of the test statistic on the replicated datasets.
*   Plot the test statistic on the original dataset (the dashed vertical line).

```python
subject_no = npr.choice(holdout_subjects)
sns.kdeplot(rep_ll_per_zi[:,subject_no]).set_title("Predictive check for subject "+str(subject_no))
plt.axvline(x=obs_ll_per_zi[subject_no], linestyle='--')
```

## Step 3: Correct for the substitute confounder in a causal inference.

**How** to estimate causal effects?

*  For simplicity, we fit a logistic regression as an outcome model here.

*  The target is the observed outcome $y_n$, $n=1,\ldots, N$.

*  The regressor is the multiple causes $\mathbf{X}_n$, $n=1,\ldots, N$.

**How** to correct for the substitute confounder?

*  We include the substitute confounder $\mathbf{Z}_n$, $n=1,\ldots, N$, into the regressors.

```python
# approximate the (random variable) substitute confounders with their inferred mean.
Z_hat = z_mean_inferred
# augment the regressors to be both the assigned causes X and the substitute confounder Z
X_aug = np.column_stack([X, Z_hat])
```

```python
# holdout some data from prediction later
X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
```

```python
dcfX_train = sm.add_constant(X_train)
dcflogit_model = sm.Logit(y_train, dcfX_train)
dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
print(dcfresult.summary())
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13267453685445485
                Iterations: 90
                Function evaluations: 90
                Gradient evaluations: 90
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      444
    Method:                           MLE   Df Model:                           10
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7974
    Time:                        03:25:06   Log-Likelihood:                -60.367
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 8.566e-96
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7940      0.259      3.068      0.002       0.287       1.301
    x1            -2.7822      0.965     -2.883      0.004      -4.674      -0.891
    x2            -1.3808      0.398     -3.472      0.001      -2.160      -0.601
    x3            -1.0748      0.722     -1.490      0.136      -2.489       0.339
    x4             0.6899      0.840      0.821      0.412      -0.957       2.337
    x5            -0.7173      0.838     -0.856      0.392      -2.359       0.925
    x6            -1.4009      1.263     -1.109      0.267      -3.877       1.075
    x7            -0.2357      0.515     -0.458      0.647      -1.245       0.774
    x8             0.6517      0.811      0.804      0.421      -0.937       2.240
    x9            -2.0235      1.631     -1.241      0.215      -5.220       1.173
    x10           -0.9468      1.768     -0.536      0.592      -4.411       2.518
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.17 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)

```python
res = pd.DataFrame({"causal_mean": dcfresult.params[:data_dim+1], \
                  "causal_std": dcfresult.bse[:data_dim+1], \
                  "causal_025": dcfresult.conf_int()[:data_dim+1,0], \
                  "causal_975": dcfresult.conf_int()[:data_dim+1,1], \
                   "causal_pval": dcfresult.pvalues[:data_dim+1]})
res["causal_sig"] = (res["causal_pval"] < 0.05)
res = res.T
res.columns = np.concatenate([["intercept"], np.array(dfX.columns)])
res = res.T
```

```python
res
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>causal_025</th>
      <th>causal_975</th>
      <th>causal_mean</th>
      <th>causal_pval</th>
      <th>causal_std</th>
      <th>causal_sig</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>intercept</th>
      <td>0.286743</td>
      <td>1.30119</td>
      <td>0.793969</td>
      <td>0.00215524</td>
      <td>0.258794</td>
      <td>True</td>
    </tr>
    <tr>
      <th>mean radius</th>
      <td>-4.67359</td>
      <td>-0.890708</td>
      <td>-2.78215</td>
      <td>0.00393983</td>
      <td>0.965039</td>
      <td>True</td>
    </tr>
    <tr>
      <th>mean texture</th>
      <td>-2.16031</td>
      <td>-0.601335</td>
      <td>-1.38082</td>
      <td>0.000516645</td>
      <td>0.397705</td>
      <td>True</td>
    </tr>
    <tr>
      <th>mean smoothness</th>
      <td>-2.48904</td>
      <td>0.339427</td>
      <td>-1.07481</td>
      <td>0.136341</td>
      <td>0.721561</td>
      <td>False</td>
    </tr>
    <tr>
      <th>mean compactness</th>
      <td>-0.957365</td>
      <td>2.33711</td>
      <td>0.689871</td>
      <td>0.411736</td>
      <td>0.840442</td>
      <td>False</td>
    </tr>
    <tr>
      <th>mean concavity</th>
      <td>-2.35923</td>
      <td>0.924694</td>
      <td>-0.717269</td>
      <td>0.391896</td>
      <td>0.837751</td>
      <td>False</td>
    </tr>
    <tr>
      <th>mean concave points</th>
      <td>-3.87681</td>
      <td>1.07505</td>
      <td>-1.40088</td>
      <td>0.267452</td>
      <td>1.26325</td>
      <td>False</td>
    </tr>
    <tr>
      <th>mean symmetry</th>
      <td>-1.24517</td>
      <td>0.773808</td>
      <td>-0.235679</td>
      <td>0.647253</td>
      <td>0.515054</td>
      <td>False</td>
    </tr>
    <tr>
      <th>mean fractal dimension</th>
      <td>-0.936826</td>
      <td>2.24028</td>
      <td>0.651725</td>
      <td>0.421338</td>
      <td>0.8105</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

We check the predictions to see if the logistic outcome model is a good outcome model.

```python
# make predictions with the causal model
dcfX_test = X_test
dcfy_predprob = dcfresult.predict(sm.add_constant(dcfX_test))
dcfy_pred = (dcfy_predprob > 0.5)
print(classification_report(y_test, dcfy_pred))
```

                 precision    recall  f1-score   support

              0       0.96      0.91      0.93        47
              1       0.94      0.97      0.96        67

    avg / total       0.95      0.95      0.95       114

# We are done!

We have computed the average causal effect of raising the causes by one unit (see the "causal mean" column above).

# Is the deconfounder worth the effort?

We finally compare the **causal** estimation (with the deconfounder) with the **noncausal** estimation (with vanilla regression).

## The classical logistic regression! Note it is noncausal :-(

```python
# regress the outcome against the causes only (no substitute confounders)
nodcfX_train = sm.add_constant(X_train[:,:X.shape[1]])
nodcflogit_model = sm.Logit(y_train, nodcfX_train)
nodcfresult = nodcflogit_model.fit_regularized(maxiter=5000)
print(nodcfresult.summary())
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13451539390572878
                Iterations: 71
                Function evaluations: 71
                Gradient evaluations: 71
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      446
    Method:                           MLE   Df Model:                            8
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7946
    Time:                        03:25:09   Log-Likelihood:                -61.205
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 3.283e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7632      0.255      2.998      0.003       0.264       1.262
    x1            -3.2642      0.904     -3.611      0.000      -5.036      -1.492
    x2            -1.7189      0.307     -5.602      0.000      -2.320      -1.117
    x3            -1.2257      0.498     -2.462      0.014      -2.202      -0.250
    x4             0.3115      0.741      0.420      0.674      -1.142       1.765
    x5            -1.1317      0.717     -1.578      0.115      -2.538       0.274
    x6            -2.0079      1.172     -1.714      0.087      -4.304       0.289
    x7            -0.5413      0.324     -1.673      0.094      -1.176       0.093
    x8             0.5775      0.677      0.853      0.394      -0.750       1.905
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)

```python
res["noncausal_mean"] = np.array(nodcfresult.params)
res["noncausal_std"] = np.array(nodcfresult.bse)
res["noncausal_025"] = np.array(nodcfresult.conf_int()[:,0])
res["noncausal_975"] = np.array(nodcfresult.conf_int()[:,1])
res["noncausal_pval"] = np.array(nodcfresult.pvalues)
res["noncausal_sig"] = (res["noncausal_pval"] < 0.05)
```

```python
res["diff"] = res["causal_mean"] - res["noncausal_mean"]
res["pval_diff"] = res["causal_pval"] - res["noncausal_pval"]
```

```python
nodcfX_test = sm.add_constant(X_test[:,:X.shape[1]])
nodcfy_predprob = nodcfresult.predict(nodcfX_test)
nodcfy_pred = (nodcfy_predprob > 0.5)
```

**Causal models do not hurt predictions here!**

```python
dcflogit_roc_auc = roc_auc_score(y_test, dcfy_pred)
dcffpr, dcftpr, dcfthresholds = roc_curve(y_test, dcfy_predprob)
nodcflogit_roc_auc = roc_auc_score(y_test, nodcfy_pred)
nodcffpr, nodcftpr, nodcfthresholds = roc_curve(y_test, nodcfy_predprob)
plt.figure()
plt.plot(nodcffpr, nodcftpr, label='Noncausal Logistic Regression (area = %0.9f)' % nodcflogit_roc_auc)
plt.plot(dcffpr, dcftpr, label='Causal Logistic Regression (area = %0.9f)' % dcflogit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()
```

**But causal models do change the regression coefficients and which features are significant.**

*   The mean smoothness is a feature **significantly correlated** with the cancer diagnosis.

*   But it does **not significantly** **causally affect** the cancer diagnosis.

*   The effect of all features are **over-estimated** with the noncausal model except the "mean compactness".

```python
res.sort_values("pval_diff", ascending=True)[["pval_diff", "causal_pval", "noncausal_pval", "causal_sig", "noncausal_sig", "causal_mean", "noncausal_mean"]]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>pval_diff</th>
      <th>causal_pval</th>
      <th>noncausal_pval</th>
      <th>causal_sig</th>
      <th>noncausal_sig</th>
      <th>causal_mean</th>
      <th>noncausal_mean</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>mean compactness</th>
      <td>-0.262641</td>
      <td>0.411736</td>
      <td>6.743764e-01</td>
      <td>False</td>
      <td>False</td>
      <td>0.689871</td>
      <td>0.311494</td>
    </tr>
    <tr>
      <th>intercept</th>
      <td>-0.000561357</td>
      <td>0.00215524</td>
      <td>2.716595e-03</td>
      <td>True</td>
      <td>True</td>
      <td>0.793969</td>
      <td>0.763172</td>
    </tr>
    <tr>
      <th>mean texture</th>
      <td>0.000516623</td>
      <td>0.000516645</td>
      <td>2.123746e-08</td>
      <td>True</td>
      <td>True</td>
      <td>-1.38082</td>
      <td>-1.718860</td>
    </tr>
    <tr>
      <th>mean radius</th>
      <td>0.00363465</td>
      <td>0.00393983</td>
      <td>3.051830e-04</td>
      <td>True</td>
      <td>True</td>
      <td>-2.78215</td>
      <td>-3.264181</td>
    </tr>
    <tr>
      <th>mean fractal dimension</th>
      <td>0.027555</td>
      <td>0.421338</td>
      <td>3.937831e-01</td>
      <td>False</td>
      <td>False</td>
      <td>0.651725</td>
      <td>0.577489</td>
    </tr>
    <tr>
      <th>mean smoothness</th>
      <td>0.122505</td>
      <td>0.136341</td>
      <td>1.383540e-02</td>
      <td>False</td>
      <td>True</td>
      <td>-1.07481</td>
      <td>-1.225668</td>
    </tr>
    <tr>
      <th>mean concave points</th>
      <td>0.180868</td>
      <td>0.267452</td>
      <td>8.658376e-02</td>
      <td>False</td>
      <td>False</td>
      <td>-1.40088</td>
      <td>-2.007921</td>
    </tr>
    <tr>
      <th>mean concavity</th>
      <td>0.277228</td>
      <td>0.391896</td>
      <td>1.146687e-01</td>
      <td>False</td>
      <td>False</td>
      <td>-0.717269</td>
      <td>-1.131677</td>
    </tr>
    <tr>
      <th>mean symmetry</th>
      <td>0.552847</td>
      <td>0.647253</td>
      <td>9.440683e-02</td>
      <td>False</td>
      <td>False</td>
      <td>-0.235679</td>
      <td>-0.541292</td>
    </tr>
  </tbody>
</table>
</div>

*   We include causes into the regression **one-by-one**.
*   The deconfounder coefficients **does not** flip signs.
*   But classical logistic regression coefficients **does** flip signs.
*   This suggests that **the deconfounder is causal**.
*   It is because **causal** coefficients **do not change** as we include more variables into the system; causal estimation already controls for confounders so that it is causal.
*   However, **correlation** coefficients **can change** as we include more variables into the system; if the added variable is a confounder, than the regression coefficients change to account for the confounding effects.

```python
# The deconfounder with causes added one-by-one
# The first i coefficient is the causal coefficient of the first i causes.
# i = 1, ..., 8.
for i in range(X.shape[1]):
    print(i, "causes included")
    # augment the regressors to be both the assigned causes X and the substitute confounder Z
    X_aug = np.column_stack([X[:,:i], Z_hat])
    # holdout some data from prediction later
    X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
    dcfX_train = sm.add_constant(X_train)
    dcflogit_model = sm.Logit(y_train, dcfX_train)
    dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
    print(dcfresult.summary())
```

    0 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.17884001225800394
                Iterations: 22
                Function evaluations: 22
                Gradient evaluations: 22
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      452
    Method:                           MLE   Df Model:                            2
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7269
    Time:                        03:51:41   Log-Likelihood:                -81.372
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 8.406e-95
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8855      0.209      4.246      0.000       0.477       1.294
    x1            -5.4067      0.590     -9.166      0.000      -6.563      -4.251
    x2            -0.7133      0.214     -3.341      0.001      -1.132      -0.295
    ==============================================================================
    1 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.15939489902128479
                Iterations: 30
                Function evaluations: 31
                Gradient evaluations: 30
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      451
    Method:                           MLE   Df Model:                            3
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7566
    Time:                        03:51:41   Log-Likelihood:                -72.525
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 2.052e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7089      0.219      3.239      0.001       0.280       1.138
    x1            -1.9883      0.523     -3.800      0.000      -3.014      -0.963
    x2            -4.2342      0.678     -6.243      0.000      -5.563      -2.905
    x3            -1.1495      0.262     -4.390      0.000      -1.663      -0.636
    ==============================================================================
    2 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.15013118448235743
                Iterations: 38
                Function evaluations: 38
                Gradient evaluations: 38
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      450
    Method:                           MLE   Df Model:                            4
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7708
    Time:                        03:51:41   Log-Likelihood:                -68.310
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 4.117e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7423      0.229      3.240      0.001       0.293       1.191
    x1            -3.2170      0.743     -4.332      0.000      -4.672      -1.762
    x2            -1.0083      0.354     -2.848      0.004      -1.702      -0.314
    x3            -2.6749      0.868     -3.082      0.002      -4.376      -0.974
    x4            -1.7639      0.377     -4.675      0.000      -2.503      -1.024
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.11 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    3 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.14017451882851403
                Iterations: 45
                Function evaluations: 45
                Gradient evaluations: 45
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      449
    Method:                           MLE   Df Model:                            5
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7860
    Time:                        03:51:41   Log-Likelihood:                -63.779
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 5.220e-99
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8874      0.248      3.583      0.000       0.402       1.373
    x1            -3.2516      0.771     -4.218      0.000      -4.763      -1.741
    x2            -1.0842      0.366     -2.960      0.003      -1.802      -0.366
    x3            -1.3478      0.478     -2.818      0.005      -2.285      -0.410
    x4            -2.9646      0.867     -3.421      0.001      -4.663      -1.266
    x5            -0.7219      0.537     -1.345      0.179      -1.773       0.330
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    4 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.1372557735182052
                Iterations: 55
                Function evaluations: 55
                Gradient evaluations: 55
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      448
    Method:                           MLE   Df Model:                            6
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7904
    Time:                        03:51:41   Log-Likelihood:                -62.451
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 1.426e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8356      0.249      3.352      0.001       0.347       1.324
    x1            -3.2847      0.774     -4.245      0.000      -4.801      -1.768
    x2            -1.0759      0.363     -2.967      0.003      -1.787      -0.365
    x3            -1.0307      0.467     -2.209      0.027      -1.945      -0.116
    x4             1.1497      0.690      1.667      0.095      -0.202       2.501
    x5            -3.7983      1.018     -3.730      0.000      -5.794      -1.802
    x6            -1.8075      0.808     -2.237      0.025      -3.391      -0.224
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    5 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13599998759971046
                Iterations: 68
                Function evaluations: 68
                Gradient evaluations: 68
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      447
    Method:                           MLE   Df Model:                            7
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7923
    Time:                        03:51:41   Log-Likelihood:                -61.880
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 7.500e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8226      0.251      3.276      0.001       0.330       1.315
    x1            -3.5500      0.816     -4.352      0.000      -5.149      -1.951
    x2            -1.1995      0.379     -3.165      0.002      -1.942      -0.457
    x3            -1.4058      0.601     -2.340      0.019      -2.583      -0.228
    x4             1.0955      0.682      1.605      0.108      -0.242       2.433
    x5            -0.7355      0.691     -1.064      0.287      -2.090       0.619
    x6            -2.9752      1.254     -2.373      0.018      -5.433      -0.517
    x7            -1.1289      1.025     -1.102      0.271      -3.137       0.880
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.18 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    6 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13417129786592752
                Iterations: 72
                Function evaluations: 72
                Gradient evaluations: 72
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      446
    Method:                           MLE   Df Model:                            8
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7951
    Time:                        03:51:41   Log-Likelihood:                -61.048
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 2.813e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7640      0.256      2.989      0.003       0.263       1.265
    x1            -3.0625      0.872     -3.511      0.000      -4.772      -1.353
    x2            -1.2930      0.379     -3.408      0.001      -2.037      -0.549
    x3            -1.0003      0.658     -1.520      0.129      -2.290       0.290
    x4             1.2091      0.693      1.744      0.081      -0.149       2.568
    x5            -0.3404      0.745     -0.457      0.648      -1.801       1.120
    x6            -1.5167      1.189     -1.276      0.202      -3.847       0.813
    x7            -2.4900      1.285     -1.938      0.053      -5.008       0.028
    x8            -1.1159      1.020     -1.094      0.274      -3.115       0.883
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    7 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13340033060780868
                Iterations: 77
                Function evaluations: 77
                Gradient evaluations: 77
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      445
    Method:                           MLE   Df Model:                            9
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7963
    Time:                        03:51:41   Log-Likelihood:                -60.697
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 1.584e-96
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7680      0.256      2.996      0.003       0.266       1.271
    x1            -3.1355      0.878     -3.569      0.000      -4.857      -1.414
    x2            -1.3825      0.400     -3.453      0.001      -2.167      -0.598
    x3            -1.2121      0.710     -1.707      0.088      -2.604       0.180
    x4             0.9028      0.798      1.131      0.258      -0.662       2.467
    x5            -0.6337      0.834     -0.760      0.447      -2.268       1.000
    x6            -1.7179      1.222     -1.406      0.160      -4.113       0.677
    x7            -0.3981      0.486     -0.818      0.413      -1.352       0.555
    x8            -1.7104      1.601     -1.068      0.285      -4.849       1.428
    x9            -0.1970      1.533     -0.128      0.898      -3.202       2.808
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)

```python
# Logistic regression with causes added one-by-one
# The first i coefficient is the causal coefficient of the first i causes.
# i = 1, ..., 8.
for i in range(X.shape[1]):
    print(i, "causes included")
    # augment the regressors to be both the assigned causes X and the substitute confounder Z
    X_aug = np.column_stack([X[:,:i]])
    # holdout some data from prediction later
    X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
    dcfX_train = sm.add_constant(X_train)
    dcflogit_model = sm.Logit(y_train, dcfX_train)
    dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
    print(dcfresult.summary())
```

    0 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.6549205599225008
                Iterations: 6
                Function evaluations: 6
                Gradient evaluations: 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      454
    Method:                           MLE   Df Model:                            0
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:              -6.217e-15
    Time:                        03:50:48   Log-Likelihood:                -297.99
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                       nan
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.5639      0.098      5.783      0.000       0.373       0.755
    ==============================================================================
    1 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.2953829271568602
                Iterations: 15
                Function evaluations: 15
                Gradient evaluations: 15
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      453
    Method:                           MLE   Df Model:                            1
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.5490
    Time:                        03:50:48   Log-Likelihood:                -134.40
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 3.955e-73
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6566      0.155      4.225      0.000       0.352       0.961
    x1            -3.5061      0.353     -9.934      0.000      -4.198      -2.814
    ==============================================================================
    2 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.2592096971350075
                Iterations: 20
                Function evaluations: 20
                Gradient evaluations: 20
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      452
    Method:                           MLE   Df Model:                            2
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.6042
    Time:                        03:50:48   Log-Likelihood:                -117.94
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 6.397e-79
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6744      0.168      4.018      0.000       0.345       1.003
    x1            -3.6479      0.390     -9.347      0.000      -4.413      -2.883
    x2            -0.9815      0.181     -5.424      0.000      -1.336      -0.627
    ==============================================================================
    3 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.162465976738943
                Iterations: 28
                Function evaluations: 29
                Gradient evaluations: 28
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      451
    Method:                           MLE   Df Model:                            3
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7519
    Time:                        03:50:48   Log-Likelihood:                -73.922
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 8.272e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9963      0.231      4.306      0.000       0.543       1.450
    x1            -4.9102      0.601     -8.170      0.000      -6.088      -3.732
    x2            -1.7394      0.284     -6.115      0.000      -2.297      -1.182
    x3            -2.2080      0.336     -6.578      0.000      -2.866      -1.550
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.12 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    4 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.15485253636195323
                Iterations: 35
                Function evaluations: 35
                Gradient evaluations: 35
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      450
    Method:                           MLE   Df Model:                            4
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7636
    Time:                        03:50:48   Log-Likelihood:                -70.458
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 3.495e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9646      0.233      4.146      0.000       0.509       1.421
    x1            -4.6125      0.614     -7.509      0.000      -5.816      -3.409
    x2            -1.6563      0.295     -5.610      0.000      -2.235      -1.078
    x3            -1.7342      0.372     -4.667      0.000      -2.463      -1.006
    x4            -0.9279      0.353     -2.631      0.009      -1.619      -0.237
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.13 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    5 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.14242454080229494
                Iterations: 44
                Function evaluations: 45
                Gradient evaluations: 44
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      449
    Method:                           MLE   Df Model:                            5
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7825
    Time:                        03:50:48   Log-Likelihood:                -64.803
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 1.444e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8234      0.242      3.396      0.001       0.348       1.299
    x1            -4.6125      0.636     -7.252      0.000      -5.859      -3.366
    x2            -1.7131      0.303     -5.648      0.000      -2.308      -1.119
    x3            -1.9515      0.395     -4.941      0.000      -2.726      -1.177
    x4             0.4077      0.512      0.797      0.425      -0.595       1.410
    x5            -1.7514      0.464     -3.777      0.000      -2.660      -0.843
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    6 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13837349335376822
                Iterations: 60
                Function evaluations: 60
                Gradient evaluations: 60
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      448
    Method:                           MLE   Df Model:                            6
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7887
    Time:                        03:50:48   Log-Likelihood:                -62.960
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 2.361e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7376      0.248      2.975      0.003       0.252       1.223
    x1            -3.5398      0.807     -4.384      0.000      -5.122      -1.957
    x2            -1.6821      0.302     -5.563      0.000      -2.275      -1.089
    x3            -1.3522      0.491     -2.756      0.006      -2.314      -0.391
    x4             0.6207      0.527      1.177      0.239      -0.413       1.655
    x5            -1.0073      0.623     -1.617      0.106      -2.228       0.214
    x6            -2.1317      1.138     -1.873      0.061      -4.362       0.098
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.14 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    7 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.135339334890979
                Iterations: 62
                Function evaluations: 62
                Gradient evaluations: 62
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      447
    Method:                           MLE   Df Model:                            7
    Date:                Wed, 26 Sep 2018   Pseudo R-squ.:                  0.7933
    Time:                        03:50:48   Log-Likelihood:                -61.579
    converged:                       True   LL-Null:                       -297.99
                                            LLR p-value:                 5.570e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7287      0.250      2.916      0.004       0.239       1.219
    x1            -3.6556      0.796     -4.592      0.000      -5.216      -2.095
    x2            -1.7323      0.307     -5.644      0.000      -2.334      -1.131
    x3            -1.1122      0.483     -2.304      0.021      -2.058      -0.166
    x4             0.7408      0.542      1.367      0.172      -0.321       1.803
    x5            -0.8364      0.614     -1.363      0.173      -2.039       0.366
    x6            -2.2682      1.139     -1.992      0.046      -4.500      -0.037
    x7            -0.5397      0.324     -1.666      0.096      -1.175       0.095
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)
    /usr/local/lib/python3.6/dist-packages/statsmodels/discrete/discrete_model.py:2405: DeprecationWarning: `chisqprob` is deprecated!
    stats.chisqprob is deprecated in scipy 0.17.0; use stats.distributions.chi2.sf instead.
      return stats.chisqprob(self.llr, self.df_model)

**We note that the causal coefficient of x4 is stable with the (causal) deconfounder but the correlation coefficent of x4 flips sign with the (noncausal) logistic regression.**

# Takeaways

*   The deconfounder is **not hard** to use.
*   We simply **fit** a factor model, **check** it, and **infer** causal effects with the substitute confounder.
*   Please **always check** the factor model!
*   The deconfounder **makes a difference**.
*   The deconfounder **deconfounds**.

# Acknowledgements

We thank Suresh Naidu for suggesting the adding-causes-one-by-one idea.