# The Deconfounder in Action

In this notebook, we are going to see **the deconfounder in action**.

We will perform **causal inference** with the deconfounder on a **breast cancer** dataset.

**Goal:**
To convince all of us that the deconfounder is **easy** to use!

The **deconfounder** operates in three steps:

1.   **Fit** a factor model to the assigned causes; it leads to a candidate substitute confounder.
2.   **Check** the factor model with a predictive check.
3.   **Correct** for the substitute confounder in a causal inference.

Let's get started!

# Getting ready to work!

```python
!pip install tensorflow_probability
```

    Requirement already satisfied: tensorflow_probability in /usr/local/lib/python3.6/dist-packages (0.9.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.12.0)
    Requirement already satisfied: gast>=0.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (0.3.3)
    Requirement already satisfied: cloudpickle>=1.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.3.0)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (1.18.2)
    Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from tensorflow_probability) (4.4.2)

```python
%tensorflow_version 1.x
import tensorflow as tf
import numpy as np
import numpy.random as npr
import pandas as pd
import tensorflow as tf
import tensorflow_probability as tfp
import statsmodels.api as sm

from tensorflow_probability import edward2 as ed
from sklearn.datasets import load_breast_cancer
from pandas.plotting import scatter_matrix
from scipy import sparse, stats
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve

import matplotlib
matplotlib.rcParams.update({'font.sans-serif' : 'Helvetica',
                            'axes.labelsize': 10,
                            'xtick.labelsize' : 6,
                            'ytick.labelsize' : 6,
                            'axes.titlesize' : 10})
import matplotlib.pyplot as plt

import seaborn as sns
color_names = ["windows blue",
               "amber",
               "crimson",
               "faded green",
               "dusty purple",
               "greyish"]
colors = sns.xkcd_palette(color_names)
sns.set(style="white", palette=sns.xkcd_palette(color_names), color_codes = False)
```

    TensorFlow 1.x selected.

    /usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```python
!pip show tensorflow
```

    Name: tensorflow
    Version: 1.15.2
    Summary: TensorFlow is an open source machine learning framework for everyone.
    Home-page: https://www.tensorflow.org/
    Author: Google Inc.
    Author-email: packages@tensorflow.org
    License: Apache 2.0
    Location: /tensorflow-1.15.2/python3.6
    Requires: absl-py, keras-preprocessing, astor, tensorboard, tensorflow-estimator, wrapt, six, google-pasta, keras-applications, numpy, opt-einsum, termcolor, grpcio, protobuf, wheel, gast
    Required-by: stable-baselines, magenta, fancyimpute

```python
!pip show tensorflow_probability
```

    Name: tensorflow-probability
    Version: 0.7.0
    Summary: Probabilistic modeling and statistical inference in TensorFlow
    Home-page: http://github.com/tensorflow/probability
    Author: Google LLC
    Author-email: no-reply@google.com
    License: Apache 2.0
    Location: /tensorflow-1.15.2/python3.6
    Requires: decorator, cloudpickle, six, numpy
    Required-by: tensorflow-gan, tensor2tensor, magenta, kfac, dm-sonnet

```python
# set random seed so everyone gets the same number
import random
randseed = 123
print("random seed: ", randseed)
random.seed(randseed)
np.random.seed(randseed)
tf.set_random_seed(randseed)
```

    random seed:  123

## The scikit-learn breast cancer dataset

*   It is a data set about **breast cancer**.
*   We are interested in how tumor properties **affect** cancer diagnosis.
*   The **(multiple) causes** are tumor properties, e.g. sizes, compactness, symmetry, texture.
*   The **outcome** is tumor diagnosis, whether the breast cancer is diagnosed as malignant or benign.

```python
data = load_breast_cancer()
```

```python
print(data['DESCR'])
```

    .. _breast_cancer_dataset:

    Breast cancer wisconsin (diagnostic) dataset
    --------------------------------------------

    **Data Set Characteristics:**

        :Number of Instances: 569

        :Number of Attributes: 30 numeric, predictive attributes and the class

        :Attribute Information:
            - radius (mean of distances from center to points on the perimeter)
            - texture (standard deviation of gray-scale values)
            - perimeter
            - area
            - smoothness (local variation in radius lengths)
            - compactness (perimeter^2 / area - 1.0)
            - concavity (severity of concave portions of the contour)
            - concave points (number of concave portions of the contour)
            - symmetry
            - fractal dimension ("coastline approximation" - 1)

            The mean, standard error, and "worst" or largest (mean of the three
            largest values) of these features were computed for each image,
            resulting in 30 features.  For instance, field 3 is Mean Radius, field
            13 is Radius SE, field 23 is Worst Radius.

            - class:
                    - WDBC-Malignant
                    - WDBC-Benign

        :Summary Statistics:

        ===================================== ====== ======
                                               Min    Max
        ===================================== ====== ======
        radius (mean):                        6.981  28.11
        texture (mean):                       9.71   39.28
        perimeter (mean):                     43.79  188.5
        area (mean):                          143.5  2501.0
        smoothness (mean):                    0.053  0.163
        compactness (mean):                   0.019  0.345
        concavity (mean):                     0.0    0.427
        concave points (mean):                0.0    0.201
        symmetry (mean):                      0.106  0.304
        fractal dimension (mean):             0.05   0.097
        radius (standard error):              0.112  2.873
        texture (standard error):             0.36   4.885
        perimeter (standard error):           0.757  21.98
        area (standard error):                6.802  542.2
        smoothness (standard error):          0.002  0.031
        compactness (standard error):         0.002  0.135
        concavity (standard error):           0.0    0.396
        concave points (standard error):      0.0    0.053
        symmetry (standard error):            0.008  0.079
        fractal dimension (standard error):   0.001  0.03
        radius (worst):                       7.93   36.04
        texture (worst):                      12.02  49.54
        perimeter (worst):                    50.41  251.2
        area (worst):                         185.2  4254.0
        smoothness (worst):                   0.071  0.223
        compactness (worst):                  0.027  1.058
        concavity (worst):                    0.0    1.252
        concave points (worst):               0.0    0.291
        symmetry (worst):                     0.156  0.664
        fractal dimension (worst):            0.055  0.208
        ===================================== ====== ======

        :Missing Attribute Values: None

        :Class Distribution: 212 - Malignant, 357 - Benign

        :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian

        :Donor: Nick Street

        :Date: November, 1995

    This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
    https://goo.gl/U2Uwz2

    Features are computed from a digitized image of a fine needle
    aspirate (FNA) of a breast mass.  They describe
    characteristics of the cell nuclei present in the image.

    Separating plane described above was obtained using
    Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree
    Construction Via Linear Programming." Proceedings of the 4th
    Midwest Artificial Intelligence and Cognitive Science Society,
    pp. 97-101, 1992], a classification method which uses linear
    programming to construct a decision tree.  Relevant features
    were selected using an exhaustive search in the space of 1-4
    features and 1-3 separating planes.

    The actual linear program used to obtain the separating plane
    in the 3-dimensional space is that described in:
    [K. P. Bennett and O. L. Mangasarian: "Robust Linear
    Programming Discrimination of Two Linearly Inseparable Sets",
    Optimization Methods and Software 1, 1992, 23-34].

    This database is also available through the UW CS ftp server:

    ftp ftp.cs.wisc.edu
    cd math-prog/cpo-dataset/machine-learn/WDBC/

    .. topic:: References

       - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction
         for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on
         Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
         San Jose, CA, 1993.
       - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and
         prognosis via linear programming. Operations Research, 43(4), pages 570-577,
         July-August 1995.
       - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques
         to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994)
         163-171.

***For simplicity, we will work with the first 10 features, i.e. the mean radius/texture/perimeter***/....

```python
num_fea = 10
df = pd.DataFrame(data["data"][:,:num_fea], columns=data["feature_names"][:num_fea])
```

```python
df.shape
```

    (569, 10)

```python
df.head()
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883

```python
dfy = data["target"]
```

```python
dfy.shape, dfy[:100] # binary outcomes
```

    ((569,),
     array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
            0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0,
            1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0,
            1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0]))

## Preparing the dataset for the deconfounder

### Only one step of preprocessing needed!

### We need to get rid of the highly correlated causes

**Why** do we need to get rid of highly correlated causes?

If two causes are **highly correlated**, a valid substitute confounder will largely **inflate the variance** of causal estimates downstream.

This phenomenon is **closely related to** the variance inflation phenomenon in linear regression.

***A more technical explanation (ignorable)***

Think of the extreme case where two causes are perfectly collinear $A_1 = 5A_2$. The only random variable Z that
$$A_1 \perp A_2 | Z,$$

$(A_1, A_2)$ **must** be a **deterministic function** of Z. For example, $Z = A_1$ or $Z = A_2$.

Such a substitute confounder Z **breaks one of the conditions** the deconfounder requires. See "***A note on overlap***" in the theory section of the paper.

**How** do we get rid of highly correlated causes?

*   We first make a **scatter plot** of **all pairs** of the causes.

*   It reveals which causes are **highly correlated**.

*   We will **exclude** these highly correlated causes by hand.

```python
sns.pairplot(df, size=1.5)
```

```python
# perimeter and area are highly correlated with radius
fea_cols = df.columns[[(not df.columns[i].endswith("perimeter")) \
                     and (not df.columns[i].endswith("area")) \
                     for i in range(df.shape[1])]]
```

```python
dfX = pd.DataFrame(df[fea_cols])

print(dfX.shape, dfy.shape)
```

    (569, 8) (569,)

### How does the dataset look like after preprocessing?

```python
# The causes
dfX.head()
```

      mean radius
      mean texture
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension

      0
      17.99
      10.38
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871

      1
      20.57
      17.77
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667

      2
      19.69
      21.25
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999

      3
      11.42
      20.38
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744

      4
      20.29
      14.34
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883

```python
# The outcome
dfy[:25]
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
           0, 0, 0])

# The dataset is ready. Let's do causal inference with the deconfounder!

## Step 1: Fit a factor model to the assigned causes; it leads to a substitute confounder.

### We start with trying out a random factor model. How about a probabilistic PCA model?

The matrix of assigned causes $X$

*   It has N=569 rows and D=8 columns.
*   N is the number of subjects/data points.
*   D is the number of causes/data dimension.

### Step 1.1: Some chores first...

#### Standardize the data
This step is optional to the deconfounder.

It only makes finding a good probabilistic PCA model easier.

```python
# dfX.std()
```

```python
# standardize the data for PPCA
X = np.array((dfX - dfX.mean())/dfX.std())
```

#### Then holdout some data!

We will later need to check the factor model with some heldout data.
So let's holdout some now.

```python
# randomly holdout some entries of X
num_datapoints, data_dim = X.shape

holdout_portion = 0.2
n_holdout = int(holdout_portion * num_datapoints * data_dim)

holdout_row = np.random.randint(num_datapoints, size=n_holdout)
holdout_col = np.random.randint(data_dim, size=n_holdout)
holdout_mask = (sparse.coo_matrix((np.ones(n_holdout), \
                            (holdout_row, holdout_col)), \
                            shape = X.shape)).toarray()

holdout_subjects = np.unique(holdout_row)
holdout_mask = np.minimum(1, holdout_mask)

x_train = np.multiply(1-holdout_mask, X)
x_vad = np.multiply(holdout_mask, X)
```

### Step 1.2: We are ready to fit a probabilistic PCA model to x_train.

This step of "**fitting** a factor model" involves **inferring latent variables** in probability models.

We will rely on **Tensorflow Probability**, a library for probabilistic reasoning and statistical analysis in TensorFlow.

There are many **other probabilistic programming toolboxes** for fitting factor models, e.g. Pyro, Stan.

Some of the latent variable models can also be fit with **scikit-learn**.

We are free to use any of these with the deconfounder!

**What does a probabilistic PCA model look like?**

*  Probabilistic PCA is a dimensionality reduction technique. It models data with a lower dimensional latent space.

*  We consider the assigned causes of the $n$th subject. We write it as $\mathbf{x}_n$, which is a $D=8$ dimensional vector.

*  The probabilistic PCA assumes the following data generating process for each $\mathbf{x}_n$, $n = 1, ..., N$:

\begin{equation*}
\mathbf{z}_{n} \stackrel{iid}{\sim} N(\mathbf{0}, \mathbf{I}_K),
\end{equation*}

\begin{equation*}
\mathbf{x}_n \mid \mathbf{z}_n
\sim N(\mathbf{z}_n\mathbf{W}, \sigma^2\mathbf{I}_D).
\end{equation*}

*  We construct a $K$-dimensional substitute confounder $\mathbf{z}_{n}$ for each subject $n$, $n = 1, ..., N$.
*  Each $\mathbf{z}_{n}$ is a $K$-dimensional latent vector, $n = 1, ..., N$.

```python
# we allow both linear and quadratic model
# for linear model x_n has mean z_n * W
# for quadratic model x_n has mean b + z_n * W + (z_n**2) * W_2
# quadractice model needs to change the checking step accordingly

def ppca_model(data_dim, latent_dim, num_datapoints, stddv_datapoints, mask, form="linear"):
    w = ed.Normal(loc=tf.zeros([latent_dim, data_dim]),
                scale=tf.ones([latent_dim, data_dim]),
                name="w")  # parameter
    z = ed.Normal(loc=tf.zeros([num_datapoints, latent_dim]),
                scale=tf.ones([num_datapoints, latent_dim]),
                name="z")  # local latent variable / substitute confounder
    if form == "linear":
        x = ed.Normal(loc=tf.multiply(tf.matmul(z, w), mask),
                    scale=stddv_datapoints * tf.ones([num_datapoints, data_dim]),
                    name="x")  # (modeled) data
    elif form == "quadratic":
        b = ed.Normal(loc=tf.zeros([1, data_dim]),
                scale=tf.ones([1, data_dim]),
                name="b")  # intercept
        w2 = ed.Normal(loc=tf.zeros([latent_dim, data_dim]),
                scale=tf.ones([latent_dim, data_dim]),
                name="w2")  # quadratic parameter
        x = ed.Normal(loc=tf.multiply(b + tf.matmul(z, w) + tf.matmul(tf.square(z), w2), mask),
                    scale=stddv_datapoints * tf.ones([num_datapoints, data_dim]),
                    name="x")  # (modeled) data
    return x, (w, z)

log_joint = ed.make_log_joint_fn(ppca_model)
```

**Let's fit a probabilistic PCA model.**

```python
latent_dim = 2
stddv_datapoints = 0.1

model = ppca_model(data_dim=data_dim,
                   latent_dim=latent_dim,
                   num_datapoints=num_datapoints,
                   stddv_datapoints=stddv_datapoints,
                   mask=1-holdout_mask)
```

The cell below implements **variational inference** for probabilistic PCA  in tensorflow probability.

You are free to fit the probabilistic PCA in your favourite ways with your favourite package.

Note: approximate inference is perfectly fine!

It is orthogonal to our discussion around the deconfounder.

Let's **ignore** that for now (and forever).

```python
def variational_model(qb_mean, qb_stddv, qw_mean, qw_stddv,
                      qw2_mean, qw2_stddv, qz_mean, qz_stddv):
    qb = ed.Normal(loc=qb_mean, scale=qb_stddv, name="qb")
    qw = ed.Normal(loc=qw_mean, scale=qw_stddv, name="qw")
    qw2 = ed.Normal(loc=qw2_mean, scale=qw2_stddv, name="qw2")
    qz = ed.Normal(loc=qz_mean, scale=qz_stddv, name="qz")
    return qb, qw, qw2, qz

log_q = ed.make_log_joint_fn(variational_model)

def target(b, w, w2, z):
    """Unnormalized target density as a function of the parameters."""
    return log_joint(data_dim=data_dim,
                   latent_dim=latent_dim,
                   num_datapoints=num_datapoints,
                   stddv_datapoints=stddv_datapoints,
                   mask=1-holdout_mask,
                   w=w, z=z, w2=w2, b=b, x=x_train)

def target_q(qb, qw, qw2, qz):
    return log_q(qb_mean=qb_mean, qb_stddv=qb_stddv,
                 qw_mean=qw_mean, qw_stddv=qw_stddv,
                 qw2_mean=qw2_mean, qw2_stddv=qw2_stddv,
                 qz_mean=qz_mean, qz_stddv=qz_stddv,
                 qw=qw, qz=qz, qw2=qw2, qb=qb)

qb_mean = tf.Variable(np.ones([1, data_dim]), dtype=tf.float32)
qw_mean = tf.Variable(np.ones([latent_dim, data_dim]), dtype=tf.float32)
qw2_mean = tf.Variable(np.ones([latent_dim, data_dim]), dtype=tf.float32)
qz_mean = tf.Variable(np.ones([num_datapoints, latent_dim]), dtype=tf.float32)
qb_stddv = tf.nn.softplus(tf.Variable(0 * np.ones([1, data_dim]), dtype=tf.float32))
qw_stddv = tf.nn.softplus(tf.Variable(-4 * np.ones([latent_dim, data_dim]), dtype=tf.float32))
qw2_stddv = tf.nn.softplus(tf.Variable(-4 * np.ones([latent_dim, data_dim]), dtype=tf.float32))
qz_stddv = tf.nn.softplus(tf.Variable(-4 * np.ones([num_datapoints, latent_dim]), dtype=tf.float32))

qb, qw, qw2, qz = variational_model(qb_mean=qb_mean, qb_stddv=qb_stddv,
                                    qw_mean=qw_mean, qw_stddv=qw_stddv,
                                    qw2_mean=qw2_mean, qw2_stddv=qw2_stddv,
                                    qz_mean=qz_mean, qz_stddv=qz_stddv)

energy = target(qb, qw, qw2, qz)
entropy = -target_q(qb, qw, qw2, qz)

elbo = energy + entropy

optimizer = tf.train.AdamOptimizer(learning_rate = 0.05)
train = optimizer.minimize(-elbo)

init = tf.global_variables_initializer()

t = []

num_epochs = 500

with tf.Session() as sess:
    sess.run(init)

    for i in range(num_epochs):
        sess.run(train)
        if i % 5 == 0:
            t.append(sess.run([elbo]))

        b_mean_inferred = sess.run(qb_mean)
        b_stddv_inferred = sess.run(qb_stddv)
        w_mean_inferred = sess.run(qw_mean)
        w_stddv_inferred = sess.run(qw_stddv)
        w2_mean_inferred = sess.run(qw2_mean)
        w2_stddv_inferred = sess.run(qw2_stddv)
        z_mean_inferred = sess.run(qz_mean)
        z_stddv_inferred = sess.run(qz_stddv)

print("Inferred axes:")
print(w_mean_inferred)
print("Standard Deviation:")
print(w_stddv_inferred)

plt.plot(range(1, num_epochs, 5), t)
plt.show()

def replace_latents(b, w, w2, z):

    def interceptor(rv_constructor, *rv_args, **rv_kwargs):
        """Replaces the priors with actual values to generate samples from."""
        name = rv_kwargs.pop("name")
        if name == "b":
            rv_kwargs["value"] = b
        elif name == "w":
            rv_kwargs["value"] = w
        elif name == "w":
            rv_kwargs["value"] = w2
        elif name == "z":
            rv_kwargs["value"] = z
        return rv_constructor(*rv_args, **rv_kwargs)

    return interceptor
```

So we just played some **magic** to **fit the probabilistic PCA** to the matrix of assigned causes $\mathbf{X}$.

**The only important thing here is: **

We have **inferred** the latent variables $\mathbf{z}_n, n=1, ..., N$ and the parameters $\mathbf{W}$.

Specifically, we have obtained from this step

```
w_mean_inferred,
w_stddv_inferred,
z_mean_inferred,
z_stddv_inferred.
```

## Step 2: Check the factor model with a predictive check.

Now we are ready to **check** the probabilistic PCA model.

The checking step is **very important** to the deconfounder.

Pleeeeeze **always** check the factor model!

**How** do we perform the predictive check?

1.  We will **generate** some replicated datasets for the heldout entries.
2.  And then **compare** the replicated datasets with the original dataset on the heldout entries.
3.  If they **look similar**, then we are good to go.

#### Step 2.1: We generate some replicated datasets first.

*  We will start with generating some **replicated datasets** from the predictive distribution of the assigned causes $X$:
\begin{align}
  p(\mathbf{X^{rep}_{n,heldout}} \,|\, \mathbf{X_{n, obs}}) =
  \int p(\mathbf{X_{n, heldout}} \,|\, \mathbf{z}_n) p(\mathbf{z_n} \,|\, \mathbf{X}_{n, obs}) \mathrm{d} \mathbf{z_n}.
\end{align}

*  That is, we generate these datasets from a probabilistic PCA model given the **inferred** latent variables $\hat{p}(\mathbf{z}_n)$ and $\hat{p}(\mathbf{W})$:

\begin{equation*}
\mathbf{z}_{n} \sim \hat{p}(\mathbf{z}_n),
\end{equation*}

\begin{equation*}
\mathbf{W} \sim \hat{p}(\mathbf{W}),
\end{equation*}

\begin{equation*}
\mathbf{x}_n \mid \mathbf{z}_n
\sim N(\mathbf{z}_n\mathbf{W}, \sigma^2\mathbf{I}_D).
\end{equation*}

*  These replicated datasets tell us what the assigned causes $X$ **should look like** if it is indeed generated by the fitted probabilistic PCA model.

```python
n_rep = 100 # number of replicated datasets we generate
holdout_gen = np.zeros((n_rep,*(x_train.shape)))

for i in range(n_rep):
    b_sample = npr.normal(b_mean_inferred, b_stddv_inferred)
    w_sample = npr.normal(w_mean_inferred, w_stddv_inferred)
    w2_sample = npr.normal(w2_mean_inferred, w2_stddv_inferred)
    z_sample = npr.normal(z_mean_inferred, z_stddv_inferred)

    with ed.interception(replace_latents(b_sample, w_sample, w2_sample, z_sample)):
        generate = ppca_model(
            data_dim=data_dim, latent_dim=latent_dim,
            num_datapoints=num_datapoints, stddv_datapoints=stddv_datapoints,
            mask=np.ones(x_train.shape))

    with tf.Session() as sess:
        x_generated, _ = sess.run(generate)

    # look only at the heldout entries
    holdout_gen[i] = np.multiply(x_generated, holdout_mask)
```

#### Step 2.2: Then we compute the test statistic on both the original and the replicated dataset.

*   We use the **test statistic** of **expected heldout log likelihood**:
\begin{align}
  t(\mathbf{X_{n,heldout}}) = \mathbb{E}_{\mathbf{Z}, \mathbf{W}}[{\log p(\mathbf{X_{n,heldout}} \,|\, \mathbf{Z}, \mathbf{W}) \,|\,
  \mathbf{X_{n,obs}}}].
\end{align}

*   We calculate this test statistic **for each $n$** and for **both** the **original** dataset $\mathbf{X_{n,heldout}}$ and the **replicated** dataset $\mathbf{X^{rep}_{n,heldout}}$.

```python
n_eval = 100 # we draw samples from the inferred Z and W
obs_ll = []
rep_ll = []
for j in range(n_eval):
    w_sample = npr.normal(w_mean_inferred, w_stddv_inferred)
    z_sample = npr.normal(z_mean_inferred, z_stddv_inferred)

    holdoutmean_sample = np.multiply(z_sample.dot(w_sample), holdout_mask)
    obs_ll.append(np.mean(stats.norm(holdoutmean_sample, \
                        stddv_datapoints).logpdf(x_vad), axis=1))

    rep_ll.append(np.mean(stats.norm(holdoutmean_sample, \
                        stddv_datapoints).logpdf(holdout_gen),axis=2))

obs_ll_per_zi, rep_ll_per_zi = np.mean(np.array(obs_ll), axis=0), np.mean(np.array(rep_ll), axis=0)
```

#### Step 2.3: Finally we compare the test statistic of the original and the replicated dataset.

*   We compare the test statistics via the $p$-values.
\begin{equation*}
  \text{$p$-value} = p\left(t(\mathbf{X_{n,heldout}^{rep}}) < t(\mathbf{X_{n, heldout}})\right).
\end{equation*}

*   The **smaller** the $p$-value is, the **more different** the original dataset is from the replicated dataset.

*   We **fail** the check if the $p$-value is **small**.

*   Note this goes in the opposite direction to the conventional usage of $p$-values.

We compute a $p$-value for each $n$ and output the average $p$-values.

```python
pvals = np.array([np.mean(rep_ll_per_zi[:,i] < obs_ll_per_zi[i]) for i in range(num_datapoints)])
holdout_subjects = np.unique(holdout_row)
overall_pval = np.mean(pvals[holdout_subjects])
print("Predictive check p-values", overall_pval)
```

    Predictive check p-values 0.10207589285714287

**We passed the check!**

The substitute confounder $\mathbf{z}_n$ constructed in Step 1 is valid. We are ready to move on!

#### An optional step

We can also peak at **the predictive check of individual subjects**.

This step is just for fun. It is how we generate Figure 2 of the paper.

*   We randomly choose a subject.
*   Plot the kernel density estimate of the test statistic on the replicated datasets.
*   Plot the test statistic on the original dataset (the dashed vertical line).

```python
subject_no = npr.choice(holdout_subjects)
sns.kdeplot(rep_ll_per_zi[:,subject_no]).set_title("Predictive check for subject "+str(subject_no))
plt.axvline(x=obs_ll_per_zi[subject_no], linestyle='--')
```

## Step 3: Correct for the substitute confounder in a causal inference.

**How** to estimate causal effects?

*  For simplicity, we fit a logistic regression as an outcome model here.

*  The target is the observed outcome $y_n$, $n=1,\ldots, N$.

*  The regressor is the multiple causes $\mathbf{X}_n$, $n=1,\ldots, N$.

**How** to correct for the substitute confounder?

*  We include the substitute confounder $\mathbf{Z}_n$, $n=1,\ldots, N$, into the regressors.

```python
# approximate the (random variable) substitute confounders with their inferred mean.
Z_hat = z_mean_inferred
# augment the regressors to be both the assigned causes X and the substitute confounder Z
X_aug = np.column_stack([X, Z_hat])
```

```python
# holdout some data from prediction later
X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
```

```python
dcfX_train = sm.add_constant(X_train)
dcflogit_model = sm.Logit(y_train, dcfX_train)
dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
print(dcfresult.summary())
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13286976182886223
                Iterations: 85
                Function evaluations: 85
                Gradient evaluations: 85
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      444
    Method:                           MLE   Df Model:                           10
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7971
    Time:                        23:28:08   Log-Likelihood:                -60.456
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 9.348e-96
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7997      0.260      3.074      0.002       0.290       1.309
    x1            -2.9778      0.964     -3.088      0.002      -4.868      -1.088
    x2            -1.4487      0.381     -3.804      0.000      -2.195      -0.702
    x3            -1.4633      0.744     -1.968      0.049      -2.921      -0.006
    x4             0.3681      0.872      0.422      0.673      -1.342       2.078
    x5            -1.1352      0.900     -1.261      0.207      -2.900       0.630
    x6            -1.6511      1.240     -1.332      0.183      -4.081       0.779
    x7            -0.5368      0.506     -1.060      0.289      -1.530       0.456
    x8             0.4164      0.765      0.544      0.586      -1.083       1.916
    x9             0.3544      1.734      0.204      0.838      -3.044       3.753
    x10           -1.0729      1.360     -0.789      0.430      -3.739       1.593
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.17 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

```python
res = pd.DataFrame({"causal_mean": dcfresult.params[:data_dim+1], \
                  "causal_std": dcfresult.bse[:data_dim+1], \
                  "causal_025": dcfresult.conf_int()[:data_dim+1,0], \
                  "causal_975": dcfresult.conf_int()[:data_dim+1,1], \
                   "causal_pval": dcfresult.pvalues[:data_dim+1]})
res["causal_sig"] = (res["causal_pval"] < 0.05)
res = res.T
res.columns = np.concatenate([["intercept"], np.array(dfX.columns)])
res = res.T
```

```python
res
```

      causal_mean
      causal_std
      causal_025
      causal_975
      causal_pval
      causal_sig

      intercept
      0.799657
      0.260116
      0.289839
      1.30947
      0.00211043
      True

      mean radius
      -2.97783
      0.964373
      -4.86796
      -1.08769
      0.0020162
      True

      mean texture
      -1.44867
      0.380799
      -2.19502
      -0.702313
      0.000142218
      True

      mean smoothness
      -1.46326
      0.743651
      -2.92079
      -0.00573425
      0.0491055
      True

      mean compactness
      0.368069
      0.87232
      -1.34165
      2.07778
      0.673067
      False

      mean concavity
      -1.13524
      0.90044
      -2.90007
      0.629588
      0.207394
      False

      mean concave points
      -1.65108
      1.23964
      -4.08072
      0.778573
      0.182893
      False

      mean symmetry
      -0.536826
      0.506479
      -1.52951
      0.455855
      0.289182
      False

      mean fractal dimension
      0.416358
      0.765082
      -1.08318
      1.91589
      0.586304
      False

We check the predictions to see if the logistic outcome model is a good outcome model.

```python
# make predictions with the causal model
dcfX_test = X_test
dcfy_predprob = dcfresult.predict(sm.add_constant(dcfX_test))
dcfy_pred = (dcfy_predprob > 0.5)
print(classification_report(y_test, dcfy_pred))
```

                  precision    recall  f1-score   support

               0       0.96      0.91      0.93        47
               1       0.94      0.97      0.96        67

        accuracy                           0.95       114
       macro avg       0.95      0.94      0.95       114
    weighted avg       0.95      0.95      0.95       114

# We are done!

We have computed the average causal effect of raising the causes by one unit (see the "causal mean" column above).

# Is the deconfounder worth the effort?

We finally compare the **causal** estimation (with the deconfounder) with the **noncausal** estimation (with vanilla regression).

## The classical logistic regression! Note it is noncausal :-(

```python
# regress the outcome against the causes only (no substitute confounders)
nodcfX_train = sm.add_constant(X_train[:,:X.shape[1]])
nodcflogit_model = sm.Logit(y_train, nodcfX_train)
nodcfresult = nodcflogit_model.fit_regularized(maxiter=5000)
print(nodcfresult.summary())
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13451539390572878
                Iterations: 71
                Function evaluations: 71
                Gradient evaluations: 71
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      446
    Method:                           MLE   Df Model:                            8
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7946
    Time:                        23:28:08   Log-Likelihood:                -61.205
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 3.283e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7632      0.255      2.998      0.003       0.264       1.262
    x1            -3.2642      0.904     -3.611      0.000      -5.036      -1.492
    x2            -1.7189      0.307     -5.602      0.000      -2.320      -1.117
    x3            -1.2257      0.498     -2.462      0.014      -2.202      -0.250
    x4             0.3115      0.741      0.420      0.674      -1.142       1.765
    x5            -1.1317      0.717     -1.578      0.115      -2.538       0.274
    x6            -2.0079      1.172     -1.714      0.087      -4.304       0.289
    x7            -0.5413      0.324     -1.673      0.094      -1.176       0.093
    x8             0.5775      0.677      0.853      0.394      -0.750       1.905
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

```python
res["noncausal_mean"] = np.array(nodcfresult.params)
res["noncausal_std"] = np.array(nodcfresult.bse)
res["noncausal_025"] = np.array(nodcfresult.conf_int()[:,0])
res["noncausal_975"] = np.array(nodcfresult.conf_int()[:,1])
res["noncausal_pval"] = np.array(nodcfresult.pvalues)
res["noncausal_sig"] = (res["noncausal_pval"] < 0.05)
```

```python
res["diff"] = res["causal_mean"] - res["noncausal_mean"]
res["pval_diff"] = res["causal_pval"] - res["noncausal_pval"]
```

```python
nodcfX_test = sm.add_constant(X_test[:,:X.shape[1]])
nodcfy_predprob = nodcfresult.predict(nodcfX_test)
nodcfy_pred = (nodcfy_predprob > 0.5)
```

**Causal models do not hurt predictions here!**

```python
dcflogit_roc_auc = roc_auc_score(y_test, dcfy_pred)
dcffpr, dcftpr, dcfthresholds = roc_curve(y_test, dcfy_predprob)
nodcflogit_roc_auc = roc_auc_score(y_test, nodcfy_pred)
nodcffpr, nodcftpr, nodcfthresholds = roc_curve(y_test, nodcfy_predprob)
plt.figure()
plt.plot(nodcffpr, nodcftpr, label='Noncausal Logistic Regression (area = %0.9f)' % nodcflogit_roc_auc)
plt.plot(dcffpr, dcftpr, label='Causal Logistic Regression (area = %0.9f)' % dcflogit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()
```

**But causal models do change the regression coefficients and which features are significant.**

*   The mean smoothness is a feature **significantly correlated** with the cancer diagnosis.

*   But it does **not significantly** **causally affect** the cancer diagnosis.

*   The effect of all features are **over-estimated** with the noncausal model except the "mean compactness".

```python
res.sort_values("pval_diff", ascending=True)[["pval_diff", "causal_pval", "noncausal_pval", "causal_sig", "noncausal_sig", "causal_mean", "noncausal_mean"]]
```

      pval_diff
      causal_pval
      noncausal_pval
      causal_sig
      noncausal_sig
      causal_mean
      noncausal_mean

      mean compactness
      -0.00130945
      0.673067
      6.743764e-01
      False
      False
      0.368069
      0.311494

      intercept
      -0.00060616
      0.00211043
      2.716595e-03
      True
      True
      0.799657
      0.763172

      mean texture
      0.000142197
      0.000142218
      2.123746e-08
      True
      True
      -1.44867
      -1.718860

      mean radius
      0.00171102
      0.0020162
      3.051830e-04
      True
      True
      -2.97783
      -3.264181

      mean smoothness
      0.0352701
      0.0491055
      1.383540e-02
      True
      True
      -1.46326
      -1.225668

      mean concavity
      0.0927253
      0.207394
      1.146687e-01
      False
      False
      -1.13524
      -1.131677

      mean concave points
      0.0963094
      0.182893
      8.658376e-02
      False
      False
      -1.65108
      -2.007921

      mean fractal dimension
      0.192521
      0.586304
      3.937831e-01
      False
      False
      0.416358
      0.577489

      mean symmetry
      0.194775
      0.289182
      9.440683e-02
      False
      False
      -0.536826
      -0.541292

*   We include causes into the regression **one-by-one**.
*   The deconfounder coefficients **does not** flip signs.
*   But classical logistic regression coefficients **does** flip signs.
*   This suggests that **the deconfounder is causal**.
*   It is because **causal** coefficients **do not change** as we include more variables into the system; causal estimation already controls for confounders so that it is causal.
*   However, **correlation** coefficients **can change** as we include more variables into the system; if the added variable is a confounder, than the regression coefficients change to account for the confounding effects.

```python
# The deconfounder with causes added one-by-one
# The first i coefficient is the causal coefficient of the first i causes.
# i = 1, ..., 8.
for i in range(X.shape[1]):
    print(i, "causes included")
    # augment the regressors to be both the assigned causes X and the substitute confounder Z
    X_aug = np.column_stack([X[:,:i], Z_hat])
    # holdout some data from prediction later
    X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
    dcfX_train = sm.add_constant(X_train)
    dcflogit_model = sm.Logit(y_train, dcfX_train)
    dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
    print(dcfresult.summary())
```

    0 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.18480230713099138
                Iterations: 22
                Function evaluations: 22
                Gradient evaluations: 22
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      452
    Method:                           MLE   Df Model:                            2
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7178
    Time:                        23:28:09   Log-Likelihood:                -84.085
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.267e-93
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9124      0.206      4.436      0.000       0.509       1.315
    x1            -0.7687      0.211     -3.643      0.000      -1.182      -0.355
    x2            -5.0965      0.543     -9.394      0.000      -6.160      -4.033
    ==============================================================================
    1 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.1627111817680479
                Iterations: 30
                Function evaluations: 30
                Gradient evaluations: 30
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      451
    Method:                           MLE   Df Model:                            3
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7516
    Time:                        23:28:09   Log-Likelihood:                -74.034
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 9.246e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7397      0.217      3.410      0.001       0.315       1.165
    x1            -2.1235      0.513     -4.143      0.000      -3.128      -1.119
    x2            -1.2129      0.253     -4.800      0.000      -1.708      -0.718
    x3            -3.8954      0.621     -6.276      0.000      -5.112      -2.679
    ==============================================================================
    2 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.15261356019021724
                Iterations: 37
                Function evaluations: 37
                Gradient evaluations: 37
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      450
    Method:                           MLE   Df Model:                            4
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7670
    Time:                        23:28:09   Log-Likelihood:                -69.439
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.268e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7502      0.227      3.309      0.001       0.306       1.195
    x1            -3.3034      0.701     -4.710      0.000      -4.678      -1.929
    x2            -1.0262      0.344     -2.985      0.003      -1.700      -0.352
    x3            -1.7916      0.355     -5.040      0.000      -2.488      -1.095
    x4            -2.4318      0.785     -3.097      0.002      -3.971      -0.893
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.10 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    3 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.14196497629747212
                Iterations: 44
                Function evaluations: 44
                Gradient evaluations: 44
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      449
    Method:                           MLE   Df Model:                            5
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7832
    Time:                        23:28:09   Log-Likelihood:                -64.594
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.173e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9137      0.247      3.700      0.000       0.430       1.398
    x1            -3.4320      0.752     -4.564      0.000      -4.906      -1.958
    x2            -1.1337      0.363     -3.119      0.002      -1.846      -0.421
    x3            -1.3748      0.462     -2.976      0.003      -2.280      -0.469
    x4            -0.7584      0.495     -1.533      0.125      -1.728       0.211
    x5            -2.6149      0.804     -3.251      0.001      -4.191      -1.039
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    4 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.1402052748348448
                Iterations: 57
                Function evaluations: 57
                Gradient evaluations: 57
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      448
    Method:                           MLE   Df Model:                            6
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7859
    Time:                        23:28:09   Log-Likelihood:                -63.793
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 5.396e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8689      0.248      3.504      0.000       0.383       1.355
    x1            -3.4797      0.755     -4.608      0.000      -4.960      -2.000
    x2            -1.1444      0.363     -3.153      0.002      -1.856      -0.433
    x3            -1.1184      0.466     -2.399      0.016      -2.032      -0.205
    x4             0.9297      0.722      1.288      0.198      -0.485       2.345
    x5            -1.6538      0.827     -2.000      0.045      -3.274      -0.033
    x6            -3.1811      0.932     -3.413      0.001      -5.008      -1.354
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    5 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13737900142942064
                Iterations: 60
                Function evaluations: 60
                Gradient evaluations: 60
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      447
    Method:                           MLE   Df Model:                            7
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7902
    Time:                        23:28:09   Log-Likelihood:                -62.507
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.395e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8515      0.251      3.393      0.001       0.360       1.343
    x1            -3.7237      0.770     -4.836      0.000      -5.233      -2.215
    x2            -1.2666      0.367     -3.451      0.001      -1.986      -0.547
    x3            -1.7016      0.617     -2.756      0.006      -2.912      -0.492
    x4             0.8248      0.694      1.189      0.235      -0.535       2.185
    x5            -1.1228      0.704     -1.594      0.111      -2.503       0.257
    x6            -0.5378      1.062     -0.506      0.613      -2.620       1.544
    x7            -2.2185      1.087     -2.041      0.041      -4.349      -0.088
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.18 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    6 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13516533343345713
                Iterations: 71
                Function evaluations: 71
                Gradient evaluations: 71
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      446
    Method:                           MLE   Df Model:                            8
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7936
    Time:                        23:28:09   Log-Likelihood:                -61.500
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 4.396e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7814      0.256      3.049      0.002       0.279       1.284
    x1            -3.1186      0.854     -3.652      0.000      -4.792      -1.445
    x2            -1.3462      0.367     -3.664      0.000      -2.066      -0.626
    x3            -1.1913      0.691     -1.724      0.085      -2.546       0.163
    x4             1.0127      0.716      1.415      0.157      -0.390       2.415
    x5            -0.5867      0.792     -0.741      0.459      -2.139       0.966
    x6            -1.6608      1.187     -1.399      0.162      -3.987       0.665
    x7            -0.6526      1.062     -0.615      0.539      -2.734       1.428
    x8            -1.8720      1.099     -1.703      0.089      -4.026       0.282
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    7 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.1331994800829468
                Iterations: 77
                Function evaluations: 77
                Gradient evaluations: 77
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      445
    Method:                           MLE   Df Model:                            9
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7966
    Time:                        23:28:09   Log-Likelihood:                -60.606
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.448e-96
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7813      0.258      3.033      0.002       0.276       1.286
    x1            -3.2282      0.861     -3.748      0.000      -4.916      -1.540
    x2            -1.4509      0.382     -3.798      0.000      -2.200      -0.702
    x3            -1.5318      0.738     -2.075      0.038      -2.978      -0.085
    x4             0.5140      0.829      0.620      0.535      -1.111       2.139
    x5            -1.0940      0.894     -1.224      0.221      -2.846       0.658
    x6            -1.8328      1.201     -1.527      0.127      -4.186       0.520
    x7            -0.6254      0.483     -1.294      0.196      -1.572       0.322
    x8             0.7810      1.558      0.501      0.616      -2.273       3.835
    x9            -0.8963      1.326     -0.676      0.499      -3.496       1.703
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.17 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

```python
# Logistic regression with causes added one-by-one
# The first i coefficient is the causal coefficient of the first i causes.
# i = 1, ..., 8.
for i in range(X.shape[1]):
    print(i, "causes included")
    # augment the regressors to be both the assigned causes X and the substitute confounder Z
    X_aug = np.column_stack([X[:,:i]])
    # holdout some data from prediction later
    X_train, X_test, y_train, y_test = train_test_split(X_aug, dfy, test_size=0.2, random_state=0)
    dcfX_train = sm.add_constant(X_train)
    dcflogit_model = sm.Logit(y_train, dcfX_train)
    dcfresult = dcflogit_model.fit_regularized(maxiter=5000)
    print(dcfresult.summary())
```

    0 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.6549205599225008
                Iterations: 6
                Function evaluations: 6
                Gradient evaluations: 6
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      454
    Method:                           MLE   Df Model:                            0
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:               7.357e-13
    Time:                        23:28:09   Log-Likelihood:                -297.99
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                       nan
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.5639      0.098      5.783      0.000       0.373       0.755
    ==============================================================================
    1 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.2953829271568602
                Iterations: 15
                Function evaluations: 15
                Gradient evaluations: 15
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      453
    Method:                           MLE   Df Model:                            1
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.5490
    Time:                        23:28:09   Log-Likelihood:                -134.40
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 3.955e-73
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6566      0.155      4.225      0.000       0.352       0.961
    x1            -3.5061      0.353     -9.934      0.000      -4.198      -2.814
    ==============================================================================
    2 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.2592096971350075
                Iterations: 20
                Function evaluations: 20
                Gradient evaluations: 20
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      452
    Method:                           MLE   Df Model:                            2
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.6042
    Time:                        23:28:09   Log-Likelihood:                -117.94
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 6.397e-79
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6744      0.168      4.018      0.000       0.345       1.003
    x1            -3.6479      0.390     -9.347      0.000      -4.413      -2.883
    x2            -0.9815      0.181     -5.424      0.000      -1.336      -0.627
    ==============================================================================
    3 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.162465976738943
                Iterations: 28
                Function evaluations: 29
                Gradient evaluations: 28
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      451
    Method:                           MLE   Df Model:                            3
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7519
    Time:                        23:28:09   Log-Likelihood:                -73.922
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 8.272e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9963      0.231      4.306      0.000       0.543       1.450
    x1            -4.9102      0.601     -8.170      0.000      -6.088      -3.732
    x2            -1.7394      0.284     -6.115      0.000      -2.297      -1.182
    x3            -2.2080      0.336     -6.578      0.000      -2.866      -1.550
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.12 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    4 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.15485253636195323
                Iterations: 35
                Function evaluations: 35
                Gradient evaluations: 35
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      450
    Method:                           MLE   Df Model:                            4
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7636
    Time:                        23:28:10   Log-Likelihood:                -70.458
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 3.495e-97
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.9646      0.233      4.146      0.000       0.509       1.421
    x1            -4.6125      0.614     -7.509      0.000      -5.816      -3.409
    x2            -1.6563      0.295     -5.610      0.000      -2.235      -1.078
    x3            -1.7342      0.372     -4.667      0.000      -2.463      -1.006
    x4            -0.9279      0.353     -2.631      0.009      -1.619      -0.237
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.13 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    5 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.14242454080229494
                Iterations: 44
                Function evaluations: 45
                Gradient evaluations: 44
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      449
    Method:                           MLE   Df Model:                            5
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7825
    Time:                        23:28:10   Log-Likelihood:                -64.803
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 1.444e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.8234      0.242      3.396      0.001       0.348       1.299
    x1            -4.6125      0.636     -7.252      0.000      -5.859      -3.366
    x2            -1.7131      0.303     -5.648      0.000      -2.308      -1.119
    x3            -1.9515      0.395     -4.941      0.000      -2.726      -1.177
    x4             0.4077      0.512      0.797      0.425      -0.595       1.410
    x5            -1.7514      0.464     -3.777      0.000      -2.660      -0.843
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.16 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    6 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.13837349335376822
                Iterations: 60
                Function evaluations: 60
                Gradient evaluations: 60
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      448
    Method:                           MLE   Df Model:                            6
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7887
    Time:                        23:28:10   Log-Likelihood:                -62.960
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 2.361e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7376      0.248      2.975      0.003       0.252       1.223
    x1            -3.5398      0.807     -4.384      0.000      -5.122      -1.957
    x2            -1.6821      0.302     -5.563      0.000      -2.275      -1.089
    x3            -1.3522      0.491     -2.756      0.006      -2.314      -0.391
    x4             0.6207      0.527      1.177      0.239      -0.413       1.655
    x5            -1.0073      0.623     -1.617      0.106      -2.228       0.214
    x6            -2.1317      1.138     -1.873      0.061      -4.362       0.098
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.14 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.
    7 causes included
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.135339334890979
                Iterations: 62
                Function evaluations: 62
                Gradient evaluations: 62
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  455
    Model:                          Logit   Df Residuals:                      447
    Method:                           MLE   Df Model:                            7
    Date:                Fri, 03 Apr 2020   Pseudo R-squ.:                  0.7933
    Time:                        23:28:10   Log-Likelihood:                -61.579
    converged:                       True   LL-Null:                       -297.99
    Covariance Type:            nonrobust   LLR p-value:                 5.570e-98
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.7287      0.250      2.916      0.004       0.239       1.219
    x1            -3.6556      0.796     -4.592      0.000      -5.216      -2.095
    x2            -1.7323      0.307     -5.644      0.000      -2.334      -1.131
    x3            -1.1122      0.483     -2.304      0.021      -2.058      -0.166
    x4             0.7408      0.542      1.367      0.172      -0.321       1.803
    x5            -0.8364      0.614     -1.363      0.173      -2.039       0.366
    x6            -2.2682      1.139     -1.992      0.046      -4.500      -0.037
    x7            -0.5397      0.324     -1.666      0.096      -1.175       0.095
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.15 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

**We note that the causal coefficient of x4 is stable with the (causal) deconfounder but the correlation coefficent of x4 flips sign with the (noncausal) logistic regression.**

# Takeaways

*   The deconfounder is **not hard** to use.
*   We simply **fit** a factor model, **check** it, and **infer** causal effects with the substitute confounder.
*   Please **always check** the factor model.
*   The deconfounder **makes a difference**.
*   The deconfounder **deconfounds**.

# Acknowledgements

We thank Suresh Naidu for suggesting the adding-causes-one-by-one idea.