# Estimating text loss in Middle Dutch chivalric epics

This English-language, Python notebook accompanies the following publication:

> Mike Kestemont and Folgert Karsdorp, "Het Atlantis van de Middelnederlandse ridderepiek. Een schatting van het tekstverlies met methodes uit de ecodiversiteit". *Spiegel der letteren* (2020).

All figures and numbers were prepared using the code below. Future updates of the code and data will be managed in an open [Github repository](https://github.com/mikekestemont/chivalric_diversity). The code block below loads all (third-party) packages and modules necessary to run the module. These can be installed from the file `requirements.txt`:

    pip install -r requirements.txt

```python
from functools import partial
from itertools import product

import numpy as np
np.random.seed(12345)
from scipy.special import erfinv
import pandas as pd

import matplotlib.pyplot as plt
plt.style.use("tufte.mplstyle")
plt.rcParams["text.usetex"] = False
%matplotlib inline

import scipy.stats as stats
from scipy.special import gammaln
```

## Data

We load the data from the spreadsheet file `mnl.xlsx`:

```python
mnl = pd.read_excel('mnl.xlsx', header=None, names=('text', 'witness'))
mnl.head(10)
```

      text
      witness

      0
      Aiol (1)
      H1

      1
      Aiol (2)
      H2

      2
      Alexanders geesten
      München, Bayerische Staatsbibliotheek, Cod. ge...

      3
      Alexanders geesten
      H3

      4
      Alexanders geesten
      H4

      5
      Alexanders geesten
      H5

      6
      Alexanders geesten
      H6

      7
      Arturs doet
      Den Haag, KB, 129 A 10, fol. 201r-238r

      8
      Arturs doet
      Rijksarchief te Antwerpen, Sint-Catharinakapit...

      9
      Aubri de Borgengoen
      H7

We are only interested in the count data, i.e. the number of witnesses per text (the technical term is "abundance data").

```python
mnl.groupby('text').size().sort_values(ascending=False).head()
```

    text
    Historie van Troyen       16
    Roman van Limborch        10
    Madelgijs                 10
    Karel ende Elegast         7
    Parthonopeus van Bloys     6
    dtype: int64

The counts per text can be plotted as follows:

```python
fig, ax = plt.subplots(figsize=(10,18))
mnl.groupby('text').size().sort_values(ascending=True).plot.barh(ax=ax);
ax.set(xlabel='aantal handschriften', ylabel='',
       title='Distributie van de (ons bekende) ridderepische teksten over tekstgetuigen')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
plt.savefig('output/Fig1.jpeg', dpi=300, transparent=True)
```

Yet a different perspective is to list the size of the frequency bins that we can distinguish within the manuscript counts:

```python
types = mnl.groupby('text').size().sort_values(ascending=False).value_counts().sort_index()
types = types.to_frame(name='aantal teksten')
types['aantal handschriften'] = types.index
types.to_excel('output/Tab1.xlsx')
types
```

      aantal teksten
      aantal handschriften

      1
      44
      1

      2
      13
      2

      3
      6
      3

      4
      3
      4

      5
      3
      5

      6
      1
      6

      7
      1
      7

      10
      2
      10

      16
      1
      16

Finally, we define the auxiliary function `species_richness` to count the number of unique texts in the data (i.e. the number of non-zero counts):

```python
def species_richness(counts):
    return np.sum(counts > 0)

print('# unique texts:', species_richness(mnl.groupby('text').size()))
print('# witnesses:', len(mnl))
```

    # unique texts: 74
    # witnesses: 164

## Jackknife

The following function computes the first-order Jackknife estimate, on the basis of the abundance data in our data frame, as well as a confidence interval (.95 be default). This approach is detailed in the following paper:

> K. Burnham & W. Overton, "Robust Estimation of Population Size When Capture Probabilities Vary Among Animals". *Ecology* (1979), 927-936.

```python
def jackknife(data, conf_lvl=0.95):
    jack_stat = species_richness(data)

    x = np.array(sum([[i] * c for i, c in enumerate(data, 1)], []))
    index = np.arange(x.shape[0])

    vals = []
    for i in range(x.shape[0]):
        t = x[index != i]
        vals.append(species_richness(np.bincount(t)))

    mean_jack_stat = np.mean(vals)
    bias = (x.shape[0] - 1) * (mean_jack_stat - jack_stat)

    estimate = jack_stat - bias

    std_err = np.sqrt(
        (x.shape[0] - 1) *
        np.mean((mean_jack_stat - vals) *
                (mean_jack_stat - vals), axis=0)
    )

    z_score = np.sqrt(2.0) * erfinv(conf_lvl)
    conf_interval = estimate + z_score * np.array((-std_err, std_err))

    return estimate, std_err, conf_interval

results = jackknife(mnl.groupby('text').size())
print('jackknife-estimate (order=1):', results[0], results[-1])
```

    jackknife-estimate (order=1): 117.73170731707278 [106.64468284 128.8187318 ]

This implementation is verbose and uses an explicit `for`-loop, which iteratively leaves out observations and tracks the drops in diversity that follow from this operation. In the code blocks below we show that the same estimate can also be obtained in a fully analytical fashion. First we calculate the frequency counts for each unique text:

```python
num_per_text = mnl.groupby('text').size()
num_per_text
```

    text
    Aiol (1)                 1
    Aiol (2)                 1
    Alexanderroman           1
    Alexanders geesten       5
    Arturs doet              2
                            ..
    Van den bere Wisselau    1
    Walewein ende Keye       1
    Willem van Oringen       1
    Willem van Oringen II    1
    Wrake van Ragisel        3
    Length: 74, dtype: int64

Next, we store the species richness (the number of unique texts) in $t$:

```python
t = species_richness(num_per_text)
t
```

    74

Then we set $s$ to the number of texts that are only attested in a single witness:

```python
s = sum(num_per_text == 1)
s
```

    44

Only the $s$ texts that occur once will affect the species richness during the iterative Jackknife procedure. We can therefore predict that we will obtain the following deviations when applying the bootstrap:

```python
mu = (((t - s) * t) + (s * (t - 1))) / t
mu
```

    73.4054054054054

That means that we can calculate the bias as follows:

```python
bias = (t - 1) * (mu - t)
bias
```

    -43.405405405405546

To account for this bias, we can subtract it from the original species richness in the observed data:

```python
t - bias
```

    117.40540540540555

## Simple example

```python
counts = [5, 4, 3, 3, 1, 1, 1, 1, 1]
names = 'ABCDEFGHI'
data = zip(counts, names)
df = pd.DataFrame(zip(names, counts), columns=('naam', 'mss'))
df.to_excel('output/Tab2.xlsx')
df
```

      naam
      mss

      0
      A
      5

      1
      B
      4

      2
      C
      3

      3
      D
      3

      4
      E
      1

      5
      F
      1

      6
      G
      1

      7
      H
      1

      8
      I
      1

```python
print('total # of witnesses:', df['mss'].sum())
```

    total # of witnesses: 20

```python
species_richness(df['mss'])
```

    9

```python
jackknife(df['mss'])
```

    (13.75, 1.8874586088176875, array([10.0506491, 17.4493509]))

```python
data = np.array(df['mss'])
x = np.array(sum([[i]*c for i, c in enumerate(data, 1)], []))
tradition = [names[i - 1] for i in x]
print(tradition)
```

    ['A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'D', 'D', 'D', 'E', 'F', 'G', 'H', 'I']

```python
bootstrap = []
for i in range(len(tradition)):
    tradition_ = [tradition[j] for j in range(len(tradition)) if i != j]

    bootstrap.append((
        (i + 1), tradition[i], ''.join(tradition_),
        len(set(tradition_)), len(set(tradition_)) - len(set(tradition))))
```

```python
df = pd.DataFrame(bootstrap, columns=('iteration', 'leftout', 'imputed tradition', 'richness', 'error'))
df.to_excel('output/Tab3.xlsx')
df
```

      iteration
      leftout
      imputed tradition
      richness
      error

      0
      1
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      1
      2
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      2
      3
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      3
      4
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      4
      5
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      5
      6
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      6
      7
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      7
      8
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      8
      9
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      9
      10
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      10
      11
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      11
      12
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      12
      13
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      13
      14
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      14
      15
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      15
      16
      E
      AAAAABBBBCCCDDDFGHI
      8
      -1

      16
      17
      F
      AAAAABBBBCCCDDDEGHI
      8
      -1

      17
      18
      G
      AAAAABBBBCCCDDDEFHI
      8
      -1

      18
      19
      H
      AAAAABBBBCCCDDDEFGI
      8
      -1

      19
      20
      I
      AAAAABBBBCCCDDDEFGH
      8
      -1

```python
mean_estimate = np.mean(df['richness'])
print('Average estimate:', mean_estimate)
print('Bias:', mean_estimate - 9)
```

    Average estimate: 8.75
    Bias: -0.25

```python
bias = 19 * (mean_estimate - 9)
bias
```

    -4.75

```python
corrected = 9 - bias
corrected
```

    13.75

```python
conf_lvl = .95

std_err = np.sqrt(
    19 * np.mean((mean_estimate - df['richness']) *
                 (mean_estimate - df['richness']), axis=0))

z_score = np.sqrt(2.0) * erfinv(conf_lvl)
conf_interval = corrected + z_score * np.array((-std_err, std_err))
conf_interval
```

    array([10.0506491, 17.4493509])

## Chao1

In the paper we eventually opt for the more recent, non-parametric formula "Chao1", which is described in this paper:

> A. Chao & L. Jost, ‘Estimating diversity and entropy profiles via discovery rates of new species". *Methods in Ecology and Evolution* (2015), 873-882.

Because we have "doubletons" in our data, we use can the following formula, where:
- $\hat{f_0}$ is the (theoretical) number of non-observed species/texts;
- $f_1$ is the number of species/texts attested exactly once ("singletons");
- $f_2$ is the number of species/texts attested exactly twice ("doubletons");
- $n$ is the total number of individuals/manuscripts in the observed data.

\begin{equation}
\hat{f_0} = \frac{(n - 1)}{n} \frac{f_1^2}{2f_2}
\end{equation}

The code block below returns the full, theoretical species richness as etimated by Chao1, i.e. it adds the estimated $\hat{f_0}$ to the species richness that was observed in the sample:

```python
def chao_richness(x):
    x, n = x[x > 0], x.sum()
    t = x.shape[0]
    f1, f2 = (x == 1).sum(), (x == 2).sum()
    return t + (n - 1) / n * ((f1 ** 2 / 2 / f2) if f2 > 0 else (f1 * (f1 - 1) / 2))
```

If we apply this function to our data, we obtain an even higher (but arguably more realistic) estimate of the loss in textual diversity for this literature. Note, however, that this estimate is still a theoretical *minimum estimate*, since the original loss could still be higher.

```python
chao_richness(num_per_text)
```

    148.00750469043152

Instead of reporting just this number, we apply a bootstrapped procedure in which we sample from the material using a multinomial distribution (see the Appendix Chao and Jost, 2015) and apply Chao1 to the resulting samples. This procedure allows us to calculate a .95 confidence interval for this value.

```python
def bt_prob(x):
    x, n = x[x > 0], x.sum()
    f1, f2 = (x == 1).sum(), (x == 2).sum()
    C = 1 - f1 / n * (((n - 1) * f1 / ((n - 1) * f1 + 2 * f2)) if f2 > 0 else
                      ((n - 1) * (f1 - 1) / ((n - 1) * (f1 - 1) + 2)) if f1 > 0 else
                      0)
    W = (1 - C) / np.sum(x / n * (1 - x / n) ** n)
    p = x / n * (1 - W * (1 - x / n) ** n)
    f0 = np.ceil(((n - 1) / n * f1 ** 2 / (2 * f2)) if f2 > 0 else
                 ((n - 1) / n * f1 * (f1 - 1) / 2))
    p0 = (1 - C) / f0
    p = np.hstack((p, np.array([p0 for i in np.arange(f0)])))
    return p

def bootstrap(x, n_iter=1000, conf=.95):
    # define a multinomial probability distribution
    # for the bootstrap procedure to sample from:
    p, n = bt_prob(x), x.sum()
    data_bt = np.random.multinomial(n, p, n_iter)

    pro = np.array([chao_richness(row) for row in data_bt])

    pro_mean = pro.mean(0)

    lci_pro = -np.quantile(pro, (1 - conf) / 2, axis=0) + pro_mean
    uci_pro = np.quantile(pro, 1 - (1 - conf) / 2, axis=0) - pro_mean

    sd_pro = np.std(pro, axis=0)

    pro = pro_mean - pro
    return (lci_pro, uci_pro, sd_pro, pro)
```

```python
def chao_estimate(x, n_iter=1000, conf=0.95):
    pro = chao_richness(x)
    (lci_pro, uci_pro, sd_pro, bt_pro) = bootstrap(x, n_iter=n_iter, conf=conf)
    lci_pro, uci_pro = pro - lci_pro, pro + uci_pro
    bt_pro = pro - bt_pro
    return (lci_pro, uci_pro, bt_pro, pro)
```

The following block applies this bootstrapped procedure to obtain the final estimates:

```python
lci_pro, uci_pro, bt_pro, pro = chao_estimate(num_per_text, n_iter=10000)
print('pro:', pro)
print('lci_pro:', lci_pro)
print('uci_pro:', uci_pro)
```

    pro: 148.00750469043152
    lci_pro: 106.21863495939421
    uci_pro: 219.01578019221017

The array `bt_pro` contains the estimates that were collected during the bootstrap (1,000 iterations by default). Below, we plot the distribution of these numbers using a rainplot: [removing rain_alpha =.3 argument on pt.RainCloud() because it is showing as invalid]

```python
import ptitprince as pt

fig, ax = plt.subplots(figsize=(8, 6))

d = list([(x, 'bootstrap') for x in bt_pro])
bt = pd.DataFrame(d, columns=('bootstrap', 'type'))

pt.RainCloud(
    data=bt, x="type", y="bootstrap", ax=ax,
    orient="h", alpha=.8, bw=.2, rain_alpha=.3, palette="Greys"
)
ax.axvline(pro, c='black', ls='--')
ax.axvline(uci_pro, c='darkgrey', ls='--')
ax.axvline(lci_pro, c='darkgrey', ls='--')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.set_yticks([])
ax.set_ylabel('')
plt.savefig('output/Fig2.png', dpi=300, transparent=True)
```

The idea that there were at least 100 texts is not completely unlikely, but it is a very
conservative estimate, at the very bottom of the probability continuum. The estimate of ~148 manuscripts (or more) is much more plausible, which would mean that *at least half of
the chivalric texts have been lost*.  Just as 100 is an extremely optimistic
estimate, ~219 is the most pessimistic estimate: in that
case, only a third of the ever available chivalric epics would have been persisted through
time, which is quite a dramatic, but not entirely unrealistic figure.

## Species accumulation curve

In what preceded, we have investigated how many unique texts may have been lost, or, more positively, how many unique texts we may have not yet seen. In this concluding section, we investigate how many texts should be retrieved before we arrive at this diversity estimate. This new estimate provides us with information about the total population size, i.e. the total number of text witnesses. We follow Hsieh, Ma and Chao (2016) to compute this estimate using "Rarefaction Extrapolation". For details about this method, see:

> Hsieh, Ma and Chao (2016): iNEXT: an R package for rarefaction and extrapolation ofspecies diversity. *Methods in Ecology and Evolution*, 7, 1451–1456.

```python
def bootstrap_re(x, fn=chao_richness, n_iter=1000, conf=.95):
    # define a multinomial probability distribution
    # for the bootstrap procedure to sample from:
    p, n = bt_prob(x), x.sum()
    data_bt = np.random.multinomial(n, p, n_iter)

    Dq = fn(x)

    pro = np.array([fn(row) for row in data_bt])

    error = stats.norm.ppf(1 - (1 - conf) / 2) * np.std(pro, 0)
    lci_pro = Dq - error
    uci_pro = Dq + error

    sd_pro = np.std(pro, axis=0)

    return (lci_pro, uci_pro, sd_pro, Dq, )

def rarefaction_extrapolation(x, max_steps):
    x, n = x[x > 0], x.sum()
    def _sub(m):
        if m <= n:
            return np.sum(1 - np.array(
                [np.exp(gammaln(n - i + 1) + gammaln(n - m + 1) -
                        gammaln(n - i - m + 1) - gammaln(n + 1)) if i <= (n - m) else
                 0 for i in x]))
        else:
            S = (x > 0).sum()
            f1, f2 = (x == 1).sum(), (x == 2).sum()
            f0 = ((n - 1) / n * f1 * (f1 - 1) / 2) if f2 == 0 else ((n - 1) / n * f1**2 / 2 / f2)
            A = n * f0 / (n * f0 + f1)
            return S if f1 == 0 else (S + f0 * (1 - A**(m - n)))
    return np.array([_sub(mi) for mi in range(1, max_steps)])
```

```python
counts = np.bincount(mnl.groupby('text').size())[1:] # ignore zero
x = np.array(sum([[i] * c for i, c in enumerate(counts, 1)], []))
```

Here too we use a bootstrap method with 100 samples:

```python
max_steps = 1000
lci_pro, uci_pro, sd_pro, Dq = bootstrap_re(
    x,
    fn=partial(rarefaction_extrapolation, max_steps=max_steps),
    n_iter=100
)
```

```python
steps = np.arange(1, max_steps)
interpolated = np.arange(1, max_steps) < x.sum()

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(steps[interpolated], Dq[interpolated], color='C0')
ax.plot(x.sum(), Dq[x.sum() - 1], 'o')
ax.plot(steps[~interpolated], Dq[~interpolated], '--', color='C0')
ax.fill_between(steps, lci_pro, uci_pro, alpha=0.3)
ax.grid()
ax.set(xlabel='# handschriften', ylabel='# teksten', title='Species Accumulation Curve')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

plt.savefig('output/Fig3.png', dpi=300, transparent=True)
```

The graph clearly show that a significant part of the text witnesses has been lost (or has not yet been found). It is only at around 600 witnesses (and more), that the curve slowly flattens towards its asymptote.


# Run Scores II for a Volume carrier

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables
import shipflowmotionshelpers.shipflowmotionshelpers as helpers

```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 461 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.dpi       : 100        ## figure dots per inch')

```python
from pyscores2.indata import Indata
from pyscores2.runScores2 import Calculation
from pyscores2.output import OutputFile
from pyscores2 import TDPError
import pyscores2
from rolldecayestimators.ikeda import Ikeda, IkedaR
from rolldecayestimators.simplified_ikeda_class import SimplifiedIkeda
import subprocess
from rolldecayestimators import measure
from src import run_scores
```

```python
df_sections = pd.read_csv('../data/interim/volume_carrier_areas.csv', sep=';')
#volume_correction = 1.005
#df_sections['area']*=volume_correction
df_sections.head()
```

      no
      area
      x
      t
      b
      r_b

      0
      0
      3.152037
      3.555946
      0.32
      15.590334
      1.529306

      1
      1
      80.056100
      13.298716
      6.77
      40.829136
      15.811695

      2
      2
      186.996780
      23.041486
      7.50
      42.999309
      13.134742

      3
      3
      263.957911
      31.392432
      7.50
      43.000004
      8.633547

      4
      4
      310.254925
      41.135203
      7.50
      43.000004
      3.948538

```python

```

```python
df_rolldecay = database.load(rolldecay_table_name='rolldecay_quadratic_b', limit_score=0.99,
                             exclude_table_name='rolldecay_exclude')
```

```python
mask = df_rolldecay['loading_condition_id'] == 134
df = df_rolldecay.loc[mask]
```

```python
df_sections['loading_condition_id'] = 134
section_groups = df_sections.groupby(by='loading_condition_id')
```

```python
df['kg']
```

    run_id
    18405    19.09
    18407    19.09
    Name: kg, dtype: float64

```python
df.KXX
```

    run_id
    18405    15.19
    18407    15.19
    Name: KXX, dtype: float64

```python
df.KZZ
```

    run_id
    18405    57.57
    18407    57.57
    Name: KZZ, dtype: float64

```python
df.gm
```

    run_id
    18405    8.02
    18407    8.02
    Name: gm, dtype: float64

```python
results = pd.DataFrame()
fi_as = np.deg2rad([1,10])
for run_name, row in df.iterrows():

    loading_condition_id=row['loading_condition_id']
    sections = section_groups.get_group(loading_condition_id)

    sections=run_scores.remove_duplicate_sections(sections)
    sections=run_scores.too_small_sections(sections)

    try:
        indata,sections_ = run_scores.define_indata(row, sections, displacement_correction=1.01, lcb_correction=0.99)
    except run_scores.DraughtError as e:
        print('Draught is too large for sections, this loading condition is skipped.')
        continue

    save_name='%s.in' % row.loading_condition_id
    save_path=os.path.join('scores',save_name)
    indata.save(save_path)

    calculation = Calculation(outDataDirectory='scores/result')

    # Run scoresII:
    try:
        calculation.run(indata=indata, b_div_t_max=None, timeout=1.0)
    except TDPError:
        print('Dissregarding the TDPError')
        continue
    except pyscores2.LcgError as e:
        print('Disregarded')
        print(e)
        continue
    except subprocess.TimeoutExpired:
        print('Disregarded, scoresII got stuck...')
        continue

    output_file = OutputFile(filePath=calculation.outDataPath)
```

    Running Scores2 for loading_condition_id_134
    Disregarded, scoresII got stuck...
    Running Scores2 for loading_condition_id_134
    Disregarded, scoresII got stuck...

```python
row.Volume
```

    49410.0

```python
indata.speedMax
```

    10.0

```python
indata.waveFrequenciesMin
```

    0.23564899342081083

```python
indata.waveFrequenciesIncrement
```

    0.017673674506560814

```python
row.scale_factor
```

    42.0

```python
row.omega0
```

    3.05436004440219


# Customer Segmentation: Estimate Individualized Responses to Incentives

Nowadays, business decision makers rely on estimating the causal effect of interventions to answer what-if questions about shifts in strategy, such as promoting specific product with discount, adding new features to a website or increasing investment from a sales team. However, rather than learning whether to take action for a specific intervention for all users, people are increasingly interested in understanding the different responses from different users to the two alternatives. Identifying the characteristics of users having the strongest response for the intervention could help make rules to segment the future users into different groups. This can help optimize the policy to use the least resources and get the most profit.

In this case study, we will use a personalized pricing example to explain how the [EconML](https://aka.ms/econml) library could fit into this problem and provide robust and reliable causal solutions.

### Summary

1. [Background](#background)
2. [Data](#data)
3. [Get Causal Effects with EconML](#estimate)
4. [Understand Treatment Effects with EconML](#interpret)
5. [Make Policy Decisions with EconML](#policy)
6. [Conclusions](#conclusion)

# Background

The global online media market is growing fast over the years. Media companies are always interested in attracting more users into the market and encouraging them to buy more songs or become members. In this example, we'll consider a scenario where one experiment a media company is running is to give small discount (10%, 20% or 0) to their current users based on their income level in order to boost the likelihood of their purchase. The goal is to understand the **heterogeneous price elasticity of demand** for people with different income level, learning which users would respond most strongly to a small discount. Furthermore, their end goal is to make sure that despite decreasing the price for some consumers, the demand is raised enough to boost the overall revenue.

EconML’s `DML` based estimators can be used to take the discount variation in existing data, along with a rich set of user features, to estimate heterogeneous price sensitivities that vary with multiple customer features. Then, the `SingleTreeCateInterpreter` provides a presentation-ready summary of the key features that explain the biggest differences in responsiveness to a discount, and the `SingleTreePolicyInterpreter` recommends a policy on who should receive a discount in order to increase revenue (not only demand), which could help the company to set an optimal price for those users in the future.

```python
# Some imports to get us started
# Utilities
import os
import urllib.request
import numpy as np
import pandas as pd

# Generic ML imports
from sklearn.preprocessing import PolynomialFeatures
from sklearn.ensemble import GradientBoostingRegressor

# EconML imports
from econml.dml import LinearDML, CausalForestDML
from econml.cate_interpreter import SingleTreeCateInterpreter, SingleTreePolicyInterpreter

import matplotlib.pyplot as plt

%matplotlib inline
```

# Data

The dataset* has ~10,000 observations and includes 9 continuous and categorical variables that represent user's characteristics and online behaviour history such as age, log income, previous purchase, previous online time per week, etc.

We define the following variables:

Feature Name|Type|Details
:--- |:---|:---
**account_age** |W| user's account age
**age** |W|user's age
**avg_hours** |W| the average hours user was online per week in the past
**days_visited** |W| the average number of days user visited the website per week in the past
**friend_count** |W| number of friends user connected in the account
**has_membership** |W| whether the user had membership
**is_US** |W| whether the user accesses the website from the US
**songs_purchased** |W| the average songs user purchased per week in the past
**income** |X| user's income
**price** |T| the price user was exposed during the discount season (baseline price * small discount)
**demand** |Y| songs user purchased during the discount season

**To protect the privacy of the company, we use the simulated data as an example here. The data is synthetically generated and the feature distributions don't correspond to real distributions. However, the feature names have preserved their names and meaning.*

The treatment and outcome are generated using the following functions:
$$
T =
\begin{cases}
  1 & \text{with } p=0.2,  \\
  0.9 & \text{with }p=0.3, & \text{if income}<1 \\
  0.8 & \text{with }p=0.5, \\
  \\
    1 & \text{with }p=0.7, \\
  0.9 & \text{with }p=0.2, & \text{if income}\ge1 \\
  0.8 & \text{with }p=0.1, \\
\end{cases}
$$

\begin{align}
\gamma(X) & = -3 - 14 \cdot \{\text{income}<1\} \\
\beta(X,W) & = 20 + 0.5 \cdot \text{avg_hours} + 5 \cdot \{\text{days_visited}>4\} \\
Y &= \gamma(X) \cdot T + \beta(X,W)
\end{align}

```python
# Import the sample pricing data
file_url = "https://msalicedatapublic.blob.core.windows.net/datasets/Pricing/pricing_sample.csv"
train_data = pd.read_csv(file_url)
```

```python
# Data sample
train_data.head()
```

      account_age
      age
      avg_hours
      days_visited
      friends_count
      has_membership
      is_US
      songs_purchased
      income
      price
      demand

      0
      3
      53
      1.834234
      2
      8
      1
      1
      4.903237
      0.960863
      1.0
      3.917117

      1
      5
      54
      7.171411
      7
      9
      0
      1
      3.330161
      0.732487
      1.0
      11.585706

      2
      3
      33
      5.351920
      6
      9
      0
      1
      3.036203
      1.130937
      1.0
      24.675960

      3
      2
      34
      6.723551
      0
      8
      0
      1
      7.911926
      0.929197
      1.0
      6.361776

      4
      4
      30
      2.448247
      5
      8
      1
      0
      7.148967
      0.533527
      0.8
      12.624123

```python
# Define estimator inputs
Y = train_data["demand"]  # outcome of interest
T = train_data["price"]  # intervention, or treatment
X = train_data[["income"]]  # features
W = train_data.drop(columns=["demand", "price", "income"])  # confounders
```

```python
# Get test data
X_test = np.linspace(0, 5, 100).reshape(-1, 1)
X_test_data = pd.DataFrame(X_test, columns=["income"])
```

# Get Causal Effects with EconML
To learn the price elasticity on demand as a function of income, we fit the model as follows:

\begin{align}
log(Y) & = \theta(X) \cdot log(T) + f(X,W) + \epsilon \\
log(T) & = g(X,W) + \eta
\end{align}

where $\epsilon, \eta$ are uncorrelated error terms.

The models we fit here aren't an exact match for the data generation function above, but if they are a good approximation, they will allow us to create a good discount policy.  Although the model is misspecified, we hope to see that our `DML` based estimators can still capture the right trend of $\theta(X)$ and that the recommended policy beats other baseline policies (such as always giving a discount) on revenue.  Because of the mismatch between the data generating process and the model we're fitting, there isn't a single true $\theta(X)$ (the true elasticity varies with not only X but also T and W), but given how we generate the data above, we can still calculate the range of true $\theta(X)$ to compare against.

```python
# Define underlying treatment effect function given DGP
def gamma_fn(X):
    return -3 - 14 * (X["income"] < 1)

def beta_fn(X):
    return 20 + 0.5 * (X["avg_hours"]) + 5 * (X["days_visited"] > 4)

def demand_fn(data, T):
    Y = gamma_fn(data) * T + beta_fn(data)
    return Y

def true_te(x, n, stats):
    if x < 1:
        subdata = train_data[train_data["income"] < 1].sample(n=n, replace=True)
    else:
        subdata = train_data[train_data["income"] >= 1].sample(n=n, replace=True)
    te_array = subdata["price"] * gamma_fn(subdata) / (subdata["demand"])
    if stats == "mean":
        return np.mean(te_array)
    elif stats == "median":
        return np.median(te_array)
    elif isinstance(stats, int):
        return np.percentile(te_array, stats)
```

```python
# Get the estimate and range of true treatment effect
truth_te_estimate = np.apply_along_axis(true_te, 1, X_test, 1000, "mean")  # estimate
truth_te_upper = np.apply_along_axis(true_te, 1, X_test, 1000, 95)  # upper level
truth_te_lower = np.apply_along_axis(true_te, 1, X_test, 1000, 5)  # lower level
```

## Parametric heterogeneity
First of all, we can try to learn a **linear projection of the treatment effect** assuming a polynomial form of $\theta(X)$. We use the `LinearDML` estimator. Since we don't have any priors on these models, we use a generic gradient boosting tree estimators to learn the expected price and demand from the data.

```python
# Get log_T and log_Y
log_T = np.log(T)
log_Y = np.log(Y)
```

```python
# Train EconML model
est = LinearDML(
    model_y=GradientBoostingRegressor(),
    model_t=GradientBoostingRegressor(),
    featurizer=PolynomialFeatures(degree=2, include_bias=False),
)
est.fit(log_Y, log_T, X=X, W=W, inference="statsmodels")
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test)
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0],
    te_pred_interval[1],
    alpha=0.2,
    label="95% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

From the plot above, it's clear to see that the true treatment effect is a **nonlinear** function of income, with elasticity around -1.75 when income is smaller than 1 and a small negative value when income is larger than 1. The model fits a quadratic treatment effect, which is not a great fit. But it still captures the overall trend: the elasticity is negative and people are less sensitive to the price change if they have higher income.

```python
# Get the final coefficient and intercept summary
est.summary()
```

Coefficient Results

      	     point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  income	        2.427	      0.075	 32.189	    0.0	    2.28	     2.575

  income^2	     -0.437	      0.026	 -16.907	   0.0	   -0.487	   -0.386

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  cate_intercept	      -3.02	      0.046	 -65.081	   0.0	   -3.111	   -2.929

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

`LinearDML` estimator can also return the summary of the coefficients and intercept for the final model, including point estimates, p-values and confidence intervals. From the table above, we notice that $income$ has positive effect and ${income}^2$ has negative effect, and both of them are statistically significant.

## Nonparametric Heterogeneity
Since we already know the true treatment effect function is nonlinear, let us fit another model using `CausalForestDML`, which assumes a fully **nonparametric estimation of the treatment effect**.

```python
# Train EconML model
est = CausalForestDML(
    model_y=GradientBoostingRegressor(), model_t=GradientBoostingRegressor()
)
est.fit(log_Y, log_T, X=X, W=W, inference="blb")
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test)
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0],
    te_pred_interval[1],
    alpha=0.2,
    label="95% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

We notice that this model fits much better than the `LinearDML`, the 95% confidence interval correctly covers the true treatment effect estimate and captures the variation when income is around 1. Overall, the model shows that people with low income are much more sensitive to the price changes than higher income people.

# Understand Treatment Effects with EconML
EconML includes interpretability tools to better understand treatment effects. Treatment effects can be complex, but oftentimes we are interested in simple rules that can differentiate between users who respond positively, users who remain neutral and users who respond negatively to the proposed changes.

The EconML `SingleTreeCateInterpreter` provides interperetability by training a single decision tree on the treatment effects outputted by any of the EconML estimators. In the figure below we can see in dark red users respond strongly to the discount and the in white users respond lightly to the discount.

```python
intrp = SingleTreeCateInterpreter(include_model_uncertainty=True, max_depth=2, min_samples_leaf=10)
intrp.interpret(est, X_test)
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, fontsize=12)
```

# Make Policy Decision with EconML
We want to make policy decisions to maximum the **revenue** instead of the demand. In this scenario,

\begin{align}
Rev & = Y \cdot T \\
    & = \exp^{log(Y)} \cdot T\\
    & = \exp^{(\theta(X) \cdot log(T) + f(X,W) + \epsilon)} \cdot T \\
    & = \exp^{(f(X,W) + \epsilon)} \cdot T^{(\theta(X)+1)}
\end{align}

With the decrease of price, revenue will increase only if $\theta(X)+1<0$. Thus, we set `sample_treatment_cast=-1` here to learn **what kinds of customers we should give a small discount to maximum the revenue**.

The EconML library includes policy interpretability tools such as `SingleTreePolicyInterpreter` that take in a treatment cost and the treatment effects to learn simple rules about which customers to target profitably. In the figure below we can see the model recommends to give discount for people with income less than $0.985$ and give original price for the others.

```python
intrp = SingleTreePolicyInterpreter(risk_level=0.05, max_depth=2, min_samples_leaf=1, min_impurity_decrease=0.001)
intrp.interpret(est, X_test, sample_treatment_costs=-1)
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, treatment_names=["Discount", "No-Discount"], fontsize=12)
```

Now, let us compare our policy with other baseline policies! Our model says which customers to give a small discount to, and for this experiment, we will set a discount level of 10% for those users. Because the model is misspecified we would not expect good results with large discounts. Here, because we know the ground truth, we can evaluate the value of this policy.

```python
# define function to compute revenue
def revenue_fn(data, discount_level1, discount_level2, baseline_T, policy):
    policy_price = baseline_T * (1 - discount_level1) * policy + baseline_T * (1 - discount_level2) * (1 - policy)
    demand = demand_fn(data, policy_price)
    rev = demand * policy_price
    return rev
```

```python
policy_dic = {}
# our policy above
policy = intrp.treat(X)
policy_dic["Our Policy"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, policy))

## previous strategy
policy_dic["Previous Strategy"] = np.mean(train_data["price"] * train_data["demand"])

## give everyone discount
policy_dic["Give Everyone Discount"] = np.mean(revenue_fn(train_data, 0.1, 0, 1, np.ones(len(X))))

## don't give discount
policy_dic["Give No One Discount"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, np.ones(len(X))))

## follow our policy, but give -10% discount for the group doesn't recommend to give discount
policy_dic["Our Policy + Give Negative Discount for No-Discount Group"] = np.mean(revenue_fn(train_data, -0.1, 0.1, 1, policy))

## give everyone -10% discount
policy_dic["Give Everyone Negative Discount"] = np.mean(revenue_fn(train_data, -0.1, 0, 1, np.ones(len(X))))
```

```python
# get policy summary table
res = pd.DataFrame.from_dict(policy_dic, orient="index", columns=["Revenue"])
res["Rank"] = res["Revenue"].rank(ascending=False)
res
```

      Revenue
      Rank

      Our Policy
      14.686241
      2.0

      Previous Strategy
      14.349342
      4.0

      Give Everyone Discount
      13.774469
      6.0

      Give No One Discount
      14.294606
      5.0

      Our Policy + Give Negative Discount for No-Discount Group
      15.564411
      1.0

      Give Everyone Negative Discount
      14.612670
      3.0

**We beat the baseline policies!** Our policy gets the highest revenue except for the one raising the price for the No-Discount group. That means our currently baseline price is low, but the way we segment the user does help increase the revenue!

# Conclusions

In this notebook, we have demonstrated the power of using EconML to:

* Estimate the treatment effect correctly even the model is misspecified
* Interpret the resulting individual-level treatment effects
* Make the policy decision beats the previous and baseline policies

To learn more about what EconML can do for you, visit our [website](https://aka.ms/econml), our [GitHub page](https://github.com/microsoft/EconML) or our [documentation](https://econml.azurewebsites.net/).

notebooks/CustomerScenarios/Case Study - Customer Segmentation at An Online Media Company.ipynb


# Problem Set 3, Spring 2021, Villas-Boas

Due <u>Tuesday March 9, end of day Pacific Time</u>

Submit materials (Jupyter notebook with all code cells run) as one pdf on [Gradescope](https://www.gradescope.com/courses/226571).

# Exercise 1: The Value of Environmental Services

## Background

This exercise is based on the paper [Does Hazardous Waste Matter? Evidence from the Housing Market and the Superfund Program by Greenstone and Gallagher 2008](https://academic.oup.com/qje/article-abstract/123/3/951/1928203?redirectedFrom=fulltext). This
paper explores individuals’ willingness to pay (WTP) for environmental quality by observing the
impact of increasing environmental quality on housing prices. This method is known as the hedonic
valuation method, whereby the price of a good is determined by each of its attributes. In the case of a house, its value is determined by all of its physical characteristics (e.g., number of bedrooms) as well as the characteristics of the neighborhood in which it is located (including environmental quality).

For this paper, the change in environmental quality results from the cleanup of hazardous
waste sites. The study focuses on Superfund sites, areas designated by the US government as
contaminated by hazardous waste and that pose a hazard to environmental/human health. The
EPA placed certain Superfund sites on the National Priorities List (NPL), which meant that these
sites were legally required to undergo remediation. If individuals value environmental quality, then
housing prices should increase after nearby NPL sites are cleaned up. To determine the extent to
which individuals value the cleanup, we can compare housing close to a hazardous waste site that
was cleaned up to *comparable* homes near a similar waste site that was not cleaned up.

## Data

The data include observations on 447 census tracts that are within 2 miles of a hazardous waste site.$^1$  This includes census tracts where the site in question was on the NPL, and thus was legally required to be cleaned up, as well as those that were not on the NPL.

The shared dataset contains the following variables for each census tract in the year 2000.

|Variable Name	|	Description |
| :----------- | :----------------------- |
| _fips_   | Federal Information Processing Standards (FIPS) census tract identifier  |
| _npl_	|	binary indicator for whether the site in a given census tract was placed on the NPL|
| _lnmdvalhs_ | log median housing value $^2$ |
| *owner\_occupied*	| % of housing that is owner occupied |
| *pop\_den*	| Population density|
| *ba\_or\_better*	| % of the population that has a Bachelors degree or higher |
| *unemprt*	| Unemployment rate |
| *povrat*	| Poverty rate |
| *bedrms1-bedrms5*	| % of housing with the indicated number of bedrooms |
|  *bedrms\_3orless*  |  % of housing with 3 bedrooms or less |
|*blt0\_10yrs* |% of housing $< 10$ years old |

$^1$ : Census tracts are statistical subdivisions of a county that are defined by the Census Bureau to allow comparisons from census to census.

$^2$ : Median Housing Value, *before logging*, is recorded in USD.

## Tips

* Do **NOT** install packages on the server. All packages that you need to complete the assignment are already installedand can be loaded with the `library()` function. Trying to install packages will create conflicts and potentially require reloading a fresh copy of the notebook.
* When submitting the notebook, do **NOT** include output of the entire dataset. This is a large dataset, and printing out all rows will take multiple pages of output and make it much harder for the GSIs to find your answers.

* `xtable` and `stargazer` are two great packages for making tables. `xtable` is great for turning a dataframe into a table output, while `stargazer` is great for making a table of your regression results.$^3$ I recommend exploring these packages when you are making your summary statistics table and regression output table in this problem set. The first half of Coding Bootcamp Part 5 goes over these two packages and is accessible [here](https://r.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fds-modules%2FENVECON-118&urlpath=tree%2FENVECON-118%2FSpring2021-J%2FSections%2FCoding+Bootcamps).

* All packages that you need for this problem set are already installed and ready for use on Datahub. \textbf{Do not try to install packages on the Datahub server}; if you want to use new packages in Datahub to solve the problem set inside the Problem Set 3 notebook, let any of the GSI's know so they ask for those packages you want to use to be installed.

$^3$: `stargazer` is great for professional-looking LaTeX or postscript tables, but if you want to produce tables for your regression output in the notebook you will need to use `type = "html"` and then copy/paste the html output into a Markdown cell.

## Preamble

#### Use the below code cell to load all your packages (we will use `haven()` and `tidyverse()`).

```R
# Add your preamble code here
```

**1.**  Load the data. This problem set will focus on census tracts that have more than 20% of homes built within the last ten years; that is for $blt0\_ 10yrs > 0.2$.
So first open the original data (`Greenstone_Gallagher_PS3_2021.dta`) and create a subset of the data to be used in this problem set. Call this data frame in R  $my\_dataPset3$.  How many observations do you lose when you focus on this subset of the data?

 (Hint: use `filter(data, condition)`)

```R
# Add any code for part 1 here.
```

➡ Type your written answer for part 1 here.

**2.** Briefly describe your $my\_dataPset3$ data set. Since these data use the impact of the NPL to estimate WTP for environmental services, you

**(a)** Want to look at the number of census tracts in both the NPL group and the non-NPL group (and the number in the sample overall),

**(b)** Create a variable that is the median value of housing price and call it $housep$, which is the median housing price in dollars (in levels, not logs) and report the average in npl=1 and in npl=0 groups, and

**(c)** compare the means of the following characteristics (covariates)  of these census tracts across the two groups. By compare, we mean report them and discuss whether the sample averages are similar across groups, and why you might expect such a result (no need to test formally if means of these covariates are similar). The covariates you are to check are percent owner\_occupied, unemployment rate, and poverty rate.

(Hint: use `group_by()` and `summarise()` to obtain separate summary statistics according to the value of $npl$)

```R
# Add any code for part 2 here.
```

➡ Type your written answer for part 2 here.

**3.** We will now compare housing prices (in levels, not logs) across the two groups (group *npl* = 1 and
*npl* = 0) using the $my\_dataPset3$ sub dataset. Draw a histogram of the median housing price in
each group.

(Hint: see [Coding Bootcamp Part 4](https://r.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fds-modules%2FENVECON-118&urlpath=tree%2FENVECON-118%2FSpring2021-J%2FSections%2FCoding+Bootcamps) for how to do this using **ggplot2**. For base R use `hist(data$variable, main = "Title", xlab = "MedianHousingPrice", ylab = "Frequency")`)

```R
# Add any code for part 3 here.
```

**4.** Overlap both histograms into the same graph and comment on differences (be precise - and explain
why the differences intuitively make sense).

(Hint: see the Histograms section of [Coding Bootcamp Part 4](https://r.datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fds-modules%2FENVECON-118&urlpath=tree%2FENVECON-118%2FSpring2021-J%2FSections%2FCoding+Bootcamps))

```R
# Add any code for part 4 here.
```

➡ Type your written answer for part 4 here.

**Show work for all steps for Questions 5-7. You can use R to calculate the sample mean,
variance, and # obs, and perform arithmetic operations (or do it by hand). No credit will
be given if you use a canned condence interval/hypothesis test function in R. Credit will
be lost if you do not clearly show your steps.**

**5.** Compute an estimate for the mean of the variable housep for the NPL group (npl=1) in the
$my\_dataPset3$ data frame. Construct a 90% confidence interval for this mean. Give an interpretation of these results in a sentence.

(Hint: use `mean()` and `sd()` to get the necessary information to construct the CI)

```R
# Add any code for part 5 here.
```

➡ Type your written work for and answer to part 5 here.

**6.** Let $D$ be the difference in $housep$ between the NPL (npl=1) and non-NPL (npl=0) groups. State
an estimator $\hat D$
for $D$ and use the estimator to compute an estimate of $D$. Compute a standard
error for $\hat D$. Derive a 95% confidence interval for $D$ and interpret in one sentence.

```R
# Add any code for part 6 here.
```

➡ Type your written work for and answer to part 6 here.

**7.** Using the $my\_dataPset3$ data frame, test whether the average of the housing values ($housep$) for
the NPL group is statistically different at the 10% significance level ($\alpha = 0.1$) from average housing values in the non-NPL group (that is, in terms of the hypothesis, the null is equal, and the alternative is not equal). (Recall
the 5 step-procedure for hypothesis testing).

```R
# Add any code for part 7 here.
```

➡ Type your written work for and answer to part 7 here.

```R
# insert code here
```

**8.** Now we want to see how adding covariates on the right hand side of our equation affects the coefficient on the treatment indicator $npl$. We run the following regressions using the data $my\_dataPset3$
in R:

\begin{align}
housep&= \beta_0+\beta_1 npl+ u  & (1)\\
housep&= \beta_0+\beta_1 npl+ \beta_2  unemprt +u    & (2)\\
housep&= \beta_0+\beta_1 npl+ \beta_2 unemprt + \beta_3 owner\_occupied +u &(3)
\end{align}

```R
# Add any code for part 8 here.
```

**(a)** Interpret (SSS) the estimated coefficient, $\hat \beta_1$, that you obtain from estimating equation (1).

➡ Type your written answer to 8 (a) here.

**(b)** Looking both at $R^2$ and the evolution of $\hat \beta_1$ as we add variables from equation (1) to (2) to (3),

1. Comment on which variable matters in explaining the outcome, and which is likely correlated with the variable $npl$ (go through equation by equation).
2. What does this tell you about how sites were selected to end up on the National Priorities List (that is, about the correlation between $npl$ and those additional variables you added? (Hint: go through the OVB formula)

➡ Type your written answer to 8 (b) here.

**9.** If you estimate

\begin{align*}housep= \beta_0+\beta_1 npl+ u\end{align*}

 using the complete data set  (not the subset $my\_dataPset3$), what happens to the standard errors of the coefficient of $npl$? Explain briefly why that is.

➡ Type your written answer to 9 here.

## Exercise 2: Attitudes toward the COVID Vaccines

*Note:* this question does not require R. If you do use R, you must show all steps used to calculate the relevant formulas from lecture. No credit will be given if a canned routine is used. Credit will be lost if values are given and work is not shown.

The Institute of Global Health Innovation (Imperial College London) released a report about Global Attitudes towards a COVID-19 vaccine. The survey covers 15 countries and was implemented at different periods in time, from November 2020 to mid January of 2021. The total number of responses was around 13,500.

In one of the questions, respondents were asked whether they strongly agreed (scale of 5), agreed (a4), neither agree nor disagree (a 3), disagree (a 2) or strongly disagree (a 1) with the statement:$^4$

"To what extent do you agree or disagree that if a COVID-19 vaccine were made available to you this week, you would defnitely get it?"

$^4$: *Aggregate view of latest week available for each country - see page 13 for exact survey dates:* [Link to Nature Article](https://www.nature.com/articles/d41586-021-00368-6)

In row 1 below are the percentage and number of respondents that agree or strongly agree with the statement, over the whole sample for all countries. Below that first row, we also report results broken down for the United Kingdom (UK) for two periods during which the survey was conducted (Nov 05-15,2020; and then Jan 11-17, 2021) in that country.

| | % Responding "agree/strongly agree"| Total Number of Respondents|
|--|--|--|
|All Countries and Periods of Survey | 54% | 13,500 |
| UK: Nov 5-15, 2020 | 55% | 1,005|
| UK: Jan 11-17, 2021 | 80% | 1,000|

Consider first the overall result (all countries/periods). Let $p$ be the fraction of individuals in these 15 countries who approve (agree or strongly agree).

**1.** Use the survey results to estimate $p$. Also estimate the standard error of your estimate.

➡ Type your written work for and answer to Part 1 here.

**2.** Construct a 95% confidence interval for $p$. Interpret.

➡ Type your written work for and answer to Part 2 here.

**3.** Construct a 90% confidence interval for $p$. Is it larger or narrower than the 95% confidence interval?
Why?

➡ Type your written work for and answer to Part 3 here.

**4.** Is there statistical evidence that more than 50% of UK in November 2020 agreed that they would
get the vaccine if given to them? Use the 5 steps for hypothesis testing with a 1% signifiance level.

➡ Type your written work for and answer to Part 4 here.

**5.** Is there statistical evidence that agreement with taking the vaccine in the UK increased in 2021
relative to November 2020 at the 5% significance level? Explain (to answer this question use the 5
steps for hypothesis testing).

➡ Type your written work for and answer to Part 5 here.


# Gaussian 中的 PUHF/PMP2 结果的重新实现

> 创建时间：2019-08-31，最后修改：2019-09-01

在这一份笔记中，我们将使用 PySCF 的功能与 NumPy 重复 Gaussian 中计算的 PUHF 与 PMP2 能量结果；并对 PUHF 与 PMP2 的推导作简单的说明。

```python
from pyscf import gto, scf, mp
```

## 参考结果与体系定义

### Gaussian 结果

在 Gaussian 中，我们使用以下输入卡可以得到 PUHF/PMP2 能量：

```
#p UMP2(Full)/6-31G nosymm

H2O

3 4
O  0.  0.  0.
H  1.  0.  0.
H  0.  1.  0.
```

对于上述分子，其中一些重要的输出结果是

1. $E_\mathrm{UHF}$：-73.0451423839

2. $E_\mathrm{UMP2, corr}$: -0.02646719276

3. $E_\mathrm{UMP2}$: -73.071609576661

4. $\langle S_z \rangle$: 1.5

5. $\langle S^{2(0)} \rangle$: 3.7531

6. $\langle S^{2(0)} \rangle + \langle S^{2(1)} \rangle$: 3.7504

7. $E_\mathrm{PUHF}$：-73.046146318

8. $E_\mathrm{PMP2}$: -73.072180589

输出文件参见 {download}`assets/PUHF_and_PMP2.out`；其中，有效的数据可以通过下述的代码获得：

```python
with open("assets/PUHF_and_PMP2.out", "r") as output:
    output_lines = output.read().split("\n")

for line_num, line_text in enumerate(output_lines):
    if any([keyword in line_text for keyword in
            ["SCF Done", "EUMP2", "<S**2>", "(S**2,1)", "E(PMP2)"]]) \
        and "Initial guess" not in line_text:
        print("line {:03d}: {}".format(line_num, line_text))
```

    line 336:  SCF Done:  E(UHF) =  -73.0451423839     A.U. after   11 cycles
    line 338:  <Sx>= 0.0000 <Sy>= 0.0000 <Sz>= 1.5000 <S**2>= 3.7531 S= 1.5008
    line 370:  (S**2,0)=  0.37531D+01           (S**2,1)=  0.37504D+01
    line 371:  E(PUHF)=      -0.73046146318D+02        E(PMP2)=      -0.73072180589D+02
    line 373:  E2 =    -0.2646719276D-01 EUMP2 =    -0.73071609576661D+02

我们的目标就是近乎准确无误地重复上述八个结果。

### PySCF 体系定义

为了获得与 Gaussian 相同的结果，我们需要定义相同的分子与电荷、多重度环境：

```python
mol = gto.Mole()
mol.atom = """
O 0. 0. 0.
H 1. 0. 0.
H 0. 1. 0.
"""
mol.charge = 3
mol.spin = 3
mol.basis = "6-31G"
mol.build()
```

    <pyscf.gto.mole.Mole at 0x7fa4e5342160>

通过 PySCF 计算 UHF 能量：

```python
scf_eng = scf.UHF(mol)
scf_eng.conv_tol = 1e-10
scf_eng.run();
```

    converged SCF energy = -73.0451423536459  <S^2> = 3.7530824  2S+1 = 4.0015409

上述结果应当能与 $E_\mathrm{UHF}$ 和 $\langle S^{2(0)} \rangle$ 对应。$\langle S_z \rangle = 1.5$ 几乎是显然的。不过，我们仍然不了解 $\langle S^{2(0)} \rangle$ 是如何生成的。

通过 PySCF 计算 UMP2 能量：

```python
mp2_eng = mp.UMP2(scf_eng)
mp2_eng.run();
```

    E(UMP2) = -73.0716095455079  E_corr = -0.0264671918619837

上述结果应当能与 $E_\mathrm{UMP2, corr}$ 和 $E_\mathrm{UMP2}$ 对应。

因此，当前的问题将是回答：如何重复

1. $\langle S^{2(0)} \rangle$: 3.7531

2. $\langle S^{2(0)} \rangle + \langle S^{2(1)} \rangle$: 3.7504

3. $E_\mathrm{PUHF}$：-73.046146318

4. $E_\mathrm{PMP2}$: -73.072180589

### 部分变量定义

首先，我们遵从大多数量化文章中的记号

- $i, j$ 代表占据分子轨道

- $a, b$ 代表非占分子轨道

- $p, q$ 代表任意分子轨道

- $\alpha, \beta$ 代表任意原子轨道

Table 1. 分子相关变量

| 变量名 | 元素记号 | 意义与注解 | 标量或区间 |
|-|-|-|-|
| `nocc_a` | $n_\mathrm{occ}^\alpha$ | $\alpha$ 自旋电子数 | $5$ |
| `nocc_b` | $n_\mathrm{occ}^\beta$ | $\beta$ 自旋电子数 | $2$ |
| `N` | $N$ | 总电子数 | $7$ |
| `nmo` | $n_\mathrm{MO}$ | 分子轨道数 | $13$ |
| `nao` | $n_\mathrm{AO}$ | 原子轨道数 | $13$ |
| `S` | $S_{\mu \nu}$ | 原子轨道重叠积分 | |
| `so_a` | | $\alpha$ 占据轨道分割 | $[0, 5)$ |
| `so_b` | | $\beta$ 占据轨道分割 | $[0, 2)$ |
| `sv_a` | | $\alpha$ 非占轨道分割 | $[5, 13)$ |
| `sv_b` | | $\beta$ 非占轨道分割 | $[2, 13)$ |
| `Sx` | $S_x$ | $x$ 分量自旋 | $0$ |
| `Sy` | $S_y$ | $y$ 分量自旋 | $0$ |
| `Sz` | $S_z$ | $z$ 分量自旋 | $3/2$ |

Table 2. UHF 计算相关变量

| 变量名 | 元素记号 | 意义与注解 |
|-|-|-|
| `C_a` | $C_{\mu p}$ | $\alpha$ 系数矩阵 |
| `C_b` | $C_{\mu \bar p}$ | $\beta$ 系数矩阵 |
| `e_a` | $e_{p}$ | $\alpha$ 轨道能 |
| `e_b` | $e_{\bar p}$ | $\alpha$ 轨道能 |
| `eo_a` | $e_{i}$ | $\beta$ 占据轨道能 |
| `eo_b` | $e_{\bar i}$ | $\alpha$ 占据轨道能 |
| `ev_a` | $e_{a}$ | $\alpha$ 非占轨道能 |
| `ev_b` | $e_{\bar a}$ | $\beta$ 非占轨道能 |
| `D2_aa` | $D_{ij}^{ab}$ | $\alpha, \alpha$ 轨道能差 |
| `D2_ab` | $D_{i \bar j}^{a \bar b}$ | $\alpha, \beta$ 轨道能差 |
| `D2_bb` | $D_{\bar i \bar j}^{\bar a \bar b}$ | $\beta, \beta$ 轨道能差 |

Table 3. UMP2 计算相关变量

| 变量名 | 元素记号 | 意义与注解 |
|-|-|-|
| `t2_aa` | $t_{ij}^{ab}$ | $\alpha, \alpha$ MP2 激发系数 |
| `t2_ab` | $t_{i \bar j}^{a \bar b}$ | $\alpha, \beta$ MP2 激发系数 |
| `t2_bb` | $t_{\bar i \bar j}^{\bar a \bar b}$ | $\beta, \beta$ MP2 激发系数 |
| `D2_aa` | $D_{ij}^{ab}$ | $\alpha, \alpha$ MP2 激发系数分母 |
| `D2_ab` | $D_{i \bar j}^{a \bar b}$ | $\alpha, \beta$ MP2 激发系数分母 |
| `D2_bb` | $D_{\bar i \bar j}^{\bar a \bar b}$ | $\beta, \beta$ MP2 激发系数分母 |

上述需要补充说明的公式有：

$$
S_z = \frac{1}{2} (n_\mathrm{occ}^\alpha - n_\mathrm{occ}^\beta)
$$

$$
D_{i \bar j}^{a \bar b} = e_i + e_{\bar j} - e_a - e_{\bar b}
$$

对于 MP2 激发系数分母，另外两种自旋情况的 $D_{ij}^{ab}$ 与 $D_{\bar i \bar j}^{\bar a \bar b}$ 也可以类似地生成。

```python
# === Molecular
# --- Definition
nocc_a, nocc_b = mol.nelec
N = nocc_a + nocc_b
nmo = nao = mol.nao
S = mol.intor("int1e_ovlp")
# --- Derivative
so_a, so_b = slice(0, nocc_a), slice(0, nocc_b)
sv_a, sv_b = slice(nocc_a, nmo), slice(nocc_b, nmo)
Sx, Sy, Sz = 0, 0, 0.5 * (nocc_a - nocc_b)

# === UHF Calculation
# --- Definition
C_a, C_b = scf_eng.mo_coeff
e_a, e_b = scf_eng.mo_energy
# --- Derivative
eo_a, eo_b = e_a[so_a], e_b[so_b]
ev_a, ev_b = e_a[sv_a], e_b[sv_b]
D2_aa = eo_a[:, None, None, None] + eo_a[None, :, None, None] - ev_a[None, None, :, None] - ev_a[None, None, None, :]
D2_ab = eo_a[:, None, None, None] + eo_b[None, :, None, None] - ev_a[None, None, :, None] - ev_b[None, None, None, :]
D2_bb = eo_b[:, None, None, None] + eo_b[None, :, None, None] - ev_b[None, None, :, None] - ev_b[None, None, None, :]

# === MP2 Calculation
t2_aa, t2_ab, t2_bb = mp2_eng.t2
```

作为对四脚标张量性质的验证，我们计算 MP2 相关能 $E_\mathrm{MP2, corr}$ 如下：

$$
E_\mathrm{MP2, corr} =
\frac{1}{4} \sum_{ijab} (t_{ij}^{ab})^2 D_{ij}^{ab} +
\frac{1}{4} \sum_{\bar i \bar j \bar a \bar b} (t_{\bar i \bar j}^{\bar a \bar b})^2 D_{i \bar j}^{a \bar b} +
\sum_{i \bar j a \bar b} (t_{i \bar j}^{a\bar b})^2 D_{i \bar j}^{a \bar b}
$$

```python
(+ 0.25 * (t2_aa**2 * D2_aa).sum()
 + 0.25 * (t2_bb**2 * D2_bb).sum()
 + (t2_ab**2 * D2_ab).sum())
```

    -0.02646719186198365

PySCF 所给出的 $E_\mathrm{MP2, corr}$ 可以给出相同的结果：

```python
mp2_eng.e_corr
```

    -0.02646719186198366

## $\langle S^2 \rangle$ 相关计算

### 分子轨道基组重叠矩阵 `S_pq` $S_{p \bar q}$

$$
S_{p \bar q} = \sum_{\mu \nu} C_{\mu p} S_{\mu \nu} C_{\nu \bar q}
$$

若用量子力学记号，上述矩阵元素可能表示为

$$
S_{p \bar q} = \int \phi_p (\boldsymbol{r}) \phi_{\bar q} (\boldsymbol{r}) \, \mathrm{d} \boldsymbol{r}
$$

注意上述的积分是空间坐标的积分，不包含自旋部分的积分。

```python
S_pq = C_a.T @ S @ C_b
S_pq.shape
```

    (13, 13)

我们以后还会使用上述矩阵的占据-占据部分 `S_ij` $S_{i \bar j}$、占据-非占部分 `S_ia` $S_{i \bar a}$ 与非占-占据部分 `S_ai` $S_{a \bar i} = S_{\bar i a}$：

```python
S_ij, S_ia, S_ai = S_pq[so_a, so_b], S_pq[so_a, sv_b], S_pq[sv_a, so_b]
[S_ij.shape, S_ia.shape, S_ai.shape]
```

    [(5, 2), (5, 11), (8, 2)]

### `S2_0` $\langle S^{2(0)} \rangle$

$\langle S^{2(0)} \rangle$ 在程序中通常写为 `<S^2>` 或 `<S**2>`。在 Gaussian 计算 PUHF 处，还写为 `(S**2,0)`。这意味着是 UHF 波函数的 $\langle S^2 \rangle_\mathrm{UHF}$。相对地，UMP2 波函数给出的对 $\langle S^2 \rangle$ 的矫正将记作 $\langle S^{2(1)} \rangle$。

参考 Chen and Schlegel [^Chen-Schlegel.JCP.1994.101] Table 1, $0 \rightarrow 0$ 或等价地，Szabo and Ostlund [^Szabo-Ostlund.Dover.1996] eq (2.271)

$$
\langle S^{2(0)} \rangle = \langle \Psi_0 | \hat S^2 | \Psi_0 \rangle = S_z (S_z + 1) + n_\mathrm{occ}^\beta - \sum_{i \bar j} (S_{i \bar j})^2
$$

```python
S2_0 = Sz * (Sz + 1) + nocc_b - (S_ij**2).sum()
S2_0
```

    3.7530823820890378

Gaussian 的参考值是 3.7531。

为了以后的记号便利，我们在这里定义 `L`

$$
L = \sum_{i \bar j} (S_{i \bar j})^2
$$

```python
L = (S_ij**2).sum()
```

### `S2_1` $\langle S^{2(1)} \rangle$

$$
\begin{align}
\langle S^{2(1)} \rangle &= 2 \langle \Psi_0 | \hat S^2 | \Psi^{(1)} \rangle = 2 \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} \langle \Psi_0 | \hat S^2 | \Psi_{i \bar j}^{a \bar b} \rangle \\
&= - 2 \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} \langle i | \bar b \rangle \langle a | \bar j \rangle = - 2 \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} S_{i \bar b} S_{a \bar j}
\end{align}
$$

上式的第一个等号是 Chen and Schlegel [^Chen-Schlegel.JCP.1994.101] eq (5) 所给出的；而第三个等号是 Table 1 $0 \rightarrow \alpha \beta (i, a: \alpha; j, b: \beta)$ 给出的。

上式的推导中有一处关于 $| \Psi^{(1)} \rangle$ 的展开的推导省略。我们知道

$$
| \Psi^{(1)} \rangle = \hat T_2 | \Psi_0 \rangle
= \frac{1}{4} \sum_{ijab} t_{ij}^{ab} | \Psi_{ij}^{ab} \rangle + \frac{1}{4} \sum_{\bar i \bar j \bar a \bar b} t_{\bar i \bar j}^{\bar a \bar b} | \Psi_{\bar i \bar j}^{\bar a \bar b} \rangle + \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} | \Psi_{i \bar j}^{a \bar b} \rangle
$$

但由于利用到 $\langle 0 | \hat S^2 | \Psi_{ij}^{ab} \rangle = \langle 0 | \hat S^2 | \Psi_{\bar i \bar j}^{\bar a \bar b} \rangle = 0$，因此在第二个等号时只从三个 $| \Psi^{(1)} \rangle$ 中留下了一项。关于 $\hat S^2$ 作用在 UHF 波函数与轨道下的性质，可以参考 Schlegel [^Schlegel-Schlegel.JCP.1986.84] eq (5) 的说明。

```python
S2_1 = - 2 * (t2_ab * S_ia[:, None, None, :] * S_ai.T[None, :, :, None]).sum()
S2_1
```

    -0.002658400959213991

因此，UMP2 矫正过的 $\langle S^2 \rangle_\mathrm{UMP2} = \langle S^{2(0)} \rangle + \langle S^{2(1)} \rangle$ 的结果是

```python
S2_0 + S2_1
```

    3.7504239811298237

Gaussian 的参考值是 3.7504。

### `S4SD` $\texttt{S4SD}$

`S4SD` 的表达式较为复杂，我们也直接使用 $\texttt{S4SD}$ 而不用其他记号表示该项：

$$
\begin{align}
\texttt{S4SD} = (n_\mathrm{occ}^\alpha - L) (n_\mathrm{occ}^\beta - L) + 2 L - 2 \sum_{i \bar j k \bar l} S_{i \bar j} S_{\bar j k} S_{k \bar l} S_{\bar l i} + \langle S^{2(0)} \rangle^2
\end{align}
$$

```python
S4SD = (nocc_a - L) * (nocc_b - L) + 2 * L - 2 * (S_ij @ S_ij.T @ S_ij @ S_ij.T).trace() + S2_0**2
S4SD
```

    14.101029785519533

该表达式的来源可能是 Amos and Hall [^Amos-Hall.PRSLA.1961.263]。该文的 eq (7·02) 下方公式中，有通过稍高阶的投影而获得的 $\langle S^2 \rangle$ 的计算方式

$$
\langle S^2 \rangle \simeq \langle S^{2(0)} \rangle + \frac{\texttt{S4SD} - \langle S^{2(0)} \rangle^2}{\langle S^{2(0)} \rangle - (S_z + 1) (S_z + 2)}
$$

通过这种方式获得的 $\langle S^2 \rangle$ 近似值可以相当精确，比 $\langle S^{2(0)} \rangle + \langle S^{2(1)} \rangle$ 还要接近精确值 $3.75$：

```python
S2_0 + (S4SD - S2_0**2) / (S2_0 - (Sz + 1) * (Sz + 2))
```

    3.749999998117527

相信 $\texttt{S4SD}$ 的存在意义是用于计算 Schlegel [^Schlegel-Schlegel.JCP.1986.84] 式 eq (24) 中的 $\langle \tilde \Phi_1 | \tilde \Phi_1 \rangle = \langle \Phi_0 | A_{s + 1}^\dagger A_{s + 1} | \Phi_0 \rangle$；但关于这一关系我还不确定是否正确。后面计算 PMP2 能量时会使用上 $\texttt{S4SD}$。

## 自旋污染矫正能量计算

### `EPUHF` $E_\mathrm{PUHF}$

根据 Schlegel [^Schlegel-Schlegel.JCP.1986.84] eq (22)，PUHF 能量可以表达为

$$
E_\mathrm{PUHF} = E_\mathrm{UHF} + \frac{1}{\langle \Psi_0 | \hat P_s | \Psi_0 \rangle} \sum_{i \bar j a \bar b} \langle \Psi_0 | \hat H | \Psi_{i \bar j}^{a \bar b} \rangle \langle \Psi_{i \bar j}^{a \bar b} | \hat P_s | \Psi_0 \rangle
$$

其中，$\hat P_s$ 算符称为 Löwdin 算符 [^Lowdin-Lowdin.PR.1955.97] eq (7)，

$$
\hat P_s = \prod_{k \neq s}^{N / 2} \frac{\hat S^2 - k (k + 1)}{s (s + 1) - k (k + 1)}
$$

相当于将自旋不纯的波函数纯化为自旋量子数为 $s$ 的态。在实际使用中，通常使用 $\hat A_{s + 1} \simeq \hat P_s$ 替代；关于这段讨论可以参考 Rossky and Karplus [^Rossky-Karplus.JCP.1980.73] section V.A 的讨论，而下面公式的形式参考 Schlegel [^Schlegel-Schlegel.JCP.1986.84] eq (14)；其中，$s$ 一般取 $S_z$：

$$
\hat A_{s + 1} = \frac{\hat S^2 - (s + 1)(s + 2)}{\langle S^{2(0)} \rangle - (s + 1)(s + 2)}
$$

关于 $\hat A_{s + 1}$，一个显然的性质是 $\langle \Psi_0 | \hat A_{s + 1} | \Psi_0 \rangle = 1$。

为了程序方便，定义下述临时变量 `Y`

$$
Y = \langle S^{2(0)} \rangle - (S_z + 1) (S_z + 2)
$$

那么 `D_EPUHF`

$$
\begin{align}
\Delta E_\mathrm{PUHF} &= \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} D_{i \bar j}^{a \bar b} \cdot \langle \Psi_{i \bar j}^{a \bar b} | \frac{\hat S^2}{Y} | \Psi_0 \rangle \\
&= - \frac{1}{Y} \sum_{i \bar j a \bar b} t_{i \bar j}^{a \bar b} D_{i \bar j}^{a \bar b} S_{i \bar b} S_{\bar j a}
\end{align}
$$

```python
Y = S2_0 - (Sz + 1) * (Sz + 2)
D_EPUHF = - 1 / Y * (t2_ab * D2_ab * S_ia[:, None, None, :] * S_ai.T[None, :, :, None]).sum()
D_EPUHF
```

    -0.0010039336320381543

因而 $E_\mathrm{PUHF} = E_\mathrm{UHF} + \Delta E_\mathrm{PUHF}$：

```python
scf_eng.e_tot + D_EPUHF
```

    -73.04614628727792

Gaussian 的参考值是 -73.046146318。

### `EPMP2` $E_\mathrm{PMP2}$

根据 Schlegel [^Schlegel-Schlegel.JCP.1986.84] eq (24)，PMP2 能量可以表达为

$$
\begin{align}
\Delta E_\mathrm{PMP2} = \Delta E_\mathrm{PUHF} \left( 1 - \frac{\langle \Phi^{(1)} | \hat A_{s + 1} | \Psi_0 \rangle}{\langle \Phi_0 | \hat A_{s + 1}^\dagger \hat A_{s + 1} | \Psi_0 \rangle} \right)
\end{align}
$$

关于上式的分数项，分子部分可以写为

$$
\begin{align}
\langle \Phi^{(1)} | \hat A_{s + 1} | \Psi_0 \rangle
= \langle \Phi^{(1)} | \frac{\hat S^2}{Y} - \frac{(s + 1)(s + 2)}{Y} | \Psi_0 \rangle = \frac{1}{2} \frac{\langle S^{2(1)} \rangle}{Y}
\end{align}
$$

而关于分子项，参考在 $\texttt{S4SD}$ 的讨论，

$$
\langle \Phi_0 | \hat A_{s + 1}^\dagger \hat A_{s + 1} | \Psi_0 \rangle \simeq \langle S^2 \rangle - \langle S^{2(0)} \rangle = \frac{\texttt{S4SD} - \langle S^{2(0)} \rangle^2}{Y^2}
$$

但作者不能断定上述论断的正确性。

将分子、分母的结果代入 $\Delta E_\mathrm{PMP2}$ 的算式中，可以得到 `D_EPMP2`

$$
\Delta E_\mathrm{PMP2} = \Delta E_\mathrm{PUHF} \left( 1 - \frac{1}{2} \frac{\langle S^{2(1)} \rangle \cdot Y}{\texttt{S4SD} - \langle S^{2(0)} \rangle^2} \right)
$$

```python
D_EPMP2 = D_EPUHF * (1 - 0.5 * S2_1 * Y / (S4SD - S2_0**2))
D_EPMP2
```

    -0.0005710125302116336

因而 $E_\mathrm{PMP2} = E_\mathrm{UMP2} + \Delta E_\mathrm{PMP2}$：

```python
mp2_eng.e_tot + D_EPMP2
```

    -73.07218055803808

Gaussian 的参考值是 -73.072180589。

至此，我们已经完成了使用 PySCF 的功能与 NumPy 重复 Gaussian 的 PUHF、PMP2 的能量结果了。

## 修订时间轴

- 2019/08/30 写完文档；文档基于 2019/08/13 的一份笔记。

- 2019/09/01 补充一部分推导。

[^Chen-Schlegel.JCP.1994.101]: Chen, W.; Schlegel, H. B. Evaluation of S2 for Correlated Wave Functions and Spin Projection of Unrestricted Moller–Plesset Perturbation Theory. *J. Chem. Phys.* **1994**, *101* (7), 5957–5968. doi: [10.1063/1.467312](https://doi.org/10.1063/1.467312).

[^Szabo-Ostlund.Dover.1996]: Szabo, A.; Ostlund, N. S. *Modern Quantum Chemistry: Introduction to Advanced Electronic Structure Theory (Dover Books on Chemistry)*; Dover Publications, 1996.

[^Schlegel-Schlegel.JCP.1986.84]: Schlegel, H. B. Potential Energy Curves Using Unrestricted Mo/ller–Plesset Perturbation Theory with Spin Annihilation. *J. Chem. Phys.* **1986**, *84* (8), 4530–4534. doi: [10.1063/1.450026](https://doi.org/10.1063/1.450026).

[^Amos-Hall.PRSLA.1961.263]: Amos, A. T.; Hall, G. G. Single Determinant Wave Functions. *Proc. R. Soc. Lond. A* **1961**, *263* (1315), 483–493. doi: [10.1098/rspa.1961.0175](https://doi.org/10.1098/rspa.1961.0175).

[^Lowdin-Lowdin.PR.1955.97]: Lowdin, P.-O. Quantum Theory of Many-Particle Systems. III. Extension of the Hartree-Fock Scheme to Include Degenerate Systems and Correlation Effects. *Phys. Rev.* **1955**, *97* (6), 1509–1520. [10.1103/physrev.97.1509](https://doi.org/10.1103/physrev.97.1509).

[^Rossky-Karplus.JCP.1980.73]: Rossky, P. J.; Karplus, M. Spin Dependent Properties of Perturbed Wave Functions: An Analytic Comparison of the Exact, UHF, and Spin-Projected UHF States. *J. Chem. Phys.* **1980**, *73* (12), 6196–6214. [10.1063/1.440115](https://doi.org/10.1063/1.440115).