# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
date_filename = "2018-03-01_2018-03-31"
```

```python
data = pd.read_csv("articles_" + date_filename + ".csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      4296661b7ec493a798bf313f1133cd1176052f30
      https://www.thetimes.co.uk/article/national-ge...
      National Geographic Traveller reveals photo co...
      2018-03-01 02:44:05.240
      2018-03-01
      4
      0.032796
      2018-03-01T23:14:09.837Z
      0
      1
      3
      The Times
      thetimes_co_uk
      0
      1280
      282.0
      False
      NaN
      NaN
      6435
      NaN

      a995ae644e9cbe39a5b4a41143466399751bdf5e
      https://www.thetimes.co.uk/article/child-safet...
      Child safety fears over director at charity fo...
      2018-03-01 00:09:18.121
      2018-03-01
      4
      0.032787
      2018-03-01T13:33:06.906Z
      0
      1
      3
      The Times
      thetimes_co_uk
      0
      1435
      270.0
      False
      NaN
      NaN
      6435
      125.0

      5e8654da9845e74a797af5a1f8fa076b47c0b6f5
      https://www.thetimes.co.uk/article/mosley-must...
      Mosley must face up to his poisonous past
      2018-03-01 00:09:10.933
      2018-03-01
      25
      0.065574
      2018-03-01T09:29:06.479Z
      4
      3
      18
      The Times
      thetimes_co_uk
      0
      1435
      270.0
      False
      NaN
      NaN
      6435
      125.0

      55c5ece98c02b70430bd19a2087558eaf0288f37
      https://www.thetimes.co.uk/article/skive-off-s...
      Skive off school and enjoy sledging, says head...
      2018-03-01 00:09:22.291
      2018-03-01
      67
      0.163845
      2018-03-01T08:28:11.729Z
      17
      37
      13
      The Times
      thetimes_co_uk
      0
      1435
      270.0
      False
      NaN
      NaN
      6435
      125.0

      19b8c2e31074fa50f06010b8f00a9899994baea1
      https://www.thetimes.co.uk/article/pop-review-...
      Pop review: Yonaka at the Garage, N5
      2018-03-01 17:05:18.708
      2018-03-01
      128
      0.541427
      2018-03-01T21:21:14.439Z
      9
      115
      4
      The Times
      thetimes_co_uk
      0
      1855
      332.0
      False
      NaN
      NaN
      6435
      126.0

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    146484.00
    mean       1519.93
    std       15132.24
    min           0.00
    50%          25.00
    75%         258.00
    90%        1597.00
    95%        4350.85
    99%       27468.06
    99.5%     51204.43
    99.9%    164145.49
    max     2915056.00
    Name: fb_engagements, dtype: float64

There's 2 articles with more than 1 million engagements this month.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      b8a75ce80f2c523e6ba16bb3b6439bdf8c37b047
      https://www.theguardian.com/science/2018/mar/1...
      Stephen Hawking: modern cosmology's brightest ...
      2018-03-14 03:49:11.754
      2018-03-14 03:46:33
      2915056
      203793.62
      2018-03-14T14:30:08.370Z
      195155
      2321470
      398431
      The Guardian
      theguardian_com
      519
      1259
      107.00
      True
      7925040.00
      2018-03-14T03:49:46.000Z
      142
      2007.00

      3a34a2b4590e487def08b1918783a5d8dd39f73a
      http://www.bbc.co.uk/news/uk-43396008
      Physicist Stephen Hawking dies aged 76
      2018-03-14 03:49:13.874
      2018-03-14 03:47:02
      2016997
      8610.41
      2018-03-14T04:12:06.080Z
      141418
      1517206
      358373
      BBC
      bbc_co_uk
      545
      1837
      59.00
      True
      46116240.00
      2018-03-14T03:50:08.000Z
      96
      66.00

```python
data.fb_engagements.mode()
```

    0    0
    dtype: int64

*november* Going back to the enagement counts, we see the mean is 1,117, mode is zero, median is 24, 90th percentile is 1,453, 99th percentile is 21,166, 99.5th percentile is 33,982. The standard deviation is 8,083, significantly higher than the mean, so this is not a normal distribution.

*december* Going back to the enagement counts, we see the mean is 1,106, mode is zero, median is 24, 90th percentile is 1,545, 99th percentile is 20,228, 99.5th percentile is 32,446. The standard deviation is 9,852, significantly higher than the mean, so this is not a normal distribution.

*january 2018* Going back to the enagement counts, we see the mean is 1,108, mode is zero, median is 26, 90th percentile is 1,621, 99th percentile is 19,918, 99.5th percentile is 32,935. The standard deviation is 8,278, significantly higher than the mean, so this is not a normal distribution.

*february 2018* Going back to the enagement counts, we see the mean is 1,237, mode is zero, median is 25, 90th percentile is 1,453, 99th percentile is 23,172, 99.5th percentile is 38,692. The standard deviation is 11,019, significantly higher than the mean, so this is not a normal distribution.

*march 2018* Going back to the enagement counts, we see the mean is 1,519, mode is zero, median is 25, 90th percentile is 1,597, 99th percentile is 27,468, 99.5th percentile is 51,204. The standard deviation is 15,132, significantly higher than the mean, so this is not a normal distribution.

Key publishers stats

```python
data.groupby("publisher_id").agg({'url': 'count', 'fb_engagements': ['sum', 'median', 'mean']})
```

      url
      fb_engagements

      count
      sum
      median
      mean

      publisher_id

      anotherangryvoice_blogspot_co_uk
      38
      78751
      1665.50
      2072.39

      bbc_co_uk
      11963
      11962778
      35.00
      999.98

      breitbart_com
      2737
      7662297
      173.00
      2799.52

      brexitcentral_com
      66
      36151
      260.50
      547.74

      buzzfeed_com
      1452
      4494915
      177.50
      3095.67

      cnn_com
      4368
      34823949
      589.50
      7972.52

      dailymail_co_uk
      25800
      16802567
      19.00
      651.26

      economist_com
      579
      170498
      35.00
      294.47

      evolvepolitics_com
      82
      342679
      2473.50
      4179.01

      foxnews_com
      5340
      21348445
      98.00
      3997.84

      ft_com
      4913
      362496
      4.00
      73.78

      huffingtonpost_com
      2564
      18008340
      849.00
      7023.53

      independent_co_uk
      5590
      4788850
      28.00
      856.68

      indy100_com
      471
      515637
      101.00
      1094.77

      lemonde_fr
      4229
      2780672
      65.00
      657.52

      libdemvoice_org
      165
      3072
      11.00
      18.62

      mirror_co_uk
      10148
      5902958
      28.00
      581.69

      nbcnews_com
      2017
      10900196
      447.00
      5404.16

      newstatesman_com
      489
      95347
      23.00
      194.98

      npr_org
      1661
      12637764
      663.00
      7608.53

      nytimes_com
      5016
      25629208
      225.00
      5109.49

      order-order_com
      264
      93065
      170.50
      352.52

      propublica_org
      43
      50915
      341.00
      1184.07

      reuters_com
      6679
      3915748
      15.00
      586.28

      rt_com
      2322
      2359592
      353.00
      1016.19

      skwawkbox_org
      124
      306273
      278.50
      2469.94

      telegraph_co_uk
      4501
      1448183
      25.00
      321.75

      thecanary_co
      258
      358003
      453.50
      1387.61

      theguardian_com
      8409
      13987337
      124.00
      1663.38

      thetimes_co_uk
      9760
      405336
      1.00
      41.53

      washingtonpost_com
      23764
      19059313
      0.00
      802.02

      westmonster_com
      255
      292326
      44.00
      1146.38

      yournewswire_com
      417
      1022159
      58.00
      2451.22

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
eng = data.fb_engagements[(data.fb_engagements < 5000)]
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(15,7))
plt.hist(eng, bins=50)
plt.title("Article count by engagements")
plt.axvline(median, label=f'Median ({median:,.0f})', color='green')
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(ninety, linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(ninetyfive, linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
# plt.axvline(ninetynine, linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   119204.00
    mean         5.03
    std          1.80
    min          3.26
    25%          3.53
    50%          4.42
    75%          6.10
    max         12.01
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   119204.00
    mean        10.61
    std         10.29
    min          0.00
    25%          2.00
    50%          7.00
    75%         17.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   146484.00
    mean         9.15
    std         90.17
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        275.00
    99.5%      579.00
    99.9%     1174.65
    max       7504.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   4086.00
    mean     328.16
    std      432.24
    min        4.00
    25%       84.00
    50%      179.00
    75%      433.50
    90%      839.50
    95%     1068.00
    99%     1594.15
    99.5%   1962.30
    99.9%   5636.62
    max     7504.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    lemonde_fr                             618
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      38.00
      5.33
      0.00
      5.33
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      108.00
      12.87
      1.50
      2.19
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      213.00
      8.35
      1.34
      0.72
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      49.00
      4.14
      1.16
      0.69
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      314.00
      11.69
      1.36
      1.80
      12.02
      12.02
      12.02
      12.02

      cnn_com
      210.00
      11.94
      2.64
      0.86
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      150.00
      11.47
      1.62
      0.99
      11.85
      11.85
      11.85
      11.85

      economist_com
      66.00
      7.62
      1.35
      0.53
      7.99
      7.99
      7.99
      7.99

      foxnews_com
      101.00
      10.54
      0.79
      2.65
      10.61
      10.61
      10.61
      10.61

      ft_com
      102.00
      7.54
      1.65
      0.54
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      162.00
      10.87
      1.35
      0.93
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      144.00
      9.69
      1.43
      0.84
      10.07
      10.07
      10.07
      10.07

      indy100_com
      138.00
      6.14
      1.68
      1.17
      5.87
      7.04
      7.04
      7.04

      lemonde_fr
      233.00
      8.15
      2.48
      0.62
      9.34
      9.34
      9.34
      9.34

      libdemvoice_org
      159.00
      4.57
      0.53
      0.78
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      6.00
      5.62
      3.11
      1.52
      3.47
      5.56
      8.23
      9.15

      nbcnews_com
      111.00
      8.71
      1.10
      0.74
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      63.00
      6.11
      0.81
      2.64
      6.35
      6.35
      6.35
      6.35

      npr_org
      174.00
      9.18
      1.04
      1.57
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      64.00
      12.53
      0.00
      12.53
      12.53
      12.53
      12.53
      12.53

      order-order_com
      262.00
      4.53
      1.64
      0.39
      3.27
      5.78
      5.78
      5.78

      propublica_org
      18.00
      6.20
      0.00
      6.20
      6.20
      6.20
      6.20
      6.20

      reuters_com
      100.00
      9.32
      1.44
      0.81
      9.66
      9.66
      9.66
      9.66

      rt_com
      166.00
      9.68
      1.65
      0.85
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      124.00
      4.69
      0.81
      1.17
      5.03
      5.03
      5.03
      5.03

      telegraph_co_uk
      102.00
      9.68
      1.91
      0.68
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      239.00
      4.90
      1.65
      0.93
      3.83
      6.11
      6.21
      6.21

      theguardian_com
      136.00
      10.89
      2.90
      1.01
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      63.00
      6.69
      0.86
      1.71
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      101.00
      11.04
      1.63
      1.71
      11.42
      11.42
      11.42
      11.42

      yournewswire_com
      170.00
      5.73
      0.61
      2.79
      5.99
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.38500569152790032, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     142399
    1.00         40
    2.00         43
    3.00         61
    4.00         60
    5.00         59
    6.00         71
    7.00        272
    8.00        196
    9.00        327
    10.00       210
    11.00       176
    12.00        81
    13.00       106
    14.00       320
    15.00       436
    16.00       379
    17.00       263
    18.00       240
    19.00       399
    20.00       346
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      14.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  0.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    lemonde_fr                         15.00
    libdemvoice_org                     7.00
    mirror_co_uk                       14.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     0.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    2.4500000000000002

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     74079
    2.00      7491
    3.00      4745
    4.00      4502
    5.00       829
    6.00       667
    7.00       534
    8.00       437
    9.00       867
    10.00      493
    11.00      242
    12.00      148
    13.00      100
    14.00       87
    15.00       18
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk      38.00
    bbc_co_uk                          14465.00
    breitbart_com                       2641.00
    brexitcentral_com                     66.00
    buzzfeed_com                       10488.00
    cnn_com                            12675.00
    dailymail_co_uk                    14438.00
    economist_com                        286.00
    evolvepolitics_com                    79.00
    foxnews_com                         6709.00
    ft_com                              3187.00
    huffingtonpost_com                  8335.00
    independent_co_uk                   3902.00
    indy100_com                          470.00
    lemonde_fr                          4216.00
    libdemvoice_org                      165.00
    mirror_co_uk                        9635.00
    nbcnews_com                         1801.00
    newstatesman_com                     482.00
    npr_org                             2919.00
    nytimes_com                         9791.00
    order-order_com                      263.00
    propublica_org                        43.00
    reuters_com                         7705.00
    rt_com                              4740.00
    skwawkbox_org                        124.00
    telegraph_co_uk                     4415.00
    thecanary_co                         255.00
    theguardian_com                    12338.00
    thetimes_co_uk                      9740.00
    washingtonpost_com                  9571.00
    westmonster_com                      246.00
    yournewswire_com                     416.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    4296661b7ec493a798bf313f1133cd1176052f30   1.00
    a995ae644e9cbe39a5b4a41143466399751bdf5e   1.00
    5e8654da9845e74a797af5a1f8fa076b47c0b6f5   1.00
    55c5ece98c02b70430bd19a2087558eaf0288f37   1.00
    19b8c2e31074fa50f06010b8f00a9899994baea1   1.00
    8fd307ee7359d9bc37ee1d0de1bdedf27c67aaec   1.00
    e9e44511e90365c6806d5d21c442dec0e5ce2289   1.00
    29df6d80a9f8a8691492021fbdb407bd873f7634   1.00
    4f23bde73ab83522dfe7d326f92500acb4c57ede   1.00
    a43485f058f904f73b53bcae2bd63b48def31851   1.00
    1eba49de1ed24dd27c87cc7f7df88eea14b77ffb   1.00
    74acd3f538a979ec306a0e23ab78376a2c02964d   1.00
    7c256cc04797c8a828adcc7a60ae5ec4c97af7e1   1.00
    0262e43862244464024660cd2cbbafb71a2cd279   1.00
    fc3207065dc6daf733c243411a60553858ad57e4   1.00
    b4b2a48f6570ecc54d037e20642fd1a72aee6d07   1.00
    20b8b495a1f324efc7fcdf563ed7c500342bc0a6   1.00
    8e5f315c408da033da14ca097efe1c1cf58f1e24   1.00
    302848dc6974b2c1f8e3c13005490801c13fab01   1.00
    ad63302360007ff4aaca5a3ca838e6b323c7f2e8   1.00
    c50275a656ed2e45255f4f34074e7b431821cdf4   1.00
    fa09e221612841335239ef657bb0e159b12a8790   1.00
    07b362eabee7ecf9852588bb5b65c0b7ba7ab3ab   1.00
    f05506729453ebf18fcc760d153805fca11a314c   1.00
    5ab3420fd5c7a9d2f4440efddc2be6d0eaf83a69   1.00
    a472d447d13cf62efc56dd028293115b27b7c7d0   1.00
    11b7bc3b9e8d7317c960b081778b156ac38d957c   1.00
    f115164f2a4becf341ed2980acf23e496b82aa1c   1.00
    1bcb84381c9c6954e7194428042f49779ea6a3a0   1.00
    de3f21baede6de63afc29bcd557aa97b85d0da56   1.00
                                               ...
    f62818af1cb8aa90e8334d4a061901c427be88f8   0.00
    9776277c068ca6b7ece5c1e0ef3581afe1958908   0.00
    9b0db6758f03513f09a33ea41cc8ecc511477282   0.00
    c8adc816a8455481a6ee2ffad6c9dc9cfbb9295f   0.00
    448c0976efe0dab8ffbd36fdd65e29f483798ac1   0.00
    723641a365d8a80e6435b5f6d1681c38e0b2bc68   1.00
    fc77940ed0980846117a7e37aceda618e310042a   1.00
    051240fb8de7e3bf81122c3432649f8e934c6d9a   0.00
    f2cb3bce762a35a29045820da033bffeb264611c   0.00
    e4ea7c7dd830a8a536c02dbb22db8d06e835ff0c   6.00
    a75381d7ee18af66a174f26fcb6e9c84c3a862dd   1.00
    09588094ea55b016b587aff909ac1e6e687b83ce   1.00
    adffbfec52cc8cfb0dd9ea8a15e6cd00e0eac35f   1.00
    7761e92901f2682bd3205f3db8d18f6f396f7092   1.00
    7b78b7d3509c6d15fcf46237d7b004d5364c3b26   1.00
    86e01feb83cf77fa60a0f942903154b315f9039e   1.00
    1ffec218f2615e0c1420a1d783d524720a5bdf27   1.00
    99bfc46e13bd0d7c42bfa28aa011e2583b732f9b   1.00
    57e16ac7e15a6b6b208c7c40d630ea40eb70739d   3.00
    fb78fef5863200d46ab89101bbc20a4ac0b7a6f8   1.00
    1cede55875753f22355802b65bc765bef3aa08b0   1.00
    78050ebf3f153ebe09b5012171df31c96f8b41b5   1.00
    a3ec6f7c4f1f492ccd315648f46de1a57ac32d86   1.00
    e0832b31125db89d7277487524a3bc3e9d410df7   1.00
    d006b8c4e6e302621dd2033eb38656e0edfc909c   1.00
    6438936238c8f4c27571c8b76ec588595314a1ce   1.00
    c6a120ac5c80edf83da06277836da094c6925955   1.00
    017ea59645d12a0f4e20385f13a9803455016734   1.00
    5a74f3bf5380e19c8b432b3ca0eddba1c4d469c9   1.00
    13db19d820572f0e174d36463e243624e16972d4   1.00
    Name: front_score, Length: 146484, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    46501656.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      38.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      bbc_co_uk
      11963.00
      0.52
      2.75
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2737.00
      0.78
      0.98
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      66.00
      0.56
      0.50
      0.00
      0.00
      1.00
      1.00
      1.00

      buzzfeed_com
      1452.00
      0.50
      0.50
      0.00
      0.00
      0.00
      1.00
      1.00

      cnn_com
      4368.00
      2.54
      4.36
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      25800.00
      0.61
      1.64
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      579.00
      2.50
      1.12
      0.00
      3.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      82.00
      0.94
      0.24
      0.00
      1.00
      1.00
      1.00
      1.00

      foxnews_com
      5340.00
      0.83
      2.07
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      4913.00
      0.49
      0.86
      0.00
      0.00
      0.00
      0.00
      2.00

      huffingtonpost_com
      2564.00
      1.84
      1.99
      0.00
      0.00
      0.00
      4.00
      4.00

      independent_co_uk
      5590.00
      0.75
      1.30
      0.00
      0.00
      0.00
      3.00
      3.00

      indy100_com
      471.00
      0.79
      0.41
      0.00
      1.00
      1.00
      1.00
      1.00

      lemonde_fr
      4229.00
      0.70
      0.95
      0.00
      0.00
      0.00
      2.00
      2.00

      libdemvoice_org
      165.00
      0.96
      0.19
      0.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      10148.00
      0.22
      0.42
      0.00
      0.00
      0.00
      0.00
      1.00

      nbcnews_com
      2017.00
      2.50
      1.94
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      489.00
      0.75
      0.43
      0.00
      1.00
      1.00
      1.00
      1.00

      npr_org
      1661.00
      1.75
      1.48
      0.00
      0.00
      3.00
      3.00
      3.00

      nytimes_com
      5016.00
      1.55
      2.31
      0.00
      0.00
      0.00
      5.00
      6.00

      order-order_com
      264.00
      0.78
      0.41
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      43.00
      0.84
      0.37
      0.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      6679.00
      0.59
      0.91
      0.00
      0.00
      0.00
      2.00
      2.00

      rt_com
      2322.00
      1.29
      0.96
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      124.00
      0.99
      0.09
      0.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      4501.00
      0.57
      0.91
      0.00
      0.00
      0.00
      2.00
      2.00

      thecanary_co
      258.00
      0.19
      0.39
      0.00
      0.00
      0.00
      0.00
      1.00

      theguardian_com
      8409.00
      0.53
      1.14
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      9760.00
      0.05
      0.22
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      23764.00
      0.11
      0.45
      0.00
      0.00
      0.00
      0.00
      2.00

      westmonster_com
      255.00
      0.29
      0.45
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      417.00
      0.18
      0.39
      0.00
      0.00
      0.00
      0.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     334591.00
    bbc_co_uk                          46501656.00
    breitbart_com                       3855739.00
    brexitcentral_com                     16040.00
    buzzfeed_com                        2941240.00
    cnn_com                            30055121.00
    dailymail_co_uk                    14651250.00
    economist_com                       8528948.00
    evolvepolitics_com                   128042.00
    foxnews_com                        16377561.00
    ft_com                              3776695.00
    huffingtonpost_com                  9872568.00
    independent_co_uk                   8204164.00
    indy100_com                          243508.00
    lemonde_fr                          4147802.00
    libdemvoice_org                        8634.00
    mirror_co_uk                        2987463.00
    nbcnews_com                         9699612.00
    newstatesman_com                     154653.00
    npr_org                             6312234.00
    nytimes_com                        15594031.00
    order-order_com                       45697.00
    propublica_org                       379791.00
    reuters_com                         3999322.00
    rt_com                              4995093.00
    skwawkbox_org                          7897.00
    telegraph_co_uk                     4425790.00
    thecanary_co                         158047.00
    theguardian_com                     7948187.00
    thetimes_co_uk                       764648.00
    washingtonpost_com                  6158264.00
    westmonster_com                       19853.00
    yournewswire_com                      28490.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.72
    bbc_co_uk                          17.65
    breitbart_com                      15.17
    brexitcentral_com                   9.68
    buzzfeed_com                       14.89
    cnn_com                            17.22
    dailymail_co_uk                    16.50
    economist_com                      15.96
    evolvepolitics_com                 11.76
    foxnews_com                        16.61
    ft_com                             15.14
    huffingtonpost_com                 16.11
    independent_co_uk                  15.92
    indy100_com                        12.40
    lemonde_fr                         15.24
    libdemvoice_org                     9.06
    mirror_co_uk                       14.91
    nbcnews_com                        16.09
    newstatesman_com                   11.95
    npr_org                            15.66
    nytimes_com                        16.56
    order-order_com                    10.73
    propublica_org                     12.85
    reuters_com                        15.20
    rt_com                             15.42
    skwawkbox_org                       8.97
    telegraph_co_uk                    15.30
    thecanary_co                       11.97
    theguardian_com                    15.89
    thetimes_co_uk                     13.55
    washingtonpost_com                 15.63
    westmonster_com                     9.90
    yournewswire_com                   10.26
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.81
    bbc_co_uk                          10.75
    breitbart_com                       8.26
    brexitcentral_com                   2.78
    buzzfeed_com                        7.99
    cnn_com                            10.31
    dailymail_co_uk                     9.59
    economist_com                       9.05
    evolvepolitics_com                  4.85
    foxnews_com                         9.70
    ft_com                              8.24
    huffingtonpost_com                  9.20
    independent_co_uk                   9.01
    indy100_com                         5.50
    lemonde_fr                          8.33
    libdemvoice_org                     2.16
    mirror_co_uk                        8.00
    nbcnews_com                         9.18
    newstatesman_com                    5.04
    npr_org                             8.75
    nytimes_com                         9.65
    order-order_com                     3.82
    propublica_org                      5.94
    reuters_com                         8.29
    rt_com                              8.52
    skwawkbox_org                       2.07
    telegraph_co_uk                     8.40
    thecanary_co                        5.06
    theguardian_com                     8.98
    thetimes_co_uk                      6.64
    washingtonpost_com                  8.73
    westmonster_com                     2.99
    yournewswire_com                    3.35
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   4.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    lemonde_fr                         12.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       3.00
    telegraph_co_uk                    12.00
    thecanary_co                        8.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     5.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   3.00
    buzzfeed_com                       11.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  6.00
    foxnews_com                        14.00
    ft_com                             11.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         7.00
    lemonde_fr                         12.00
    libdemvoice_org                     2.00
    mirror_co_uk                       11.00
    nbcnews_com                        13.00
    newstatesman_com                    6.00
    npr_org                            12.00
    nytimes_com                        14.00
    order-order_com                     4.00
    propublica_org                      8.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        6.00
    theguardian_com                    13.00
    thetimes_co_uk                      9.00
    washingtonpost_com                 12.00
    westmonster_com                     3.00
    yournewswire_com                    4.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     21250
    1.00     19394
    2.00     12021
    3.00      8741
    4.00      6532
    5.00      5408
    6.00      4333
    7.00      3791
    8.00      3415
    9.00      3065
    10.00     2821
    11.00     2518
    12.00     2342
    13.00     2350
    14.00     2266
    15.00     2083
    16.00     1894
    17.00     1902
    18.00     1892
    19.00     1765
    20.00     1715
    21.00     1635
    22.00     1592
    23.00     1578
    24.00     1500
    25.00     1551
    26.00     1484
    27.00     1382
    28.00     1311
    29.00     1324
             ...
    66.00      107
    67.00       82
    68.00       70
    69.00       65
    70.00       52
    71.00       51
    72.00       45
    73.00       32
    74.00       43
    75.00       35
    76.00       19
    77.00       28
    78.00       36
    79.00       27
    80.00       28
    81.00       17
    82.00       14
    83.00       15
    84.00        6
    85.00       15
    86.00        8
    87.00        7
    88.00       15
    89.00        7
    90.00        4
    91.00        6
    92.00        3
    93.00        2
    94.00        4
    95.00        1
    Name: attention_index, Length: 96, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      3a34a2b4590e487def08b1918783a5d8dd39f73a
      http://www.bbc.co.uk/news/uk-43396008
      Physicist Stephen Hawking dies aged 76
      2018-03-14 03:49:13.874
      2018-03-14 03:47:02.000
      2016997
      8610.41
      2018-03-14T04:12:06.080Z
      141418
      1517206
      358373
      ...
      46116240.00
      2018-03-14T03:50:08.000Z
      96
      66.00
      50.00
      20.00
      10.00
      15.00
      45.00
      95.00

      e7607247fffe2d26b41a40d3e538ca6aa3713c6a
      https://www.cnn.com/2018/03/25/us/emma-gonzale...
      Emma Gonzalez profile: What you need to know a...
      2018-03-25 13:04:19.014
      2018-03-25 12:56:50.000
      233961
      787.19
      2018-03-25T15:48:12.330Z
      29785
      189759
      14417
      ...
      29898261.00
      2018-03-25T15:30:01.000Z
      105
      851.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      424489ba1c79b4b8d4da2e45f7fd358322947ebc
      https://www.cnn.com/2018/03/21/us/austin-explo...
      Austin bomber's motive is to cause 'mayhem and...
      2018-03-21 06:10:16.915
      2018-03-21 06:06:52.000
      167127
      381.36
      2018-03-21T13:15:12.369Z
      39679
      99918
      27530
      ...
      29873177.00
      2018-03-22T00:03:09.000Z
      105
      688.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      0fd72776c86f7d09898cb4f9e59b79d589127ae8
      https://www.cnn.com/2018/03/31/politics/nra-me...
      Ted Nugent calls Parkland survivors 'liars' an...
      2018-03-31 23:34:16.919
      2018-03-31 23:30:04.000
      149760
      251.02
      2018-04-01T00:57:05.961Z
      60523
      78564
      10673
      ...
      29918466.00
      2018-04-01T00:21:12.000Z
      105
      567.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      8d4daa22650daac24f1278a5e31c1f7204f47380
      https://www.cnn.com/2018/03/20/us/maryland-sch...
      Lone resource officer's quick action stopped t...
      2018-03-20 19:19:19.249
      2018-03-20 19:14:25.000
      233418
      1128.83
      2018-03-21T13:08:07.973Z
      41196
      169456
      22766
      ...
      29865951.00
      2018-03-20T22:30:14.000Z
      105
      694.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      7664d105c1e404927d190c8ef9106192165226d0
      https://www.cnn.com/2018/03/16/politics/andrew...
      Ex-FBI deputy director Andrew McCabe fired
      2018-03-17 02:09:08.055
      2018-03-17 02:00:06.000
      289576
      1156.07
      2018-03-17T03:14:08.419Z
      100382
      162935
      26259
      ...
      29844606.00
      2018-03-17T02:08:52.000Z
      105
      113.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      8848efde83e686020d400bbc7505803567b9331c
      http://www.bbc.co.uk/news/world-europe-43525267
      French police 'hero' dies of wounds
      2018-03-24 05:13:12.232
      2018-03-24 05:10:28.000
      113294
      285.37
      2018-03-24T08:40:11.781Z
      10422
      91883
      10989
      ...
      46201300.00
      2018-03-24T08:30:01.000Z
      96
      83.00
      48.00
      20.00
      10.00
      15.00
      45.00
      93.00

      0705ea630bb0def04ffcbe69420b63a500879d43
      http://www.bbc.co.uk/news/world-europe-43324406
      International Women's Day: Trains cancelled as...
      2018-03-08 03:25:11.956
      2018-03-08 03:22:33.000
      91511
      237.61
      2018-03-08T17:58:06.745Z
      3914
      78771
      8826
      ...
      46026903.00
      2018-03-08T17:40:00.000Z
      96
      289.00
      47.00
      20.00
      10.00
      15.00
      45.00
      92.00

      2e5fa0c72c95854d8fec1ad919caef746a97894e
      https://www.cnn.com/2018/03/16/politics/tennes...
      Resolution denouncing white supremacists and n...
      2018-03-16 15:10:20.763
      2018-03-16 15:07:46.000
      113404
      380.13
      2018-03-17T15:08:09.009Z
      23082
      69399
      20923
      ...
      29843740.00
      2018-03-16T16:31:06.000Z
      105
      398.00
      48.00
      20.00
      9.00
      15.00
      44.00
      92.00

      a487a5b5754e7ef5e0cdbbadbf0195f88347ce4b
      https://www.cnn.com/2018/03/23/politics/transg...
      White House announces policy to ban most trans...
      2018-03-24 01:34:16.864
      2018-03-24 01:29:02.000
      114599
      249.08
      2018-03-24T03:08:06.356Z
      30346
      73204
      11049
      ...
      29880855.00
      2018-03-24T01:55:03.000Z
      105
      87.00
      48.00
      20.00
      9.00
      15.00
      44.00
      92.00

      b2688f1ef5dfcb7d0cfa192634af8c318db75736
      https://www.cnn.com/2018/03/14/health/stephen-...
      Stephen Hawking, renowned scientist, dies at 76
      2018-03-14 04:07:13.160
      2018-03-14 04:03:53.000
      711418
      2328.02
      2018-03-14T04:19:09.362Z
      58142
      545087
      108189
      ...
      29827961.00
      2018-03-14T04:09:41.000Z
      105
      653.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      50436ebae1b3cdca23bc56a93b67aeb35087c01a
      https://www.cnn.com/2018/03/27/politics/donald...
      Trump suggests US military foot the bill for b...
      2018-03-27 18:34:22.226
      2018-03-27 18:29:24.000
      125500
      409.71
      2018-03-28T13:03:08.153Z
      47570
      65167
      12763
      ...
      29916236.00
      2018-03-31T10:00:21.000Z
      105
      482.00
      49.00
      20.00
      7.00
      15.00
      42.00
      91.00

      0be4709b4b6141a6344aa3ae90df89aba5c96eb4
      https://www.buzzfeed.com/arianelange/john-kric...
      Creator Of “Ren & Stimpy” Accused Of Preying O...
      2018-03-29 20:40:13.743
      2018-03-29 20:37:53.000
      94263
      125.60
      2018-03-30T00:54:08.282Z
      31786
      51169
      11308
      ...
      2936785.00
      2018-03-29T21:20:54.000Z
      147
      4111.00
      47.00
      19.00
      14.00
      11.00
      44.00
      91.00

      328f86eb43abe871db40cac05697a816367bb89b
      https://www.cnn.com/2018/03/28/politics/donald...
      An unheard-of problem: The President can't fin...
      2018-03-28 05:31:22.960
      2018-03-28 05:26:52.000
      181848
      460.77
      2018-03-28T13:07:05.137Z
      37122
      126413
      18313
      ...
      29908332.00
      2018-03-28T13:29:04.000Z
      105
      1173.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      a459ce106c02660968f2836220d0118a9a2a6824
      http://money.cnn.com/2018/03/14/news/companies...
      Toys 'R' Us will close or sell all US stores
      2018-03-14 22:44:11.213
      2018-03-14 22:38:00.000
      159559
      515.25
      2018-03-14T23:18:03.648Z
      40065
      96445
      23049
      ...
      29837453.00
      2018-03-14T22:45:04.000Z
      105
      nan
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      7f88871e141af09b1e5de790cd2d0b440d1adf4c
      https://www.cnn.com/2018/03/22/politics/john-d...
      Dowd resigns as Trump's lawyer
      2018-03-22 15:37:16.332
      2018-03-22 15:31:14.000
      182692
      855.50
      2018-03-22T16:00:09.907Z
      31654
      128300
      22738
      ...
      29875784.00
      2018-03-22T15:45:06.000Z
      105
      76.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      25dc33625aea1e8d559a344bbf837ea986bd4a08
      https://www.huffingtonpost.com/entry/andrew-mc...
      Read Andrew McCabe's Response To Being Fired T...
      2018-03-17 02:39:14.557
      2018-03-17 02:33:26.000
      161658
      687.37
      2018-03-17T19:09:08.459Z
      40559
      97409
      23690
      ...
      9870240.00
      2018-03-17T14:30:04.000Z
      215
      954.00
      50.00
      17.00
      10.00
      13.00
      40.00
      90.00

      7ffffa6d7974335419e3c65059fc4ea3de49af3d
      https://www.cnn.com/2018/03/12/us/texas-austin...
      Austin explosions: 2 deadly blasts connected, ...
      2018-03-12 17:22:18.966
      2018-03-12 17:19:09.000
      72077
      197.98
      2018-03-12T19:07:11.845Z
      12615
      43108
      16354
      ...
      29816162.00
      2018-03-12T18:44:02.000Z
      105
      200.00
      46.00
      20.00
      9.00
      15.00
      44.00
      90.00

      1c6abd59d584927743ac3606cc6783edc6653805
      https://www.cnn.com/2018/03/15/politics/robert...
      NYT: Mueller subpoenas Trump Organization
      2018-03-15 18:07:22.216
      2018-03-15 18:03:15.000
      94218
      532.91
      2018-03-15T19:13:13.249Z
      19521
      64878
      9819
      ...
      29842457.00
      2018-03-15T18:15:04.000Z
      105
      92.00
      47.00
      20.00
      8.00
      15.00
      43.00
      90.00

      40168add43bafc4da17ca035e9b52e967055e680
      https://www.cnn.com/2018/03/08/politics/donald...
      Trump accepts offer to meet Kim Jong Un
      2018-03-09 00:25:18.624
      2018-03-09 00:20:14.000
      90701
      210.60
      2018-03-09T02:12:10.970Z
      34902
      42937
      12862
      ...
      29798449.00
      2018-03-09T00:50:02.000Z
      105
      34.00
      47.00
      20.00
      8.00
      15.00
      43.00
      90.00

      9b25e487d0eaa9639787f6a8e205a98b4acc0ac0
      https://www.buzzfeed.com/tomnamako/ralph-peters
      An "Ashamed" Fox News Commentator Just Quit Th...
      2018-03-20 20:07:19.832
      2018-03-20 20:00:15.000
      76239
      172.22
      2018-03-20T22:44:19.065Z
      15047
      43957
      17235
      ...
      2935278.00
      2018-03-20T20:30:51.000Z
      147
      603.00
      46.00
      19.00
      13.00
      11.00
      43.00
      89.00

      505077913306a55b2cf4e8cd7ca01fc810f78749
      https://www.cnn.com/2018/03/17/politics/mccabe...
      McCabe has memos documenting his conversations...
      2018-03-17 17:43:13.101
      2018-03-17 17:36:54.000
      83981
      711.60
      2018-03-17T19:08:04.728Z
      15804
      60628
      7549
      ...
      29849494.00
      2018-03-17T18:46:10.000Z
      105
      146.00
      47.00
      20.00
      7.00
      15.00
      42.00
      89.00

      3e9f91659c6cfd2ef4e3f1db04d19f81c24d59f3
      https://www.cnn.com/2018/03/01/politics/ivanka...
      Exclusive: FBI counterintel looks at Ivanka Tr...
      2018-03-01 23:01:21.164
      2018-03-01 22:59:33.000
      58344
      294.07
      2018-03-02T16:15:13.750Z
      12208
      36614
      9522
      ...
      29763429.00
      2018-03-01T23:15:05.000Z
      105
      1276.00
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      9afbf17229fbe2035b78545478034672f64db046
      https://www.cnn.com/2018/03/08/politics/trump-...
      Trump upset with Sanders over Stormy Daniels r...
      2018-03-08 13:49:18.486
      2018-03-08 13:42:56.000
      75453
      313.91
      2018-03-08T15:47:10.491Z
      22492
      46863
      6098
      ...
      29794913.00
      2018-03-08T15:30:12.000Z
      105
      153.00
      46.00
      20.00
      8.00
      15.00
      43.00
      89.00

      6b7e492f18852c185ce675613126ecc1dc635ad8
      https://www.cnn.com/2018/03/09/us/florida-gov-...
      Florida Gov. Rick Scott signs gun bill
      2018-03-09 20:19:13.516
      2018-03-09 20:14:08.000
      63286
      191.69
      2018-03-09T22:18:14.865Z
      21148
      34852
      7286
      ...
      29803315.00
      2018-03-09T20:30:44.000Z
      105
      597.00
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      0bc531fbd47a9cbf242866e491b5912b92138391
      https://www.huffingtonpost.com/entry/florida-p...
      Exclusive: Florida Public School Teacher Has A...
      2018-03-03 17:04:20.369
      2018-03-03 17:02:22.984
      348983
      343.78
      2018-03-03T21:52:08.809Z
      98997
      200766
      49220
      ...
      9855456.00
      2018-03-03T17:30:30.000Z
      215
      1163.00
      50.00
      17.00
      9.00
      13.00
      39.00
      89.00

      e0c9b941ff0b4d0875f3da9f2faa3229e902bf7c
      https://www.huffingtonpost.com/entry/teens-kil...
      73 Teens Shot To Death In The 37 Days Since Th...
      2018-03-23 23:46:11.064
      2018-03-23 23:32:05.860
      205380
      153.51
      2018-03-24T14:10:06.582Z
      16975
      145972
      42433
      ...
      9872116.00
      2018-03-24T13:30:06.000Z
      215
      852.00
      50.00
      17.00
      9.00
      13.00
      39.00
      89.00

      4d6502bd4e3a0c3973855e284d48f12c1038cb52
      https://www.nytimes.com/interactive/2018/obitu...
      Women We Overlooked in 167 Years of New York T...
      2018-03-08 06:01:13.571
      2018-03-08 06:00:29.000
      175255
      937.13
      2018-03-08T14:03:12.334Z
      9797
      124907
      40551
      ...
      15386187.00
      2018-03-08T12:55:06.000Z
      120
      nan
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      ffc8f88a3fb0bf314c61ca1b09e8a66ffc2a75d2
      https://www.cnn.com/2018/03/15/us/miami-bridge...
      Pedestrian bridge collapses in Miami
      2018-03-15 18:25:24.501
      2018-03-15 18:22:27.000
      99543
      609.18
      2018-03-15T19:09:15.074Z
      19426
      65311
      14806
      ...
      29842448.00
      2018-03-15T18:29:06.000Z
      105
      43.00
      48.00
      20.00
      5.00
      15.00
      40.00
      88.00

      773d4d6e929b7214b91c99af479d3428243d8834
      https://www.huffingtonpost.com/entry/nra-march...
      NRA Takes Aim At ‘March For Our Lives' Rally, ...
      2018-03-25 12:59:13.260
      2018-03-25 12:49:22.000
      120306
      243.07
      2018-03-25T14:45:14.160Z
      39242
      69841
      11223
      ...
      9872381.00
      2018-03-25T14:00:10.000Z
      215
      456.00
      49.00
      17.00
      9.00
      13.00
      39.00
      88.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      589381e0e53a564fcbf2b7f7d686fb4ffb7de28f
      https://www.washingtonpost.com/national/10-thi...
      10 Things to Know for Today
      2018-03-14 10:19:12.084
      2018-03-14 10:07:15.000
      0
      0.00
      2018-03-14T10:31:10.552Z
      0
      0
      0
      ...
      nan
      NaN
      191
      393.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      374eded6396cc66aac5f262bbf861d651040f5f1
      http://www.bbc.co.uk/news/uk-scotland-tayside-...
      Man charged with assault after CCTV appeal
      2018-03-14 10:13:09.772
      2018-03-14 10:09:32.000
      1
      0.02
      2018-03-15T09:41:07.716Z
      0
      0
      1
      ...
      nan
      NaN
      96
      65.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      235c18ee0dcdfc6d5ba97a5606413a4977bf289a
      https://www.buzzfeed.com/hazelshearing/these-s...
      These Sexual Assault Victims Say Bristol Unive...
      2018-04-27 08:01:13.447
      2018-03-14 10:10:14.000
      1
      0.00
      2018-04-27T08:12:12.343Z
      0
      0
      1
      ...
      nan
      NaN
      147
      2604.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ff34bc30c66bd4dc84aaee72eb5b399915ee8473
      https://www.washingtonpost.com/business/techno...
      Edmunds: These 6 new SUVs are worth the wait
      2018-03-14 10:22:16.719
      2018-03-14 10:15:04.000
      0
      0.00
      2018-03-14T10:34:11.039Z
      0
      0
      0
      ...
      nan
      NaN
      191
      940.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      a3a7576ea53b61ef6e26e68c2f99315e9578de9e
      https://www.washingtonpost.com/sports/wizards/...
      Cavaliers’ Kyle Korver attends funeral for you...
      2018-03-27 14:13:19.120
      2018-03-27 14:10:16.000
      0
      0.00
      2018-03-28T13:41:16.514Z
      0
      0
      0
      ...
      nan
      NaN
      191
      155.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0d7d4e6c8e63fb95d01c0b23691656917bd9c329
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      Dave Hughes' crooked smile motivated Joe Hilde...
      2018-03-14 09:40:21.414
      2018-03-14 09:35:54.000
      1
      0.02
      2018-03-14T13:55:14.174Z
      0
      0
      1
      ...
      nan
      NaN
      158
      797.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f4b9ffbc56344633583386f7a3e2aa90541be0a9
      http://www.bbc.co.uk/news/live/uk-politics-par...
      AMs quiz education watchdog
      2018-03-14 09:37:12.559
      2018-03-14 09:33:40.000
      0
      0.00
      2018-03-14T09:48:13.624Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      af9129581e2869e0cc8c3be7426c5fdcd4879c8f
      https://www.washingtonpost.com/world/asia_paci...
      Nepal plane crash survivors say it’s a miracle...
      2018-03-14 09:37:16.154
      2018-03-14 09:30:21.000
      0
      0.00
      2018-03-14T09:48:13.627Z
      0
      0
      0
      ...
      nan
      NaN
      191
      142.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c9b58956ac88ddc224ad13eb4071079b2380bc1b
      https://edition.cnn.com/2016/04/13/sport/nba-f...
      NBA Finals: The greatest records
      2018-03-27 14:28:22.960
      2018-03-27 14:19:59.000
      0
      0.00
      2018-03-28T14:58:12.849Z
      0
      0
      0
      ...
      nan
      NaN
      105
      312.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      88d2f8ebf711b99ac5054cd9dc253913bcbb36fc
      https://www.washingtonpost.com/national/missin...
      Missing Georgia teenager’s remains found in sh...
      2018-03-14 08:40:10.520
      2018-03-14 08:30:09.000
      0
      0.00
      2018-03-14T10:52:10.127Z
      0
      0
      0
      ...
      nan
      NaN
      191
      249.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      116c491935236e1476a014d7ebca706125a22c07
      https://www.washingtonpost.com/entertainment/w...
      World Video Game Hall of Fame announces 2018 f...
      2018-03-27 14:31:13.686
      2018-03-27 14:19:12.000
      1
      0.02
      2018-03-27T19:46:09.073Z
      0
      0
      1
      ...
      nan
      NaN
      191
      150.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      3bee1f7ead7f873c4782aaef3e51a5e6af581888
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      Michelle Bridges flaunts her taut visage after...
      2018-03-14 08:40:13.393
      2018-03-14 08:37:02.000
      0
      0.00
      2018-03-14T08:52:10.822Z
      0
      0
      0
      ...
      nan
      NaN
      158
      447.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      cf30cb097a3db61a654544abb4602c4252c69cd0
      http://www.independent.co.uk/news/business/new...
      Balfour Beatty more than doubles profits as tu...
      2018-03-14 08:43:20.864
      2018-03-14 08:42:38.000
      1
      0.02
      2018-03-14T09:55:11.653Z
      0
      0
      1
      ...
      nan
      NaN
      386
      218.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ccb73b31b27ad487ff278e430a7d598bcc32d7a1
      https://www.washingtonpost.com/world/europe/it...
      Italy detains Egyptian imam accused of preachi...
      2018-03-27 14:31:11.680
      2018-03-27 14:16:14.000
      1
      0.02
      2018-03-27T15:43:13.263Z
      0
      0
      1
      ...
      nan
      NaN
      191
      149.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1c38f71e043b6cba1674a51d23db3c9f09ced0c4
      http://www.bbc.co.uk/news/uk-england-cambridge...
      Hinchingbrooke Country Park death: Arrest in S...
      2018-03-27 14:19:15.403
      2018-03-27 14:16:02.000
      1
      0.02
      2018-03-27T16:32:08.406Z
      0
      0
      1
      ...
      nan
      NaN
      96
      207.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b12039e3ab812c96a686904511beb8a2210a3702
      https://www.washingtonpost.com/business/ecb-he...
      ECB head: Higher inflation needed to end stimulus
      2018-03-14 08:49:15.380
      2018-03-14 08:44:17.000
      0
      0.00
      2018-03-14T11:02:10.420Z
      0
      0
      0
      ...
      nan
      NaN
      191
      146.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      68f4e9cb71bbd0b9140f03174aef3a75ac4026c0
      http://www.independent.co.uk/sport/tennis/indi...
      Caroline Wozniacki blows chance to overtake Si...
      2018-03-14 08:49:18.512
      2018-03-14 08:48:51.000
      1
      0.10
      2018-03-14T09:00:15.662Z
      0
      0
      1
      ...
      nan
      NaN
      386
      407.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e543cde9cf07954b432586b505e04440a5abc652
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      MAFS' Tracey Jewel shows off her breast augmen...
      2018-03-14 08:58:22.486
      2018-03-14 08:53:34.000
      0
      0.00
      2018-03-14T09:10:08.806Z
      0
      0
      0
      ...
      nan
      NaN
      158
      462.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f3cbcdfa23c724c3eb4cd6ca3bbe07260e0b8c2b
      https://www.washingtonpost.com/national/waymo-...
      Waymo to buy 20,000 Jaguars for robotic ride-h...
      2018-03-27 14:19:20.518
      2018-03-27 14:14:31.000
      0
      0.00
      2018-03-27T20:35:09.526Z
      0
      0
      0
      ...
      nan
      NaN
      191
      151.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1892c26277a06c54a16378d085f77d3b096c88fe
      https://www.washingtonpost.com/world/europe/ge...
      Germany’s parliament elects Angela Merkel for ...
      2018-03-14 09:10:11.394
      2018-03-14 08:57:10.000
      1
      0.10
      2018-03-14T09:21:09.832Z
      0
      0
      1
      ...
      nan
      NaN
      191
      31.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      38ddc94c67c6f0250fe52d853385f3d73f9e00ec
      https://www.washingtonpost.com/business/techno...
      France to sue Apple and Google for abusive pra...
      2018-03-14 09:07:08.901
      2018-03-14 08:58:10.000
      0
      0.00
      2018-03-14T09:18:11.748Z
      0
      0
      0
      ...
      nan
      NaN
      191
      135.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      adf35886949348793ba290bed12c3c3f4e40e42e
      https://www.nytimes.com/2018/03/14/learning/is...
      Is Childhood Today Too Sterilized?
      2018-03-14 09:01:26.994
      2018-03-14 09:00:02.000
      0
      0.00
      2018-03-14T09:13:04.797Z
      0
      0
      0
      ...
      nan
      NaN
      120
      210.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1860084c3babc992a5d7bf1b8a2e0d0a90a6b149
      http://www.bbc.co.uk/news/entertainment-arts-4...
      Music News LIVE: 14 March
      2018-03-14 09:04:09.735
      2018-03-14 09:03:18.000
      0
      0.00
      2018-03-14T09:15:08.370Z
      0
      0
      0
      ...
      nan
      NaN
      96
      58.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      dc489191d695f2ea21c5056bf52f04040cf63e92
      https://www.washingtonpost.com/national/parent...
      Parents to AP: Military ignored boy’s abuse of...
      2018-03-14 09:10:11.876
      2018-03-14 09:05:30.000
      0
      0.00
      2018-03-14T11:22:15.494Z
      0
      0
      0
      ...
      nan
      NaN
      191
      148.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      73dd45ea4708dd4d1899583d534158ff654cf9a4
      https://www.washingtonpost.com/national/how-ap...
      How AP’s investigation into childhood sex assa...
      2018-03-14 09:22:12.358
      2018-03-14 09:11:12.000
      0
      0.00
      2018-03-14T11:35:07.289Z
      0
      0
      0
      ...
      nan
      NaN
      191
      545.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c5e55d09fd84c72d1bdb4958a4915114f52b6341
      https://www.washingtonpost.com/business/spain-...
      Spain: Future of pensions is main priority, sa...
      2018-03-14 09:22:13.062
      2018-03-14 09:13:12.000
      0
      0.00
      2018-03-15T06:50:03.633Z
      0
      0
      0
      ...
      nan
      NaN
      191
      146.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      eac51a65487c71639689314af6a89923cff1c95a
      https://www.washingtonpost.com/business/recove...
      Recovering Cathay Pacific Flies Into a Threat ...
      2018-03-14 09:46:19.383
      2018-03-14 09:24:36.000
      0
      0.00
      2018-03-15T03:11:13.923Z
      0
      0
      0
      ...
      nan
      NaN
      191
      494.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      7c7311d19dbdc874fb1a1635e939523f833fd554
      https://www.washingtonpost.com/lifestyle/food/...
      COOKING ON DEADLINE: Leftover Brisket Barley Soup
      2018-03-27 14:19:22.409
      2018-03-27 14:11:13.000
      0
      0.00
      2018-03-28T11:46:10.965Z
      0
      0
      0
      ...
      nan
      NaN
      191
      586.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      504b31c73509ae9657d6592a4998caee9e839819
      http://www.bbc.co.uk/news/live/uk-northern-ire...
      RHI Live: OFGEM officials face inquiry grilling
      2018-03-14 09:31:12.851
      2018-03-14 09:27:36.000
      1
      0.02
      2018-03-14T16:48:06.143Z
      0
      0
      1
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6205205421bc3e02bb31acdc0656147578c677f4
      https://www.washingtonpost.com/national/charge...
      Charges: Man shot dead after trying to record ...
      2018-03-15 23:07:17.810
      2018-03-15 22:58:32.000
      0
      0.00
      2018-03-16T10:27:07.148Z
      0
      0
      0
      ...
      nan
      NaN
      191
      157.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

146484 rows × 26 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      42d519353c1ffa2303ad54ba8f6afe6ce0ed7b8f
      https://www.buzzfeed.com/albertsamaha/good-guy...
      She Thought She’d Shot A Burglar. Then She Rea...
      2018-03-23 11:31:19.815
      2018-03-22 11:28:05
      0
      0.00
      2018-03-23T11:42:14.365Z
      0
      0
      0
      ...
      2018-03-23T14:22:09.000Z
      147
      3003.00
      0.00
      19.00
      14.00
      11.00
      44.00
      44.00
      44.00

      512f3efdc422623da2139c574ddf9cf485f813fc
      https://www.buzzfeed.com/charliewarzel/the-mou...
      The Mountain West Is Experiencing A Second Gol...
      2018-03-12 00:19:22.840
      2018-03-07 16:18:51
      1
      0.09
      2018-03-12T00:31:05.739Z
      0
      0
      1
      ...
      2018-03-12T05:04:02.000Z
      147
      3252.00
      0.00
      19.00
      13.00
      11.00
      43.00
      43.00
      43.00

      719512ac4c86426735c50c9773d2294fef708519
      http://www.bbc.co.uk/news/world-europe-43552165
      Putin: Mall fire was criminal negligence
      2018-03-27 06:04:19.988
      2018-03-26 01:50:59
      8
      0.64
      2018-03-27T06:16:05.655Z
      0
      0
      8
      ...
      2018-03-27T07:44:41.000Z
      96
      707.00
      2.00
      20.00
      9.00
      15.00
      44.00
      46.00
      42.00

      5f76f987898ae0564ad528783cc4cb4a18369676
      https://www.buzzfeed.com/borzoudaragahi/baghda...
      After 15 Years Of War, Baghdad's Night Life Is...
      2018-05-05 16:19:11.986
      2018-03-30 13:39:53
      0
      0.00
      2018-05-05T16:31:10.611Z
      0
      0
      0
      ...
      2018-05-06T20:04:46.000Z
      147
      909.00
      0.00
      19.00
      12.00
      11.00
      42.00
      42.00
      42.00

      737c0b787113751672e74a807bb490576e2cf943
      https://www.buzzfeed.com/louispeitzman/theater...
      Theater Professionals Say Eliminating Sexual H...
      2018-03-31 13:19:09.974
      2018-03-29 20:27:00
      5
      0.40
      2018-03-31T13:30:07.064Z
      0
      0
      5
      ...
      2018-03-31T15:11:47.000Z
      147
      3656.00
      1.00
      19.00
      13.00
      11.00
      43.00
      44.00
      42.00

      515df476d5cfcd80af00d8a9cf6b9530a8def2a7
      http://www.bbc.co.uk/news/live/entertainment-a...
      Oscars 2018
      2018-03-02 19:25:13.001
      2018-03-02 19:22:31
      0
      0.00
      2018-03-03T11:51:08.649Z
      0
      0
      0
      ...
      2018-03-05T01:22:58.000Z
      96
      nan
      0.00
      20.00
      6.00
      15.00
      41.00
      41.00
      41.00

      044f712902ec6bc040d61e1e0b621c18dfe8b5ef
      https://www.buzzfeed.com/nidhisubbaraman/faceb...
      Here’s How A Cabbage Juice “Cult” With 58,000 ...
      2018-03-17 13:44:15.842
      2018-03-15 20:27:43
      6
      0.46
      2018-03-17T13:56:02.749Z
      0
      2
      4
      ...
      2018-03-17T18:36:18.000Z
      147
      4576.00
      2.00
      19.00
      13.00
      11.00
      43.00
      45.00
      41.00

      e44ea39a4c91c5d841a7ccf22e197ba1dd941010
      https://www.buzzfeed.com/kendalltaggart/secret...
      Secret NYPD Files: Hundreds Of Officers Commit...
      2018-03-05 11:01:26.053
      2018-03-02 19:27:57
      4
      0.28
      2018-03-05T11:13:08.100Z
      0
      2
      2
      ...
      2018-03-05T11:00:19.000Z
      147
      4197.00
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      bbcec1194a52059beffa7f0170bf6fbb8c7c0742
      https://www.buzzfeed.com/lamvo/oscars-script-d...
      We Compared Several Years Of Oscars Best Pictu...
      2018-03-03 00:13:18.340
      2018-03-02 23:10:08
      11
      0.07
      2018-03-03T05:30:11.035Z
      0
      0
      11
      ...
      2018-03-04T18:13:00.000Z
      147
      1368.00
      2.00
      19.00
      13.00
      11.00
      43.00
      45.00
      41.00

      07d6bb8bd3c9e6df61c120dcf554c7a4d243d77f
      https://www.buzzfeed.com/danvergano/heroine-op...
      This Town Is At The Center Of The US Opioid Cr...
      2018-03-03 15:09:14.487
      2018-03-02 15:20:47
      0
      0.00
      2018-03-03T15:21:15.398Z
      0
      0
      0
      ...
      2018-03-04T01:02:29.000Z
      147
      2772.00
      0.00
      17.00
      12.00
      11.00
      40.00
      40.00
      40.00

      31bd466fdfce3e9ae907f91a2178b6a12a460f1b
      https://www.buzzfeed.com/alisonwillmore/ready-...
      “Ready Player One” Is An Accidental Horror Mov...
      2018-04-01 15:54:15.397
      2018-03-31 16:59:26
      22
      0.37
      2018-04-01T17:05:18.157Z
      0
      15
      7
      ...
      2018-04-01T22:06:30.000Z
      147
      1408.00
      4.00
      19.00
      14.00
      11.00
      44.00
      48.00
      40.00

      a7ed7f09b6f3909a1bf6188c556291884c66ab06
      http://www.bbc.co.uk/news/live/uk-politics-435...
      English local elections 2018
      2018-03-26 11:04:06.928
      2018-03-26 11:01:52
      1
      0.02
      2018-03-27T03:26:12.038Z
      0
      0
      1
      ...
      2018-05-03T22:56:23.000Z
      96
      nan
      0.00
      20.00
      4.00
      15.00
      39.00
      39.00
      39.00

      22ebd1f10c19bdb54727cb59989d51e814428c53
      https://www.buzzfeed.com/danvergano/ale-shooti...
      Rich People Will Soon Be Able To Buy Fake Mete...
      2018-03-25 14:44:14.146
      2018-03-24 13:17:38
      2
      0.09
      2018-03-25T14:56:11.764Z
      0
      0
      2
      ...
      2018-03-25T15:44:26.000Z
      147
      845.00
      1.00
      18.00
      11.00
      11.00
      40.00
      41.00
      39.00

      d573ec8e1d6a43be40ca46acf2ee249c1398d745
      https://www.buzzfeed.com/hannahalothman/this-i...
      This Is What It's Like To Be Put To Sleep So Y...
      2018-03-17 11:52:16.580
      2018-03-16 19:12:33
      43
      0.25
      2018-03-17T13:04:14.124Z
      6
      19
      18
      ...
      2018-03-18T07:44:10.000Z
      147
      3579.00
      6.00
      19.00
      14.00
      11.00
      44.00
      50.00
      38.00

      180467647e6639290cf5684d36823b38f1f0a2f5
      https://www.buzzfeed.com/albertsamaha/filipino...
      I Followed My Uncle’s Legend To Italy, And Fou...
      2018-03-31 15:04:13.911
      2018-03-30 16:29:47
      13
      0.08
      2018-03-31T16:16:10.694Z
      0
      3
      10
      ...
      2018-03-31T21:13:09.000Z
      147
      6252.00
      3.00
      19.00
      11.00
      11.00
      41.00
      44.00
      38.00

      041fcd233fce6a0a01754aefa84d6928f91bd695
      https://www.buzzfeed.com/sylviaobell/black-act...
      Here’s What Black Actors Say They Need From Ti...
      2018-03-26 23:59:13.622
      2018-03-26 21:20:45
      40
      0.20
      2018-03-27T01:11:18.458Z
      4
      25
      11
      ...
      2018-03-28T03:30:43.000Z
      147
      1534.00
      6.00
      19.00
      14.00
      11.00
      44.00
      50.00
      38.00

      e7d299569233947afddaeeddacf390842a30f733
      https://www.buzzfeed.com/charliewarzel/silicon...
      Silicon Valley Is Having An Existential Crisis...
      2018-03-13 14:49:15.190
      2018-03-13 03:53:47
      29
      0.15
      2018-03-14T03:10:11.630Z
      0
      11
      18
      ...
      2018-03-14T02:07:22.000Z
      147
      1188.00
      5.00
      19.00
      12.00
      11.00
      42.00
      47.00
      37.00

      abecd354933cbaca20fc43f619a379b682fd1d01
      https://www.buzzfeed.com/rahawahaile/selma-civ...
      Hiking The Selma Montgomery Trail
      2018-04-01 13:09:10.601
      2018-03-30 14:23:54
      2
      0.20
      2018-04-01T13:20:07.352Z
      0
      0
      2
      ...
      2018-04-01T18:11:32.000Z
      147
      3269.00
      1.00
      19.00
      7.00
      11.00
      37.00
      38.00
      36.00

      fb80249318c8be37a4d870ea4ba997d31263b3ab
      https://www.buzzfeed.com/tylerkingkade/what-it...
      What It’s Like For School Shooting Survivors T...
      2018-03-20 13:04:24.495
      2018-03-19 23:47:29
      39
      0.27
      2018-03-20T17:18:12.228Z
      1
      22
      16
      ...
      2018-03-20T16:24:07.000Z
      147
      2188.00
      6.00
      19.00
      12.00
      11.00
      42.00
      48.00
      36.00

      49d1f5a72ccfe6e37cba6657efd46528bf2b2cc1
      https://www.buzzfeed.com/alexkantrowitz/facebo...
      Facebook's Political Nightmare Is About To Get...
      2018-03-06 14:58:17.643
      2018-03-05 23:06:47
      59
      0.32
      2018-03-06T19:13:08.185Z
      6
      11
      42
      ...
      2018-03-06T18:36:18.000Z
      147
      1841.00
      7.00
      19.00
      13.00
      11.00
      43.00
      50.00
      36.00

      50fec0d08992fc98fb0111ef4aff5efadb107b93
      https://www.buzzfeed.com/tamerragriffin/miguna...
      This Lawyer Was Born In Kenya But The Governme...
      2018-03-27 19:40:20.520
      2018-03-27 19:17:41
      55
      0.20
      2018-03-27T19:51:12.488Z
      0
      25
      30
      ...
      2018-03-27T20:25:26.000Z
      147
      858.00
      7.00
      19.00
      12.00
      11.00
      42.00
      49.00
      35.00

      b94fd95755d5c76accf8df4202c931e5bddbfe85
      https://www.buzzfeed.com/gabrielsanchez/24-of-...
      24 Of The Most Iconic Pictures From Oscars His...
      2018-03-04 17:46:29.898
      2018-03-01 20:44:40
      2
      0.18
      2018-03-04T17:58:10.670Z
      0
      0
      2
      ...
      2018-03-05T00:12:00.000Z
      147
      nan
      1.00
      19.00
      6.00
      11.00
      36.00
      37.00
      35.00

      b11cb512de184a6cf579f3726a1ddcdc07a172ec
      https://www.buzzfeed.com/mattberman/david-shul...
      Trump's Veterans Affairs Secretary David Shulk...
      2018-03-28 21:43:15.112
      2018-03-28 21:39:27
      71
      0.30
      2018-03-28T21:54:14.461Z
      16
      30
      25
      ...
      2018-03-29T02:10:56.000Z
      147
      293.00
      8.00
      19.00
      13.00
      11.00
      43.00
      51.00
      35.00

      4afb8c4ac700d5419654f8445622ec01cb454b8a
      https://www.buzzfeed.com/adambvary/5-reasons-w...
      5 Reasons Why This Year's Best Picture Is The ...
      2018-03-03 16:49:11.480
      2018-03-03 00:13:26
      58
      0.29
      2018-03-03T23:20:43.186Z
      1
      48
      9
      ...
      2018-03-03T22:02:54.000Z
      147
      1124.00
      7.00
      19.00
      12.00
      11.00
      42.00
      49.00
      35.00

      63896d10a68d35756e94dfe8076a28fa2937c098
      https://www.buzzfeed.com/hazelshearing/russia-...
      Russia Will Expel 23 British Diplomats As Tens...
      2018-03-17 08:49:19.698
      2018-03-17 08:49:02
      78
      0.48
      2018-03-17T18:07:02.743Z
      7
      46
      25
      ...
      2018-03-17T17:15:14.000Z
      147
      229.00
      8.00
      19.00
      12.00
      11.00
      42.00
      50.00
      34.00

25 rows × 27 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      50274b99fd61c30213190e1f47e61c06b0fe8444
      https://www.cnn.com/2018/03/24/us/naomi-wadler...
      11-year-old speaks up for the black girls whos...
      2018-03-24 17:58:21.797
      2018-03-24 17:54:04.000
      220229
      229.14
      2018-03-24T22:19:05.478Z
      10005
      189918
      20306
      ...
      NaN
      105
      400.00
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      220bc1e5ff5023dabe5ff4aab0cf50d422d64a6d
      https://www.nytimes.com/2018/03/13/us/ice-spok...
      ICE Spokesman Resigns, Saying He Could No Long...
      2018-03-13 15:28:06.745
      2018-03-13 15:26:00.000
      490246
      592.80
      2018-03-13T17:12:05.255Z
      43354
      363565
      83327
      ...
      NaN
      120
      659.00
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      5217dabb2788a25fb6ab6cf6903a5a565fbfdd3b
      https://www.cnn.com/2018/03/14/us/nikolas-cruz...
      Nikolas Cruz: Not guilty plea entered on Parkl...
      2018-03-14 18:16:14.627
      2018-03-14 18:10:07.000
      319745
      312.88
      2018-03-17T13:07:04.378Z
      106801
      141537
      71407
      ...
      NaN
      105
      327.00
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      665b158f2d4382c825e9c95181e19a880d0ea444
      https://www.huffingtonpost.com/entry/man-award...
      Innocent Man Awarded $1 Million After Spending...
      2018-03-22 23:04:19.286
      2018-03-22 22:45:08.830
      166773
      224.08
      2018-03-26T01:05:03.403Z
      45996
      99141
      21636
      ...
      NaN
      215
      666.00
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      4e0f5018cbbd109c03fb7b5c2b115b6f83787461
      http://www.foxnews.com/lifestyle/2018/03/01/ca...
      California has worst 'quality of life' in US, ...
      2018-03-01 06:04:37.010
      2018-03-01 05:50:40.000
      158939
      376.80
      2018-03-02T04:07:09.673Z
      46730
      88830
      23379
      ...
      NaN
      285
      262.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      811b11341fd9cf3570bfc99a75c00d80380e741f
      https://www.washingtonpost.com/news/morning-mi...
      High school students are marching 50 miles to ...
      2018-03-26 10:34:13.083
      2018-03-26 10:28:00.000
      228839
      725.92
      2018-03-27T13:03:05.545Z
      19054
      188846
      20939
      ...
      NaN
      191
      929.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      49c7c48264287c418b4f325a27cafa6f66f2c05c
      https://www.nytimes.com/2018/03/14/us/politics...
      Democrat’s Strong Performance in Pennsylvania ...
      2018-03-14 09:49:14.808
      2018-03-14 09:48:15.000
      130754
      396.00
      2018-03-14T13:00:10.561Z
      13571
      109852
      7331
      ...
      NaN
      120
      839.00
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      8fc7a94c76f872356ed483b7ac17adac5512eeaf
      http://www.dailymail.co.uk/health/article-5559...
      NHS BANS over-the-counter remedies on prescrip...
      2018-03-29 15:31:23.764
      2018-03-29 15:30:11.000
      192930
      2579.73
      2018-03-29T21:07:11.238Z
      50619
      113504
      28807
      ...
      NaN
      158
      844.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      3cefbaccf74c67435f2d9e3ce03a833a6a031dd9
      http://www.foxnews.com/politics/2018/03/16/for...
      Former FBI Deputy Director Andrew McCabe fired
      2018-03-17 02:09:08.956
      2018-03-17 02:05:10.000
      153446
      723.92
      2018-03-17T02:31:08.779Z
      34418
      101832
      17196
      ...
      NaN
      285
      63.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      2029aee2f236d917e866b77fff279d24bf704f53
      https://www.rt.com/news/421292-russia-uk-skrip...
      Russia-UK standoff over Skripal case: Ex-Londo...
      2018-03-14 15:10:29.252
      2018-03-14 15:09:53.000
      181999
      2973.62
      2018-03-14T18:25:05.286Z
      25191
      11859
      144949
      ...
      NaN
      365
      32.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      b81059e9e80c976fb540b0e2072e2e0de13d8237
      https://www.theguardian.com/world/2018/mar/15/...
      Marielle Franco, councillor and police critic,...
      2018-03-15 05:25:08.366
      2018-03-15 05:22:02.000
      144482
      148.20
      2018-03-15T13:30:07.659Z
      7767
      115115
      21600
      ...
      NaN
      142
      307.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      4b263c68cdef616305925dbc652ca78ea2893928
      http://www.foxnews.com/politics/2018/03/09/tru...
      Trump pardons former Navy sailor imprisoned fo...
      2018-03-09 19:44:16.711
      2018-03-09 19:35:00.000
      171069
      698.29
      2018-03-10T14:06:13.319Z
      22645
      129024
      19400
      ...
      NaN
      285
      34.00
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      c3660f6b3c146aaf04faf55e0c0bf241e29e8989
      https://www.washingtonpost.com/news/morning-mi...
      Texas woman sentenced to 5 years in prison for...
      2018-03-30 10:14:11.434
      2018-03-30 10:01:00.000
      174415
      129.75
      2018-03-30T21:35:06.218Z
      37119
      108128
      29168
      ...
      NaN
      191
      886.00
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      c47143c6da051a69409d08f1b0a625b4e98459aa
      https://www.npr.org/sections/thetwo-way/2018/0...
      Man Who Bid For Border Wall Contract Charged I...
      2018-03-14 22:29:12.121
      2018-03-14 21:44:29.000
      118176
      48.82
      2018-03-15T23:34:10.721Z
      15836
      76427
      25913
      ...
      NaN
      594
      790.00
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      06a46e015b59b2b7a4e9f3bbd178631499f84d0b
      https://www.indy100.com/article/donald-trump-m...
      Scientists discovered a moth with golden hair ...
      2018-03-27 15:24:16.483
      2018-03-27 15:15:44.000
      108340
      8.62
      2018-03-28T02:43:16.019Z
      10078
      84470
      13792
      ...
      NaN
      5014
      233.00
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      ad709da694f5f5cfdabb47c2f7f7e7554ae12c7c
      http://www.foxnews.com/politics/2018/03/13/hil...
      Hillary Clinton blames 'pressure' from men for...
      2018-03-13 12:24:18.762
      2018-03-13 12:20:50.000
      124599
      260.88
      2018-03-13T16:50:09.695Z
      51816
      61737
      11046
      ...
      NaN
      285
      404.00
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      29a501b1c0a690e5cd174ab7e6d13b11122a93b8
      https://www.cnn.com/2018/03/28/politics/alton-...
      White House calls police shootings of black me...
      2018-03-29 15:07:20.076
      2018-03-29 14:55:23.000
      128890
      289.15
      2018-03-29T17:06:13.811Z
      30686
      73449
      24755
      ...
      NaN
      105
      454.00
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      0d311bd9bc88f483d8cd59c3bdd0f72f6fbe7d75
      https://www.nbcnews.com/feature/nbc-out/oregon...
      Oregon Supreme Court suspends judge who refuse...
      2018-03-19 03:37:20.964
      2018-03-19 03:34:39.000
      96376
      145.45
      2018-03-19T17:08:06.367Z
      14772
      72125
      9479
      ...
      NaN
      826
      nan
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      418cc03dd36ea891048b13fe7d67ac936ded0b4e
      https://www.nbcnews.com/storyline/paris-terror...
      Eagles of Death Metal frontman calls Parkland ...
      2018-03-26 16:10:08.842
      2018-03-26 16:08:53.000
      84246
      179.64
      2018-03-27T16:54:13.990Z
      23308
      55832
      5106
      ...
      NaN
      826
      380.00
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      7a45be3e4cae44045719b087b12c263686a10f54
      http://www.foxnews.com/opinion/2018/03/07/tamm...
      Tammy Bruce: California chaos -- Unchallenged ...
      2018-03-07 15:29:11.736
      2018-03-07 15:22:08.000
      87396
      356.16
      2018-03-08T22:07:12.830Z
      10953
      63055
      13388
      ...
      NaN
      285
      936.00
      47.00
      0.00
      2.00
      0.00
      2.00
      49.00
      -45.00

      d4ea734b1b76e25df1d75bf955cfee3415c67d2e
      http://yournewswire.com/rockefeller-big-pharma...
      How Rockefeller Founded Big Pharma And Waged W...
      2018-03-27 17:29:26.929
      2018-03-27 16:39:37.000
      206259
      58.82
      2018-03-29T13:51:13.201Z
      34034
      114928
      57297
      ...
      2018-04-05T08:20:00.000Z
      22568
      742.00
      50.00
      0.00
      1.00
      4.00
      5.00
      55.00
      -45.00

      8e677451fa0eb8ac53c9431bd3c780e24c15313a
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      Married At First Sight's Ash now dating Justin
      2018-03-21 21:19:16.396
      2018-03-21 21:16:49.000
      74126
      85.91
      2018-03-22T03:09:07.965Z
      57990
      14690
      1446
      ...
      NaN
      158
      101.00
      46.00
      0.00
      1.00
      0.00
      1.00
      47.00
      -45.00

      c7ff5d9a2aacb0bbb6cf09c2137416a2b0b5ea8d
      http://yournewswire.com/london-churches-mosques/
      London Closes 500 Churches; Opens 423 New Mosques
      2018-03-05 17:49:22.185
      2018-03-05 17:34:50.000
      265318
      69.38
      2018-03-06T21:13:09.677Z
      67888
      147744
      49686
      ...
      2018-03-06T22:43:36.000Z
      22568
      1064.00
      50.00
      0.00
      1.00
      4.00
      5.00
      55.00
      -45.00

      9dba841e93a95329b60a2d259aa5cc088fff53af
      https://www.huffingtonpost.com/entry/justice-s...
      Justice System Failed These 23 Black Victims
      2018-03-28 21:04:22.008
      2018-03-28 20:55:15.999
      194436
      51.85
      2018-03-30T14:59:11.295Z
      14885
      123614
      55937
      ...
      NaN
      215
      nan
      50.00
      0.00
      5.00
      0.00
      5.00
      55.00
      -45.00

      bc9d99af54ec89e3e286b5ed111c2020a51d91c9
      https://www.nytimes.com/2018/03/13/opinion/nra...
      Dear National Rifle Association: We Won’t Let ...
      2018-03-13 23:52:04.852
      2018-03-13 23:49:41.000
      93521
      159.67
      2018-03-14T12:54:10.917Z
      6482
      75593
      11446
      ...
      NaN
      120
      611.00
      47.00
      0.00
      3.00
      0.00
      3.00
      50.00
      -44.00

25 rows × 27 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_" + date_filename + ".csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}

```python

```

```python

```