# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
date_filename = "2018-04-01_2018-04-30"
```

```python
data = pd.read_csv("articles_" + date_filename + ".csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      eb8e9bb4465037557f2b72a02463ef2240048a0a
      https://www.thetimes.co.uk/article/mccann-pres...
      McCann presence helps Dundee secure valuable p...
      2018-04-01 17:09:07.450
      2018-04-01
      0
      0.000000
      2018-04-01T17:20:06.737Z
      0
      0
      0
      The Times
      thetimes_co_uk
      0
      350
      392.0
      False
      NaN
      NaN
      6435
      125.0

      58f51d1d8e99d7c9909b84201706b10411632331
      https://www.thetimes.co.uk/article/anthony-jos...
      Anthony Joshua showed he has the maturity to r...
      2018-04-01 17:09:08.170
      2018-04-01
      0
      0.000000
      2018-04-01T17:20:06.739Z
      0
      0
      0
      The Times
      thetimes_co_uk
      0
      350
      392.0
      False
      NaN
      NaN
      6435
      125.0

      0b1a1a27c7e6f3aff21586ae2d447b4f1944e597
      https://www.thetimes.co.uk/article/saracens-eu...
      Saracens’ European reign ended by familiar foes
      2018-04-01 17:09:08.413
      2018-04-01
      0
      0.000000
      2018-04-01T17:20:06.740Z
      0
      0
      0
      The Times
      thetimes_co_uk
      0
      350
      392.0
      False
      NaN
      NaN
      6435
      125.0

      3f9169b7e2d845613721fe006eba63b9bd8f6ffb
      https://www.thetimes.co.uk/article/labour-part...
      Labour Party donor Sir David Garrard quits par...
      2018-04-01 11:09:13.260
      2018-04-01
      1
      0.016429
      2018-04-01T12:21:04.387Z
      0
      0
      1
      The Times
      thetimes_co_uk
      0
      709
      381.0
      False
      NaN
      NaN
      6435
      126.0

      2a743b32a30e803f38b966ef27e1cbf9f5ccd7ba
      https://www.thetimes.co.uk/article/jordan-pete...
      Jordan Peterson: the bestselling author and cl...
      2018-04-01 20:33:13.945
      2018-04-01
      293
      1.820941
      2018-04-01T20:45:07.257Z
      86
      144
      63
      The Times
      thetimes_co_uk
      0
      0
      NaN
      True
      763761.0
      2018-04-01T20:30:02.000Z
      6435
      125.0

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    138275.00
    mean       1301.27
    std       10881.11
    min           0.00
    50%          25.00
    75%         236.00
    90%        1467.00
    95%        4082.00
    99%       24069.16
    99.5%     42726.87
    99.9%    127818.48
    max     1282889.00
    Name: fb_engagements, dtype: float64

There's 2 articles with more than 1 million engagements this month.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      a2c14a5be2fd2120a995789a59f63751734d8312
      http://www.foxnews.com/entertainment/2018/04/1...
      'Full Metal Jacket' actor R. Lee Ermey dies at...
      2018-04-15 23:23:18.706
      2018-04-15 23:20:18
      1088522
      3701.16
      2018-04-15T23:55:11.721Z
      111839
      781170
      195513
      Fox News
      foxnews_com
      0
      2204
      154.00
      True
      16331201.00
      2018-04-15T23:21:55.000Z
      285
      359.00

      1c67b7d5633798a8217ff854d6e4854d64ac1b78
      https://www.npr.org/sections/ed/2018/04/25/605...
      High-Paying Trade Jobs Sit Empty, While High S...
      2018-04-25 21:01:27.348
      2018-04-25 20:00:00
      1282889
      833.15
      2018-04-26T16:49:11.642Z
      216494
      829474
      236921
      NPR
      npr_org
      0
      965
      30.00
      True
      6298047.00
      2018-04-26T04:22:00.000Z
      594
      1673.00

```python
data.fb_engagements.mode()
```

    0    0
    dtype: int64

*november* Going back to the enagement counts, we see the mean is 1,117, mode is zero, median is 24, 90th percentile is 1,453, 99th percentile is 21,166, 99.5th percentile is 33,982. The standard deviation is 8,083, significantly higher than the mean, so this is not a normal distribution.

*december* Going back to the enagement counts, we see the mean is 1,106, mode is zero, median is 24, 90th percentile is 1,545, 99th percentile is 20,228, 99.5th percentile is 32,446. The standard deviation is 9,852, significantly higher than the mean, so this is not a normal distribution.

*january 2018* Going back to the enagement counts, we see the mean is 1,108, mode is zero, median is 26, 90th percentile is 1,621, 99th percentile is 19,918, 99.5th percentile is 32,935. The standard deviation is 8,278, significantly higher than the mean, so this is not a normal distribution.

*february 2018* Going back to the enagement counts, we see the mean is 1,237, mode is zero, median is 25, 90th percentile is 1,453, 99th percentile is 23,172, 99.5th percentile is 38,692. The standard deviation is 11,019, significantly higher than the mean, so this is not a normal distribution.

*march 2018* Going back to the enagement counts, we see the mean is 1,519, mode is zero, median is 25, 90th percentile is 1,597, 99th percentile is 27,468, 99.5th percentile is 51,204. The standard deviation is 15,132, significantly higher than the mean, so this is not a normal distribution.

*april 2018* Going back to the enagement counts, we see the mean is 1,301, mode is zero, median is 25, 90th percentile is 1,467, 99th percentile is 24,069, 99.5th percentile is 42,726. The standard deviation is 10,881, significantly higher than the mean, so this is not a normal distribution.

Key publishers stats

```python
data.groupby("publisher_id").agg({'url': 'count', 'fb_engagements': ['sum', 'median', 'mean']})
```

      url
      fb_engagements

      count
      sum
      median
      mean

      publisher_id

      anotherangryvoice_blogspot_co_uk
      37
      182291
      2207.00
      4926.78

      bbc_co_uk
      12059
      10830500
      26.00
      898.13

      breitbart_com
      2589
      7211253
      145.00
      2785.34

      brexitcentral_com
      42
      39391
      398.50
      937.88

      buzzfeed_com
      1290
      2463775
      191.50
      1909.90

      cnn_com
      3996
      24266804
      516.50
      6072.77

      dailymail_co_uk
      24494
      17875059
      18.00
      729.77

      economist_com
      504
      245341
      32.50
      486.79

      evolvepolitics_com
      65
      255755
      1553.00
      3934.69

      foxnews_com
      5009
      22811621
      103.00
      4554.13

      ft_com
      4416
      270005
      4.00
      61.14

      huffingtonpost_com
      2250
      10943965
      785.00
      4863.98

      independent_co_uk
      5290
      3943093
      35.50
      745.39

      indy100_com
      395
      257944
      111.00
      653.02

      lemonde_fr
      3649
      2023160
      71.00
      554.44

      libdemvoice_org
      141
      2837
      13.00
      20.12

      mirror_co_uk
      9204
      5571882
      32.00
      605.38

      nbcnews_com
      1878
      7042840
      420.50
      3750.18

      newstatesman_com
      439
      97708
      19.00
      222.57

      npr_org
      1824
      10768007
      346.00
      5903.51

      nytimes_com
      4627
      18669054
      209.00
      4034.81

      order-order_com
      257
      81152
      150.00
      315.77

      propublica_org
      51
      52191
      346.00
      1023.35

      reuters_com
      6179
      3171709
      17.00
      513.30

      rt_com
      2302
      3609677
      376.50
      1568.06

      skwawkbox_org
      121
      115099
      545.00
      951.23

      telegraph_co_uk
      6009
      1703445
      16.00
      283.48

      thecanary_co
      276
      345275
      645.00
      1251.00

      theguardian_com
      7989
      8911855
      122.00
      1115.52

      thetimes_co_uk
      8887
      295032
      1.00
      33.20

      washingtonpost_com
      21307
      13599144
      0.00
      638.25

      westmonster_com
      302
      497503
      42.50
      1647.36

      yournewswire_com
      397
      1778898
      45.00
      4480.85

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
eng = data.fb_engagements[(data.fb_engagements < 5000)]
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(15,7))
plt.hist(eng, bins=50)
plt.title("Article count by engagements")
plt.axvline(median, label=f'Median ({median:,.0f})', color='green')
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(ninety, linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(ninetyfive, linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
# plt.axvline(ninetynine, linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   113072.00
    mean         4.98
    std          1.76
    min          3.26
    25%          3.53
    50%          4.38
    75%          5.99
    max         11.76
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   113072.00
    mean        10.62
    std         10.38
    min          0.00
    25%          2.00
    50%          7.00
    75%         17.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   138275.00
    mean         9.26
    std         90.28
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        284.00
    99.5%      580.00
    99.9%     1199.73
    max       9840.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   3928.00
    mean     326.03
    std      428.55
    min        4.00
    25%       84.00
    50%      184.00
    75%      424.00
    90%      804.30
    95%     1074.00
    99%     1599.73
    99.5%   1832.37
    99.9%   4428.57
    max     9840.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    lemonde_fr                             618
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      37.00
      5.33
      0.00
      5.33
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      105.00
      12.97
      1.31
      2.19
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      205.00
      8.18
      1.63
      0.72
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      35.00
      4.49
      0.65
      0.77
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      321.00
      11.74
      1.24
      1.00
      12.02
      12.02
      12.02
      12.02

      cnn_com
      210.00
      12.29
      2.00
      2.15
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      154.00
      11.31
      1.89
      0.99
      11.85
      11.85
      11.85
      11.85

      economist_com
      48.00
      7.65
      1.23
      1.33
      7.99
      7.99
      7.99
      7.99

      foxnews_com
      114.00
      10.50
      0.62
      6.90
      10.61
      10.61
      10.61
      10.61

      ft_com
      98.00
      7.59
      1.70
      0.68
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      148.00
      11.13
      0.40
      7.26
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      134.00
      9.55
      1.89
      0.84
      10.07
      10.07
      10.07
      10.07

      indy100_com
      146.00
      5.82
      1.83
      0.59
      4.61
      7.04
      7.04
      7.04

      lemonde_fr
      185.00
      8.18
      2.51
      0.62
      9.34
      9.34
      9.34
      9.34

      libdemvoice_org
      133.00
      4.55
      0.58
      1.49
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      3.00
      0.97
      0.49
      0.61
      0.69
      0.76
      1.14
      1.52

      nbcnews_com
      127.00
      8.63
      1.33
      0.74
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      79.00
      6.02
      1.10
      1.48
      6.35
      6.35
      6.35
      6.35

      npr_org
      150.00
      9.23
      0.87
      3.13
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      63.00
      12.39
      0.62
      9.19
      12.53
      12.53
      12.53
      12.53

      order-order_com
      255.00
      4.50
      1.55
      0.48
      3.27
      5.29
      5.78
      5.78

      propublica_org
      16.00
      6.20
      0.00
      6.20
      6.20
      6.20
      6.20
      6.20

      reuters_com
      96.00
      9.22
      1.78
      0.81
      9.66
      9.66
      9.66
      9.66

      rt_com
      138.00
      9.36
      2.12
      0.68
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      121.00
      4.73
      0.81
      1.17
      5.03
      5.03
      5.03
      5.03

      telegraph_co_uk
      98.00
      9.87
      1.37
      0.85
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      260.00
      4.82
      1.70
      0.93
      3.52
      6.11
      6.21
      6.21

      theguardian_com
      152.00
      11.60
      1.95
      0.81
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      57.00
      6.84
      0.00
      6.84
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      88.00
      11.06
      1.92
      0.76
      11.42
      11.42
      11.42
      11.42

      yournewswire_com
      152.00
      5.48
      1.11
      1.00
      5.76
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.4812571144098754, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     134348
    1.00         46
    2.00         50
    3.00         60
    4.00         73
    5.00         64
    6.00         78
    7.00        216
    8.00        194
    9.00        272
    10.00       234
    11.00       164
    12.00        67
    13.00       278
    14.00       276
    15.00       241
    16.00       336
    17.00       265
    18.00       226
    19.00       437
    20.00       350
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      13.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  0.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    lemonde_fr                         14.00
    libdemvoice_org                     7.00
    mirror_co_uk                        2.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     0.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    2.4500000000000002

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     69443
    2.00      6888
    3.00      4403
    4.00      4357
    5.00       843
    6.00       606
    7.00       537
    8.00       443
    9.00       876
    10.00      323
    11.00      372
    12.00      145
    13.00       39
    14.00       27
    15.00        9
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk      37.00
    bbc_co_uk                          13580.00
    breitbart_com                       2494.00
    brexitcentral_com                     42.00
    buzzfeed_com                        8720.00
    cnn_com                            12377.00
    dailymail_co_uk                    14392.00
    economist_com                        244.00
    evolvepolitics_com                    61.00
    foxnews_com                         6264.00
    ft_com                              2983.00
    huffingtonpost_com                  7806.00
    independent_co_uk                   3799.00
    indy100_com                          392.00
    lemonde_fr                          3645.00
    libdemvoice_org                      140.00
    mirror_co_uk                        8722.00
    nbcnews_com                         1659.00
    newstatesman_com                     432.00
    npr_org                             2754.00
    nytimes_com                         9206.00
    order-order_com                      257.00
    propublica_org                        50.00
    reuters_com                         7125.00
    rt_com                              4641.00
    skwawkbox_org                        121.00
    telegraph_co_uk                     4279.00
    thecanary_co                         274.00
    theguardian_com                    11959.00
    thetimes_co_uk                      8866.00
    washingtonpost_com                  8982.00
    westmonster_com                      287.00
    yournewswire_com                     386.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    eb8e9bb4465037557f2b72a02463ef2240048a0a   1.00
    58f51d1d8e99d7c9909b84201706b10411632331   1.00
    0b1a1a27c7e6f3aff21586ae2d447b4f1944e597   1.00
    3f9169b7e2d845613721fe006eba63b9bd8f6ffb   1.00
    2a743b32a30e803f38b966ef27e1cbf9f5ccd7ba   0.00
    1ebabeaa2db7cc62305cae75986d6d108165c7f2   1.00
    772fc50f1134a566dc5e001423994070b8c58a93   1.00
    4a7806d2ce55afcb6faf9b968afddf9081e6fc3b   1.00
    14f2c2f6943742da6e4b3bd6e55c540f84d14f89   1.00
    731c40614f7299ff148188cb4935ac42f78504e2   1.00
    9630301c907b8df80e414e6cf4b91db234fd49e8   1.00
    164ebf483c3d37fce9663e48bc38a6d8f193c948   1.00
    0aa736272acb499eb78d5ef0f82e95c4e1306e66   1.00
    408e7e5138bfe286f3a3ace50e288b73ec7d70e8   1.00
    f975e668a58e5dd418b2d27115bf355b0e2c8f26   1.00
    d624b8ac09af1d50be8fbc87b7a3ef344cdf94fb   1.00
    8092f3fbd0b033689e9d2d6c435d9b7018d54ed5   1.00
    e5bb231bb892da4e3b4d949897fcbc97d2265a93   1.00
    af3f789eacbc6f448f7c3d4406e6ee44b7b0b2b4   1.00
    d75b82fea9ec27dc8b4b7dbb822c213270b6e22f   1.00
    e8898b366e9e8cead7377d91ea84142845b8c411   1.00
    2b0e0562b22b3dbc89a555d4255e9c0d50cfbc7a   1.00
    84c47ecb160d6eb722b3d9a676172a83c83e64c1   1.00
    0f6160fa161885292958f8939319b06e96eb27f6   1.00
    b656446d299b94eef2d9a520cbbd7f5c03242b66   0.00
    174bff4b69cf24f94f8ffef87f3bce022a799d26   1.00
    ce462dd900e5b9112e3e504939c9e85fca009759   1.00
    82370cce427a53970d07ea97961cc8bd83df2bce   0.00
    a89da1db2fceae2c0172f6b9943d11f8f1807ec3   7.00
    c141350d3c336f0b1575c3902072a8daea26c4e2   1.00
                                               ...
    2646b9fb92e1507df71cd22ba19522bf4c0ddd51   0.00
    4c179dcf62c5162ac662580e0a4021d7bd49ecf1   0.00
    2ac60e5ac1a705ef696ffa7b13061db9e8c53de2   1.00
    573aaeb31b9bc173835c20abb30b6a7bac16b6fe   1.00
    19cffd6ccddfa7c0a84d15002f8a2b68f57fcec2   1.00
    f458dd8bbe2486580f244aee5d505ecc683c2170   3.00
    7fc403dc2a7605e0435326b7d52f45771e3f9401   0.00
    e4b21080e26a3f50b17984ad19baf8449e598d63   1.00
    ba12871d0bf8d8f0ffdb8a9726999cf6400aa1f0   1.00
    b8e032c9aee9e4128325e1da3569e41fea9f1d46   1.00
    749a5ac507b521df52a24d39fadb11edb16bc3d6   3.00
    3f95cc42dc55f01fa4c9d6022f0e6ea5ae21dd71   1.00
    1ea319842636cc047ef3558d643892a482b4fe2e   0.00
    07408efb98ea300ac74c1239aad537ad045e5833   1.00
    acc344aea665093d76a2e41e26e4f5aaf6e1b9cb   7.00
    ceac8ec3c65784c78710353bc4b045db1261fa40   0.00
    fa23176a6667a466654205a8b2d6eeea1147a8c7   0.00
    5d8703f945464f3cf37e8c3a90bd93baf49abdfc   0.00
    6ca69dc36f67723d91c5c02a4926569799c48db2   8.00
    f0e0390e80c6d660b0e371e7b73e5176f3406787   1.00
    16d3197f7e2389632ae6d43705157ee50d65831d   1.00
    ab951d17a3ffd353feb36859793a91d81140324c   0.00
    ab0fb3586222b04d4dcd9c979142a4160b87efc0   1.00
    a425d96edc27f152fb8252a26a7e19c9348f893f   0.00
    727a72fa4cbb7f2fb531daf2a7889345285d656e   1.00
    7533597b2e527be93f8b07d724faa489a0744cb0   1.00
    dcf4cb3ce9b1f599a3ccc0d7be469b4c6b363aae   0.00
    a0e6141f0661e04d65e5927da5fa4529423acd85   0.00
    d1d89005f667d2f9e8ac41934cb04ade769c9678   2.00
    8f46bb9e8b6cb89255338a22bdb50ab22604e87d   1.00
    Name: front_score, Length: 138275, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    46496164.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      37.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      bbc_co_uk
      12059.00
      0.47
      2.62
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2589.00
      0.79
      0.98
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      42.00
      0.95
      0.22
      0.00
      1.00
      1.00
      1.00
      1.00

      buzzfeed_com
      1290.00
      0.55
      0.50
      0.00
      0.00
      1.00
      1.00
      1.00

      cnn_com
      3996.00
      2.61
      4.39
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      24494.00
      0.63
      1.66
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      504.00
      2.24
      1.30
      0.00
      0.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      65.00
      0.94
      0.24
      0.00
      1.00
      1.00
      1.00
      1.00

      foxnews_com
      5009.00
      0.82
      2.06
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      4416.00
      0.49
      0.86
      0.00
      0.00
      0.00
      0.00
      2.00

      huffingtonpost_com
      2250.00
      1.96
      2.00
      0.00
      0.00
      0.00
      4.00
      4.00

      independent_co_uk
      5290.00
      0.82
      1.34
      0.00
      0.00
      0.00
      3.00
      3.00

      indy100_com
      395.00
      0.91
      0.29
      0.00
      1.00
      1.00
      1.00
      1.00

      lemonde_fr
      3649.00
      0.75
      0.97
      0.00
      0.00
      0.00
      2.00
      2.00

      libdemvoice_org
      141.00
      0.93
      0.26
      0.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      9204.00
      0.25
      0.43
      0.00
      0.00
      0.00
      0.25
      1.00

      nbcnews_com
      1878.00
      2.53
      1.93
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      439.00
      0.78
      0.42
      0.00
      1.00
      1.00
      1.00
      1.00

      npr_org
      1824.00
      1.57
      1.50
      0.00
      0.00
      3.00
      3.00
      3.00

      nytimes_com
      4627.00
      1.81
      2.67
      0.00
      0.00
      0.00
      5.00
      6.00

      order-order_com
      257.00
      0.88
      0.32
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      51.00
      0.88
      0.33
      0.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      6179.00
      0.64
      0.93
      0.00
      0.00
      0.00
      2.00
      2.00

      rt_com
      2302.00
      1.28
      0.96
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      121.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      6009.00
      0.40
      0.80
      0.00
      0.00
      0.00
      0.00
      2.00

      thecanary_co
      276.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      theguardian_com
      7989.00
      0.54
      1.15
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      8887.00
      0.05
      0.22
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      21307.00
      0.11
      0.45
      0.00
      0.00
      0.00
      0.00
      2.00

      westmonster_com
      302.00
      0.31
      0.46
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      397.00
      0.25
      0.44
      0.00
      0.00
      0.00
      1.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     334841.00
    bbc_co_uk                          46496164.00
    breitbart_com                       3866566.00
    brexitcentral_com                     16895.00
    buzzfeed_com                        2941222.00
    cnn_com                            30207108.00
    dailymail_co_uk                    14628916.00
    economist_com                       8541804.00
    evolvepolitics_com                   128802.00
    foxnews_com                        16366386.00
    ft_com                              3777860.00
    huffingtonpost_com                  9872554.00
    independent_co_uk                   8182856.00
    indy100_com                          243768.00
    lemonde_fr                          4140995.00
    libdemvoice_org                        8631.00
    mirror_co_uk                        2993874.00
    nbcnews_com                         9705196.00
    newstatesman_com                     154289.00
    npr_org                             6312422.00
    nytimes_com                        15597020.00
    order-order_com                       45695.00
    propublica_org                       379263.00
    reuters_com                         4011707.00
    rt_com                              5093479.00
    skwawkbox_org                          8235.00
    telegraph_co_uk                     4425088.00
    thecanary_co                               nan
    theguardian_com                     7949540.00
    thetimes_co_uk                       778877.00
    washingtonpost_com                  6161035.00
    westmonster_com                       20335.00
    yournewswire_com                      28501.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.72
    bbc_co_uk                          17.65
    breitbart_com                      15.17
    brexitcentral_com                   9.73
    buzzfeed_com                       14.89
    cnn_com                            17.22
    dailymail_co_uk                    16.50
    economist_com                      15.96
    evolvepolitics_com                 11.77
    foxnews_com                        16.61
    ft_com                             15.14
    huffingtonpost_com                 16.11
    independent_co_uk                  15.92
    indy100_com                        12.40
    lemonde_fr                         15.24
    libdemvoice_org                     9.06
    mirror_co_uk                       14.91
    nbcnews_com                        16.09
    newstatesman_com                   11.95
    npr_org                            15.66
    nytimes_com                        16.56
    order-order_com                    10.73
    propublica_org                     12.85
    reuters_com                        15.20
    rt_com                             15.44
    skwawkbox_org                       9.02
    telegraph_co_uk                    15.30
    thecanary_co                         nan
    theguardian_com                    15.89
    thetimes_co_uk                     13.57
    washingtonpost_com                 15.63
    westmonster_com                     9.92
    yournewswire_com                   10.26
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.81
    bbc_co_uk                          10.75
    breitbart_com                       8.26
    brexitcentral_com                   2.83
    buzzfeed_com                        7.99
    cnn_com                            10.32
    dailymail_co_uk                     9.59
    economist_com                       9.05
    evolvepolitics_com                  4.86
    foxnews_com                         9.70
    ft_com                              8.24
    huffingtonpost_com                  9.20
    independent_co_uk                   9.01
    indy100_com                         5.50
    lemonde_fr                          8.33
    libdemvoice_org                     2.16
    mirror_co_uk                        8.00
    nbcnews_com                         9.18
    newstatesman_com                    5.04
    npr_org                             8.75
    nytimes_com                         9.65
    order-order_com                     3.82
    propublica_org                      5.94
    reuters_com                         8.30
    rt_com                              8.54
    skwawkbox_org                       2.11
    telegraph_co_uk                     8.40
    thecanary_co                         nan
    theguardian_com                     8.98
    thetimes_co_uk                      6.66
    washingtonpost_com                  8.73
    westmonster_com                     3.01
    yournewswire_com                    3.35
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   4.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    lemonde_fr                         12.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       3.00
    telegraph_co_uk                    12.00
    thecanary_co                        0.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     5.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          15.00
    breitbart_com                      11.00
    brexitcentral_com                   3.00
    buzzfeed_com                       11.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  6.00
    foxnews_com                        14.00
    ft_com                             11.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         7.00
    lemonde_fr                         12.00
    libdemvoice_org                     2.00
    mirror_co_uk                       11.00
    nbcnews_com                        13.00
    newstatesman_com                    6.00
    npr_org                            12.00
    nytimes_com                        14.00
    order-order_com                     4.00
    propublica_org                      8.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                         nan
    theguardian_com                    13.00
    thetimes_co_uk                      9.00
    washingtonpost_com                 12.00
    westmonster_com                     3.00
    yournewswire_com                    4.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     20053
    1.00     18335
    2.00     11291
    3.00      8327
    4.00      6147
    5.00      4740
    6.00      4091
    7.00      3663
    8.00      3196
    9.00      2938
    10.00     2605
    11.00     2492
    12.00     2324
    13.00     2214
    14.00     2021
    15.00     2035
    16.00     1897
    17.00     1768
    18.00     1725
    19.00     1655
    20.00     1595
    21.00     1494
    22.00     1574
    23.00     1476
    24.00     1451
    25.00     1352
    26.00     1344
    27.00     1254
    28.00     1272
    29.00     1218
             ...
    66.00      100
    67.00       56
    68.00       70
    69.00       60
    70.00       52
    71.00       49
    72.00       56
    73.00       36
    74.00       41
    75.00       32
    76.00       19
    77.00       34
    78.00       29
    79.00       30
    80.00       20
    81.00       28
    82.00       17
    83.00       21
    84.00       19
    85.00        9
    86.00       10
    87.00       10
    88.00       10
    89.00        7
    90.00        4
    91.00        5
    92.00        3
    93.00        3
    94.00        3
    97.00        1
    Name: attention_index, Length: 96, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      8fc308aeac5b6b75c0baae4d5becdaa91a489988
      http://www.bbc.co.uk/news/world-europe-43841194
      Swedish DJ Avicii dies at 28
      2018-04-20 17:43:12.450
      2018-04-20 17:42:04
      355685
      2261.13
      2018-04-20T18:05:11.938Z
      48245
      233667
      73773
      ...
      46410824.00
      2018-04-20T18:42:14.000Z
      96
      119.00
      50.00
      20.00
      12.00
      15.00
      47.00
      97.00

      2a08dfaff24d92a3788b4ddb85bcbecec66c975f
      https://www.cnn.com/2018/04/21/entertainment/v...
      Verne Troyer, 'Mini-Me' in Austin Powers films...
      2018-04-21 20:58:12.888
      2018-04-21 20:52:09
      134799
      595.71
      2018-04-21T21:20:07.393Z
      12457
      91067
      31275
      ...
      30053099.00
      2018-04-21T21:10:35.000Z
      105
      151.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      f42e1b7e147355839db1cca071667c4dfcde1fbc
      http://www.bbc.co.uk/news/uk-43933056
      Legal battle toddler Alfie Evans dies
      2018-04-28 06:04:26.131
      2018-04-28 06:02:47
      140579
      3930.28
      2018-04-28T09:06:06.565Z
      21382
      106308
      12889
      ...
      46476225.00
      2018-04-28T06:37:18.000Z
      96
      66.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      4bfcac581b66efe4cd843c30d0fa748fff2c47c6
      https://www.cnn.com/2018/04/22/us/waffle-house...
      Hero prevented further bloodshed at Tennessee ...
      2018-04-22 14:10:19.077
      2018-04-22 14:05:18
      178909
      138.33
      2018-04-22T22:50:11.682Z
      21312
      137121
      20476
      ...
      30054437.00
      2018-04-22T16:30:09.000Z
      105
      195.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      8145abc78ad81e5f5cc7f88a8b41c26b85becc37
      https://www.cnn.com/2018/04/17/politics/barbar...
      Barbara Bush, former first lady, dies at 92
      2018-04-17 23:49:24.539
      2018-04-17 23:45:19
      348473
      1649.76
      2018-04-18T01:05:10.294Z
      37209
      277520
      33744
      ...
      30040473.00
      2018-04-17T23:49:02.000Z
      105
      26.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      7b9b3d5c8ee0581d1fda95ad1f2cbe7c3e4fda2b
      http://www.bbc.co.uk/news/world-middle-east-43...
      US allies launch strikes on Syria chemical sites
      2018-04-14 01:10:11.543
      2018-04-14 01:08:58
      168448
      466.39
      2018-04-14T07:43:14.177Z
      49569
      94543
      24336
      ...
      46369696.00
      2018-04-14T07:29:14.000Z
      96
      67.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      0847758466cc97ab4bdb7cebc75668dd3fcbae44
      https://www.cnn.com/2018/04/16/politics/michae...
      Michael Cohen reveals client list ahead of hea...
      2018-04-16 15:25:20.997
      2018-04-16 15:22:19
      128047
      5411.53
      2018-04-16T21:04:20.495Z
      36278
      78404
      13365
      ...
      30034998.00
      2018-04-16T19:16:02.000Z
      105
      592.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      7664a38d1979ec11da8cee1213834830cabec0c2
      https://www.cnn.com/2018/04/23/politics/george...
      Former President George H.W. Bush in intensive...
      2018-04-23 22:49:18.709
      2018-04-23 22:45:50
      488393
      1739.91
      2018-04-23T23:11:09.019Z
      98937
      343905
      45551
      ...
      30060241.00
      2018-04-23T22:51:47.000Z
      105
      20.00
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      5e638ea0ab5dfab129588fc29658028ca6e4c086
      https://www.cnn.com/2018/04/02/politics/trump-...
      Trump says 'our country is being stolen' due t...
      2018-04-02 11:55:17.513
      2018-04-02 11:48:33
      119509
      145.19
      2018-04-02T12:49:14.831Z
      31414
      73469
      14626
      ...
      29924986.00
      2018-04-02T12:35:09.000Z
      105
      252.00
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      bd0987ad82d9fa682846f43a66035d5ee412550d
      http://www.bbc.co.uk/news/world-europe-43683687
      Van drives into pedestrians in Germany
      2018-04-07 14:46:14.949
      2018-04-07 14:43:38
      65118
      455.28
      2018-04-07T15:08:06.644Z
      11506
      42716
      10896
      ...
      46300901.00
      2018-04-07T15:00:54.000Z
      96
      63.00
      47.00
      20.00
      10.00
      15.00
      45.00
      92.00

      b9d308491d3ca84f74d821e854fe8fb426257956
      https://www.cnn.com/2018/04/07/americas/canada...
      Multiple deaths reported in crash involving Ca...
      2018-04-07 06:55:14.007
      2018-04-07 06:51:18
      106273
      268.64
      2018-04-07T18:01:11.959Z
      14999
      80856
      10418
      ...
      29939419.00
      2018-04-07T17:49:03.000Z
      105
      165.00
      49.00
      18.00
      9.00
      15.00
      42.00
      91.00

      adcef7a8ae7abc514986679dfa03c9f4947e2ec2
      http://www.bbc.co.uk/news/uk-politics-43944988
      Amber Rudd resigns as home secretary
      2018-04-29 21:07:14.462
      2018-04-29 21:04:39
      129951
      1288.21
      2018-04-29T21:19:09.751Z
      31021
      82656
      16274
      ...
      46486435.00
      2018-04-29T21:09:43.000Z
      96
      57.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      9e8412abedfd92ea5e87329a1b658037c50846c8
      http://www.bbc.co.uk/news/world-africa-43621112
      South Africa's Winnie Mandela dies at 81
      2018-04-02 14:07:10.496
      2018-04-02 14:04:43
      96795
      382.00
      2018-04-02T14:19:13.732Z
      12601
      68162
      16032
      ...
      46265981.00
      2018-04-02T14:07:59.000Z
      96
      63.00
      49.00
      20.00
      7.00
      15.00
      42.00
      91.00

      6777aba349821201bf84afc471fda75754cba9da
      https://www.cnn.com/2018/04/25/us/golden-state...
      Golden State Killer: Authorities to announce a...
      2018-04-25 15:40:16.853
      2018-04-25 15:38:11
      93358
      145.91
      2018-04-25T18:04:12.614Z
      20897
      57937
      14524
      ...
      30065961.00
      2018-04-25T17:14:01.000Z
      105
      805.00
      49.00
      20.00
      7.00
      15.00
      42.00
      91.00

      846a6dd6430beec9ca5b072bd748c03641d0949f
      https://www.cnn.com/2018/04/01/politics/trump-...
      Trump tweets: 'NO MORE DACA DEAL'
      2018-04-01 14:31:17.767
      2018-04-01 14:26:13
      95196
      389.45
      2018-04-01T15:25:12.600Z
      27537
      57743
      9916
      ...
      29920840.00
      2018-04-01T15:07:21.000Z
      105
      206.00
      49.00
      20.00
      7.00
      15.00
      42.00
      91.00

      f9602d17fe7442a7f5255e5017f4b564dbc9cf13
      https://www.cnn.com/2018/04/22/us/tennessee-wa...
      3 killed at Waffle House in Tennessee
      2018-04-22 10:49:11.824
      2018-04-22 10:47:42
      83475
      165.92
      2018-04-22T13:04:11.763Z
      25384
      43519
      14572
      ...
      30054050.00
      2018-04-22T11:34:29.000Z
      105
      54.00
      48.00
      20.00
      7.00
      15.00
      42.00
      90.00

      81e11c509e7d040fb81f13783a4bd4f693f9b5f0
      https://www.cnn.com/2018/04/05/politics/donald...
      Trump says he didn't know about Stormy Daniels...
      2018-04-05 21:04:23.035
      2018-04-05 20:57:57
      59913
      210.91
      2018-04-05T21:27:06.734Z
      22767
      32093
      5053
      ...
      29936495.00
      2018-04-05T21:15:04.000Z
      105
      128.00
      46.00
      20.00
      9.00
      15.00
      44.00
      90.00

      042308324da49129060ee9dc165806b0f5159483
      https://www.cnn.com/2018/04/13/politics/trump-...
      Trump to speak on Syria tonight
      2018-04-14 00:52:16.872
      2018-04-14 00:47:09
      209940
      1151.91
      2018-04-14T01:24:12.988Z
      61583
      110339
      38018
      ...
      29991525.00
      2018-04-14T01:09:52.000Z
      105
      62.00
      50.00
      20.00
      5.00
      15.00
      40.00
      90.00

      10ffadc59e881959a8dd491ac60eeb5c78017c31
      https://www.cnn.com/2018/04/18/politics/donald...
      Trump blasts 'breeding' in sanctuary cities. T...
      2018-04-18 15:40:21.812
      2018-04-18 15:32:53
      103393
      267.80
      2018-04-20T03:05:08.025Z
      26070
      65348
      11975
      ...
      30042648.00
      2018-04-18T18:02:06.000Z
      105
      766.00
      49.00
      20.00
      6.00
      15.00
      41.00
      90.00

      a5662ddd2382861e5a9ef5b26932f1b4dcffd630
      https://www.cnn.com/2018/04/07/us/trump-tower-...
      Fire at Trump Tower injures one person, says FDNY
      2018-04-07 23:28:15.603
      2018-04-07 23:23:02
      53774
      123.56
      2018-04-08T01:11:12.454Z
      14405
      27647
      11722
      ...
      29939819.00
      2018-04-07T23:45:03.000Z
      105
      53.00
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      fcc2264ec50e478c1e05c31db2516a4914403fd9
      http://money.cnn.com/2018/04/16/news/companies...
      Starbucks CEO calls arrest of black men at sto...
      2018-04-16 12:44:14.313
      2018-04-16 12:38:16
      49696
      93.81
      2018-04-16T16:04:13.722Z
      16433
      28082
      5181
      ...
      30032452.00
      2018-04-16T13:00:03.000Z
      105
      nan
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      fbddd30d040459755d4aadd9010ad9e96517f7ad
      https://www.cnn.com/2018/04/23/world/toronto-c...
      Van strikes multiple pedestrians in Toronto, p...
      2018-04-23 18:04:19.402
      2018-04-23 17:59:01
      126333
      264.93
      2018-04-23T19:40:14.774Z
      23484
      84584
      18265
      ...
      30058323.00
      2018-04-23T18:11:02.000Z
      105
      46.00
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      4294fdff174879ab0589bfc53c432e3d115104a5
      http://www.bbc.co.uk/news/world-asia-india-436...
      Bollywood star Khan guilty of poaching
      2018-04-05 06:19:14.755
      2018-04-05 06:14:36
      34874
      56.96
      2018-04-05T12:15:07.276Z
      4438
      28109
      2327
      ...
      46279403.00
      2018-04-05T06:30:08.000Z
      96
      65.00
      43.00
      20.00
      11.00
      15.00
      46.00
      89.00

      47b0b99715235252b60c2bc5c1de478334be7867
      https://www.cnn.com/2018/04/29/politics/michel...
      5 takeaways on Michelle Wolf's hugely controve...
      2018-04-29 14:16:47.533
      2018-04-29 14:15:53
      49809
      115.91
      2018-04-29T16:29:08.442Z
      21092
      26190
      2527
      ...
      30124025.00
      2018-04-29T15:30:10.000Z
      105
      1111.00
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      4e3362cec33419a17ee9d809178c2b274dcd45d2
      https://www.cnn.com/2018/04/26/us/bill-cosby-t...
      Bill Cosby trial: The jury has reached a verdi...
      2018-04-26 17:52:20.410
      2018-04-26 17:46:39
      260387
      1232.41
      2018-04-26T19:09:07.571Z
      75822
      152598
      31967
      ...
      30070171.00
      2018-04-26T18:02:37.000Z
      105
      84.00
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      258942f24b6df728b6df3cd609e424c57cfbd661
      http://money.cnn.com/2018/04/29/news/companies...
      T-Mobile and Sprint agree to merge, finally
      2018-04-29 16:39:12.269
      2018-04-29 16:26:01
      71353
      249.97
      2018-04-29T17:43:10.038Z
      20059
      37990
      13304
      ...
      30124334.00
      2018-04-29T17:30:05.000Z
      105
      nan
      47.00
      20.00
      7.00
      15.00
      42.00
      89.00

      2888e0693674f83fd2b2e34eab57a69175617c9e
      https://www.nytimes.com/2018/04/27/world/asia/...
      North and South Korean Leaders Discuss Ending ...
      2018-04-27 07:46:13.972
      2018-04-27 07:43:03
      114526
      156.47
      2018-04-27T14:31:05.214Z
      18540
      84074
      11912
      ...
      15565770.00
      2018-04-27T08:00:01.000Z
      120
      nan
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      3c3e190480d1c9c7f6f753fb2b08afbcc5d5ccd3
      https://www.cnn.com/2018/04/20/politics/democr...
      Dems file suit alleging Russia, Trump, WikiLea...
      2018-04-20 15:52:11.889
      2018-04-20 15:46:16
      40104
      127.26
      2018-04-20T18:40:11.023Z
      12741
      21490
      5873
      ...
      30049423.00
      2018-04-20T18:27:02.000Z
      105
      152.00
      44.00
      20.00
      9.00
      15.00
      44.00
      88.00

      e524f53f0c6dd3152719ea965c2ccda50c314ddf
      https://www.nytimes.com/2018/04/09/us/politics...
      F.B.I. Raids Office of Trump’s Longtime Lawyer...
      2018-04-09 19:55:09.840
      2018-04-09 19:54:08
      718482
      10488.43
      2018-04-10T01:05:16.730Z
      198533
      445591
      74358
      ...
      15495843.00
      2018-04-09T20:05:08.000Z
      120
      291.00
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      223c62b3f6dfe70425b3c19cdc5ed0072c1a1810
      https://www.nytimes.com/2018/04/26/arts/televi...
      Bill Cosby Found Guilty of Sexual Assault Afte...
      2018-04-26 17:52:13.445
      2018-04-26 17:51:03
      369459
      1265.01
      2018-04-26T18:26:06.957Z
      88104
      247358
      33997
      ...
      15563107.00
      2018-04-26T17:55:55.000Z
      120
      nan
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      12d11fd698296de7cd1936b65c364c837a5548bb
      https://www.mirror.co.uk/news/world-news/jens-...
      Driver who ploughed into crowd in Germany kill...
      2018-04-09 12:13:18.411
      2018-04-07 21:25:00
      0
      0.00
      2018-04-09T12:25:06.507Z
      0
      0
      0
      ...
      nan
      NaN
      706
      242.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5cb80e681bf2979339c44745afdf172ff16d0f5f
      http://www.dailymail.co.uk/home/event/article-...
      Pressure at the Park Theatre review: Tense and...
      2018-04-07 21:10:19.358
      2018-04-07 21:06:44
      1
      0.02
      2018-04-08T09:28:08.107Z
      0
      0
      1
      ...
      nan
      NaN
      158
      1286.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4b2649eec0ecdb576cf686e621b062bb3da4f749
      https://www.washingtonpost.com/national/arizon...
      Arizona mother pleads not guilty in children’s...
      2018-04-13 21:46:24.110
      2018-04-13 21:34:07
      0
      0.00
      2018-04-13T21:58:14.200Z
      0
      0
      0
      ...
      nan
      NaN
      191
      152.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4d1dd8c218f64c1828dc7663f26a5838609b24c9
      https://www.washingtonpost.com/national/girl-3...
      Girl, 3, shot in head outside Detroit gas station
      2018-04-07 21:37:11.637
      2018-04-07 21:27:17
      0
      0.00
      2018-04-08T15:57:12.491Z
      0
      0
      0
      ...
      nan
      NaN
      191
      156.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0dadebe20f82a53b629fb6ce0a7d2dd092f92849
      https://www.washingtonpost.com/national/us-coa...
      US Coast Guard stops Haitian migrant boat with...
      2018-04-11 17:16:19.774
      2018-04-11 17:11:11
      1
      0.02
      2018-04-12T00:32:18.756Z
      0
      0
      1
      ...
      nan
      NaN
      191
      146.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4c3ec7580f425f9d26d199e6a5b02ed3bcf5c12a
      https://www.washingtonpost.com/opinions/how-to...
      How to pay for national parks
      2018-04-13 21:46:22.382
      2018-04-13 21:35:17
      1
      0.02
      2018-04-14T17:11:06.725Z
      0
      0
      1
      ...
      nan
      NaN
      191
      184.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      639e2c9e5742ad9331cab2f25b5f86c9f2c4041e
      https://www.washingtonpost.com/national/iowa-g...
      Iowa governor signs into law ban on ‘sanctuary...
      2018-04-11 17:16:19.772
      2018-04-11 17:11:10
      0
      0.00
      2018-04-12T07:38:08.777Z
      0
      0
      0
      ...
      nan
      NaN
      191
      159.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      57a85483548404c97eae5258828b2c05ec9eb327
      https://www.mirror.co.uk/sport/football/news/e...
      "Everyone has to start somewhere": Mansfield T...
      2018-04-07 21:31:22.017
      2018-04-07 21:30:00
      0
      0.00
      2018-04-08T09:50:15.671Z
      0
      0
      0
      ...
      nan
      NaN
      706
      680.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2bd202acccdc311daaf65a86f5c83f3a8fe9180b
      https://www.washingtonpost.com/national/prosec...
      Prosecutor says DNA tied serial killing suspec...
      2018-04-25 19:34:17.395
      2018-04-25 19:30:11
      0
      0.00
      2018-04-26T05:53:09.433Z
      0
      0
      0
      ...
      nan
      NaN
      191
      35.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      54a443baefc9e8fe645628c4042a6a64b7285965
      https://www.mirror.co.uk/tv/tv-news/stars-ship...
      Where are the stars of Shipwrecked now? Big Br...
      2018-04-17 10:19:20.594
      2018-04-17 10:05:00
      1
      0.09
      2018-04-17T10:31:09.731Z
      0
      0
      1
      ...
      nan
      NaN
      706
      824.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c3dc9c8c80c8ac22bc1a6c2c5d60172a77db565c
      https://www.economist.com/news/21740383-millen...
      Generation uphill
      2018-04-11 18:13:24.021
      2018-04-11 17:13:02
      0
      0.00
      2018-04-12T17:42:13.297Z
      0
      0
      0
      ...
      nan
      NaN
      1825
      91.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0cf15930b628ef55a48119addf25ef21b0a4af7a
      https://www.washingtonpost.com/national/indian...
      Indiana traffic stop nets 78 pounds of marijuana
      2018-04-17 10:19:12.100
      2018-04-17 10:12:15
      0
      0.00
      2018-04-18T08:44:13.403Z
      0
      0
      0
      ...
      nan
      NaN
      191
      82.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e0cc6335e5db8ac87ea3e85bff8e0f61a7a95804
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      BIP star Eden Schwencke reveals the real reaso...
      2018-04-23 11:22:18.822
      2018-04-23 11:18:58
      0
      0.00
      2018-04-23T11:34:11.810Z
      0
      0
      0
      ...
      nan
      NaN
      158
      594.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      9e14b5cdb439151d9fbf0f0a264d8f7b1ec9948f
      http://www.dailymail.co.uk/news/article-558963...
      JUSTIN WEBB: Mixing up generations brought sun...
      2018-04-07 21:04:22.854
      2018-04-07 21:01:19
      0
      0.00
      2018-04-08T06:22:10.253Z
      0
      0
      0
      ...
      nan
      NaN
      158
      462.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ae0c487e677a4be16ba61a99db796b04b2a96f4a
      http://www.dailymail.co.uk/sport/rugbyunion/ar...
      England and Leicester team-mates George Ford a...
      2018-04-13 21:34:39.967
      2018-04-13 21:30:20
      1
      0.02
      2018-04-13T23:47:09.938Z
      0
      0
      1
      ...
      nan
      NaN
      158
      867.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b561a9358c84fea3636da29592f9604628a68f30
      https://www.washingtonpost.com/sports/dcunited...
      FIFA bans Brazilian official Del Nero for life...
      2018-04-27 12:43:18.583
      2018-04-27 12:35:33
      0
      0.00
      2018-04-27T18:59:11.265Z
      0
      0
      0
      ...
      nan
      NaN
      191
      123.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      32aa570d5c404d80feb23b5e0012df5a7c62aa60
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      MAFS' Patrick Miller cruises around Melbourne ...
      2018-04-17 10:13:13.742
      2018-04-17 10:09:21
      0
      0.00
      2018-04-18T08:40:13.458Z
      0
      0
      0
      ...
      nan
      NaN
      158
      634.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2a8fdd4963fc770cbd8cb0bb9b31f2be58c5246f
      http://www.dailymail.co.uk/home/event/article-...
      Event critics give their take on live orchestr...
      2018-04-07 21:04:18.792
      2018-04-07 21:02:46
      1
      0.02
      2018-04-08T09:23:10.354Z
      0
      0
      1
      ...
      nan
      NaN
      158
      1081.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      d7e7577e3305efb7bc36b09a6b978e661bc62ba2
      https://www.independent.co.uk/sport/golf/maste...
      Masters 2018: Rory McIlroy’s beautiful front n...
      2018-04-07 21:07:20.220
      2018-04-07 21:04:02
      1
      0.10
      2018-04-07T21:18:12.122Z
      0
      0
      1
      ...
      nan
      NaN
      386
      449.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      68e5f5dee89a81e5bfc804b966babcc7fc408f77
      https://www.washingtonpost.com/politics/us-eco...
      US economic growth slowed to 2.3 percent pace ...
      2018-04-27 12:46:09.121
      2018-04-27 12:34:17
      0
      0.00
      2018-04-28T01:06:09.572Z
      0
      0
      0
      ...
      nan
      NaN
      191
      145.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      760632750b46d0823ef61867a0039b485291cb07
      https://www.washingtonpost.com/world/europe/co...
      Correction: Armenia-Protests story
      2018-04-23 11:34:16.922
      2018-04-23 11:19:10
      0
      0.00
      2018-04-24T05:56:11.987Z
      0
      0
      0
      ...
      nan
      NaN
      191
      287.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      7e12cadbfb711fa81c5ec89699031c31a2f99621
      http://www.bbc.co.uk/news/av/uk-wales-43925442...
      Moment masked robber sprayed shop worker with ...
      2018-04-27 12:34:15.844
      2018-04-27 12:33:06
      1
      0.02
      2018-04-27T21:52:12.973Z
      0
      0
      1
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      80ac4446da66d50b4c5b6fa235adfbfc9ae7f72d
      http://www.bbc.co.uk/sport/live/horse-racing/4...
      Listen to live Scottish Grand National Festiva...
      2018-04-17 10:10:17.460
      2018-04-17 10:05:32
      0
      0.00
      2018-04-17T10:22:06.496Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5bb39645b3c0bc2978f8c42a4c92e7cfd61c3226
      http://www.dailymail.co.uk/home/event/article-...
      Parsons gets everything right with minimum of ...
      2018-04-07 21:07:17.571
      2018-04-07 21:05:20
      1
      0.10
      2018-04-07T21:18:12.114Z
      0
      0
      1
      ...
      nan
      NaN
      158
      832.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      69bee8508b05df1cd0be36bcf93ff713d616115f
      http://www.bbc.co.uk/news/uk-scotland-glasgow-...
      Murder accused found sister assault claims 'fu...
      2018-04-11 17:16:16.294
      2018-04-11 17:13:48
      1
      0.02
      2018-04-11T21:31:04.641Z
      0
      0
      1
      ...
      nan
      NaN
      96
      525.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ea72509784337c57159df275fe9c67083bb30ecd
      https://www.independent.co.uk/arts-entertainme...
      Faceless, Park Theatre, London, review: A comp...
      2018-04-23 11:25:16.814
      2018-04-23 11:20:36
      1
      0.02
      2018-04-24T08:51:05.664Z
      0
      0
      1
      ...
      nan
      NaN
      386
      560.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5c8927ba9144e73cc655cf31125be468e38afe33
      http://www.dailymail.co.uk/news/article-566487...
      Manhunt for Maine cop killer enters day three ...
      2018-04-27 20:58:19.756
      2018-04-27 12:33:16
      1
      0.02
      2018-04-27T22:11:13.773Z
      0
      0
      1
      ...
      nan
      NaN
      158
      596.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2465261fcdbe82198242a41dd4ad953e43202021
      https://www.washingtonpost.com/politics/us-eco...
      US economic growth slowed to 2.3 percent rate ...
      2018-04-27 12:46:09.533
      2018-04-27 12:33:13
      0
      0.00
      2018-04-27T19:01:17.482Z
      0
      0
      0
      ...
      nan
      NaN
      191
      34.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      296b2ff2960aa46ff7dd1b19ce105175c3353a3f
      https://www.washingtonpost.com/world/asia_paci...
      Kim Jong Un returns to North Korea, ending sum...
      2018-04-27 12:46:11.318
      2018-04-27 12:33:11
      0
      0.00
      2018-04-27T21:03:10.921Z
      0
      0
      0
      ...
      nan
      NaN
      191
      43.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4b98aec507b39c8afbb7f4cde8152f94ff1e8dbe
      https://www.washingtonpost.com/national/foreca...
      Forecasters warn of rare, life-threatening wil...
      2018-04-17 11:37:19.856
      2018-04-17 11:29:18
      1
      0.02
      2018-04-17T21:56:09.086Z
      0
      0
      1
      ...
      nan
      NaN
      191
      151.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

138275 rows × 26 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      8b67316e22d9bbbbe26a97a39dd1365ff9d175d3
      https://www.buzzfeed.com/katienotopoulos/teens...
      Teens Also Sometimes Quit Social Media
      2018-04-07 17:04:24.234
      2018-04-04 17:37:46
      0
      0.00
      2018-04-07T17:16:12.351Z
      0
      0
      0
      ...
      2018-04-08T01:04:02.000Z
      147
      1176.00
      0.00
      19.00
      13.00
      11.00
      43.00
      43.00
      43.00

      3cb27a4e5d7b0b0d6972afec2737c2a99cc76be2
      https://www.buzzfeed.com/theresatamkins/twin-s...
      Twin Sisters With Severe OCD Have Died In A Po...
      2018-04-05 19:16:17.257
      2018-04-04 19:35:26
      19
      0.31
      2018-04-05T20:28:07.825Z
      1
      10
      8
      ...
      2018-04-08T18:04:00.000Z
      147
      130.00
      4.00
      19.00
      15.00
      11.00
      45.00
      49.00
      41.00

      f05031efa6a829f220e82331a2c91c48b33ddd91
      https://www.buzzfeed.com/kendalltaggart/nypd-p...
      Here’s Why BuzzFeed News Is Publishing Thousan...
      2018-04-16 09:34:16.432
      2018-04-13 23:12:43
      0
      0.00
      2018-04-16T09:45:13.729Z
      0
      0
      0
      ...
      2018-04-16T13:23:00.000Z
      147
      1304.00
      0.00
      19.00
      11.00
      11.00
      41.00
      41.00
      41.00

      acd63345be81904294a37e8e687c1ffcd41c7c1f
      https://www.buzzfeed.com/claudiakoerner/teache...
      This Teacher Says She Can’t Afford To Stay On ...
      2018-04-11 12:01:20.976
      2018-04-09 21:34:31
      0
      0.00
      2018-04-11T12:13:11.080Z
      0
      0
      0
      ...
      2018-04-11T17:20:19.000Z
      147
      2476.00
      0.00
      19.00
      11.00
      11.00
      41.00
      41.00
      41.00

      4d77d72e13dbba05f89bf209fde926d9347a3548
      https://www.buzzfeed.com/theresatamkins/ocular...
      This Rare Eye Cancer Is Affecting Young Women ...
      2018-05-01 17:07:48.450
      2018-04-30 17:03:45
      5
      0.45
      2018-05-01T17:19:18.075Z
      0
      3
      2
      ...
      2018-05-02T18:50:17.000Z
      147
      1162.00
      1.00
      19.00
      11.00
      11.00
      41.00
      42.00
      40.00

      aaeccc954251b991b87742305992c21b5a07e4fb
      https://www.buzzfeed.com/krishrach/south-korea...
      The #MeToo Era Is Happening In South Korea Whe...
      2018-04-17 12:40:15.914
      2018-04-13 13:44:41
      2
      0.19
      2018-04-17T12:52:07.422Z
      0
      0
      2
      ...
      2018-04-18T13:30:17.000Z
      147
      2552.00
      1.00
      19.00
      11.00
      11.00
      41.00
      42.00
      40.00

      adf38c51ab0e882926d1294199ad563e9efa8c71
      https://www.buzzfeed.com/albertsamaha/san-anto...
      This Police Force Is Considered Progressive, B...
      2018-04-24 12:19:26.784
      2018-04-19 18:00:51
      0
      0.00
      2018-04-24T12:30:13.329Z
      0
      0
      0
      ...
      2018-04-24T13:24:55.000Z
      147
      4976.00
      0.00
      19.00
      10.00
      11.00
      40.00
      40.00
      40.00

      261f8a165d69b04445a514ae9d7693b67d003c6b
      https://www.buzzfeed.com/katebubacz/heres-why-...
      Here's Why People Are Seeking Asylum In The US
      2018-04-30 11:34:19.256
      2018-04-28 19:04:52
      1
      0.10
      2018-04-30T11:45:08.069Z
      0
      0
      1
      ...
      2018-04-30T14:11:07.000Z
      147
      1486.00
      0.00
      19.00
      9.00
      11.00
      39.00
      39.00
      39.00

      d91e16c87837630173d243077657f028199a5e29
      https://www.buzzfeed.com/johnpaczkowski/make-d...
      What To Expect At Facebook And Google's Big Sh...
      2018-04-23 16:31:31.209
      2018-04-21 00:10:58
      2
      0.18
      2018-04-23T16:43:09.589Z
      0
      1
      1
      ...
      2018-04-23T16:40:19.000Z
      147
      824.00
      1.00
      19.00
      10.00
      11.00
      40.00
      41.00
      39.00

      c9b4fe9d6bb974728cf3be7416843b6a59134771
      https://www.buzzfeed.com/monicamark/dapchi-gir...
      There’s A Truther Movement In Nigeria That Thi...
      2018-04-28 16:19:14.197
      2018-04-27 18:40:36
      11
      0.20
      2018-04-28T16:30:13.514Z
      0
      2
      9
      ...
      2018-04-29T23:34:00.000Z
      147
      4457.00
      2.00
      19.00
      10.00
      11.00
      40.00
      42.00
      38.00

      01710ce2fee069ba68fb22291549cd4822c66081
      https://www.buzzfeed.com/munzeralawad/syrian-r...
      Syrian Rebels Hope Strikes Aren’t Just For Sho...
      2018-04-09 21:28:19.824
      2018-04-09 21:09:10
      36
      0.46
      2018-04-09T21:40:08.801Z
      2
      20
      14
      ...
      2018-04-10T01:00:57.000Z
      147
      848.00
      6.00
      19.00
      14.00
      11.00
      44.00
      50.00
      38.00

      1c0fa02e27e11772590021e15381451d072f937f
      https://www.buzzfeed.com/kateaurthur/joe-pater...
      Watching "Paterno" During The #MeToo Era
      2018-04-07 14:04:19.812
      2018-04-06 22:47:16
      43
      0.22
      2018-04-07T16:17:11.923Z
      9
      22
      12
      ...
      2018-04-08T21:03:28.000Z
      147
      1071.00
      6.00
      19.00
      14.00
      11.00
      44.00
      50.00
      38.00

      8a5c5d33f825e7d9497f31fb9c1bd70b1013bb13
      https://www.buzzfeed.com/scottpham/cocaine-fen...
      The Opioid Crisis Is Turning Into A Cocaine Cr...
      2018-05-01 13:39:11.249
      2018-04-30 14:43:47
      35
      0.52
      2018-05-01T14:51:11.209Z
      2
      15
      18
      ...
      2018-05-01T14:42:35.000Z
      147
      2614.00
      5.00
      19.00
      11.00
      11.00
      41.00
      46.00
      36.00

      8607d8bcd0338c95626444386547442a95dbe452
      https://www.buzzfeed.com/karlazabludovsky/the-...
      The Caravan That Has Trump Angry Is An Excepti...
      2018-04-02 21:34:26.612
      2018-04-02 20:19:41
      56
      0.15
      2018-04-03T01:49:09.479Z
      3
      29
      24
      ...
      2018-04-03T00:56:32.000Z
      147
      939.00
      7.00
      19.00
      13.00
      11.00
      43.00
      50.00
      36.00

      472557d89afbef234765488b15cf82a8d7364158
      https://www.buzzfeed.com/buzzfeednews/bill-cos...
      Live Updates: Bill Cosby Starts Under The Glar...
      2018-04-09 16:07:43.899
      2018-04-09 16:04:52
      53
      0.15
      2018-04-09T23:24:14.462Z
      6
      38
      9
      ...
      2018-04-09T17:56:12.000Z
      147
      382.00
      7.00
      19.00
      13.00
      11.00
      43.00
      50.00
      36.00

      17951ed09b77c76391414d3a9921c5900581066b
      https://www.buzzfeed.com/adambvary/chappaquidd...
      This New Film Is Resurrecting Ted Kennedy’s Da...
      2018-04-07 01:19:14.059
      2018-04-07 00:51:00
      85
      0.23
      2018-04-07T23:44:12.003Z
      20
      35
      30
      ...
      2018-04-08T23:04:41.000Z
      147
      1128.00
      9.00
      19.00
      14.00
      11.00
      44.00
      53.00
      35.00

      b2166204c46fb2999a8b47d79c35173cb3be618f
      https://www.buzzfeed.com/laurageiser/californi...
      California's Trees Are Dying At A Catastrophic...
      2018-04-22 16:01:16.983
      2018-04-17 15:47:49
      0
      0.00
      2018-04-22T16:12:11.498Z
      0
      0
      0
      ...
      2018-04-22T19:33:44.000Z
      147
      1719.00
      0.00
      19.00
      4.00
      11.00
      34.00
      34.00
      34.00

      1f401a0e408e81d6fbff3e6590f6c4f6191c34d7
      https://www.buzzfeed.com/adolfoflores/caravan-...
      Not Over Yet: With Legal Papers In Hand, The M...
      2018-04-06 22:52:30.325
      2018-04-06 22:21:23
      119
      0.27
      2018-04-07T22:18:14.605Z
      31
      61
      27
      ...
      2018-04-07T00:40:13.000Z
      147
      1060.00
      11.00
      19.00
      15.00
      11.00
      45.00
      56.00
      34.00

      b21e43b2d43b7fdb8d7919c808cd2c718135b62e
      https://www.buzzfeed.com/koviebiakolo/the-rach...
      The Rachel Dolezal Documentary Has Become A De...
      2018-04-28 14:14:12.074
      2018-04-28 01:58:16
      29
      0.12
      2018-04-28T15:26:02.727Z
      6
      10
      13
      ...
      2018-04-29T14:04:00.000Z
      147
      3089.00
      5.00
      19.00
      9.00
      11.00
      39.00
      44.00
      34.00

      2cd85f75342343e471684a899fbca70a1da5c7b0
      https://www.buzzfeed.com/salvadorhernandez/nat...
      Hundreds Of National Guard Troops Are Headed T...
      2018-04-07 01:16:17.047
      2018-04-07 01:08:37
      143
      0.58
      2018-04-07T21:41:06.364Z
      27
      88
      28
      ...
      2018-04-07T20:30:38.000Z
      147
      496.00
      11.00
      19.00
      15.00
      11.00
      45.00
      56.00
      34.00

      a12bb350a8e994e03d79a58a26a0ccad0d67fe3b
      https://www.buzzfeed.com/skbaer/how-the-golden...
      How California's Most Notorious And Prolific S...
      2018-04-27 23:55:31.649
      2018-04-27 02:59:31
      19
      0.20
      2018-04-28T01:07:16.868Z
      0
      2
      17
      ...
      2018-04-28T20:35:06.000Z
      147
      1304.00
      4.00
      19.00
      8.00
      11.00
      38.00
      42.00
      34.00

      46a709c479a116fa560575e778ffadcaad98cd24
      https://www.buzzfeed.com/bensmith/facebook-you...
      What Comes After The Social Media Empires
      2018-04-18 18:04:27.256
      2018-04-18 18:01:44
      68
      0.25
      2018-04-18T19:17:04.867Z
      5
      27
      36
      ...
      2018-04-18T18:42:49.000Z
      147
      1228.00
      8.00
      19.00
      12.00
      11.00
      42.00
      50.00
      34.00

      79f3ea9c5f0c08b32c975851e7aea661ab91776b
      https://www.buzzfeed.com/adolfoflores/caravan-...
      Organizers Say Caravan Will Continue And Will ...
      2018-04-04 00:37:26.127
      2018-04-04 00:29:33
      117
      0.42
      2018-04-04T04:51:10.902Z
      33
      57
      27
      ...
      2018-04-04T03:18:58.000Z
      147
      748.00
      10.00
      19.00
      13.00
      11.00
      43.00
      53.00
      33.00

      f7de523afb549ce453cf8e07321eabc8fe1db5d2
      https://www.buzzfeed.com/coralewis/new-sanctio...
      The US Is Placing New Sanctions On 38 Russian ...
      2018-04-06 12:40:13.441
      2018-04-06 12:37:40
      67
      0.27
      2018-04-06T12:52:07.859Z
      8
      28
      31
      ...
      2018-04-06T15:20:15.000Z
      147
      295.00
      8.00
      19.00
      11.00
      11.00
      41.00
      49.00
      33.00

      e228b9b4729b1e26b70d184c84b07192c813b026
      https://www.buzzfeed.com/craigsilverman/youtub...
      How YouTube’s Channel Recommendations Push Use...
      2018-04-12 19:07:36.597
      2018-04-12 16:15:27
      58
      0.15
      2018-04-13T15:33:13.988Z
      8
      14
      36
      ...
      2018-04-16T18:07:25.000Z
      147
      1148.00
      7.00
      19.00
      10.00
      11.00
      40.00
      47.00
      33.00

25 rows × 27 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      98e23165dce61669ac8bc420f87cb39d5682f90b
      https://www.cnn.com/2018/04/26/world/korea-sum...
      South Korea credits Trump for talks with North
      2018-04-26 11:55:10.250
      2018-04-26 11:49:17
      488344
      539.73
      2018-04-27T17:04:09.730Z
      78939
      344837
      64568
      ...
      NaN
      105
      443.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      697c0030f63bf62fbaf47e740a04af8b45a51f0c
      http://yournewswire.com/california-obscene-lgb...
      California Bans Parents From Pulling Kids From...
      2018-04-23 16:24:28.801
      2018-04-23 15:22:43
      115948
      67.65
      2018-04-26T22:37:11.361Z
      43498
      54777
      17673
      ...
      NaN
      22568
      765.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      d475a7270eadc9e9b419abc63a59e54478399d13
      https://www.cnn.com/2018/04/27/asia/read-full-...
      READ: Full declaration of North and South Kore...
      2018-04-27 10:13:20.763
      2018-04-27 10:10:33
      97886
      204.93
      2018-04-28T09:06:07.845Z
      16821
      68860
      12205
      ...
      NaN
      105
      1144.00
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      c46c58ebe5cb89d87b3b3e734d799f697e8de70c
      http://www.foxnews.com/us/2018/04/18/former-ac...
      Former ace Navy pilot ID’d as hero who landed ...
      2018-04-18 05:44:09.443
      2018-04-18 05:37:56
      186147
      651.87
      2018-04-18T14:11:11.142Z
      8710
      157521
      19916
      ...
      NaN
      285
      412.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      42c82ece9cb64042cbed2d151e3d0aa5d43c608e
      http://www.dailymail.co.uk/tvshowbiz/article-5...
      Nasser Sultan 'moves on with a man' after spli...
      2018-04-08 22:31:23.503
      2018-04-08 22:26:54
      118419
      215.18
      2018-04-09T03:34:08.228Z
      101766
      14833
      1820
      ...
      NaN
      158
      424.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      f56a0d6c345e8f99deeb019ba69a335ef193f54e
      http://www.dailymail.co.uk/news/article-560591...
      NSW man nearly burnt alive by barbeque
      2018-04-12 01:46:20.183
      2018-04-12 01:42:16
      113588
      443.69
      2018-04-12T15:07:10.107Z
      25066
      61963
      26559
      ...
      NaN
      158
      206.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      e0db1773b7e34469b8624958df28c93997f5048e
      http://www.dailymail.co.uk/news/article-563067...
      Body cam shows police fatally shoot black male...
      2018-04-18 16:34:23.675
      2018-04-18 16:31:01
      91087
      31.41
      2018-04-26T13:51:12.301Z
      23820
      40915
      26352
      ...
      NaN
      158
      699.00
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      02d16220f444d881aac0dff48bd84c204edeb189
      https://www.washingtonpost.com/news/politics/w...
      China’s retaliatory tariffs will hit Trump cou...
      2018-04-04 18:01:16.010
      2018-04-04 17:56:19
      110272
      84.71
      2018-04-05T02:17:11.648Z
      31539
      59270
      19463
      ...
      NaN
      191
      328.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      fa16823eda18f3c7cd15c117d53c7c6504736c84
      https://www.washingtonpost.com/news/politics/w...
      FBI seizes records related to Stormy Daniels i...
      2018-04-09 20:34:12.849
      2018-04-09 20:27:56
      102076
      327.17
      2018-04-09T22:32:12.679Z
      25888
      63815
      12373
      ...
      NaN
      191
      145.00
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      6ed8d0b4c37eed6ca2a44109c5b1c2bf160370a5
      http://www.foxnews.com/entertainment/2018/04/0...
      Jimmy Kimmel mocks Melania Trump's accent, tro...
      2018-04-03 16:39:16.948
      2018-04-03 16:26:39
      101329
      199.70
      2018-04-03T19:28:07.769Z
      41034
      54055
      6240
      ...
      NaN
      285
      564.00
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      e38732e2f458927cf77118b432ba912d197770ba
      https://www.cnn.com/2018/04/20/politics/mike-p...
      Indiana abortion law signed by Mike Pence rule...
      2018-04-20 20:34:29.230
      2018-04-20 20:30:33
      141991
      105.32
      2018-04-21T15:17:16.755Z
      13900
      114761
      13330
      ...
      NaN
      105
      367.00
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      5779932dc4708265140b6f5c57d63e553b1a2a42
      http://www.foxnews.com/health/2018/04/02/teens...
      Teens snort condoms then pull them through mou...
      2018-04-02 16:59:17.583
      2018-04-02 16:51:40
      103517
      545.63
      2018-04-03T05:03:10.892Z
      49901
      40170
      13446
      ...
      NaN
      285
      395.00
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      1884d5b8859e0dd1c2c252c4a5cbb3e416a96f18
      https://www.buzzfeed.com/caseyrackham/watch-co...
      Pixar's "Coco" Is Coming To Netflix In May, So...
      2018-04-24 14:14:07.919
      2018-04-24 14:09:14
      100220
      340.67
      2018-04-24T18:46:10.985Z
      19390
      74627
      6203
      ...
      NaN
      147
      nan
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      574b18606f4757686e9ee04a479a85f3abb0982c
      http://www.foxnews.com/entertainment/2018/04/1...
      Roseanne defends pro-Trump Diamond and Silk af...
      2018-04-11 18:24:10.969
      2018-04-11 17:21:34
      68570
      1257.53
      2018-04-12T01:11:16.004Z
      2981
      63390
      2199
      ...
      NaN
      285
      411.00
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      ee874f553d5fdf610abf75e9e100d40dd7a1dbcd
      http://www.washingtonpost.com/video/politics/t...
      Trump claims ‘millions and millions’ vote ille...
      2018-04-05 19:28:14.478
      2018-04-05 19:11:23
      58822
      40.83
      2018-04-06T00:32:04.144Z
      23824
      27390
      7608
      ...
      NaN
      191
      nan
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      2e57f337f489f4a986044c55a13684946df9fabe
      https://www.washingtonpost.com/news/arts-and-e...
      Forty years ago, ‘Rock Lobster’ launched the c...
      2018-04-06 11:04:29.589
      2018-04-06 11:00:24
      72671
      397.24
      2018-04-06T23:06:09.762Z
      9873
      56191
      6607
      ...
      NaN
      191
      982.00
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      2d671f7213cf85d0a97ca1db26c60b805740fa59
      https://www.nytimes.com/2018/04/22/nyregion/ch...
      New York to Expand Use of Doulas to Reduce Chi...
      2018-04-22 22:52:15.228
      2018-04-22 22:51:09
      55909
      60.36
      2018-04-23T14:02:10.892Z
      5271
      44099
      6539
      ...
      NaN
      120
      684.00
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      0ba20825bcbbd5fd6b4f812e89f2632c3182b313
      http://www.bbc.co.uk/news/uk-england-london-43...
      Man, 78, arrested over Hither Green 'burglar' ...
      2018-04-04 10:16:08.368
      2018-04-04 10:14:48
      100900
      132.18
      2018-04-05T09:34:11.481Z
      41029
      49364
      10507
      ...
      NaN
      96
      140.00
      49.00
      0.00
      3.00
      0.00
      3.00
      52.00
      -46.00

      e7ce2eb7d78463c80e6747eb0f961e2bb8e5b141
      https://www.reuters.com/article/us-usa-biofuel...
      Exclusive: U.S. EPA grants biofuels waiver to ...
      2018-04-30 11:04:20.311
      2018-04-30 11:02:46
      59982
      152.27
      2018-04-30T15:30:16.669Z
      7488
      39088
      13406
      ...
      NaN
      497
      nan
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      800b7219c01f879055a94d83af8387aae9aa3213
      http://yournewswire.com/queen-elizabeth-bans-o...
      Queen Elizabeth Bans Obamas From Royal Wedding
      2018-04-15 16:54:33.316
      2018-04-15 16:22:54
      75033
      25.45
      2018-04-17T23:34:14.132Z
      17465
      47923
      9645
      ...
      NaN
      22568
      1146.00
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      be0c99a91cade39290e46f145c9ed8bff606a225
      https://www.washingtonpost.com/opinions/robert...
      Robert Redford: The biggest Scott Pruitt scand...
      2018-04-26 23:46:22.593
      2018-04-26 23:36:59
      73360
      59.73
      2018-04-30T01:31:09.202Z
      4778
      56113
      12469
      ...
      NaN
      191
      608.00
      47.00
      0.00
      2.00
      0.00
      2.00
      49.00
      -45.00

      e641e3fdb87663c53282c8e2e5b74373a9e4f905
      http://www.foxnews.com/politics/2018/04/04/oba...
      Obama State Dept used taxpayer dollars to fund...
      2018-04-04 21:59:16.809
      2018-04-04 21:56:39
      56230
      70.48
      2018-04-04T23:14:07.972Z
      8452
      33225
      14553
      ...
      NaN
      285
      802.00
      46.00
      0.00
      1.00
      0.00
      1.00
      47.00
      -45.00

      ad1282930d08eea4d6b20ac2ea99a087f593183e
      http://www.bbc.co.uk/news/uk-england-london-43...
      No further action after 'burglar' death
      2018-04-06 17:25:14.934
      2018-04-06 17:23:05
      320098
      1413.60
      2018-04-06T17:59:10.398Z
      42599
      241401
      36098
      ...
      NaN
      96
      68.00
      50.00
      0.00
      5.00
      0.00
      5.00
      55.00
      -45.00

      c2ff33a69eecfbeaafb92f967d2434a758da7bc8
      https://yournewswire.com/smallville-star-sold-...
      ‘Smallville’ Star Confesses She Sold Children ...
      2018-04-29 00:09:32.401
      2018-04-29 00:07:39
      276577
      800.85
      2018-04-30T23:04:08.422Z
      93935
      123872
      58770
      ...
      2018-05-01T16:12:00.000Z
      22568
      865.00
      50.00
      0.00
      1.00
      4.00
      5.00
      55.00
      -45.00

      2d9b8a4b0d7b4055b3f2d416f1f87d264d7f61a5
      http://yournewswire.com/california-brown-ban-b...
      California Gov. Jerry Brown To Ban Sales Of Th...
      2018-04-20 17:04:25.516
      2018-04-20 16:27:56
      552344
      422.89
      2018-04-22T07:05:08.991Z
      259819
      208358
      84167
      ...
      2018-04-21T19:52:56.000Z
      22568
      437.00
      50.00
      0.00
      1.00
      4.00
      5.00
      55.00
      -45.00

25 rows × 27 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_" + date_filename + ".csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}