# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
data = pd.read_csv("articles_2017-12-01_2017-12-31.csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      ddd1ccb2bc026b5b44204e5f4f08de1d75d4280a
      https://www.thetimes.co.uk/article/winter-sun-...
      Winter sun: 12 of the best holiday destinations
      2017-12-04 00:04:58.520
      2017-12-01
      2
      0.000000
      2017-12-04T00:16:03.826Z
      0
      1
      1
      The Times
      thetimes_co_uk
      0
      1435
      237.0
      False
      NaN
      NaN
      6435
      126.0

      d8409c3f40ce732ab4917e88f3991fba6e34052c
      https://www.thetimes.co.uk/article/news-in-pic...
      News in pictures
      2017-12-01 03:24:07.825
      2017-12-01
      246
      1.150685
      2017-12-01T18:50:03.239Z
      26
      216
      4
      The Times
      thetimes_co_uk
      0
      1235
      293.0
      False
      NaN
      NaN
      6435
      NaN

      8ec1e46d261cde4b7eb7818481e02a439a344966
      https://www.thetimes.co.uk/article/the-best-ch...
      The best Christmas photos from around the UK
      2017-12-02 00:04:48.863
      2017-12-01
      0
      0.000000
      2017-12-02T00:16:18.566Z
      0
      0
      0
      The Times
      thetimes_co_uk
      0
      1434
      328.0
      False
      NaN
      NaN
      6435
      NaN

      a0b6c78af7ce197a41eb7f2e8514929e40cf0f01
      https://www.thetimes.co.uk/article/nationalist...
      Inés Arrimadas: Nationalist’s wife leads call ...
      2017-12-01 07:39:11.810
      2017-12-01
      0
      0.000000
      2017-12-01T07:50:11.497Z
      0
      0
      0
      The Times
      thetimes_co_uk
      0
      565
      294.0
      False
      NaN
      NaN
      6435
      125.0

      d6b3d8608c147417d6f83042303ad18d48df5069
      https://www.thetimes.co.uk/article/winners-of-...
      Winners of the mini-reshuffle
      2017-12-01 00:08:35.154
      2017-12-01
      1
      0.016389
      2017-12-01T10:30:08.613Z
      0
      0
      1
      The Times
      thetimes_co_uk
      0
      1435
      291.0
      False
      NaN
      NaN
      6435
      125.0

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    139145.00
    mean       1105.88
    std        9852.92
    min           0.00
    50%          24.00
    75%         243.00
    90%        1545.00
    95%        4086.00
    99%       20228.36
    99.5%     32446.04
    99.9%     87128.53
    max     1741886.00
    Name: fb_engagements, dtype: float64

There's 3 articles with more than 1 million engagements this month.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count

      id

      5c2b2bee6d23ef061ff65a0adcd0fb2e1d5374bf
      http://www.dailymail.co.uk/health/article-2054...
      Bad news for dads: Babies 'should share mother...
      2017-12-08 16:18:25.150
      2017-12-07 21:00:00
      1502932
      284.84
      2017-12-09T04:07:10.793Z
      573487
      672710
      256735
      Daily Mail
      dailymail_co_uk
      0
      0
      nan
      True
      13574153.00
      2017-12-08T16:15:11.000Z
      158
      494.00

      5bf276f7b22af522d1a15b94ad40f1b3858d3e61
      https://www.nytimes.com/2017/12/12/us/politics...
      Once a Long Shot, Democrat Doug Jones Wins Ala...
      2017-12-13 03:28:08.107
      2017-12-13 03:25:52
      1054926
      8809.66
      2017-12-13T04:12:18.429Z
      86607
      906432
      61887
      New York Times
      nytimes_com
      840
      1319
      120.00
      True
      15020753.00
      2017-12-13T03:28:45.000Z
      120
      1454.00

      823cc4c44f81ebb54e4adef2e98c20f95480055f
      https://www.washingtonpost.com/national/health...
      CDC gets list of forbidden words: fetus, trans...
      2017-12-16 00:08:50.251
      2017-12-15 23:53:00
      1741886
      1850.97
      2017-12-16T02:04:09.495Z
      457275
      1017806
      266805
      The Washington Post
      washingtonpost_com
      0
      3749
      89.00
      True
      6099192.00
      2017-12-16T00:27:14.000Z
      191
      766.00

```python
data.fb_engagements.mode()
```

    0    0
    dtype: int64

*last month* Going back to the enagement counts, we see the mean is 1,117, mode is zero, median is 24, 90th percentile is 1,453, 99th percentile is 21,166, 99.5th percentile is 33,982. The standard deviation is 8,083, significantly higher than the mean, so this is not a normal distribution.

Going back to the enagement counts, we see the mean is 1,106, mode is zero, median is 24, 90th percentile is 1,545, 99th percentile is 20,228, 99.5th percentile is 32,446. The standard deviation is 9,852, significantly higher than the mean, so this is not a normal distribution.

Key publishers stats

```python
data.groupby("publisher_id").agg({'url': 'count', 'fb_engagements': ['sum', 'median', 'mean']})
```

      url
      fb_engagements

      count
      sum
      median
      mean

      publisher_id

      anotherangryvoice_blogspot_co_uk
      20
      40120
      1809.00
      2006.00

      bbc_co_uk
      10821
      7646802
      28.00
      706.66

      breitbart_com
      2640
      11722200
      270.50
      4440.23

      brexitcentral_com
      45
      24409
      188.00
      542.42

      buzzfeed_com
      1182
      2772469
      165.00
      2345.57

      cnn_com
      3085
      14990814
      638.00
      4859.26

      dailymail_co_uk
      22066
      15935337
      29.00
      722.17

      economist_com
      340
      90374
      32.00
      265.81

      evolvepolitics_com
      39
      274563
      1592.00
      7040.08

      foxnews_com
      5043
      15670336
      72.00
      3107.34

      ft_com
      3870
      268372
      5.00
      69.35

      huffingtonpost_com
      8947
      10716052
      9.00
      1197.73

      independent_co_uk
      5718
      4911814
      31.00
      859.01

      indy100_com
      500
      290099
      52.00
      580.20

      lemonde_fr
      3644
      2019719
      82.50
      554.26

      libdemvoice_org
      142
      1988
      8.00
      14.00

      mirror_co_uk
      9514
      6559353
      54.50
      689.44

      nbcnews_com
      1793
      5202028
      446.00
      2901.30

      newstatesman_com
      362
      61350
      23.00
      169.48

      npr_org
      1878
      7021350
      239.50
      3738.74

      nytimes_com
      4541
      17519582
      215.00
      3858.09

      order-order_com
      170
      55692
      162.50
      327.60

      propublica_org
      55
      71414
      529.00
      1298.44

      reuters_com
      5213
      1842218
      20.00
      353.39

      rt_com
      2572
      2600470
      264.00
      1011.07

      skwawkbox_org
      102
      68492
      263.50
      671.49

      telegraph_co_uk
      6285
      2283897
      18.00
      363.39

      thecanary_co
      184
      220508
      609.00
      1198.41

      theguardian_com
      7662
      9020836
      122.00
      1177.35

      thetimes_co_uk
      8869
      338754
      1.00
      38.20

      washingtonpost_com
      21217
      12472172
      0.00
      587.84

      westmonster_com
      274
      371790
      51.50
      1356.90

      yournewswire_com
      352
      792901
      165.50
      2252.56

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
eng = data.fb_engagements[(data.fb_engagements < 5000)]
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(15,7))
plt.hist(eng, bins=50)
plt.title("Article count by engagements")
plt.axvline(median, label=f'Median ({median:,.0f})', color='green')
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(ninety, linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(ninetyfive, linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
# plt.axvline(ninetynine, linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   112066.00
    mean         4.99
    std          1.77
    min          3.22
    25%          3.50
    50%          4.41
    75%          6.06
    max         11.38
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   112066.00
    mean        11.33
    std         10.89
    min          0.00
    25%          2.00
    50%          8.00
    75%         18.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   139145.00
    mean        10.25
    std        107.35
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        305.00
    99.5%      594.00
    99.9%     1359.00
    max      13930.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    4013.00
    mean      355.24
    std       526.37
    min         4.00
    25%        94.00
    50%       190.00
    75%       425.00
    90%       888.20
    95%      1160.60
    99%      1890.20
    99.5%    2723.16
    99.9%    5808.77
    max     13930.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    lemonde_fr                             618
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      20.00
      5.31
      0.10
      4.88
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      106.00
      12.80
      1.80
      1.97
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      197.00
      8.43
      1.19
      0.72
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      39.00
      4.45
      0.57
      1.91
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      275.00
      11.80
      1.16
      1.80
      12.02
      12.02
      12.02
      12.02

      cnn_com
      209.00
      12.57
      1.37
      2.15
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      171.00
      11.57
      1.22
      1.98
      11.85
      11.85
      11.85
      11.85

      economist_com
      37.00
      7.66
      1.44
      0.53
      7.99
      7.99
      7.99
      7.99

      foxnews_com
      105.00
      10.55
      0.62
      4.25
      10.61
      10.61
      10.61
      10.61

      ft_com
      69.00
      7.69
      1.54
      0.54
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      165.00
      11.00
      0.88
      5.40
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      134.00
      9.68
      1.51
      0.84
      10.07
      10.07
      10.07
      10.07

      indy100_com
      166.00
      5.79
      1.86
      0.47
      4.69
      7.04
      7.04
      7.04

      lemonde_fr
      184.00
      8.59
      1.93
      0.78
      9.34
      9.34
      9.34
      9.34

      libdemvoice_org
      135.00
      4.58
      0.55
      0.78
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      333.00
      8.51
      1.80
      0.61
      9.15
      9.15
      9.15
      9.15

      nbcnews_com
      111.00
      8.45
      1.67
      0.60
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      55.00
      6.01
      1.23
      1.48
      6.35
      6.35
      6.35
      6.35

      npr_org
      135.00
      9.07
      1.27
      1.57
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      59.00
      12.25
      1.57
      2.09
      12.53
      12.53
      12.53
      12.53

      order-order_com
      168.00
      4.75
      1.52
      0.48
      3.83
      5.78
      5.78
      5.78

      propublica_org
      22.00
      5.96
      1.10
      1.03
      6.20
      6.20
      6.20
      6.20

      reuters_com
      87.00
      9.11
      1.84
      0.81
      9.66
      9.66
      9.66
      9.66

      rt_com
      162.00
      9.29
      2.39
      0.68
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      102.00
      4.76
      0.79
      0.75
      5.03
      5.03
      5.03
      5.03

      telegraph_co_uk
      87.00
      9.78
      1.42
      2.37
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      148.00
      4.63
      1.81
      0.41
      3.11
      5.38
      6.21
      6.21

      theguardian_com
      157.00
      11.48
      2.11
      1.01
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      77.00
      6.52
      1.24
      1.03
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      83.00
      11.30
      1.15
      0.95
      11.42
      11.42
      11.42
      11.42

      westmonster_com
      74.00
      4.68
      1.26
      0.78
      5.22
      5.22
      5.22
      5.22

      yournewswire_com
      141.00
      5.57
      0.81
      2.69
      5.59
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.41405622016623894, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     135133
    1.00         46
    2.00         40
    3.00         47
    4.00         45
    5.00         51
    6.00         53
    7.00        230
    8.00        224
    9.00        245
    10.00       146
    11.00       189
    12.00        61
    13.00        81
    14.00       581
    15.00       363
    16.00       345
    17.00       266
    18.00       244
    19.00       402
    20.00       353
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      14.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  0.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    lemonde_fr                         15.00
    libdemvoice_org                     7.00
    mirror_co_uk                       14.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     8.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    2.4500000000000002

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     65696
    2.00      6558
    3.00      4381
    4.00      4043
    5.00       713
    6.00       467
    7.00       491
    8.00       446
    9.00       947
    10.00      205
    11.00      192
    12.00      387
    13.00      215
    14.00       88
    15.00       68
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk      20.00
    bbc_co_uk                          14840.00
    breitbart_com                       2576.00
    brexitcentral_com                     45.00
    buzzfeed_com                        9269.00
    cnn_com                            11015.00
    dailymail_co_uk                    12847.00
    economist_com                        196.00
    evolvepolitics_com                    37.00
    foxnews_com                         6244.00
    ft_com                              2475.00
    huffingtonpost_com                  7800.00
    independent_co_uk                   4067.00
    indy100_com                          499.00
    lemonde_fr                          3639.00
    libdemvoice_org                      142.00
    mirror_co_uk                        9036.00
    nbcnews_com                         1715.00
    newstatesman_com                     358.00
    npr_org                             2406.00
    nytimes_com                         9063.00
    order-order_com                      170.00
    propublica_org                        55.00
    reuters_com                         6295.00
    rt_com                              4391.00
    skwawkbox_org                        102.00
    telegraph_co_uk                     4161.00
    thecanary_co                         183.00
    theguardian_com                    12282.00
    thetimes_co_uk                      8832.00
    washingtonpost_com                  8507.00
    westmonster_com                      256.00
    yournewswire_com                     352.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    ddd1ccb2bc026b5b44204e5f4f08de1d75d4280a   1.00
    d8409c3f40ce732ab4917e88f3991fba6e34052c   1.00
    8ec1e46d261cde4b7eb7818481e02a439a344966   1.00
    a0b6c78af7ce197a41eb7f2e8514929e40cf0f01   1.00
    d6b3d8608c147417d6f83042303ad18d48df5069   1.00
    617651116dbfa17ce3192113309e3fc36c653bb7   1.00
    895b257635c005e56f8c33804d1e1b15adafa4ac   1.00
    aaf6aea336edb397ab00ba5b07afa76a7b5a62f0   1.00
    8669975be98765a81a7c563c33cc8477778b2d96   1.00
    d1689b6457ebc3c471a836aa2e1638e54635f09d   1.00
    7864cbd80daceedb03462a2e8acc99af0af843eb   1.00
    a0000c6cbabd41b73833f2e03d737a65f0310192   1.00
    46cf09d0abaebe8e43560dd298a324bbf3061925   1.00
    b68f6aea1926d28ad82dd59c88f6afb048ad50c3   1.00
    0459ed970d4bf508bfe956f838175ca95a26e63d   1.00
    7a32f7ef1be604c817442af980c9caea6c554728   1.00
    6c5f507bcf7468a5958705907bc57bfcf07189ff   1.00
    a19c0121b70fa21942d150f3df8ae94414e9a929   1.00
    a4aac0567b150accb92fce3b239d99428e63964d   1.00
    7feb3b9ca2006c3ba3ed1b500785a828efd661e3   1.00
    f4847375127b100848f654f00665aa39471b4966   1.00
    411003c86ad9dcfae6bbae44563abfc3ae30ee68   1.00
    457fcb987305dcf9c5494ec3769ec4c8810ed666   1.00
    7e061ce50c3d37a8ff7f24cbfe03cbb4b59196b1   1.00
    c9571e9beb0d861bbe4dd6a84e53c3c06ab11e38   1.00
    34d49d78a8945fd1a66b05a74cc914a839fe8ccf   1.00
    f36ba216b4dee0e67e94d5a65c22248b1d6bab53   1.00
    6a579d2a5b6a597aceb38ce26146c45badc25e84   1.00
    b5c1335d4e30cffb2fe1068ef2cb1033ae3e5737   1.00
    bfe41b70413238cac3c7507812d245ac0602cdaa   1.00
                                               ...
    bfae49c09de2767f82e72a12f5b12764c1ce7820   0.00
    70c0842823086bd113b46a4e59d65ddb65fffbe4   0.00
    8c411f2a9c58264e5bd070efb6571e3a1fdf1c3c   1.00
    7877d1f78013c05e3f27af8397db90cad54dfbda   1.00
    300a44e0beba24a2e51dffdf9836a2dfc85b9476   1.00
    e870230a01a0f4644b05730e10e951f40dbead3b   0.00
    56f96111213c53307bfc938a13a93e91ca279a82   1.00
    1613e5ee04e452487058b1a366aa1f043ac083cd   0.00
    8974f3e2076e9f7a603d83f217618daa039593c2   0.00
    ef76454179a8471433f96bfa479afecc10ba135c   0.00
    1e4c5f31a5ad70b5d95d0c2dbcb1b00b7c015f93   1.00
    a05a77d7acbb4a660e89f9d756e736bf7d9ef44d   6.00
    2550727576c66a34027959450c0bbbd4294af766   4.00
    f8160c9f615f4454fcd667b0013e108e76344e68   0.00
    9deb9c716841e1238e8fdbcf930ad61ad6ef7543   0.00
    6d429eb2b256224d88dc65c7cec8f30223d5e513   0.00
    5ecf4d8ee26077a04e67b131825596003b62c37c   0.00
    bdbf38c1bd2d8b34a4dd30bfb23d06a2bbbda741   1.00
    dd188e1f4c1e9d3c52436bab86dccc632b7e976a   0.00
    10bc19ceb4744dadeebb54b131f00b27d523dc6e   0.00
    b84a4946cbd1485cbd093be665d701c2c7715e6b   0.00
    dea7aa4d75a3ab52308f89b2afa370e18ae252d8   2.00
    d49f5492306ecdf8b0584aeb4ca6e5366101bfeb   1.00
    95869409b267dd18f18b99bd3587fcc37777443c   3.00
    f4eafe68c3ab197dfe036046a04c28b3d5ecac65   0.00
    ffd528fd949229fa7684ac6bc966c8d3fc0bc687   1.00
    10357eed59889a5ee0da5195c5cd6b110e4e31cf   0.00
    fcb700f805f54fa7b8468b8008881115917018eb   1.00
    af42fc6be7c3e331ad9831c18406ffebdf49128e   0.00
    9869d24d1829f06aa84b50d93e2ba029f28bd514   0.00
    Name: front_score, Length: 139145, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    45331565.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      20.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      bbc_co_uk
      10821.00
      0.63
      3.00
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2640.00
      0.85
      0.99
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      45.00
      0.91
      0.29
      0.00
      1.00
      1.00
      1.00
      1.00

      buzzfeed_com
      1182.00
      0.54
      0.50
      0.00
      0.00
      1.00
      1.00
      1.00

      cnn_com
      3085.00
      3.07
      4.61
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      22066.00
      0.61
      1.64
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      340.00
      2.44
      1.17
      0.00
      3.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      39.00
      0.95
      0.22
      0.00
      1.00
      1.00
      1.00
      1.00

      foxnews_com
      5043.00
      0.87
      2.11
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      3870.00
      0.48
      0.85
      0.00
      0.00
      0.00
      0.00
      2.00

      huffingtonpost_com
      8947.00
      0.47
      1.28
      0.00
      0.00
      0.00
      0.00
      4.00

      independent_co_uk
      5718.00
      0.60
      1.20
      0.00
      0.00
      0.00
      0.00
      3.00

      indy100_com
      500.00
      0.69
      0.46
      0.00
      0.00
      1.00
      1.00
      1.00

      lemonde_fr
      3644.00
      0.80
      0.98
      0.00
      0.00
      0.00
      2.00
      2.00

      libdemvoice_org
      142.00
      0.95
      0.22
      0.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      9514.00
      0.27
      0.44
      0.00
      0.00
      0.00
      1.00
      1.00

      nbcnews_com
      1793.00
      2.29
      1.98
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      362.00
      0.79
      0.41
      0.00
      1.00
      1.00
      1.00
      1.00

      npr_org
      1878.00
      1.42
      1.50
      0.00
      0.00
      0.00
      3.00
      3.00

      nytimes_com
      4541.00
      1.62
      2.42
      0.00
      0.00
      0.00
      5.00
      6.00

      order-order_com
      170.00
      0.76
      0.43
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      55.00
      0.76
      0.43
      0.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      5213.00
      0.56
      0.90
      0.00
      0.00
      0.00
      2.00
      2.00

      rt_com
      2572.00
      1.03
      1.00
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      102.00
      0.99
      0.10
      0.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      6285.00
      0.58
      0.91
      0.00
      0.00
      0.00
      2.00
      2.00

      thecanary_co
      184.00
      0.99
      0.10
      0.00
      1.00
      1.00
      1.00
      1.00

      theguardian_com
      7662.00
      0.52
      1.13
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      8869.00
      0.05
      0.22
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      21217.00
      0.18
      0.71
      0.00
      0.00
      0.00
      0.00
      3.00

      westmonster_com
      274.00
      0.29
      0.46
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      352.00
      0.10
      0.30
      0.00
      0.00
      0.00
      0.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     330509.00
    bbc_co_uk                          45331565.00
    breitbart_com                       3770975.00
    brexitcentral_com                     13487.00
    buzzfeed_com                        2826049.00
    cnn_com                            29374374.00
    dailymail_co_uk                    13812505.00
    economist_com                       8415152.00
    evolvepolitics_com                   119272.00
    foxnews_com                        16099176.00
    ft_com                              3724840.00
    huffingtonpost_com                  9829303.00
    independent_co_uk                   7950318.00
    indy100_com                          233256.00
    lemonde_fr                          3976365.00
    libdemvoice_org                        8618.00
    mirror_co_uk                        2945691.00
    nbcnews_com                         9469485.00
    newstatesman_com                     154787.00
    npr_org                             6264453.00
    nytimes_com                        15161266.00
    order-order_com                       45291.00
    propublica_org                       375168.00
    reuters_com                         3936474.00
    rt_com                              4956977.00
    skwawkbox_org                          6461.00
    telegraph_co_uk                     4411426.00
    thecanary_co                         156579.00
    theguardian_com                     7832684.00
    thetimes_co_uk                       731034.00
    washingtonpost_com                  6107196.00
    westmonster_com                       16551.00
    yournewswire_com                      27641.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.71
    bbc_co_uk                          17.63
    breitbart_com                      15.14
    brexitcentral_com                   9.51
    buzzfeed_com                       14.85
    cnn_com                            17.20
    dailymail_co_uk                    16.44
    economist_com                      15.95
    evolvepolitics_com                 11.69
    foxnews_com                        16.59
    ft_com                             15.13
    huffingtonpost_com                 16.10
    independent_co_uk                  15.89
    indy100_com                        12.36
    lemonde_fr                         15.20
    libdemvoice_org                     9.06
    mirror_co_uk                       14.90
    nbcnews_com                        16.06
    newstatesman_com                   11.95
    npr_org                            15.65
    nytimes_com                        16.53
    order-order_com                    10.72
    propublica_org                     12.84
    reuters_com                        15.19
    rt_com                             15.42
    skwawkbox_org                       8.77
    telegraph_co_uk                    15.30
    thecanary_co                       11.96
    theguardian_com                    15.87
    thetimes_co_uk                     13.50
    washingtonpost_com                 15.62
    westmonster_com                     9.71
    yournewswire_com                   10.23
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.80
    bbc_co_uk                          10.72
    breitbart_com                       8.24
    brexitcentral_com                   2.60
    buzzfeed_com                        7.95
    cnn_com                            10.29
    dailymail_co_uk                     9.53
    economist_com                       9.04
    evolvepolitics_com                  4.78
    foxnews_com                         9.69
    ft_com                              8.22
    huffingtonpost_com                  9.19
    independent_co_uk                   8.98
    indy100_com                         5.45
    lemonde_fr                          8.29
    libdemvoice_org                     2.15
    mirror_co_uk                        7.99
    nbcnews_com                         9.16
    newstatesman_com                    5.04
    npr_org                             8.74
    nytimes_com                         9.63
    order-order_com                     3.81
    propublica_org                      5.93
    reuters_com                         8.28
    rt_com                              8.51
    skwawkbox_org                       1.87
    telegraph_co_uk                     8.39
    thecanary_co                        5.05
    theguardian_com                     8.97
    thetimes_co_uk                      6.59
    washingtonpost_com                  8.72
    westmonster_com                     2.81
    yournewswire_com                    3.32
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   4.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    lemonde_fr                         12.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       3.00
    telegraph_co_uk                    12.00
    thecanary_co                        8.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     4.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   3.00
    buzzfeed_com                       11.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  6.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         7.00
    lemonde_fr                         12.00
    libdemvoice_org                     2.00
    mirror_co_uk                       11.00
    nbcnews_com                        13.00
    newstatesman_com                    7.00
    npr_org                            12.00
    nytimes_com                        14.00
    order-order_com                     5.00
    propublica_org                      8.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        7.00
    theguardian_com                    13.00
    thetimes_co_uk                      9.00
    washingtonpost_com                 12.00
    westmonster_com                     3.00
    yournewswire_com                    4.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     23004
    1.00     17206
    2.00     10703
    3.00      7293
    4.00      5918
    5.00      4993
    6.00      3946
    7.00      3437
    8.00      3179
    9.00      2891
    10.00     2645
    11.00     2521
    12.00     2284
    13.00     2285
    14.00     2069
    15.00     1863
    16.00     1822
    17.00     1696
    18.00     1614
    19.00     1579
    20.00     1460
    21.00     1503
    22.00     1509
    23.00     1468
    24.00     1450
    25.00     1404
    26.00     1391
    27.00     1324
    28.00     1325
    29.00     1252
             ...
    65.00      154
    66.00      132
    67.00       78
    68.00       81
    69.00       82
    70.00       65
    71.00       42
    72.00       46
    73.00       53
    74.00       44
    75.00       27
    76.00       48
    77.00       39
    78.00       32
    79.00       38
    80.00       19
    81.00       16
    82.00       20
    83.00       28
    84.00       13
    85.00       17
    86.00        9
    87.00       12
    88.00       20
    89.00       12
    90.00        3
    91.00        4
    92.00        4
    93.00        2
    94.00        6
    Name: attention_index, Length: 95, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      63152b4a2aa41e82016aa771891fdf1333128985
      http://www.cnn.com/2017/12/05/politics/jeff-fl...
      GOP Sen. Jeff Flake writes check to Democrat o...
      2017-12-05 22:07:18.152
      2017-12-05 22:01:48.000
      115293
      563.96
      2017-12-05T22:49:09.264Z
      13890
      93202
      8201
      ...
      29256092.00
      2017-12-05T22:30:17.000Z
      105
      173.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      6654a151fcbef618eb610580d5448ad83303b5a8
      http://www.cnn.com/2017/12/16/health/cdc-banne...
      Word ban at CDC includes 'vulnerable,' 'fetus,...
      2017-12-16 16:04:22.124
      2017-12-16 16:01:09.000
      339451
      3406.06
      2017-12-16T22:04:03.814Z
      94158
      201048
      44245
      ...
      29319882.00
      2017-12-16T16:40:20.000Z
      105
      447.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      33d3d70eb4c8c44c81f87f7cfc62a61e8f051150
      http://www.cnn.com/2017/12/01/politics/senate-...
      Uncertainty looms over Senate's tax bill
      2017-12-01 06:16:20.820
      2017-12-01 06:11:21.000
      76401
      121.19
      2017-12-02T17:18:08.986Z
      24222
      43912
      8267
      ...
      29233578.00
      2017-12-02T07:34:25.000Z
      105
      962.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      abe03a536d43790cd2c89fbeb8d3f3953376454d
      http://www.cnn.com/2017/12/15/politics/judicia...
      Trump judicial nominee struggles to answer bas...
      2017-12-15 14:07:14.292
      2017-12-15 14:02:30.000
      108798
      157.13
      2017-12-16T02:46:13.544Z
      36967
      56279
      15552
      ...
      29318332.00
      2017-12-16T02:29:02.000Z
      105
      327.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      46d8c8a0e17496bd7af92576c33efb6c80327f22
      http://www.cnn.com/2017/12/06/us/ventura-fire-...
      Ventura fire: Southern California grapples wit...
      2017-12-06 08:55:34.234
      2017-12-06 08:53:45.000
      76140
      73.96
      2017-12-07T01:03:07.285Z
      18241
      48944
      8955
      ...
      29259398.00
      2017-12-06T11:00:22.000Z
      105
      810.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      49974d79caf78028259583d5b24b6a272c81ba16
      http://www.bbc.co.uk/news/world-middle-east-42...
      Jerusalem: UN resolution rejects Trump's decla...
      2017-12-21 17:25:06.643
      2017-12-21 17:21:33.000
      104052
      1024.29
      2017-12-21T18:18:13.675Z
      14367
      82849
      6836
      ...
      45205436.00
      2017-12-21T17:51:41.000Z
      96
      119.00
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      d479366e1dab01635b8321713b5122e26d0e1e43
      http://www.cnn.com/2017/12/10/politics/kfile-r...
      Roy Moore in 2011: Getting rid of amendments a...
      2017-12-10 23:10:12.827
      2017-12-10 23:06:56.000
      97289
      156.92
      2017-12-11T04:44:07.308Z
      23777
      56347
      17165
      ...
      29294730.00
      2017-12-10T23:30:04.000Z
      105
      1243.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      b23787d954cd054eafa4e6bf352d71dfb793df17
      http://www.cnn.com/2017/12/05/us/ventura-count...
      1 dead, hundreds evacuate in fast-moving South...
      2017-12-05 07:13:18.431
      2017-12-05 07:06:21.000
      86026
      79.91
      2017-12-05T14:14:04.504Z
      25060
      51634
      9332
      ...
      29253039.00
      2017-12-05T14:00:14.000Z
      105
      196.00
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      4fbd6316c740d293ad9c2adb45c8b3e73a2225ea
      http://www.bbc.co.uk/news/business-42353545
      Walt Disney buys Murdoch's Fox for $52bn
      2017-12-14 12:10:24.949
      2017-12-14 12:08:08.000
      40757
      214.01
      2017-12-14T12:31:24.627Z
      8386
      26851
      5520
      ...
      45140390.00
      2017-12-14T12:20:08.000Z
      96
      64.00
      46.00
      20.00
      11.00
      15.00
      46.00
      92.00

      454bb14b5b7017aae1cd27b5ba6fd159b00b5f6d
      http://www.bbc.co.uk/news/world-us-canada-4223...
      Supreme court backs Trump travel ban
      2017-12-04 21:22:10.709
      2017-12-04 21:19:04.000
      64007
      363.73
      2017-12-04T21:34:10.895Z
      9688
      49710
      4609
      ...
      45022998.00
      2017-12-04T21:23:49.000Z
      96
      67.00
      49.00
      20.00
      8.00
      15.00
      43.00
      92.00

      579a2932b11423a6064a13062f0cf4291501f041
      http://www.cnn.com/2017/12/04/politics/trump-m...
      Trump fully endorses Roy Moore
      2017-12-04 12:19:19.168
      2017-12-04 12:15:22.000
      68054
      139.41
      2017-12-04T14:07:15.216Z
      25100
      35731
      7223
      ...
      29245912.00
      2017-12-04T12:45:01.000Z
      105
      223.00
      49.00
      20.00
      8.00
      15.00
      43.00
      92.00

      6b9e111e87a519cfdf58f463da0827f8f426d668
      http://www.cnn.com/2017/12/17/us/atlanta-airpo...
      Atlanta's Hartsfield-Jackson airport crippled ...
      2017-12-17 19:46:16.423
      2017-12-17 19:42:18.000
      72981
      1472.54
      2017-12-18T02:05:07.404Z
      23851
      36683
      12447
      ...
      29322298.00
      2017-12-17T21:01:26.000Z
      105
      173.00
      49.00
      20.00
      8.00
      15.00
      43.00
      92.00

      4bc906b992d5dafee16bef7f580b3b3d438e329e
      http://www.bbc.co.uk/news/world-us-canada-4233...
      Democrat Doug Jones projected to defeat Roy Mo...
      2017-12-13 03:37:11.629
      2017-12-13 03:34:12.000
      57498
      718.45
      2017-12-13T04:11:10.090Z
      6585
      45624
      5289
      ...
      45130284.00
      2017-12-13T03:56:33.000Z
      96
      73.00
      48.00
      20.00
      8.00
      15.00
      43.00
      91.00

      63fa3fe45932328190916f1d4a8a98e683dae037
      http://www.cnn.com/2017/12/21/politics/haley-u...
      Haley tells UN: Jerusalem vote 'will be rememb...
      2017-12-21 16:10:15.550
      2017-12-21 16:06:13.000
      79027
      277.93
      2017-12-21T19:17:18.583Z
      24369
      46433
      8225
      ...
      29337642.00
      2017-12-21T19:02:11.000Z
      105
      176.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      51de2b073b9e2a4e399682bfb6618ff3e05843e8
      https://www.buzzfeed.com/juliareinstein/turner...
      Brock Turner Wants To Overturn His Sexual Assa...
      2017-12-02 22:04:22.377
      2017-12-02 21:48:03.000
      48345
      45.92
      2017-12-03T01:29:09.818Z
      9638
      35158
      3549
      ...
      2793599.00
      2017-12-03T01:05:00.000Z
      147
      413.00
      47.00
      19.00
      14.00
      11.00
      44.00
      91.00

      6c09a9c935b40dc9eee1925a42ed82d1d3df037e
      http://www.cnn.com/2017/12/01/politics/michael...
      Flynn charged with one count of making false s...
      2017-12-01 14:19:32.564
      2017-12-01 14:16:50.000
      148032
      1214.16
      2017-12-01T22:04:10.122Z
      48459
      79607
      19966
      ...
      29230687.00
      2017-12-01T14:20:28.000Z
      105
      32.00
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      8d343bda0de91991a08cc37fe49e8f6a6223bd9a
      http://www.cnn.com/2017/12/12/politics/alabama...
      Alabama election: Roy Moore, Doug Jones vote
      2017-12-12 20:25:16.987
      2017-12-12 20:21:05.000
      131891
      750.91
      2017-12-13T04:38:10.262Z
      16392
      105180
      10319
      ...
      29308172.00
      2017-12-13T04:30:14.000Z
      105
      504.00
      50.00
      20.00
      5.00
      15.00
      40.00
      90.00

      7f49688d72b8a9fcf37890e3b9921d7264cc86fb
      http://www.cnn.com/2017/12/28/politics/trump-g...
      Trump tweets that 'cold' East Coast 'could use...
      2017-12-29 01:22:18.203
      2017-12-29 01:19:39.000
      55372
      186.73
      2017-12-29T02:47:06.680Z
      21668
      28990
      4714
      ...
      29357992.00
      2017-12-29T02:30:03.000Z
      105
      548.00
      48.00
      20.00
      7.00
      15.00
      42.00
      90.00

      cd9e7316c9cb90a1bd629a32e85e7c61beaa0ea9
      http://www.cnn.com/2017/12/08/politics/roy-moo...
      Roy Moore's incredible 'even though we had sla...
      2017-12-08 16:07:24.145
      2017-12-08 16:03:52.000
      58332
      799.25
      2017-12-08T20:05:15.464Z
      18324
      32060
      7948
      ...
      29282563.00
      2017-12-08T16:09:38.000Z
      105
      697.00
      48.00
      20.00
      7.00
      15.00
      42.00
      90.00

      0a7a0ee92a87f1b5c0d92ae0bbc22e52607c0f01
      https://www.huffingtonpost.com/entry/senate-pa...
      Senate Passes Tax Cuts For The Rich
      2017-12-02 06:58:11.961
      2017-12-02 06:52:32.000
      334487
      383.56
      2017-12-02T16:01:08.036Z
      71795
      232203
      30489
      ...
      9812652.00
      2017-12-02T06:57:19.000Z
      215
      1482.00
      50.00
      17.00
      9.00
      13.00
      39.00
      89.00

      f3980d577d96f300046cb4a27181f1e006a003d8
      http://www.bbc.co.uk/news/world-us-canada-4220...
      US Senate passes sweeping tax overhaul bill
      2017-12-02 06:55:08.770
      2017-12-02 06:51:32.000
      19240
      46.62
      2017-12-02T08:21:05.294Z
      4377
      12733
      2130
      ...
      45006183.00
      2017-12-02T08:05:33.000Z
      96
      539.00
      41.00
      20.00
      13.00
      15.00
      48.00
      89.00

      d220f05a70f23435eab05889ca8d55566c02b0d2
      http://www.cnn.com/2017/12/03/politics/flynn-f...
      Trump's lawyer says he was behind @realDonaldT...
      2017-12-03 17:13:26.205
      2017-12-03 17:08:43.000
      46289
      127.72
      2017-12-03T18:36:13.543Z
      21157
      19761
      5371
      ...
      29242216.00
      2017-12-03T18:00:09.000Z
      105
      375.00
      47.00
      20.00
      7.00
      15.00
      42.00
      89.00

      64cc07af65787b72659c9d657f3b9b10c73ad3a8
      http://www.cnn.com/2017/12/04/politics/trump-j...
      Trump's personal lawyer claims the President c...
      2017-12-04 15:13:15.175
      2017-12-04 15:08:42.000
      39569
      143.76
      2017-12-04T16:28:09.312Z
      15364
      18413
      5792
      ...
      29247000.00
      2017-12-04T16:16:01.000Z
      105
      383.00
      46.00
      20.00
      8.00
      15.00
      43.00
      89.00

      709cfecc8cbf26f5fdaae8eca2970004cc3f78c3
      http://www.cnn.com/2017/12/26/us/erie-pennsylv...
      With 53 inches, Erie breaks Pennsylvania's two...
      2017-12-26 16:58:20.633
      2017-12-26 16:51:20.000
      38825
      76.62
      2017-12-26T19:22:08.868Z
      7071
      26834
      4920
      ...
      29354093.00
      2017-12-26T19:00:30.000Z
      105
      362.00
      46.00
      20.00
      8.00
      15.00
      43.00
      89.00

      df3abea4bb6331dcb57dfe4f7e9b2f8a59fcdb60
      http://www.cnn.com/2017/12/23/politics/donald-...
      WH denies Trump said Haitian immigrants 'all h...
      2017-12-23 19:19:24.386
      2017-12-23 19:15:10.000
      41299
      189.14
      2017-12-23T20:13:12.024Z
      21076
      15543
      4680
      ...
      29344064.00
      2017-12-23T20:00:12.000Z
      105
      329.00
      46.00
      20.00
      8.00
      15.00
      43.00
      89.00

      05a25081fd942b7a42156ab4d38cd1c53ff754df
      https://www.huffingtonpost.com/entry/trump-jud...
      Trump Judicial Nominee Can't Answer Basic Ques...
      2017-12-15 07:59:09.559
      2017-12-15 07:47:05.000
      141324
      195.94
      2017-12-15T16:09:11.998Z
      47819
      73532
      19973
      ...
      9819968.00
      2017-12-15T12:30:21.000Z
      215
      415.00
      50.00
      17.00
      9.00
      13.00
      39.00
      89.00

      b8a53fdd9cc4107e4900e166ff97cb7c42988ec9
      http://www.bbc.co.uk/news/world-us-canada-4219...
      Flynn charged for 'making false statement'
      2017-12-01 14:22:06.935
      2017-12-01 14:18:35.000
      24302
      169.26
      2017-12-01T14:33:12.629Z
      5169
      15786
      3347
      ...
      44999420.00
      2017-12-01T14:24:45.000Z
      96
      64.00
      43.00
      20.00
      11.00
      15.00
      46.00
      89.00

      5a7d8d46133d592cab3c0b75f6156ac752792f64
      https://www.buzzfeed.com/emmaloop/the-senates-...
      The Senate’s Russia Investigation Is Now Looki...
      2017-12-18 21:55:21.465
      2017-12-18 21:48:41.000
      51571
      99.85
      2017-12-18T22:51:11.460Z
      19301
      25483
      6787
      ...
      2811436.00
      2017-12-18T22:07:36.000Z
      147
      739.00
      47.00
      19.00
      12.00
      11.00
      42.00
      89.00

      072d4c81fd2c28c098ee4dbf783bd91603fd41f4
      http://www.cnn.com/2017/12/06/politics/preside...
      Trump set to make historic Jerusalem announcement
      2017-12-06 17:52:22.847
      2017-12-06 17:46:10.000
      79082
      378.95
      2017-12-06T18:35:12.151Z
      29100
      43531
      6451
      ...
      29263371.00
      2017-12-06T18:18:47.000Z
      105
      202.00
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      1280bd6134195750988f31f51812f0df5b17a8d6
      http://www.bbc.co.uk/news/uk-politics-42277040
      Deal agreed in Brexit talks - Juncker
      2017-12-08 06:52:06.829
      2017-12-08 06:49:03.000
      20639
      53.36
      2017-12-08T08:08:08.346Z
      7353
      10331
      2955
      ...
      45059591.00
      2017-12-08T06:53:25.000Z
      96
      66.00
      42.00
      20.00
      12.00
      15.00
      47.00
      89.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      acf7e0fc60442a0153833d21fd007f8c1fdf29e7
      https://www.huffingtonpost.com/entry/the-marke...
      The Marketing Industry's $7.2-Billion Dollar P...
      2017-12-11 14:19:20.222
      2017-12-11 14:13:59.085
      0
      0.00
      2017-12-11T17:33:15.998Z
      0
      0
      0
      ...
      nan
      NaN
      215
      1006.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      48c4d5bb19ce559ab0a9c52f6c8acaf3b42f0c88
      http://www.bbc.co.uk/sport/football/42376184
      Albion Rovers v East Fife
      2017-12-23 16:01:23.302
      2017-12-23 15:58:48.000
      0
      0.00
      2017-12-24T16:35:05.847Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0714370ea5ec336030a82fe9a4358cb385f3bdd5
      http://www.bbc.co.uk/news/av/stories-42313332/...
      Meet Me On The Bridge
      2017-12-11 14:10:11.907
      2017-12-11 14:07:08.000
      0
      0.00
      2017-12-11T17:24:13.872Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      eefe920607837a80539c8e5982adcfd80dd0ca09
      http://www.bbc.co.uk/sport/football/42376186
      Airdrieonians v Ayr United
      2017-12-23 16:01:15.338
      2017-12-23 15:58:37.000
      1
      0.02
      2017-12-23T17:13:15.272Z
      0
      0
      1
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c6244d2a5b121292baccf4a1e1a73fc3d766b44c
      http://www.telegraph.co.uk/travel/destinations...
      Holualoa Inn
      2017-12-11 14:01:14.396
      2017-12-11 13:57:32.000
      0
      0.00
      2017-12-11T17:16:06.570Z
      0
      0
      0
      ...
      nan
      NaN
      370
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      cfe1b0664355bdc3abf78f09e22abd92b86ce1b2
      http://www.bbc.co.uk/sport/football/42376114
      Dunfermline Athletic v Brechin City
      2017-12-23 16:01:22.692
      2017-12-23 15:57:48.000
      0
      0.00
      2017-12-24T16:35:05.841Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ea6f934c295029edbfc5d89797912deced2dbee4
      http://www.bbc.co.uk/sport/football/42376118
      Dundee United v Livingston
      2017-12-23 16:01:08.777
      2017-12-23 15:57:37.000
      0
      0.00
      2017-12-24T16:34:10.823Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      d7d5cd31f3a1a161d2068ae872575bf4b4646eaf
      https://www.huffingtonpost.com/entry/how-to-ha...
      How to Hack Networking
      2017-12-23 16:04:22.715
      2017-12-23 15:56:49.939
      0
      0.00
      2017-12-24T16:36:10.390Z
      0
      0
      0
      ...
      nan
      NaN
      215
      1606.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c16bce03679e384220eec2e958c262ed90f3247e
      https://www.huffingtonpost.com/entry/we-photog...
      We Photographed A Magical Underground World Ca...
      2017-12-11 14:01:21.698
      2017-12-11 14:00:31.689
      0
      0.00
      2017-12-11T16:15:09.192Z
      0
      0
      0
      ...
      nan
      NaN
      215
      127.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1d3f75dd24e566ce43d85a9eb2fc4dab89f1b16a
      https://www.independent.co.uk/life-style/insta...
      23 incredible photos by daredevil Instagrammer...
      2017-12-11 14:13:23.182
      2017-12-11 14:00:32.000
      0
      0.00
      2017-12-11T16:27:11.268Z
      0
      0
      0
      ...
      nan
      NaN
      386
      510.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      45fdcd86028597cdd7b36adcef0fac0c1dc01180
      https://www.huffingtonpost.com/entry/kim-field...
      Kim Fields Goes Behind the Scenes with 'Blesse...
      2017-12-11 14:19:32.824
      2017-12-11 14:02:38.133
      0
      0.00
      2017-12-11T17:33:16.170Z
      0
      0
      0
      ...
      nan
      NaN
      215
      1327.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      9ad1b443d52bdd29a976b5b6d9be1dba3f1f2809
      http://www.bbc.co.uk/news/uk-scotland-edinburg...
      Men charged over knifepoint robbery in Edinburgh
      2017-12-11 14:10:08.312
      2017-12-11 14:05:49.000
      0
      0.00
      2017-12-11T17:24:13.869Z
      0
      0
      0
      ...
      nan
      NaN
      96
      173.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0e257747adf85b01a259b2aea687fdbeb3cd46c5
      http://www.bbc.co.uk/news/uk-northern-ireland-...
      Drugs and cash stolen from pharmacy in south B...
      2017-12-23 15:55:06.436
      2017-12-23 15:52:46.000
      0
      0.00
      2017-12-24T16:28:13.817Z
      0
      0
      0
      ...
      nan
      NaN
      96
      101.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f71064283aec021662857a374c050b510e9b9d70
      http://www.washingtonpost.com/video/world/pale...
      Palestinians clash with Israeli security force...
      2017-12-23 16:01:18.223
      2017-12-23 15:51:26.000
      0
      0.00
      2017-12-24T16:35:04.135Z
      0
      0
      0
      ...
      nan
      NaN
      191
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      d2a7afef4a59bb20421a317e6738d2af68407266
      https://www.washingtonpost.com/national/liquor...
      Liquor store owner, worker, suspect shot to de...
      2017-12-23 16:01:20.688
      2017-12-23 15:51:25.000
      0
      0.00
      2017-12-24T16:35:05.827Z
      0
      0
      0
      ...
      nan
      NaN
      191
      134.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2f943ac17e96bd307577f7ab44400021b891811f
      http://www.telegraph.co.uk/business/2017/12/11...
      Founder steps up war of words with Van Elle bo...
      2017-12-11 14:10:07.514
      2017-12-11 14:07:19.000
      1
      0.02
      2017-12-11T15:22:14.172Z
      0
      0
      1
      ...
      nan
      NaN
      370
      402.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f4e53fb2256515ae5d8a853f8625e74cddcb3079
      https://www.huffingtonpost.com/entry/the-year-...
      The Year I Almost Ruined Christmas for My Son
      2017-12-11 14:19:27.046
      2017-12-11 14:13:21.139
      0
      0.00
      2017-12-11T17:33:16.047Z
      0
      0
      0
      ...
      nan
      NaN
      215
      990.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      08077ace1da9ba7ae34b6dabe65d27bd51bd9711
      http://www.bbc.co.uk/news/av/stories-42313330/...
      The school for bullied and traumatised children
      2017-12-11 14:10:14.944
      2017-12-11 14:07:28.000
      1
      0.02
      2017-12-11T15:22:14.191Z
      0
      0
      1
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      681670f0206e9855091aae54978a776aaf2dcfb6
      https://www.huffingtonpost.com/entry/how-the-v...
      How The Visual Internet Influences Advertising
      2017-12-11 14:19:29.017
      2017-12-11 14:08:05.522
      1
      0.02
      2017-12-11T15:31:14.886Z
      0
      0
      1
      ...
      nan
      NaN
      215
      620.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e3533a2d1883674641f147866cf247fa36b97bf8
      https://www.washingtonpost.com/national/man-co...
      Man convicted of sexually assaulting sleeping ...
      2017-12-23 15:58:13.359
      2017-12-23 15:47:16.000
      0
      0.00
      2017-12-24T16:32:02.911Z
      0
      0
      0
      ...
      nan
      NaN
      191
      127.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b8146c1503518b775eff4e15c6583841307e8efa
      http://www.bbc.co.uk/news/av/stories-42313327/...
      The teachers who pay cash to keep their pupils...
      2017-12-11 14:10:14.189
      2017-12-11 14:08:12.000
      0
      0.00
      2017-12-11T17:24:13.889Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1259575fab7f5a1f5ec1895d453e9a5155b7a7ae
      https://www.huffingtonpost.com/entry/a-christm...
      A Christmas Story for Grown-Ups
      2017-12-23 15:49:21.255
      2017-12-23 15:45:03.913
      0
      0.00
      2017-12-24T16:23:09.715Z
      0
      0
      0
      ...
      nan
      NaN
      215
      730.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      997cde166f1d60c9586c66f133a77d984a333775
      https://www.washingtonpost.com/sports/dcunited...
      Milan paired with Ludogorets in last 32 of Eur...
      2017-12-11 14:16:11.860
      2017-12-11 14:10:20.000
      0
      0.00
      2017-12-11T16:29:13.683Z
      0
      0
      0
      ...
      nan
      NaN
      191
      149.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      23805de5378971233a9e11fc63e47dbf791525f8
      https://www.huffingtonpost.com/entry/telcoin-i...
      Telcoin is the Mobile Cryptocurrency Solution ...
      2017-12-11 14:19:28.664
      2017-12-11 14:11:45.542
      0
      0.00
      2017-12-11T17:33:16.136Z
      0
      0
      0
      ...
      nan
      NaN
      215
      846.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      77a14bd9632990ab6211642cdfd4368965becbbb
      http://www.bbc.co.uk/sport/cricket/42312025
      Ashes: Jonny Bairstow says there is 'a lot of ...
      2017-12-11 14:13:12.688
      2017-12-11 14:11:55.000
      0
      0.00
      2017-12-11T16:26:16.039Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0298beaac81177795390ec51422e1e86bfb56e1f
      https://www.washingtonpost.com/business/teache...
      Teacher cuts finger, still wins Food Network c...
      2017-12-23 15:52:17.903
      2017-12-23 15:40:14.000
      0
      0.00
      2017-12-24T16:26:11.381Z
      0
      0
      0
      ...
      nan
      NaN
      191
      151.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      326e0be2dd3bf2e3198075ceb293048646f3e307
      http://www.independent.co.uk/news/long_reads/r...
      How an American road trip helped a family heal...
      2017-12-19 00:08:07.222
      2017-12-11 14:12:12.000
      0
      0.00
      2017-12-19T00:19:13.804Z
      0
      0
      0
      ...
      nan
      NaN
      386
      2299.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      939248ed8885691ea2dd8d8b4d647407133655af
      http://www.telegraph.co.uk/travel/destinations...
      Beaches Negril Resort & Spa
      2017-12-11 14:16:08.636
      2017-12-11 14:12:44.000
      0
      0.00
      2017-12-11T16:29:13.658Z
      0
      0
      0
      ...
      nan
      NaN
      370
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f0cbb47eae03db1557ba467b97dd3ca567f74f34
      http://www.telegraph.co.uk/travel/destinations...
      Hotel Belle Epoque
      2017-12-11 14:16:06.688
      2017-12-11 14:13:15.000
      0
      0.00
      2017-12-11T16:29:13.638Z
      0
      0
      0
      ...
      nan
      NaN
      370
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      9869d24d1829f06aa84b50d93e2ba029f28bd514
      http://www.bbc.co.uk/newsround/38878494
      Tennis player accidentally hits umpire in eye ...
      2018-01-01 00:04:19.034
      2017-12-31 23:59:59.000
      0
      0.00
      2018-01-01T00:15:14.296Z
      0
      0
      0
      ...
      nan
      NaN
      96
      nan
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

139145 rows × 26 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      60dde39afccfcabaf8cf2a8ed28a3d507020545b
      https://www.buzzfeed.com/kateaurthur/2018-tv-a...
      Everything You Need To Know About TV And Movie...
      2017-12-30 17:04:21.638
      2017-12-21 22:23:16
      1
      0.09
      2017-12-30T17:16:03.579Z
      0
      0
      1
      ...
      2017-12-30T20:22:25.000Z
      147
      nan
      0.00
      19.00
      15.00
      11.00
      45.00
      45.00
      45.00

      2f11fdaca32431287b0f6fba6e02e024cbbec99d
      https://www.buzzfeed.com/jtes/for-women-lookin...
      For Women Looking For Jobs After Prison, It Do...
      2017-12-07 13:01:23.490
      2017-12-05 19:01:09
      1
      0.09
      2017-12-07T13:13:14.909Z
      0
      0
      1
      ...
      2017-12-10T02:00:00.000Z
      147
      3739.00
      0.00
      19.00
      14.00
      11.00
      44.00
      44.00
      44.00

      8705fb3da20761c6674f087884089ba3068e63fa
      https://www.buzzfeed.com/nidhisubbaraman/nativ...
      Native Americans Were Anxious About Trump Head...
      2017-12-27 16:04:15.611
      2017-12-20 17:50:41
      4
      0.39
      2017-12-27T16:15:16.931Z
      0
      0
      4
      ...
      2017-12-27T16:19:01.000Z
      147
      1380.00
      1.00
      19.00
      15.00
      11.00
      45.00
      46.00
      44.00

      32034e45b629a27a33e035a77757e67a27c59d71
      https://www.buzzfeed.com/stephaniemlee/here-ar...
      Here Are 13 Tech Startups And Products That Fl...
      2017-12-27 18:13:24.230
      2017-12-21 23:52:53
      4
      0.36
      2017-12-27T18:25:11.821Z
      0
      0
      4
      ...
      2017-12-28T02:12:57.000Z
      147
      1260.00
      1.00
      19.00
      15.00
      11.00
      45.00
      46.00
      44.00

      eb5b52c981dd04408660f23967ba814262156a5f
      https://www.buzzfeed.com/charliewarzel/youtube...
      YouTubers Made Hundreds Of Thousands Off Of Bi...
      2017-12-11 15:46:21.876
      2017-12-08 22:35:56
      0
      0.00
      2017-12-11T15:57:19.321Z
      0
      0
      0
      ...
      2017-12-11T16:05:29.000Z
      147
      1916.00
      0.00
      19.00
      14.00
      11.00
      44.00
      44.00
      44.00

      24f5ef2ea5bb64cc3a2def2b55092a2ce5751de8
      https://www.buzzfeed.com/rosalindadams/how-a-g...
      How a Giant Psychiatric Hospital Company Tried...
      2017-12-27 16:40:28.231
      2017-12-22 23:17:31
      6
      0.46
      2017-12-27T16:52:08.743Z
      0
      0
      6
      ...
      2017-12-27T17:13:02.000Z
      147
      1831.00
      2.00
      19.00
      15.00
      11.00
      45.00
      47.00
      43.00

      a2c6e7241dd8ffcfad898e5d44f9b00873f4f637
      https://www.buzzfeed.com/monicamark/niger-secr...
      Inside The Botched Raid That Left Four US Sold...
      2017-12-09 15:58:19.223
      2017-12-07 15:22:08
      0
      0.00
      2017-12-09T16:10:11.079Z
      0
      0
      0
      ...
      2017-12-11T18:03:00.000Z
      147
      3392.00
      0.00
      19.00
      13.00
      11.00
      43.00
      43.00
      43.00

      851d61f0cda588c4405a4dd959b187bcca3edf3c
      https://www.buzzfeed.com/melissasegura/how-to-...
      How To Close Two Stubborn Murder Cases In One ...
      2017-12-28 20:46:16.651
      2017-12-22 22:51:14
      2
      0.18
      2017-12-28T20:58:09.837Z
      0
      0
      2
      ...
      2017-12-28T21:22:08.000Z
      147
      4647.00
      1.00
      19.00
      14.00
      11.00
      44.00
      45.00
      43.00

      aecd5ea40f84a071b2b98deb901d374a1af82bea
      https://www.buzzfeed.com/davidmack/heres-how-t...
      Here's How To Find Out How The Republican Tax ...
      2017-12-19 21:19:19.316
      2017-12-16 15:28:34
      0
      0.00
      2017-12-19T21:31:07.547Z
      0
      0
      0
      ...
      2017-12-19T23:44:50.000Z
      147
      nan
      0.00
      19.00
      13.00
      11.00
      43.00
      43.00
      43.00

      f9b56421c207cc659ad9d2b9f02e41a3e2235852
      https://www.buzzfeed.com/tasneemnashrulla/cali...
      Powerful Winds Are Pushing The Destructive Cal...
      2017-12-07 17:34:17.052
      2017-12-06 18:44:10
      5
      0.07
      2017-12-07T18:46:14.159Z
      0
      1
      4
      ...
      2017-12-07T19:33:00.000Z
      147
      15.00
      1.00
      19.00
      13.00
      11.00
      43.00
      44.00
      42.00

      6b7ae56b46b3c625f1be5c21c435edb5d44e6716
      https://www.buzzfeed.com/zahrahirji/us-climate...
      11 Ways The US Is No Longer Fighting Climate C...
      2017-12-29 17:24:22.641
      2017-12-19 15:13:10
      20
      0.00
      2017-12-29T17:36:12.253Z
      0
      5
      15
      ...
      2017-12-31T02:22:00.000Z
      147
      1161.00
      4.00
      19.00
      15.00
      11.00
      45.00
      49.00
      41.00

      9fa212fb91158bb2e772b47de5619f7910b7f704
      https://www.buzzfeed.com/patricksmith/how-the-...
      How The Brother Of A Victim Of The Manchester ...
      2017-12-24 07:01:19.403
      2017-12-22 13:10:34
      2
      0.18
      2017-12-24T07:13:08.102Z
      0
      0
      2
      ...
      2017-12-25T02:22:00.000Z
      147
      2347.00
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      54a867bda1060aefd4e1de41690360d03f7d738e
      https://www.buzzfeed.com/monicamark/a-pastor-f...
      A Pastor Found A Massive Diamond. Now He Hopes...
      2017-12-16 16:16:20.407
      2017-12-13 19:01:57
      0
      0.00
      2017-12-16T16:28:08.626Z
      0
      0
      0
      ...
      2017-12-16T19:54:00.000Z
      147
      1880.00
      0.00
      19.00
      11.00
      11.00
      41.00
      41.00
      41.00

      f26fbab786c63264349418944f20666059288a7d
      https://www.buzzfeed.com/marisacarroll/stories...
      15 Stories About Justice That Outraged Us In 2017
      2017-12-21 19:43:21.195
      2017-12-21 19:12:01
      7
      0.05
      2017-12-22T16:15:06.173Z
      3
      0
      4
      ...
      2017-12-26T13:33:16.000Z
      147
      nan
      2.00
      19.00
      13.00
      11.00
      43.00
      45.00
      41.00

      11446451655a07177c9ba67543d908b7b9934c2e
      https://www.buzzfeed.com/katenocera/rep-ruben-...
      Nevada Democrat Ruben Kihuen Won't Seek Reelec...
      2017-12-16 21:31:22.428
      2017-12-05 18:48:53
      2
      0.18
      2017-12-16T21:43:05.715Z
      0
      0
      2
      ...
      2017-12-16T21:46:46.000Z
      147
      700.00
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      13045f42cde8a48bbcadd8e7ce042e4ca090e7f3
      https://www.buzzfeed.com/keelyflaherty/xx-rand...
      15 Random, Weirdly Fascinating Facts About Our...
      2017-12-11 13:14:16.245
      2017-12-06 18:55:29
      1
      0.00
      2017-12-11T13:25:16.547Z
      0
      0
      1
      ...
      2017-12-17T03:35:01.000Z
      147
      345.00
      0.00
      19.00
      10.00
      11.00
      40.00
      40.00
      40.00

      d83709ffbc6b9cf3ae4462dbb57f06c4f4f1400b
      https://www.buzzfeed.com/sallytamarkin/weight-...
      13 Experts On Why They No Longer Recommend Diets
      2018-01-01 18:49:23.289
      2017-12-01 20:43:38
      6
      0.55
      2018-01-01T19:01:06.672Z
      0
      0
      6
      ...
      2018-01-02T06:51:25.000Z
      147
      3123.00
      2.00
      19.00
      12.00
      11.00
      42.00
      44.00
      40.00

      e9b5702b01be019dc5f0e67703b78663974a9d06
      https://www.buzzfeed.com/adolfoflores/trump-ha...
      Trump Has Tied "Chain Migration" To Terrorism....
      2017-12-31 20:40:20.698
      2017-12-29 20:02:50
      1
      0.09
      2017-12-31T20:52:07.486Z
      0
      0
      1
      ...
      2018-01-01T01:34:32.000Z
      147
      1903.00
      0.00
      19.00
      10.00
      11.00
      40.00
      40.00
      40.00

      3c7cfda35f965505ba968c777ab7a31f162f608a
      https://www.buzzfeed.com/laurageiser/these-pho...
      These Photos Of Santas-In-Training Will Lift Y...
      2017-12-20 01:31:25.031
      2017-12-06 00:02:59
      0
      0.00
      2017-12-20T01:43:11.216Z
      0
      0
      0
      ...
      2017-12-20T15:56:46.000Z
      147
      129.00
      0.00
      18.00
      10.00
      11.00
      39.00
      39.00
      39.00

      513ef968cfbcdb1c6c8c1fc39d1d1e6af032ecee
      https://www.buzzfeed.com/danvergano/epa-media-...
      2017 Was The Year The EPA Declared War On The ...
      2017-12-23 15:09:13.991
      2017-12-22 20:17:48
      18
      0.15
      2017-12-23T19:24:12.663Z
      0
      6
      12
      ...
      2017-12-23T22:01:54.000Z
      147
      952.00
      4.00
      19.00
      13.00
      11.00
      43.00
      47.00
      39.00

      e529e130efcd7e0c4687e94ae8d14bc898ceec0e
      https://www.buzzfeed.com/alannabennett/ways-ho...
      6 Ways Hollywood Can Do Better In 2018
      2017-12-30 01:29:11.817
      2017-12-29 22:02:12
      40
      0.21
      2017-12-30T19:58:09.135Z
      2
      26
      12
      ...
      2017-12-30T18:22:00.000Z
      147
      nan
      6.00
      19.00
      15.00
      11.00
      45.00
      51.00
      39.00

      d9fa9656eb388266ef0ef42314a1af0747c2d729
      https://www.buzzfeed.com/doree/silicon-valley-...
      Silicon Valley Punished Some Badly Behaved Men...
      2017-12-22 20:10:15.285
      2017-12-22 20:08:31
      26
      1.18
      2017-12-22T20:21:14.378Z
      0
      3
      23
      ...
      2017-12-25T23:33:40.000Z
      147
      1180.00
      5.00
      18.00
      15.00
      11.00
      44.00
      49.00
      39.00

      0e5bb71c21836af42e79cf180822df178ca05faa
      https://www.buzzfeed.com/annehelenpetersen/mom...
      Moms Are Main Characters, Too
      2017-12-10 16:39:12.901
      2017-12-08 21:21:30
      24
      0.30
      2017-12-10T16:50:13.895Z
      2
      15
      7
      ...
      2017-12-10T23:30:00.000Z
      147
      1843.00
      4.00
      19.00
      12.00
      11.00
      42.00
      46.00
      38.00

      515685a5a5560ea7980d63c011a2fe814c5af832
      https://www.buzzfeed.com/matthewzeitlin/one-cr...
      One Crucial Thing Republicans Agreed On In The...
      2017-12-07 22:04:22.607
      2017-12-07 21:56:31
      31
      0.28
      2017-12-07T23:16:09.641Z
      4
      19
      8
      ...
      2017-12-07T22:25:00.000Z
      147
      643.00
      5.00
      19.00
      13.00
      11.00
      43.00
      48.00
      38.00

      deddc425e9e2a6c7ccd8c7f9898210d63b2e1802
      https://www.buzzfeed.com/sarahmarshall/20-year...
      The Insane True Story Of How "Titanic" Got Made
      2017-12-17 16:39:14.602
      2017-12-15 22:50:55
      31
      0.40
      2017-12-17T16:50:11.752Z
      2
      9
      20
      ...
      2017-12-18T01:04:50.000Z
      147
      8636.00
      5.00
      19.00
      12.00
      11.00
      42.00
      47.00
      37.00

25 rows × 27 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_time
      alexa_rank
      word_count
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      6db0269552b047b879c328010314c66ddcaec6da
      https://www.nytimes.com/2017/12/13/opinion/sal...
      Salma Hayek: Harvey Weinstein Is My Monster Too
      2017-12-13 23:49:10.353
      2017-12-13 23:45:20.000
      824459
      5490.26
      2017-12-14T03:03:13.004Z
      101937
      599446
      123076
      ...
      NaN
      120
      2800.00
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      81c5b9be259fc28bcdef151800f9460767cf7690
      https://www.washingtonpost.com/news/politics/w...
      In an upset, Democrat Doug Jones is projected ...
      2017-12-13 03:29:13.065
      2017-12-13 03:26:00.000
      139737
      1302.34
      2017-12-13T03:40:17.040Z
      19636
      111733
      8368
      ...
      NaN
      191
      102.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      1cd3a4c15d1b80136d4cf9bc6ef4a8e1ec03a3ad
      http://yournewswire.com/nestle-ceo-water-is-no...
      Nestle CEO: Water Is Not A Human Right, Should...
      2017-12-14 17:44:25.584
      2017-12-14 16:30:44.000
      111833
      1.74
      2017-12-14T20:59:13.680Z
      42856
      44725
      24252
      ...
      NaN
      22568
      135.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      d18643b71da717ca3fbe0357ecf425b197853b16
      https://www.washingtonpost.com/news/capital-we...
      Alaska just reported one of the most extreme s...
      2017-12-07 17:04:33.494
      2017-12-07 16:57:03.000
      117687
      533.02
      2017-12-08T18:06:11.882Z
      19815
      83900
      13972
      ...
      NaN
      191
      580.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      87f55392a64a0600bbbb60e7a61e641dd51e75cc
      http://www.foxnews.com/sports/2017/12/09/linds...
      Lindsey Vonn suffers back injury in World Cup ...
      2017-12-10 01:29:22.598
      2017-12-10 01:28:54.000
      77394
      45.61
      2017-12-15T08:13:17.970Z
      31471
      41903
      4020
      ...
      NaN
      285
      650.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      74a8e4a79e8e0d97e2e803ecdb912533007bfe21
      http://www.telegraph.co.uk/art/artists/lubiana...
      Lubiana Himid's 2017 Turner Prize victory is a...
      2017-12-05 19:34:16.136
      2017-12-05 19:30:17.000
      127045
      2073.74
      2017-12-05T22:48:10.208Z
      11691
      25685
      89669
      ...
      NaN
      370
      540.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      8c012e0ed5e7d1facf7e68848ae595de4be07f2d
      https://www.theguardian.com/business/2017/dec/...
      UK banks join multinationals in pledge to come...
      2017-12-12 11:55:44.686
      2017-12-12 11:53:19.000
      79893
      76.13
      2017-12-15T10:22:10.889Z
      3370
      68212
      8311
      ...
      NaN
      142
      386.00
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      8804453dcfb7fc6bb6185ef0037846ad46b4290a
      https://www.buzzfeed.com/michaelblackmon/profe...
      People Are Thirsting Pretty Hard For James McA...
      2017-12-05 20:34:16.372
      2017-12-05 20:06:02.000
      135627
      263.18
      2017-12-06T01:32:03.730Z
      31320
      100289
      4018
      ...
      NaN
      147
      nan
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      1cd4d1d98c6f91e32bb2dcaba710f524bc8d8a1d
      http://www.foxnews.com/entertainment/2017/12/0...
      Media elite laughed at 2008 roast sending Matt...
      2017-12-05 20:54:22.240
      2017-12-05 20:44:23.000
      75257
      71.06
      2017-12-09T05:31:06.998Z
      14766
      50287
      10204
      ...
      NaN
      285
      1157.00
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      c2298f24d323275ee6ff00f56b1c8caabb457aeb
      https://www.theguardian.com/books/2017/dec/24/...
      'The difficulty is the point': teaching spoon-...
      2017-12-23 22:40:10.841
      2017-12-23 22:37:54.000
      59004
      14.06
      2017-12-24T22:01:04.177Z
      11099
      38299
      9606
      ...
      NaN
      142
      3097.00
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      4dcf50426af48e5e999bf526b55176a714074ef1
      https://www.washingtonpost.com/news/politics/w...
      Ex-Trump national security adviser Michael Fly...
      2017-12-01 14:34:13.986
      2017-12-01 14:29:00.000
      72220
      468.95
      2017-12-01T18:24:12.055Z
      20328
      41964
      9928
      ...
      NaN
      191
      69.00
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      b9e27ec3d3786ed575f5491b9b1756552d00cbfe
      http://money.cnn.com/2017/12/04/media/pbs-char...
      PBS announces 'Amanpour' as interim replacemen...
      2017-12-04 22:19:15.603
      2017-12-04 21:55:23.000
      58791
      102.26
      2017-12-05T01:45:12.208Z
      6515
      48812
      3464
      ...
      NaN
      105
      nan
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      c17364bddb1b1e1881d26321215078fcf9376136
      https://www.washingtonpost.com/national/health...
      Uranium firm urged Trump officials to shrink B...
      2017-12-09 00:43:21.734
      2017-12-09 00:28:33.000
      85616
      72.66
      2017-12-10T03:36:10.952Z
      11415
      56897
      17304
      ...
      NaN
      191
      1050.00
      50.00
      0.00
      3.00
      0.00
      3.00
      53.00
      -47.00

      fcdf75b812ac05992fcf139ed1fa9fbe26d2d313
      http://www.mirror.co.uk/news/world-news/one-eu...
      One of Europe's brightest Christmas trees ligh...
      2017-12-02 00:01:28.689
      2017-12-01 23:58:50.000
      57934
      50.43
      2017-12-02T10:25:11.817Z
      2392
      48859
      6683
      ...
      NaN
      706
      213.00
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      e67159816fb58bef23f7ec7a956bab62abcdb70d
      http://www.foxnews.com/entertainment/2017/12/0...
      ABC News suspends Brian Ross over 'serious err...
      2017-12-02 23:09:17.105
      2017-12-02 22:50:59.000
      64863
      273.60
      2017-12-03T00:13:21.277Z
      17007
      38340
      9516
      ...
      NaN
      285
      210.00
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      81dbab2e69af8a1dece821f1e7437c621967814f
      http://www.cnn.com/2017/12/14/us/sandy-hook-ne...
      Sandy Hook shooting victims remembered
      2017-12-14 10:34:21.685
      2017-12-14 10:28:31.000
      158920
      247.38
      2017-12-14T23:46:14.233Z
      17348
      122515
      19057
      ...
      NaN
      105
      3714.00
      50.00
      0.00
      4.00
      0.00
      4.00
      54.00
      -46.00

      ef8588c058d4bce262f04a4cf3f58b507662626d
      https://www.washingtonpost.com/news/wonk/wp/20...
      GOP eyes post-tax-cut changes to welfare, Medi...
      2017-12-01 17:52:16.868
      2017-12-01 17:39:19.000
      206995
      197.00
      2017-12-01T22:58:05.475Z
      48607
      126093
      32295
      ...
      NaN
      191
      686.00
      50.00
      0.00
      4.00
      0.00
      4.00
      54.00
      -46.00

      eb6b9898f665d8019751a2270ab567a6a0dc09d0
      https://www.theguardian.com/music/2017/dec/10/...
      Fiona Maddocks’s best classical music of 2017
      2017-12-10 08:04:34.787
      2017-12-10 08:00:31.000
      39600
      7.49
      2017-12-11T00:31:13.938Z
      159
      39308
      133
      ...
      NaN
      142
      nan
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      24104ea0009010519c5307f562acaa533ce66cf0
      http://www.telegraph.co.uk/travel/inspiring-to...
      Be wowed in the north of India by wonderful wi...
      2017-12-29 12:07:05.118
      2017-12-29 10:30:00.000
      38901
      0.03
      2017-12-29T16:22:12.948Z
      7482
      27387
      4032
      ...
      NaN
      370
      595.00
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      68c2b2ad7566bf375e6a6e1f3b3fbea39083d42b
      https://www.washingtonpost.com/news/wonk/wp/20...
      The richest 1 percent now owns more of the cou...
      2017-12-06 15:25:17.266
      2017-12-06 15:16:10.000
      44858
      51.37
      2017-12-06T16:37:07.585Z
      8579
      29245
      7034
      ...
      NaN
      191
      1029.00
      46.00
      0.00
      1.00
      0.00
      1.00
      47.00
      -45.00

      496fcccdb268532134f1625b26aca823b89bb936
      https://www.nytimes.com/2017/12/05/us/climate-...
      4 Takeaways From a Gathering of Mayors on Clim...
      2017-12-06 00:22:12.248
      2017-12-06 00:18:44.000
      33564
      55.53
      2017-12-06T23:05:08.990Z
      376
      32637
      551
      ...
      NaN
      120
      767.00
      45.00
      0.00
      0.00
      0.00
      0.00
      45.00
      -45.00

      e26fb062a26f15584865b2509a27138d7e4e9ed8
      https://www.theguardian.com/world/2017/dec/15/...
      Extreme poverty in America: read the UN specia...
      2017-12-15 17:05:40.703
      2017-12-15 17:00:38.000
      33833
      7.63
      2017-12-16T17:39:02.478Z
      5307
      19585
      8941
      ...
      NaN
      142
      692.00
      45.00
      0.00
      0.00
      0.00
      0.00
      45.00
      -45.00

      b1d48f4a374627c1047af6ff8044e6202509e0b4
      https://www.huffingtonpost.com/entry/the-gop-t...
      The GOP Tax Plan is A Major Assault in Their W...
      2017-12-02 16:46:13.616
      2017-12-02 16:32:34.160
      36793
      42.84
      2017-12-02T16:57:08.929Z
      7323
      25056
      4414
      ...
      NaN
      215
      1044.00
      45.00
      0.00
      0.00
      0.00
      0.00
      45.00
      -45.00

      5f3c3c9b3fe935c782751681584d574adf487220
      https://www.theguardian.com/us-news/2017/dec/2...
      'We're under attack': young conservatives gath...
      2017-12-23 12:04:14.888
      2017-12-23 12:00:21.000
      48012
      123.49
      2017-12-23T20:35:06.298Z
      11240
      35271
      1501
      ...
      NaN
      142
      1155.00
      47.00
      0.00
      2.00
      0.00
      2.00
      49.00
      -45.00

      79b5cf5842c27212d4d8f95d693eb3b7aa22c01c
      http://yournewswire.com/denzil-washington-obam...
      Denzil Washington: ‘Criminal-In-Chief’ Obama ‘...
      2017-12-01 17:24:41.981
      2017-12-01 16:54:17.000
      73830
      34.34
      2017-12-02T20:09:19.285Z
      15179
      45429
      13222
      ...
      2017-12-02T15:40:00.000Z
      22568
      382.00
      49.00
      0.00
      1.00
      4.00
      5.00
      54.00
      -44.00

25 rows × 27 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_2017-10-01_2017-10-31.csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}

```python

```

```python

```