# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
data = pd.read_csv("articles_2017-09-01_2017-09-30.csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      171a1db515d4d4f81e9cdaefacdb8dba16c93b75
      https://www.economist.com/news/middle-east-and...
      Kenya’s high court annuls the presidential ele...
      2017-09-01 14:15:09.267
      2017-09-01
      312
      15.020862
      2017-09-01T18:48:02.383Z
      51
      177
      84
      The Economist
      economist_com
      0
      10099
      33.0
      False
      NaN
      NaN
      1825

      ea4e6959acdbcf288c76f9d25ac6baefb6410463
      http://www.foxnews.com/us/2017/08/31/atlantic-...
      Atlantic hurricane season 2017: What you shoul...
      2017-09-01 00:15:08.851
      2017-09-01
      85
      0.484001
      2017-09-01T01:30:01.288Z
      7
      32
      46
      Fox News
      foxnews_com
      0
      599
      86.0
      False
      NaN
      NaN
      285

      65b41c17cbc8fe96f2eb7d3d8086fbefa0a03cf7
      http://www.telegraph.co.uk/travel/ski/resort-g...
      Laax: piste guide
      2017-09-01 13:21:00.323
      2017-09-01
      1
      0.000000
      2017-09-01T13:34:01.749Z
      0
      0
      1
      The Telegraph
      telegraph_co_uk
      0
      0
      NaN
      False
      NaN
      NaN
      370

      cf3c6ce8b392031a436a585ec7ffc2c1f94729c1
      http://www.telegraph.co.uk/travel/ski/resort-g...
      Laax: restaurants and entertainment
      2017-09-01 13:21:02.106
      2017-09-01
      1
      0.000000
      2017-09-01T13:34:01.753Z
      0
      0
      1
      The Telegraph
      telegraph_co_uk
      0
      0
      NaN
      False
      NaN
      NaN
      370

      96854242b80f8ea0c960c8a3c1b19f1f27d82db6
      http://www.telegraph.co.uk/travel/ski/resort-g...
      Ski Laax: resort guide
      2017-09-01 13:21:02.491
      2017-09-01
      2
      0.000000
      2017-09-01T13:34:01.755Z
      0
      0
      2
      The Telegraph
      telegraph_co_uk
      0
      0
      NaN
      False
      NaN
      NaN
      370

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    152658.00
    mean       1315.77
    std        9625.56
    min           0.00
    50%          26.00
    75%         266.00
    90%        1717.00
    95%        4658.00
    99%       24236.33
    99.5%     42666.43
    99.9%    114308.39
    max     1143483.00
    Name: fb_engagements, dtype: float64

There's a few articles there with 1 million plus engagements, let's just double check that.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      a85688580da47024b77bb7e8718eb06ebbda917a
      https://www.nytimes.com/2017/09/07/us/harvey-i...
      Harvey and Irma, Married 75 Years, Marvel at t...
      2017-09-07 19:33:01.260
      2017-09-07 19:32:07
      1143483
      1005.42
      2017-09-08T01:59:59.960Z
      45548
      956285
      141650
      New York Times
      nytimes_com
      0
      0
      nan
      True
      14529828.00
      2017-09-07T20:25:01.000Z
      120

```python
data.fb_engagements.mode()
```

    0    0
    dtype: int64

Going back to the enagement counts, we see the mean is 1,315, mode is zero, median is 26, 90th percentile is 1,717, 99th percentile is 24,236, 99.5th percentile is 42,666. The standard deviation is 9,625, significantly higher than the mean, so this is not a normal distribution.

Key publishers stats

```python
data.groupby("publisher_id").agg({'url': 'count', 'fb_engagements': ['sum', 'median', 'mean']})
```

      url
      fb_engagements

      count
      sum
      median
      mean

      publisher_id

      anotherangryvoice_blogspot_co_uk
      8
      28818
      3320.50
      3602.25

      bbc_co_uk
      12116
      12265072
      37.00
      1012.30

      breitbart_com
      2472
      11792580
      263.50
      4770.46

      brexitcentral_com
      62
      49203
      291.50
      793.60

      buzzfeed_com
      2026
      7582843
      163.00
      3742.77

      cnn_com
      3444
      20150473
      655.00
      5850.89

      dailymail_co_uk
      23716
      17973676
      28.00
      757.87

      economist_com
      522
      174870
      44.00
      335.00

      evolvepolitics_com
      132
      222865
      537.00
      1688.37

      foxnews_com
      6191
      18136364
      41.00
      2929.47

      ft_com
      3250
      347966
      9.00
      107.07

      huffingtonpost_com
      10566
      20805259
      11.00
      1969.08

      independent_co_uk
      6127
      10609711
      48.00
      1731.63

      indy100_com
      517
      1682810
      160.00
      3254.95

      lemonde_fr
      3883
      2086707
      82.00
      537.40

      libdemvoice_org
      134
      2698
      10.00
      20.13

      mirror_co_uk
      10010
      8262666
      48.00
      825.44

      nbcnews_com
      1710
      5260240
      523.00
      3076.16

      newstatesman_com
      749
      232068
      11.00
      309.84

      npr_org
      1986
      6582394
      179.00
      3314.40

      nytimes_com
      4934
      19844645
      216.00
      4022.02

      order-order_com
      268
      94214
      135.00
      351.54

      propublica_org
      39
      142388
      480.00
      3650.97

      reuters_com
      6047
      1682993
      16.00
      278.32

      rt_com
      2137
      2641695
      390.00
      1236.17

      skwawkbox_org
      114
      36440
      190.50
      319.65

      telegraph_co_uk
      7376
      3943188
      26.00
      534.60

      thecanary_co
      344
      359005
      346.00
      1043.62

      theguardian_com
      8567
      10717627
      135.00
      1251.04

      thetimes_co_uk
      9147
      329695
      1.00
      36.04

      washingtonpost_com
      23360
      15896784
      0.00
      680.51

      westmonster_com
      289
      338151
      41.00
      1170.07

      yournewswire_com
      415
      587343
      97.00
      1415.28

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   123953.00
    mean         5.07
    std          1.79
    min          3.30
    25%          3.56
    50%          4.47
    75%          6.14
    max         11.65
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   123953.00
    mean        11.12
    std         10.73
    min          0.00
    25%          2.00
    50%          8.00
    75%         18.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   152658.00
    mean         9.02
    std         89.37
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        264.00
    99.5%      564.71
    99.9%     1240.72
    max       7546.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   4208.00
    mean     327.33
    std      430.79
    min        4.00
    25%       84.00
    50%      174.00
    75%      414.25
    90%      839.00
    95%     1113.25
    99%     1656.90
    99.5%   2172.34
    99.9%   4612.71
    max     7546.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    lemonde_fr                             618
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      8.00
      5.33
      0.00
      5.33
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      94.00
      12.55
      2.36
      0.88
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      190.00
      8.40
      1.21
      0.58
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      55.00
      4.39
      0.75
      1.45
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      270.00
      11.75
      1.38
      1.80
      12.02
      12.02
      12.02
      12.02

      cnn_com
      166.00
      12.04
      2.68
      1.07
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      150.00
      11.63
      1.24
      1.98
      11.85
      11.85
      11.85
      11.85

      economist_com
      46.00
      6.54
      2.61
      0.53
      5.99
      7.99
      7.99
      7.99

      evolvepolitics_com
      64.00
      5.10
      0.28
      2.91
      5.13
      5.13
      5.13
      5.13

      foxnews_com
      119.00
      10.51
      0.59
      6.19
      10.61
      10.61
      10.61
      10.61

      ft_com
      94.00
      7.74
      1.21
      1.36
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      149.00
      11.11
      0.56
      5.59
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      139.00
      9.46
      1.74
      1.68
      10.07
      10.07
      10.07
      10.07

      indy100_com
      202.00
      5.65
      1.91
      0.47
      4.58
      7.04
      7.04
      7.04

      lemonde_fr
      138.00
      8.38
      2.22
      0.62
      9.34
      9.34
      9.34
      9.34

      libdemvoice_org
      134.00
      4.62
      0.45
      1.10
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      342.00
      8.72
      1.24
      0.61
      9.15
      9.15
      9.15
      9.15

      nbcnews_com
      92.00
      8.47
      1.59
      0.74
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      78.00
      6.20
      0.70
      2.01
      6.35
      6.35
      6.35
      6.35

      npr_org
      140.00
      9.01
      1.37
      0.78
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      55.00
      12.53
      0.00
      12.53
      12.53
      12.53
      12.53
      12.53

      order-order_com
      262.00
      4.50
      1.57
      0.48
      3.27
      5.78
      5.78
      5.78

      propublica_org
      23.00
      6.13
      0.32
      4.65
      6.20
      6.20
      6.20
      6.20

      reuters_com
      85.00
      9.38
      1.13
      3.06
      9.66
      9.66
      9.66
      9.66

      rt_com
      171.00
      9.21
      2.28
      0.68
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      114.00
      4.62
      0.97
      0.75
      5.03
      5.03
      5.03
      5.03

      telegraph_co_uk
      109.00
      9.75
      1.77
      0.85
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      222.00
      5.00
      1.55
      0.93
      3.62
      6.11
      6.21
      6.21

      theguardian_com
      140.00
      11.75
      1.61
      1.01
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      75.00
      6.63
      0.74
      3.42
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      86.00
      11.02
      1.75
      0.95
      11.42
      11.42
      11.42
      11.42

      westmonster_com
      63.00
      4.78
      1.16
      0.87
      5.22
      5.22
      5.22
      5.22

      yournewswire_com
      133.00
      5.81
      0.66
      1.00
      5.99
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.46948415885821004, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     148451
    1.00         41
    2.00         42
    3.00         56
    4.00         78
    5.00         85
    6.00         67
    7.00        247
    8.00        271
    9.00        300
    10.00       237
    11.00       199
    12.00        70
    13.00       274
    14.00       498
    15.00       214
    16.00       359
    17.00       269
    18.00       229
    19.00       383
    20.00       288
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      13.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  8.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    lemonde_fr                         14.00
    libdemvoice_org                     7.00
    mirror_co_uk                       14.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     8.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    2.4500000000000002

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     71064
    2.00      7858
    3.00      4538
    4.00      4303
    5.00       857
    6.00       543
    7.00       557
    8.00       577
    9.00       765
    10.00      228
    11.00      283
    12.00      418
    13.00      150
    14.00       22
    15.00        3
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk       8.00
    bbc_co_uk                          15181.00
    breitbart_com                       2420.00
    brexitcentral_com                     62.00
    buzzfeed_com                        9582.00
    cnn_com                            11168.00
    dailymail_co_uk                    13797.00
    economist_com                        284.00
    evolvepolitics_com                   130.00
    foxnews_com                         7753.00
    ft_com                              3064.00
    huffingtonpost_com                  7607.00
    independent_co_uk                   4402.00
    indy100_com                          516.00
    lemonde_fr                          3787.00
    libdemvoice_org                      134.00
    mirror_co_uk                        9419.00
    nbcnews_com                         1671.00
    newstatesman_com                     694.00
    npr_org                             2352.00
    nytimes_com                         9407.00
    order-order_com                      267.00
    propublica_org                        39.00
    reuters_com                         7080.00
    rt_com                              4254.00
    skwawkbox_org                        114.00
    telegraph_co_uk                     6210.00
    thecanary_co                         341.00
    theguardian_com                    12545.00
    thetimes_co_uk                      9122.00
    washingtonpost_com                  9177.00
    westmonster_com                      271.00
    yournewswire_com                     403.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    171a1db515d4d4f81e9cdaefacdb8dba16c93b75   1.00
    ea4e6959acdbcf288c76f9d25ac6baefb6410463   1.00
    65b41c17cbc8fe96f2eb7d3d8086fbefa0a03cf7   0.00
    cf3c6ce8b392031a436a585ec7ffc2c1f94729c1   0.00
    96854242b80f8ea0c960c8a3c1b19f1f27d82db6   0.00
    a6768245f519a1a2dd1da633b5d9f186e66b5f86   0.00
    3ab8cb96ad784c7646b327a2ff6c2fb0875a9726   0.00
    5d2369b6efcae2693582b2dc4b7444c64bfcb488   0.00
    1a3a69741a4e3eca3399022e9bc1c710d8df86ee   0.00
    0e6649a1d87a2e67e592e7fbec5f002cbb32a44c   1.00
    c31f4c3f12d749682a00fa03b8ad4eb8b2f11e27   3.00
    d43c9606cd44696362eb724037818c279f29a918   0.00
    05dba6d3c8d602f6ef0004b3265e30a5099479de   0.00
    095c2c1a6ac01b5606dc7c540d389fa04546e3c4   0.00
    5c839e9b3c6c059c43ba5b364671817115eec39b   0.00
    40ab7a42b352900d4c2b01e4826b3553cf159fb3   1.00
    48d50d925c7c2f68536e25de1f15429531aa47cb   1.00
    600931b1ef7a8b93fa9e34f5b61a1503bd50754e   1.00
    4fbb0d728589c31c533e86269389f592563e801f   0.00
    db43e4bac26e26ac5eeb0ab224778e55a80a170e   1.00
    ef918777a190d2a6f40b0370be447e31e4b131a5   5.00
    6c084bc2f2d8f6b3ed33d9d951108ad8ddc3c9b1   1.00
    637b56e7e012fabe04d54ca328bc3d010f01afb9   0.00
    562f6351aba92fabb666b284cb678cc19ff5d15c   1.00
    d1b1f3421b3d56dfb7cdd6b7ee7f5c02f536d7cf   1.00
    ca17823d9c51d4b32d525a15a58e0b2b25a9f6ba   0.00
    037fad0847b147bc9d0933bfcc63aa902ee13d71   0.00
    b3ce5114c925176798351df253d835bc0232a9ea   0.00
    9a0a167585ab16d2bd99f4d355f77afc901f52aa   0.00
    9c323ecb10718cc856007c8c8bfd0aac5d10bd3e   3.00
                                               ...
    371cc6c3e409283441a2471f2ec214e7ac8682cc   0.00
    160f46e03ff51a3944bed74510c738e9e29fdce3   0.00
    3f9489916269f0caaf46674464088b3810a2e692   0.00
    6bb5e2634ad849cf69abde949d029b2b0dd18b4b   2.00
    03405d7d4b535851e8b6b395df8a1bf772b3e017   0.00
    3dd84c45d04ee158e27c0a36623886130a758a05   0.00
    c7443ba3b937d485c0e6ef68c747218f1daccc71   0.00
    9186449842926329474a32475c37b7dc7dcf7064   1.00
    35001ab1fb3d11fd417d10e33bebbd984b1031c4   0.00
    d9fc1c214777b60a24d0cb3da9a35bb8492643ae   0.00
    7e3710af4f37d280336dd704b2ebe05802e8fa6a   0.00
    25f16c16f436aad31ae3baaa7fd800680e0e69b2   0.00
    20bc3574168f9a27239bfd4f7090317204e02464   0.00
    47f92d6da64d872f9dc15a199cd75d549bc16d31   1.00
    21ee0e5f55e63a51cdc07d25df92cf661ed8da39   0.00
    af8bed7dd1ba9ff7e6718144ef69a6e8f335decc   0.00
    0276d4428f791bc09a0142469182e0e9123563f5   0.00
    28dbafb480db5c1e673938765a5c3daa0617cf84   0.00
    3d7c2f8d8ce531b045f1d1373d25185e963e91e6   0.00
    fa09dcb5be9c60cfa6596342cd304656259e60b9   0.00
    0a0847e09b1c77bcec4cd6152b86c51e9e005f36   1.00
    f3127c22bd153b941224ea41e96d8e6f1e7acda0   0.00
    c39da6648ab2f20cea074de5e1326c249626f303   0.00
    d0b55c216b94420ba2d401af5e299e642db69f09   1.00
    4df995cccb46e21fc0e642565a06dd6c97e0ee24   0.00
    634883b02e9c7a11bd19623d1f3bb6eba03f97a7   0.00
    f9eaf89d61d8b2ff051e893c3e3ac1b3a12d00d0   0.00
    3606fdc0b002f417710fbc16ea2045a5b04c5f62   1.00
    d6e923f137e8ad3cc4c2694557f30b1e27b6d38a   0.00
    a58d5a7df0e47588212248dc1209a15053fb1ff1   2.00
    Name: front_score, Length: 152658, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    44213994.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      8.00
      0.88
      0.35
      0.00
      1.00
      1.00
      1.00
      1.00

      bbc_co_uk
      12116.00
      0.59
      2.92
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2472.00
      0.86
      0.99
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      62.00
      0.89
      0.32
      0.00
      1.00
      1.00
      1.00
      1.00

      buzzfeed_com
      2026.00
      0.23
      0.42
      0.00
      0.00
      0.00
      0.00
      1.00

      cnn_com
      3444.00
      2.79
      4.49
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      23716.00
      0.57
      1.59
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      522.00
      2.14
      1.36
      0.00
      0.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      132.00
      0.67
      0.47
      0.00
      0.00
      1.00
      1.00
      1.00

      foxnews_com
      6191.00
      0.57
      1.75
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      3250.00
      0.62
      0.92
      0.00
      0.00
      0.00
      2.00
      2.00

      huffingtonpost_com
      10566.00
      0.48
      1.30
      0.00
      0.00
      0.00
      0.00
      4.00

      independent_co_uk
      6127.00
      0.65
      1.24
      0.00
      0.00
      0.00
      0.00
      3.00

      indy100_com
      517.00
      0.62
      0.49
      0.00
      0.00
      1.00
      1.00
      1.00

      lemonde_fr
      3883.00
      0.73
      0.96
      0.00
      0.00
      0.00
      2.00
      2.00

      libdemvoice_org
      134.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      10010.00
      0.25
      0.43
      0.00
      0.00
      0.00
      1.00
      1.00

      nbcnews_com
      1710.00
      2.03
      2.00
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      749.00
      0.58
      0.49
      0.00
      0.00
      1.00
      1.00
      1.00

      npr_org
      1986.00
      1.36
      1.49
      0.00
      0.00
      0.00
      3.00
      3.00

      nytimes_com
      4934.00
      1.49
      2.29
      0.00
      0.00
      0.00
      5.00
      5.00

      order-order_com
      268.00
      0.82
      0.38
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      39.00
      0.79
      0.41
      0.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      6047.00
      0.55
      0.89
      0.00
      0.00
      0.00
      2.00
      2.00

      rt_com
      2137.00
      1.06
      1.00
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      114.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      7376.00
      0.55
      0.89
      0.00
      0.00
      0.00
      2.00
      2.00

      thecanary_co
      344.00
      0.66
      0.47
      0.00
      0.00
      1.00
      1.00
      1.00

      theguardian_com
      8567.00
      0.53
      1.14
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      9147.00
      0.05
      0.23
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      23360.00
      0.20
      0.74
      0.00
      0.00
      0.00
      0.00
      3.00

      westmonster_com
      289.00
      0.28
      0.45
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      415.00
      0.17
      0.37
      0.00
      0.00
      0.00
      0.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     330078.00
    bbc_co_uk                          44213994.00
    breitbart_com                       3680635.00
    brexitcentral_com                      9347.00
    buzzfeed_com                        2714356.00
    cnn_com                            28782078.00
    dailymail_co_uk                    12712687.00
    economist_com                       8315962.00
    evolvepolitics_com                   113280.00
    foxnews_com                        15924468.00
    ft_com                              3674434.00
    huffingtonpost_com                  9753503.00
    independent_co_uk                   7610080.00
    indy100_com                          225454.00
    lemonde_fr                          3895889.00
    libdemvoice_org                        8593.00
    mirror_co_uk                        2873844.00
    nbcnews_com                         9302574.00
    newstatesman_com                     154405.00
    npr_org                             6179264.00
    nytimes_com                        14688146.00
    order-order_com                       44296.00
    propublica_org                       350212.00
    reuters_com                         3875248.00
    rt_com                              4588236.00
    skwawkbox_org                          5276.00
    telegraph_co_uk                     4365583.00
    thecanary_co                         155885.00
    theguardian_com                     7739677.00
    thetimes_co_uk                       695491.00
    washingtonpost_com                  6028517.00
    westmonster_com                       14257.00
    yournewswire_com                      26752.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.71
    bbc_co_uk                          17.60
    breitbart_com                      15.12
    brexitcentral_com                   9.14
    buzzfeed_com                       14.81
    cnn_com                            17.18
    dailymail_co_uk                    16.36
    economist_com                      15.93
    evolvepolitics_com                 11.64
    foxnews_com                        16.58
    ft_com                             15.12
    huffingtonpost_com                 16.09
    independent_co_uk                  15.84
    indy100_com                        12.33
    lemonde_fr                         15.18
    libdemvoice_org                     9.06
    mirror_co_uk                       14.87
    nbcnews_com                        16.05
    newstatesman_com                   11.95
    npr_org                            15.64
    nytimes_com                        16.50
    order-order_com                    10.70
    propublica_org                     12.77
    reuters_com                        15.17
    rt_com                             15.34
    skwawkbox_org                       8.57
    telegraph_co_uk                    15.29
    thecanary_co                       11.96
    theguardian_com                    15.86
    thetimes_co_uk                     13.45
    washingtonpost_com                 15.61
    westmonster_com                     9.57
    yournewswire_com                   10.19
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.80
    bbc_co_uk                          10.70
    breitbart_com                       8.21
    brexitcentral_com                   2.24
    buzzfeed_com                        7.91
    cnn_com                            10.27
    dailymail_co_uk                     9.45
    economist_com                       9.03
    evolvepolitics_com                  4.73
    foxnews_com                         9.68
    ft_com                              8.21
    huffingtonpost_com                  9.19
    independent_co_uk                   8.94
    indy100_com                         5.42
    lemonde_fr                          8.27
    libdemvoice_org                     2.15
    mirror_co_uk                        7.96
    nbcnews_com                         9.14
    newstatesman_com                    5.04
    npr_org                             8.73
    nytimes_com                         9.59
    order-order_com                     3.79
    propublica_org                      5.86
    reuters_com                         8.26
    rt_com                              8.43
    skwawkbox_org                       1.66
    telegraph_co_uk                     8.38
    thecanary_co                        5.05
    theguardian_com                     8.95
    thetimes_co_uk                      6.54
    washingtonpost_com                  8.70
    westmonster_com                     2.66
    yournewswire_com                    3.29
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   4.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    lemonde_fr                         12.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       3.00
    telegraph_co_uk                    12.00
    thecanary_co                        8.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     4.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   2.00
    buzzfeed_com                       11.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  6.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         7.00
    lemonde_fr                         12.00
    libdemvoice_org                     2.00
    mirror_co_uk                       11.00
    nbcnews_com                        13.00
    newstatesman_com                    7.00
    npr_org                            12.00
    nytimes_com                        14.00
    order-order_com                     5.00
    propublica_org                      8.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        7.00
    theguardian_com                    13.00
    thetimes_co_uk                      9.00
    washingtonpost_com                 12.00
    westmonster_com                     3.00
    yournewswire_com                    4.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     23428
    1.00     19237
    2.00     12755
    3.00      8699
    4.00      6444
    5.00      5114
    6.00      4503
    7.00      3939
    8.00      3523
    9.00      3331
    10.00     3043
    11.00     2770
    12.00     2681
    13.00     2512
    14.00     2433
    15.00     2259
    16.00     2187
    17.00     1982
    18.00     1884
    19.00     1726
    20.00     1719
    21.00     1632
    22.00     1647
    23.00     1542
    24.00     1544
    25.00     1436
    26.00     1429
    27.00     1384
    28.00     1352
    29.00     1302
             ...
    66.00       93
    67.00       85
    68.00       76
    69.00       60
    70.00       34
    71.00       45
    72.00       53
    73.00       33
    74.00       35
    75.00       41
    76.00       32
    77.00       34
    78.00       19
    79.00       27
    80.00       34
    81.00       17
    82.00       21
    83.00       16
    84.00       18
    85.00        7
    86.00       15
    87.00        9
    88.00       19
    89.00       10
    90.00        5
    91.00        6
    92.00        7
    93.00        7
    94.00        6
    95.00        1
    Name: attention_index, Length: 96, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      6067d56573cffca0809e48adf887dc244f0071bc
      http://www.bbc.co.uk/news/world-asia-41275614
      N Korea 'fires missile from Pyongyang'
      2017-09-14 22:18:02.020
      2017-09-14 22:14:57.000
      116513
      834.83
      2017-09-14T22:44:01.001Z
      22667
      76135
      17711
      ...
      True
      43989977.00
      2017-09-15T01:49:45.000Z
      96
      50.00
      20.00
      10.00
      15.00
      45.00
      95.00

      e5916f9bd62e8c6c9cb564d4d4b4ea8a99c8e1d2
      http://www.cnn.com/2017/09/18/politics/paul-ma...
      US government wiretapped former Trump campaign...
      2017-09-18 22:57:09.206
      2017-09-18 22:55:12.000
      167763
      540.50
      2017-09-18T23:10:01.591Z
      46913
      97863
      22987
      ...
      True
      28570339.00
      2017-09-18T23:01:49.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      68f2a3b5ccfe8c92d511232c1e284f4f42852394
      http://www.cnn.com/2017/09/03/politics/obama-t...
      Read the Inauguration Day letter Obama left fo...
      2017-09-03 12:48:16.001
      2017-09-03 12:45:03.000
      184591
      416.92
      2017-09-03T13:14:00.480Z
      25453
      132713
      26425
      ...
      True
      28191321.00
      2017-09-03T12:52:17.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      6dd1f438183f2d3b123ce529eda57ce6e1a5be97
      http://www.bbc.co.uk/news/world-latin-america-...
      Large earthquake strikes near Mexico
      2017-09-08 05:12:00.922
      2017-09-08 05:09:57.000
      87552
      516.50
      2017-09-08T05:38:01.097Z
      13470
      61106
      12976
      ...
      True
      43867811.00
      2017-09-08T05:13:41.000Z
      96
      49.00
      20.00
      10.00
      15.00
      45.00
      94.00

      a4ec2e243a5327b4c92ac310dc8e8c028b069e1d
      http://www.cnn.com/2017/09/20/americas/hurrica...
      Hurricane Maria: Puerto Rico awaits direct hit
      2017-09-20 09:03:20.771
      2017-09-20 08:59:03.000
      143472
      388.67
      2017-09-20T23:54:02.368Z
      15363
      109288
      18821
      ...
      True
      28601101.00
      2017-09-20T19:30:25.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      e915fdae2abc802ebd8edc374dc070ade3465019
      http://www.cnn.com/2017/09/18/americas/atlanti...
      Hurricane Maria forecaset to be Category 4
      2017-09-18 07:51:12.471
      2017-09-18 07:44:05.000
      143711
      484.50
      2017-09-18T21:06:02.694Z
      33386
      88010
      22315
      ...
      True
      28565647.00
      2017-09-18T08:31:05.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      8e73a1eb14f6a69156ddf755d404b8a464e22ef8
      http://www.cnn.com/2017/09/23/politics/nfl-goo...
      FL commissioner: Trump showed 'lack of respect...
      2017-09-23 14:43:16.807
      2017-09-23 14:37:43.000
      98752
      211.70
      2017-09-24T19:06:05.123Z
      32897
      53603
      12252
      ...
      True
      28618169.00
      2017-09-23T15:20:06.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      588089e0a7c0e9eabe610bf658f2d4205b31e0cc
      http://www.cnn.com/2017/09/05/us/hurricane-irm...
      Hurricane Irma: Florida and Puerto Rico brace ...
      2017-09-05 06:57:19.474
      2017-09-05 06:51:25.000
      263683
      257.52
      2017-09-05T16:02:00.306Z
      88414
      144215
      31054
      ...
      True
      28201804.00
      2017-09-05T08:30:27.000Z
      105
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      c7e29470ae37b7a78df9a3d784e4f66e4ca951a2
      http://www.bbc.co.uk/news/world-middle-east-41...
      Saudi women 'given right to drive'
      2017-09-26 19:13:08.658
      2017-09-26 19:09:45.000
      330324
      2935.87
      2017-09-26T21:08:04.432Z
      36855
      265864
      27605
      ...
      True
      44163937.00
      2017-09-26T19:13:41.000Z
      96
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      6755d9c120cfcf187f90e6c4ad4974cb7d1447a7
      http://www.bbc.co.uk/news/world-europe-41376577
      Germany election: Merkel seeks fourth term
      2017-09-24 03:49:15.814
      2017-09-24 03:47:32.000
      57302
      320.86
      2017-09-24T16:32:02.690Z
      7320
      45782
      4200
      ...
      True
      44137706.00
      2017-09-24T16:14:49.000Z
      96
      46.00
      20.00
      12.00
      15.00
      47.00
      93.00

      cb9529046059677cc97532667a6e4b6a85e21790
      http://www.bbc.co.uk/news/world-asia-41391978
      N Korea accuses US of declaring war
      2017-09-25 15:07:10.151
      2017-09-25 15:04:19.000
      163895
      1682.78
      2017-09-25T17:12:12.061Z
      43397
      96735
      23763
      ...
      True
      44155104.00
      2017-09-26T02:49:43.000Z
      96
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      75bd9111d5a725787b322d8212c72beade7fb8a6
      http://money.cnn.com/2017/09/27/media/hugh-hef...
      Playboy founder Hugh Hefner dead at 91
      2017-09-28 03:34:14.532
      2017-09-28 03:27:23.000
      112943
      808.77
      2017-09-28T03:46:12.227Z
      20372
      65226
      27345
      ...
      True
      28720237.00
      2017-09-28T03:37:38.000Z
      105
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      84c9cdda4951e8791077a683f93e2a5e5a9d9bfb
      http://www.cnn.com/2017/09/17/politics/trump-t...
      Trump retweets GIF of him hitting Clinton with...
      2017-09-17 13:48:21.304
      2017-09-17 13:43:23.000
      136292
      311.77
      2017-09-17T15:15:59.744Z
      60117
      63645
      12530
      ...
      True
      28562039.00
      2017-09-17T14:54:04.000Z
      105
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      e3fee93183c369e2eb37069a142e6c7142dab705
      http://money.cnn.com/2017/09/01/technology/mic...
      Tech billionaire Michael Dell pledges $36 mill...
      2017-09-01 16:45:10.351
      2017-09-01 15:29:21.000
      82021
      296.50
      2017-09-01T16:58:01.790Z
      5412
      69977
      6632
      ...
      True
      28178745.00
      2017-09-01T16:40:38.000Z
      105
      49.00
      20.00
      9.00
      15.00
      44.00
      93.00

      d6ff153e02b24da93ad3444297ab046d5cd62d97
      http://www.cnn.com/2017/09/24/us/nfl-trump-tak...
      NFL Sunday kicks off amid beef between players...
      2017-09-24 14:16:17.026
      2017-09-24 14:11:58.000
      148932
      432.54
      2017-09-24T15:28:12.004Z
      33865
      104721
      10346
      ...
      True
      28621607.00
      2017-09-24T14:35:38.000Z
      105
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      12d0d6ce64cee12a5b117c70fe8f2272378d8053
      http://www.cnn.com/2017/09/22/politics/donald-...
      Trump: NFL owners should fire players who protest
      2017-09-23 01:34:27.121
      2017-09-23 01:29:11.000
      126459
      163.48
      2017-09-23T02:46:08.560Z
      57819
      57768
      10872
      ...
      True
      28614688.00
      2017-09-23T01:48:52.000Z
      105
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      b86ac39e4b7fdf8c537b54840eee3aa4fe2d28ab
      http://www.cnn.com/2017/09/23/politics/donald-...
      The dark racial sentiment in Trump's NBA and N...
      2017-09-23 21:07:20.631
      2017-09-23 21:03:20.000
      84670
      145.59
      2017-09-23T23:11:08.062Z
      25997
      46214
      12459
      ...
      True
      28619235.00
      2017-09-23T22:00:12.000Z
      105
      49.00
      20.00
      8.00
      15.00
      43.00
      92.00

      73518249283178cf7ff417e30956880b1bc1878f
      http://www.cnn.com/2017/09/17/americas/atlanti...
      Atlantic hosts three storms, with Maria on Irm...
      2017-09-17 05:36:14.222
      2017-09-17 05:31:54.000
      76800
      123.25
      2017-09-17T17:41:59.655Z
      15773
      46849
      14178
      ...
      True
      28562524.00
      2017-09-17T17:30:29.000Z
      105
      48.00
      20.00
      9.00
      15.00
      44.00
      92.00

      2562369665718a1858add5e041951a9c9f10cd80
      http://www.bbc.co.uk/news/uk-41278545
      Incident reported on London Tube train
      2017-09-15 07:48:02.690
      2017-09-15 07:45:25.000
      101874
      396.05
      2017-09-15T08:02:01.607Z
      22092
      56920
      22862
      ...
      True
      43997776.00
      2017-09-15T07:49:19.000Z
      96
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      355a3ab21fa58fd0c251c8fb6d39065852d5d867
      http://www.bbc.co.uk/news/world-asia-41431057
      China to shut down North Korean companies
      2017-09-28 13:40:11.625
      2017-09-28 13:36:30.000
      45061
      111.36
      2017-09-28T20:06:12.105Z
      1843
      41077
      2141
      ...
      True
      44185601.00
      2017-09-28T19:22:19.000Z
      96
      45.00
      20.00
      12.00
      15.00
      47.00
      92.00

      566c188bb9929c658a3c6d40088f87743081151a
      http://www.bbc.co.uk/news/world-us-canada-4142...
      Playboy founder Heffner dies aged 91
      2017-09-28 03:31:10.364
      2017-09-28 03:29:02.000
      78003
      257.46
      2017-09-28T03:53:05.355Z
      16105
      44826
      17072
      ...
      True
      44181244.00
      2017-09-28T03:38:13.000Z
      96
      48.00
      20.00
      9.00
      15.00
      44.00
      92.00

      1134a234e872bbf57eec6bcf1fa40eee11e2c999
      http://www.cnn.com/2017/09/06/us/hurricane-irm...
      Hurricane Irma hits Caribbean island of Barbuda
      2017-09-06 06:39:19.049
      2017-09-06 06:34:13.000
      73503
      79.75
      2017-09-06T18:05:59.675Z
      18627
      41020
      13856
      ...
      True
      28218940.00
      2017-09-06T07:00:10.000Z
      105
      48.00
      20.00
      8.00
      15.00
      43.00
      91.00

      158596a091671070e926a74ce3c1e8d252e08133
      http://www.cnn.com/2017/09/21/politics/kim-jon...
      North Korea's Kim: Trump 'will pay dearly' for...
      2017-09-21 23:01:24.885
      2017-09-21 22:56:44.000
      67623
      275.96
      2017-09-22T01:07:12.468Z
      23918
      32365
      11340
      ...
      True
      28610759.00
      2017-09-21T23:07:59.000Z
      105
      47.00
      20.00
      9.00
      15.00
      44.00
      91.00

      cdd1e7a21d1430598e553c8146eb3f5a93b5a486
      http://www.cnn.com/2017/09/19/americas/mexico-...
      7.1 earthquake strikes central Mexico
      2017-09-19 18:57:14.566
      2017-09-19 18:55:08.000
      139662
      359.50
      2017-09-19T19:58:00.234Z
      20286
      96085
      23291
      ...
      True
      28583302.00
      2017-09-19T19:42:15.000Z
      105
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      2847600e75fdbf6357bfa32c08916aad2d9b6f83
      http://www.cnn.com/2017/09/19/americas/hurrica...
      Hurricane Maria hits Dominica with Category 5 ...
      2017-09-19 07:33:14.696
      2017-09-19 07:29:02.000
      62093
      61.17
      2017-09-20T02:22:00.321Z
      11081
      41740
      9272
      ...
      True
      28572026.00
      2017-09-19T08:00:33.000Z
      105
      47.00
      20.00
      9.00
      15.00
      44.00
      91.00

      1c534e814b943941f4db2387b9d96eda77d57570
      https://www.buzzfeed.com/adolfoflores/daca-res...
      A Paramedic With DACA Status Rescues Harvey Vi...
      2017-09-02 03:24:21.448
      2017-09-02 03:22:27.000
      96700
      119.25
      2017-09-02T20:24:01.207Z
      12230
      70455
      14015
      ...
      True
      2662925.00
      2017-09-02T14:55:59.000Z
      147
      49.00
      19.00
      12.00
      11.00
      42.00
      91.00

      df0494b09e5e2991b2774626f0e313590b598948
      http://www.cnn.com/2017/09/25/politics/north-k...
      North Korea accuses Trump of declaring war
      2017-09-25 15:16:27.965
      2017-09-25 15:09:18.000
      122525
      509.35
      2017-09-25T17:11:11.339Z
      37299
      62908
      22318
      ...
      True
      28627649.00
      2017-09-25T16:09:44.000Z
      105
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      12b2893bac6075f49fb8f552f321dc6d49cfbd12
      https://www.buzzfeed.com/davidmack/kylie-jenne...
      Holy Crap, Kylie Jenner Is Pregnant
      2017-09-22 21:24:15.447
      2017-09-22 21:13:50.000
      59174
      509.54
      2017-09-22T23:09:05.482Z
      26805
      29763
      2606
      ...
      True
      2699166.00
      2017-09-22T21:29:43.000Z
      147
      47.00
      19.00
      13.00
      11.00
      43.00
      90.00

      d7b8dca19b15e0b52d4a21fad60b8dac7b38f5fd
      http://www.cnn.com/2017/09/30/politics/trump-t...
      Trump attacks San Juan mayor over hurricane re...
      2017-09-30 13:10:21.979
      2017-09-30 13:05:07.000
      324522
      600.79
      2017-09-30T14:45:08.993Z
      112864
      181960
      29698
      ...
      True
      28737538.00
      2017-09-30T13:20:59.000Z
      105
      50.00
      20.00
      5.00
      15.00
      40.00
      90.00

      a3c5b9f1315aad42f86037db387093743e949368
      http://www.cnn.com/2017/09/05/politics/daca-tr...
      Trump ends DACA, but gives Congress window to ...
      2017-09-05 15:18:16.587
      2017-09-05 15:12:27.000
      106307
      389.12
      2017-09-05T15:44:01.931Z
      27624
      65130
      13553
      ...
      True
      28204756.00
      2017-09-05T15:22:03.000Z
      105
      50.00
      20.00
      5.00
      15.00
      40.00
      90.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      3cae93c5f6e373e9939f52c20ca2f6c119dd695b
      http://www.huffingtonpost.com/entry/revive-you...
      Revive Your Family on the Best Winter Vacation...
      2017-09-15 18:18:16.858
      2017-09-15 18:15:38.431
      0
      0.00
      2017-09-16T03:50:00.969Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      505dbb2bf9e0a7a809dd33a0b67e5432492214ed
      https://www.washingtonpost.com/national/coast-...
      Coast Guard suspends search for N Carolina men...
      2017-09-03 17:24:07.265
      2017-09-03 17:18:18.000
      0
      0.00
      2017-09-04T15:20:00.383Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1cee3dfc94d901a4f5ee0a674e93b07ed244fa8e
      http://www.dailymail.co.uk/sport/cricket/artic...
      England batsman Keaton Jennings set to leave D...
      2017-09-26 09:01:24.271
      2017-09-26 08:59:16.000
      1
      0.02
      2017-09-26T10:14:05.025Z
      0
      0
      1
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      1393e93bb1c4eac4faf1ba014ef98c9a021e4488
      https://www.washingtonpost.com/national/hurric...
      Hurricane watch in effect for Florida Keys, pa...
      2017-09-07 15:24:12.613
      2017-09-07 15:14:28.000
      1
      0.02
      2017-09-07T16:40:01.169Z
      0
      0
      1
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ad11c7355e1101976c3257970a9d360ee01ff77c
      https://www.washingtonpost.com/politics/long-t...
      Long-term US mortgage rates fall to lowest lev...
      2017-09-07 15:24:07.391
      2017-09-07 15:14:27.000
      0
      0.00
      2017-09-07T15:38:02.849Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      fb30f48fbf7656287634ba122cf6a31ae394205f
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘DC 4 Midday’ game
      2017-09-15 18:27:11.304
      2017-09-15 18:20:14.000
      0
      0.00
      2017-09-16T05:00:01.737Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b23c6ee69b4ef5c31583c2f4ae30ac88852649c8
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘Pick 4 Day’ game
      2017-09-15 18:27:14.499
      2017-09-15 18:20:20.000
      0
      0.00
      2017-09-16T05:00:01.753Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6320989548e7b6b931c86fd8f26aec50a964cff3
      https://www.washingtonpost.com/world/national-...
      Trump adviser says military option not preferr...
      2017-09-15 18:27:09.992
      2017-09-15 18:20:28.000
      0
      0.00
      2017-09-16T05:00:01.732Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      fa5a3963d1a16211c2b79e0f608d216edb76d640
      https://www.washingtonpost.com/local/dc-lotter...
      DC Lottery
      2017-09-15 18:27:12.368
      2017-09-15 18:20:33.000
      0
      0.00
      2017-09-16T05:00:01.739Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      9eeb8123352c7675b26f497bf862f298233655a4
      https://www.washingtonpost.com/politics/federa...
      Trump administration offers a new travel ban
      2017-09-26 09:04:16.466
      2017-09-26 08:57:22.000
      0
      0.00
      2017-09-26T11:17:07.377Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      cae3bcbaedc662e3cca8e7a91f19103301ddca37
      http://www.bbc.co.uk/news/av/world-asia-411418...
      Worried about North Korea?
      2017-09-03 17:18:01.083
      2017-09-03 17:16:41.000
      0
      0.00
      2017-09-04T15:14:00.442Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f8fc25f41827dc3a09f22b1454d75d31087edf19
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘DC 5 Midday’ game
      2017-09-15 18:27:10.106
      2017-09-15 18:20:37.000
      0
      0.00
      2017-09-16T05:00:01.735Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6572aea3fe9c36c73ae0b082d6e6a738530d61d4
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘DC 3 Midday’ game
      2017-09-15 18:27:08.802
      2017-09-15 18:20:44.000
      0
      0.00
      2017-09-16T05:00:01.730Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      814370ad5fb03aa2782c8920046011f3113d91f3
      http://www.huffingtonpost.com/entry/how-your-u...
      How Your Unusual Idea Can Spark Change
      2017-09-15 18:33:16.512
      2017-09-15 18:21:13.948
      1
      0.02
      2017-09-16T17:30:00.688Z
      0
      0
      1
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b341e964f21b813dddb39e2544f92abd99467617
      https://www.washingtonpost.com/world/middle_ea...
      Turkey warns Iraq: Kurdish referendum security...
      2017-09-15 18:27:13.191
      2017-09-15 18:21:15.000
      1
      0.02
      2017-09-15T21:46:01.315Z
      0
      0
      1
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5429bdbedb7217d1a005978744b70e5d24e7adf3
      https://www.washingtonpost.com/sports/colleges...
      8 was enough: Texas Tech defense rematch vs. A...
      2017-09-15 18:27:12.568
      2017-09-15 18:21:23.000
      0
      0.00
      2017-09-16T05:00:01.748Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0d82f1792e97a5e694bd5d5d6c560d365e10ebb9
      http://www.washingtonpost.com/video/world/bees...
      Bees invade Australian home
      2017-09-26 09:01:18.240
      2017-09-26 08:54:11.000
      0
      0.00
      2017-09-26T11:14:05.859Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      efe770bbfac7362c79d46ab36a4f684f71b86057
      http://www.foxnews.com/sports/2017/09/15/royal...
      Royals-Indians game Saturday airs on FS1
      2017-09-15 18:40:03.668
      2017-09-15 18:21:43.000
      1
      0.00
      2017-09-16T04:12:01.504Z
      0
      0
      1
      ...
      False
      nan
      NaN
      285
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2300922d72f1dfd75bd914641edee108e42b40dc
      http://www.bbc.co.uk/news/live/uk-england-suss...
      BBC Live: Sussex on Friday 8 September 2017
      2017-09-07 15:18:02.781
      2017-09-07 15:14:07.000
      0
      0.00
      2017-09-07T15:32:03.378Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      878f6e8733a7c3ee626fbcc37ae157684e66516a
      http://www.dailymail.co.uk/sport/article-48488...
      Froome keeps Vuelta lead as Miguel Angel Lopez...
      2017-09-03 17:21:18.961
      2017-09-03 17:17:41.000
      0
      0.00
      2017-09-04T15:18:01.063Z
      0
      0
      0
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      8af19589757e499bc8ea91662faca754a8a18d2a
      http://www.bbc.co.uk/sport/football/41287263
      Kilmarnock's Lee McCulloch delivers 'home trut...
      2017-09-15 18:24:03.556
      2017-09-15 18:22:43.000
      0
      0.00
      2017-09-16T06:00:01.867Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      3de1d53c2c31b35f1d38e5bbdcf26eef937a63cf
      https://www.washingtonpost.com/sports/national...
      Cubs shortstop Russell nears return from foot ...
      2017-09-15 18:30:10.390
      2017-09-15 18:24:22.000
      0
      0.00
      2017-09-16T04:02:01.249Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      3f762ebe86a2e48983ea058c8e00287056eb2eff
      http://www.dailymail.co.uk/sport/cricket/artic...
      Lyon helps Australia record victory over Bangl...
      2017-09-07 15:15:22.842
      2017-09-07 15:13:17.000
      0
      0.00
      2017-09-07T15:28:03.225Z
      0
      0
      0
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b60931c502c565965921d33aa3da0fb062f2b0dc
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘Play 4 Day’ game
      2017-09-15 18:30:09.226
      2017-09-15 18:25:09.000
      0
      0.00
      2017-09-16T04:02:01.248Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      69b7813fdfbf3dbb654e7fe2ea45a9cc41a0190b
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘Play 3 Day’ game
      2017-09-15 18:30:09.191
      2017-09-15 18:25:17.000
      0
      0.00
      2017-09-16T04:02:01.246Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      131f76f9485a82dfe2f80f5fa7e0a0e188a14cdc
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘Cash 5 Day’ game
      2017-09-15 18:39:08.018
      2017-09-15 18:25:22.000
      0
      0.00
      2017-09-16T04:10:01.012Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      b1c8f82f99814a195bae6da2873c843544629cb8
      https://www.washingtonpost.com/sports/auto-rac...
      Tony Kanaan says IndyCar finale will be last w...
      2017-09-15 18:39:09.365
      2017-09-15 18:25:28.000
      0
      0.00
      2017-09-16T04:10:01.016Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c4f7fc9d58ff8dc8b105412b99a7e455ee9a9549
      https://www.washingtonpost.com/politics/federa...
      A tough start for GOP in a high-stakes week
      2017-09-26 08:52:13.514
      2017-09-26 08:51:41.000
      0
      0.00
      2017-09-26T11:05:09.073Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      329f1f0ea56ea1b61fa88204b2f898a649585362
      http://www.huffingtonpost.com/entry/body-shami...
      Body Shaming and The Rampant Judgment of Women
      2017-09-15 18:33:14.571
      2017-09-15 18:26:14.101
      0
      0.00
      2017-09-16T05:06:01.794Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      25938ff119dd44a82dd96ff1593e4903541aaf9c
      https://www.washingtonpost.com/sports/colleges...
      Totten runs for 2 TDs as NCCU beats Div. II Sh...
      2017-09-10 02:12:16.743
      2017-09-10 02:08:07.000
      0
      0.00
      2017-09-10T07:36:01.184Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

152658 rows × 25 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      b149ea3a151e68636d95ed9276493cc30e848afe
      http://www.bbc.co.uk/news/world-africa-41123329
      Judges annul Kenya presidential election
      2017-09-01 08:54:00.852
      2017-09-01 08:50:37
      0
      1981.84
      2017-09-01T22:33:59.613Z
      0
      0
      0
      ...
      43741129.00
      2017-09-01T09:39:04.000Z
      96
      0.00
      20.00
      8.00
      15.00
      43.00
      43.00
      43.00

      dfc27a9222b7b195234505922dddcb80c81e588e
      https://www.buzzfeed.com/johnpaczkowski/apple-...
      Coming Up: The Biggest Announcements From Appl...
      2017-09-12 12:54:19.843
      2017-09-08 19:31:51
      3
      0.25
      2017-09-12T13:08:01.784Z
      0
      0
      3
      ...
      2675254.00
      2017-09-12T17:19:36.000Z
      147
      1.00
      19.00
      13.00
      11.00
      43.00
      44.00
      42.00

      ada53b337d27f18d07e0887771e480401dcfd3e7
      https://www.buzzfeed.com/johnpaczkowski/iphone...
      How Apple Built An iPhone Camera That Makes Ev...
      2017-09-22 13:49:20.295
      2017-09-20 21:58:16
      1
      0.10
      2017-09-22T14:00:07.330Z
      0
      0
      1
      ...
      2699863.00
      2017-09-24T01:30:00.000Z
      147
      0.00
      19.00
      12.00
      11.00
      42.00
      42.00
      42.00

      495281dd90c11682f9d07d81e498c14b7eb3c24d
      https://www.buzzfeed.com/adambvary/summer-box-...
      Why Hollywood Bombed So Badly This Summer
      2017-09-03 16:20:05.028
      2017-09-01 21:50:22
      2
      0.08
      2017-09-03T16:34:00.954Z
      0
      0
      2
      ...
      2664073.00
      2017-09-03T22:31:01.000Z
      147
      1.00
      19.00
      13.00
      11.00
      43.00
      44.00
      42.00

      5d3a94d93655aedcc6c067b8dc6e59cdb472e92d
      https://www.buzzfeed.com/jtes/women-are-spendi...
      Women Are Spending Years In Prison Because Wyo...
      2017-09-19 12:06:15.097
      2017-09-15 20:37:05
      0
      0.00
      2017-09-19T12:20:01.415Z
      0
      0
      0
      ...
      2696756.00
      2017-09-20T18:34:00.000Z
      147
      0.00
      19.00
      8.00
      11.00
      38.00
      38.00
      38.00

      47c7627b33e6ac909d597b753965d91ed1cdf932
      https://www.buzzfeed.com/albertsamaha/they-can...
      Why A Cop With A Tattoo That Looks Like A Nazi...
      2017-09-22 12:07:28.784
      2017-09-20 15:40:15
      1
      0.10
      2017-09-22T12:18:10.717Z
      0
      0
      1
      ...
      2698893.00
      2017-09-22T14:16:06.000Z
      147
      0.00
      19.00
      7.00
      11.00
      37.00
      37.00
      37.00

      0feb3bc421d776d3efcbf5e4aef07fd3be80b028
      http://www.dailymail.co.uk/news/article-488703...
      Explosion on underground train in south London
      2017-09-15 07:45:12.603
      2017-09-15 07:41:53
      0
      194.08
      2017-09-15T09:10:01.432Z
      0
      0
      0
      ...
      12417198.00
      2017-09-15T20:00:22.000Z
      158
      0.00
      18.00
      1.00
      14.00
      33.00
      33.00
      33.00

      f0e103cc56d6fb0e459bedf75d9db013d20a7bf7
      https://www.buzzfeed.com/buzzfeednews/watch-am...
      Watch Our New Morning Show AM To DM!
      2017-09-25 12:31:22.415
      2017-09-24 22:49:46
      39
      0.23
      2017-09-25T14:42:10.589Z
      1
      28
      10
      ...
      2700472.00
      2017-09-25T13:57:00.000Z
      147
      6.00
      19.00
      9.00
      11.00
      39.00
      45.00
      33.00

      61afb108d4605bff4febdb14172eaa3ce34b8f91
      http://www.dailymail.co.uk/news/article-488457...
      Fights break out ahead of Cologne Arsenal match
      2017-09-14 14:48:25.961
      2017-09-14 14:45:23
      0
      244.27
      2017-09-15T10:06:01.808Z
      0
      0
      0
      ...
      12394441.00
      2017-09-14T15:30:02.000Z
      158
      0.00
      18.00
      1.00
      14.00
      33.00
      33.00
      33.00

      f7fdcfd9351b25d1769fe7def13c015e14ed5539
      http://www.dailymail.co.uk/news/article-485784...
      Hurricane Irma slams into Caribbean
      2017-09-06 12:18:23.755
      2017-09-06 12:16:18
      0
      57.50
      2017-09-06T13:46:01.586Z
      0
      0
      0
      ...
      12305933.00
      2017-09-07T00:00:20.000Z
      158
      0.00
      18.00
      1.00
      14.00
      33.00
      33.00
      33.00

      d3cfc4bed55b56f152d3b746296c544a8634d325
      https://www.buzzfeed.com/jonathanwachtel/we-ma...
      We Made Allies Of Our WWII Enemies. Why Not No...
      2017-09-28 21:19:27.919
      2017-09-28 19:52:36
      61
      0.16
      2017-09-28T23:32:04.353Z
      9
      46
      6
      ...
      2709015.00
      2017-10-01T23:00:00.000Z
      147
      8.00
      19.00
      10.00
      11.00
      40.00
      48.00
      32.00

      e664865f79868e9e592eb856e9089233a10f81ed
      https://www.buzzfeed.com/lesterfeder/a-priest-...
      People Are Watching This Election To See If Fa...
      2017-09-16 13:21:16.484
      2017-09-14 17:13:19
      0
      0.00
      2017-09-16T13:34:02.317Z
      0
      0
      0
      ...
      nan
      NaN
      147
      0.00
      19.00
      13.00
      0.00
      32.00
      32.00
      32.00

      005982e350987fc505cde0b5c01d8b311d574522
      https://www.buzzfeed.com/vanessagrigoriadis/se...
      Campus Rapes Drop When Women Learn Self-Defens...
      2017-09-01 19:06:15.799
      2017-09-01 17:14:50
      116
      0.23
      2017-09-02T17:02:00.227Z
      13
      68
      35
      ...
      2664820.00
      2017-09-04T15:32:44.000Z
      147
      10.00
      19.00
      12.00
      11.00
      42.00
      52.00
      32.00

      bcca0a691fb67db10058303b6ce8840849d5e0b0
      https://www.buzzfeed.com/terrelljstarr/ukraine...
      Ukraine Wants Trump To Make Up His Mind About ...
      2017-09-20 14:57:14.858
      2017-09-18 21:02:38
      1
      0.08
      2017-09-20T15:10:02.596Z
      0
      0
      1
      ...
      nan
      NaN
      147
      0.00
      19.00
      12.00
      0.00
      31.00
      31.00
      31.00

      e0fc5343a1736075c389de6d0c15c1a60aadc1fd
      https://www.buzzfeed.com/franciswhittaker/sout...
      South Korea Says The North May Be Preparing An...
      2017-09-04 08:54:16.171
      2017-09-04 08:39:38
      144
      0.67
      2017-09-04T09:08:01.596Z
      31
      87
      26
      ...
      2664924.00
      2017-09-04T17:33:18.000Z
      147
      12.00
      19.00
      13.00
      11.00
      43.00
      55.00
      31.00

      ef7366fbf79ac24d0aea3bef2dad525edeaa5394
      https://www.buzzfeed.com/karstenschmehl/german...
      Trolls Are Trying To Hijack The German Electio...
      2017-09-04 08:48:17.834
      2017-09-01 15:30:52
      1
      0.08
      2017-09-04T09:02:00.661Z
      0
      0
      1
      ...
      nan
      NaN
      147
      0.00
      19.00
      12.00
      0.00
      31.00
      31.00
      31.00

      67850d9d2c463095a15a63bf7bc7d2722c5bc096
      https://www.buzzfeed.com/leticiamiranda/workin...
      Working In America Is Way Better Now Than It W...
      2017-09-04 12:03:20.422
      2017-09-03 14:05:41
      2
      0.03
      2017-09-04T13:18:01.230Z
      0
      0
      2
      ...
      nan
      NaN
      147
      1.00
      19.00
      13.00
      0.00
      32.00
      33.00
      31.00

      ec5e19ae42d089e82f0d3591a678525f526f7d62
      https://www.buzzfeed.com/katiehasty/louis-ck-i...
      Louis C.K.’s Asking For It
      2017-09-14 20:20:05.667
      2017-09-14 03:32:55
      70
      0.47
      2017-09-15T01:44:01.911Z
      24
      31
      15
      ...
      2679509.00
      2017-09-16T21:45:00.000Z
      147
      8.00
      19.00
      9.00
      11.00
      39.00
      47.00
      31.00

      d4ef0955d3cd7455a199a93ea4f896c9380048f7
      https://www.buzzfeed.com/charliewarzel/mark-zu...
      Mark Zuckerberg Can’t Stop You From Reading Th...
      2017-09-25 00:19:29.007
      2017-09-23 19:06:04
      2
      0.18
      2017-09-25T00:31:05.857Z
      0
      0
      2
      ...
      nan
      NaN
      147
      1.00
      19.00
      11.00
      0.00
      30.00
      31.00
      29.00

      ca2ec806e28f815f4803cafb9947ce3d1f03f0fc
      http://www.huffingtonpost.com/entry/independen...
      Selling An Abortion Clinic Might Be Even Harde...
      2017-09-01 02:10:06.117
      2017-09-01 02:01:18
      94
      0.40
      2017-09-02T01:10:01.229Z
      45
      35
      14
      ...
      9708625.00
      2017-09-01T21:01:19.000Z
      215
      9.00
      17.00
      8.00
      13.00
      38.00
      47.00
      29.00

      a6953cfdcd36c970df9859458d7570a99e41c674
      http://www.independent.co.uk/news/uk/politics/...
      Brexit: deportations of EU citizens soar since...
      2017-09-10 20:57:12.905
      2017-09-07 18:09:13
      3
      0.17
      2017-09-10T21:10:01.051Z
      0
      0
      3
      ...
      7421886.00
      2017-09-11T11:46:00.000Z
      386
      1.00
      16.00
      1.00
      13.00
      30.00
      31.00
      29.00

      f14ab3386893187ad35ff2431d8f69f5055b3168
      https://www.buzzfeed.com/amberjamieson/beaumon...
      Beaumont Evacuees Are Flown To Dallas As Flood...
      2017-09-01 22:00:21.801
      2017-09-01 21:59:50
      244
      0.84
      2017-09-02T18:54:00.747Z
      35
      156
      53
      ...
      2663099.00
      2017-09-02T17:03:00.000Z
      147
      14.00
      19.00
      12.00
      11.00
      42.00
      56.00
      28.00

      97ec34ddd972abea0c8eaf1e5c5eedb69f34b830
      https://www.buzzfeed.com/buzzfeednews/live-upd...
      Live Updates: Floridians Struggle To Begin Cle...
      2017-09-13 11:45:16.474
      2017-09-13 11:44:39
      18
      0.08
      2017-09-13T16:06:01.902Z
      0
      0
      18
      ...
      nan
      NaN
      147
      3.00
      19.00
      12.00
      0.00
      31.00
      34.00
      28.00

      9ebf1287ea5d6cef8d4225c9a8ade06dbccc62ac
      http://www.bbc.co.uk/news/live/uk-england-lond...
      BBC Local Live: Updates from London on Monday ...
      2017-09-10 15:33:04.406
      2017-09-10 15:29:43
      0
      0.00
      2017-09-11T01:04:00.103Z
      0
      0
      0
      ...
      43999822.00
      2017-09-15T09:19:55.000Z
      96
      0.00
      0.00
      13.00
      15.00
      28.00
      28.00
      28.00

      c7537e28cc23047329945e9c80c7323055d41ffe
      https://www.buzzfeed.com/zanromanoff/how-the-k...
      How The Kardashians Reflect Changing Ideas Aro...
      2017-09-15 21:50:05.919
      2017-09-15 14:48:23
      313
      1.71
      2017-09-16T14:48:01.071Z
      56
      201
      56
      ...
      2679435.00
      2017-09-16T16:44:00.000Z
      147
      16.00
      19.00
      13.00
      11.00
      43.00
      59.00
      27.00

25 rows × 26 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      847e11ad09dcdffaaf4746032ad1664a1067da8a
      http://www.huffingtonpost.com/entry/if-the-pro...
      If The Protest Outrages You, But The Cause Doe...
      2017-09-25 20:46:24.106
      2017-09-25 20:40:30.716
      109288
      235.98
      2017-09-27T00:20:11.580Z
      15892
      80756
      12640
      ...
      nan
      NaN
      215
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      2812dae6a98764e8b1c9b2f3fc6c370047793546
      https://www.buzzfeed.com/javieraceves/las-tien...
      Canadá envió 1500 tiendas de campaña y el SAT ...
      2017-09-28 17:19:32.150
      2017-09-28 17:10:22.000
      337311
      367.95
      2017-09-29T03:01:05.580Z
      40454
      199045
      97812
      ...
      nan
      NaN
      147
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      bc24ca5aedc56346e619a6b1c05374338f7ff826
      https://www.washingtonpost.com/local/education...
      Senate panel rejects Trump’s proposed  federal...
      2017-09-07 22:12:09.409
      2017-09-07 22:05:00.000
      111176
      209.29
      2017-09-10T19:04:00.023Z
      6724
      96145
      8307
      ...
      nan
      NaN
      191
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      423bab68e6dbb29153f05f4036e5adf84decefc2
      http://www.cnn.com/2016/01/11/entertainment/da...
      David Bowie, a master of music and makeovers, ...
      2017-09-08 16:51:14.029
      2017-09-08 16:46:06.000
      118977
      0.03
      2017-09-08T18:06:02.670Z
      8971
      100422
      9584
      ...
      nan
      NaN
      105
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      0e6098488737a461a546cb6ddadc70db9c220fc3
      https://www.washingtonpost.com/national/health...
      Police officer who arrested Utah nurse fired f...
      2017-09-05 21:39:12.763
      2017-09-05 21:33:13.000
      246436
      2698.89
      2017-09-06T00:58:01.409Z
      30634
      189357
      26445
      ...
      nan
      NaN
      191
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      2ac8e6d2202d8c603b8fb20f6b3defa24f357c61
      http://www.washingtonpost.com/video/local/weat...
      San Juan mayor: 'There is horror in the streets'
      2017-09-23 21:58:22.408
      2017-09-23 20:41:42.000
      109615
      213.36
      2017-09-25T23:04:02.843Z
      15658
      72891
      21066
      ...
      nan
      NaN
      191
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      d10e7ebabab09ec6a72423d3995796f9536c23bc
      https://www.buzzfeed.com/tatianafarah/veja-30-...
      Veja 30 obras da exposição censurada no Santan...
      2017-09-12 21:24:18.661
      2017-09-12 20:53:22.000
      174607
      154.33
      2017-09-13T02:28:01.800Z
      57836
      102143
      14628
      ...
      nan
      NaN
      147
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      22f025d7af4929e90cbb43329af351961cbe3dbd
      http://www.cnn.com/2017/09/18/americas/hurrica...
      Hurricane Maria strengthens to Category 5
      2017-09-19 00:03:16.021
      2017-09-18 23:59:39.000
      144371
      1366.45
      2017-09-19T03:22:00.487Z
      33573
      88346
      22452
      ...
      nan
      NaN
      105
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      3942b12bb0a8dddd8b1a5c962a5b0d2f800684d6
      https://www.nytimes.com/2017/09/01/business/mi...
      Michael Dell, Tech Billionaire, Pledges $36 Mi...
      2017-09-01 12:18:01.060
      2017-09-01 12:16:21.000
      134143
      141.86
      2017-09-01T16:12:00.598Z
      7974
      112734
      13435
      ...
      nan
      NaN
      120
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      6d54a9c6501d7a327248c579b685de99ae0c3456
      https://www.nytimes.com/2017/09/28/us/jones-ac...
      Jones Act Suspended, Easing Aid Shipments to P...
      2017-09-28 12:43:08.600
      2017-09-28 12:41:38.000
      81931
      191.13
      2017-09-28T13:55:03.833Z
      17266
      52888
      11777
      ...
      nan
      NaN
      120
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      d81a87a3b13f2622a78aeb0c6405f0cfafee7deb
      http://www.dailymail.co.uk/news/article-489852...
      Navy staff probed over 'sick' pictures with ne...
      2017-09-20 07:09:16.336
      2017-09-19 10:40:08.000
      83538
      26.65
      2017-09-20T12:42:00.165Z
      22255
      55205
      6078
      ...
      nan
      NaN
      158
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      5505aed7fe7dec69e83ab5cdeb8e030fe9a98d46
      https://www.reuters.com/article/us-storm-maria...
      U.S. denies request for Puerto Rico shipping w...
      2017-09-26 20:04:17.320
      2017-09-26 19:57:23.000
      83962
      332.54
      2017-09-27T07:05:11.220Z
      22198
      47714
      14050
      ...
      nan
      NaN
      497
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      97b4941e7757e6073c9d85c39b347c611278e0f6
      http://www.dailymail.co.uk/news/article-484567...
      John Lewis ditches 'boys' and 'girls' labels o...
      2017-09-01 22:51:20.523
      2017-09-01 22:48:46.000
      96611
      6659.17
      2017-09-03T21:19:59.809Z
      28346
      62353
      5912
      ...
      nan
      NaN
      158
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      be8f14c73f781b72f4f095506860d428f0e6b4da
      https://www.reuters.com/article/storm-maria-pu...
      U.S. denies request for Puerto Rico shipping w...
      2017-09-26 19:49:09.431
      2017-09-26 19:45:42.000
      84393
      745.32
      2017-09-27T01:05:07.085Z
      22381
      47901
      14111
      ...
      nan
      NaN
      497
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      442a1ac6371ba39f4b0ae21937357ab8eda0b9bd
      http://insider.foxnews.com/2017/09/11/tomi-lah...
      Tomi Lahren: Final Thoughts on Sunday Kneeling...
      2017-09-11 23:05:04.997
      2017-09-11 18:15:00.000
      141071
      608.27
      2017-09-11T23:18:02.042Z
      16388
      108693
      15990
      ...
      nan
      NaN
      285
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      300aa0307f1b99da03abd9d37f28ef2e75a9d9ae
      https://www.buzzfeed.com/stephaniemcneal/love-...
      This Bride Had A Bouquet Made Of Doughnuts On ...
      2017-09-12 16:57:24.308
      2017-09-12 16:57:06.000
      130965
      419.33
      2017-09-13T19:07:59.758Z
      69608
      56040
      5317
      ...
      nan
      NaN
      147
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      74bb44bb13cdf421f83f15272dfd5f15801b4629
      http://www.foxnews.com/politics/2017/09/27/mic...
      Michelle Obama slams women who voted for Trump
      2017-09-27 20:54:16.608
      2017-09-27 20:45:00.000
      75781
      197.16
      2017-09-28T16:32:08.848Z
      28174
      42243
      5364
      ...
      nan
      NaN
      285
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      af7e6bb3fdf38ac1bf37f696bb94fe1099af7552
      https://www.washingtonpost.com/news/post-polit...
      Trump’s son-in-law Jared Kushner used a privat...
      2017-09-24 22:58:11.527
      2017-09-24 22:53:59.000
      78368
      3045.95
      2017-09-25T03:38:09.964Z
      15447
      53819
      9102
      ...
      nan
      NaN
      191
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      aef9af4a52f05dee48fb98ece49cc8f870fd5197
      https://www.washingtonpost.com/news/checkpoint...
      Clinton pressed Trump to deploy hospital ship ...
      2017-09-26 20:37:16.766
      2017-09-26 20:33:12.000
      127262
      135.18
      2017-09-27T02:50:03.001Z
      22279
      87002
      17981
      ...
      nan
      NaN
      191
      50.00
      0.00
      3.00
      0.00
      3.00
      53.00
      -47.00

      d4a0b1812294d9d7df73a5bea80e8c9fc3f38118
      http://www.foxnews.com/entertainment/2017/09/2...
      Canada's prime minister ridiculed for Chewbacc...
      2017-09-20 23:10:06.252
      2017-09-20 22:41:25.000
      78990
      312.46
      2017-09-22T05:02:11.097Z
      19270
      55883
      3837
      ...
      nan
      NaN
      285
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      d26aabc0297fb47a6f71c65954275629c462c029
      http://www.foxnews.com/entertainment/2017/09/0...
      Country music star Troy Gentry, 50, dies in Pe...
      2017-09-08 20:20:04.613
      2017-09-08 20:05:00.000
      94949
      292.25
      2017-09-08T21:10:02.516Z
      14841
      61660
      18448
      ...
      nan
      NaN
      285
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      a02df93727c5975bc2506b78c5d9ed65f813daff
      http://www.foxnews.com/world/2017/09/22/doomsd...
      Doomsday is not on Saturday after all, Christi...
      2017-09-22 14:49:14.541
      2017-09-22 14:33:18.000
      79705
      80.43
      2017-09-23T11:24:04.973Z
      25033
      45219
      9453
      ...
      nan
      NaN
      285
      48.00
      0.00
      2.00
      0.00
      2.00
      50.00
      -46.00

      d50425525120f1de4f0438b91189e93eafd7aa39
      https://www.nytimes.com/2017/09/16/health/bras...
      Como a Grande Indústria Viciou o Brasil em Jun...
      2017-09-16 12:51:00.660
      2017-09-16 12:50:37.000
      52380
      33.13
      2017-09-17T16:51:59.718Z
      5229
      37901
      9250
      ...
      nan
      NaN
      120
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      11134532da6de27a27478bf59ee8b7ea49788ce7
      https://www.buzzfeed.com/florapaul/habilite-o-...
      Mulheres, atualizem o iPhone de vocês para o i...
      2017-09-21 15:22:25.861
      2017-09-21 15:05:50.000
      54545
      104.86
      2017-09-22T14:23:07.298Z
      9430
      41056
      4059
      ...
      nan
      NaN
      147
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      d24f06cf636952c454dcda24fa51db7e4f899987
      https://www.washingtonpost.com/news/acts-of-fa...
      Why this Muslim-turned-Christian speaker reson...
      2017-09-17 20:15:12.625
      2017-09-17 20:04:55.000
      90744
      135.13
      2017-09-17T22:32:00.253Z
      5436
      71191
      14117
      ...
      nan
      NaN
      191
      49.00
      0.00
      3.00
      0.00
      3.00
      52.00
      -46.00

25 rows × 26 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_2017-09-01_2017-09-30.csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}

```python

```