This post isn’t meant to pick on NewsWhip. They do good work and offer a helpful product. But the focus of this report has become all too common: Ignoring what is important while obsessing over flawed metrics that are often inaccurate, buggy and impossible to verify.

It’s the Faceboocalypse!

NewsWhip’s content discovery platform Spike tracks shares and social engagement to help alert PR professionals and journalists of the stories getting the most attention. Analysis of this data was the source of a reported drop in Facebook shares beginning in mid-February that continued into March.

Here’s a chart showing a clear drop in daily Facebook shares of 10 major publishers…

Looks pretty bad, right?

It was found that after analyzing the 100 most-shared English language stories, there was a clear drop of shares from 16.4 Million in January to 10.2 Million in February.

So… what in the world could be behind this?

NewsWhip went on to — naturally — connect this drop to marketing’s favorite obsession: The Reach metric…

It’s understood that a reduction in the reach of these links – how many people see them in their news feed – is behind the fall-off in engagements.

It’s understood. But not proven or verified.

Whatever was happening, the sky was clearly falling. There was no other explanation.

A staff member at one major UK publisher described the reduce in engagement as a ‘Faceboocalypse’, and said that his team had noticed what he described as ‘a change to news feed algorithm which drastically reduced the reach of many news sites’ posts.’

Shares appeared to be down. We can then assume this is due to a drop in reach (though not verified). Let’s not stop assuming about what all of this means…

It seems as though Facebook is less concerned with sending major traffic to external sites than it is with ensuring that its users can consume high quality content without having to leave the network’s familiar surrounds.

Except…

There was a pretty clear hole in the findings. NewsWhip noted that “likes and comments on the stories don’t seem to have been affected as much.” In fact, engagements remained up significantly year-over-year:

However, it’s important to note that engagements for top publishers remain very high, compared to 12 months ago. Indeed, many publishers in the top 25 have increased their engagements from 12 months ago significantly. NBC went from 9.2 million total Facebook interactions in February 2014 to over 19.8 million in February 2015. Likewise, the Daily Mail almost doubled their engagements, from 7.6 million to 14.5 million. The vast majority of sites are experiencing much higher levels of engagement than they did 12 months ago, reinforcing the fact that this is less wipe-out, more restructuring.

That’s weird, right?

It’s appreciated that NewsWhip provided those details. But concerning that they were glossed over.

You’d think that if you spotted a significant drop in social shares, you’d do a little more digging. What about other engagements? What about traffic? Is this a reporting bug, which seems to happen once or twice per year?

Things weren’t adding up. But let’s just assume that it’s because reach is dying and Facebook doesn’t want us sending traffic to our websites.

What Wasn’t Reported

There would have been a very simple way to verify this: How much traffic was being sent to these websites?

The assumptions were as follows:

Reach is down for pages

Facebook wants to keep users on Facebook, devaluing links

Shares are down because reach is down

If reach was down — causing a significant drop in shares — it would undoubtedly result in a huge drop in traffic.

So… did that happen? We don’t know. This was never reported.

Of course, the assumption that Facebook wants to keep people on the site at the expense of user experience conflicts with recent findings that reach for links for pages is way up lately — higher than any other post type.

The Correction

The NewsWhip article was updated on April 1. They were contacted by a Facebook representative who told them that a bug was behind the fall-off in reach.

We experienced an issue with Page Insights logging in February that failed to count some viral reach accurately. Importantly, actual delivery of posts was not affected by this issue; this was a reporting issue only. We identified and resolved this issue in early March and Page Insights are now correctly reporting organic reach. We apologize for any unintended impact this may have had to our partners.

Was this also behind the drop in shares? Neither the quote nor NewsWhip clarify on that. But since not all shares are publicly available information and Facebook mentioned a bug in viral reporting, it’s very likely that the drop in reach and shares are connected.

I won’t assume anything, of course. But we do know that there is a common weakness in both reach and shares that can lead to bad reporting.

The Problem with Reach and Shares

There is a major problem with obsessing over reach: It’s a flawed, fuzzy metric that cannot be verified. We trust what Facebook reports.

1. You do not know how many people actually saw your post. Facebook reports this based on impressions. But there is no way to verify it.

Contrast this with the reporting of post likes, comments or website clicks. I can dig into a post to see just how many comments and likes are attached to it. And I can compare Facebook data to website data to track traffic.

2. Facebook can change the way they report reach on a whim.This happened. And it can happen again.

There was a time when a person could be counted as being reached both organically and paid. I share my post and reach a fan. I then reach that person with the same post when it’s promoted. That counted as being reached once organically and once paid.

That is no longer the case. Now I can reach that person organically. As soon as that same person is also reached with the promoted post, they are no longer counted as being reached organically, but paid only. That way, adding Organic Reach + Paid Reach = Total Reach.

Call it what you will, but I consider that inaccurate. You can’t just discount that a person was reached organically so that the numbers add up neatly. Because of that, I firmly believe that organic reach of posts that also receive promotion is underreported.

3. There are often bugs in the reporting of reach. This is not the first time a bug in reach reporting has happened, and it won’t be the last.

Because reach is a fuzzy metric that is impossible to verify, we rely on Facebook accurately reporting it. But that doesn’t always happen. It’s not 100% dependable.

While such fuzziness is not typically associated with shares, the problem exists here as well. What Facebook reports and what you can verify are not the same.

A recent post of mine received 57 shares. When I clicked to view those shares, I get this pop-over…

When I count the shares, I only see 15 entries. Why?

Oh, there’s this little gray message at the bottom…

Facebook will only show me shares I’m supposed to see based on privacy settings. So that will typically be public shares or those from my friends.

As a result, there’s a gaping hole in this report. I can only confirm that 26% of the reported shares actually happened.

What this means is that there is plenty of room for error in the other 74%. Facebook could report that only 20 people shared the post. They could report that 2,000 people shared it.

I couldn’t verify it either way. All I know for sure is that 15 people did. And as a result, it’s dangerous to make any assumptions about surges in that data.

Does Reach Matter?

Let’s assume for a moment that reach actually did drop. If all engagement remained healthy — including website clicks and conversions — what does that drop in reach mean?

It would mean that Facebook was showing your content to people most likely to engage favorably — which is what we as marketers and users would want.

It would mean that your engagement rate would improve and the measurable actions would at least remain the same.

It would mean that Facebook is no longer showing your content to people who would have otherwise ignored you, cluttering their news feed.

If all of these things — which you can verify — remained the same while your reach dropped, the distinct possibility also exists that there is a bug in Facebook’s reporting. Or that Facebook changed the way that they report reach.

In either case, how much does your reach really matter if your traffic and conversions aren’t negatively impacted?

The typical response will be, “But, Jon! I won’t get any traffic if I don’t reach anyone!”

That’s not the argument here. No one is claiming that website traffic — the most important metric for these publishers — is down. Only that social shares and reach — two things that can’t be verified — are down.

Unless you have a report that emphatically shows a drop in website traffic and other metrics that matter, keep it to yourself. It’s misleading and helps no one.

The Metrics That Matter Most

We need to stop obsessing over fluff, vanity metrics that are often flawed and impossible to verify. Too many marketers lose sleep over them while ignoring what actually matters.