This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

What To Do About False-Positive Alerts, Non-Collaborative BI and Inaccurate Metrics

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives. Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since. A piece of the solution to both problems is an engine for setting and modifying metrics definitions.

I've been hinting at some points for quite a long time, without really spelling them out in written form. So let's fix that. I believe:

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives.

Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since.

A large, necessary piece of the solution to both problems is a great engine for setting and modifying metrics definitions.

I shall explain.False positives on alarm systems are a huge problem. Alarms were ignored (indeed, turned off) on the Deepwater Horizon due to too many being false. When I had a fire in my house, the smoke alarm wasn't on because of its penchant for earsplitting false positives. And false positives held back the software category of intrusion detection systems until it was eventually subsumed by related kinds of security appliance.
Similarly, false positives are an under-appreciated problem that restrains the advance of business intelligence software. It's no big inconvenience when, out of your 3 biggest problems this week, your dashboard calls attention to 17 of them. But suppose you want to make alerts more granular, take them up in quantity a couple of orders of magnitude, and have your smartphone pinged each time the alert condition is met. Whoa. Now those bogus warnings start to be more of a consideration.
Why would you want some kind of "push" alerting? Use case circumstances include but are not limited to anything that might herald:

A sudden market trend change (if you're in the kind of consumer market where significant developments an occur in hours or less)

An investment opportunity or threat, or a financial risk threshold being breached (if you're an investor, if you buy or sell commodities, or if you're a corporate treasurer).

An attitude change on the part of a specific important customer (sales contact, service contact, news development, whatever).

The advisability of rerouting or rescheduling something.

Why would these alerts ideally be granular and high-volume? Well, you might have lots of machines, lots of trucks, lots of customers, lots of potential investments, and so on. I could list many more examples, but these probably suffice to at least start the discussion.

I'm agnostic about what the UI would be. Sounds going off on your mobile phone? Email on your mobile device? An app more like a chat window? Something more like a mobile dashboard? Whatever it is, false positives would really screw up your work day.

The simpler and more top-down all this is, the better it creates enterprise-wide unity of vision/focus/purpose/blahblahblah.

Yes, you can do drilldown or even data exploration, QlikView-style or otherwise. You can communicate around the BI tools, whether via general portal technology or something built-in. Vendors keep trying to make it easier for you to build your own reports, charts, and dashboard elements. But you can't very easily define and track your own metrics, and I don't see a lot of effort toward making that better.

Perhaps vendors don't try to provide strong metrics management because doing so might lead to -- gasp! -- more than "one version of the truth." If so, such thinking is regrettably misguided. Examples of formulas -- i.e., metrics -- that should and can not be cast in stone* include:

Profitability of a customer

Expected lifetime value of a customer

Anything else which boils down to "importance of a customer"

Profitability, value, or importance of a lead or prospect action

Significance of a social media flare-up

Cost of a schedule slip

Risk of a portfolio

*At least as a general rule -- we surely can think of some simple cases or exceptions where extreme cookie-cutterness is just fine.

It's not that there shouldn't be preferred or default metrics for these quantities. Rather, my point is that individuals should be allowed to:

Question those metrics

Consider and test alternatives to those metrics

Make decisions based on those alternatives (as long as they also keep the results and implications of the approved/default versions of the metrics in mind)

Why? Because the person closest to the situation may be the one best able to assess it, and also because recognizing that fact is a really good idea from the standpoints of employee morale, engagement, and even development. (Or you might say -- shiny buzzword! -- employee "empowerment".) In particular, when a boss and her subordinate disagree about the ideal way to calculate a metric, they should be able to track both versions side-by-side. When there's a significant difference between the two, that could be a fine time for an out-of-the-norm decision.* If that ability were available, portal-style BI collaboration would almost automatically come into its own.

*What kind of decision that is of course depends on the specifics of the case. If only one of two competing metrics says you should bother giving extra care to a specific customer, the answer is easy - you give it. In other cases, an actual meeting or other conversation may be in order to decide what to do.

A metric is just a function. To offer BI users the flexibility I want, it should be very easy to use these functions as inputs into other, custom functions. I.e., I'm talking about something with the flexibility of a stripped down Excel, although the UI would be more like that for a rules engine. And if you think about it, that's exactly the same functionality needed to personalize your alerts and weed out false positives.

One last point -- what I'm calling for could greatly increase the BI performance burden, perhaps even by three orders of magnitude. Well, that's what we have all this great new super price-performance analytic database technology for. And anyhow the throughput hit won't come overnight. It will be interesting to see where the boundaries end up between DBMS and inside-the-BI-tool analytic processing. But overall, the analytic DBMS industry -- and the hardware vendors backing them up -- should be able to handle anything the BI vendors throw at them.

Related links

Gooddata may be a little more oriented toward what I'm talking about here than some bigger vendors are.

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives. Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since. A piece of the solution to both problems is an engine for setting and modifying metrics definitions.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.

This IT Trend Report will help you gain insight into how quickly and dramatically data science is influencing how enterprises are managed and where they will derive business success. Read the report today!