There is a fundamental rule in database theory that when data is recorded in a database, it is "immutable". It cannot be changed. Applications may act on the data, but the integrity of the underlying data is intact.

Consider a database tracking temperatures over time. The temperature sensor at my house might record into the database that it is 31 degrees F right at this time and date.

That data should always remain intact.

If I query the temperature database tomorrow for today's temperature at 8am, the database should say that it was 31 degrees F. If I query the database 5 months from now... or 5 years from now... the database should always spit back the 31 degree temperature.

The historical answer will always be identical.

This is just a fundamental principle of databases that are tracking data over a period of time.

But personally I'd love history to be kept intact. Show the change in the algorithm NOW. Sure, the trend graph would show a big drop. Okay. Then, like in Google Analytics, we can all make a notation that the algorithm was changed on such-and-such a date and our score now reflects the new algorithm. No big deal.

The Counterpoint

But what if the algorithm had a fundamental error in it? Shouldn't you go back and revise all the data?

Consider my temperature example - what if I found that the thermometer in my house was actually off by 4 degrees? That it was actually 4 degrees colder outside that it was showing?

Wouldn't it make sense to go back and change all the historical readings for that sensor to be 4 degrees colder? (Assuming I could pinpoint the time at which it started being inaccurate... or just made the assumption that it had always been inaccurate.)

And yes... there's certainly a school of thought that says you should go back and revise history. The other school of thought would be to leave history alone and indicate that from this point forward the sensor data will now be more accurate.

It's obvious which school of thought Klout fits in.

Klout's Ecosystem "Problem"

The "problem" Klout has... and I put "problem" in quotes because it's the kind of "problem" any small startup would LOVE to have... is that they've had a lot of companies and developers using Klout's APIs to build other applications and systems that interact with Klout's metric. In fact, Klout is claiming over 3500 "partners and developers".

And you have to imagine that some % of those developers are engaging in tracking Klout scores over time. They want to track the trend of their own score... or their competitors score... or their clients' scores... or whatever.

All of that trend data just got rendered inaccurate.

It doesn't matter if Application X says that your client had a Klout score of 43 last week.... the official Klout database now says that the client's score was really 32... and it never was 43.

Oops. Now the application has "bogus" data.

Klout's Reporting Problem

Plus, if you were presenting reports or charts regularly to a client (or your management) showing them their Klout score, now you have to go back to the client and say "I'm sorry, but Klout revised their algorithm and you never had that score I told you."

You look like an idiot for trusting a metric that changes like this.

Of course, you're not alone, as Bob LeDrew so eloquently pointed out in his post yesterday "A Klout Upside The Head"... obviously many people are taking Klout's metric very seriously. (And way more seriously than I would even remotely consider.)

The fact that some people are using Klout's metric for business decisions would, in my mind, point to Klout needing to consider historical accuracy/integrity a bit stronger.

Sure, change the algorithm if you need to... but keep the history intact so that your partners and users don't look like idiots.

A Wake-up Call?

In the end will this kerfuffle make people be a little bit more critical of the Klout Score?Will people realize it is only one of the metrics they should consider? Will they take a look at other metrics that are emerging?

Will people realize that they shouldn't blindly rely on one simple metric?

While I'd love to believe people might - and we can only hope that at least some people will, I guess I'm cynical enough to think that people want nice, simple, easy metrics... and Klout is delivering that. Give it a few days for all this to blow over and sadly people will probably be right back caring about their Klout Score.

Only now perhaps they'll take occasional screenshots to be able to back up later claims about the score whenever Klout does its next revision of history...

Comments

Klout's Other Major Fail: Violating Historial Integrity/Accuracy

There is a fundamental rule in database theory that when data is recorded in a database, it is "immutable". It cannot be changed. Applications may act on the data, but the integrity of the underlying data is intact.

Consider a database tracking temperatures over time. The temperature sensor at my house might record into the database that it is 31 degrees F right at this time and date.

That data should always remain intact.

If I query the temperature database tomorrow for today's temperature at 8am, the database should say that it was 31 degrees F. If I query the database 5 months from now... or 5 years from now... the database should always spit back the 31 degree temperature.

The historical answer will always be identical.

This is just a fundamental principle of databases that are tracking data over a period of time.

But personally I'd love history to be kept intact. Show the change in the algorithm NOW. Sure, the trend graph would show a big drop. Okay. Then, like in Google Analytics, we can all make a notation that the algorithm was changed on such-and-such a date and our score now reflects the new algorithm. No big deal.

The Counterpoint

But what if the algorithm had a fundamental error in it? Shouldn't you go back and revise all the data?

Consider my temperature example - what if I found that the thermometer in my house was actually off by 4 degrees? That it was actually 4 degrees colder outside that it was showing?

Wouldn't it make sense to go back and change all the historical readings for that sensor to be 4 degrees colder? (Assuming I could pinpoint the time at which it started being inaccurate... or just made the assumption that it had always been inaccurate.)

And yes... there's certainly a school of thought that says you should go back and revise history. The other school of thought would be to leave history alone and indicate that from this point forward the sensor data will now be more accurate.

It's obvious which school of thought Klout fits in.

Klout's Ecosystem "Problem"

The "problem" Klout has... and I put "problem" in quotes because it's the kind of "problem" any small startup would LOVE to have... is that they've had a lot of companies and developers using Klout's APIs to build other applications and systems that interact with Klout's metric. In fact, Klout is claiming over 3500 "partners and developers".

And you have to imagine that some % of those developers are engaging in tracking Klout scores over time. They want to track the trend of their own score... or their competitors score... or their clients' scores... or whatever.

All of that trend data just got rendered inaccurate.

It doesn't matter if Application X says that your client had a Klout score of 43 last week.... the official Klout database now says that the client's score was really 32... and it never was 43.

Oops. Now the application has "bogus" data.

Klout's Reporting Problem

Plus, if you were presenting reports or charts regularly to a client (or your management) showing them their Klout score, now you have to go back to the client and say "I'm sorry, but Klout revised their algorithm and you never had that score I told you."

You look like an idiot for trusting a metric that changes like this.

Of course, you're not alone, as Bob LeDrew so eloquently pointed out in his post yesterday "A Klout Upside The Head"... obviously many people are taking Klout's metric very seriously. (And way more seriously than I would even remotely consider.)

The fact that some people are using Klout's metric for business decisions would, in my mind, point to Klout needing to consider historical accuracy/integrity a bit stronger.

Sure, change the algorithm if you need to... but keep the history intact so that your partners and users don't look like idiots.

A Wake-up Call?

In the end will this kerfuffle make people be a little bit more critical of the Klout Score?Will people realize it is only one of the metrics they should consider? Will they take a look at other metrics that are emerging?

Will people realize that they shouldn't blindly rely on one simple metric?

While I'd love to believe people might - and we can only hope that at least some people will, I guess I'm cynical enough to think that people want nice, simple, easy metrics... and Klout is delivering that. Give it a few days for all this to blow over and sadly people will probably be right back caring about their Klout Score.

Only now perhaps they'll take occasional screenshots to be able to back up later claims about the score whenever Klout does its next revision of history...