The Data is the Query

As each new piece of data arrives in the enterprise, the enterprise just learned something. And with each new observation one should ask, “How does this relate to what I already know? Does this matter and, if so, to whom?"

This is the world of sense and respond, situational awareness, sensemaking or whatever you want to call it.

Example: An employee in the bank’s credit department changes his home address in the payroll system. What if the employee's new address is the same address currently under investigation by the bank's own fraud department? How would the bank know?

They wouldn't.

When the data is the query, a change to a home address in the payroll system is determined, at that split second, to be the same address under investigation by the fraud department. And, at that split second this is determined to be relevant, so the fraud department is notified.

Real-time, sub-second.

When organizations can process arriving observations for relevance … organizations will be more competitive, or might even, for the first time, seem to be “awake.”

Note: The systems/technologies that are going to do this are very different than what organizations have in place today. Existing operational systems cannot do this. Neither can master data management systems, data warehouses, operational data stores, data mining engines … not even Hadoop Map/Reduce.

Comments

You can follow this conversation by subscribing to the comment feed for this post.

I agree that the systems/technologies that an org will need to create situational awareness with their data will be very different than what they have today. I also agree that data management environments (big data, rdbms, etc) in of themselves will not solve this problem, however, I believe the beginning steps in the right direction are establishing a robust scalable data layer not bound by the constrains of a typical rdbms, it is this scalable data layer that will enable the advanced analytics (which also do not exist in most orgs) to begin to answer some of the questions you pose.

I applaud your efforts to elevate this topic to a business discussion. Creating meaningful use cases will help bridge what has been, for too long, localized and academic efforts to solve parts of this problem.

Architectural patterns like pub-sub/ESB, data warehouse/datamart, and classification/association have been used to attack this problem "in the small" - not always at the enterprise level (as in, scope of business and range of data) as you suggest.

I don't think it's a problem of lack of tools and methods, as there has been a lot of technical work on this in the KDD (Knowledge Discovery and Data Mining) community, as well as for text the IR (Information Retrieval) community.

The problem is in the way businesses (and IT) think about and organize their approach to data management. More attention to the possibilities from the business might drive better solutions.

Architectural patterns like pub-sub/ESB, data warehouse/datamart, and classification/association have been used to attack this problem "in the small" - not always at the enterprise level (as in, scope of business and range of data) as you suggest.