Turn away from the dark side: 80% of all data is considered dark

“It takes great strength to resist the dark side. Only the weak embrace it." — Obi-Wan Kenobi
At CoEfficient, we not only love great Star Wars analogies, we also love data of all types, not just accounting data. Accounting data + contextual data always leads to better decisions. We found this presentation by Stanford professor Chris Ré about dark data fascinating: https://youtu.be/_fBMmQo-Z4E

The presentation focuses on scientific data and the problem that 80% of all data is considered “dark". It’s unstructured and largely inaccessible. Finding ways to bring this data into a usable format or to artificially read / process this data could greatly accelerate innovation.

So what does this have to do with accounting and data-driven decision making? How much of the data generated by your company every day is “dark”? Is the data generated by the activities of your company collected, curated, and reported in a way that leads to timely and informed decisions?

Here are a few ways we think most companies have unnecessarily dark data:

No connection between data sources / data in silos: Many companies have adopted multiple software tools to address different problems in different parts of the organization. Many times that data is evident to that part of the organization, but not available in relation to other operating data nor to the entire team. Bringing siloed data into the context of financial results or other operational data can lead to deeper insights and better decisions.

Inconsistent application of data standards: This is a massive problem we see with CRM systems. Different sales people using different fields to record similar activities or using different definitions for certain sales stages. The result is data that can be misleading or just worthless. We realize getting sales people to act in a uniform way is akin to herding cats, but with some outside data management support and clear definitions, good habits can be reinforced.

N = 1 (limited summarization): This seems like a simple one, but often data from activities like quality management data sets doesn’t have smart grouping, so everything looks like an anomaly and its hard to prioritize actions. In the example of quality control data, if every cause has an N-size of 1, how do you prioritize your improvement efforts? Smart summary data will help you make sense of bigger operational data sets. Establishing the right buckets is an important step and getting it wrong can also lead to poor decisions – so be careful when creating summary buckets.

Limited trending: The human brain loves to see patterns. Patterns can lead to insight. Given this, why do we often settle for snapshot data without the context of trending data from past results? This is a really simple step, but if you take the right metrics and look at trends over time, patterns or problems will stand out. Optimal decisions will be clearer.

While Professor Ré’s presentation was about scientific data, we believe most companies suffer from similar problems, and the 80% ratio is a pretty good guess for the amount of dark data lurking. This is where CoEfficient steps in. By leveraging solid accounting, smart analysis, data management, and CoEfficient View as a reporting tool, we can help you see your data in a new light.