Yesterday, I had the pleasure of doing a webinar on measurement and nonprofit for Kivi at the Nonprofit Marketing Guide. One of the questions I was asked, ”Why Data Informed? Why Not Data-Driven?”

In our book, “Measuring the Networked Nonprofit“,’ KD Paine and I explain how being data-informed is something very different from a data-driven culture. The term “data-driven” has been used to describe organizations that rely solely on cold hard data to make decisions. Being data-driven sounds great—in theory. But, because it doesn’t acknowledge the importance of basing decisions on multiple information sources, it can doom an organization to epic failures.

Eric Petersen was one of the first of the data geeks that I’ve read to suggest that the phrase “data-informed” is a far more useful label. Data-informed describes agile, responsive, and intelligent businesses that are better able to succeed in a rapidly changing environment. The concept of being data-informed resonates with nonprofit and public sector practitioners as well. Data-informed cultures are not slaves to their data. Mario Morino uses the phrase “information-based introspection” to refer to using and applying data in context to excel.

Data-informed cultures have the conscious use of assessment, revision, and learning built into the way they plan, manage, and operate. From leadership, to strategy, to decision-making, to meetings, to job descriptions—a data-informed culture has continuous improvement embedded in the way it functions. Key Performance Indicators (KPIs) are the specific quantifiable metrics that an organization agrees are necessary to achieve success. They are the mileposts that tell a data-informed organization whether they are making progress toward their goals. Too often organizations chose KPIs that simply reflect activity.

Measurement is a tool that data-informed cultures use to improve their programs; they observe the results of their programs, and then learn from those results to improve and refine their next programs. Data-informed cultures design measurement into their projects—not just so they have measurable outcomes, but so they provide the data necessary to guide how to improve them.

Measurement can be used for many things, some of them undesirable, like proving a point. A data-informed culture uses measurement to continuously improve.

The article makes the point that “more data” does not help with decision-making and uses the Tesla situation as a case in point.

New York Times columnist David Brooks nails this in an op-ed piece, wherein he argues that Big Data, while very useful for guiding our intuitions, gets some things very wrong. Like the value of social connections. Or the context for answering a question. In fact, he speculates, Big Data might actually obscure Big Answers by complicating decisions and making it even harder to determine which statistically signifiant correlations between data are informative and not simply spurious.

Such thinking won’t be surprising to anyone that has read Nassim Taleb’s book The Black Swan, which posits that the more data we analyze, the more likely our conclusions will be wrong. Taleb writes:

In business and economic decision-making, data causes severe side effects – data is now plentiful thanks to connectivity; and the share of spuriousness in the data increases as one gets more immersed into it. A not well-discussed property of data: it is toxic in large quantities – even in moderate quantities.In other words, the more data you collect, the harder it can become to interpret that data. And even if you can interpret your data correctly, are you actually going to listen to that interpretation?

Which brings me back to again to the importance of not getting lost or distracted by too much data or the tools to collect them. Pick that one data point that tells you whether you or not been successful – and bring your organization’s wisdom to understanding and applying it to improve what you’re doing.

What do you think? Is there a distinction between data informed and data driven?

It seems like data for nonprofits can be more complicated than data for a company. there’s no clear bottom line, especially if you’re trying to measure “impact.”
This makes it all the more critical to pick metrics that will say something about your success, while realizing that you can’t do everything.

Great post Beth! I agree that data can sometimes make the waters a little murky. @jdl, I think it’s less about an organization type when it comes to messy data and more about undefined or unclear expectations and goals.

Having worked mostly in the “corporate” world, I can say that it’s far too common for large decisions to be made on small, unsound bits of data…

Allison: Yes, definitely -messy is something we see a lot — and is usually a case of collecting “just in case data” because no one has focused on defining a clear result and the One Metric That Matters!

[...] and self-consciousness to ensure that we don’t enslave ourselves to it. As Kanter writes in a blog post, “From leadership, to strategy, to decision-making, to meetings, to job descriptions – a [...]

[...] "Data-informed cultures are not slaves to their data. Mario Morino uses the phrase “information-based introspection” to refer to using and applying data in context to excel.Data-informed cultures have the conscious use of assessment, revision, and learning built into the way they plan, manage, and operate. From leadership, to strategy, to decision-making, to meetings, to job descriptions—a data-informed culture has continuous improvement embedded in the way it functions. Key Performance Indicators (KPIs) are the specific quantifiable metrics that an organization agrees are necessary to achieve success. They are the mileposts that tell a data-informed organization whether they are making progress toward their goals. Too often organizations chose KPIs that simply reflect activity." [...]

[...] where data is used to inform decisions and shape programs. Beth Kanter summarizes it well in this post: “Data-informed cultures have the conscious use of assessment, revision and learning built into [...]

[...] results and make informed decisions is the hallmark of a highly successful nonprofit organization. Beth Kanter reminds us that “a data-informed culture is committed to continuous improvement”. But few of us [...]

Useful article to introduce to non-profits some of the key concepts that statisticians are aware of. Despite being someone who is embedded in data-driving decisions, I acknowledge that there is a lot of interplay between less measurable factors, such as self-esteem, culture and sociology that impact metrics such as KPIs. Culture can encourage the improvement of the the processes that the KPIs measure and vice versa.

What the above article also cites is the key problem of ‘overfitting’. One can analyse the metrics and tonnes of them, but the inclusion of metrics which are ‘too specific’ lead to high specificity and hence, low generalisability. Then the best you can hope for is that the model works that one *time*, but even that isn’t possible because of *time*

Factor analysis or the use of Taguchi matrices when designing lean experiments to analyse the effect of changing factors in the organisation will allow us to move towards an understanding of the high level impact of a much smaller set of variables. That keeps generalisability high and almost always avoids over-fitting. Plus, it’s easer to evolve over time. After all, Bonini’s Paradox basically states that if we model the situation 100%, the model is no easier to understand than the original problem. So it would be fruitless in that regard.

[...] One of the dangers we face by relying on large amounts of data for decision making is hidden biases. Social data has issues of reliability and validity at this aggregate level. The results do not generalize well. [...]