Surveys should ask not for ranges when collecting numbers, but actual numbers

Poring over a compliance function survey now underway by PricewaterhouseCoopers and Compliance Week, I noted that they ask for the number of employees in the function and gave nine ranges to choose: 1, 2, 3-5, 6-10, up to “more than 400.”

While ranges may make it easier for respondents to give approximate answers, range data cripples calculations of ratios. At best, if you use a mid-point of a range, you can pretend to create a percentage, such as employees per billion of company revenue. At best, I say, because that makes for very sloppy results.

Other benchmark surveys collect data in ranges and report results in those terms: “One third of the respondents had 3-5 employees.” What can a general counsel do with such a factoid except say, “Well, I am in a common or not-so-common demographic group.”

A third problem with ranges, shared by all multiple-choice questions, stems from the risk that a less useful mix of ranges is picked. For example, if the first range is “1-3 employees” you might have more than half your respondents in it. In the PwC survey, the employee ranges broaden as they increase, as if the designers of the survey would like approximately equal numbers of respondents in each bucket.

If researchers want a modicum of precision and analytical potential, ask for absolute numbers (See my post of March 26, 2007: ranges criticized in survey of time to fill in-house positions.).

One response to “Surveys should ask not for ranges when collecting numbers, but actual numbers”

This point is well taken, though at least in this case people are, if somewhat crudely, attempting to move towards actual numbers. Where I find survey data particularly poor and unhelpful is the omnipresent use of the 5 point rating scale to make subjective evaluations of performance (e.g., “Rate this law firm on level of competence from a 1-5, with 5 being the highest level of competence”). Within legal departments and law firms, it is also extremely popular to internally rate personnel in this manner. This seems to be an easy way to offer the appearance of metrics based evaluations without actually using metrics that are predictive of high quality performance.