How many atheists are there in America? The answer to that question is a lot tougher to nail down than you may think, writes PRRI’s Dan Cox in his latest article in FiveThirtyEight. Here’s why:

The American religious tapestry is continually being re-stitched as new religious groups slowly gain acceptance. Atheists have been on the fringe for quite some time. They remain one of the country’s most disliked “religious” groups: Only 30 percent of Americans have a “warm” view of atheists.Research has also shown that even as America has grown increasingly accepting of religious diversity, atheists have been the exception.

Americans express a considerable degree of intolerance toward atheists. More than half of Americans believe atheists should not be allowed to put up public displays that celebrate their beliefs (for example, a banner highlighting Americans’ freedom from religion under the Bill of Rights). More than one-third believe atheists should be banned from becoming president, and similar numbers believe they should be denied the opportunity to teach in public schools or the right to hold a rally.

And therein lies the problem: The stigma attached to the atheist label may prevent Americans from claiming it or sharing their beliefs with others. In certain parts of the country, pressure to conform to prevailing religious practices and beliefs is strong. A reporter with The Telegraph writing from rural Virginia, for example, found that for many atheists, being closeted makes a lot of sense. “The stakes are high,” said a Virginia Tech graduate who was raised Christian but is now an atheist. “Do I want to be supported by my friends and family, or am I going to risk being kicked out of clubs and organizations? It’s tempting just to avoid the whole issue.”

Cox also points out that polling structure creates another problem, as many survey questionnaires only let respondents check off one answer for religious affiliation.