Last week’s Heritage Foundation panel on the 2012 attacks in Benghazi was bound to be an ugly affair, what with the presence of panelist Brigitte Gabriel, a self-described “terrorism analyst” with a laundry list of offensive statements about Islam and Arabs.

Sure enough, when attendee Saba Ahmed, an American University law school student, explained that not all Muslims are terrorists, Gabriel retorted that “the peaceful majority were irrelevant” in the Sept. 11, 2001 attacks in the way that peaceful Germans were irrelevant during the Holocaust.

That prompted much hand-wringing, primarily on cable news, about the supposed silence of “moderate Muslims” in this supposed age of Islamist extremism. What no one on either side of the debate questioned, though, was the legitimacy of the phrase “moderate Muslims” itself.

In the years that I’ve spent writing about and studying the phenomenon of Islamophobia, that phrase has always troubled me.

Muslims and non-Muslims alike bandy it about, though the latter usually demand that the former prove that they are such. What bothers me is not that there aren’t “moderate Muslims”—from my perspective, there certainly are. The unacknowledged problem is how that phrase informs our judgments.

Brigitte Gabriel, for instance, told the Australian Jewish News in 2008, “Every practicing Muslim is a radical Muslim,” meaning “moderates” must be only those who don’t practice their religion. (About time to redefine “practicing religious person”)

Celebrity atheist Sam Harris writes that “moderate Muslims” are those who express skepticism over the divine origins of the Quran and “surely realize that all [sacred] books are now candidates for flushing down the toilet.”

Then there’s conservative columnist John Hawkins, who enumerates 7 criteria that Muslims must meet in order to be considered “moderate” while the queen of Muslim-bashing, Pamela Geller, asks in typical fashion, “What’s the difference? Today’s moderate is tomorrow’s mass murderer.”

To be fair, it’s not just the wackos. Newsweek, NPR, the Wall Street Journal,Reuters, TIME, The New Republic and many others have used this phrase to describe Muslims who fit a certain preferred profile. Many Muslims themselves have bought into this dichotomy, if only to distance themselves from the so-called radicals and extremists—to assure paranoid non-Muslims, in other words, “I’m not that kind of Muslim.”

How is it that we talk about Muslims much like we talk about Buffalo wings, their “potency” being measured not by some objective rubric but rather by our personal preferences?

It’s the mild ones that we seem to search out: not so spicy in their religious practices that they burn us, yet not so bland that they dilute our religious diversity altogether.

The idea of a “moderate Islam” or “moderate Muslim” is intellectually lazy because it carves the world up into two camps:

the “good” Muslims and the “bad” Muslims, as Columbia University professor Mahmood Mamdani has noted. (Saba Ahmed herself used the word “bad” in her remarks at the Heritage panel.) Until proven good, or in this case “moderate,” all Muslims are perceived as “bad,” or potentially extreme.

We certainly don’t spend our time searching out “moderate” Christians or Jews, but rather reckon that the Westboro Baptists, Jewish Defense League, and others are aberrations. And sure, Muslims give us plenty of bad examples, but it’s our own fault if we allow those examples to constipate our ability to perform basic logic.

During the panel, Gabriel argued that “15-25 percent” (how they get these stats? and why this huge margin?) of the world’s Muslims are extremists, and that the remaining “moderates” are “irrelevant” (I shouldn’t have to explain that it’s usually the majority of a given group that makes the minority irrelevant, not vice versa).

Based on the lower end of that range, that’s 240 million of the world’s 1.6 billion Muslims — the equivalent of every single Muslim in Sub-Saharan Africa, or nearly 6 times the number of all Muslims on the entire continent of Europe. Where are the examples of such supposedly widespread extremism? Even if a mere 1% of the world’s 1.6 billion Muslims is committed to violence, why is it that we haven’t seen 16 million violent attacks?

Proving one’s “moderation” is a trap, anyway. (And why you are required to prove moderation? Are the Inquisitions in progress and kicking?)

Is the only way to do it to meet the criteria set forth by the person making the demand?

For Gabriel and others, it’s by supporting Western foreign policies in the Middle East, cheering continued military aid to Israel, and even rejecting certain Islamic tenets.

It’s why a figure like Zuhdi Jasser, a darling of the Republican Party and Peter King’s star witness in the “radicalization” hearings, is held up like a trophy while Saba Ahmed is mocked.

That’s the problem with this “moderate Muslim” nonsense: it empowers anti-Muslim activists by implying that the degree to which a Muslim digests their religious faith is indicative of their status as a potential terrorist.

Thus, “moderately” subscribing to the teachings of the Quran is OK, but should they cross over into the world of daily prayers, Friday afternoons at the mosque, and, God forbid, Ramadan, they’re suddenly flirting with extremism.

That way of thinking is predicated on the unfounded notion that pious religious orthodoxy necessarily entails Muslims behaving badly. It also implies that religious “moderation” involves swallowing up one particular political narrative.

Lastly, calling on “moderate Muslims” to condemn violence or other loathsome acts presumes that anyone who doesn’t is a terrorist lying in wait (should every Moslem claim the 5th and not incriminate himself?).

It gives credence to the idea that only those who are at the beck and call of Islam’s credential police are the peaceful ones, and that the ones posing the question—“where are the moderate Muslims”—are sufficient arbiters of what Islam really is and isn’t.

In order to arrive at a more peaceful and equitable place in our society, we must divorce ourselves from the notion that we are authorities on the faith traditions of others and as such are entitled to prescribe how they must interpret them in order to be welcomed.

The diversity within religious traditions is just as important to the pluralistic fabric of America as the diversity of religious traditions. Carving up our Muslim compatriots into categories that fit our idea of what they should be isn’t going to get us there.