The X-Men are perceived as a threat, an emerging new species that - as the films keep telling us - could do what Cro-Magnon did to the Neanderthal. The general policy of hiding in the shadows to avoid discrimination is sensible, but is perceived by humans as making them even more unknown, furtive and sinister. They're perceived to be vigilantes with hidden agendas.

The FF were born normal humans and were public figures who have never hidden away. They're not perceived to be the next stage in human evolution, they are seen as victims of accidental radiation exposure.

Those people who have powers but weren't born with them (what Marvel calls 'mutates' rather than mutants and what DC calls 'metahumans') also come in for suspicion and criticism if they hide their identities, as with Spider-Man and Jameson's relentless campaign against him.

I think it's a lot to do with being secretive and therefore seeming to have mysterious motives. That can happen with gay people too; those who are out, loud, proud and obvious are often more happily accepted in society than those who 'hide' their sexuality.