For anyone who appreciates and uses social media free speech should be inviolable. That includes ensuring abuses of that freedom do not infringe on the ability of anyone to exercise it. To challenge, call out, parody or criticise someone is to practise freedom of speech.

To threaten them with rape is not.

There is no freedom to be recognised in the sustained attacks on Caroline Criado-Perez, on Twitter, following the success of her campaign to retain women on banknotes. Women in public life – whether politicians, actresses, writers, musicians or academics – face a barrage of sexualised, misogynistic messages. Complain and the cry goes up that we should go back to the kitchen if we can't take the heat of social media.

Twitter tell me we should simply block those who "offend us", as though a rape threat is matter of bad manners, not criminal behaviour. When you push them, as I did, they say report it to the police who can "accurately" assess a threat, implying victims may be overreacting. New legislation passed last year gave prosecutors powers to address cyber harassment, yet this casual attitude to the safety of the majority of their users – 53% of Twitter account holders are female – show how little attention companies themselves are paying to helping end violence against women.

This behaviour is nothing to do with technology and everything to do with inequality. If Twitter didn't exist such individuals would find another way to try to shut down the voices of those they despise. So the seriousness of this cannot be underestimated. Some tweets could be the escalation of aggression that can spill out offline. Others reflect a world in which men all too often can – and do – get to determine what matters about women. None are about sexual arousal.

We don't just need a strong response from those who profit from our custom as users of platforms such as Twitter or are there to enforce public order. Projects such as everydaysexism show how these platforms can help change this culture. Men react with surprise to the extent of passive-aggressive sexism exposed. Women express relief it's not just them who feel threatened.

That both have stood up for Caroline show this isn't a "women's" issue. It's a human rights issue.

This is about more than making Twitter somewhere fun for everyone to enjoy. If we want a world where everyone can live hassle-free, then everyday expressions of inequalities need to be confronted– not least as they help ease the big inequalities.

Casual denigration in the workplace makes pay gaps seem somehow more inevitable. Lewd aggression online can numb what should be a burning shock and revulsion at the physical violence faced by women offline. Wanting an end to this doesn't mean policing relationships between the sexes or shutting down Twitter. It's about seeking a world in which everyone can achieve their potential.

It is not for us to learn to cope with this abuse, but for those involved to change their behaviour. That requires not just asking users to stand up. Women deserve a better response that addresses these evolving ways in which technology is used to harass and distress. Its time for us all to act to take back Twitter.