Witches never look quite like you imagine. The latest "victim" of what has been called a "celebrity witch-hunt" is an elderly, baffled-looking man. He is Stuart Hall aged 83. It turns out he is not a victim at all. He is a perpetrator who has pleaded guilty to sexually assaulting 13 young girls, the youngest of whom was nine. Yes, nine years old.

I want to emphasise this because, ever since the Savile scandal broke, there has been a rumbling discussion about the culture of the 70s, "permissiveness" and a haze of confusion about what constitutes consent. Sexual abuse and rape by men who were supposedly trusted and revered by the public has resulted in massive displacement, a smog of guilt – why didn't we know, or did we know and ignore it? – meaning all kinds of justifications have been made. Some have tried to turn this into a story about the evils of the BBC or of the Leveson inquiry, because in some ways that is easier to deal with than the banality of "light entertainment" being the home of sordid abuse.

At first, remember Savile was regarded as an eccentric, merely doing what "the talent" at the time did. When the full extent of his wickedness was revealed, we put him in a box marked "monster". As he was conveniently dead, that box was then moved, for fear of desecration. Savile, we saw, was a vile predator, and yet given access to already damaged and institutionalised girls.

The whole "groupie" excuse did not wash. Instead we learned of the seedy reality of the tracksuit bottoms whipped off, and serious sexual assault on a 10-year-old boy. We shuddered, looked away and assigned it to the past.

Well, some of us did. From the moment I started writing about Savile in early October, I began to receive letters and emails from women detailing their own abuse, often by a family "friend" and often, horribly enough, with their parents knowing. Thankfully the writer Yasmin Alibhai-Brown received one such letter from a woman Hall had assaulted. Yasmin is such a concerned and persistent person that she took it to the police. That started the inquiries rolling, for which we must thank her.

The common themes of many of the missives I received were that women who have been abused feel guilty, responsible and so ashamed that they have never spoken about it. The Savile case was churning up terrible memories. This experience was not completely new to me.

Once, when I was teaching an MA course, an extremely competent and mature student came to see me after a film I had shown triggered all sorts of memories for her about abuse. Apart from referring her to organisations that deal with "survivors", I found myself shocked to see someone so knocked sideways by things that had happened so many years earlier.

This feeling is now there again, but this time collectively. We still do not know how to deal with abuse. We are reeling from these revelations in denial, disbelief and distress.

For instance, there are those who criticise Operation Yewtree (set up post-Savile) on the grounds that we cannot apply today's attitudes to the sex crimes of yesteryear. There also exists the feeling that perhaps the police are themselves indulging in some retrospective guilt trip: as they did not act during the Savile era, despite complaints being made very early on in his career, they have now gone into overdrive, hanging out innocent men to dry. The list grows: Max Clifford, Rolf Harris, Dave Lee Travis, Jim Davidson, Bill Roache and Freddie Starr have all had serious allegations made against them.

The piecing together of historical evidence, as there is no physical or forensic evidence, is what will make or break these cases. Once again a different, murky time is demarcated for these alleged crimes: the blur of the late 60s and 70s, which we are told was both a more "innocent" time and also a time of great promiscuity. Promiscuity for whom remains the crucial question.

As each of these "family" entertainers is arrested, I hear people – often men – saying, "Is all my childhood going to be destroyed?" Which is unfortunate, because yes, that's exactly what abuse does. I don't share this nostalgia for these "wholesome" presenters – they were always a bit creepy – and the new details of their extraordinary behaviour bear this out. The immunity that protects celebrities was clearly in play 40 years ago. Was it really OK that when Hall turned up at a school in Cheshire in 1967 to hand out school prizes, he insisted on kissing the young schoolgirls? Was it OK that, as Linda McDougall recalls, he occupied the medical room at the Look North studios for his dalliances with "lady friends" or that "he was one of those people who had his hands all over you and all over any female that came in". Was it OK when, eight years ago, he told Radio 5 listeners, "Your average 10-year-old can instruct you in oral and anal sex"? These remarks were dismissed as satirical banter.

Anyone my age knows how some men did behave in the workplace, and you got as far away from them as fast as possible. But there were always girls who didn't or couldn't. For the 70s was pre-Aids, a time when we were expected to be free and easy. What did not exist then was the idea that one could be sexually harassed, and could complain and be taken seriously.

This is why today, when I hear the voices of Hall's victims, I would argue that naming the suspects (which enabled some women to come forward) remains powerful. Some of these women have been silent for a very long time. When Susan Harrison, whom Hall plied with alcohol then assaulted, told her father after returning home in tears, he said: "He is famous and we are nobodies. Nobody is going to believe you if you tell." Unless victims waive their right to anonymity, the victims do indeed remain nobodies. Harrison felt guilty, worrying that Hall may go on to abuse others, which he did. She later suffered from depression.

Kim Wright, another of Hall's victims, was watching the news about Savile when she decided to make a formal complaint. She is a policewoman who has worked in child protection. "I've always felt what Stuart Hall did to me is not something he'd do once, that it was part of some kind of modus operandi." Wright felt that if the police could tell other victims they were not the only one, they would have the courage to come forward. And she was right. "When he picked on me," she said, "he picked on the wrong person."

Indeed. Though surely sex without consent is always picking on the wrong person. We can now choose to recognise that or not. We worry about our young, about "sexualisation", about access to porn, and yet we are still – some of us – hazy about consent. Girls are still deemed to be "asking for it" if they are out late or wearing the wrong clothes.

It is much easier to assign all these horrible stories to a different time. But is it really so much better now? Are young women believed when they speak up today?

These men, who now look old and pathetic, clearly at the time felt immune and able to damage who they liked. Those who criticise Operation Yewtree should listen to the calls coming into Childline for a few nights. Abuse didn't stop with the arrest of a few sad DJs. I see Peter McKay in the Daily Mail quoting Thomas Macaulay: "We know of no spectacle so ridiculous as the British public in one of its periodical fits of morality."

Maybe so, but what culture thinks it's all right to sexually assault a nine-year-old? None that I want any part of. Now let the victims' voices come to the fore. They were pushing off Hall and Savile. They were saying no. They were crying. They were gagging. They were bleeding. They were hurting. Child abuse and sexual assault has not suddenly stopped, but if we cannot speak about the past, we cannot speak about the present. Now we need to listen.