Throughout last autumn, Broughton Hall high school was riding high. The 2011 GCSE results at this big Catholic girls' school in a rundown area of Liverpool were better than any other comprehensive in the city – with 97% achieving five good grades. The school's 1,300 pupils even outperformed those at some independent schools, and the progress they'd made won their school a national award.

The local MP – the shadow education secretary, Stephen Twigg – visited in November and December, and peppered his blog afterwards with words like "fantastic" and "inspirational". There was a brand new state-of-the-art building, and an innovative curriculum designed by the Royal Society of Arts.

And then, in January, Ofsted came. The headteacher, Gerard Murphy, jokes that he'd been thinking of inviting the inspectors in sooner, given how well the school was doing. But within hours of the inspection team's arrival, it was clear this would be a very different experience from Ofsted's last visit. In 2007, the inspectors declared Broughton Hall a good school and its sixth form "outstanding".

To be fair, all schools had been warned that under a new inspection framework – introduced at the start of the spring term – they could expect tougher judgments. Ofsted used to inspect a whole range of areas, such as whether a school's provision met local employment needs or contributed to community cohesion, but it now looks at just four: achievement, teaching, leadership and behaviour. Broughton Hall was one of the first to be inspected under the new regime.

Halfway through the first morning of the two-day inspection, Murphy says, the lead inspector broke some shocking news: she had seen some lessons that she deemed inadequate – although she wouldn't say which – and was considering whether to put the school into "special measures".

"This is my 37th year in teaching, and 25th year in senior management as a deputy and as a head – and I've never experienced anything like that before," Murphy says. "I've been deemed an outstanding head in previous Ofsted reports. I felt as though my feet had been swept away from under me."

By the end of the second day the threat had receded, he says, and the school was given a "satisfactory" rating. Under yet another framework, to be introduced in September, it will still mean the school must improve or face sanctions.

Murphy – backed by his local authority, the diocesan authorities and Twigg, who has spoken to the chief inspector, Sir Michael Wilshaw, on Broughton Hall's behalf – is fighting back. He's one of a growing number of headteachers who say they've been treated unfairly under the new rules.

New figures released by Ofsted to BBC Radio 4's File on 4 programme reveal that in the first five months of the new framework, 262 schools – one in 12 of those inspected – made a formal complaint afterwards. Ofsted did not release comparable data for last year, but the figure certainly represents a rise. The Association of School and College Leaders says its monitoring suggests the number of complaints has doubled. At the same time, the number of schools judged inadequate rose from 6% during the last full academic year to 9% in the spring term of 2012.

Meanwhile, the inspectorate says its own post-inspection monitoring suggests nine out of 10 schools are happy with the process.

The rise in complaints may also be linked to the fact that Ofsted has a new, tough-talking chief inspector, who is not afraid to upset schools. Wilshaw, who took up his role in January after leaving the headship of Mossbourne academy in Hackney, London, believes 5,000 headteachers – around one in five – lack leadership skills.

At Broughton Hall, it wasn't just the inspectors' negative judgment that caused consternation. Murphy says the draft report on his school contained numerous errors. The inspectors wrongly claimed the school had failed to meet key targets, he says, and even got its name wrong, describing it as Broughton Hall Technology College.

Those mistakes have now been corrected after a formal complaint, but the judgment on the school has not. And as controversy grows around Ofsted's new inspection regime, there is also increasing concern about the quality of the inspectors themselves.

At Burnholme Community College in York, which was inspected last month, there is particular outrage that an inspector who was described at the inspection as an English specialist appeared to be nothing of the sort. "When you have an inspection, you're given a mini-CV of the inspectors," explains the school's headteacher, Simon Gumn. "Ours said this inspector had experience of inspecting schools, but more in behaviour and safeguarding – nothing about English. When the English department were observed, that particular inspector didn't seem to understand the English curriculum."

Gumn, like Murphy, has made a formal complaint, and as a result Ofsted has made some changes in advance of the publication of the school's report on Wednesday. But the report is expected to contain a formal notice to improve, and Gumn is still unhappy. "I will be held to account by the outcome of the inspection, but who holds Ofsted to account?" he asks.

It is thought the inspector who visited Burnholme's English department was a former lay inspector – an obsolete category introduced in the 1990s to allow non-professionals to work with inspectors, looking at non-core areas of provision. Now, with just four areas under scrutiny, it seems these inspectors are being used to judge core subjects.

Other inspectors working for the three private companies that now carry out school inspections on behalf of Ofsted – Serco, CfBT and Tribal – are former lay inspectors who have never been teachers. And some former headteachers who were forced out because their schools were judged failing are also working as inspectors. Both Ofsted and the inspection companies have refused to comment on individual cases, but all say inspectors are rigorously trained. Former lay inspectors have many years' inspection experience, they add.

While some schools are pursuing their cases through Ofsted's complaints system, others have decided to take more radical action. Furness academy in Cumbria has applied to take Ofsted to judicial review after inspectors judged the entire school inadequate in January because its pupils had made insufficient progress in maths over five years. The school only opened in 2009, and last year's GCSE candidates spent their first three years of secondary education in three other local schools, two of which were in special measures.

The school's principal, Douglas Blackledge, says he could not allow the judgment to stand. "I embrace entirely the need for a robust inspection and monitoring framework. But we were worried a report which said that the academy was inadequate would be used to drain the confidence of our students and parents. We felt that was not legitimate at all, so we felt that we had to fight it," he says.

Wilshaw refuses to comment on individual cases, but does say inspectors who have never taught or who have failed as school leaders should be rooted out. "If that is happening, we need to address it. When an inspector's in a classroom judging teaching, I would expect them to know what good teaching looks like," he says.

He adds that he wants to encourage more outstanding heads to become part-time inspectors, but on the subject of the new framework he is generally unrepentant. "We're focusing on the key issues," he says. "And if the teaching isn't good enough, and leadership isn't good enough, and the culture of the school isn't much good, and the children aren't making the necessary progress, we're going to say it's not good. Until headteachers and governing boards realise what's really important, possibly we're going to have these sorts of figures."

At Broughton Hall, Murphy predicts the level of unrest over Ofsted will continue to rise. The talk among other heads, he says, has been defiant – some are even threatening to throw out inspectors who seem incompetent or overly harsh. "I think we're going to get to that stage," he says. "You can beat a person so far to the ground, but there comes a point where they may get back up and fight back."

• File on 4's investigation into Ofsted will be broadcast on Tuesday at 8pm on BBC Radio 4, and will be repeated on Sunday at 5pm