In a small office in a high school in a formerly industrial area in the West Midlands, I am talking to a psychotherapist. Each week up to 20 pupils at this mixed comprehensive have an appointment in this brightly lit, windowless room. They come to talk about problems with parents or friends, but also more nameless anxieties that can't be traced to a single cause.

Sometimes they come a few times, sometimes for a year. Initial consultations can be confidential – a letter sent to parents lets them know about the service and asks them to write back if they don't want their children to use it – but if sessions are regular, parents are told.

The headteacher has asked the Guardian not to name the school, not because of concerns around stigma or confidentiality, but because the project is a new one. It is less than two years since he took the decision to upgrade the counselling service to a full-time psychotherapist, funding the shortfall from the pupil premium. As yet there are no outcomes to report.

But the head, whom I'll call Mr O'Brien, believes the experiment is working, and cites improved attendance and GCSE results (59% of pupils here get A-C in five GCSEs including English and maths) and a reduced rate of exclusions. "I think our kids are happier, more settled," he says. "It's a stable school."

This isn't solely down to psychotherapy, and many schools achieve improved results without it. But O'Brien believes recruiting a highly qualified professional as part of the school's pastoral team has had measurable results.

"I felt too much low-level need was being addressed at too high a level," he says, and gives the example of heads of year sorting out rows between friends. "Schools can be such busy places and when you are busy, your emotional intelligence falls off a cliff. If we want them to do well in their GCSEs we have to address the underlying problems."

These can range from crises around sexual identity to symptoms including hearing voices. "So many of us are well-meaning, but are we really equipped?" he says.

Cuts to Camhs (child and adolescent mental health services) dating back to the last government mean the process of referring such children to specialist health teams is fraught. Because of local variations there is no typical scenario, but teachers and providers agree that thresholds have become higher – meaning it can be difficult to get a child accepted by Camhs at all unless they have a diagnosed mental illness – and waiting lists can be several months long.

Devolution of education budgets from local authorities to individual schools under Labour, and increasing fragmentation since, have also resulted in headteachers having far more responsibility than they did 10 years ago for pastoral care. Siôn Humphreys, policy adviser at the National Association of Headteachers and a former deputy head, says heightened awareness may also be a factor. "To my way of thinking, the mental health issue is as important as obesity," he says, "but schools' ability to offer a network of statutory and voluntary agencies is under stress. The voluntary sector has been under the cosh in terms of funding cutbacks and there aren't as many clubs in the golf bag any more."

For a school to have an NHS-trained psychotherapist on its payroll is highly unusual. But research suggests the number of secondary schools offering some form of counselling has increased to around 60%. Data is patchy, but in a review published this year Prof Mick Cooper of the University of Strathclyde estimates that with up to 90,000 episodes of school-based counselling each year, the number of secondary-age children seen by school counsellors could be roughly equivalent to the number seen in Camhs.

In the past few years, the Welsh and Northern Irish governments have made school-based counselling a statutory service. When Mindfull, a new online support service for 11-17s, launched this summer it made counselling in all schools one of its goals. Meanwhile primary schools have also begun to use counsellors, with national charity Place2Be planning to expand from 200 to 250 schools next year and offering counselling for parents.

But while the idea of offering emotional support in schools may have gained currency in some areas, a national roll-out looks unlikely. When asked what ministers think about school counselling, a Department for Education spokesperson said "the decision is one for schools".

Official advice is not encouraging. Ofsted no longer inspects and assesses schools on care, guidance and support, and the future of the social and emotional aspects of learning (Seal) programme established by Labour is uncertain. In June, MPs voted against adding personal, social and health education (PSHE) to the national curriculum.

But fans of these services have evidence, in England as well as Wales that proves they work. The West Midlands school I visited has yet to produce its first evaluation, but 48% of primary-age children seen by a Place2Be counsellor in 2011-12 and 62% of those seen by Tavistock Outreach Psychotherapy Service (Tops) in 2012-13 showed improvements in their learning. Parents of 82% of Place2Be children with "severe behavioural difficulties" saw a general improvement, while 44% of referrals to Tops took less than a week.

Mandi Howells is deputy head at Carlton primary in Camden, one of five London schools currently signed up to Tops. She says one advantage of the service, based in school one day a week, is speedy access to hard-to-reach families.

"In some cases it can be very complex to get children seen in Camhs, say if a parent won't agree to take them." Then there are the waiting lists. "If a referral process takes three months, sometimes people change their minds," she says. Tops, which provides discussion groups for staff as well as psychotherapy for children, has become "part of our ethos, an integral part of what we do. People talk a lot more about emotional wellbeing than they did."

Place2Be employs 304 people and has placed 1,300 volunteer counsellors in schools over the past year. It also runs its own training and qualifications. Its chief executive, Benita Refson, would like to see awards for "mentally healthy schools, like you have for excellence in people and families", and quotes the headteacher who told her "you've made us into a talking school".

Not everyone is so enthusiastic. Psychotherapist Helen Randall, who works in several Birmingham schools as part of a Camhs project, sees children in a clinic because the sessions can stir things up. "I always tell families the therapist's job is to be the dustbin man who takes away the rubbish and if you can leave that mess in the health centre, that leaves school uncontaminated," she says.

Humphreys says "schools shouldn't have to provide these services, just as we don't provide medicine. If there's a co-ordinated approach within a cluster of schools, fine. If schools are ploughing their own furrow that can be a problem. There needs to be a co-ordinated policy approach and if the correct signals were to come from the top it would help a lot."

In the West Midlands, headteacher O'Brien agrees that in a perfect world services might be organised around a local hub away from school, but says in practice such arrangements are hard to manage. And there are advantages to having health professionals on site where they can get to know staff.

There are many of the problems of low aspiration and attainment here associated with traditionally white, working-class areas where unemployment is high, and 43% of children get free school meals. But the school is doing well. There is a "no shouting" policy, and a boy holds the door open when we pass.

But dig down and, as in any community of 1,000 people, there are problems. O'Brien says teachers use the term "ghosts" to describe the pupils in any school "who make themselves wallpaper – usually their attendance, their homework will be just good enough" to avoid attracting attention. Sometimes this masks serious difficulties, such as parental alcoholism. He is convinced social and emotional deprivation are part of the reason why children on free school meals, targeted by the pupil premium, do less well.

"The key is early intervention and building resilience," he adds. "I don't want to provide superb levels of support and see them in tears when they walk out the door. Sometimes we congratulate ourselves on how sad they are to leave, but the best chicks that leave the nest are those that don't look back."