The fact that Britain has the highest rate of teenage pregnancy in western Europe – and the second-highest in the world after the US – is nothing new. What's far more interesting is why this is still so, after concerted policy intervention since 1997. While the teenage pregnancy rate declined between 1998 and 2007, it now looks increasingly likely that the government will leave power with the dubious record of having presided over a marginal increase in 2007, the latest year for which data is available.

Why the government's strategy appears to have failed in spite of public investment – from advertising campaigns raising awareness and increasing access to contraception, to sex education in schools – remains something of a mystery even to experts in the field.

Today's results, reported in the British Medical Journal, of an independent evaluation of a government scheme to tackle teen pregnancy shed little further light on how public funds can best be spent. Results published show that young women attending the programme were "significantly" more likely to fall pregnant than those in a comparison group: 16% of the young people development programme group fell pregnant compared with 6% in the other group, which was a youth programme not receiving YPDP funds. They also reported early heterosexual experience (58% v 33%) and 34% compared to 24% in the control group expected teen parenthood in spite of the fact that they received sex and drugs education, were no more sexually active than the control group and some sites distributed free condoms. Young women in the YPDP group were also more likely to have not used contraception when they most recently had sex.

Douglas Kirby, a senior research scientist based in the US, also writing in the British Medical Journal, said the results of the English study showed that, at best, the programme had no impact, and at worst had a negative impact – and at an estimated cost of £2,500 per individual teenager participating in the programme over three years. In many ways this report makes depressing reading, not least because there will be many who will use the results as an argument to turn the tide away from more effective sex education in schools and community youth services and who will argue that the liberal approach has failed.

Yet this would be a mistake. As Madeleine Bunting has already cogently argued, where there has been a concerted local push to invest in services for teenagers that are accessible and appropriate, and where sexual education has been well-delivered in schools, some of the most unexpected, most deprived areas have shown dramatic improvements. Hackney, one of London's most deprived boroughs, saw a 25% drop in its teen pregnancy rate; Blackburn, also with high levels of deprivation, saw a comparable improvement. Their success shows that it is possible to bring down teen pregnancy rates even in areas of high deprivation, and that well-designed services can break the link between poverty and teen pregnancy.

The results of this latest evaluation suggests that the reasons for failure with initiatives aimed at reducing the rate of teen pregnancy, are complex. The success of the New York initiative on which the UK scheme was modelled, for example, was not easily replicated in other states. There is, it seems, no easy win-win solution. Kirby, in his editorial for the British Medical Journal, comments on the UK evaluation:

This does not mean that all youth development approaches are ineffective. Programmes may be more effective when implemented by charismatic staff, when they facilitate access to reproductive health services, when the staff connect with the teenage participants or when the staff give a strong clear message about avoiding unprotected sex. Programmes may be less effective when one or more of these conditions are not met.

The authors of the evaluation of the government scheme, themselves at a loss for a definite explanation, conclude that similar future programmes might be better targeted at young people in areas of social disadvantage, and that ultimately it is the wider social, economic and educational influences on young people's health that need to be addressed, rather than the behavioural profile itself.

What is clear is that young teens need pathways out of their current lifestyles and habits, rather than programmes simply focused on changing behaviour. If the foundations of their life are shaky or without aspiration, there will be a tendency to continue with escapist patterns of behaviour that produce short-term highs (unprotected sex, drugs or alcohol) to cope with daily lows. There is also the fact that, as US initiatives suggest, programmes need to explicitly target young potential fathers, as well as potential teen mums, to preach a shared message of responsibility. Experience there suggests this is best done in same-sex groups over time, often with staff who can act as wiser role models, and are rooted in the communities they are seeking to help. Interestingly, in the US such initiatives originally emerged more than a decade ago as add-on policy initiatives to welfare-to-work programmes for welfare mums and unemployed fathers.

As the recession continues to bite, and as educational and employment prospects for our nation's youth appear to diminish, it is not hard to imagine a future in which the teen pregnancy rate exponentially increases. What we need now are practical and holistic policy interventions that start with education, training and welfare-to-work initiatives, as well as sex and drugs education in our schools and communities from inspirational community leaders, teachers and practitioners.