A House of Lords committee concluded that it would take more than nudges to tackle the UK's obesity epidemic, which could cripple the NHS. Photograph: Alamy

If I had a pound for every time I've heard a politician or government official urging me to "do the right thing" I would be a rich man. Do you save enough for your pension, drink within the official guidelines, volunteer in your community, try to cut your carbon footprint, maintain your body mass index below 25? If we all did the right thing in these, and many other ways, life would be a lot easier for policy makers. But unfortunately a lot of us don't.

Governments have traditionally used a combination of three kinds of tool to get us to do the right thing: legislation, financial carrots or sticks, and education. We are encouraged not to drink and drive by laws against it and penalties for getting caught. We are discouraged from driving into central London by the congestion charge and encouraged to insulate our homes with subsidies. We are told about the dangers of smoking, unprotected sex and eating an unhealthy diet in various government advertising campaigns.

There are big success stories in government attempts to change the behaviour of the population as a whole. Sixty years ago nearly three quarters of us were smokers, today it is less than a quarter. When seat belts were introduced we were reluctant to wear them, but now we do so without thinking. It used to be normal to drink and drive, but the roadside breathalyser and penalties have changed our attitudes and behaviour.

These changes in our behaviour resulted from a mixture of legislation, education and in the case of tobacco, taxation, although the precise contribution of each is not known.

But in recent years a new fashion has emerged, drawing on research in behavioural sciences, especially in psychology, behavioural economics and behavioural biology. The central idea is that many of our moment-to-moments decisions are made on autopilot, without much conscious reflection. By tapping into our semi-automatic decision-making, we can be subtly persuaded to change our behaviour and do the right thing.

The book Nudge, by Chicago academics Richard Thaler and Cass Sunstein has caught the eye of politicians, as has Robert Cialdini's Influence, in which he analyses how were are persuaded to buy stuff by subtle psychological manipulation.

Pensions are a classic example. If the default for a pension scheme is "opt in", far more people pay into the scheme than if the default is "opt out". Apparently we tend to accept the default without really bothering to think too much about an active choice.

There are now numerous documented examples of where "nudging" can contribute to getting people to do the right thing. It is attractive to politicians because it is a light touch approach that avoids the heavy hand of regulation and the nanny state. The Behavioural Insights Team (or "nudge unit") in the Cabinet Office is encouraging government departments to use nudging to achieve policy objectives such as getting people to pay their road tax or reduce their carbon footprint. The team is not taking it for granted that nudging works: they are setting up randomised control trials to find out.

Here are a couple of examples of where nudging has an effect. If householders' energy bills indicate their energy consumption relative to others in the street, together with tips for reducing energy use, people reduce their consumption by a few percent. In another study at Newcastle University people were about twice as likely to clear up their rubbish in a cafeteria if the sign encouraging them to do so had a pair of eyes on it than if it had a bunch of flowers.

Although nudging is a relatively new idea, the notion of using insights from behavioural science to get people to do the right thing is not. More than 50 years ago the American psychologist B F Skinner wrote a Utopian novel, Walden Two, about a community in which everyone did the right thing as a result of "behavioural engineering". Many commentators were horrified at the implications of Skinner's vision for freedom of choice. Similar ethical questions could be asked about any measures that get us to change our behaviour by working through our subconscious.

But how useful is nudging for tackling society's hardest problems? It is one thing for marketeers to persuade us to buy the latest gadget, this season's fashions, or a new beauty product. Here subtle psychological ploys are working alongside our own wish for immediate gratification. It is quite another to persuade us to do things that have a long-term benefits to us, such as losing weight, or a benefit to future generations, such as reducing our environmental footprint.

When the House of Lords Science and Technology Select Committee looked into this last year, it concluded that soft approaches such as nudging are not, on their own, enough. The Committee looked at two challenges in particular: obesity and transport.

Some experts estimate that the adverse health consequences of obesity could cripple the NHS if nothing is done to get people to lose weight. There is no magic bullet for fixing the obesity epidemic, now affecting about a quarter of the UK population, but the select committee's report concluded that the huge change in behaviour needed will not be achieved by nudging alone.

Likewise, persuading us to leave our cars in the garage and take public transport, walk or get on our bikes requires a massive shove rather than a nudge.

Although governments are reluctant to be more interventionist, a recent Mori survey of 18,600 people in 24 countries showed surprisingly high levels of support for regulations and other strong measures to tackle the major problems facing society.

The behavioural sciences have an important role to play, but our understanding of how to persuade people to change their behaviour is still rudimentary. In the mean time, tough problems will need tough action from government.