David Cameron has promised to protect the NHS budget, and increase its funding every year, but on the ground managers are getting used to having less money to work with. Photograph: Alastair Grant/AFP/Getty Images

In the little village of Mortimer, near Reading, Berkshire, Bill Walters sits most of the day in the conservatory he built before his hip gave out. He can get up the stairs with crutches, but the pain, he says, half-way up and breathing heavily with the effort, is "excruciating". Doctors agree he needs a hip replacement. On 9 December he was told the waiting time was 18 weeks. He now has a "pencilled-in" date in July, when he will have been waiting for 31 weeks.

A little over a year ago, David Cameron stood at a lectern in the first televised leaders debate of a British general election and declared his passionate devotion to the NHS following the illness and death of his son, Ivan. "What it did for my family and my son I will never forget," he said. His party would make an exception of the NHS, he promised, sparing the axe and increasing spending in real terms so that "we improve it, we expand it, we develop it".

Bill Walters could be forgiven for wondering how Cameron's pledge squares with his present pain. Waiting lists like the one he is on are growing. The Patients Association, hearing hundreds of stories from people complaining of delayed and cancelled operations, put in freedom of information requests to every acute NHS Trust in England in January, asking about nine common surgical procedures including hip and knee replacements. Among the 62 Trusts that replied, it concluded, 10,757 patients had not received an operation because they were unfortunate enough to need it in 2010 instead of 2009.

Cameron and the health secretary Andrew Lansley keep repeating their pledges that the NHS's budget is being protected and will increase in real terms every year of this parliament. But on the ground the reality is very different. An array of financial pressures is forcing primary care trusts (PCTs) and hospitals across England to get used to having less money to spend just at a time when their costs are rising. NHS bosses are gloomy. Increasingly their NHS organisations are having to ration access to healthcare just when demand for it is growing. More and more of them are shrinking their workforces – by as many as 1,600 staff over the next four years, in the case of the Heart of England NHS Trust, which runs hospitals in Birmingham and Solihull.

No wonder health policy experts warn that even when the health and social care bill has finally ceased to be controversial for the coalition, the NHS's difficult finances could prove to be a rolling bad news story ministers had not anticipated.

But how can a solemn undertaking to give the NHS more money translate into delays and cuts? The answer lies, in part, in the growing numbers of us who, like Bill, have every expectation of living long into retirement and every hope that the NHS will sort out our infirmities. Average life expectancy in the UK rose from 71 in 1960 to over 80 in 2009. Demographic change is adding £1bn a year to NHS costs, the influential health thinktank the King's Fund says.

But the NHS also has to find an extra £1bn this year to cover pay increments, guaranteed cash fillips which many of its 1.4m staff get in any year. January's VAT increase will cost another £250m. Some £200m has to be found to fund the government's Cancer Drugs Fund. Transitional costs of the NHS preparing for the big shakeup anticipated in the stalled health bill will be around £500m, according to the regulatory impact assessment for the legislation itself. Then there is the unknown cost of the NHS being expected to provide brilliant but increasingly expensive new technologies, from fertility treatment to novel cancer drugs.

Plus, crucially, PCTs are being forced by the NHS to get their finances in order so they are not carrying debts which their successors, the new clinical consortiums due to start commissioning care in 2013, would refuse to assume.

NHS leaders have to contend with all that but then also confront the biggest challenge of all: the need to make £15bn-£20bn in efficiency savings by April 2015, to try to plug the widening gap between the demands being placed on the service and its ability to meet them. This is the so-called Nicholson challenge, named after the service's chief executive, Sir David Nicholson.

The NHS was launched in 1948 with a budget of £437m, which would equate to £9bn now. In 1997, when Labour came to power promising to restore an underfunded health service and make it as successful as the best in Europe, the budget was £35bn. By the time they left office, it was around £110bn – and yet productivity, in terms of the amount of healthcare we get for the money (not an easy thing to measure) has gone down. The Office for National Statistics estimates that between 1995 and 2008, it fell by 3.3% – an average of 0.3% a year. Some people say the money was soaked up by increases in doctors' pay (allegedly up to 40% went on salary increases). Lansley says the NHS under Labour was inefficient – which bolsters his argument that it can make savings – but Labour and sympathisers say performance is incredibly hard to measure. You can count the number of hip operations, or ambulance trips or treatments for depression, but since some patients end up coming back, does that translate into fitter, healthier, happier people?

The recession put an end to the big spending days of the NHS. It quickly became clear that economies would have to be made if rising healthcare costs were not to spiral out of control. In 2009, Labour called in the management consultants. McKinsey were given a contract to figure out how and by how much the sails of the NHS could be trimmed.

Labour didn't publish the McKinsey report – but the coalition did, once in power. Headlines in the Health Service Journal, which got a leak in September 2009, explain why. "Department of Health is told 137,000 NHS posts must go in next five years," it said. They had recommended slashing 10% of the workforce.

The McKinsey report's bottom line was that if all hospitals and GP practices were as efficient and effective as the best, the NHS could save £13 to £20bn a year over the next three to five years – or up to 22% of its budget a year.

The report was delivered in March 2009. In June, behind closed doors, Nicholson delivered his challenge – calling on NHS finance directors to save between £15 and £20bn over four years in which they would receive just a 0.1% real terms annual increase in funding.

The election brought promises to save and stand up for the NHS and protestations of love for it, but nobody was going to pledge a load more money. Cameron's promise to give the NHS a real-terms increase in funding year on year was as good as it got. But inflation estimates have gone up and – allowing for the rising costs of healthcare – this means a real terms cut of 0.9% over four years, according to John Appleby, the King's Fund's chief economist.

Appleby, with the Institute of Fiscal Studies, did his own report around the time of the McKinsey report, based on the figures in Derek Wanless's review of NHS spending in 2002. Appleby, too estimated the amount of savings needed at around £20bn (he has since revised the figure to £28bn if costs keep rising and productivity does not increase). But he did not fully agree with McKinsey that it was possible.

Alan Maynard, professor of health economics at York University, was chair of York NHS Trust for 12 years until last year. In reality, he says, "you squeeze everything you can, but the only way you can reduce expenditure is to cut labour costs, which means closing wards, so waiting times go up. You have to do emergency stuff or they die in your car park."

Nonetheless, the Nicholson challenge has concentrated minds and in the last two years, hospitals and primary care trusts and GP practices have been working on ways of saving money. But if they are becoming more efficient, it is not immediately apparent. The King's Fund found in April that waiting times for hospital treatment had reached their highest level for nine years. Bill was among nearly 15% of hospital patients who had waited more than 18 weeks for treatment. Most of the finance directors it surveyed were already warning they were unlikely to meet their productivity targets in 2011-12. Nearly half said ward closures and cuts in services were one of the main ways they were attempting to meet their target.

Warwickshire PCT is one of those trying to find smarter ways of doing more for less. They found they were spending too much money on elective surgery, such as hip and knee replacements, because GPs were sending everybody who might need an operation to hospital – and hospitals, used to payment by results, were processing them as fast as they could. "The PCT was spending more money than it had," said chief executive Stephen Jones.

They came up with their Fast, Slow, Stop programme. The PCT got GPs and hospital consultants together and asked them to discuss – as they had not before – which sort of patients actually needed surgery straight away. These were allocated to the Fast stream.

Those which could reasonably wait until the next financial year were placed in the Slow stream. The latter included some joint operations where the patient was still quite mobile but also a list of other procedures, such as cataract removal, weight loss surgery and laser treatment for skin conditions.

On the Stop list were procedures they felt the NHS really should not be paying for at all, such as acupuncture and tattoo removal. "It was unpopular," Jones admitted. "That's one of the things that was a challenge. Because Warwickshire had always balanced the books, there was a sense of disbelief that the problem was actually there."

PCT officials visited the GPs to talk it through and there were difficult conversations about whether the changes would work and whether people who needed urgent treatment would be denied it. Dr Francis Campbell, a GP who is also on the PCT, said some patients were anxious. "The biggest concern was not knowing what was going on," he said. "People came in saying I haven't heard about my hip operation or I have had a letter and what is going on here? Once I'd explained to the patients I talked to, they said yes - I understand."

Like Jones, he said that it is important to ensure patients get the treatment they really need. "After hip and knee operations, 25% of people say they have no improvement in the quality of their life. Mobility might not have been their biggest problem."

Hospitals were upset because their expected income dropped without much warning, but it avoided savage cuts. Jones describes it as "the difference between slamming the brakes on and driving at the right speed".

Since April, he has been chief executive of a combined Coventry and Warwickshire PCT with a budget of £1.4bn and aiming to save 5%, which will be "a challenge". Among the other areas he will scrutinise is GP prescribing, which accounts for about £1 in every £12 of PCT spending. The PCT estimates it spends £80m on drugs, of which 10% is wasted.

Warwickshire is merely one of scores of PCTs that since last autumn have been cutting their costs by restricting care by, for example, making patients wait longer or refusing them treatment altogether. Many have produced lists of treatment they deem to be of low or no clinical value, often sparking rows with doctors or surgeons, who fear that the NHS's increasingly desperate bid to avoid serious financial problems will mean more pain and inconvenience for people like Bill Walters.

No longer removing tattoos is one thing. But denying many thousands of people a hernia operation, or surgery to fit grommets to help a child hear better, could easily become a political issue.

When PCTs cut their spending, it is hospitals that feel the pain. Appleby says hospitals are being disproportionately leaned on to make savings, as the biggest spending sector of the NHS. "As part of the DH strategy in terms of closing the productivity gap, they are putting huge pressure on hospitals and there are only a few levers they can pull."

Chief among those is the tariff – the cost the NHS will pay for each treatment. So there is a fixed price for kidney dialysis or for a heart bypass. "They have set the tariff this year at a figure which is below the rise in inflation," said Appleby.

Getting less money for operations is a strong incentive to do them more efficiently – for instance by getting people fit faster so they can leave hospital sooner, reducing costly stays –and the risk of contracting a superbug on the ward. Hospitals, anyone will tell you, are potentially dangerous places, especially at the weekend and overnight.

How much hospitals must save is variable. The average is meant to be 4% but Monitor, the regulator of the Foundation Trusts (those high-performing hospitals that have more freedom to innovate and spend money), has said any that want to apply to join the club will need to be look at nearer 7%. It wants ever more efficiency from the best.

"Different hospitals have different financial histories," said Appleby. There are hospitals that have built up a surplus through canny management, recognising that tougher times are coming. Among the Foundation Trusts, said Appleby, that amounts to £3 to £4bn. Part of the surplus will now have to be used to cover their losses, as the PCTs and then GP commissioners cut back on the amount of treatment they buy – postponing hip operations, for instance or cancelling procedures that might – and might not – work.

"There may be 20 or 30 hospitals in an almost impossible situation financially," he said. "The solution is almost definitely merger and closing services. It's a big issue in London where we have maybe 40 hospitals. Maybe we really need 10. There are big mergers in the pipeline." Barts and the London is in talks with Whipps Cross and Newham, for instance.

King's College hospital is not in crisis. It chief executive is the only one in the NHS to have come from big business, but Tim Smart says the hospital he took over in 2008 is more efficient than BT Global, which he used to run. "I would say we are at least 50% more efficient than BT," he said. "The secretary of state talks about taking 30% of management costs out of the system. I couldn't take 30% out of here and run the hospital. I couldn't take out 10%." PCTs, however, he implies, are another story.

King's has found itself with less income thanks to the flattened tariff and wage rises agreed about five years ago. Smart says the move to community care and emphasis on public health will also reduce the role of hospitals. Kings is prepared. Last year it saved about 8% and plans to do the same this year. "We made efficiency savings of about £50m on turnover of about £600m," he said.

They reduced the number of agency doctors and nurses by about 80%. "We are being much more rigorous about planning people's time, making sure that not everybody is on holiday in Easter week or August. It's good for staff – it means we are preserving permanent jobs ... and staff say having lots of agency people on the wards causes them to do more work."

Among other measures, intensive care consultants have been persuaded to help in A&E when there is the need. And the average length of stay is being reduced to what they think best for each procedure. "What we have been really determined about is that patient safety should never be compromised. Hospitals are not the safest places," he said.

Smart is probably more comfortable about business ventures than most chief executives. King's now has joint procurement with Guy's and St Thomas's, reducing costs and enabling bulk buying, which last year saved about £1m. He has now embarked on a joint venture with the FTSE 100 company Serco, and again Guy's and St Thomas's, which will carry out hospital pathology work.

King's can think big and be ambitious, as a world-class centre for specialities like brain surgery and liver transplants. It could become smaller, losing much of the routine treatment, and still make money. For the average district hospital, the future could be much harder.

And there are real doubts over what is possible. Appleby bases his on history – even 2% has never been done before, let alone 4%. "Personally I'm not sure it's do-able, but I could be wrong. I'm not sure we will know after four years whether we have got the extra value in the NHS. Productivity is not the same as efficiencies and certainly not the same as savings. Simply sacking some nurses and saving money – the NHS can usually do that – and close down some wards. But that isn't the exercise. It's much more difficult than that."

He thinks the £20bn ought to be described in a different way – "such as the number of lives saved. When you express it in money terms, people immediately think that's what you have got to save."