In July 1837 Britain opened an engineering marvel: the world's first, fast intercity railway between London and Birmingham. A century and a half later, the route came to symbolise Britain's disastrous transport planning. Taxpayers spent £9bn upgrading the line, only for trains to run more slowly and less frequently than promised. Figures last week showed that the west coast route is one of the least reliable services in the country.

The west coast shambles is the reason that the transport secretary and rail engineers want to escape from the make-do-and-mend approach which has seen Britain slip down the railway league table. While Japan, France and now Spain made the leap into a new generation of rail technology and high-speed routes, Britain neglected its trains.

"Railways were not the priority. The priority was to complete the motorway network and manage the railways as best we could on an annual budget," says Jim Steer, former head of strategy at the Strategic Rail Authority.

A succession of transport ministers aimed to keep trains out of the headlines.

"In the 1980s British Rail was expected to manage the railways better, to be more customer-responsive. There was no thought that rail might do more – or in fairness, less – than it had done over the previous 10 years. Any investment expenditure was expected to be more or less self-financing," he said.

In tight circumstances, BR did its best, investing in some electric routes and the InterCity 125 diesel express, still the mainstay of many services. But modernisation always faltered in the face of political confusion and a lack of cash.

In 1955, the recently nationalised British Railways promoted an upgrade plan that aimed to put the network back in profit. Instead much of the cash was wasted.

"British Rail was obsessed with setting out a vision, but never defined what the railways were for or what they were good at," said Tim Leunig, an economic historian at the LSE. "The government threw huge sums at BR, which always promised to break even and never did. It was the most bailed-out organisation in Britain."

While France was setting out plans for its TGV fast network, Britain was experimenting with incremental improvements, including the short-lived APT tilting train from London to Scotland.

"Nobody was prepared to find the money for the investment under the Tories, and not a lot under us," admitted John Prescott, transport secretary from 1997 to 2001.

Stephen Glaister, professor of transport and infrastructure at Imperial College London, said: "British governments have been much more concerned about value for money and have been much less willing to spend taxpayers' money on any form of transport and big infrastructure. The French and Spanish have taken a completely different line."

Decline set in long before privatisation, which ministers hoped would free railways from the political neglect that had left them underfunded.

"The principles behind privatisation were to create a train-operating industry with a number of players," said Sir George Young, the transport secretary who oversaw the process. "The aim was to increase capital investment by placing the industry in the private sector so it was no longer at the end of the queue for Treasury investment."

Bound by tight Treasury spending rules, however, there was never enough long-term money for Britain to copy France and build new routes.

"In an ideal world unconstrained by limited resources, I would be wholly in favour of new high-speed rail lines," said Young. "But if the funding for them comes out of a highly constrained rail transport budget, one has to ask some hard questions about priorities."

Privatisation, completed in 1997 as Labour came to power, put the brakes on strategic thinking.

Companies bought new trains and passenger numbers climbed by 40%, but costs rose too: the current rail subsidy of almost £5bn a year is well above levels under nationalisation.

Prescott said that in Tony Blair's first term the government was preoccupied with bailing out the high-speed Channel tunnel route from Dover to London and the task of dealing with the now-defunct Railtrack, the owner of Britain's privatised tracks, stations and signals which was taken over by the government-backed Network Rail in 2002.

"That was a real bloody problem because we had people in control who had more interest in getting profits than doing the work properly," he said.

Prescott's lessons from the Channel tunnel rail link, since renamed High Speed 1, contained a veiled warning about the private sector's ability to underwrite a major transport project.

HS1 was completed only after the government stepped in to guarantee bonds issued to pay for the £5.7bn route: "The government was the only one that could find the resources, so we nationalised it [HS1] – and it came out on time and on schedule."

• This article was amended on Wednesday 5 August 2009. The tilting train we mentioned is called APT, not ATP. This has been corrected.