An unexpected summer crisis is developing in the NHS, with performance in accident and emergency departments deteriorating, bucking the normal trend of waiting times being worse in winter than summer, according to the Labour party.

The figures show a record number of patients attending hospital A&E departments in any one week and a record number then admitted to hospital wards. The figures show a total of 296,667 going to A&E in the last week of May, again raising questions about the adequacy of GP services. Of these 77,745 were admitted to hospital.

A&E departments are supposed to see, treat, and admit or discharge 95% of patients within four hours of arrival at hospital, but in the last four weeks the number of patients waiting more than four hours has ranged from 22,231 a week to 24,503.

By comparison, the numbers waiting in the four weeks of November for the equivalent period were between 13,938 a week and 17,372. The figures waiting this summer are also double the number of patients waiting more than four hours in the summer three years ago.

A total of over 16,000 patients were waiting up to 12 hours on trolleys for a ward bed.

Figures published last week revealed a record 300,000 patients arriving at A&E departments and record numbers admitted to wards.

The president of the College of Emergency Medicine, Dr Clifford Mann, said: "These figures underline the need for significant action. The college has exhorted the NHS to decongest A&E departments by providing accessible alternatives for the 2.1 million patients who could be safely redirected from triage and to mandate full capacity plans to address the pernicious problem of 'exit block'; failure to do so will mean these figures go from bad to worse and lives are endangered."

Labour is keen to see the future of the NHS rise up the political agenda, and was struck by polling released this week by the Conservative peer Lord Ashcroft showing its importance to voters.

When voters were asked what they saw as the most important issues facing their families, the NHS topped the list when it came to people's personal priorities: 57% named the NHS as in their top three, just pipping the economy and jobs (56%)

Jamie Reed, shadow health minister, said: "People can see that the NHS is heading downhill under this government. A&Es are facing the worst year in a decade – there's now a summer crisis that's worse than the winter one.

"Ministers have taken social care support away from older people and made it harder for others to get a GP appointment. A&Es are struggling to deal with the extra pressure.

"Hospitals are operating above safe levels – A&Es and hospital wards are full to bursting."

At the same time Labour sounded the alarm on the extent of the use of agency workers in the NHS as spending on temporary staff in foundation trusts hit £1.4bn in 2013-14.

In a sign of the staffing crisis hitting the NHS, Monitor's latest report revealed that while foundation trusts planned to spend £523m on agency staff last year, they instead spent 162% more than expected.

Foundation trusts' annual accounts for the past five years show that the proportion of staffing costs spent on agency workers has increased by 20%.

The shadow care minister Liz Kendall said: "It's now clear that ministers have lost control and that NHS finances are going backwards."

The government insists it is trying to recruit extra staff for A&E, but claims Labour is trying to create a crisis.