"It's been the worst year in living memory," says Jonathan Lukies, who farms 288 hectares (720 acres) of arable and fruit orchards near Stansted, Essex. "It was horrific."

This year's weather has been a rollercoaster for British farmers that most now just want to forget. With a record drought afflicting most of England in the early spring – one so severe it prompted a series of emergency meetings with government – farmers desperately needed above-average rainfall to replenish the soil for planting. Their prayers for rain were answered – but in the worst possible way, with the wettest early summer ever recorded, followed by a near-sunless summer and torrential downpours in many areas late in the growing season.

This combination of extreme weather was disastrous for staple crops such as wheat and vegetables, first putting off growth and then washing out crops and preventing them from ripening.

"Starting from Easter weekend, we had a year's rainfall in three months," says Lukies. "That was the killer. I've never seen anything like it, and my father who is in his 60s says he's never seen the like either."

His views are echoed across the farming community. Guy Gagen, crops adviser at the National Farmers' Union (NFU), says: "Speaking to farmers who have been in business for decades, they don't remember anything being as difficult as this year. There have been bad years before, of course, but this has been terrible right across the growing season, from beginning to end.

"One of the problems was that it was just so dark – there was too little sunlight for crops to grow. If you think back, some days in June were like November. That really reduced productivity."

Every sector of farming has been hit. Arable farmers have seen yields of wheat fall by 14%, according to the NFU, reducing the UK's wheat crop to levels not seen since the 1980s – before many farmers invested in modern technology such as grain driers. Vegetable growers have suffered, with half the pea crop wiped out across the country. Meat producers, from poultry to pig farming, have seen their overheads soar due to the poor global grain harvest raising feed prices. Salad and fruit growers have also had a dreadful year, with fresh produce being thrown away or fetching abnormally low prices during June and July, as people were simply not buying summery foods because of the miserable weather.

Prices to consumers are rising sharply as a result of all these factors. That might appear to mean a bonanza for farmers – at least the food they have been able to grow is now fetching top prices. But most farmers are not profiting.

For many arable farmers, one reason is that they hedge their risks on future yields by contracting to sell their grain at an agreed price before the harvest, which gives them a guaranteed income on what they can produce. Lukies was typical, contracting for about two-thirds of his crop in advance. But prices early in the season were low – about £150 a tonne. As the harvest came in, prices soared to £205. Lukies was unable to cash in even on the third he had not sold in advance, as most of his crop was of too poor quality, owing to the weather. Thousands of farmers are likely to have found themselves in a similar bind.

For other farmers, such as vegetable growers, any price rises are outweighed by the much lower yields of their crops. "People have invested money in growing these crops, and seen the yields sinking," says James Hallett, chief executive of the British Growers Association.

As has been increasingly the case in recent years, the fate of most UK farmers will be dictated by the supermarkets. The "big four" have such a huge market share that they can determine the margins farmers receive. They even manage to soak up much of the one source of income farmers can rely on even in tough years – farm subsidies.

An average family of four in the UK was paid £426 a year in farm subsidies under the EU's common agricultural policy (CAP) in 2009, according to the Department for Environment, Food and Rural Affairs (Defra), and poor people are hit hardest as that represents a higher proportion of their disposable income than for the rich. That figure, amounting to more than £8 on every weekly shop, is likely to have increased since 2009 but no new research has been done. The money is supposed to support hard-pressed farmers. But the stranglehold of the supermarkets means that much of it goes to the big firms' bottom line.

That is because the big retailers know how much farmers receive in subsidy – it is public information – and can calculate the farmers' cost of production, so that they push down the prices they pay farmers. For thousands of farmers, this means producing food such as milk, poultry and pigs at a loss – what the supermarkets pay is below the cost incurred. Only the subsidies they receive keep these farmers in business, making them just about profitable enough to carry on.

So the subsidies paid by UK households are in effect boosting the supermarkets' profits more than the farmers – the big retailers can reap higher margins from the food they buy from farmers by paying less than the cost of production, leaving taxpayers to pick up the bill for keeping farmers in business. That is one of the reasons consumers on mainland Europe pay less for their shopping – the rise of 6.4% to UK consumers' weekly shop is double the EU average.

Defra said the government was trying to reform the CAP. However, while the big supermarkets are able to dictate prices, farmers and consumers are likely to continue to be squeezed.

Matthew Sinclair, chief executive of the TaxPayers' Alliance, said: "The CAP is a disaster for Britain, forcing up our food bills and costing taxpayers a fortune in administration – it undermines the competitiveness of British farmers while cheaper imported food becomes more expensive because of tariffs and quotas. This keeps prices artificially high at a time when so many families are struggling to even put food on the table."