Bobbing on the port captain's launch off the coast of Gibraltar on Tuesday there was no evidence of how this calm stretch of sea could have caused such an international storm. A handful of craft fishermen cast their nets, seemingly oblivious to the fact that 10 metres below their dinghies lay the unlikely catalyst for a political row that has embroiled David Cameron in the UK's bitterest battle with Spain over "the Rock" since Franco.

Here, late last month, Gibraltar dumped an artificial reef on a fishing ground favoured by Spanish scallop dredgers. Now the ripples from those dozens of concrete blocks are rocking a 300 year old British enclave that for some is an emblematic imperial redoubt and for others an awkward colonial hangover.

On Tuesday Gibraltar said it was preparing for legal action against Madrid over its retaliation for the reef, which has taken the form of a frontier control go-slow that has caused residents to queue for up to six hours in scorching summer temperatures. Gibraltar has accused Spain of inhumane behaviour and is gathering evidence that could be used at the European Court of Human Rights.

But a widespread view in Gibraltar and the Spanish borderlands is that this bitter row is not fundamentally about fishing nor overheating drivers, but Spain's attitude towards Gibraltar's self-determination and Madrid's desire to deflect attention from an awkward corruption scandal in the ruling party and worsening economic woes, especially in the south.

On Tuesday as the former Labour minister Peter Hain called for fresh negotiations with Spain towards a joint sovereignty deal, tit-for-tat retaliations continued.

Long queues persisted at the frontier between the two countries as Spanish border guards stopped most vehicles for checks. Gibraltar's leadership believes Spain is ready to continue its tactics for some time, and that life in the colony will get worse before it gets better.

Still in place are threats by the Spanish foreign minister, Jose Garcia-Margallo, to impose a €50 border fee, investigate the tax status of 6,000 Gibraltarians owning properties in Spain, close Spanish airspace to flights headed to Gibraltar and bring its huge online gambling industry under Madrid's tax jurisdiction.

"We are compiling statistical and anecdotal evidence about the operation at the border and we are now intensifying that record-keeping," Joseph Garcia, deputy first minister of the British colony, told the Guardian. "We are collecting individual complaints in case we need to take action on the basis that an individual's rights have been infringed. The Spanish actions at the border are totally inhumane."

Gibraltar also complains that Spanish authorities have even stopped people in pleasure boats in the disputed fishing waters and insisted on seeing their papers, while it says Spain is treating the border like a tap it can turn on and off.

The British border guards still have sufficient communication with their opposite numbers just a few metres over the border to know when the tap will be turned off, it is claimed.

"The other day we knew it would be 5.30pm and then at about 5.15pm, sure enough, it started clogging up," said one official.

Spain, by contrast, believes the reef is stopping its citizens from working, a highly sensitive issue when there is reportedly 40% unemployment in the borderlands by Gibraltar. It believes the reef is making the marine ecology worse, not better, and argues that the move was simply intended to frustrate Spanish fishermen.

"They must take the blocks away and allow the fishermen to work," said Carmen Crespo, the government delegate in Andalucia. "All this government is doing is controlling its border against fiscal fraud and smuggling."

Residents on both sides have empathy with those over the border, and are united in their frustration at the politics. In La Linea, the impoverished Spanish settlement where gutted fish hang to dry outside blue-and-white-tiled cottages and the number of working fishermen has halved in the last year to 160, Gibraltar's new reef is seen as "a real problem" .

"Our fishermen have been fishing this area their whole lives," said a spokesman for the town's fisherman's union. "In 1713 [when Britain took authority over Gibraltar from Spain] their ancestors were fishing there – and now they tell us we have to stop fishing."

Susie Barranco, the town's deputy mayor, said people were frustrated at their fortunes being buffeted by international politics between London, Madrid and Gibraltar. "We want to fish in peace," she said.

Ordinary Gibraltarians feel victimised by Spain's response. "Spain is being a bully, it is very sad," said one, Regina, after an hour-long queue for the frontier. "Spain has always targeted Gibraltar quite badly. The people get on well but it is just the politicians. If Madrid had treated Gibraltar with respect it would have been Spanish by now."

"It is a disgrace," said Julie Thompson as her car crawled towards customs. "It is so much time wasted daily."

For all its pubs called The Piccadilly, red phone boxes and Leyland red double decker buses that lend Gibraltar the air of being frozen in the mid-1980s, it is a cosmopolitan town with a population of 30,000 with roots in the UK, Spain, Malta, Portugal and Italy, as well as a thriving community of Sephardic Jews. As many as 12,000 Spaniards also rely on Gibraltar for work, many of them women in domestic and care work.

Protesting in the shade of a tree on the Spain side of the border was Juan Jose Uceda, a leader of the Association of Spanish Workers in Gibraltar, who was furious at Madrid for making a bad situation worse for people who can't find work in Spain.

"The government should be discussing this in official circles, but they land us with the problem and make us queue daily for up to six hours," he said.

Peter Howitt, a lawyer who advises the online gambling industry which employs 2,900 people in Gibraltar – accounting for more than 15% of its GDP, works close to Casemates Square, where English tourists mill among pungent "traditional British fish and chips" shops. But Gibraltar, for him, has a more international draw with workers coming from across Europe, meaning that the border spat "has a massive impact".

"This is a tiny place on the border of a superpower that it relies on for access to the European Union, and that is not a fight you want to get into," he said.

Inside Gibraltar's equivalent of 10 Downing Street, a converted convent school, there has been a sense of siege. Beneath portraits of the Queen and Prince Philip, a rumour circulated on Tuesday that a couple of cars owned by Gibraltarians had been vandalised on the other side of the border. The first minister has been on the phone from his holiday in the Algarve wanting updates.

In the waiting room copies of the daily news sheet, Panorama, scream: "Gibraltar MEPs on the warpath" and "Boycott Spain" as well as an item about a Union Jack-carrying Gibraltarian protestor who said the Spanish police threatened to beat him up after he dropped his trousers and mooned at them at the frontier.

Gibraltar is trying to make as much noise as possible about the situation because it needs public opinion on its side against a powerful neighbour with which its population interacts happily on a daily basis. Joseph Garcia, who has a PhD in Gibraltar's political history, was not ready to tone down the rhetoric.

"We have seen it all before under General Franco," he said referring to the 13 years in which the dictator closed the border. "The overwhelming issue then was about sovereignty and today it is the same. This is not about fishing. Spain has tried it before and it was counter-productive. This will probably turn a whole new generation of Gibraltarians against them. The Spaniards are having a public relations disaster."

For many the comparisons with the violently oppressive dictatorship of Franco and, earlier this week, North Korea seem shrill.

But Garcia says it is proportionate: "It is not about being abrasive. It is a question of defending our interests and it is a fact Franco used to behave like this."