Theresa May at the European council in Brussels, June 2017.
Photograph: Olivier Hoslet/EPA

Theresa May and her circle of advisers did not understand how the European Union works, and consequently followed a negotiating strategy in 2016 that was doomed to fail, the former UK ambassador to the EU Sir Ivan Rogers has said.

Speaking to the Institute for Government on Monday, Rogers said the people around the prime minister at the start of the article 50 process “didn’t know very much about European councils or that much about the EU”.

Rogers, who resigned a year ago and has developed a reputation for producing some of the most caustic assessments of the misunderstandings between the UK and the EU, said the UK lived under the illusion that it could circumvent Brussels by making direct deals with the major capitals.

He said: “Capitals obviously matter, but I think having lived through this with a number of prime ministers, a number of different negotiations ... that reflex in the British system always to think that we can deal direct with the organ grinders and not the monkeys: it never works like that.

“It didn’t work like that in the Cameron renegotiation either. That stuff is not done in the way British politics works, leader to leader. It’s done via the bureaucrats, and the sherpas, and the people at the top of the institutions.”

Rogers also warned that the chances of a no-deal Brexit sticking for long were close to zero. “The UK and EU know there is no chance of no-deal Brexit being the long-term end state, as the UK would quickly come back to the negotiating table,” he said. “There is not a world where we are going to end up with no deal.”

He also warned the UK could face a legal challenge to the European court of justice if it sought to stay temporarily in the EU by extending article 50 for more than three months, but also tried to avoid participation in the May European parliamentary elections.

He said the only talks extension on offer to the UK from the EU may be as long as nine to 12 months, since the EU will not wish to grant an extension simply because the UK is in a state of political chaos. He added if unelected UK MEPs remained in the European parliament, it was possible that any newly appointed commission, endorsed by that parliament, would be struck down as illegal.

Rogers said the issue of whether the UK should contribute into the EU budget would come up in April. He also said there was no chance that the UK would be able to disentangle itself from the EU even if Brexit goes ahead.

He said: “These fantasies of release and liberation – they are fantasies. We are going to be negotiating on everything from aviation to farming for evermore with our biggest neighbour. We cannot live in glorious isolation. Talk to the Swiss and to the Norwegians – they live in a permanent state of negotiation with the EU.”

He also said he was deeply concerned that May had in the negotiations prioritised goods over services “even though it’s a modest proportion of the economy”.

Rogers predicted a huge blame game between the EU and May if she is unable to get an adjusted deal through the Commons, followed by a brutal row over the UK’s requirement to pay its outstanding negotiated payments to the EU.