Justice Secretary Michael Gove, one of the most prominent figures backing “Brexit” in June's referendum, is being touted as the most likely figure to have leaked details to The Sun that the Queen is claimed to have told ex-Lib Dem leader Nick Clegg she believed the union was heading in the wrong direction.

The Sun on Wednesday

Both Buckingham Palace and Clegg have dismissed the claims.

Now Labour has said it will write to Jeremy Heywood, the Cabinet Secretary, to register concern over leaks of the Queen’s private conversations.

It also emerged Gove helped David Cameron, who is campaigning to stay “In”, to prepare for today’s Prime Minister’s Questions in the House of Commons.

The Queen speaks to then Lord Chancellor Michael Gove

A Labour Party spokesman said: “(Labour MP) Wes Streeting is writing a letter. It will be interesting to see the response. Let’s see what comes back.”

After PMQs, the Prime Minister’s spokesman was asked repeatedly whether Cameron discussed the row with Gove this morning.

He said: “I wouldn’t guide you towards that conclusion.”

According to official Court Circular records, there were four other people at the Windsor Castle lunch where the exchange between Her Majesty and the then Deputy Prime Minister took place on April 7, 2011.

They were Gove, Cheryl Gillan, the then Welsh Secretary, Lord McNally, who was a justice minister and Judith Simpson, a clerk.

Buckingham Palace has complained to the press watchdog, the Independent Press Standards Organisation, over the article.

In response, the paper said it "stands by its story", saying it was "based upon two impeccable sources and presented in a robust, accessible fashion. A spokesman added: "The Sun will defend this complaint vigorously".

At the lunch, the monarch gave Clegg a dressing down on the direction of the EU and European integration, according to the tabloid.

The Queen is meant to remain strictly neutral on political matters and the palace has complained, saying the story is inaccurate.

A palace spokesperson said: "We can confirm that we have this morning written to the chairman of the Independent Press Standards Organisation to register a complaint about the front page story in today's Sun newspaper.

"The complaint relates to clause one of the Editors' Code of Practice."

Clause one of the code relates to accuracy and says the press must not publish "inaccurate, misleading or distorted information".

The paper cited a "highly reliable source" who relayed details of the conversation but Clegg later said: "I've no recollection of this happening & its not the sort of thing I would forget."

Tom Newton Dunn, The Sun's political editor, called this a "non denial denial," a term used by journalists to describe an attack on a story that does not actually refute its assertions.

Speaking on LBC Radio, he said: "He's not confirming it. He's not denying it in any way, shape or form. As slightly ruder people than me have already said today, it's one of those 'non-denial denials' we hear from politicians.

"A form of words quite carefully constructed to make you believe they're denying the story when they're actually not... We do believe this incident has happened. We do believe Her Majesty has strong views on Europe."

Re Sun story. As I told the journalist this is nonsense. I've no recollection of this happening & its not the sort of thing I would forget

He said the palace and Clegg conferred before responding to his request for comment, prompting host Nick Ferrari, an ex-Sun journalist, to say: "It's got to be true... They don't ring each other unless they're nervous about something."

The paper's source told the paper: "People who heard their conversation were left in no doubt at all about the Queen’s views on European integration.

"It was really something, and it went on for quite a while. The EU is clearly something Her Majesty feels passionately about."

In a separate piece, Newton Dunn wrote the paper would not have published the story if the comments "had not come from two different and impeccably placed sources".

Clegg's spokesperson later added: "This is categorically untrue. Nick has no recollection of this conversation and it is not the sort of conversation you forget."

Labour has called for an inquiry into the source of the leak and written to Cabinet Secretary Sir Jeremy Heywood to ask how private conversations became public.