"I don't know how The Sun got all its information and I don't think it's really worth my adding to what has already been said about this story," he said.

Asked if he thought the Queen would be pleased to see him when he next has an audience with her, he replied: "I don't know. I wouldn't speculate on a matter like that."

Buckingham Palace has dismissed the story and complained to the Independent Press Standards Organisation (IPSO).

Darren McCaffrey, Sky News Politics Correspondent, said: "Speculation will continue about who the source is and it will take time for IPSO to report on its findings.

"What it does demonstrate is the high stake politics involved, that the Justice Secretary, a close friend of David Cameron, might be involved in the leaking of a private conversation with the Queen - a sign of how combative the referendum campaign is becoming."