However, a palace spokesperson tells PEOPLE that Charles and William were never supposed to take part in the meeting.

“In line with standard practice for guest of government visits the President was received by The Queen as Head of State. The arrangements were mutually agreed between the Palace and the U.K. and U.S. governments,” the spokesperson says. “There was never a suggestion that other members of the royal family would take part.”

Getty Images

Adds Dickie Arbiter, former spokesman for the Queen and author ofOn Duty with the Queen, “It was always going to be a short visit and as soon as you have too many people it gets longer. It was a courtesy call – that’s all it was.”

The Trumps were welcomed at the Quadrangle of the castle, where a Guard of Honor gave a royal salute and the U.S. national anthem was played. The Queen and President Trump inspected the Guard of Honor before watching the military march past. The president and first lady then joined the Queen inside the castle for afternoon tea.

In addition, Trump was slammed online forawkwardly blocking the Queenduring their walk while inspecting her Guard of Honor. In a viral clip, the monarch appears to make a hand gesture as they turn the corner. She then attempts to pass behind the president, who stops walking and completely eclipses Queen Elizabeth from view for a moment. She returns to his right side and continues their walk.

Grant Harrold, a former butler for the royal family, told ITV News that letting the Queen lead in every scenario is a top rule in royal protocol.

Memorably, Prince George stayed up late to greet the commander-in-chiefclad in a bathrobe, matching gingham blue pajamas and slippers before taking a ride a rocking horse – a gift from the Obamas when he was born.

The special evening – which followed lunch Buckingham Palace with the Queen – was arranged by William on behalf of all three royals, and “they were delighted that [the First Couple] were able to accept,” a palace source told PEOPLE.

“The three of them have built up a relationship with the President and First Lady over the years,” added the source.

Harry also hosted Michelle and daughters Malia and Sasha – along with grandma Marian Robinson – for tea at Kensington Palace back in June 2015. In October 2015, the Obamas played host to a visiting Prince Harry – where the chemistry between Michelle and the royal was on adorable display.