The music mogul was due to make an appearance on the talent contest but backed out of a return to the show

Pressured Simon Cowell has axed his big return to UK X Factor as the furore over his affair with a friend’s wife gathered pace.

The show boss was due to invite judges Gary Barlow, Sharon Osbourne, Louis Walsh and Nicole Scherzinger on to his £45million yacht in the South of France to dish out their categories.

But after it was revealed last week that married American Lauren Silverman was expecting his baby, Cowell has put ?his comeback on hold – and ?ITV bosses are said to be furious.

All four stars had been due to fly to St Tropez on a private jet this Thursday, to meet Cowell as he holidays.

He would then have told them in person which group they were receiving. It was his return to the show after leaving the panel in 2011.

Splash

Simon's yacht Slipstream

But in an 11th hour change of plan, he phoned the judges yesterday to tell them they wouldn’t be coming.

The cameras will still follow the X Factor boss in the South of France, but he will tell the stars his verdict by phone.

A show source said: “Simon has delayed the original date he was going to set sail on his boat and it is not going to work in his schedule now, so it has been cancelled.

“His problems and drama off screen are now starting to impact on his TV shows. He even called the UK judges personally to apologise for messing them around.

“ITV bosses are furious at the last-minute change. Simon’s return was going to be a big coup for them, but now that he’s just doing exactly the same as he did last year, it’s no big draw for the viewers. It’s really disappointing.”

Gettt

Simon was due to return to the show

In a twist on the usual format, this year’s mentors will find out which category they receive before the bootcamp stage, which begins this weekend.

Yesterday, the Mirror revealed that of Cowell’s usual harem of beauties only Sinitta will accompany him for the holiday aboard his yacht Slipsteam. Meanwhile, the star is also filming the US show until the end of the year.

Last week, bootcamp in the States was cancelled after 220 contestants had waited to perform for 15 hours. Producers chose their Top 40 acts based on initial auditions alone.

Once the show finishes, Cowell will have to look at juggling his ?TV career with bringing up a new baby in 2014.

Despite Lauren being hit with divorce papers in which her estranged husband Andrew accused her of adultery with Cowell, the pair spent the weekend with their son.

Although Lauren is expected to get up to ?£3million under a ?prenuptial deal, the couple will reportedly fight over child custody.

US reports yesterday insisted she was surprised to be pregnant as ?she was ?unsure she could still conceive.