"We will miss them. We will have to up our game now. It will be a different show," the presenter told The Mirror. "Whoever they bring in, no one can replicate Simon. There will only ever be one real Mr Nasty. He is a force of nature, he's our boss.

Advertisement - Continue Reading Below

"It's disconcerting because I signed up to host the show this year not knowing who the new judges would be so it will be a challenge not having Simon there week in, week out. Whenever I have a problem or a question, I go to him. Now I'll have to figure it out for myself. We all have to work a little bit harder."

O'Leary continued: "I signed early but you have faith in the show - it's the biggest in the country. I'd be the biggest idiot not to do it. It feels like home."

However, the 37-year-old admitted that he expects Cowell to remain in contact throughout the next series.

"Simon is so unpredictable. He is so dedicated to the show that I'm sure he'll turn up in some way and surprise us," he said. "I'll always have my phone on me. I have no doubt he'll be ringing."

On Cole's rumoured exit, he added: "I'm so happy for Cheryl doing the US X Factor. She will be great but we're going to miss her here."