Of course, when one celebrity judge leaves, it opens the floodgates for all of the speculation as to who should be passing out the compliments and critiques when the show returns for another season. If the newest rumors are true, Simon is going to be the sole dude with some very talented ladies at the judge's table.