Simon Cowell is staying mum about his baby-to-be, but the rest of the world sure is talking about the future daddy!

There have been rumors that the X Factor judge is "angry" with baby mama Lauren Silverman, but a source tells E! News exclusively this is "not true," explaining that the twosome are "fine" and "in constant touch."

Simon, 53, is vacationing at the moment—something he "always" does this time of year, per a source. Lauren, 36, on the other hand, has stayed close to home in New York, taking care of her and Andrew Silverman's 7-year-old child. "Her main priority is her son," says a source. "She wants to disrupt his life as little as possible during this transition."

Both Lauren and Andrew issued statements last week expressing their commitment to protecting their son's best interests. There hasn't been any public mud-slinging between the spouses of 10 years, but that doesn't mean things couldn't get dirty.

A prenup signed by the former couple reportedly states that Lauren is entitled to a $4 million payout if she and Andrew were married 10 years prior to the time of their divorce. One source tells E! News Andrew didn't find out about the affair and pregnancy until he and Lauren had "passed the 10-year mark of marriage," noting that the mom of one would tell her husband she was visiting her mother to keep her visits with Simon under wraps.

But don't think Lauren is planning to run to Simon for support. A source tells E! News that even though things are fine between the NYC socialite and her baby daddy, they're "in no rush to see each other."