"It started out as a goof," he told listeners of his Sirius XM radio show Tuesday morning. "Now it's becoming a job."

Stern joined "AGT" back in season seven and returned for the next one, even though he originally only planned be a reality TV panelist for a single season.

The decision to come back a third time wasn't an easy one for Stern, and it involved "much neurotic whining to everyone," he explained. But in the end, he admitted that there's a "big, black dark hole that lives inside of me that can never be filled with enough attention," so he just had to come back.

When he announced the news on his show, he assured co-host Robin Quivers there's a limit to this return engagement.

"I am going back for one season more," he said.

As for the fate of his fellow panelists for the ninth season — Howie Mandel, Heidi Klum and Mel B —Stern wasn't certain about any of them, but said, "I'm assuming it's the same crew."