Receive the latest entertainment-news updates in your inbox

Speculation was rampant regarding who would replace David Letterman on the "Late Show" after the veteran host announced his retirement on April 3. One week later CBS revealed Comedy Central star Stephen Colbert will be taking over Letterman's duties when he exits sometime in 2015.

According to Neil Patrick Harris, Colbert wasn't the only TV comedy star being considered for the gig.

"I think I would get bored of the repetition fast, and the structure of it is so set that I don't have any interest in doing monologue, commercial, sketch, guest, guest, musical act, goodnight," said Harris, who told Stern the idea was pitched over sushi.

Stephen Colbert's Top Moments

"I was surprised that he pitched me that idea, and I sort of — I sat for a minute with it ... and told him the things that concerned me about the longevity in that kind of gig," said Harris, 40, whose nine-season stint on "How I Met Your Mother" ended with the series finale in March.

While Harris is currently busy performing on Broadway in "Hedwig and the Angry Inch," the notion of hosting a weekly, rather than nightly, televised variety show holds appeal.

"I think if it is weekly and people really want to see it and you trust that there's a lot of really great (expletive) on there, then you are going to get the guests to want to come on," he told Stern. "You'd have more time to prep for stuff, you could do pre-taped stuff that would be really exciting, you could flesh it out a bit more."