When Jon Stewart took over "The Daily Show" from Craig Kilborn in 1999, he was not immediately made welcome and even considered quitting the show, he revealed in an interview with Stephen Colbert Friday night.

"I walk in the door, into a room with the writers and producers, and the first thing they say is, 'This isn't some MTV bull----.' ... And then I was told not to change the jokes or improvise," he told the former "Daily Show" correspondent, now host of "The Colbert Report," according to Third Beat Magazine.

Stewart told his agent, "get me the f--- out of this. These people are insane ... I had to be talked down from a moderately high cliff ... What I did not realize is, a lot of the people who worked there were a-------."

Two and a half years later, Stewart had the staff and the show he wanted.

The conversation between the two titans of late-night satire took place as part of a fundraiser for the Montclair Film Festival in New Jersey. The pair sat in curved brown comfy chairs in the Wellmont Theatre, simulating a fireside chat, and talked about their lives, their jobs and other stories to the delight of the sold-out crowd.

Further fertile territory for discussion involved show guests -- Stewart revealed that his least-favorite was Hugh Grant, who displayed divalike behavior during his appearance to promote "Did You Hear About the Morgans?" in 2009.

"He's giving everyone s--- the whole time, and he's a big pain in the a--," said Stewart, who added that Grant complained about the movie clip provided to the show by the film's publicist. He said Grant asked, "What is that clip? It's a terrible clip." Stewart replied, "Well, then make a better f------ movie."

He said he'd never let Grant back on the show.

Even more tricky territory came from Stewart's inclusion of Yusuf Islam (who changed his name from Cat Stevens) in the "Rally to Restore Sanity" in 2010. Stewart, Ozzy Osbourne, the O'Jays and Colbert joined Islam in a train-themed song parody (Islam/Stevens famously recorded "Peace Train"). But afterward, Stewart heard from author Salman Rushdie, who was disappointed to see Islam included, believing he had supported the 1989 fatwa leveled against Rushdie after the publication of "The Satanic Verses."

Stewart relayed that he got a call from Rushdie, and assured him he hadn't known of Islam's support of the fatwa. "So I'm like, I'm sure he doesn't believe that people should be put to death for apostasy," said Stewart, who promised to call Islam and get it smoothed over.

His chat with Islam later didn't go as he'd hoped. "We get into a whole conversation, and it becomes very clear to me that he is straddling two worlds in a very difficult way," said Stewart. "It broke my heart a little bit. I wish I had known that. I wouldn't have done (the routine), I don't think ... Because that to me is a deal-breaker. Death for free speech is a deal breaker."

It wasn't all about Stewart and the ups and downs of working with "The Daily Show," but Colbert's recollections were largely less polarizing -- although he did surprise the audience and Stewart with the revelation that he used to sleepwalk -- and act out his dreams.

Colbert revealed that when he and his wife were still newlyweds, he dreamed ghosts were coming out an imaginary tracheotomy hole in her neck -- and he had to hold them in. She woke up to find him with his hands on her neck.

When she asked what he was doing, Colbert replied, "I'm keeping the ghosts in."