Diff'rent strokes, indeed. The late former child actor Gary Coleman worked with prolific porn star Ron Jeremy four times: on the 1998 Kid Rock music video "Cowboy"; a 2002 Weakest Link episode; a 2004 episode of VH1's The Surreal Life; and in the 2009 film comedy Midgets vs. Mascots.

"I've known Gary for years," Jeremy tells UsMagazine.com of Coleman, who passed away at age 42 on Friday. "He was always very nice to me, but he was tough to get along with."

Case in point? During Coleman's cameo on The Surreal Life, he butted heads with another faded star, Vanilla Ice. When the rapper teased him and asked Coleman to repeat his trademark Diff'rent Strokes catchphrase, he stormed off the set. "He was so pissed off," Jeremy recalls. "He was a sweet guy, but he hated being asked to repeat 'Whatchu talkin' 'bout Willis?'…He didn't want to live in the past."

To be fair, Vanilla Ice's ribbing didn't stop there, Jeremy explains to Us: "He's carrying Gary around and going 'Get in My Belly,' like they do in Austin Powers. Vanilla Ice is carrying him around and threatens to put him in the deep fryer. Gary's going 'I hate this guy. I hate this guy. Let go of me.'"

Coleman "didn't like people making jokes about him," Jeremy observes. "He had a very sensitive nature."

The 4'8" star was frank to the adult film actor — and the public — about his complete lack of sexual experience. When the duo appeared together on game show The Weakest Link, Coleman, then in his mid-30s, introduced himself as "'the opposite of Ron Jeremy. I'm still a virgin…Whatever Ron has done, I've done none of it.'" Jeremy remembers. ""I think everyone thought he was joking. Gary was like, 'I'm serious, I'm a virgin.'"

(Coleman married Shannon Price in 2007; she was at his bedside when he passed away.)

Naturally, Coleman confided in Jeremy about sex. "He did ask me, "When you're with a lot of women, do you get emotional with them?'…He made it clear that he wasn't into frivolous sex. He could have gotten some considering he was a celebrity, but he was more into having an emotion behind it."

Jeremy's happiest memory of his pal? "He really enjoyed being in Kid Rock's video…Just seeing him with a big smile. He didn't smile a lot, but on the video, he had his whole cowboy outfit with the hat, and it's really cute, and he really enjoyed it."

The porn legend adds that his late friend, who was making a reported $100,000 an episode for Diff'rent Strokes in his tweens, didn't have a "real childhood." "It took its toll," he opines. "Kids need to have their childhood."