The "Portlandia" star married the "Mad Men" actress, 30, in October 2009 after dating for one year. Though he's trying to work on his bad behaviors, Armisen said he gets "very caught up in the beginning."

"I want it all -- fast," he explained. "I want to be married ... the amount of girls I've lived with right away ... and then somewhere around a year, two years, I get freaked out."

"Do you feel entitled to more women?" Stern asked.

"I don't want to admit that out loud to myself," Armisen said, "but that probably is it."

And to end his relationship, the "SNL" star admits he starts doing things to make the person hate him.

"[It's] awful ... you sort of withdraw and you sort of get into all the things you think make you weird, like 'I'm really into playing Xbox ... [or my] record collection.' You sort of disappear into the things that you think are just your hobbies."

"It must be so shocking to the woman because you're this guy who is so super-attentive in the beginning ... and you are like the best boyfriend in the beginning, and then you become the worst boyfriend," Stern explained.

"I feel bad for everyone I've gone out with," Armisen said.

Armisen's marriage to Moss was his second. He was previously married to English singer and songwriter Sally Timms from 1998 to 2004, but told Stern their marriage was "more for her green card."

Shortly after splitting from Moss, Armisen began dating his "SNL" co-star Abby Elliott, but the romance "just fizzled," he told Stern.

The co-workers ended things on "so-so terms." Though Elliott, 25, has since left the show, Armisen said, "Once in a while, it would kind of be a little weird at work, but I have to say, for the most part, it was fine. The year that I worked with her where I was dating her, it was non-eventful."

When Stern asked if he had her fired, Armisen replied, "I would never do anything like that."