The "Mad Men" star has garnered a huge female fanbase since shooting to fame as Don Draper in the hit TV series, and he is regularly considered one of the sexiest men in Hollywood.

But the actor admits he's taken aback by the interest in his image, and just wants to be recognized as a good actor.

He tells the Pittsburgh Post-Gazette, "I don't necessarily think of myself like the handsome guy. That's reserved for Brad Pitt and Ryan Reynolds and those guys. It's certainly nice when people say nice things about you, don't get me wrong. I guess I never really thought of myself that way. I just wanted to be a regular person and try to portray parts as varied as I could."

"It's a weird experience to go through something like this, especially when it happens so fast. It seemed like 'Mad Men' sort of hit overnight and was this crazy sensation. We won the Golden Globe and I won the Golden Globe, and we won the Emmy and all this stuff happened, and you think, 'Oh my God this is a crazy swirl'. I live a pretty stable life. I mean, I have a relationship and a house and a dog, and it's kind of like, 'Well, now we are living a life that has this weirdness in it.' It is a strange shift."