Actually, since before we were even born, men have been restricted from wearing the clothing of women and if found would usually be punished by either hanging, stoning, or other unbelievable acts of violence. That's probably why society is so biased today, since we've been raised by the costumes of our ancestors. I believe the same went for women as well.

Though I honestly wouldn't want to see a man in a skirt (because it freaks me out) I say do what feels comfortable. I can't help but wear men's clothing sometimes. It's just so comfy.