"I have a huge crush on Blake Lively. She's drop-dead gorgeous," Lochte, 28, told PEOPLE at New York's Ainsworth Park at the Windsor Custom Clothes Fashion's Night Out event. "I haven't met her yet. If I do, I'd definitely be starstruck. She makes my toes curl."

The Olympic gold medalist, who is on a strict diet and trains about six hours a day during competitions, admits he'd break his healthy regimen for Lively, who has a passion for baking sweets and has been linked to Reynolds since Oct. 2011.

"For her, I would totally cheat [on my diet]," he says. "I love homemade food, but I do have to keep to my regimen because I'm still training for the 2016 Olympics."

When it comes to style and clothes, the swimming champ is attracted to women who dress similarly to Lively – a clean look with an edge.

"I like girls who are confident enough to wear sexy dresses and have a wild night out and [then] be the complete opposite and be able to chill and wear shorts and a football jersey around the house," Lochte says. "A woman wearing an oversized jersey – or especially a guy's button-down shirt with no pants – is by far the sexiest thing."

As for his own style, "I'm a jeans and a T-shirt guy," he says, "and I like to accessorize by wearing a watch or necklace or I'll wear a sports jacket with some jeans. I don't wear suits a lot. I feel more comfortable walking around in a Speedo than being in a suit."

And what does Lochte prefer to wear to bed? "I'm naked or in boxers," he said with a sly grin. "I don't like to wear too much clothing when going to bed. I like to feel free."