"If I was, I would say I was," the actor told British TV show GMTV on Monday.

Butler, 39, has been recently and repeatedly linked to Aniston, with whom he's currently filming The Bounty in Manhattan.

But even though the two look cozy in photos, he says there is nothing to it. "It's just that if you're not married in Hollywood and you're working with somebody else who's single, you're having a thing with her."

Not that there's anything wrong with Aniston, 40. Butler described her to PEOPLE last month as "classy." He added to GMTV, "I'm not embarrassed by her ... Maybe she's embarrassed by me though!"

And just to add more fire to the speculation, Butler says he's looking forward to the day on The Bounty set when they get to share some kisses. "I haven't done it yet," said, "but I'm excited ... I'm lucky – yeah, I know."