The actor, 37, and pop star, 20, allegedly got into an altercation at Cipriani restaurant in Ibiza, Spain, Wednesday night, when Bloom reportedly took a swing at Bieber after the men exchanged words, a source tells PEOPLE.

"Orlando took one swing at Justin and missed," the source says. "They were separated and Justin stayed at Cipriani with friends for a while before leaving."

According to the New York Post, the two men were seated at different tables, but when Bieber and his entourage walked past Bloom's table, the actor supposedly refused to shake the pop star's hand.

The alleged dis didn't sit well with Bieber, the Post reported of the exchange, part of which was caught on video. The source claimed that "Bieber said something rude to Orlando, like 'She was good' " – a possible allusion to Bloom's ex-wife, Miranda Kerr, perhaps? (Another source told the paper no such comment was made.)

Bloom and Kerr, 31 – who have a 3-year-old son together, Flynn – split in 2013, but Bieber was reportedly seen getting chummy with the model at a 2012 Victoria's Secret fashion show, fueling rumors of tension between Bieber and Bloom.

Bloom, meanwhile, spent time with Bieber's ex-girlfriend Selena Gomez, 22, back in April, when the two were spotted together outside a Chelsea Handler show in Los Angeles.