Just after they wed in 2014, Jolie, 41, and Pitt, 52, jetted off to the Mediterranean nation of Malta with their six children to begin filming the ’70s drama about a troubled married couple, directed and written by Jolie.

In real life, the pair’s relationship already appeared to be rocky. Those on the set of the movie tell PEOPLE that Jolie and Pitt seemed distant and unhappy.

“They hardly spoke when they were together,” a set source tells PEOPLE, adding that they often took separate business trips on the weekends.

Jolie and Pitt rented three villas for themselves, the kids and eight nannies, the source says. Due to their celebrity status, the family lived mostly in isolation while filming. The kids played in the pool, but “they couldn’t go anywhere on the island because of who they are,” says the source. Their oldest son, Maddox, 15, pitched in as a production assistant and was “a really nice guy,” the source says.

When they weren’t filming, Jolie and Pitt “were quite distant” with each other, adds the source.

During his off-time, a downbeat Pitt would go to a local bar for a light beer, never more than two.

“He would be very quiet and just keep to himself,” says an onlooker. “Angelina was never with him.”