While filming SWATH, Sanders, 41, seemed infatuated with Stewart, 22, but it never went any further, a source tells PEOPLE. One of the reasons could be that Sanders's wife, actress-model Liberty Ross, starred alongside Stewart in the film, playing Snow White's mother. Sanders even cast the couple's two children, Skyla, 7, and Tennyson, 5, as extras in the film's final scene.

Another source insists the hook-up between the young actress and director was far from a full-blown affair.

"She wasn't having an affair with Rupert. It was just a fleeting moment that shouldn't have happened," the source says. "She never meant to hurt anyone. She's a good person who just made a bad choice."

Dad: "Sanders Will Survive This"

Sanders's father, Michael, spoke to PEOPLE about the tryst between his son and the Twilight actress.

"From what I gather nothing really happened and it is all a lot of fuss about nothing. I should imagine it was something very brief," he said. "She is a very pretty girl and when you work that closely with someone for so long, sometimes things happen."

Michael also defended his son's character, arguing that he is far from the philander the press is making him out to be. "He is very honorable, hardworking and [a] dedicated father and filmmaker."

As for Sanders's relationship with Ross recovering from this shocking incident, Michael remains optimistic.

"Of course their relationship will survive this. This is nothing – they will definitely survive all of this. It is all a lot of fuss about nothing."

Stewart publicly apologized for the fling on Wednesday. "This momentary indiscretion has jeopardized the most important thing in my life, the person I love and respect the most, Rob. I love him, I love him, I'm so sorry," she said, referring to longtime boyfriend Robert Pattinson.