Multiple sources connected to Arnold and Patty tell TMZ they have spoken over the phone several times -- including the night the scandal broke -- but they have yet to meet in person ... despite recent reports.

We're told Patty hasn't been back to the Schwarzenegger home or even her own home in Bakersfield ... because of all the media attention.

But it sounds like a reunion is in the works. Our sources say, "Both families are open to a meeting, it just hasn't happened yet."