Though Woods, 34, told ESPN in his first Q&A since the sex scandal broke in November that he and his wife, 30, are working to repair their marriage, "Nordegren is not happy" because "the trust is gone," a source tells PEOPLE.

On Friday, the former model from Sweden took the children Sam, 2, and Charlie, 1, on the yacht at Old Port Cove Marina – 150 miles away from their father.

Nordegren and the kids, along with a nanny, spent two nights aboard the triple-deck, $22 million vessel, but never ventured out to sea.