Middleton, 25, was set to steer an all-female crew in a traditional Chinese dragon boat across the 21-mile English Channel. They were to vie against an all-male crew in an record-breaking attempt scheduled for Aug. 25.

"She's upset she can't do it, but she's made a call and taken the whole team into consideration. By pulling out she's doing it for everyone else," says Sayle.

Contacted by PEOPLE, a palace spokesman refused to comment.

During their training sessions on the River Thames in the west of London, the women's crew had found themselves under the glare of photographers.

"It's dangerous enough as it is, without having to take into consideration the world's [paparazzi] causing us to capsize the whole time," says Sayle, who was set to share the helming role with Middleton. "They've been a nightmare."

Early this year Middleton's lawyers issued a warning to photographers asking that they respect her privacy.

The crew rep says it is now too late to find a new recruit.

"I will have to helm the whole thing myself," says Sayle of the journey from Dover to Cap Gris Nez, near the French port of Calais.

Meanwhile, sources close to the men's crew reveal that although the 20-strong women's team appear stoic in public, they are disappointed about Middleton's departure.

"The girls are slightly perplexed – this should have been mentioned three months ago, not two-and-a-half weeks before the race," an insider tells PEOPLE.