WATCH: Rita Wieber Wonders at What Age Children Should Specialize in a Sport: Jordyn ‘Had to Settle into Just Gymnastics’ Early

The women on the latest edition of PEOPLE’s Mom Talk discuss that when their kids were very young, they wanted to make sure they were well-rounded in the world of athletics.

But at some point, they realized that to be top in one sport, they would have to steer them in that direction. The trouble, though, was figuring out when that should happen.

“When my kids were little, it seemed like we’d let them get involved in a number of different sports,” says Rita Wieber, whose daughter Jordyn Wieberis a former Olympian who helped her team win gold at the 2012 Summer Olympics in London.

“And then Jordyn had to settle into just gymnastics at a certain point,” Rita continues of her daughter, who recently turned 21. “Because if you want to go to a higher level, you can’t really do a lot of different things.”

For more of PEOPLE‘s Olympic coverage, pick up our collector’s edition, The Best of the Games, on sale now.

Courtesy Rita Wieber

Though the Wieber family obviously found a great balance in figuring out the right age for Jordyn to pursue strictly gymnastics, Rita admits it wasn’t easy figuring out what that timeframe should be.

“At what point do you decide, ‘They need to spend all of their time doing this particular sport because they have an incredible amount of talent?’ ” she asks.