On Aug. 6, during the most dogged soccer match at the London Olympics, Megan Rapinoe blasted two shots past the Canadian goalie to help Team USA secure a spot in Thursday’s final. Even more impressive, however, may have been Rapinoe’s resolve when she came out as a lesbian just weeks before the Olympics. “I feel like sports in general are still homophobic,” she said in an interview with Out.com on July 5. “People want — they need — to see that there are people like me playing soccer for the good ol’ U.S. of A.”

In the high-profile world of Olympic competition, Rapinoe is among a small but growing number of gay athletes who have publicly acknowledged their sexual orientation. According to Outsports, a media-watchdog and sports-news site, of the 14,690 athletes participating in the Olympic and Paralympic Games this year, only 23 are openly gay. That’s around 0.16%. Even so, it’s a big improvement over the 2004 Games in Athens, which counted just 11 out athletes. In Beijing in 2008 there were only 10.

Their reasons for keeping a low profile vary, but closeted Olympians share one thing in common: they have trained their entire lives to represent their countries at the Games. Coming out, they fear, could cause sponsors to pull out of deals, and negative stereotypes may leave coaches and teammates questioning their abilities. “The most important thing to every athlete is their position and standing,” says Blake Skjellerup, a gay speedskater who represented New Zealand at the 2010 Winter Olympics. “They wouldn’t want anything as trivial as their sexuality to jeopardize that.”

The organizers behind Pride House — “a welcoming space for all athletes, staff, spectators and friends” — hope to show that being gay and being competitive aren’t incompatible. To that end, they have held informal gatherings like a recent 5-km run, are staging an exhibition on gay athletes and provide a space for athletes and nonathletes alike to watch the Olympics. “We’re putting a little flag in the sand and saying that within this environment, which isn’t inclusive and welcoming, we are an inclusive and welcoming space,” says Louise Englefield, the founding director of Pride Sports, an LGBT sports-development and equality organization. “If that means that people realize there is an alternative, then great.”

The inaugural Pride House at the Vancouver Games played a big role in Skjellerup’s coming out. Although he had told his family ahead of the Olympics, he had not contemplated coming out publicly. He sat at a Starbucks opposite the house before deciding to step inside. After strolling through a photo exhibition of gay athletes — including Olympic gold medalists like Greg Louganis and Matt Mitcham — he soon found himself telling staff members his secret. “It was quite a big thing, coming out to strangers,” he says. “I felt really good with myself after doing that.”

Coming out seems more daunting for male athletes. Of the 23 out Olympians this year, only four are men. “Constructions of masculinity within sport are incredibly rigid,” says Englefield, who adds that the “macho environment” entrenches homophobia. It’s a different story for gay women, she notes: “Lesbians who maybe don’t conform to heterosexual stereotypes of femininity can just get on with it and be themselves.”

No gay athlete, closeted or out, wants to hear homophobic slurs bandied about in the locker room. And yet having to fight against more than just your opponent may partly explain the success of openly gay sportsmen and sportswomen at the Olympics. “When you’re closeted, it’s quite hard on you mentally,” says Skjellerup. “But there is a lot of mental toughness that comes with being an athlete. For me, homophobic comments actually spear me on and encourage me more.”

He may not be alone. Outsports has identified 104 out athletes who have participated in Summer Games. More than half of them have won Olympic medals. Gay men and lesbians seem poised for similar success in London. Equestrian Carl Hester, as part of the British dressage team, became the first out athlete to win gold in this Olympics. Other notables include German Judith Arndt, who bagged a silver in cycling, and American Lisa Raymond, who walked off the tennis court with a bronze. Other likely medalists include Seimone Augustus, a star of the U.S. women’s basketball team; Rapinoe of the U.S. soccer squad; and four members of Holland’s field-hockey team.

But the biggest rewards have nothing to do with medals. In the week before Karen Hultzer, a 46-year-old archer from South Africa, competed at the Olympics, journalists began to chatter online that she might be gay. Although she had always lived her life openly among friends and family, she had never publicly acknowledged that she was a lesbian. That changed on July 30, shortly after she was eliminated from archery. “I am an archer, middle-aged and a lesbian,” she said in a statement. “I am also cranky before my first cup of coffee. None of these aspects define who I am, they are simply part of me.”

Hultzer doesn’t see herself as a trailblazer. But having visited Pride House and spoken with several journalists about her sexual orientation, it’s clear she sees the benefits of speaking up. “People aren’t out, and people don’t speak openly, so people don’t realize how many gay people there are,” she tells TIME. “They’re everybody. Your waitress at the bar, your nurse, the guy who drives your train.” And thanks to Hultzer, Skjellerup, Rapinoe and others, they’re your Olympians too.