At home, running after her 2-year-old son Phoenix, Michaels wants to make one thing clear: "I don't want fans to think I feel like I'm too big for the show or that I'm seeking greener pastures. I'm so grateful to the show and so sad to leave."

Yet, Michaels, 40, says, "there were some fundamental differences [with Loser's producers] that have existed for a while."

One main issue: how she was being portrayed in recent seasons. "In the beginning of the show it was tough love. You saw the tough, and you saw the love," she says. But more recent episodes, Michaels says, seemed to focus more on her being harsh, rather than helpful. "You saw none of the relationships, none of the bonds that I build with my clients," she says.

As a result, says the trainer, "millions of people have this warped negative perception of me." It's a perception that started to affect her personal life with partner Heidi Rhoades, son Phoenix and their daughter Lukensia, 4.

While commentators on TV shows would call her "abusive" – and one even quipped "about what kind of mom I must be," Michaels recalls with a shudder – things only got worse when, this past season, she was accused of cheating by allowing her Loser team to take caffeine pills (a move Michaels continues to stand behind).

Picking up her daughter from camp, she says someone had told her little girl, "I know who your mom is – she's a cheater. I saw it on TV."

And after last season's controversy over winner Rachel Frederickson appearing extremely thin at the finale, "I had to take a hard look at my work," Michaels says. "I came to the conclusion that moving forward, I need to be able to have an impact on the outcome of what I do.”

For more on Michaels' decision to leave Loser and her life at home now, pick up this week's issue of PEOPLE, on newsstands Friday