“The fights that everyone thought we were having were fictional,” Levine told reporters at Friday’s press event for the show. “We never hated each other. We never had these secret battles that everyone thought we were having.”

Levine, 33, even goes so far as to stand up for Aguilera, 31, who has been the object of criticism for her weight. He says the naysayers should “grow up.”

“The one thing about the culture right now, celebrity culture particularly, is people feel like they can just say nasty things about other people whether it’s Christina or whether it’s me,” he says. “She gets a lot and it pisses me off. Of course I have her back and of course I defend her.”

Aguilera herself recently spoke out about her body, telling PEOPLE, “I’ve always been one to make it very clear, love me or hate me, take it or leave it, this is who I am. I embrace my body, and I embrace everything about myself.”

Levine, who says he’ll miss Aguilera on the next season of The Voice, believes that mean words about her appearance are akin to “bullying.”

“It’s not cool. I’ve never liked it. I think it’s horrible and I’ll totally defend [Christina] when people say things like that because it’s no one’s business,” he adds.