"People are always gonna talk. But I'm in a positive place. I consider myself a grown-ass man," says the star, whose new album F.A.M.E. stands for "Forgiving All My Enemies."

Speaking of forgiveness, Brown says he's done apologizing for the brutal Rihanna attack. "At the end of the day," muses the "Forever" singer, "if I walk around apologizing to everybody, I'm gonna look like a damn fool."

A judge recently downgraded a restraining order barring contact between Brown and Rihanna, 23. (Brown is serving five years probation for the crime, and completed court-mandated domestic violence classes plus 180 hours of hard labor last year.)

In the wake of that shocking scandal, Brown says "a handful of people stuck by me," including fellow musicians Pitbull and producer Pharrell Williams.

Still, he complains that many folks "turn[ed] away. They don't want to get involved with it because they don't want their name attached to anything negative. Unknowingly, they kind of show their true colors when they do that."

He doesn't exactly fault those former friends for walking away after the assault — which he calls a "mishap."

"You can't blame people for how they want to be portrayed or if they don't want to be associated with somebody who had a particular mishap."

"The last two years, everybody dissed me," Brown says. "But my fans were so dedicated. The way I look at it is, you can't walk around mad, because then you just prove everybody right that you're an angry person."