"When it's done, then we can take a deep breath," NBA commissioner Adam Silver told CNN's Rachel Nichols in an exclusive interview Sunday.

Until then, Silver notes, Sterling still hasn't withdrawn a billion-dollar lawsuit against the NBA.

He's also wary of the litigious Sterling, whom he says he's known for decades.

"He's unsold his club several times over the years," Silver said. "There's well-noted incidents in the league when he was right there at a closing and at the last minute decided not to sell. And until he signs that document, we still have a pending litigation with him."

In his first extended interview since recorded racist remarks made by Sterling were leaked online, Silver gave Nichols a behind-the-scenes glimpse into the scandal that thrust the rookie commissioner into the spotlight just months into the job. The racist remarks sparked a media firestorm and prompted players to threaten to boycott in the midst of the playoffs if action wasn't taken.

"You deal with the cards as they're dealt," Silver told Nichols. "And we dealt with it."

Silver, who hasn't even moved into predecessor David Stern's office yet, banned Sterling from the league for lifejust days after the recordings were leaked, fined him $2.5 million and pushed through a charge to terminate all of his ownership rights in the franchise.

Silver earned near-universal praise for the swift and decisive action he took, even if the fledgling boss didn't quite grasp the magnitude of what he was doing at the time.

"I [hadn't] walked out on such a large stage, really, ever in my career with that many cameras, that many members of the media," he told Nichols about the April 29 press conference in which he announced the punitive action against Sterling. "Up until the last second, I was really writing what I was going to say."

"I don't have a good answer for that," he said. "I was at the league during that time, so I don't want to run from it."

But he speculated that the pervasiveness of social media helped catapult the latest scandal into his office.

I think in the old days it was a little bit different," he said. "Conduct that had nothing to do with what happened in the NBA ... didn't come to our attention in the same way. ... Clearly there's a different standard now in part because of social media and because there's a much greater awareness now of how that behavior can affect people and impact our league."

Still, Silver hasn't yet absorbed what his action against Sterling symbolizes for the African-American community as a whole. And he doesn't see that happening anytime soon, since he is still "working around the clock virtually on trying to end this" scandal.

Silver also addressed a non-Sterling-related scandal that he has been challenged with in his brief tenure: "Cramp-gate."