"The audio recording posted by TMZ is truly offensive and disturbing and we intend to get to the bottom of it as quickly as possible," Silver said.

The commissioner said Sterling agreed not to attend the Clippers playoff game against the Golden State Warriors on Sunday.

Silver said he hopes the investigation will be "wrapped up in the next few days."

In the audio, a man TMZ alleges is Sterling tells a woman that he doesn’t want her to “broadcast” that she is associating with black people. The comments were made after Sterling’s girlfriend posted a photo with Magic Johnson on her Instagram, TMZ reported.

“Yeah, it bothers me a lot that you want to broadcast that you’re associating with black people,” the man says in the recording.

“You can do anything, but don’t put him on an Instagram for the world to have to see,” he adds. “And don’t bring him to my games.”

On Saturday morning, Johnson took to Twitter to voice his disapproval.

"We have heard the tape on TMZ. We do not know if it is legitimate or it has been altered. We do know that the woman on the tape -- who we believe released it to TMZ -- is the defendant in a lawsuit brought by the Sterling family alleging that she embezzled more than $1.8 million, who told Mr. Sterling that she would 'get even,'" Roeser said.

The woman, V. Stiviano, is challenging the case.

NBC News has not been able to authenticate the audio-tapes.

Los Angeles Mayor Eric Garcetti released a statement condemning the comments attributed to the Clippers' owner.

"These statements are offensive and despicable and have no place in Los Angeles," Garcetti said. "I urge the NBA to act swiftly. L.A. fans deserve and demand better."

During a press conference in Malaysia, President Barack Obama responded to questions about the incident.

"The owner is reported to have said some incredibly offensive, racist statements that were published,” Obama explained to the audience. “When ignorant folks want to advertise their ignorance, you don’t really have to do anything. You just let them talk, and that’s what happened here."

The NAACP also issued a response to the allegations made against Sterling.

"If these allegations are proven true, we are extremely disappointed in Mr. Sterling," said Alice Huffman, president of the NAACP California State Conference. "Recent remarks like these, and those of Cliven Bundy, remind us that racism is not a footnote of our past, but a reality of our present that we must confront head on."

The statement went on to say that Sterling feels "terrible" about the allegations made against him.

"He is also upset and apologizes for sentiments attributed to him about Earvin Johnson. He has long considered Magic a friend and has only the utmost respect and admiration for him--both in terms of who he is and what he has achieved," the statement read.

The Clippers organization is also conducting an investigation into the allegations.

Huffman also called on the Los Angeles branch of the organization to remove Sterling from an honoree list for next month's 100th anniversary gala. A post for the event on the chapter's website noted that Sterling will be honored with a lifetime achievement award on May 15.

"If true, these are very hurtful remarks and Mr. Sterling, whose team talents and fan base is majority minority, should offer an apology to Californians," Huffman said.

It’s not the first time Sterling is being accused of racially insensitive behavior. In 2009 he agreed to pay $2.73 million to settle allegations that he discriminated against blacks, Hispanics and families with children at apartment buildings he controls in Los Angeles.

Sterling admitted no wrongdoing in the settlement. The U.S. Department of Justice said at the time it was the largest settlement of its kind.