(CNN) -- Fox News host Bill O'Reilly says his review of famed Harlem restaurant Sylvia's is being taken out of context and was not racist.

Fox News host Bill O'Reilly says his comments about a Harlem restaurant have been taken out of context.

On his September 19 radio show, O'Reilly said he took civil rights leader Al Sharpton to the Lenox Avenue fixture and "couldn't get over the fact that there was no difference between Sylvia's Restaurant and any other restaurant in New York City."

"I mean, it was exactly the same, even though it's run by blacks, primarily black patronship," he told listeners. "It was the same, and that's really what this society's all about now here in the U.S.A. There's no difference."

And later, speaking with National Public Radio correspondent and Fox analyst Juan Williams, O'Reilly said there "wasn't any kind of craziness at all" during his dinner with Sharpton: "There wasn't one person in Sylvia's who was screaming, 'M.F.-er, I want more iced tea.' "

"It was like going into an Italian restaurant in an all-white suburb in the sense that people were sitting there and they were ordering and just having fun," he continued. Watch reaction to O'Reilly's comments »

The comments, part of a broader discussion on the persistence of prejudice in American society, were circulated by Media Matters for America, a left-leaning media watchdog group that has been highly critical of O'Reilly. Other media organizations then picked up the comments.

But on his Fox News television program Tuesday night, O'Reilly said the remarks were being distorted.

"Anybody who listened to it ... would, I think, have appreciated the discussion for being an exposition of why people are afraid not only of blacks but of everybody they don't know," he said.

Elsewhere in the September 19 program, O'Reilly said he was trying to reassure whites -- such as his grandmother, whom he said had an "irrational fear" of blacks -- that rap-star stereotypes of African-Americans "are not true."

"This is what white America doesn't know, particularly people who don't have a lot of interaction with black Americans," he said. "They think the culture is dominated by Twista, Ludacris and Snoop Dogg."

"It's outrageous, because the discussion we had was about how too often the images on TV are these rappers glorifying drugs, glorifying violence, degrading women," Williams said.

But Suzan Johnson Cook, a former Clinton administration official, said O'Reilly sounded surprised that a black crowd was so civil.

"Was he expecting people to be jumping all over the place?" she said on CNN's "Out in the Open" on Tuesday. "It's very sad that, in 2007, a man who influences so many people, number one, has that kind of ignorance, who has not been exposed to our culture."

And Sharpton, who is scheduled to appear on O'Reilly's television program Wednesday night, said he has had dinner with the voluble host twice. He told CNN that O'Reilly "said nothing at dinner that time or the time before that was offensive to me."

"I don't know the whole context of it," he said. "What I've read was certainly not complimentary. And I think that if I said that -- anything halfway like that -- about other races, people would not take it as complimentary and they'd jump all over me -- probably some of his colleagues at Fox.

"But at the same time, I've been quoted out of context," he continued. "That's why I want to ask him what was the context."

Sharpton said he was already scheduled to appear on O'Reilly's program to discuss the "Jena 6," the black Louisiana teens charged with beating a white classmate amid racial tensions in a small town.

"I happen to think the future of the criminal justice system is a little more important than me and Bill O'Reilly having dinner in Harlem," he said.

In a written statement, Bill Shine, Fox's vice president for programming, added, "This is nothing more than left-wing outlets stirring up false racism accusations for ratings. It's sad."E-mail to a friend