Also a retired New York City firefighter, Brown says he lost 93 colleagues that day and like so many others, is deeply offended with Rauf's efforts to build a mosque and cultural center near ground zero.

"We find that it's extremely insensitive to the families and to those of us who lost so much on that day," Brown said.

The protestors clashed with others who support the Imam.

Rauf spoke about his international upbringing and naturalized U.S. citizenship. He believes Muslims share the same values as Americans.

"Our founding fathers established a country based upon the concept of religious freedom which is something that's fundamental," Rauf said. "Islam fundamentally demands that."

That message connected with some who traveled a good distance to listen.

"I'm recently married to an Egyptian Muslim and I'm Christian, so a lot of what he was to say resonates with me," said Lucretia King, attendee from Greenville, N.C.

Imam Faul received $20,000 for his appearance at UNC, but the school says the fee came from private donations. Faul will speak at Duke University Thursday.