With all the discussion, debate and conjecture surrounding plans to construct a $100m Islamic cultural centre in lower Manhattan – benignly named Park51 after its street address but controversially dubbed the "Ground Zero Mosque" by opponents – the focus of attention and criticism has landed squarely on the project's chief proponent, Imam Faisal Abdul Rauf.

As founder and chairman of the Cordoba Initiative, the organisation behind the plans for Park51, Imam Rauf proposed the centre as a "platform for multi-faith dialogue" promoting "inter-community peace", a place for healing which includes a "9/11 victims memorial". Despite purporting to speak as a voice of moderate Islam, Imam Rauf's words ring hollow to many and his efforts have struck a raw nerve nationwide, with 68% of Americans and 71% of New Yorkers polled opposing the project – amid a controversy that caused even President Barack Obama to lend his voice to the dialogue.

At a White House dinner in honour of Ramadan on 13 August 2010, President Obama said "Muslims have the same right to practise their religion as everyone else in this country", including the right to "build a place of worship and a community centre on private property in Lower Manhattan". Interpreted as tacit approval of the "Ground Zero Mosque", many decried the comment as insensitive to 9/11 victims. In follow-up remarks, the president doubted "the wisdom of making a decision to put a mosque there", indicating some measure of agreement with the position of civil rights groups such as the Anti-Defamation League whose national director Abe Foxman had previously said that building a mosque in the shadow of Ground Zero "is not a question of rights, but a question of what is right".

At one end of the spectrum, some say Park51 is an Islamic provocation and a front for "interests tied to terrorism", though no proof for that allegation has come forward. In a recent Guardian article about the implications of Park51's architecture, Ed Pilkington reported that Park51's design plans are not "militantly hardcore", are "more festive than threatening", and, therefore, "designed to be multi-faith and also secular".

To sceptics, the semiotics of the centre's design will hardly be a more convincing argument than the sound constitutional claim in support of the centre. While recognising the right of Muslims to practise their faith under the US constitution's first amendment, which proscribes government from "prohibiting the free exercise" of religion, Park51's opponents maintain that proximity of an Islamic cultural centre to Ground Zero is insensitive to victims because Islamic extremists perpetrated the 9/11 attacks.

But this argument, in its simplest form, is not very strong either. For 27 years, Imam Rauf has been the spiritual leader of the Masjid al-Farah congregation in lower Manhattan, situated 12 blocks from Ground Zero. Although Park51 is 10 blocks closer, Pilkington did rightly observe that, from the criticism it has received, one would think Park51 will be built as "an Islamic citadel right on top of Ground Zero", which is clearly not the case.

Yet, there remains something about Imam Rauf's presentation that feels disingenuous, making people uncomfortable but unable to articulate why, forcing them back on the argument that proximity to Ground Zero is "insensitive". Timothy Garton Ash's recent piece in the Guardian articulated some other denunciations of Park 51, including Newt Gingrich's comparison that "Nazis don't have the right to put up a sign next to the Holocaust Museum in Washington."

Gingrich's emotive plea is not, in fact, entirely correct. In 1977, the United States supreme court affirmed the first amendment right of neo-Nazis to demonstrate peacefully in a neighbourhood substantially occupied by Holocaust survivors (National Socialist Party of America v Village of Skokie, 432 US 43). So, also, even if it is personally offensive to some, Muslims have a constitutional right to build an Islamic centre in a location of their choosing, assuming it is not otherwise prohibited.

It is clear that a neo-Nazi demonstration in the backyard of Holocaust victims is outrageous and intended to offend, but since Islamic provocation in this instance cannot be proven and simple proximity to Ground Zero is not the most persuasive argument, what is it about building Park51 that strikes people as so unwise, not right, or insensitive?

This is where the dispassionate analyses of Pilkington, who focuses on the festivity of the architecture, and Garton Ash, who broadly claims that there is "no reasonable objection" to the Islamic cultural centre, fall short. They do not present an alternative rationale, which is that, regardless of where the centre is built, opposition may arguably be derived from a certain latent nuance in Imam Rauf's language that demonstrates an imperviousness to the hurt that New Yorkers still feel. While he professes to be a bridge-builder who is using Park51 to promote inter-community peace and healing, his lack of sensitivity is exhibited for many in a certain evasiveness over Islamic culpability in the 9/11 attacks. In short, he has never conceded that Muslims committed them.

In response to the question posed on his Cordoba Initiative website – "Isn't the location insensitive given that the 9/11 attackers were Muslims?" – the response recognises "legitimate concerns and sensitivity" to Park51, but it also said the terrorism "was cloaked in the guise of Islam" and that "extremists who profess to be Muslim perpetrated murder… in the name of Islam". This clearly intimates that the extremists were, in fact, not true Muslims or even Muslims at all. For the answer to really respond to the question, it should say something to the effect that it recognises that Muslims perpetrated the 9/11 attacks and that is why the Park51 proponents understand the sensitivity to the centre's location.

In his many public remarks, Imam Rauf's words seem conciliatory, yet on closer examination, he seems to say nothing substantive. In an interview he gave to CNN's Soledad O'Brien on 8 September 2010, Imam Rauf was again equivocal on terrorism. O'Brien referred to a previous interview in which Rauf was asked whether the state department was correct in designating Hamas a terrorist group. O'Brien accused Rauf of having ducked the issue – that he "went on a long time… but there was really sort of no answer to it". So she posed the question to him again, offering him the opportunity to clarify his view.

Imam Rauf answered, "I condemn everyone and anyone who commits acts of terrorism. And Hamas has committed acts of terrorism." At first glance, that appears as though he answered the question, but re-read, Rauf does not actually allow that Hamas is a terrorist organisation. This is exactly the kind of equivocation that creates confusion and makes people distrustful of his claims of moderation.

There are certainly those who would, out of prejudice, dismiss any gesture by Imam Rauf and the Muslim community in New York. What most New Yorkers, and most Americans, are looking for, though, is an absolutely unambiguous acknowledgment by moderate Muslim leaders, specifically Imam Rauf, stating that Muslims did, indeed, perpetrate the terrorist attacks of 11 September 2001 in the name of Islam, even if it is a version of Islam they themselves abhor.

One may think this has already been said, but in Rauf's case, the statements on the subject are not unambiguous. This may seem like nothing more than a matter of parsing words, but words matter here; and actions even more so. No matter where it is located, near Ground Zero or elsewhere, Park51 will never be able to realise its ambition as a place of conciliation until Imam Rauf clears up any doubt about his position regarding Islamist terrorism, affirms the Muslim identity of the 9/11 perpetrators, and takes the proper steps to address the legitimate concerns of New Yorkers.