Angelina Jolie shares Easter eggs with Bosnian women on a visit as a goodwill ambassador for the United Nations High Commissioner for Refugees last April. Photograph: UNHCR via AP

A pack of dogs is basking in the sun in the old Jewish cemetery on the hill overlooking the district of Grbavica in Sarajevo. During the siege of Sarajevo in the Bosnian war, the Serbs placed their guns up here to fire into the city. Fifteen years after the war's end, this scruffy neighbourhood has become the centre of a new conflict.

It is thought to be one of the locations where Angelina Jolie would like to direct her debut film, dealing in part with the experience of a Muslim woman who was a victim of the notorious rape camps. The film has provoked a bitter battle over who has the right to interpret one of the conflict's dark episodes – and how. The dispute has even split groups that speak for rape survivors.

What started as a vague rumour that Jolie intended to depict a love affair between a Serb rapist and his Muslim victim has come to represent much more: a fierce debate over the political and social influence of war victims' groups in a still troubled society.

It has seen the country's culture minister, Gavrilo Grahovac, withdraw Jolie's permission to film before being forced a few days later into an embarrassing climbdown.

The scandal has dominated TV news and ordinary conversation – with many backing Jolie. On Thursday the country's leading political weekly magazine, Dani, dedicated 16 pages to the affair, with a picture of Jolie above the acid-eaten words "Welcome to Sarajevo".

Bakira Hasecic, a rape victim and the head of the Association of Women Victims of War, is one of those with the strongest objections to Jolie's film. Her voice – Hasecic's critics say – has had undue influence in Bosnian politics. She argues that any depiction of a relationship between a Serb rapist and his victim would be offensive.

Not all rape victims support her. Enisa Salcinovic, who was raped in a camp in Foca, split with Bakira in 2006 and now heads the women's section of the Association of Concentration Camp Victims. "Fifty per cent of the victims who called me after the row over the film escalated told me they do not support Bakira," she says.

While her group's members would also like to be reassured about the contents of the film's script, she insists they are deeply unhappy about Hasecic's aggressive tactics and says she should not "talk in all our name".

"Angelina Jolie went to meet women victims in Gorazde [a city in the east of the country]," she adds. "I don't believe she would want to hurt the victims of the war." She accuses Hasecic of using her position to advance her own political interests, not least her close connections with Bosnian nationalist politicians. What is certainly true is that neither the culture minister, who at first withdrew the filming permit, nor Hasecic, had seen the script before mounting their protests.

The row has also prompted sharp reactions among other Bosnians with some – including writers for Dani – breaking a long-existing taboo against criticising war victims' organisations and their influence in Bosnian society.

Among those who felt impelled to speak out is Belma Becirbasic, a senior editor and writer who had carried out academic research on the Bosnian rape camps, who believes the controversy reflects a far deeper social malaise in a society still struggling with its demons 15 years on and that is undergoing an increasing radicalisation, producing leaders who have exploited war victims for their own ends.

"Behind the story of Angelina Jolie and her film," she said, "is the inability of a society and culture to go forward and put the war behind us. This is the first time I've taken the course of writing about the real Bakira and the other side of the war victims' organisations.

"Because if you write about that other side, how they exploit public discourse, you offend them. What has happened around this film has been an excuse. Next week there will be something else for them to take issue with. And in Bakira's case what has happened is that she has monopolised the discussion of Bosnia's raped."

The clumsy handling of permission for Jolie to film has also incensed members of Bosnia's small but vibrant film industry, among them Nenad Dizdarevic, a director and dean of Sarajevo's Film Academy whose film, The Awkward Age, was premiered in Sarajevo in 1994 in the midst of the siege. He insists that films dealing with "problematic themes" are "the most interesting", pointing to Charlotte Rampling and Dirk Bogarde in The Night Porter, about a masochistic love affair between a former Nazi and a concentration camp victim, as a "small masterpiece".

"What has happened has been a crime against Bosnia-Herzegovina and its film industry," says Dizdarevic, who was one of the first to go on television – at his own insistence, he says – to defend the film project. "I am ashamed of what has happened and the potential damage it has done to film-making here."

Bakira Hasecic is in her association's tiny office on the outskirts of Sarajevo on the ground floor of an apartment block. Members of her group are gathered for a meeting about Jolie's film, still deeply unhappy about the prospect of filming without further details on the script.

The walls are plastered with press cuttings and posters and political photos given to her by her high-placed supporters. The "Heroine of Visegrad", the town in eastern Bosnia whose Muslims suffered terrible depredations at the hands of the Serbs, is a wiry and intense woman. She admits quickly that she still has not seen the script. Her objections she says were based on internet rumours. And that rumour of a love affair between rapist and raped is what the women cling to. Other things rankle, including the fact that, despite letters and requests, Jolie has not spoken to her since the row first erupted at the end of August.

But there is something more as I listen to these women's stories. It is repeated again and again: an insistent belief that because no one can "understand what they have been through" outsiders have no right to describe it.

Bakira admits they considered making their own film to tell the "real stories" but could get no funding. At one point she encourages the other women to speak "so it is not just Bakira, Bakira, Bakira". But another woman, giving us coffee, admits that they are "all Bakira". Then she plays a phone message from a member in Bihac. It is short and moving – albeit referring to a plot line that may not even exist. "If you meet Angelina Jolie," the voice says, "ask if she would fall in love with her rapist."

With filming due to begin in a few weeks, I ask what is the next step?

"We want Angelina Jolie to come here to us and assure us that she is not going to offend us." And if she doesn't come? "We should make a film about her life," Bakira says grimly.

I ask if it is possible to make a feature film about the Bosnian rape camps that would feel appropriate to their still caustic pain. "Not impossible," answers one of the women.