An official UN inquiry into Britain's record on tackling violence against women was prevented from investigating conditions inside Yarl's Wood, Britain's biggest, most controversial immigration detention centre for women, it has been claimed.

Campaigners are furious that Rashida Manjoo, the UN special rapporteur on violence against women, did not visit the Bedfordshire detention facility during her two-week inquiry, despite having told several groups that she was keen to inspect Yarl's Wood.

The Home Office organised her itinerary, the first UK visit of its kind by an independent expert charged by the UN human rights council to monitor violence against women. In a statement delivered at the start of her investigation, Manjoo vowed: "[I] will look at violence that is perpetrated or condoned by state authorities, and violence encountered by women facing new vulnerabilities due to the increased influx of immigrant women, asylum seekers and refugees."

Yarl's Wood, which houses up to 400 women, has been criticised for its treatment of female detainees. The recent death of a 40-year-old woman led to fresh calls for a full inquiry into conditions inside the centre.

A number of representatives from women's groups claim they were told by Manjoo that she wanted to visit Yarl's Wood – one said the UN special investigator had said she would raise the issue with the Home Office. However, it is understood that the Home Office instead arranged for her to visit Colnbrook, a facility in Berkshire primarily for men.

Manjoo began her investigation on 31 March, the day after Jamaican Christine Case, 40, died inside Yarl's Wood and during a high-profile campaign that failed to prevent teenage student Yashika Bageerathi being deported from Yarl's Wood to Mauritius two days later.

During a meeting at the start of her investigation, representatives from the End Violence Against Women (EVAW) coalition urged Manjoo to make an inspection of Yarl's Wood a priority.

Debora Singer of Asylum Aid said: "The impression she gave was that she would ask her Home Office contact if she could do that. She seemed to be trying to get an appointment at Yarl's Wood but only had an appointment at Colnbrook. It's very disappointing. One of the relatively few places where we have concerns about violence against women potentially perpetrated by people working for the state is Yarl's Wood. For her to have the opportunity to talk to women inside Yarl's Wood was very important. For that not to have happened is quite strange."

Sarah Green, campaigns manager for EVAW, said: "Manjoo is uniquely qualified to make an authoritative expert assessment for the UN of the UK's current performance in tackling violence against women. Her ability to make a well-informed assessment will be considerably compromised, however, if she is prevented from visiting Yarl's Wood, the UK's key immigration detention centre for women and a place which has sparked numerous concerns and controversies about the treatment of women and girl detainees.

"Manjoo was invited as a guest by the government and is visiting the UK in good faith. The UK would judge harshly a foreign government which declined a specific request from a special rapporteur to visit a site of concern. It does the UK's own credibility on women's human rights no good."

Natasha Walter of the charity Women for Refugee Women said they had also advised Manjoo that her trip needed to involve Yarl's Wood. "She should see at first hand the effect of the government's immigration policies on vulnerable women who seek asylum in the UK. If a visit by the UN special rapporteur on violence against women to Yarl's Wood has not been facilitated by the UK government, we need to know why. What have they got to hide?"

In a statement at the start of her UK investigation, Manjoo, a South African academic, also promised: "I will visit shelters to obtain first-hand information from individual survivors of gender-based violence."

Manjoo's trip to the UK will culminate in a press conference in London on 15 April at which she will unveil her findings and concerns before sharing them officially with the UN human rights council.

It is unclear whether Manjoo officially requested the Home Office to allow her visit Yarl's Wood. A statement from the Home Office said: "A visit to Yarl's Wood immigration removal centre was never agreed as part of this fact-finding mission. However, as part of her visit, the special rapporteur has met the home secretary. Violence against women and girls in any form is unacceptable and the government has shown its commitment to ending it."

Last autumn the UN's special rapporteur on adequate housing, Brazilian Raquel Rolnik, irked the coalition government during an unofficial visit to Britain when she demanded the abolition of the "bedroom tax".