Channel 4's Benefits Street has been branded "an old Victorian freak show" by a Birmingham community leader after the broadcaster declined to apologise to residents featured in the controversial documentary series.

The Rev Steve Chalke was speaking after Channel 4 issued a statement saying it was proud of the "fair and balanced observational documentary". The broadcaster added that it was confident members of the community around Birmingham's James Turner Street would want to take part in a live debate show to be broadcast after the series ends next month.

He responded after Channel 4 said on Friday that it would not be apologising by saying the broadcaster owed it to the community around James Turner Street, in Birmingham's Winson Green, "to allow them to speak on their own terms".

Chalke said: "Benefits Street is like an old Victorian freak show. Everyone that took part in it thinks it's ridiculous. This debate doesn't count as a right of reply. No one wants to take part in anything."

However, Nick Mirsky, Channel 4 head of documentaries, said the broadcaster knew that Chalke's views "do not represent the opinions of all the residents".

Mirsky added: "This was apparent in reports from the residents' meeting Steve refers to – no consensus view was reached. The team from Love Productions, who have been on James Turner Street for approaching two years, are still in place and have daily contact with the contributors. Many are supportive of the series because they show strong friendships and community spirit during difficult times."

Benefits Street has been dogged by controversy, with critics labelling it "poverty porn", and complaints from people who claimed they were tricked into taking part. It has polarised opinion between those who believed it victimised people on benefits and those who claimed it highlighted the flaws in the welfare state.

However, the five-part series is also Channel 4's biggest hit since its 2012 Paralympics coverage, with last Monday's second episode focusing on immigration watched by more than 5 million viewers.

The live 60-minute studio debate, to be made by Question Time producer Mentorn, is scheduled for Monday 10 February, immediately after the last episode.

A Channel 4 spokeswoman said: "We are proud of the programme as being a fair and balanced observational documentary series." It is a sympathetic, humane and objective portrayal of how people are coping with continuing austerity and cuts in benefits, and accurately reflects the reality of life on a street where the majority of households depend on benefits."

Channel 4 was "confident that there are members of the community who would want to engage in the debate", the spokeswoman added.

The controversy surrounding the documentary has drawn the media and even sightseers to James Turner Street.

Chalke claimed some contributors would not speak out "because there are threats on the street". He added: "There are cars driving up and down the street throwing things at the houses."

He organised the public meeting for residents on Wednesday night, which was attended by about 100 people, initially saying the proposed TV debate would give them the opportunity to have their say.

Another local community leader, Birmingham city councillor Chaman Lal, said: "I know people are very angry with Channel 4 and I don't think they want to have any dealings with them again.

"The general consensus appears to be that they would like a live public debate to air their views but with the BBC and not Channel 4."