Furious residents have demanded that the chair of the public inquiry into the Grenfell Tower fire bring those responsible for the fire’s catastrophic spread to justice, with many rejecting his explanation that he has no power to bring prosecutions.

A meeting at the Notting Hill Methodist church, close to the blackened shell of the tower, saw survivors and local people vent their anger against Sir Martin Moore-Bick and his panel of advisers, who are consulting on the inquiry’s terms of reference.

Members of the public turned against one another several times in a sign of the distress and discord that the tragedy has inflicted on the community.

A string of speakers from the floor of the meeting called for criminal prosecutions to be brought against members of Kensington and Chelsea council and those who supplied cladding and other materials in the tower’s 2014 refurbishment.

Moore-Bick, who at times struggled to be heard in the raucous meeting, said: “A public inquiry is designed to find out exactly what happened. I have no power to do anything in relation to criminal responsibility.”

In response to a direct question about whether there had ever been a successful prosecution for corporate manslaughter, Moore-Bick said: “I think the answer to your question is that there have been no successful prosecutions.”

Residents voiced their fears that vital evidence was being destroyed by officials or company employees. “We’ve all heard that evidence is being shredded,” said one woman.

Some of those present asked for specific matters to be included in the terms of reference of the inquiry, such as the specifications of the cladding used or the lack of sprinklers or fire drills in the tower.

But others asked for a broader approach from the inquiry, looking at the role and management of social housing in the borough since the 1970s, or the social composition of its residents.

One woman said she had no confidence in the panel presented at the meeting, which consisted of six people, five of whom were white. “Look at the people in this room, and then look at the panel,” she said. “I have no faith in the justice system.”

Another woman said the inquiry must look at issues of discrimination: “Who sleeps safely in their beds at night, and who gets burnt to death”.

She added: “We want the inquiry to look into why these people were killed and how their rights were violated … Democracy will be judged on how it protects minorities like us.”

One person raised four specific issues the inquiry should address: building regulations, cladding and insulation, piped gas in the tower block and the role of the supervision of works. Moore-Bick said yes to each issue separately.

The consultation period for the public inquiry has been extended to 4 August. Moore-Bick expects to agree the terms of reference by the middle of August and begin gathering evidence in September. He told the meeting: “I can’t tell you how long [the inquiry] will take, but we will work as quickly as possible.”

A police investigation is also under way.

Earlier on Tuesday, lawyers supporting survivors of the Grenfell fire called for a shake-up of the way the public inquiry is being handled.

BMElawyers4Grenfell sent a letter to Moore-Bick, demanding responses to a series of questions about the selection process and criteria for Moore-Bick and the inquiry team. They say that given that so many of the victims and survivors are from black and minority ethnic backgrounds the inquiry team should reflect that diversity.

The lawyers say that the Equality Act may have been breached in the selection process.

The letter said: “There is a perception that your team is narrow in its ethnic and socio-economic background and will thus come to their task from an equally narrow viewpoint. Credibility is key to legitimacy in the public eye and is key to inspiring confidence.

“The choice of the inquiry team however can still be adjusted at this early stage however without any harm being caused.”