Conditions at Lincoln prison had deteriorated sharply, the chief inspector said, as a result of holding 50% more prisoners than it was certified to accommodate. Photograph: Lincolnshire Echo/PA

Conditions at a grossly overcrowded Victorian city centre prison are so unsafe that prisoners are too frightened to leave their wings to go to work, the chief inspector of prisons has warned.

Nick Hardwick raised such damning concerns about Lincoln prison during his inspection in August that the governor was replaced the next day.

He says in his report, published on Tuesday, that conditions at the prison had deteriorated sharply since a previous inspection in 2010 as a result of holding 50% more prisoners than it was certified to accommodate.

The report depicts a jail where assaults and bullying are a regular feature of life in impoverished and filthy conditions with drugs in easy supply.

Hardwick cites a "shocking" case where the inspectors found two foreign national prisoners who had been incarcerated for lengthy periods after the end of their sentences, one of them for nine years. "It seemed, for reasons neither man can control, a return [to their own countries] is extremely difficult if not impossible. It cannot be right that they continue to be detained for so long without the authority of the court."

He says the uncertain future of HMP Lincoln may have contributed to low morale and poor management. It faces being turned into a holding centre for asylum seekers or downgraded to a category C prison.

"The prison was not safe … A third of prisoners told us they had been victimised by other prisoners. The number of reported fights and assaults was high and there were evident opportunities for bullying," says the report. "There was little attempt to investigate either individual incidents or patterns of violence. Prisoners who were too frightened to leave their wings to go to work but who refused to name the perpetrator were punished by being sent to the segregation unit."

The report adds that although the vulnerable prisoner unit was mainly for sex offenders, inmates from the main wings also sought sanctuary there. The external and communal areas of the prison were dirty and some areas did not appear to have been cleaned for some time.

The inspectors found that up to half of the prisoners were locked in their cells during the working part of the day and a new academy for work and education was almost empty when they visited. "The staff were unable to explain why this was so," the report said.

Hardwick said Lincoln's strengths, such as good staff and prisoner relationships, were undermined by a serious lack of professionalism in many areas that compromised safety and the smooth running of the prison. He said changes were needed urgently.

Michael Spur, chief executive officer of the National Offender Management Service, said the decline in the prison's performance was not acceptable: "We have taken urgent action to address the chief inspector's concerns. The governor has been replaced and measures are being implemented at pace to improve safety, decency and regime provision across the prison."

The Howard League for Penal Reform said the inspection report was one of the worst they had seen: "If you want an example of the terrible consequences of overcrowding in our prison system, you need look no further than Lincoln," said the League's Andrew Neilson.

"This cramped and dirty prison holds 50% more people than it's meant to, with 20% fewer staff in post than there were three years ago. Its shocking levels of violence and bullying are a symptom of this."