Hundreds of thousands of slaves freed during the American civil war died from disease and hunger after being liberated, according to a new book.

The analysis, by historian Jim Downs of Connecticut College, casts a shadow over one of the most celebrated narratives of American history, which sees the freeing of the slaves as a triumphant righting of the wrongs of a southern plantation system that kept millions of black Americans in chains.

But, as Downs shows in his book, Sick From Freedom, the reality of emancipation during the chaos of war and its bloody aftermath often fell brutally short of that positive image. Instead, freed slaves were often neglected by union soldiers or faced rampant disease, including horrific outbreaks of smallpox and cholera. Many of them simply starved to death.

After combing through obscure records, newspapers and journals Downs believes that about a quarter of the four million freed slaves either died or suffered from illness between 1862 and 1870. He writes in the book that it can be considered "the largest biological crisis of the 19th century" and yet it is one that has been little investigated by contemporary historians.

Downs believes much of that is because at the time of the civil war, which raged between 1861 and 1865 and pitted the unionist north against the confederate south, many people did not want to investigate the tragedy befalling the freed slaves. Many northerners were little more sympathetic than their southern opponents when it came to the health of the freed slaves and anti-slavery abolitionists feared the disaster would prove their critics right.

"In the 19th century people did not want to talk about it. Some did not care and abolitionists, when they saw so many freed people dying, feared that it proved true what some people said: that slaves were not able to exist on their own," Downs told the Observer.

Downs's book is full of terrible vignettes about the individual experiences of slave families who embraced their freedom from the brutal plantations on which they had been born or sold to. Many ended up in encampments called "contraband camps" that were often near union army bases. However, conditions were unsanitary and food supplies limited. Shockingly, some contraband camps were actually former slave pens, meaning newly freed people ended up being kept virtual prisoners back in the same cells that had previously held them. In many such camps disease and hunger led to countless deaths. Often the only way to leave the camp was to agree to go back to work on the very same plantations from which the slaves had recently escaped.

Treatment by union soldiers could also be brutal. Downs reconstructed the experiences of one freed slave, Joseph Miller, who had come with his wife and four children to a makeshift freed slave refugee camp within the union stronghold of Camp Nelson in Kentucky. In return for food and shelter for his family Miller joined the army. Yet union soldiers in 1864 still cleared the ex-slaves out of Camp Nelson, effectively abandoning them to scavenge in a war-ravaged and disease-ridden landscape. One of Miller's young sons quickly sickened and died. Three weeks later, his wife and another son died. Ten days after that, his daughter perished too. Finally, his last surviving child also fell terminally ill. By early 1865 Miller himself was dead. For Downs such tales are heartbreaking. "So many of these people are dying of starvation and that is such a slow death," he said.

Downs has collected numerous shocking accounts of the lives of freed slaves. He came across accounts of deplorable conditions in hospitals and refugee camps, where doctors often had racist theories about how black Americans reacted to disease. Things were so bad that one military official in Tennessee in 1865 wrote that former slaves were: "dying by scores – that sometimes 30 per day die and are carried out by wagonloads without coffins, and thrown promiscuously, like brutes, into a trench".

So bad were the health problems suffered by freed slaves, and so high the death rates, that some observers of the time even wondered if they would all die out. One white religious leader in 1863 expected black Americans to vanish. "Like his brother the Indian of the forest, he must melt away and disappear forever from the midst of us," the man wrote.

Such racial attitudes among northerners seem shocking, but Downs says they were common. Yet Downs believes that his book takes nothing away from the moral value of the emancipation.

Instead, he believes that acknowledging the terrible social cost born by the newly emancipated accentuates their heroism.

"This challenges the romantic narrative of emancipation. It was more complex and more nuanced than that. Freedom comes at a cost," Downs said.