Barack Obama's election win in 2008 was hailed by some as ushering in a post-racial age in the US. However, recent books and surveys have shown that black American progress has often either halted or declined.

From increasing segregation in the workplace, to hundreds of thousands of young black men in prison, to stuttering levels of black voting and a black middle class sent into reverse by the recession, the election of America's first black president – and his fight to win a second term – seem to have had little impact on any of this.

Some of the most shocking revelations are detailed in a new book called Invisible Men by sociology professor Becky Pettit from the University of Washington. Pettit realised that many surveys conducted by government agencies exclude people in the prison population from their research and findings. When Pettit added them in, she found that it dramatically altered the picture of the status of black America, as the number of black Americans in jail is disproportionately high. About half of the 2.3 million people in US prisons are black.

The results of Pettit's work, some argue, have exploded what she calls "the myth of black progress" since the civil rights era of the 1960s.

"This work dispels the notion that we live in a post-racial society. It not only deconstructs the myth of black progress, but also the myth of American progress overall," said Inimai Chettiar, a director at the Brennan Centre for Justice at New York University's school of law.

For example, adding the prison population to the voting statistics shows that black voter turnout in 2008 – believed to have been a historic high as Obama was elected – was overestimated by 13%. A greater percentage of young black high school dropouts turned out to vote in the 1980 election, when Jimmy Carter lost to Ronald Reagan, than when Obama beat John McCain in 2008.

When prisoners are included, the employment rate for young black men who have dropped out of school sinks from an already low 42% to 26%. Far from advancing over the past half-century since Martin Luther King championed the civil rights struggle, the picture being painted is one of troubled decline. "We have developed a distorted view of how black Americans are faring in our society," Pettit said. The reason given for this in Pettit's work is the high rate of incarceration of black Americans. The rate is so steep that government estimates suggest that eventually one in three of all black male adults will spend some time in prison if current trends continue.

In the 1930s, blacks were three times more likely to be incarcerated than whites, but the figure now is seven times more likely. Some experts put this down to the "war on drugs", which has affected black communities far more than others, seeing increased arrests of blacks, often for non-violent offences. "There is no evidence that drug use is dramatically different by race or ethnicity, but the pattern of arrests is very different," said Ernest Drucker, author of a recent book, A Plague of Prisons.

The recession, too, has taken a huge swipe at what gains the black middle class may have made. A swath of recent data has revealed a major reverse. The Pew Charitable Trust showed that 68% of middle-class Americans are predicted to see their economic status decline in the next generation. The National Urban League civil rights group also showed that from 2005 to 2009 the average black household's wealth fell by more than half. Nor has economic decline stopped. From 2009 to 2012, median annual household income for blacks fell by 11.1%, compared with a drop of 5.2% for whites and 4.1% for Hispanics. The current black unemployment rate of 14% is roughly double that of the white jobless. However, it is not just recent economic turmoil that has dampened black progress in America.

Another new book, Documenting Desegregation, has examined racial equality in the workplace since the 1960s and found that progress largely halted in 1980 and has gone into reverse in some industries since then. Racial segregation between white and black men is increasing in one in six industries.

In fact, far from painting a picture of black progress, the book's examination of more than five million private sector workplaces revealed that it is white men who have gained access to managerial jobs since the 1960s.

Documenting Desegregation has a grim but simple conclusion that stands in stark contrast to the general perception of a racial breakthrough that accompanied the election of Obama as the country's first black president. "The United States is no longer on the path to equal employment opportunity," it says.