Brent Staples was that rare bird in 70s America: a young black man who was also a gilt-edged success. At 22, he was already on the flightpath to a PhD from the University of Chicago; later he would become a bigshot on the New York Times. To strangers, however, he was just another "black man – a broad six feet two inches with a beard and billowing hair . . . indistinguishable from the muggers who occasionally seeped into the area from the surrounding ghetto".

On seeing him, white people would cross the street. Couples locked arms. Women ran. Staples knew the stats about street crime but, as he wrote in a 1986 essay: "These truths are no solace against the kind of alienation that comes of being ever the suspect, a fearsome entity with whom pedestrians avoid making eye contact."

To reassure everyone that this African American meant no harm, Staples took to whistling popular classics. "Virtually everybody seems to sense that a mugger wouldn't be warbling bright, sunny selections from Vivaldi's Four Seasons. It is my equivalent of the cowbell that hikers wear when . . . they are in bear country." A top student sporting a cowbell: Staples sums up the knottiness of modern prejudice – and the damage it can do.

Racism and sexism are usually talked about as treatable social diseases. You legislate against them. You wait for economics to take its course, as (horrible phrase, this) a "black middle class" emerges and businesses spot all this untapped talent. Then arrives utopia, where everyone at least has an equal shot at success.

In which case Britain must now be that promised land. Over the last few decades, we have published enough anti-discrimination laws and HR codes of conduct to wallpaper the Taj Mahal. We have thrown money at the problem in training schemes and diversity projects. Not enough, certainly – but sufficient to make a big dent. And then there is the talent and application of all those ethnic minorities and women who would simply like to get on.

The results are disappointing. When it comes to female representation in the House of Commons, the UK lags behind Afghanistan – with one in five MPs in Westminster versus two in five in Kabul. Indian and Chinese students in the UK now outperform their white counterparts, but the jobs and wages they go on to don't reflect that.

Perhaps with time these imbalances will level out. Trouble is, we are talking a very long time: a 2008 report for the Equality and Human Rights Commission projects that at the current rate it will take another 40 general elections for the number of women MPs to match men.

Or maybe the old levers of laws and market forces aren't enough. As Brent Staples could tell you, governments can't legislate against thought bubbles. And as the economists George Akerlof and Rachel Kranton argue, their profession has completely ignored the importance of people's identities in shaping their lives. Their new book, Identity Economics, points out that their discipline teaches that discrimination should largely have disappeared by now. After all, racism and sexism are inefficient ways to choose new undergraduates or workers – and economics has a hard time getting to grips with inefficiency.

Yet those phantom stereotypes continue to haunt even those who have made it. In his forthcoming book Whistling Vivaldi (the title is a reference to Staples), the social psychologist Claude Steele shows just how we end up conforming to the cliches of ethnicity and gender. He describes an experiment in which black and white students are asked to play 10 holes of golf in a university lab. The first lot are told this is a test of their "natural athletic ability" – and the white students play shockingly badly, living up to the old white-men-can't-jump stereotype. The black subjects are completely unaffected.

The next group are told that they are being tested for "sports strategic intelligence" – and there is a complete racial reversal. As Steele writes of the black putters, "any mistake makes them feel vulnerable to being judged and treated like a less intelligent black kid". They take on average four strokes more to get through the course. And these are Princeton students, among the brightest and best of their generation; just imagine doing a similar test on school dropouts.

Psychologists term this a "stereotype threat" and it can take many forms. Merely reminding a group of Asian women of their sex before a maths test means that they do badly (women can't handle numbers, of course). Asking another group of Asian women about their ethnic background produces a better performance – since Asians are supposed to be good at maths.

"Multitasking" is how Steele describes the effect on a person labouring under a negative stereotype: he or she has to get on with the job in hand at the same time as warding off the demons that say they shouldn't be doing it in the first place. And it can affect everyone, from the female head of a university science faculty to a black graduate student in mid-70s Chicago. Steele treats class as a creation of the land of Beefeaters, but it's easy to see how his work applies to that category, too.

"You can do everything right," he says. "Study hard, work hard, come from a good family. But mentally dealing with a stereotype is another challenge altogether." Dealing with the nuts and bolts of discrimination is hard enough; shadowboxing stereotypes is far trickier.