Next week, at the international black film festival in Nashville, Bill Duke and D Channsin Berry will premiere their new documentary, Dark Girls. The film looks at the everyday experiences of dark-skinned black women in America. The blurb from the official site promises the directors will "[pull] back our country's curtain to reveal that the deep-seated biases and hatreds of racism – within and outside of the black American culture – remain bitterly entrenched".

When the film-makers released a preview of Dark Girls in May, it spread like wildfire across social media sites and black entertainment blogs. Commenters wrote about being moved to tears by the nine minutes of film they'd seen and many mentioned how long in coming such a film was. Why did the documentarians decide to tackle this subject and why now? For Duke, a veteran of Hollywood – co-star of Car Wash and Predator – it was down to personal experience. "It came from me being a dark-skinned black man in America, and also observing what [dark-skinned] relatives like my sister and niece have gone through. The issue exists externally of our race, but a lot of it comes within the race itself and our perception of ourselves." Berry recalls being called "darkie" at elementary school by his fellow classmates, "and even some family members were like: 'He is really dark. Why is he so dark?' It left a scar. So when Bill came to me, within the first couple of seconds, I was on board."

I am a dark-skinned girl. I always have been – I was never fair-skinned, not as a baby (like my sister), not as a child (like one of my brothers) nor an adolescent. My parents did not wait for my colour to "come in". I was born a deep brown, and have pretty much remained so all my life. My extended family is pretty diverse-looking – from my second cousin Ruka, who looks white in certain lights, to my cousin Baraka, who is dark as night; I never had any real inclination to be lighter-skinned, but almost every Nigerian Briton I spoke to while writing this article reported having seen bad bleach jobs at weddings, church and parties.

Growing up in Nigeria during the 90s, I remember being offered a soft drink, and my hostess jokingly telling me to choose something other than a Coke because it "would make me darker". Even being a fairly confident and logical child, and despite understanding that a drink had no effect on my complexion, I changed my mind. During a decade there, it would not be the only time I would alter my drink order.

The women in Dark Girls discuss the role melanin has played in their lives. One woman recalls asking her mother to add bleach to her bathwater so she "could escape the feelings that I had about not being as beautiful, as lovable". Another says: "It was so damaging, it made us feel like we were 'less than'." The preview also shows a clip from a 2010 pilot study in which schoolchildren were asked to select from pictures of dolls ranging from light to dark. The researcher asks a five-year-old black girl to show her the smart child. The girl points to the image of the lightest child. She does the same when the researcher asks her to pick the good-looking child. Her reasons are "because she's white" and "because she's light-skinned". By contrast, she selects the darkest child when asked to pick out the "ugly" child and the "dumb" child. This time, her reason is "'cause she's black."

It is an update on the doll experiments carried out by African-American psychologists Kenneth and Mamie Clark in the 1940s, which sought to find out children's self-perception as related to race. They concluded that black children had internalised the racism caused by discrimination and segregation. "Our goal is to take that little girl's black finger off the picture that looks like her," says Duke. "If we can do that, maybe we will impact things, because the truth is that all of the dolls in that picture are beautiful."

The origins of colourism are widely believed to be in the "pigmentocracy" of slavery. Ruth Fisher, project manager of the Understanding Slavery Initiative, says: "Generally speaking on plantations, you had what you would call the house slaves and the field slaves. The delineation of shade in that regard would be those who were darker would be in the fields while those who were fairer or of mixed heritage would be the house slaves. Part of it was because of the fear factor; those who were more closely associated with being African or those who were new to the plantation would be darker and more resistant than those who were born on the plantation and therefore considered to be less aggressive, less rowdy.

"That started a divide within the African community on the plantation, because then those who were closer to the house had some of the less back-breaking work and therefore they felt that they were a bit more privileged."

Heidi Safia Mirza, professor of equalities studies in education at the Institute of Education, University of London, says: "Pigmentocracy in the Caribbean as a kind of social hierarchical system emulated from the slave days where there was favouritism if you were fairer, particularly if you were a woman." Mirza, who has been conducting her own research looking at young black and minority ethnic women in schools, tells the story of a Sierra Leonean teenager who reported being made fun of because of her very dark skin. "It was not uncommon for dark-skinned girls to be vilified and teased and called names like 'blick', which means 'blacker than black'."

Debbie Weekes-Bernard, senior research and policy analyst for education at the Runnymede Trust, wrote Shades of Darkness, a report on the way "darker-skinned girls reflect upon themselves against lighter-skinned (in this case mixed-parentage) girls" as part of her PhD. The subjects were girls between the ages of 12 and 16.

"The thing that struck me the most is that there were things at work societally, which place all women, but certainly black women, on a hierarchy of beauty," she says. "And the hierarchy of beauty for black women is different from the hierarchy for white women. For white women, it's about size and shape [thinness] but for black women it's all of those things, but also the shape of one's nose and lips, the texture of your hair and all those other things which are bound up within how 'womanly' or not you look. The issue, then, is that we have people being quite essentialist and saying you can only be really, truly black if you are darker skinned, compared with other lighter-skinned women who say they aren't considered to be truly black because they're lighter." She concludes: "There were darker-skinned girls who felt they were policing what it meant to be black; policing the boundaries of blackness, because they're tired of other people doing it for them."

Simone Bresi-Ando, founder of I'mPossible, a social enterprise for women of colour in the UK, thinks the film Dark Girls is "important and necessary" and also believes "it's so important that we start looking above and beyond tones and hairstyles and colour. We really have to be focusing on things that have trapped a whole race of people for so long. It's time to push it on."

It's a stance shared by Baroness Lola Young. "I have to voice disappointment that people still feel that they can't shake it off," she says. "Last month, yet another young black man got stabbed in London. Then there's Damilola Taylor. And all of that is going on and there's people worrying what shade of black they are and going out and buying bleaching creams from shops." She also has little time for the argument about the part the media has to play in the issue. "In my view, it's not good enough to keep saying, 'Oh, we're flooded with images of light-skinned black women in the media and therefore that makes people want to be light-skinned because they think it's something that is acceptable.' Because if you said that, then what about all the years in which we waited and never had any black people on television at all? Did that mean we wanted to obliterate ourselves totally in order to conform to a particular world view? It just doesn't make any sense to me at all. People need to get a grip." She continues: "How can we argue against Satoshi Kanazawa [an evolutionary psychologist who claimed black women are "objectively less attractive" than women of other races] when we're saying the same thing? It's racism in that case, but what is it when black people say it? Crazy."

Shadism and colourism have continued to flourish even in mono-ethnic societies. Last July, Vaseline launched a skin-whitening app on Facebook in India, enabling users to make their skin whiter in their profile pictures. The app was designed to promote the brands's range of skin-lightening creams for men, a fast-growing market on the subcontinent. In 2010 Jamaican dancehall star Vybz Kartel came under fire after lightening his skin. He said: "I feel comfortable with black people lightening their skin. It's tantamount to white people getting a suntan." In a statement to Vibe.com, he defended his use of "cake soap", saying: "When black women stop straightening their hair and wearing wigs and weaves, when white women stop getting lip and butt injections and implants … then I'll stop using the 'cake soap' and we'll all live naturally ever after." This month he will launch his range of 'skin brightening' products including moisturiser, soaps and fragrances in the Caribbean. He has high hopes for his range: "I wanna see them in Macy's and all other fine retailers worldwide," he told the Tribune newspaper.

At one point in Dark Girls, a young black man states his preference for "light skin, pretty girl, long hair", something Weekes-Bernard heard "frequently" when she was talking to her subjects. "There is an implicit assumption that there are black boys out there who only want to date either white or lighter-skinned women," she says. "Young dark-skinned girls are still facing this horrible burden," Duke says. He talks of a rapper who "literally stated in the casting announcement [for his music video] that no dark-skinned women need apply for the audition".

"This is this year!" he laughs incredulously. "We're not suggesting on any level that all black men are only attracted to light-skinned black women but we would be liars were we not to say that the predominant standard of beauty when black men look at women, to a great extent, is light-skinned, so-called 'good' hair and fair eyes.

"If you take a look at some of our celebrities – let's take the sports world for a moment – and look at some of the choices these gentlemen have made in terms of their girlfriends and mates, I think one would be hard-pressed to find a woman of dark complexion," says Berry. "I think they buy into [the idea that] once you have the money, you get a status symbol. And she doesn't look like your mother."

Duke thinks it is of "enormous importance" that Michelle Obama, the wife of the world's most powerful man, is dark-skinned. "I don't think he'd be president of the United States if [Michelle] was a light-skinned black woman," he says. "There are a lot of black women who would not have voted for him because the implication would've been that women of a dark hue are not acceptable to him, so why does he deserve their vote? We don't talk about it very much but it runs very, very deep. You're expected, if you're a successful man, to have a light-skinned trophy on your arm. With a dark woman on your arm, it means, for whatever reason, and this sounds horrible, you had to settle. I mean, this thing of colour in our culture is deep."

Mirza points to the commercialisation of "darkness". "Now, being dark can be appropriated and turned in on itself and turned into a 'style'. Consumption and commercialisation has come in – it sells records, cosmetics, and has become a vehicle for capitalism. But it is still entrenched in racist meaning. Nowadays it may be less about social mobility and more about desirability." Pigmentocracy still exists, she says, only the forms of mobility have changed. "It's about celebrity now, being famous and beautiful and how that's defined is to be thin and white, and fair and black. People are caught up in it 100%. They used to call it false consciousness. You could call it that, but in a sense, it's about presentation and identity. How you see yourself is through representation – how the world represents you. You want what you are shown, what is presented and promoted as privileged."

The producers of Dark Girls hope their film will have the cross-continental appeal of Chris Rock's 2009 documentary, Good Hair.

"We had more than 725,000 hits to the preview in 28 days," says Berry. "And we had more hits from France and Germany and the Netherlands than from South Africa and Jamaica. It tells us what the world wants." He and Berry hope that their film will start a healing, something that, according to Berry, needs to start at home. "Reinforce that your child is beautiful. Don't only tell them, show them," he says.

Bresi-Ando says: "If we want to get to the next level, for other people to respect us as well, we have to respect and love ourselves. If you live your life making choices based around not liking yourself in the full glory of what you can be, why would anybody respect you?"