As I sat in the heat of a Jamaican afternoon within touching distance of the Caribbean sea listening to Zadie Smith, Salman Rushdie, Ngugi Wa Thiong'o and Jamaica Kincaid read from their respective works, my mind kept returning to the phrase coined by Nigerian writer Chimamanda Ngozi Adichie, "the single story". Adichie's phrase describes the process by which entire nations, even continents, have their reality excised and a single, usually misguided or at the very least limited, image is allowed to become the dominant picture of a place and its people.

I was in Jamaica to record a BBC Radio 4 documentary about Rastafari and to perform alongside the above names at the Calabash International Literature Festival. Founded in 2001 and located in the grounds of Jake's Treasure Beach hotel, St Elizabeth, this three-day festival is the brainchild of the writers Colin Channer and Kwame Dawes, and the producer/film-maker Justine Henzell. Witnessing the audience of more than 4,000 people, and with homegrown artists such as Jah9 and Jesse Royal, it was one of the most inspiring events I've ever attended.

I am someone whose worldview was born of the African-Caribbean radical tradition. Yet I was surprised at just how well managed and well attended the festival was. Why? Could I have internalised a single story about the Caribbean, that still sees it more as a place of carnival than cognition? Or could it be that thousands of people intently listening to writers read for six hours a day, in that heat, is just genuinely a unique phenomena?

You see, beyond the blood, the beach and the banana, there is another Caribbean that – unlike its violent, virile, rum-soaked, ganja-smoking cousin – seems to be of little interest to the outside world. I could not help but notice that, despite the top-class international line-up and Jamaica's huge tourist numbers, almost the entire audience was made of people from Jamaica and its diaspora. Perhaps this only reflects the festival's location on the quieter south coast of the island. Or could it be that what Jamaicans want from their island is at odds with what most visitors demand?

Either way, the festival is clearly thriving, so it really does not matter – but being at Calabash caused me to think about how much more interesting the real Jamaica is than the tiny stereotypical slither we are usually offered.

I have seen numerous documentaries about sex tourism, violence and homophobia – and even ones revolving around the absurd suggestion that "black men are fast because of slavery". But never have I seen anything on the Caribbean that produced Walter Rodney, CLR James, Marcus Garvey and Toussaint L'Ouverture. The Caribbean whose revolutions broke the back of the "nefarious trade".

The impacts of this lopsided representation are manifold: those of us who internalise stereotypes of the lands from which we descend are likely to be racked by the kinds of psychic maladies that Frantz Fanon (another Caribbean genius) wrote about so insightfully; the nations being projected upon suffer the external stereotypes; but also those in the wider, whiter culture that is doing the projecting lose the chance to expand their own humanity by appreciating the fullness of another's.

While none of this asymmetric cultural trade-off is exclusive to a Caribbean/western world relationship, the former slave colonies, along with their African cousins, seem to me to suffer the greatest distortion of their reality. Just look at the resurgence of those "please save a starving black child" adverts.

So my challenge is twofold: for the Caribbean intellectuals and creatives to scream even louder so they can be heard above the general din; and for those of us of African-Caribbean origin (and anyone else interested in a fuller human experience) to put more pressure on our media decision-makers to reflect the wholeness of our humanity. Oh, and if you can, visit a literature festival in the Caribbean to see it all for yourself.