Sticks and stones... graffiti obliterates the French words from a bilingual Brussels signpost. Photograph: Mark Renders

Words don't do physical damage, but they can be real live wires. The London Word Festival launch had a darn good go at hotwiring the lexicon after Tom Chivers and company invited the crowd to stick their neologisms to the wall. My favourites included 'vomage' (physical homage to a night on the lash) and 'frosy' (those very cold, very sunny days we love).

But words are not about individual creativity, however witty or wicked. We don't carry an arsenal of our own words around with us, ready to trip off the tongue, Babel-style. Words are meaningless if they only have meaning for us.

We make language between us, and the hammering-out of what words mean doesn't happen in a vacuum. Movements in politics, society and technology, shifts in how we see ourselves - all these changes have a bearing on the words we use. We agree on the meanings of words from moment to moment, and because society changes, the meanings of words evolve. As Chris Powers' recent blog on etymythology illustrated, the origins of words can be rooted in the remote past, and lexical origins can take on a mythology of their own. Words can up-anchor from what they once meant. Old etymologies can slide down the back of the sofa to re-emerge years later. Today, recovered etymologies are a political minefield, ready to blow up in our faces.

Even when they share a common language, different people speak differently, peppering their sentences with argot, slang and patois. "You look like the wreck of the Hesperus" was a scolding phrase my Irish mum used constantly when we were kids. In 1970s north London, far removed from any knowledge of Longfellow, a snatch of poetry was somehow being given idiomatic oxygen. Some phrases have far more dubious origins. When I need to get things done, I have a habit of furrowing my brow and declaring, "Right, let's get down to the nitty-gritty," and I was horrified to find that the phrase allegedly has its roots in the slave trade. For me it's the same as saying "let's get down to the nuts and bolts", and I've never used or heard it used in a racially abusive sense. So should I carry on comfortably trotting it out? Does it matter any more what I am trying to say when I speak those words? This is not as straightforward as you might think. Identity politics matter, and identity politics are often played out in language.

I have been giving serious thought to these things in advance of two events next week. I'm chairing a Bath Literature Festival debate on identity politics and, like any good chair, I've been mugging up on the speakers. Julian Baggini's Welcome to Everytown has a striking passage where he describes the casual use of "Paki" in Rotherham. He gradually comes to the conclusion that Rotherham's use of a word many a liberal would balk at is descriptive rather than racist, and its use by Rotherham's white population is "not primarily a symptom of race hatred but of a divided nation". It is a point worth dwelling on. If 'who we are' rather than 'what we do' now defines our public selves, where does that leave the words we use? Is our common language under threat, with certain words ring-fenced and segregated, meaning and offending differently depending on who uses the word? Even within a community, words can be a battleground. The rehabilitation of 'Nigga' by the hip-hop generation continues to infuriate older activists from the civil rights era. In the UK, too, the n-word irks.

When we start worrying about words, we have failed politically. Becoming over-anxious about 'Paki', 'nigga' or 'nitty-gritty' won't change the world or the ideas that these words were formulated to express. It's only by allowing people to say what they truly think - using words we find offensive if they choose - that we can have the argument, and begin to change the ideas that shape our words and, ultimately, our world. This is why Tuesday night's Manifesto Club matters. It brings MPs, journalists and rap artists together to argue for unbridled free speech. I, for one, am hungry for neologisms that come out of real politics: words that do not focus on who we are, but ask us what we are going to do.