Both Elvis Costello and the Roots are what you might call serial collaborators. At an age when a lot of artists sink into comforting nostalgia – making albums that wilfully evoke the albums that made them famous in the first place – Costello seems instead to have embarked on a quest to find new partners to spark his muse: from Burt Bacharach to Allen Toussaint to Swedish mezzo-soprano Annie Sofie von Otter to octogenarian jazz pianist Marian McPartland. As for the Roots, there's a reason one online biography of their drummer/producer Ahmir "Questlove" Thompson opens with the adjective "ubiquitous". It sometimes seems like the only way an artist with a even passing interest in vintage soul can guarantee their album won't feature a guest appearance from him is by turning all the lights in the studio off, lying on the floor and pretending to be out when he turns up, perhaps after piling furniture against the door as an extra precaution. In the last few years alone, he's appeared on records by John Mayer, John Legend, Betty Wright, Al Green, Joss Stone, Duffy, Amy Winehouse, Dido, Corinne Bailey Rae, Joe Jackson, Al Jarreau, Erykah Badu, Jill Scott and Mark Ronson.

Nevertheless, a certain sense of trepidation surrounds Wise Up Ghost and Other Songs, which arrives in a sleeve that mimics the jackets of old City Lights poetry editions – most famously home to Allen Ginsberg's Howl – presumably to underscore Costello's lyrical approach: like the cover, it's rooted in writing from the past, re-editing and juxtaposing lines from old songs. Perhaps it has something to do with the last major cross-generic collaboration unleashed on the world, Lou Reed and Metallica's Lulu, an album that provides a fairly definitive answer to the question you suspect artists from differing ends of the musical spectrum ask when they decide to work together: "Well, what's the worst that can happen?" Perhaps it's down to creeping disquiet about a middle-aged white rocker teaming up with a band whose background lies in hip-hop, never the most edifying form for middle-aged white rockers to dabble in. Costello, though, has a lot of what the police would term "previous" when it comes to working in black musical genres: the New Orleans blues of his album with Toussaint, The River in Reverse, the taut, Motown-inspired miniatures of 1980's Get Happy!, the reggae rhythms that frequently underpinned his early work with the Attractions.

Curiously, it's the latter that Wise Up Ghost most regularly evokes. Although Costello offers a fairly restrained vocal performance here, never unleashing his patent contemptuous sneer – the one that sounds like his eyes are bulging out of his head with incredulous disgust – his is still an enormously distinctive voice, which means building the right scenery around it is a tough call. The sound the Roots settle on is sparse, Thompson's rolling, syncopated funk patterns offset by stabbing guitar and organ, sudden bursts of brass and a host of Jamaican-influenced sounds. Instruments suddenly swoop in and out of the mix, as on a dub album. Opener Walk Us Uptown features a stately skanking bassline; Wake Me Up strikingly counterpoints a vocal so close-miked you keep checking your ears for flecks of spittle with a guitar coated in disorientating echo.

That Costello's lyrical approach automatically invites comparison with the past is occasionally the album's undoing. Stick Out Your Tongue is a languid retooling of 1983's Pills and Soap, perhaps the most cryptic of his livid Thatcher-era state of the nation addresses: it's hard to get around the fact that the new version doesn't have the same menacing drama as the original, which is unfortunate, given that the overall message of the song appears to be that nothing has really changed in the intervening 30 years. But more often it works to considerable effect. The album's sound is frugal and full of space, but doesn't feel airy so much as eerie. There's a queasy, drugged feeling about the music on Wake Up Ghost that fits both the fragmentary lyrics and at least one of their themes. Costello has always been good at conjuring up a sense of imminent apocalypse – from 1986's Tokyo Storm Warning to The River in Reverse's Broken Promise Land – but here the chaos appears to happening at a remove from the songs' narrator. Costello frequently sounds like a man experiencing the unreal sensation of watching tumultuous events unfold on TV, aware that however distantly they're happening, they'll ultimately directly affect him. "Just because I don't speak the language doesn't mean I'm blind to the threat," he sings on Tripwire, to a Southern soul backing drifting languorously along behind a layer of distortion, "But I thought there was more to forgiveness than we conveniently forget." The title track is more densely packed with sound – woozy, spiralling orchestral samples, guitars feeding back – but the effect is the same, dreamlike and troubled, heightened by the way Costello sings in a kind of torpid murmur. He might sound half-asleep, but, for a man who announced five years ago that he couldn't really be bothered making any more albums, Elvis Costello seems as animated as ever.