For a moment, let us banish the nagging suspicion that the world may house people more deserving of our sympathy than a happily married knight of the realm, globally acknowledged as a peerless genius and with a rumoured personal fortune of £762m, and spare a thought for Paul McCartney. Despite his reputation for irrepressible chirpiness - the man who, for a generation of 1980s Smash Hits readers, will always be Fab Macca Wacky Thumbs Aloft - it can't be easy being him at 63. Your best work was completed four decades ago. The hits have dried up (his last top 10 single was in 1987). Every new effort is greeted with little more than a yawn, a shrug and at least one twerpish critic bringing up the subject of The Frog Chorus: even a wildly successful world tour couldn't hoist 2001's Driving Rain higher than number 46.

Obviously the unimaginable wealth, rapturous reception at Glastonbury and Live8 and official title as the Most Successful Songwriter in the History of Popular Music must make life a smidge easier, but none of it answers the question: now what? Over the past 20 years, he has tried virtually everything, embarking on projects that presumably whiled away the time between world tours pleasantly enough, but that only the bona fide nutjobs would listen to twice: ambient techno, classical music, old rock'n'roll covers, fitful attempts to reignite the spark with new collaborators, even a compilation of his late wife Linda's musical efforts.

On first glance, Chaos and Creation in the Backyard looks like more dabbling. It teams McCartney up with Nigel Godrich, the modish producer of Radiohead, Beck and, perhaps less laudably, Band Aid 20's Do They Know It's Christmas? In interviews, McCartney has made the sessions sound hard work: "painful", "a plunge into the darkness" and "like being pulled through a hedge backwards". Godrich first dismissed McCartney's idea to make an Indian-themed album, then dismissed his backing band, then started dismissing his songs.

The largely one-man-band results resemble the more ramshackle albums from the first decade of McCartney's post-Beatles career: McCartney, Ram, 1980's McCartney II. But those albums were sunlit, quirky and marked by a daffy, occasionally grating sense of humour. Chaos and Creation in the Backyard is muted and crepuscular. Godrich's measured, dry production means that even the love songs seem strangely downbeat: the chirpy Promise to You Girl sounds as out of place here as a burst of Ob-La-Di-Ob-La-Da at a funeral.

Some of the sessions' tension has seeped into the songs, with surprising results. At the Mercy sounds bewildered and despairing. Riding to Vanity Fair is notable not only for a glorious chorus that rises from the song's murky strings and minor chords in a way that is so inimitably, ridiculously McCartney-esque, you can virtually feel your thumbs involuntarily twitching aloft, but also because it offers a previously unheard noise: Paul McCartney sounding bitter. It's an emotion he has previously avoided, presumably because he spent his golden years collaborating with a songwriter who could do vicious, sneering, bug-eyed bitterness better than anyone. Even when Lennon turned his sights on him - on How Do You Sleep?, an early draft of which tactfully labelled McCartney a "cunt" - he never responded in kind, preferring the bemused, disappointed shrug of Dear Friend and Let Me Roll It. But someone has clearly riled him in a way that Lennon could not. Peppered with withering "apparently"s and "I wouldn't dare to presume"s, Riding to Vanity Fare takes McCartney, emotionally at least, into new territory. It's all rather bracing.

Not all the album's pleasures are so unexpected. It does a brisk and highly enjoyable trade in Beatles references. English Tea offers a string arrangement that is one part Eleanor Rigby to two parts Martha My Dear and a witty lyrical nod to the author's saccharine public image ("very twee," he notes, "very me"). Friends to Go has a distinct Two of Us swing. A charming bit of Latin-inflected fluff called A Certain Softness recalls Step Inside Love, the charming bit of Latin-inflected fluff he wrote for Cilla Black in the mid-1960s. The delightful Jenny Wren could no more obviously signpost its links to The White Album's Blackbird if it were called Listen to This, It Sounds a Bit Like Blackbird off The White Album.

For all the nods to the past, not a note of Chaos and Creation in the Back Yard comes close to Beatle standards: it's an intriguing diversion rather than a major addition to the canon. What it has is a sense of purpose, lovely tunes in abundance, and charm. It mints an unassuming and idiosyncratic style with which McCartney could see out his career. At last, it seems he's found an answer to the previously imponderable question: now what?