There is a great story about Led Zeppelin's arrival in LA in December 1968. It was the moment at which they began their campaign to conquer the US, something they proceeded to achieve in remarkably short order, with perhaps more mythic swagger than any band before or since. Like so many of the subsequent stories of excess and debauchery, it involves drummer John Bonham and a hotel employee. This time, however, the employee wasn't on the receiving end of a drunken threat of violence or required to clear up the debris once the drummer had finished enjoying himself. As Led Zeppelin checked into the Chateau Marmont, Bonham meekly inquired of the desk clerk if there was an oven in his bungalow: he and singer Robert Plant were tired and intimidated and desperately homesick, a situation the man who would soon become known as Bonzo, scourge of the global hotel industry, thought he might alleviate by cooking them all a nice roast dinner.

Just as the legend of Led Zeppelin doesn't have much room for the notion of Bonham donning a pinny and dishing out the sprouts, so the image of Plant, isolated, vulnerable and pining for Kidderminster, has been crowded out of the popular imagination by the more potent image of Plant, bare-chested Golden God of Rock, bestriding America like a priapic colossus. Nevertheless, it's brought to mind by Lullaby and … the Ceaseless Roar, an album whose message frequently appears to be: there's a feeling I get when I look to the west and it largely involves wishing I was back in Worcestershire, or possibly Aberystwyth. On its centrepiece track, Turn It Up, we find the singer's voice adrift over a dank, disturbing mass of electronic creaking and crackling, ominous, thudding drums, guitar-playing somewhere between heavy-duty riffing and the desert blues of Tinariwen, with the west African lute of recent recruit Juldeh Camara distorted until the figure he plays becomes claustrophobic. Plant is "alone with disconnection … lost inside America … blinded by neon, the righteous and the might". "Let me out!" he howls. At the other extreme, there's Somebody There, a lovely, warm, insistent pop song – not something Plant normally specialises in – featuring the singer rambling on contentedly through a verdant landscape, ruminating on the loveliness of the passing of the seasons, or Embrace Another Fall, on which Plant is "home again", "out upon the shire, all through the frost and rain". The track's chilly ambience eventually resolves, gorgeously, into a verse from Marwnad yr Ehedydd, a Welsh song thought to date back to the 15th century, sung by Julie Murphy.

Lullaby and … the Ceaseless Roar is at least partly inspired by the end of Plant's relationship with singer and former collaborator Patty Griffin, and the increasing cultural isolation he says he felt in Austin, Texas, where the couple had relocated. The result is a set of remarkably personal, moving songs ruminating not only on romantic disappointment – beautifully rendered on the stark piano ballad A Stolen Kiss – and the horror of finding himself "adrift … high and lonesome" in the US, as Pocketful of Golden puts it, but on the agonies and pleasures of ageing. It all feels striking coming from an artist whose legend is based on a certain bombast.

The songs' impact is aided by his hugely impressive backing band, the Sensational Space Shifters, essentially a rebooted version of his noughties cohorts Strange Sensation, a disparate collection of musicians variously drawn from the ranks of Portishead, Jah Wobble's Invaders of the Heart, Britpoppers Cast and modern jazzers the Loop collective. To use a phrase that might appeal to Plant – famously partial to a bit of mysticism – there's something almost alchemical about their ability to draw together incongruent musical influences into something coherent and exciting. The hypnotic repetitions of Camara's lute and the scrape of his riti, a single-stringed Gambian fiddle, never feel worthily grafted on to their sound: similarly, the electronics provided by John Baggott sound integral, rather than an impulsive lunge for contemporaneity. On paper, the collection of influences – world music, acid house, British folk, clattering funk breakbeats and country – that makes up opener Little Maggie looks like something painfully well-meaning you might chance upon in an outlying field at Glastonbury, shortly before running as fast as possible in the opposite direction. In reality, it sounds fantastic: gritty, dark and satisfying.

You could apply those adjectives to almost all of Lullaby and … the Ceaseless Roar. The optimistic Led Zeppelin diehard, patiently waiting for a reunion tour, might sighingly note that the Plant they find here – vulnerable, dignified, restrained, backed by a thoroughly modern-sounding band – is as far away as it's possible to be from the Plant they fantasise about seeing again, wallowing in nostalgia, stomping about a stadium stage and urging the ladies to squeeze his lemon. But even they might be forced to admit how well Plant's current mode seems to suit him.