Has alternative music ever been as smooth as it is right now? From Rhye to London Grammar, Blood Orange to Jessie Ware, pop's leftfield is littered with acts fusing the soft-rock sounds of yesteryear with suave, melancholic R&B. The danger of this current vogue for soft-pedalled sounds, of course, is that the resultant music can sound suffocatingly tasteful, bloodless and rootless, all surface no feeling.

Woman's Hour are different. Their refined blend of yearning bedsit indie pop and sleek 80s soul is meticulously smooth, for sure, but the slickness is a front, a way to channel the band's own niggling insecurities, middle-class neuroses and romantic failures. A quartet founded by siblings Will and Fiona Burgess and completed by keyboardist Josh Hunnisett and bassist Nicolas Graves, their music is a beautifully ruminative, after-hours carouse that expands deftly on the xx's stark intimacy and the quiet rapture of Kendal compatriots Wild Beasts. Ask them how they hope to stand out in a scene where "tasteful" and "restrained" acts are 10-a-penny, and they distinguish themselves by refusing to distinguish themselves.

"I think it's dangerous to start thinking that way," says Fiona. "If you're too concerned about whether you're going to be considered original or not, that can be incredibly damaging to making music. Because as soon as you start doing that you start prohibiting creativity, you're putting barriers up. Of course, subconsciously, we are the product of our generation. If you look at any movements in art history, the artists might not have been in communication, but you can still identify similarities in the way people are making art. But as long as what we do doesn't feel forced, we hope it'll come across in the music."

Fiona talks like she sings, in a soft Cumbrian burr. Expansively chatty, she's given to lapsing into artspeak, talking of "problematising" the band's image with their sleeve art. Will, on the other hand, is a quiet, self-deprecating presence, chiming in hesitantly only when his sister, tired of being the band's mouthpiece, shoots him glances from across the table. Fiona was a performance art student at the University of London when she had the idea of fronting the band with her brother back in 2011. Amazingly, Will claims he had never heard her sing before – "I was worried she wouldn't be able to do it!" he says – a fact that Fiona traces to a knockback suffered at the age of 10.

"I wanted to try singing, so I tried joining a choir, and was basically told I couldn't sing! I know that sounds really lame, but throughout my teenage years I had it in the back of my head that I couldn't sing, so I didn't. Definitely not on my own, anyway." Since then, Fiona has had voice coaching, and has even installed the odd pre-gig yoga session to ensure the band get into the right headspace.

"I'm still nowhere near touching my toes," grumbles Will.

"The most nerve-racking thing is being myself," continues Fiona. "When I've been performing before in theatre pieces I could escape somewhere, you're performing a role that's not yourself, which I love. Whereas when I'm onstage with a band I fear I'm being judged as me. And that is scary."

Nonetheless, the band worked up a short set's worth of material, with each song title, like their name, cribbed from the Radio 4 schedule (the station was required listening in the Burgess household when they were growing up). But after the release of debut single, Jenni, and a few months of gigging in 2011, they opted to retreat from the frontline to spend more time honing their craft. "We knew we hadn't got to a place where we could look each other in the eye and say, 'This reflects something in me that I feel really proud of,'" says Fiona.

The breakthrough came with the gorgeous Our Love Has No Rhythm (working title: Book At Bedtime), whose shattered, haunting mood owes something to Fleetwood Mac's Tusk, while the lyric's clear-eyed appraisal of a failing relationship points up themes that would come to define the album. The swooning dream-pop of Darkest Place finds Fiona picking over the bones of an affair with brutal honesty ("If there's anything left, we left it dead"). And the hypnotic, Sade-conjuring To The End offers a similarly grown-up take on relationship strife, setting to music the inner musings of a hopeless introvert.

The lyrics feel like an articulation of doubts its protagonist is unable to express in real life, a hunch confirmed when Will turns bright red when the topic is raised, amid laughter from his bandmates. "But isn't that what art is?" he protests sheepishly.

Not for the first time, Fiona jumps in to rescue her brother. "It's an artistic right not to have to talk about those things!" she insists.

But as with their music, what Woman's Hour don't or won't say is as telling as what they do. Their very British reserve and obvious discomfort with discussing their feelings and motivations is reflected in Conversations' cautious grooves; it's what makes the album so poignant and relatable. In a field of smooth operators, Woman's Hour's human frailties are what make them stand out.

Woman's Hour play Field Day, Victoria Park, E3, 7 Jun; Comversations is out in the UK on 14 Jul