Before Punch Brothers' show at west London's Bush Hall last month, I saw a young man turn to his friend at the bar and ask: "What are you supposed to drink at a gig like this?" I understood his pain. There's nothing easily classifiable about this five-piece ensemble who have proved as welcome at Carnegie Hall as they are headlining festivals. In their late 20s and early 30s, they play traditional bluegrass instruments, wear vintage tailoring and sing raucously about whisky. But their sound is as sophisticated as their New York home and equally cerebral: they regularly play Bach as an encore. Their music defies attempts to label it, although that's not to say people haven't tried: "bluegrass rock", "folk-jazz" and "avant roots string band" have all failed to stick.

Punch Brothers aren't even brothers. Their name comes from a Mark Twain tale in which a traveller is rendered unable to do anything when a catchy jingle takes his ear. "We thought that was a good goal for any band," says guitarist Chris Eldridge. It's working: they recently returned from touring with Paul Simon, whose son, Gabriel, is a fan. Rapturous receptions in London and Paris last month, as well as a sell-out gig at Celtic Connections, suggest that the Europeans are about to follow.

Chris Thile, their charismatic frontman, has been described in the Washington Post as probably "the most virtuosic American ever to play the mandolin"; he looks like a cross between Jude Law and Jonny Lee Miller and plays his instrument with swivel-hipped motions. Thile created the band six years ago to record an experimental, 40-minute quintet he had composed. And while the blazing ambition of their music has not dimmed a single watt, their latest album, Who's Feeling Young Now?, the follow-up to 2010's Antifogmatic, is the one that promises to transmit it to a wider audience.

"We'd been having fun playing with musical ideas the way some people play with puzzles, you know, sudoku or whatever," says Thile. "We were erring on the side of the intellectual. We're now quick to discard something that doesn't hit us in the heart or the gut."

Who's Feeling Young Now? targets both. Thile's lyrics major on broken or longed-for relationships, of connections missed or misunderstood; some songs, such as "No Concern of Yours", seem to be a manifesto for better community. In the otherwise lighthearted "Patchwork Girlfriend", its serially unfaithful narrator pleads: "I'm not sure what I'm saying, I just need you to believe me." Thile says he's fascinated by "the way we well-meaningly deceive each other and ourselves and justify our activity or try to take mulligans [second chances]. Can I have a do-over please?" He pauses. "Unfortunately, sometimes the answer's 'no'."

Musically, the album seems about to burst its skin, with influences from Swedish folk to punk to minimalism's Steve Reich. There's even a woozy, bluesy interpretation of "Kid A": Radiohead covers have long been a Punch Brothers speciality (check out their "Packt Like Sardines in a Crushd Tin Box" on YouTube). "Everyone talks about how depressing Radiohead are. I don't hear it," says Thile. "They've created their own universe and it is dimly lit, but it's not inherently dark."

Punch Brothers recorded in Nashville's famous Blackbird studio last autumn; I was there to see them work on "Don't Get Married Without Me", the track that is now the album's finale. The needles on the sound desk hopped to the song's distinctive bluegrass chop while fiddle player Gabe Witcher made arpeggios flutter like birds. The songs' complex arrangements were roughly a year in the making, but when they play live each member improvises their individual solos, and it's a spontaneity they're keen to capture in the studio; there's little distinction between a warm-up and a take. "We can trust each other enough to leave some things open-ended," says Witcher. "We're trying to capture on the record the first time these songs really come together."

Their instrumental talents are prodigious: Witcher and bassist Paul Kowert are classically trained; Eldridge was mentored by acclaimed acoustic guitarist Tony Rice. Noam Pikelny was recently chosen by a panel of the world's best banjoists (including the instrument's eminence grise, Earl Scruggs) as winner of the inaugural Steve Martin prize for excellence in banjo and bluegrass.

It might sound like a joke award, but it came with a very serious $50,000 cheque from the comedian's own pocket. I ask what he spent the money on. "It afforded me the luxury of travelling separately from the rest of the band," he deadpans. "For one day." No treats for his bandmates? "Yes, I bestowed on them the gift of a solo banjo album," he laughs. Beat the Devil and Carry a Rail came out earlier this year and includes contributions from the rest of the band.

Of the five, only Eldridge, whose father was a founding member of influential bluegrass outfit the Seldom Scene, actually comes from the south. Thile grew up in California, where he formed Nickel Creek with brother and sister Sean and Sara Watkins; when they recorded their first album, with Thile's dad playing bass, he was only 13. "I didn't have stage parents," he says, "and sometimes I've envied people who did because I felt like, I guess, I'm compulsively worried I'm not accomplishing enough. Since I was little, I've always put a lot of pressure on myself."

By the time Nickel Creek disbanded in 2007, their 18-year career boasted platinum-selling records, a passionate following and a reputation for pushing musical boundaries. Thile, recovering from a divorce at the tender age of 24, channelled his hurt into The Blind Leading the Blind, the four-movement bluegrass suite that was to be Punch Brothers' first project. Not all of Thile's fanbase were ready to follow him in his new direction, however. "It was a case of, sorry, guys, if that's what you're looking for, better go check off one of the Nickel Creek knock-off bands..." smiles Thile.

Thile's current oeuvre establishes him as one of the most progressive musicians in the States. When not on the road with Punch Brothers, he has been mixing it with major classical and jazz practitioners such as Hilary Hahn and Brad Mehldau, and last year he collaborated with Yo-Yo Ma on the cellist's bluegrass/classical hybrid TheGoat Rodeo Sessions. "I just want the opportunity to transcend my personal boundaries and the only way you can do that is by latching on to other people's coat-tails," says Thile.

So it was no surprise to see Punch Brothers joined on stage, at the end of their Bush Hall set, by Marcus Mumford, whom they met at the Telluride bluegrass festival in Colorado a couple of years ago. Together, they sang an Irish drinking song that they recently recorded for the Coen brothers' latest film, set in the world of 70s folk; when Inside Llewyn Davis is released next year, Punch Brothers can hope for a lot more exposure. For Thile, it couldn't come at a better time. "This is where to start listening to us." He laughs and pounds the table. "We're ready to be heard!"