A short walk from Lord's cricket ground is a pocket of St John's Wood that seems for ever suspended in the 70s. Around the corner from a clutch of Peabody estates, the Oslo Court restaurant still serves prawn cocktail, peach melba and veal Holstein. It's 1.59pm on a Thursday at the height of this summer's heatwave, and as an MCC member passes by in his egg-and-bacon tie, a black Mercedes pulls up outside a grand but anonymous building. From the car emerge two women clutching union jack carrier bags. The one with the coiffed blonde flick and the Gucci espadrilles is Mary J Blige.

The much-garlanded soul singer isn't here for the heritage eats. Blige is starting her two o'clock shift at RAK recording studios, itself synonymous with eras long departed. The blue plaque above her head records the name of founder Mickie Most, producer of hits for Hot Chocolate, the Yardbirds, Suzi Quatro and more. But Blige is at RAK for a project that is entirely of the moment, decamping from New York for a creative stint in London that promises to deliver the most innovative and unusual album of her 22-year career. For two weeks she has been collaborating with some of the most exciting young artists in contemporary music, a disparate group who are in their own way as ineffably English as any on the roster in the glory days of Most.

The movement that began with Amy Winehouse and Adele and now sees British acts making up 40% of sales in the US charts has been followed by a second wave of UK musicians being embraced by America. In particular this summer, the insistent pop house music of Surrey brothers Disclosure and the impassioned torch songs of their friend Sam Smith have cut a swath from east coast to west. As well as performing, all three are accomplished songwriters, and it is with Disclosure, Smith and their coterie of regular collaborators that Blige has been hard at work. Ten new songs, co-written and recorded by the singer in one month in postcodes W6, NW8 and NW5, will be released this autumn as Mary J Blige: The London Sessions.

Baronial as RAK may be, the studio that Blige is using is no more than 10ft inside the front door, and in turn the booth where her microphone awaits no more than 6ft inside that. By 2.01pm she is ready for action, the closest to a litany of diva demands being a whispered "Can I get my tea, please?" Primed for this businesslike approach to the recording process is a team of 12 blokes, ranging from tea boy to session players and finally the producer of the day, Jimmy Napes, 20 years her junior, who stands before the mixing desk. Napes co-wrote much of Disclosure's immaculate debut album Settle and has done the same for Sam Smith's debut In the Lonely Hour. It may be Napes's bicycle that is propped against the desk. He signals for the track to be run, and Blige attacks her opening note.

But where is she? A window the width of the room looks into the cavernous recording area that holds the musicians. Through it can be spied a bass player and pianist, barely illuminated, and at the back a distant drummer. Blige, however, is invisible. She is definitely in there somewhere: her voice begins to dance across the music in tones that are rich and plaintive and entirely assured. In just a line or two she fills the mixing room with life. Then she audibly digs down and summons up something extra. In the bottom left-hand corner of the window her outstretched hands appear, reaching up from her spot in the shadows. Napes, however, is not convinced.

"Something not quite right with your phrasing at the beginning of verse two," he says. 'Can we go again please Mary?'

She laughs.

"What's happening is I can't flip the page quick enough to get to that part," she says. Napes peers over to see her struggling with the sheaf of lyrics. "But I'm warming up," she says brightly. "We're getting there!" Which, as she was born in the Bronx, is pronounced "We gittin they!"

Now she sings again, and what is striking is not just the timbre, but the rhythm. With her voice dipping in and out of it, an initially unremarkable beat has taken on new punch and syncopation. Blige was first launched as "the queen of hip-hop soul", and it was over thumping breakbeats that she first sang for producer Sean "Puffy" Coombs in the early 1990s, and then again a decade later with Dr Dre. To compete with beats takes real vocal talent; to somehow elevate them too is one of Blige's special gifts.

Napes wants another take. Then another. On the seventh, Blige has appreciably gone up a gear. By the ninth, however, the engineer is exchanging glances with the boy operating the computer behind a spaghetti vongole of wiring. She may have made 11 studio albums and been a guest vocalist on more than 120 tracks, but truculence is a known aspect of Blige's character. Are the toys about to exit Mary's pram?

"OK, Mary," says Napes. "Let's do the last take." "You just say that to get the best performance," says the engineer. "Then you get her to do one more after."

With her vocal duties done, Blige comes into the mixing room to listen to the playbacks, sinking into a sofa next to the project's American executive producer, Rodney "Darkchild" Jerkins, a don of the production world who has worked most notably with Michael Jackson, Beyoncé and Lady Gaga. Aside from the song, the two also appear keen to talk about food, and in particular fish and chips, the consumption of which Blige feels is central to what she calls "the London-scene experience". Napes meanwhile is in the RAK kitchen, which has hosted the tea breaks of everything from Radiohead's The Bends to Kim Wilde's Kids in America. I ask him how he came to be part of the album.

"It has been a little bit surreal to be honest," he says, 'I grew up listening to Mary J Blige, and then there she is with the Disclosure boys in Kentish Town, where my studio is, next to a car park. She had loved Stay With Me, which I wrote with Sam Smith for his album, and that's how I got the opportunity to work on this project. We were writing for a week or so – we've got some wicked songs. It does take time to get the confidence to say to her, 'I think you can sing that better' – but it's crucial. She is counting on me to call her on stuff, because she wants to make the best record she can."

Disclosure were the starting point for Blige's fascination with London's new crop of music-makers. Late last year she saw the duo's single F for You on the Vevo video site, and got in touch with them about covering the song. The resulting release was not a cover, but an astutely balanced duet, composed by Blige, that retained the vocals of the group's 20-year-old singer Howard Lawrence and married them with a set she had written herself. Her contribution brought new depth to the original: driving, gutsy, soulful – it made Disclosure sound immeasurably funkier and was the most original thing she had done in years.

"I love what Mary did on that track," Howard tells me on the phone. "She hadn't just changed everything, written entirely new stuff and gotten rid of everything we'd done. She wrote something we believe totally fits with the track stylistically – it's an excellent house music vocal. But she also chose to send it over and say 'what do you guys think'? As opposed to putting it straight out and saying 'I don't care what those guys think, I'm Mary J Blige."'

And what about when it came to working face-to-face with the 30-times-nominated, nine-time Grammy winner in deepest Kentish Town? "The track Right Now started with some chords I made on Jimmy Napes's piano," he says. "We took that and gave it a Disclosurey feel with some drums that Guy [his brother] made. Mary leaves the instrumental side to us and gets much more involved when it comes to writing vocals. So she puts an idea in, then we all have a dog-fight until someone comes up with something that they agree on. It's a really good way of working, because no one is afraid to just pitch ideas. We did the majority of the songwriting within about two and half hours, and then spent a lot of time editing and polishing, because that's how Guy and me work. We'll write a whole song very roughly, then go in and proof-read it for a long, long time and change it so much that you wouldn't even recognise it from when you started.

"Mary has amazing vocal control but she also has such a level of soul to her voice. That's why she is the queen of that style, of writing using ad libs. She'll just go off on one over a track, and get really into it – just sing whatever comes naturally. And that normally ends up making the final take."

Mary J Blige in London: 'Working with these guys, it's like they're my family. We have a good chemistry together.' Photograph: D'Andre Michael

And what about her notion of a London sound? What does he think she believes she is getting from this month spent collaborating here? "The sound in London at the moment is house music. That is what the majority of people are producing their songs like. But the ones that get truly successful are the ones using proper songwriting. Rudimental for example – they write proper songs and then produce them like dance music. And that is exactly what we're trying to do, along with a few other people. But that applies to any genre, not just dance music. You could take the songs off Sam Smith's album, produce them in a completely different way and they would still be a huge success – you could produce them like acid jazz and I still feel like they'd get somewhere."

Back in the studio, Blige is preparing to leave. It is the third anniversary of the death of Amy Winehouse and she has arranged to have a get-together with Amy's father, Mitch Winehouse, whom she does not know but wants to meet. After some discussion with the assembled locals she has decided on dinner at the nearby Sea Shell of Lisson Grove – conceivably London's poshest chippy. When she has gone, Rodney Jerkins, an affable pear-shaped man umbilically attached to a MacBook Pro, plays rough edits of some of the London Sessions tracks. "It's all comin' from here right now," says Jerkins, whose embrace of "yookay" cuisine consists of taking constant furtive nibbles at a supply of Cornish pasties.

"On the final record, we wanna keep the feeling of London," he says. A film crew are making a documentary of the recording, and he plans to drop in audio clips from it between tracks. "You have so much different music here. Variety births the next generation. In California the music kind of all feels the same. That 90s house vibe you have right now – it feels fresh. Dancin', celebratin' – feelin' good about life. We're making a Mary Blige record, but she can introduce new styles to the world." He says they will make 13 tracks and then choose a final 10. We listen to three and the variety is striking.

Mary J Blige duets with Sam Smith on Stay With Me for an encore at Smith's show at the Apollo theatre, New York, in June. Photograph: Theo Wargo/Getty Images for Capitol Records

It is certainly not an out-and-out house record. Pick Me Up, produced by Watford's own Naughty Boy and co-written with Emeli Sandé, mixes sub-bass with clarinet and a percussion sound that recalls early 00s UK garage. Therapy, written by Napes and Sam Smith, has a doo-wop feel, while the Disclosure number is house plus additional squelches. Disparate as the writers and producers may be, Jerkins agrees that these all sound unquestionably like Mary J Blige tracks: 'She knows how to cement her voice to the lyric. She knows how to make you know what she's been through." The feeling that she conveys of a life lived though all its triumphs and defeats is certainly at the heart of Blige's sound.

The next day she is back at RAK to work with another young UK producer, Sam Romans. Romans thinks he knows the attraction of Blige's "London scene": "I'm doing a track at the moment with Naughty Boy – a soul track – and he was saying it's amazing how a Pakistani singer from Watford and a Jewish producer are making something that would be described as black music. That is the interesting thing about England."

And what of Blige herself? At the end of the Romans session we sit down to talk at her hotel and she starts with her own vision for The London Sessions. "Our idea was to become part of London," she says, "to really embrace the culture – to really live in it. Not that I haven't been here before, but I've never had the chance to really soak in it the way I have this time. To make records ["rekkits"] from the London-scene perspective." What is that exactly? What does she think London has? "Freedom," she says. "The music is free over here the way it used to be in the States. Artists are just free to do what they love. Listening to the radio you can hear the freedom. The music is living and breathing – you can hear that from Adele's last album. It was massive – a big deal. But she did what she loved."

She has clearly relished the combination of creative freedom and the discipline of a time limit. "Working with these guys," she says, "it's like they're my family. We all seem to have a good chemistry together. And when I sign on, I sign on – I can't go hang out! Because I have to make sure that my music is right. And I love to work. I love creating, and once I'm in it, I'm just in it. I just love being all in the midst of the creativity."

Does she always find it so easy to enter a studio and summon up such potent and emotive performances? "Once you agree you're ready, you ready," she says. "You walk in. You already know what the mindset is, you know what the song is. You read the words, and get everything from here [hand on heart]. When I'm singing, I don't think about anything but what I'm doing. I could look crazy in that moment, it doesn't matter to me. I'm just trying to get all this stuff out. Because it feels good to get it out. It feels good to sing. It's like you can fly almost, when you singing that stuff."

Of her meeting with Mitch Winehouse, she says this: "When you lose a child… I can't even imagine what that must feel like. But he's a beautiful person. It was like he gave me the chance to hug her through him. She was one of the special ones, and I never got a chance to meet her. Her voice had a lot of pain, a lot of joy, a lot of struggle, trying to figure out 'Who am I'? And she was just so free in her expression. She said whatever she wanted to say – and that's what I admired about her. She was talking to you: 'It is what it is – like it or not.'"

In everyone connected with The London Sessions, this quest for emotional honesty was a recurrent theme. Never arch or ironic, all the young Londoners were notable for an earnestness that somehow chimed with Blige's years of singing with a raw and open heart. The final part of the jigsaw came in a telephone conversation with Sam Smith, writer of four songs for the Blige project and riding high in Los Angeles on the success of his debut album and its anthemic single Stay With Me.

"With my record – and when you think about Adele, and Amy Winehouse and Ed Sheeran – we're not worried about the way we're looking, the way we're coming across in our music," he said. "We're just saying what's in our minds and in our hearts. Some people think that when you are singing about heartbreak, or how lonely you are, or how sad, that you are admitting to weakness. But I don't see that as weakness. I see that as strength – to be able to face your issues and your sadness head-on. That's what I've tried to do in my music, and I think that's what this Mary album is about – a fearless vulnerability.'

Mary J Blige: The London Sessions will be released in November on Capitol Records