Chart R'n'B is in rude health, with the likes of Rita Ora and Flo Rida scoring massive hits with anodyne, cookie-cutter stompers about going hell-for-leather in a club. However, on the fringes there has been a surprising reaction to this homogenised sound. Acts including The Weeknd, Miguel and Solange have broken free of the Top 40's castle walls and are re-inventing the sound of R'n'B via a boundary-pushing style some are calling "eclec-tric."

For many, it began with Frank Ocean. His rise to prominence began with a mixtape (February 2011's Nostalgia, Ultra) that leant heavily on its alternative music samples of MGMT and Radiohead, conjuring up an utterly unique sonic and lyrical world in the process. But it was this year's Channel Orange from Ocean that reminded the music intelligentsia that R'n'B music could have a complicated depth. Like Ocean, The Weeknd's (aka Canadian vocalist Abel Tesfaye) unique narrative poetry began in the mixtape world. The three mixtapes which began with House of Balloons in March 2011, and have now been banded together and released last month as Trilogy, brilliantly flipped the nocturnal tropes of R'n'B on their heads. Instead of signalling hedonistic fun at anonymous clubs, The Weeknd's songs signalled paranoia and despair, all to an abstract soundscape which sampled Siouxsie and the Banshees, Beach House and the Cocteau Twins.

Cleverly, Tesfaye has maintained a distance from his musical creation, declining to be interviewed and making listeners play a guessing game about where his real life ends and the narrator of his songs begins. At a rare live show at London's Supper Club a few weeks ago, the singer played off this dichotomy perfectly. On the one hand, there were fans who were screaming and whooping like he was heartthrob Trey Songz and he'd just whipped his shirt off. On the other, he neared the end of his set with the static drama of "Enemy", and, as the glassy textures of the song faded out, he repeated the strangely familiar chorus, a capella: "Please, please, please, let me, let me, let me, get what I want… this time." The setting of Morrissey's moribund lyrics may have changed, but the spirit of desperate desire remained. Although "Enemy" is yet to appear on a Weeknd album, the fact that Trilogy – a truly "eclec-tric" work steeped in the elusive vocabulary of alternative music – topped the Billboard R'n'B/hip-hop charts is indicative of the sea-change that has been taking place in the last couple of years.

Alongside Tesfaye and Ocean, more R'n'B acts have incorporated avant garde elements to their sound, from Drake working with Jamie Smith from The xx, to more mainstream artists like Rihanna and Usher dipping their toes into abstract, atmospheric sounds, whose trademark downbeat synths and sparse drum patterns, and emotionally transparent lyrics, have more in common with indie, fringe dance music and goth than the furious BPM of the Top 40.

In this spirit, singer Miguel's second album Kaleidoscope Dream combined strands of swoony psychedelic rock with dreamy soul harmonies. The singer, who coined the phrase "eclec-tric", has been credited with pushing collaborators like Alicia Keys out of their comfort zones. He also recently tweeted a picture of himself with Beyoncé, suggesting he's been working on her next album. He summed up the musical shift to LA Times: "I hope more artists who are R'n'B at the core and pushing the boundaries will join me in embracing the fact that [the genre] did become a stereotype." He told them "I'm proud to wave the flag for what's coming." Other, established singers like Beyoncé's little sister Solange, and Dawn Richard, have had to turn their backs on major label demands in order the fight against the stereotypes of the genre.

After two major label albums, Solange's attempts to break out of the shadow of her sister, first as a tween-friendly pop star, secondly as a Motown-channelling chanteuse, seemed unsuccessful. But she had garnered a reputation, and fans, as the "hipster Knowles" – she infamously introduced her brother-in-law Jay-Z to the sounds of Grizzly Bear.

Dropped from Interscope Records, Solange moved to Brooklyn and began collaborating with the likes of American indie bands Chairlift and The Dirty Projectors. For the resulting mini-album True (released on Terrible Records, the label of Grizzly Bear's Chris Taylor) she has stuck with Dev Hynes, who has brought the distinctive sounds of his lo-fi rock album Blood Orange to the project. The meeting of Solange's unmistakably R'n'B vocals with Hynes' spidery guitars and lo-fi synths sounds like the songs have been snatched off an album by Ariel Pink's Haunted Graffiti. They are utterly compelling.

"We're more open to the crossover of genres," explains Dawn Richard who, like Solange, began her career at the poppier end of the spectrum before going indie. Six years under the management of P Diddy (first with reality-show girl group Danity Kane then with Diddy – Dirty Money) and Bad Boy Records, she left in 2011 to self-release a series of eclectic one off singles and EPs.

"You see urban influences in bands like Linkin Park and System of a Down. They've mixed R'n'B elements into rock and people have accepted it," says the singer, who releases Goldenheart, the first part of a proposed futuristic trilogy of albums, in January.

"It's a scary moment for us in R'n'B, but it's happening. We're trying to start something new and fresh. It's exciting to see people appreciate it."

Dawn Richard's 'Goldenheart' is released on 15 January. Miguel is touring the UK, beginning on 12 January at Shepherd's Bush Empire, London W12