'This is the gateway," says an animated Bobby Ray Simmons Jr, rearranging glasses of water on a table in a central London hotel to represent a portal. The rapper, already a veteran at 21, is trying to explain why his career to date has been defined by the tussle between his experimental inclinations and commercial realities, between going his own way and following the gangs/guns/girls crowd. And in so doing he is providing an object lesson in how hip-hop – once a music defined by its resistance to compromise – operates as an increasingly dysfunctional business in 2010.

"You have everybody filing in," he continues, marching a pepper pot and a salt cellar between the drinks to represent the rappers who are content to simply supply music to the mainstream market. "And it's easy, because it's just straight up-and-down, everybody followin' the format. But when you wanna do something different, you have to go all the way around …" At which point Simmons picks up his own drink – an Earl Grey Martini, rather than the rapper stereotype of Moët or brandy – and sweeps it to the left, around the side of the gateway.

"And a lot o' people don't wanna do that," he smiles. "They'd rather just go straight in, rather have a surefire way of makin' it as an artist. I kinda started that way, but it was like, 'Aargh! I just can't do it!' I can't. I can't conform, it's not in my nature. So it took me a long time."

The pop fans who have been buying his songs in their droves this summer could be forgiven for considering Simmons to be the quintessential rapper who went "straight in", but he has a back story that belies his youth. Rapping since his teens, he signed his contract with a subsidiary of the Atlantic label in 2006, and his current album, B.o.B Presents the Adventures of Bobby Ray, is the first he has released for the major. But he has given away at least six mixtapes of his own material since 2007, each helping generate buzz and carve out a reputation while defining who, musically, he wants to be.

The long slog has been the making of him. His first major label single, Nothin' on You, topped charts in America and Britain at the end of May; its follow-up, Airplanes, has already given him his second UK No 1, even though the single itself isn't released on CD until Monday (download sales of the album version were enough to make it the biggest-selling track in Britain by the middle of July). And while chart success is gratifying, B.o.B is just as pleased that he's been able to achieve it on what he has come to realise are his own terms.

"I went around," he continues, moving the Martini in an exaggerated arc past the gateway. "And when you're takin' that route, it's frustrating. Sometimes you feel like it's unfair, sometimes you feel like, 'Man, this isn't right; I just wanna change it.' But eventually, finally, everything came together."

That moment, Simmons believes, came in November last year, when he teamed up with songwriter Bruno Mars to create Nothin' on You. A glorious, euphoric, essentially old-fashioned kind of love song, it marries Mars's melody with verses from Simmons that find him alternating insouciant sung passages with raps in which his lovestruck grin is almost audible. A sonic approximation of a summer day, it would stand apart from other 2010 rap records even without B.o.B's lyric, which eschews the in-the-club, eyeing-up-the-booty cliches that have come to pass for romance in hip-hop, and turns them inside out. Simmons thinks it's comparable in spirit to A Tribe Called Quest's Bonita Applebum, but that may be its only antecedent in the hip-hop canon.

"I felt like I was experienced enough as an artist to put a verse together that didn't sound corny, and didn't just sound like a rap on a verse," he explains. "If it was just that, the personality wouldn't have been the same. It took a marriage of me doin' what I wanted to do, and compromisin' a little bit for the system. But it was kinda like playin' their game by my rules. And Nothin' on You changed my life: I finally feel that I reached the point where I wanna be at. At times I questioned whether it was worth the sacrifice, but now I see it was."

On the face of it, Simmons shouldn't have had to struggle so hard to get to a point where he felt able to find and use his own voice. Though a native of North Carolina, he grew up in and around Atlanta, Georgia, a city where the role models for a rapper keen to go his own way were hard to ignore. From the likes of Lil' Jon and the Yin Yang Twins, who made crunk a commodity by doggedly sticking with the music they believed in, to mavericks such as Cee-Lo Green of Gnarls Barkley, Atlanta is home to plenty of hip-hop iconoclasts. Simmons even took his rap acronym from the title of a song by the sine qua non of Atlanta rap individualists, OutKast. Yet he felt under tremendous pressure to fit in, and make music that, while coming somewhat from the heart, nevertheless expressed only a narrow slice of his creative range – a fate that seems to have befallen several of his peers, whose records can often seem indistinguishable from one another.

"The industry can get so caught up in the formula that people'll get taken for granted," he says. "There is a formula, though you can't really pinpoint where it's comin' from. It's just like there's this collective agreement, this unspoken rule of what somethin' is gonna sound like, or what's gonna be accepted. And it's a circular relationship. A lot of artists go in the studio and say, 'OK, whaddaya want me to do? Is it gonna be a hit? I'll do it. Is it gonna get played on the radio? I'll do it.' So they start makin' these songs, and they fall in the same tempo, same category, same this, same that, and it'll just all sound the same. And what I was fighting was that mindset."

While he feels Nothin' on You was the point where he met the mainstream on equal terms, the turning point in his journey there was a song called Generation Lost. A single verse rapped over a sparse piano and pizzicato string loop, it's barely two minutes long; but it was pivotal in Simmons' quest to understand who he was as an artist, and who he wanted to become.

In it, he admits to adopting gangsta styles to fit in, expresses dismay that so many rappers feel they have to dumb their music down to make a living, and says he adopted his B.o.B moniker because he was ashamed of his real name. Daringly, for someone who had yet to release a proper record, he demolished the weed-dealer, slack-talking persona he'd constructed on previous mixtapes. Instead, like a penitent sinner, he vowed henceforth to "play my guitar, rap about aliens and sing about stars", because to do otherwise wouldn't represent his own truth.

"I had it all in my head, and that was me gettin' it out," he says. But he didn't expect anyone to understand. He shot a video for the song (bit.ly/bobgenerationlost), in which he's walking along the edge of a road in the middle of nowhere, rapping to the camera, cars occasionally whizzing past, taking no notice. "That was partly a metaphor," he says, acknowledging that the views he was expressing weren't likely to be universally well received. "In a sense, that track was like tellin' people there's no Santa Claus." But any resistance he may have encountered was irrelevant, because the most important audience for Generation Lost's lyrics was Simmons himself.

"My music, I feel, has always been experimental," he says, "but it had got to a point where I felt disconnected from it completely. I didn't want to be a Clark Kent/Superman: I couldn't really say, 'Well, B.o.B's the old me, and Bobby Ray's the new me.' I had to just make a point."

He was as good as the song's word, and reinvented himself: performing at the South By Southwest festival in Austin, Texas, sitting on a stool, playing acoustic guitar, singing love songs – the Dirty South rapper behind the US radio hit Haterz seemingly banished. But on his current album he's found a way for the two to coexist. It's little wonder the album sounds so mature in both execution and outlook, or that he is capable of inspiring his elders.

A second version of Airplanes ends the record, in which he takes the conceit of the chorus – "can we pretend that airplanes in the night sky are like shooting stars/ I could really use a wish right now" – and meticulously interrogates it, asking whether he would have been happy if everything in his career had been easy, and wondering about how wrong it all could have gone. The final verse comes from Eminem, and is probably the best thing he's recorded since Lose Yourself: he takes B.o.B's two perspectives and applies them simultaneously to his own life story, and ends up with a passionate paean to the integrity of artistic struggle.

"It was an honour that he took what I said and it could relate to him," Simmons says. "I feel like the story's similar, in terms of the passion. I really relate to him." The feeling is evidently mutual.

Airplanes is out on Monday on Atlantic; B.o.B Presents the Adventures of Bobby Ray is out now. Most of his earlier mixtapes can be downloaded from bobmixtapes.wordpress.com