Professor Green once made a career out of cussing people. In the dingy nightclubs of Neasden and King's Cross he'd go head-to-head in a ring with other MCs. A small-scale weed dealer at the time, it's not difficult to imagine how he earned his stage name. Back then, he had to be thick-skinned in receipt and callous in delivery. In 2005 he won a rap battle final and a £2,000 prize by telling black rapper Roky: "You got slewed, my bars are harder too/ I'm the reason your little sister ain't as dark as you".

Green's life has changed almost beyond recognition in the ensuing six years. Last night he played an acoustic set at the Groucho club as part of a Great Ormond Street charity benefit. This afternoon he's rushing over to T4 before playing a freshers' week gig in Hull.

In person, he's got grandad-like qualities: he comes in to the interview complaining about a dicky tum. At the photo shoot he quaintly refuses to roll up his sleeves for fear of scuffing them up. He's become more sensitive to criticism, too. He says his upcoming fly-on-the-wall TV show "scares the fucking life out of me, people are gonna see me and think I'm a prick. It's gonna be the start of people coming up to me on the street and calling me a wanker."

Green's transformation, from aggro battle rapper to self-deprecating TV star, shares a timespan with the rise of grime. But his music doesn't have the same glitchy, claustrophobic feel. It's funnier, the beats are more expansive, there are no unintelligible, 32-bar tirades. Rather it exists in the phantom genre of UK hip-hop, something that never quite took off.

"I can't dance," he tells the Guardian, sipping on mineral water in the plush offices of his record label. "So I was never really into garage in the first place. The grime thing never interested me. I grew up influenced by the US hip-hop scene, where everything was about the lyrics." Green was signed to Mike Skinner's The Beats label but was never given the chance to release an album. At the end of 2007 Skinner's label shut down and Green was handed his P45. "They all had our best interest at heart but just didn't really know what they were doing," he says looking back. "I don't hear a peep from Mike now."

'My dad treated me like shit as a kid, but I loved that man'

Photograph: Felix Clay for the Guardian

From then, Green's life quickly descended into horror. A few months after he was dropped, his father killed himself. Born Stephen Manderson, Green was raised mostly by his grandmother and great-grandmother, and had always had a turbulent relationship with his dad, who was rarely around: "People looked after me, I wasn't unloved, but I suppose you always want what you can't have. With my dad, the sun shone out of his backside. He treated me like shit as a kid, but I loved that man."

In 2009 Green signed a new deal with Virgin but a few months later his own life was almost taken away from him. In May 2010 he was stabbed in the neck with a broken bottle. On the street outside the club where it happened, he called his mother and said goodbye. He got lucky and survived. But by then he had suffered too many setbacks. He'd had enough.

"That summer, I really thought about jacking it all in," he recalls. "I was back selling drugs, harder stuff, stuff I don't even really condone. As I say on the record, 'I was selling wraps of coke, not raps and flows'. Then, from one conversation with Lily Allen, it really did change direction. She picked me up and put me back on my feet."

Allen had found out Green had been working on a version of Beats International's Dub Be Good To Me. They turned it into a duet which they debuted at Bestival 2009, and the resulting single went on to be the second of Green's top five hits the following year.

"It's nice to meet someone with no ulterior motive. She stood to gain nothing from that," he says. "I think as you get older your circles become smaller but you also become a lot closer with like-minded people. Lily is one of those people."

'People tell me how strong I am because of what I've dealt with, but I haven't … a lot of shit's unresolved'

Photograph: Felix Clay for the Guardian

Green's own success converged with that of his chums in the grime world. Failing commercially, they'd begun to rap about late-night shopping over Vengaboys beats, and Green – still trying to process what had happened to his life – found himself on the coat-tails of a scene he'd always rejected. "I didn't have any time to cope,' He says. "People tell me how strong I am because of what I've dealt with, but I haven't. There was a lot of shit that was unresolved. A lot of it still is. I don't know if you ever truly get over anything like that. Stress is a motherfucker for me."

His latest release, Read All About It, responds directly to accusations made by Green's stepmother that his first record was "cashing-in" on her husband's death: "A lot of people face that craving for acceptance and wanting to feel appreciated. How am I not allowed to talk about how that affected me? It was a massive part of my life."

Green shuffles awkwardly along the sofa, and starts to choke up: "In the video there's one scene in particular where the dad holds his son's hand and then lets go. That moment for me is uncomfortable. Yeah, it's not … easy."

The song is the first single from new album, At Your Inconvenience. It's a record that recalls a golden-age of egocentric hip-hop, when Eminem and Biggie had lives worth talking about. But because it's Pro Green, each tender uncomfortable moment is quickly doused in slick British humour (our current favourite line: "I'm only into spooning if it leads to forking").

There's such narrative depth and musical creativity, that it seems bizarre to compare it with recent releases from some of his pap-rap peers. Green, though, is still implicated in the musical crimes of Roll Deep, Chipmunk and their Eurotrash assault on the charts. Does he ever worry that he's fallen in with the wrong crowd?

'I've changed an awful lot since I was 18 and I was a little aggy yoot. But when I lose my temper, I get immature again'

"I'd like to be taken seriously," he says. "People probably do see it as a bit of a fad and a lot of it is quite throwaway. As much as I'm a part of the scene and I support it, I want people to judge me on my own merit."

So have parts of "the scene" started to embarrass themselves? "I think it's time for a change. "People have become too comfortable. They know that if they do a song with this producer and that person on the hook then they'll go to No 1. I suppose some of those people are gonna look back and not be entirely happy with what they did." You sense that he's giving the restrained answer. But isn't that Green's whole attitude now? He's a reflective, polite pop star who is putting his cussing matches behind him.

Green himself is impressed with how the change in his demeanour has been reflected in his audience: "Everyone comes down to the shows: young, old, black, white." In his battling days Green would often tease opponents about their sexuality. But now, "even gay couples come to my shows. That's something that would never have happened two years ago. It surprises me, but I'm happy they can feel comfortable there."

Yet in other ways Green still feels tethered to his past. He talks about a connection with those involved in the riots; not condoning their actions, but understanding their situation. "We come from nothing. You get a little bit of something and you want to show it off," he says. "That's where all of this has come from. I've changed an awful lot since I was 18 and I was a little aggy yoot. But when I lose my temper, I get immature again and I become that person. There's still a part of me that admires that shotter that I saw when I was a kid. Who had the car, who had the girl … who had whatever he wanted."

So perhaps, rather than undergoing a transformation, Green finds himself in one last battle: between the young rapper trying to "merk everyone" and the 27-year old pop star trying to move on. For him, it seems like this is sometimes torture, but from ringside it makes for a spectacular show.

"I have the same insecurities I've had since I was a kid … being unsure of myself; problems that everyone goes through," he admits. "A man's problems are his own, I just have an avenue in which I can moan about it."

Hackney-born rapper Professor Green visits the Guardian studios with his full band in tow to play a special version of Astronaut from his new album At Your Inconvenience – and talk about some of the serious issues behind it