We learned the big and the small about Sol Campbell when his book, Sol Campbell: The Authorised Biography, came out in March. The former footballer expressed his belief that he would have captained England more often were he not black – that was big, a statement fiery enough to get him booked on to Newsnight, Victoria Derbyshire and the Today programme.(Hat trick!) We also learned from the book that Sol, 39, still knows his first numberplate off by heart and that, as a boy, he "never left a single flake" of his Special K. In among this lesser biographical bric-a-brac was the fact that Campbell prefers to take professional meetings close to his Chelsea home, and that, by habit, he's often 15 minutes late.

Sure enough, when we meet on a sunny June day, it is at the door of his Chelsea home, and he is 15 minutes late. The Authorised Biography contains a whole, heavy page on Sol haggling down the price of his first conservatory – the book making clear that he's proud of having been a financially literate footballer, a defender who negotiated some of his own contracts, and I supposed before meeting him that this 15-minutes-late business was some sort of learned technique, a subtle bit of power posturing. Actually, after saying hello, it's immediately obvious he's far too open a guy for anything so underhand. A guts-on-the-table sort.

After kissing one of his children goodbye Campbell slings a blue blazer over his shoulder, steps out, and we're hardly 10 paces along Chelsea Embankment before he's talking through his timescale for having more kids: two, ideally soon. He's not quite clean shaven, his cropped hair a little grown out, and white bristles show through. It's something Campbell attributes to the stress of running an interior design business with his wife, Fiona, but I can't see it. He seems relaxed.

I wasn't expecting this, I tell him, as we walk towards Battersea Park and a bench. In the pictures you glowered.

"The only pictures you saw of me when I was playing, I was crashing into a centre forward!" he says. "Or running down a line to save my team. I'm not gonna be happy, be smiling." Does he feel happier now, since retiring in 2012? Though a winner of major trophies, and a scorer for England in the 2002 World Cup, Campbell had a difficult career. His move from Tottenham to club rivals Arsenal in 2001 began a decade-long campaign of abuse from upset fans that was essentially unhinged in terms of its degree and its duration. Really, Campbell's hair might have begun to turn white way back then, in about 2002, when Spurs supporters started chanting merrily about the chances of him having Aids...

"Look at me then, and look at me now, I'm two different people," Campbell says. "I don't want to be the same person I was on the pitch. I was an angry man, focused, wanted to win… Generally, now, I'm a happy guy. Dry sense of humour. I love the banter. I have a laugh. Of course, I'm gonna be a little bit guarded for press…"

He isn't, luckily. Guarded. As we walk in the park Campbell chats about the need for a "bloodbath" in the FA following England's World Cup exit. Bring out the guillotine, he says. He gossips about former colleagues and even offers theories about how to solve the developing international crisis in the Crimea – ban Russia from the Olympics, maybe.

Campbell has agreed to meet the Observer to promote the work of the Black Cultural Archives, a London-based organisation that celebrates the history of black Britons. "Nothing like this was around when I was young. Nothing I knew of, anyway." Campbell grew up, he says, having no real clue about the depth and variety of black British culture. "It's not only about slaves. There were guys who were very wealthy. A lot of the pub landlords, back in the day, were black. A lot of fantastic artists. Sports men and women. The truth has always been hidden, for whatever reason, and I don't understand that… Why are we learning about Egyptians [in schools]? Why are we learning about Romans? Hello! There's a big area here we're not learning about."

This month, the Black Cultural Archives moves to a new museum building in Brixton, south London. Settling on a bench by the river, shades on against the sun, Campbell talks about items from the archive's collection that have caught his interest. On a recent visit he found a picture of an Edwardian football team, a lone black player lining up among the first XI in 1908. Once a kid who used to plaster his walls with posters of black footballers, the postcard tickled Campbell. He was also captivated by the story of Mary Seacole, a Jamaican woman who supported British troops on the front line during the Crimean war. "And there were others, black people helping this country in its wars, and that has to be celebrated, that has to be told. We did play a part."

Campbell announces, with honest feeling, that the archive's relocation to a new, better facility is "the biggest thing to happen to this country". The biggest thing in terms of…? "History! Museums! The biggest thing to happen for black history in this country."

A photograph of a 1908 football team that Campbell found among the Black Cultural Archives.

This is his way. Exuberant, unpredictable, a little wild. It makes him intriguing company. And the man knows how to boast – full bore, no false modesty. He drops into conversation, twice, that he was once considered the best defender in the whole world; that, without his mother and father moving to the UK from Jamaica to find work, "I wouldn't be here, I wouldn't have played for my country, and represented it in a fantastic way. I've contributed tenfold, even more than that, in terms of what I've done in the game, and that would have been lost." He plans to offer a copy of his biography, he says, to the cultural archive.

Unlike the vast majority of his colleagues in sport, Campbell doesn't automatically edit away any interesting statements before they're out. "Yes, I was a footballer," he says, "but there's more to me than meets the eye. I've got more levels. So many levels. I'm not going away – and I'm going to keep on talking until things start changing."

This issue of change is important to him. In retirement, while running his design business with his wife, and raising their two children (Campbell has a third, older child from a previous relationship), and studying for coaching badges, Campbell has set out to try to improve the racial imbalance in English football. Not easy.

On the surface our national sport is diverse, 22 men trotting out on to pitches in London and the Midlands and Manchester and Tyneside every weekend, a great many skin colours, and backgrounds, and religions represented. But look closer, look up. The game is white at the top. At the start of last season, only two out of 92 clubs had black managers. Campbell's has been one of the loudest voices complaining about this.

"I don't want to sit by the edge, like some other guys who have played their game. Earned money, have nice lifestyles, and don't say anything."

As a result he's taken on a curious role, as a kind of freelance pundit, not often formally invited on to BBC or ITV sofas to dissect games, but chucking verbal grenades into conversations around football anyway, usually via the news programmes. Last October, Campbell was among the first to point out that a newly established FA commission – formed with a brief to better domestic football – was all white. And therefore, of course, all wrong.

Sol Campbell playing for Spurs in 1993. Photograph: Colorsport/Rex

Right from its start, Campbell's career was influenced by racial clumsiness, outright racism. As a young teen he trained informally with West Ham, the club closest to his family home in Stratford, east London. He recalls playing a practice match there when he was 13. "I'm tired, I'm not going to be smiling, and this guy comes up to me, who worked at West Ham, I think he still works there, and he said: 'Cheer up, Sol, you're 2-1 up.'

"I thought he must have meant the practice match we'd just played. But he said: 'In the cricket. The West Indies are 2-1 up against England.' " The inference, as Campbell took it, was that "I'm not English. And this was where it got really complex. He was mixed race. So I'm black, he's mixed race… In his eyes, I'm not English. I'm not English." Campbell repeats this as if he's still trying to work out the logic of it.

What happened next? "I left and never went back. Quite strong, at 13 years old, but when it's right and it feels right I'll do it. I wasn't going to stand around and train with people like that."

According to the writer of Campbell's biography, Simon Astaire, Sol's older brother also left a youth team – Charlton Athletic's – because of a racist remark. John Campbell, "who could have made it", Sol thinks, never got his career restarted afterwards. Sol fared better, switching to train with Tottenham Hotspur. In 1993, he formally graduated from Spurs' youth team to the senior side – with whom, during a match against Sunderland, he was subjected to something much more brazen than a comment about cricket. "I played right back that day," Campbell recalls, "and every time I touched the ball – monkey chants. Every single time."

This was in January 1995. I'd naively imagined that this nonsense had been shamed out of British football by then. "Mad thing is," Campbell says, "the captain of Sunderland was black." (The captain, Gary Bennett, asked on Twitter recently if he remembered the chants, said: "No, not at all I was busy making myself big in the goal.")

What did Campbell say to his team-mates at the time? "I was a youngster, what am I going to talk about?" Didn't they hear the monkey noises? "No one said anything. It was my first or second season in football. They [the opposition fans] might have been trying to put me off. But now I'm older I think, fucking hell, what was going on there? Whole sections of the crowd doing monkey chants – and no one said anything… I was young. I was just happy to be in the team. But no one came up to me to say, 'You all right, Sol?' "

That none of his team-mates offered support or sympathy seems odd. I wonder if it's possible, having read his book, that Campbell's preference for solitude when he was younger made the situation worse. A former coach says in the book: "Whenever we travelled away and there was a spare single room on offer, Sol was the first to put up his hand."

You get a sense, reading, that he was not the most popular guy at his football clubs. After scoring his first goal for Tottenham it is mentioned, casually, that nobody joined him to celebrate. And his move from Spurs to Arsenal in 2001 was such a dramatic event, in part, because the press didn't know a thing about it until the deal was done; unthinkable discretion in a gossip-rippled sport, made possible because Campbell let no peers in on the secret. Tony Adams, an ex-team-mate, told Campbell's biographer: "If you can get into Sol's head, you're doing a far better job than I ever did."

When Campbell's feelings about England and the captaincy were publicised in March ("I believe if I was white", the controversial sentence ran, "I would have been England captain for over 10 years") other ex-team-mates including Martin Keown, Paul Merson and Ian Wright spoke against the claim. Paul Ince and John Barnes, half a generation up, contradicted Campbell too. He has since qualified the words, shying from that mention of 10 years, but insisting, still, that he would have been captain more often were race not a factor. Was he disappointed that no black players – no players at all, as far as I can tell – stepped out to support him?

"Yes. Big time," he says. "There's guys who've had bananas thrown at them [which happened to John Barnes in the 80s]. You almost say to yourself: who are you? Are you actually black, or what? You've actually gone through it. Bananas being thrown. Can't get a job in the game. Do you understand what's actually happening to you?"

I wonder if Campbell might have received more support, or at least less of a kicking, if he had closer friends in the game. A cuddlier public image. While we sit in Battersea Park a jogger diverts his route for a running high-five with Campbell and shouts, "Respect!" But, honestly, widely, and especially when compared with a Barnes or a Wright or an Adams, Campbell doesn't command a huge amount of public fondness. It's something he's aware of.

"Football-wise, I'm more respected around the world than in my own country. It's true. If I had this type of career somewhere else, I'd probably be a sir in that country by now."

His voice rises in pitch. He's half joking, half not. "I've got no OBE, MBE, nothing. And no one says anything… How can I have this career? And be who I am? And no one's said anything? To get an MBE, just three people have got to write in. Three people have never written in about Sol!"

Campbell playing for Arsenal in the 2006 game against West Ham in which he drove away from the stadium at half time. Photograph: Javier Garcia/Back Page Images

Along with some excellent gossip and a few hash-settling insults ("I had muppets as team-mates who were on treble the money"), there are significant disclosures in the authorised biography that peer deep into Campbell's character.

Sulzeer – Sol's given name – was a nine-year-old who used to get annoyed at other nine-year-olds who messed around in the swimming pool, because he wanted to improve his stroke. He was the only child in a classroom full of howling pupils to stand up and tell the teacher his flies were open. As a teenager he changed his name from Sulzeer to the simpler Sol after scoring in a youth game. The announcer botched his name and Campbell thought: "Enough of this."

Campbell was the youngest of 12 and his parents, Wilhelmina and Sewell, appear in the book as hard-working, providing parents. They were not demonstrative with their affection, however. When Sol scored on his debut for Tottenham, it was not only his team-mates who left him to celebrate alone – the goal went unmentioned at home. Nobody congratulated him on becoming captain of Spurs either, and in fact his father used to claim he only found out about it when it was mentioned at the pub. This wounding story comes up twice in the book. Later there are sad pages that recount Sewell Campbell's death and how Sol, after a lifetime being kept at a physical and emotional distance, was able to touch his father's face for the first time.

After Campbell's transfer from Spurs to Arsenal, his brother John got into a fight with someone who'd slighted Sol. John was jailed for assault in the aftermath and "we never as a family discussed what had happened", Sol says in the book. Later, when he returned to Tottenham for the first time as an Arsenal player, he looked out into the stands and saw that another of his brothers, Tony, was in among the jeering Spurs fans. The two hardly spoke again.

Campbell was under all sorts of strain in the wake of that move – he tells me he was so miserable he considered leaving Arsenal after six months – and there were eruptions later. Half way through a torrid game for Arsenal against West Ham in 2006, Sol asked his manager to substitute him. Rare enough. But then he drove away from the stadium at half time, still wearing his number 23 shirt, and fled to Brussels without telling anybody. When I ask about Brussels, an incident referred to as "the lost weekend" in his book, Campbell says: "I look back now and say to myself, that was the build-up, absolutely. The build-up of all the pressure… It was too much. For one man, it was too much."

The biggest source of pressure was the terraces. He took abuse from spurned Tottenham fans, certainly, but from other fans too, who seemed to target Campbell simply because the precedent was there to do so. Many of the popular anti-Sol chants of the day riffed on the rumour that he was a closet homosexual. Not true, says Campbell (something he once had to tell a representative from Puma, hopeful about endorsing football's first openly gay player).

Rival fans, says Campbell, "were abusing my life, abusing my name, abusing my liberty, abusing who I am. Blackening my name continually." It was only after his "lost weekend" in Brussels that he first thought about making a larger complaint about this. It took a couple of years, but in 2008 "I got external institutions involved, ie the police. Because the FA weren't doing anything. The Premier League weren't doing anything. Newspapers weren't doing anything."

A footballer calling the cops about what he was hearing from the stands was unprecedented. Looking back, it marked a shift in accepted standards of behaviour in stadiums. Crude insults didn't vanish from the sport after Campbell acted in 2008. Crude insults still haven't vanished from the sport. But today, at least, there isn't a regular football-goer left who can plausibly claim not to know where the limits lie, and what the punishments are (bannings, arrest) when those limits are overstepped. It's something. Campbell's contribution was vital.

As he'll tell you, by the way, often. "Ultimately, I started the ball rolling," he says. And: "People stuck their heads in the sand, hoping it would go away. But it didn't go away, not until I started pursuing it." And: "I started to think: how do I turn this around? How do I stem the flow? That's what I started to do, on my own. No PR. No X,Y,Z. Me!"

Towards the end of our conversation I try to address this, Campbell's tendency to be a bit boastful. A frank, unpredictable guy, he stopped being invited on to TV to be a pundit a while ago. In order for you to properly tackle (I begin) the problems surrounding race in football...

"One man can't do it all," he says.

...wouldn't it be helpful if you were invited to speak on TV more often? If you were, for want of a better word, safer? Is your campaigning work actually hindered by you being a contentious guy, an open talker?

"What do you mean, I'm not safe? When I talk, I talk sense."

You're not bland, Sol. Telly needs bland.

"I'm colourful. I always like to fight for things. And this is worth fighting for. I'm not just gonna sit down and be the same, like the others. It's a calling. Some people have this naturally inside them, that they will do the right things, that they will stand up for the right things. For me? I don't care if I don't ever get a job on TV, in management, whatever. I'd rather get a job being me. You go further being you."

Last August, at the start of the new season, there were two black managers in English football; both had lost their jobs by March. As a new season approaches, there are none. Not one, out of 92 teams, when a quarter of all professional players in the domestic game are black. How to fix this? In American football, its ratios similarly askew, there have been moves to insist that a set number of black candidates are interviewed for coaching positions. Campbell puts forward something else: more of a hope than a plan. He expects an African billionaire to buy a British club before long, and maybe then, when someone black is signing the cheques...

We finish by talking some more about the Black Cultural Archives, the reason for our meeting in the first place. As well as that photograph of the Edwardian footballer, and the story of Mary Seacole, Campbell says he was struck by an old coin he found in the archive's collection. It showed the face of the Roman emperor Septimius Severus, black and African-born. "A black Roman emperor!" says Campbell. "I had no idea." It prompts him to rather a good joke. No black managers in English football. No black chairman of the FA or of the Premier League to date. But the ancient Roman Empire. Now there was diversity.• This article was amended on 16 July 2014 to correct the date of a match between Tottenham Hotspur and Sunderland and to include a comment on Twitter from the former Sunderland captain, Gary Bennett.