As the vigil continues outside the hospital, we don't know how close to the final freedom Nelson Mandela is. But after the strange denials that this old, sick man is dying I want to talk not with pity but of his power. Before the pygmy politicians line up to pay tribute to this giant, I want to remember how he lived so much for so many. Part of my memory is that he was not a living saint to the very people whose staff will now be writing their "heartfelt" speeches.

Really, I have no desire to hear them from leaders of parties who described his organisation as terrorist, who believed that sanctions were wrong, whose jolly young members wore T-shirts demanding he be strung up. Of course, not all Tories were pro-apartheid, but I can already feel the revisionism revving up.

So we must recall how it really was. The struggle against apartheid was the one thing that unified the left. I came to it accidentally. Isn't that how politicisation happens sometimes? Via extraordinary people, unlikely meetings, chance encounters?

Like this one: in 1981 I had just come back from travelling around South America and got a job in a care home with Haringey Social Services in north London. Some of the local kids were in big trouble – the girls were on the game at 14, the boys breaking into houses and stealing cars. A large, in every sense of the word, African woman became my ally there. She was always encouraging them to be lawyers despite their constant truanting. We were an unlikely pair, but she believed in "discipline" and I believed in "manners" so we would talk late into the night. She was one of the poshest people I had ever met – she drank Perrier water, which at that time was exotic beyond belief. Sometimes she would weep after receiving calls from South Africa and talk of murders and assassinations. Sometimes she would take me out for cocktails and get diplomatic cars from embassies to take me home. Her name was Adelaide Tambo, the wife of Oliver. They were the exiled leaders of the ANC.

I began to know what this meant. How Mandela had ridden to power in 1952 in the Defiance Campaign, how he was harassed and, of course, finally taken to Robben Island. To that tiny cell. The Tambos had to leave much later. One night she called me as she was locked out of her house in Muswell Hill. "Can't you just break a window? "No Suzanne," she said. "The windows are all bullet proof glass." That's how they lived.

This personal introduction to the ANC is my story but everyone I knew opposed apartheid. Indeed, who could support such barbarism? This was more than racism – there is only one race, called the human race. Botha's regime did not regard black people as humans but as animals.

By 1984 Jerry Dammers had written Free Nelson Mandela. But apartheid continued to exist, propped up by the Tories. Some of their elder statesmen, such as Norman Tebbit, still see Thatcher's policy as a success. David Cameron denounced it in 2006, saying she had been wrong to condemn the ANC as terrorists and to have opposed sanctions. Too late for those veteran campaigners such as Peter Hain, who had seen the massacres in the townships and knew it was a life-or-death struggle.

Indeed, when I saw Mandela in later years having his garden surreally being "made over" by Alan Titchmarsh or being cuddled by random Spice Girls, I wondered if they had ever heard Gil Scott-Heron's Johannesburg (1975) or been at the anti-apartheid demos outside the South African embassy where we were all kettled.

When we hear Cameron's inevitable tribute, don't forget that in 1989, aged 23, he went on a "jolly" to South Africa paid for by a firm that did not want sanctions busted. This does not mean he supported apartheid, but by then it would have been impossible not to know of the regime's brutality. Many people knew, and boyotted South African goods.

I see Dylan Jones, a Cameron fan, has written a book on Live Aid, defining the 80s as caring: more anodyne revision. The key concert of the 80s was the more political and consciousness-raising Free Nelson Mandela one, not long after. Mandela himself was there on stage with that smile that came from the centre of the Earth. The glare of his grin made us cheer and cry. The glare of the sun, when he was breaking rocks in Robben Island, had permanently damaged his eyesight, but not his mind. When he walked to freedom he wrote, that unless he left bitterness and hatred behind, "I would still be in prison."

This is wonderful, but do not let his story be rewritten, do not let those who opposed his struggle pretend they didn't.

"There is no passion to be found playing small," he said. He told his own people to recall the past. I ask simply, before we are inundated with those who want to bask in his afterglow, that we remember our own past too. It is sad, but let him go. I just wanted to remind you of how it was before he passes and before the "official" rewrite of history begins. Forgiveness is possible. Forgetting isn't.

Alf Kumalo chronicled the Mandela family for 60 years, photographing the children as they were growing up and the man himself after his release. David Smith met the photojournalist before his death in 2012