In June 1957, Eric Griffith-Jones, the attorney general of the British administration in Kenya, wrote to the governor, Sir Evelyn Baring, detailing the way the regime of abuse at the colony's detention camps was being subtly altered.

From now on, Griffith-Jones wrote, for the abuse to remain legal, Mau Mau suspects must be beaten mainly on their upper body, "vulnerable parts of the body should not be struck, particularly the spleen, liver or kidneys", and it was important that "those who administer violence … should remain collected, balanced and dispassionate".

Almost as an after-thought, the attorney general reminded the governor of the need for complete secrecy. "If we are going to sin," he wrote, "we must sin quietly."

More than 50 years later, with the imperial endgame long over, evidence of those sins remained quietly concealed in a secret archive within one of the British government's most secure facilities. Set deep in the Buckinghamshire countryside and surrounded by 16ft-high fences topped with razor wire, lies Hanslope Park, home of Her Majesty's Government Communications Centre, where teams of scientists – real-life versions of Q, the fictional boffin of the James Bond films – devise technical aids for the Foreign Office, MI5 and MI6.

What better place to bury Griffith-Jones's letter to Baring, along with thousands more documents from colonial-era Kenya and countless others from 36 other former colonies and protectorates? Were this secret archive to be stacked upright, it would create a tower 200 metres tall. And every document was selected for concealment on the basis of an instruction that nothing should be handed over to any post-independence government that might "embarrass HMG or other government" or cause problems for any colonial policeman, civil servant or member of the armed forces.

Incredibly, perhaps, the Foreign Office maintained until last year that it had no idea its secret archive existed. When lawyers representing five veterans of the Mau Mau rebellion, who were seeking leave to sue the British government in the high court in London, repeatedly demanded disclosure of further documents, the FCO insisted there were none. Finally it admitted the existence of what it termed a "migrated archive" at Hanslope Park.

William Hague, the foreign secretary, conceded that under the obligations of the Public Records Act 1958, the FCO should have assessed the documents and passed any of historical interest to the National Archives at Kew in Surrey. He also commissioned an inquiry by Anthony Cary, the former British high commissioner to Canada, to establish what had gone wrong and what lessons could be learned.

Cary's report (pdf) describes how the files were first stored at Hayes, in west London, before being moved to Hanslope Park, where staff were led to believe that they belonged to another organisation called Hayes and not to the FCO. "According to a canard that was widely shared and passed down during handovers," Cary wrote, the FCO was holding the archive because there had been a fire at Hayes.

Cary said the way the archive was handled "reflected a failure by successive senior managers to grip what should have been seen to be an unresolved and potentially explosive problem".

He concluded that some staff were aware they were FCO files, however, with one retired archivist telling him she was aware that it "might contain potentially sensitive/interesting material which could become the focus of [freedom of information] requests".

When the first batch of Kenyan documents was handed over to the Mau Mau veterans' lawyers, it could be seen to contain material that could only be described as incendiary. The documents detailed the way suspected insurgents had been beaten to death, burned alive, castrated – like two of the high court claimants – and kept in manacles for years.

The papers also showed that ministers and senior civil servants in London were fully aware of the details of the horrific, systemic abuse and murder of detainees during the 1950s emergency – abuses that they had repeatedly told the British public were not happening.

Hague ordered an independent review of the "migrated archive" before its transfer to Kew, overseen by Professor Tony Badger, master of Clare College, Cambridge. The first documents, representing about a sixth of the total archive, are now available at Kew, with Badger promising that very few have been redacted, usually to conceal the identities of informants.

Many historians remain suspicious of the FCO and believe it may seek to retain some of its secret files. Caroline Elkins, the Pulitzer prize-winning historian of the Mau Mau rebellion, warns that the FCO's history of concealment and denial is such that the public should also continue to sceptical.

As the files come available , Badger admits that many of his colleagues wondered whether the FCO was "up to its old tricks again", and adds: "Given the failure of the Foreign Office to acknowledge the existence of the migrated archives, I understand the legacy of suspicion. It is difficult to overestimate the degree of suspicion." But he believes the depth of embarrassment suffered by the FCO over the Hanslope Park scandal offers the best reassurance that it will now finally offer up the full archive.

It may be significant, he adds, that Hague and David Lidington, the junior foreign minister responsible for the transfer process, are both historians and should be conscious of the potential for further "reputational risk" if the FCO continues to conceal documents.

Among the first papers transferred to Kew are a handful of files that show many of the British empire's most sensitive and incriminating documentation was not hidden at Hanslope Park but simply destroyed – sometimes shredded, occasional dumped at sea, but usually incinerated – as the British withdrew from one colony after another.

In a number of colonies, as files were destroyed a certificate was completed and sent to London to show that the job had been done. Could these certificates also be stored at Hanslope Park, providing a glimpse of the contents of each file that was destroyed? The FCO was refusing to say on Tuesday, and insisted that any queries about such certificates should be the subject of an freedom of information request.

Furthermore, Cary's report states that a separate inquiry is now examining the fate of a number of files that were lost or destroyed after they were returned to the UK. The FCO failed to answer a number of questions about that inquiry, stating only that the files remained missing despite an extensive search.

• This article was amended on 6 June 2013 to change the name of Eric Griffiths-Jones to the correct spelling of Eric Griffith-Jones.