Hanslope Park, where the secret archive of colonial papers was held. Photograph: Martin Argles for the Guardian

In June 1957, Eric Griffiths-Jones, the attorney general of the British administration in Kenya, wrote to the governor, Sir Evelyn Baring, detailing the way the regime of abuse at the colony's detention camps was being subtly altered.

From now on, Griffiths-Jones wrote, for the abuse to remain legal, Mau Mau suspects must be beaten mainly on their upper body, "vulnerable parts of the body should not be struck, particularly the spleen, liver or kidneys", and it was important that "those who administer violence … should remain collected, balanced and dispassionate".

Almost as an after-thought, the attorney general reminded the governor of the need for complete secrecy. "If we are going to sin," he wrote, "we must sin quietly."

More than 50 years later, with the imperial endgame long over, evidence of those sins remained quietly concealed in a secret archive within one of the British government's most secure facilities.

Set deep in the Buckinghamshire countryside and surrounded by 16ft-high fences topped with razor wire, lies Hanslope Park, home of Her Majesty's Government Communications Centre, where teams of scientists – real-life versions of Q, the fictional boffin of the James Bond films – devise technical aids for the Foreign Office, MI5 and MI6.

What better place to bury Griffiths-Jones's letter to Baring, along with thousands more documents from colonial-era Kenya and countless others from 36 other former colonies and protectorates?

Were this secret archive to be stacked upright, it would create a tower 200 metres tall.

And every document was selected for concealment on the basis of an instruction that nothing should be handed over to any post-independence government that might "embarrass HMG or other government" or cause problems for any colonial policeman, civil servant or member of the armed forces.

Incredibly, perhaps, the Foreign Office maintained until last year that it had no idea its secret archive existed.

When lawyers representing five veterans of the Mau Mau rebellion, who were seeking leave to sue the British government in the high court in London, repeatedly demanded disclosure of further documents, the FCO insisted there were none.

Finally it admitted the existence of what it termed a "migrated archive" at Hanslope Park.

William Hague, the foreign secretary, conceded that under the obligations of the Public Records Act 1958, the FCO should have assessed the documents and passed any of historical interest to the National Archives at Kew in Surrey.

He also commissioned an inquiry by Anthony Cary, the former British high commissioner to Canada, to establish what had gone wrong and what lessons could be learned.

Cary's report (pdf) describes how the files were first stored at Hayes, in west London, before being moved to Hanslope Park, where staff were led to believe that they belonged to another organisation called Hayes and not to the FCO.

"According to a canard that was widely shared and passed down during handovers," Cary wrote, the FCO was holding the archive because there had been a fire at Hayes.

Cary said the way the archive was handled "reflected a failure by successive senior managers to grip what should have been seen to be an unresolved and potentially explosive problem".

He concluded that some staff were aware they were FCO files, however, with one retired archivist telling him she was aware that it "might contain potentially sensitive/interesting material which could become the focus of [freedom of information] requests".

When the first batch of Kenyan documents was handed over to the Mau Mau veterans' lawyers, it could be seen to contain material that could only be described as incendiary.

The documents detailed the way suspected insurgents had been beaten to death, burned alive, castrated – like two of the high court claimants – and kept in manacles for years.

The papers also showed that ministers and senior civil servants in London were fully aware of the details of the horrific, systemic abuse and murder of detainees during the 1950s emergency – abuses that they had repeatedly told the British public were not happening.

Hague ordered an independent review of the "migrated archive" before its transfer to Kew, overseen by Professor Tony Badger, master of Clare College, Cambridge.

The first documents, representing about a sixth of the total archive, are now available at Kew, with Badger promising that very few have been redacted, usually to conceal the identities of informants.

Many historians remain suspicious of the FCO and believe it may seek to retain some of its secret files.

Caroline Elkins, the Pulitzer prize-winning historian of the Mau Mau rebellion, warns that the FCO's history of concealment and denial is such that the public should also continue to sceptical.

As the files come available , Badger admits that many of his colleagues wondered whether the FCO was "up to its old tricks again", and adds: "Given the failure of the Foreign Office to acknowledge the existence of the migrated archives, I understand the legacy of suspicion. It is difficult to overestimate the degree of suspicion."

But he believes the depth of embarrassment suffered by the FCO over the Hanslope Park scandal offers the best reassurance that it will now finally offer up the full archive.

It may be significant, he adds, that Hague and David Lidington, the junior foreign minister responsible for the transfer process, are both historians and should be conscious of the potential for further "reputational risk" if the FCO continues to conceal documents.

Among the first papers transferred to Kew are a handful of files that show many of the British empire's most sensitive and incriminating documentation was not hidden at Hanslope Park but simply destroyed – sometimes shredded, occasional dumped at sea, but usually incinerated – as the British withdrew from one colony after another.

In a number of colonies, as files were destroyed a certificate was completed and sent to London to show that the job had been done.

Could these certificates also be stored at Hanslope Park, providing a glimpse of the contents of each file that was destroyed?

The FCO was refusing to say on Tuesday, and insisted that any queries about such certificates should be the subject of an freedom of information request.

Furthermore, Cary's report states that a separate inquiry is now examining the fate of a number of files that were lost or destroyed after they were returned to the UK.

The FCO failed to answer a number of questions about that inquiry, stating only that the files remained missing despite an extensive search.

Hanslope Park, where the Foreign Office kept a secret archive of colonial papers. Photograph: Martin Argles for the GuardianThousands of documents detailing some of the most shameful acts and crimes committed during the final years of the British empire were systematically destroyed to prevent them falling into the hands of post-independence governments, an official review has concluded.

Those papers that survived the purge were flown discreetly to Britain where they were hidden for 50 years in a secret Foreign Office archive, beyond the reach of historians and members of the public, and in breach of legal obligations for them to be transferred into the public domain.

The archive came to light last year when a group of Kenyans detained and allegedly tortured during the Mau Mau rebellion won the right to sue the British government.

The Foreign Office promised to release the 8,800 files from 37 former colonies held at the highly-secure government communications centre at Hanslope Park in Buckinghamshire.

The historian appointed to oversee the review and transfer, Tony Badger, master of Clare College, Cambridge, says the discovery of the archive put the Foreign Office in an "embarrassing, scandalous" position. "These documents should have been in the public archives in the 1980s," he said. "It's long overdue."

The first of them are made available to the public on Wednesday at the National Archive at Kew, Surrey.

The papers at Hanslope Park include monthly intelligence reports on the "elimination" of the colonial authority's enemies in 1950s Malaya; records showing ministers in London were aware of the torture and murder of Mau Mau insurgents in Kenya, including a case of aman said to have been "roasted alive"; and papers detailing the lengths to which the UK went to forcibly remove islanders from Diego Garcia in the Indian Ocean.

However, among the documents are a handful which show that many of the most sensitive papers from Britain's late colonial era were not hidden away, but simply destroyed.

These papers give the instructions for systematic destruction issued in 1961 after Iain Macleod, secretary of state for the colonies, directed that post-independence governments should not get any material that "might embarrass Her Majesty's government", that could "embarrass members of the police, military forces, public servants or others eg police informers", that might compromise intelligence sources, or that might "be used unethically by ministers in the successor government".

Among the documents that appear to have been destroyed were: records of the abuse of Mau Mau insurgents detained by British colonial authorities, who were tortured and sometimes murdered; reports that may have detailed the alleged massacre of 24 unarmed villagers in Malaya by soldiers of the Scots Guards in 1948; most of the sensitive documents kept by colonial authorities in Aden, where the army's Intelligence Corps operated a secret torture centre for several years in the 1960s; and every sensitive document kept by the authorities in British Guiana, a colony whose policies were heavily influenced by successive US governments and whose post-independence leader was toppled in a coup orchestrated by the CIA.

The documents that were not destroyed appear to have been kept secret not only to protect the UK's reputation, but to shield the government from litigation.

If the small group of Mau Mau detainees are successful in their legal action, thousands more veterans are expected to follow.

It is a case that is being closely watched by former Eoka guerillas who were detained by the British in 1950s Cyprus, and possibly by many others who were imprisoned and interrogated between 1946 and 1967, as Britain fought a series of rearguard actions across its rapidly dimishing empire.

The documents show that colonial officials were instructed to separate those papers to be left in place after independence – usually known as "Legacy files" – from those that were to be selected for destruction or removal to the UK. In many colonies, these were described as watch files, and stamped with a red letter W.

The papers at Kew depict a period of mounting anxiety amid fears that some of the incriminating watch files might be leaked.

Officials were warned that they would be prosecuted if they took any any paperwork home – and some were.

As independence grew closer, large caches of files were removed from colonial ministries to governors' offices, where new safes were installed.

In Uganda, the process was codenamed Operation Legacy.

In Kenya, a vetting process, described as "a thorough purge", was overseen by colonial Special Branch officers.

Photograph: The National ArchivesClear instructions were issued that no Africans were to be involved: only an individual who was "a servant of the Kenya government who is a British subject of European descent" could participate in the purge.

Photograph: The National ArchivesPainstaking measures were taken to prevent post-independence governments from learning that the watch files had ever existed.

One instruction states: "The legacy files must leave no reference to watch material. Indeed, the very existence of the watch series, though it may be guessed at, should never be revealed."

When a single watch file was to be removed from a group of legacy files, a "twin file" – or dummy – was to be created to insert in its place.

If this was not practicable, the documents were to be removed en masse.

There was concern that Macleod's directions should not be divulged – "there is of course the risk of embarrassment should the circular be compromised" – and officials taking part in the purge were even warned to keep their W stamps in a safe place.

Many of the watch files ended up at Hanslope Park.

They came from 37 different former colonies, and filled 200 metres of shelving.

But it is becoming clear that much of the most damning material was probably destroyed. Officials in some colonies, such as Kenya, were told that there should be a presumption in favour of disposal of documents rather than removal to the UK – "emphasis is placed upon destruction" – and that no trace of either the documents or their incineration should remain.

When documents were burned, "the waste should be reduced to ash and the ashes broken up".

Some idea of the scale of the operation and the amount of documents that were erased from history can be gleaned from a handful of instruction documents that survived the purge.

In certain circumstances, colonial officials in Kenya were informed, "it is permissible, as an alternative to destruction by fire, for documents to be packed in weighted crates and dumped in very deep and current-free water at maximum practicable distance from the coast".

Photograph: The National ArchivesDocuments that survive from Malaya suggest a far more haphazard destruction process, with relatively junior officials being permitted to decide what should be burned and what should be sent to London.

Dr Ed Hampshire, diplomatic and colonial record specialist at the National Archive, said the 1,200 files so far transferred from Hanslope Park represented "gold dust" for historians, with the occasional nugget, rather than a haul that calls for instant reinterpretation of history.

However, only one sixth of the secret archive has so far been transferred.

Chevron-shaped Object Seen over Longview, Texas Published: 1:36 PM 4/12/2012 Longview, Texas - 04-11-12 I was heading Eastward on I-20 at 8:30 P.M. on Thursday the 5th of April 2012. I was passing... more

The European Stabilization Mechanism, Or How the Goldman Vampire Squid Just Captured Europe Posted on April 18, 2012 by Ellen Brown The Goldman Sachs coup that failed in America has nearly succeeded... more