Some of the Kenyans seeking compensation from Britain for their Mau Mau-era torture: from left, Wambugu Wa Nyingi, who spent 10 years in detention, Jane Muthoni Mara, who was raped in detention aged 15, and Paulo Nzili, who was castrated by his captors. Photograph: Peter Macdiarmid/Getty

Secret government files from the final years of the British empire are still being concealed despite a pledge by William Hague, the foreign secretary, that they would be declassified and opened to the public.

The withheld files are among a huge cache of documents that remained hidden from view for decades at an undisclosed Foreign and Commonwealth Office (FCO) archive, in breach of laws governing the handling of official papers. Once the existence of the archive became known to lawyers for a group of elderly Kenyans who are trying to sue the British government over the abuses they suffered during the Mau Mau insurgency, Hague ordered an inquiry and promised disclosure.

He told MPs: "I believe that it is the right thing to do for the information in these files now to be properly examined and recorded and made available to the public through the National Archives. It is my intention to release every part of every paper of interest subject only to legal exemptions."

However, it emerged this week that the Foreign Office is holding back significant numbers of documents, using a legal exemption contained within a catch-all clause within the very law that it had breached by maintaining the secret archive.

Among the papers known to have been withheld by the FCO are a file that contains the minutes of many of the meetings of the cabinet of the British colonial government in Kenya in 1963, the year before independence; a file about compensation that was paid after the 1946 bombing of the British military HQ at the King David Hotel in Jerusalem; and a file containing telegrams that British diplomats sent to London from Mauritius in 1968, the year that colony proclaimed independence.

Other files that are known to remain hidden from public view include a series of five concerning a visit that Prince Philip made to Singapore in 1956; three concerning a proposed royal visit to Nigeria the same year; and a proposed royal visit to Mauritius in 1968.

Sections of some files that have been declassified and handed over the National Archives at Kew, south-west London, are also being withheld. These include parts of a 1950 file about the "indoctrination of Malay Chinese" travelling to China, which the Foreign Office wishes to remain concealed until 2032; Singapore intelligence reports from the 1950s; and a 1960 file concerning Northern Cameroons, both of which the Foreign Office plans to withhold until 2029. These files remain classified under the terms of Section 3.4 of the 1958 Public Records Act, which permits government departments to withhold from public view any historic document "required for administrative purposes" or that "ought to be retained for any other special reason".

The Foreign Office declined to disclose the reasons for deciding that specific files should be withheld, with the result that the reason for the continuing secrecy is itself, at this stage, a secret. A Foreign Office spokesperson said: "We are committed to making as many files as we can from our colonial archives available to the public. These files are an important part of our history. By the end of our programme, we'll have released some 18,000 files. On average, only 1% of material has been withheld from release."

The secret archive was discovered by a number of historians, including David Anderson, professor of African politics at Oxford, who were working for the legal team that represented the Mau Mau veterans as they fought for the right to sue to the British government for the mistreatment they suffered as prisoners of the British in 1950s Kenya. The government had acknowledged that their accounts of appalling torture are true, but is continuing to contest their right to claim damages, saying that a fair trial is impossible after so many years.

During a series of hearings the Foreign Office acknowledged that the archive existed, and that it contained 200 boxes of files: 1.5 tons of paper that covered about 200 metres of shelving. Lord Howell, then Foreign Office minister, said that the documents had been brought to the UK not only from Kenya, but from 37 different colonies and protectorates including Aden, Ceylon, Cyprus, Malaya, Malta, Nigeria and Northern Rhodesia.

For several years the secret archive was housed at Hanslope Park in Buckinghamshire, inside the highly secure premises of Her Majesty's Government Communications Centre. This is a facility where teams of scientists – real-life versions of Q, the fictional boffin of the James Bond films – work behind five metre-high fences topped with razor wire to devise technical aids for MI5 and MI6.

An inquiry commissioned by Hague established that the files were first stored in Hayes, west London, before being moved to Hanslope Park, where staff were led to believe that they belonged to another organisation called Hayes and not to the FCO.

Anthony Cary, the former British high commissioner to Canada, who conducted the investigation, reported that "a canard that was widely shared and passed down during handovers" included the explanation that the FCO was holding the archive because there had been a fire at Hayes. In this way it was never assessed for declassification, as the law required, and placed beyond reach of the Freedom of Information Act.

Once the bulk of the material began to be handed over to the National Archive, it became clear that many of Britain's late colonial-era official papers were not archived anywhere, but destroyed.

Thousands of documents containing evidence of official misdeeds committed in the years in the years before independence were either incinerated or dumped at sea. Surviving scraps of documents show that colonial officials in some colonies, such as Kenya, were told that there should be a presumption in favour of disposal of documents rather than removal to the UK – "emphasis is placed upon destruction" – and that no trace of either the documents or their incineration should remain. When documents were burned, officials were told that "the waste should be reduced to ash and the ashes broken up".

Some idea of the scale of the destruction operations can be gleaned from instruction documents that survived the purge. In certain circumstances, officials in Kenya were informed, "it is permissible, as an alternative to destruction by fire, for documents to be packed in weighted crates and dumped in very deep and current-free water at maximum practicable distance from the coast".