There is an instructive aspect, even for home secretaries of more modern times, in the circumstances under which Lord Allen of Abbeydale, who has died aged 95, was appointed as permanent undersecretary at the Home Office in the early summer of 1966.

Roy Jenkins had become home secretary at the turn of the year and was convinced that "a certain Home Office approach to life" had to be broken if he was to avoid what he called "the St Sebastian-like fate" of his predecessors. Dutiful defeat had become the appropriate demeanour for any home secretary, which, in Jenkins' view, was largely due to the hierarchical authority exercised by the then PUS, Sir Charles Cunningham. So he embarked on a "high-noon shoot-out" with his top official, secured his early retirement and his replacement with Sir Philip Allen, as he then was.

Although Allen had joined the Home Office in 1934, having come top in that year's civil service examination, he had worked at the Treasury for three years, attended to the war cabinet and done a five-year stint at the Ministry of Housing and Local Government. "As he had spent a full decade away, he did not count as a troglodyte," Jenkins observed. Jenkins commended Allen as an example of mandarin excellence, which indeed was the mark of his long and high-achieving years as a public servant.

Allen's approach was one of positive social reformism, happily coinciding with Jenkins' view. This was symbolically marked by lifting the air of "inspissated gloom" in the department, not least by replacing the indicator board on which the names of criminals awaiting the death sentence had been shown; a refrigerator was installed in its place.

Allen was already a profound opponent of capital punishment, following two high-profile cases with which he was personally involved. As deputy chairman of the Prison Commission for England and Wales (1950-52), he had advised against a reprieve for Timothy Evans, who was hanged in 1950 for the murder of his baby daughter at 10 Rillington Place, north Kensington. At the time he had expressed the view that Evans was also guilty of the murder of his wife (for which Evans had not been prosecuted). It was later revealed that John Christie, Evans's landlord, was responsible for strangling his wife and five other women, and he also confessed to the murder of Evans's wife and baby. In 1966, 16 years after his death, Evans was posthumously pardoned, and when the Home Office files were published 30 years later, Allen expressed his deep regret at the advice he had given.

Earlier, he had unsuccessfully urged a reprieve for Derek Bentley, who was hanged in 1953 for the murder of a policeman in Croydon, south London. Bentley had allegedly called to an armed friend, Christopher Craig: "Let him have it Chris!" when they were caught trying to burgle a sweet factory. Forty-five years later, in 1998, Bentley received a full exoneration.

Allen was born in Sheffield, the son of Arthur Allen and Louie Tipper. He went on a scholarship to King Edward VII school and developed a lifelong enthusiasm for Sheffield Wednesday, before another scholarship took him to Queens' College, Cambridge, where he took a first in history. He was awarded the Whewell scholarship in international law on graduation and became an honorary fellow of Queens' in 1974.

While already employed in the civil service, he spent a year on a Commonwealth fellowship studying crime in the United States - his expertise was always in criminal justice and prison reform. He was widely regarded as having had an important impact on the Metropolitan police reforms of 1972, following a series of corruption scandals.

Allen retired from the Home Office in 1972, and was made a life peer in 1976 - he had been knighted in 1964 - sitting as a crossbencher. Two years earlier he had been appointed to the Royal Commission on standards in public life, serving until 1976. He was also a member of the Diplock security commission for 18 years (1973-91), investigating, among other things, the salacious circumstances which brought about the resignation of Lords Jellicoe and Lambton in 1973 and the spy scandals inside MI5.

He was a member of the Royal Commission on Civil Liability and Compensation for Personal Injury (1973-78) and of the tribunal of inquiry into the Crown Agents ((1978-82). When Harold Wilson resigned in 1976, Allen was put in charge of the inquiry into how the "Lavender list" of proposed resignation honours was leaked. He also served as chairman of the Occupational Pensions Board (1973-78), the National Council of Social Service (1973-77) and the Gaming Board (1977-85). From 1985 to 1992, he chaired the council of Royal Holloway college.

Allen was a teetotaller - the Tory home secretary Reginald Maudling wrote that "I could never persuade him to consume anything but bitter lemon" - but he did have a taste for speed and excitement in sports cars. His lifetime was spent as an unseen member of the establishment, except for the occasion on which, as returning officer of the 1975 referendum on Britain's continuing membership of the EEC, he announced the result from the platform of the Royal Albert Hall. His wife, Marjorie, whom he married in 1938, died in 2002. They had no children.