For more than 40 years, Bill McCarthy, who has died aged 87, tracked the progress of Britain's industrial relations from his base at Nuffield College, Oxford. He charted the growth of shop-steward power in the 1960s, was enmeshed in the struggles of Labour and Conservative governments to regulate the unions in the 1970s and 80s, and, after the Thatcherite revolution swept away many of the assumptions of his working life, tussled with the TUC and Labour to find an industrial relations consensus. His understanding of the intricacies of collective bargaining guaranteed him a role as an arbitrator of knotty disputes, from train drivers to teachers, and later as a Labour spokesman on employment in the Lords.

McCarthy was brought up in Islington, north London, attended Holloway county school and became an assistant in a gentleman's outfitter. The shopworkers' union Usdaw recognised a precocious intelligence and secured him a scholarship to Ruskin College, Oxford, in 1953. Glittering progress followed, though his self-confidence did not always endear him to others. After a distinction in his diploma, there was a first at Merton College in philosophy, politics and economics before, in 1958, he embarked on a DPhil at Nuffield, becoming a research fellow the following year. He remained at Nuffield as faculty fellow from 1969 and emeritus fellow from 1992.

At Ruskin he met Margaret Godfrey, the daughter of an Oxford midwife. They married in 1957. It was a close partnership; she encouraged him when his self-belief faltered and they battled side by side in the Oxford Labour party, where both held office. From Ruskin, too, came a lifelong friendship with Ruth and Derek Gladwin, later a major figure in the GMB union and chairman for years of the Labour party's conference arrangements committee.

McCarthy's DPhil was on the closed shop: it was a timely look at the growing influence of shop-steward power amid rising political worries about the breakdown of centralised union control and the growth of "unofficial" locally endorsed action, labelled in newspaper headlines as "wildcat strikes". When this concern was translated into Lord Donovan's Royal Commission on Trade Unions and Employers' Associations by Harold Wilson's government in 1965, McCarthy was an obvious candidate for research director. His colleagues remember his energy and enthusiasm and the quality of the research papers, while the report's famously permissive conclusions, against restrictive legislation, in favour of union involvement in broader business questions, chimed with his convictions.

These broadly followed the "Oxford School", a group around Allan Flanders and Hugh Clegg, who became McCarthy's supervisor and mentor. Flanders had emphasised the importance of organisations involving trade unions, insisting that it was not so much outcomes as the extent of involvement which mattered. The idea of a trade-off between giving unions a say on broader national issues and demanding greater responsibility became a critical theme for McCarthy.

As prices and incomes policy continued to preoccupy the Labour government, McCarthy was summoned to the economic research department of the Department of Employment and Productivity set up by Barbara Castle in 1968. His appetite for the political inside track took him deep into the government's discussions about statutory intervention in collective bargaining. Although this had been rejected by Donovan, ministers were now increasingly drawn to it as a response to growing industrial action.

McCarthy was at the Sunningdale conference in 1968 when Castle unveiled her blueprint for what would become the In Place of Strife policy – an attempt to regulate union behaviour by sanctions including cooling-off periods, in exchange for extending workers' rights. He noted: "She came down, this tiny little person and sat in this great chair, and it was marvellous; it was how we were going to thread our way through all these difficulties and she asked me to write it."

But the consequences were bitter. The draft, Partners in Progress, was attacked in cabinet and by the TUC after it was leaked. Redrafted as In Place of Strife, it came up against the "terrible twins", the new trade union leadership of Jack Jones and Hugh Scanlon, who joined with Jim Callaghan, then home secretary, and party doubters to defeat the idea of legislation. Instead an agreement was cobbled together, with the commitment of the TUC to take effective action on unconstitutional strikes.

As an industrial relations adviser and arbitrator, McCarthy had to work hard to regain the confidence of the big union battalions. Ironically, his own views were permissive: "We have no certainties to put in place … what we offer is merely a greater appreciation of the limits and constraints [on change in trade unions]," he wrote in 1981, shortly before Margaret Thatcher shifted the landscape irrevocably.

His academic colleagues saw him as a great empiricist, without a strong theoretical framework, fascinated by minutiae. "Give him an obscure rule book," said one friend, "and Bill is never happier." For 30 years he produced a book almost every year. The role of shop stewards or employers associations, wage inflation, New Labour at work, employment legislation, were all analysed. He was recognised as an outstanding teacher, but preferred practice to theory, and was regularly called on to help with union mergers and for arbitrations and inquiries. But he would resist being named as the union choice, guarding his independence, in spite of his transparent politics.

He and Margaret were renowned for their work as canvassers, and Bill's chairmanship of the Oxford Labour party in the 1960s and 70s was marked by a mastery of procedure and the way he moderated between vociferous proponents of opposing beliefs from CND to proto-Social Democrats. His own views were centrist; a close friend of Tony and Susan Crosland, he was most comfortable with moderate union leaders such as Gladwin, David Basnett and Sid Weighell. His wider political career was subject to Labour's factionalism, and ended in some disappointment. Elevated to the Lords in 1975 by Wilson to buttress its industrial relations expertise, he was cold-shouldered by Callaghan, with whom he had clashed years before over immigration policy and union reform. A man of strong likes and dislikes, and a sharp tongue, he would tell friends, "never trust a word that Jim says."

After the Thatcherite assault on union rights, McCarthy and Bill Wedderburn, his legal academic counterpart, became a powerful double act as TUC advisers and in the Lords. But the arrival there of heavyweight union leaders such as David Basnett and Muriel Turner diluted their influence. Unimpressed when Tony Blair became the party's employment spokesman, he was still hurt when Blair declined to give him a frontbench post when he became prime minister. After the Iraq war, he would describe Blair as worse than Ramsay MacDonald.

Shaped by the 1960s and 70s Labour party, he found some difficulty in adapting to new currents in Labour and TUC thinking. An opponent of entry into the Common Market, he remained sceptical, even scornful, of European trade union activity which increasingly absorbed the TUC. He remained in his element, and in demand, dealing with complex arbitration issues, although the results were not always clearcut. His chairmanship of the Railway Staff National Tribunal from 1973 to 1986 coincided with major industrial unrest but never seemed to impinge. A colleague described him as "a good sticky-wicket batsman. He could always come up with a good answer. If it didn't satisfy, at least it kept them quiet." Typically, his experience was translated into academic studies of the arbitration and conciliation process.

Outside industrial relations and politics, his keenest interest was theatre and its history. Always liable to break into a quotation from Shakespeare, he and Margaret were passionate supporters of the Royal Shakespeare Company.

He is survived by Margaret.

Geoffrey Goodman writes: Bill McCarthy was a luminous member of that celebrated team of academics assembled by Nuffield College's longest-serving warden, the remarkable Norman Chester. At the start of his tenure, he set about turning the newly formed Oxford college into a unique British institution by convincing its founder, Lord Nuffield – William Morris, the great British motor magnate of the 1930s – to make it an outstanding forum for social and economic thinking.

Bill was one of its star dons: a wonderful teacher on that complex subject, industrial relations, about which he cut through the often obscure jargon with a humorous rationality for which he became nationally renowned. His Nuffield seminars on his subject drew the best minds among lecturers with undergraduate and graduate students flocking into the college rooms.

He was a warm and very dear friend – Bill sponsored my journalist fellowship at Nuffield – but never allowed friendship to prevent him disagreeing when required, as it often was. Though a devoted socialist, he remained ever sceptical of adopting political heroes – with the one exception of Aneurin Bevan. He was a walking encyclopedia of Britain's industrial history, not merely of trade unions, and should have been made a minister after Wilson brought him into the Lords. Instead McCarthy perfected his role as a prince of industrial relations peacemakers well before bodies such as Acas were born.

The two Bills – McCarthy and Wedderburn, who died earlier this year – were probably unique in being the only two professors to sit on the Labour benches in the Lords and offer such a working partnership in their specific subject. Parliament has now lost its two outstanding experts on labour law and industrial relations.

• William Edward John McCarthy, Lord McCarthy, scholar of industrial relations and arbitrator, born 30 July 1925; died 18 November 2012