Ronald Fraser, who died earlier this month, was an oral historian who, in his haunting book, In Search of a Past, focused his researches on his own privileged childhood. Through long interviews with the servants who used to look after him and transcripts of sessions with his psychoanalyst, his book unravels the tortuous dynamic between employers and servants in the manor house near Aldermaston where he was brought up in the years leading up to and during the second world war.

This period coincides with that covered by the current BBC series Upstairs Downstairs, although that is set in Belgravia rather than Berkshire. A common objection to Sunday night dramas from the era of domestic service is that they paint a roseate picture of life below stairs, contributing to a paternalist myth that imagines that skivvies were quite happy in their subordinate roles. But Upstairs Downstairs is barely about the relationship between upstairs and downstairs at all. As with Downton Abbey, a house with a retinue of servants is a dramatic device for bringing together contrasting characters. Sir Hallam Holland and his punctilious butler Mr Pritchard are not defined by their relationship in Upstairs Downstairs; it is just part of the period furniture, like the art deco beds and the strange way people held cigarettes in the 1930s.

Fraser's book, by contrast, suggests that the relationship between servants and their employers dominated their lives, and that it was as much a psychic as a social divide. Employing servants was far more than a financial contract or even an expression of the class system: rather, it was an emotional minefield, full of resentment, guilt and grudging obligation on both sides. Fraser's manor house is a world in which everyone feels trapped in a social performance. Even the lives of the idle rich, buried behind their newspapers or playing monopoly in the afternoon, do not seem enviable.

Following Fraser's book, a number of historians, such as Carolyn Steedman, Alison Light and Selina Todd, have sought to uncover the previously neglected history of domestic service. The main insight of these histories is that the very intimacy of the relationship between servant and employer bred particular tensions and frustrations that were not simply reducible to questions of money or social class. The resentment of the servants is unsurprising. The memoirs of Margaret Powell, which partly inspired the original series of Upstairs Downstairs and which have just been republished, bristle with the sense that "servants were not real people with minds and feelings. They were possessions."

More surprising, perhaps, is the resentment felt by their employers. "The cook was a good cook, as cooks go; and as cooks go, she went", wrote the short story writer Saki in 1904. A common upper-middle-class complaint in the first half of the last century was the problem of getting decent staff, and the tendency of bolshy servants to demand perks or more pay on the threat of handing in their notice. As Alison Light reveals in her book Mrs Woolf and the Servants, Virginia Woolf complained bitterly about her servants, and had an intense, difficult relationship with them that a psychoanalyst would surely define as co-dependence. The arguments and appeasements between Woolf and her cook and housemaid sound quite draining and far more work than not having servants at all, and indeed Woolf found it a relief later in her life when she took on some of the domestic tasks herself.

Wherever there is repetitive, poorly paid work to be done and it is distributed unequally, this issue returns. The green baize door may no longer exist, but today's arguments about the "slave labour" of unpaid work experience and "job snobs" are strangely reminiscent of what used to be called the servant problem. On last Thursday's edition of This Week, Michael Portillo argued that "people have to be willing to do things right at the bottom". He claimed that a tenth of Americans begin their working lives "flipping a hamburger" and that in Spain, unlike in Britain, waiting on tables was seen as a profession. For years, young people in this country have been fed the rhetoric of meritocratic elitism and social aspiration. Now, older notions of the dignified, vocational nature of "service" are being revived – and not only on Sunday night television.