"It was 1962," says Sir Brian McMaster, with what sounds almost like a chuckle. "It was absolutely fantastic." It is a rare moment of animation from a man renowned for his reserve. But then McMaster is remembering his first ever Edinburgh festival. Still in his teens, high in the gods of the city's theatres and concert halls, he revelled in opera, drama, music, comedy: the collected works of Schoenberg, Joan Sutherland singing Lucia, Giulini conducting The Barber of Seville.

"I went to everything," he says. "It was Lord Harewood's first festival, and if you look at the programme it was just fantastic, and the fringe, too, all fantastic." He had gone on his own, a bookish 19-year-old from Hertfordshire, ostensibly on his way to a career in law, but with a growing fascination for the arts. "I found the arts for myself," he says. "I heard a bit of music and something went click, and my God this was a wonderful place to discover it."

That Edinburgh summer was a life-changing experience. The legal career would never materialise and instead McMaster moved into arts administration. Now the 63-year-old is preparing to step down as director of the Edinburgh International Festival after a record 15 years.

Quiet, dapper and self-effacing, McMaster may not have been as much of a showman as some of his predecessors, such as Frank Dunlop, few could argue that he has not left his mark on one of the world's most famous arts extravaganzas. It is in rude health despite ongoing financial constraints and growing international competition.

He is leaving with "surprisingly little regret", feeling strongly that 15 years is long enough. And while he insists that he has not gone out of his way to make his last festival a special affair, he admits that others, including donors and some of the artists with whom he has been most closely associated, such as the renowned German theatre director Peter Stein and Spanish director Calixto Bieito, want to see him leave with a flourish.

So this year's festival will be as close to a definitive McMaster as anyone can pin down, with Stein directing Troilus and Cressida; the world premiere of The Assassin Tree, a specially commissioned opera by the young Scottish composer Stuart MacRae; Bieito's adaptation of Michel Houellebecq's controversial novel Platform, about the sex industry in Thailand; Scottish Ballet dancing Balanchine and Sir Charles Mackerras conducting Beethoven.

Born in Hitchin, Hertfordshire, in 1943, McMaster was brought up by his mother, a typist, after his father was killed in the war a few months before his son's birth. After leaving school, he read law at Bristol University. "I had no idea what I wanted to do." Despite his studies, he continued to immerse himself in the arts. In 1964 he slept for days outside Covent Garden to ensure a seat to hear Maria Callas sing Tosca. Others in the queue included Nicholas Payne, who went on to become director of English National Opera, and Neil MacGregor, director of the British Museum.

"I knew Brian when he first came to London and was trying to be a solicitor, and because he came regularly to Covent Garden and Sadler's Wells and so on," recalls Payne, who is currently chief executive of Opera Europa. "We met in the upper reaches of the house, in the cheap seats."

Offered a scholarship to the University of Strasbourg to do a course in comparative law, McMaster accepted, principally, he says, because of the excellent opera house in the city. He stopped short of a PhD, however, and returned to the UK, where he got a job as an articled clerk. But once he had qualified as a solicitor he left law, and in 1967 he enrolled in one of the Arts Council's new arts administration courses. His first arts job was at EMI Records, where he was posted to the International Classical Division and found himself able to indulge his passion for opera, and casting major recordings. After five years he moved to English National Opera to be controller of planning. Three years later, in 1976, he landed the job of managing director at Welsh National Opera, where he met up again with Payne.

"We were both in our early thirties," says Payne, "but his direction certainly transformed the company, and I think that his openness to European influences and production values in particular transformed the way that opera was done in this country quite a bit before the famous ENO powerhouse."

McMaster's workload increased further when he also took on the role of artistic director at Vancouver Opera, in effect commuting between Cardiff and Canada. "It was a nightmare," he says. "It was wonderful to have done it, but it was quite stressful."

He stayed at WNO for 16 years, building such a reputation that, in 1991, he was seen as a strong choice to replace Frank Dunlop, who was standing down after eight years as director of the Edinburgh International Festival. Dunlop boosted the festival's profile as much by sheer force of personality as by the acts he brought to the city, whereas McMaster was the quiet opera buff who shunned the limelight.

"McMaster and Dunlop are the antithesis of each other," says Catherine Lockerbie, McMaster's counterpart at the Edinburgh International Book Festival. "Dunlop was an extrovert, he was very ebullient. But people are wrong if they think that Brian's been the invisible man. His programmes are more fiercely idiosyncratic than those of many other directors. He just does not jump up and down and shout about it."

McMaster's first decision was to move to Edinburgh, making him the only director to have ever lived in the city. At the time, the festival was run from a London office. McMaster closed it. "There was opposition," he recalls. "Someone said you will win Edinburgh but you will lose the world. There was the feeling that it made it parochial to run it from Edinburgh ... Now that idea seems totally absurd."

Where Dunlop was sometimes accused of being too populist, the accusation that has most persistently dogged McMaster is that he has relied too heavily on a small band of artists who have returned year after year, such as Peter Stein, the American dance supremo Mark Morris and the pianist and conductor András Schiff.

McMaster makes no apology for developing and nurturing such associations. "From 1947, the festival was the subjective choice of its director," he says. "That has been part of the Edinburgh tradition. It's one of the things that have led to the success of the event." McMaster has always been guided by his own instincts and preferences. But despite strong feelings about some of his choices, people have often struggled to define what characterises a McMaster festival."It's quite unpredictable in some ways and predictable in others," muses Lockerbie. "There is also a kind of quirky intellectual independence. In theatre, it is quite esoteric or quite vulgar or somewhere in between."

For McMaster it has been a ceaseless search for work that inspires him. In previous years, he could go for months without having a weekend at home. "We have to find the new, create the new," he says. "We live in an age when the arts are being dumbed down. But because of the excitement generated by the festival, the audience we have built up, the international profile that has been secured over 60 years, I think we can resist that. We have to encourage people, through programming, to get on a plane in Los Angeles, get on a plane in Tokyo, spend a lot of money on hotels and have an experience that is worth it. But also I believe we can put on work that will change the life of people in Wester Hailes [Edinburgh] and we need to programme for that too."

He has always been adamant that the festival is not elitist, insistent that certain events should be affordable to all - this year includes a series of £10 orchestral concerts. He is deeply irritated by the People's Festival, an alternative political initiative, first launched in 1951 by trade unionists and members of the artistic community who felt the international event was too inaccessible for many of Edinburgh's citizens. It ran for four years and was relaunched in 2002, with support from the Scottish Socialist Party, staging a series of concerts, comedy shows and exhibitions on the city's edges.

"It seems to me to be patronising to lots of people. It says, look, you can only accept the second rate or something done especially for you. Actually, no, excellence is for everybody. Beethoven's fifth symphony: people can respond to it as an ad on television, why can't they respond to the whole thing? Why can't they respond to a great performance of it? Of course they can, everybody can. So you can come and hear a great performance this year for 10 quid."

It is unusual to glimpse this kind of emotion. Even after 15 years in such a public role, associates say they know little of what makes McMaster tick. Few know what he likes to do in his meagre private time beyond scouring the city's wealth of second-hand bookshops and spending downtime in Spain, a country he adores.

He has already handed the reins of the 2007 festival to Jonathan Mills, the 42-year-old Australian chosen as his successor, and will only say his plans are to stay in Edinburgh. Initially.

He is equally enigmatic when asked what he wishes for his successor and for the festival that he has guided for 15 years. "Mills will do things different ways; there will be changes, which is fantastic. I just hope that in 50 years it will do to others what it did to me in 1962. That's always been my ambition in a way, to try and recreate that experience for other people."