Worthy Farm lies between two limestone ridges on the southern tip of the Mendip Hills, six miles east of Glastonbury. This is a region famed for its legends and its leylines, but for much of the year the farm's acres stretch in broad green tranquillity, preoccupied with the rhythms of dairy farming; the grazing and milking of nearly 400 cows that begins each day at 4am.

For 40 years, however, these fields have also played host to Europe's largest and most revered outdoor music and arts festival. In 1970, it welcomed 1,500 people; this June, it will be more than 175,000. Then, it was Marc Bolan playing the headline slot, with Roy Harper and Quintessence on the bill and free milk for all; this year, it will be Gorillaz and Muse and Stevie Wonder, a circus, vintage Indian motorcycles, sushi bars and the Free University, all broadcast around the world by the BBC. As this monumental anniversary approaches, and this year's jamboree draws close (it starts on 23 June), one can't help but wonder just why Glastonbury has survived – flourished, in fact, when so many other festivals have disappeared or become over-sponsored shadows of their former selves.

That first festival was an ad-hoc thing, the hare-brained scheme of a young farmer named Michael Eavis, who had been inspired by a trip to the Bath festival. "We could hear the music from the show, and I didn't have a clue what was going on," he recalls. "So we went along. I'd never seen a hippy in my life, and there were 150,000 of them, real flower power types and they were splendid!" It was a life-changing moment for Eavis, something thrillingly new, but also oddly familiar: "It struck a chord with my Methodist love-and-peace upbringing in a way, with the thinking I had been weaned on, but moved it into the 20th century," he explains. Almost immediately, he decided to try to launch his own music festival on the family farm. "But I didn't know where to start," he says. "I called the venue in Bristol where we went to local concerts, told them I was looking for the Kinks, and they gave me their manager's number, and so I called him and he said they would do it for £500."

'Marc Bolan's set was extraordinary'

The Kinks, in fact, did not play that year – balking after the Melody Maker carried a story with the headline "Kinks for Mini Festival". "I think they were a bit insulted by that," Eavis says jovially. "After all, they were at the top of the charts with Lola at the time." When they pulled out, Eavis, floundering, called their manager again. "And he said, 'Well, Marc Bolan's going past on his way to play Butlins in Minehead. He'll do it for the same money.' So I said, 'Fantastic' – I mean, T Rex was just really taking off then. All the posters still said it was the Kinks playing, though."

It was very much a one-man operation that first year; Eavis charged his festival-goers £1 to get in, collecting all the money himself, as well as organising the stage and dealing with the musicians. "Marc Bolan arrived in what I think was a Thunderbird," he says, "a big wide car trying to make it down this narrow track, and he probably got a few scratches from the hedge, so he was quite grumpy when he turned up." Still, he recovered enough to play, and that performance remains one of Eavis's favourite memories of the past 40 years: "It was extraordinary, and with that set I thought, if I can produce this, the festival's got a future."

He could not, at that moment, have predicted just what that future might hold – from highlights such as David Bowie, the Levellers, Orbital, PJ Harvey, Pulp, Jay-Z, Radiohead and Blur, to the inevitable lowpoints: rain, mud, fires and stage invasions, disputes with new age travellers, gatecrashers and fence-jumpers. I'm speaking to Eavis on the morning that U2 unexpectedly pulled out, but he seems surprisingly unfazed; perhaps that's what 40 years of juggling a festival does for you.

The modern Glastonbury has become extraordinarily large; indeed, for the few days of its reign Glastonbury becomes a settlement roughly the size of Newcastle. "When Neil Young drove in, he made the tourbus stop on the hill to take a picture, because they don't have anything this huge anywhere else," says Emily Eavis, Michael's daughter. Each year it seems the debate rages over whether the festival might have grown too large, but Emily notes that she recently found a newspaper article from 1971, "And even then it said the festival should be smaller!" She laughs. "But when you're there, you find it has different camps and tribes almost, and that is really what makes it magic."

'Glastonbury's part of your upbringing'

Still, the sheer size of the festival has brought a few practical considerations, and eventually, Eavis enlisted the assistance of some professionals, Melvin Benn and Vince Power, to help with the nitty-gritty of organisation. Benn is now head of Festival Republic, which stages some of the UK's biggest festivals, but he learned his art, he says, on Worthy Farm.

"Glastonbury's part of your upbringing, in the way you can't walk away from it. It's a bit like the Catholic church is to a good Catholic: rooted in you," says Benn. Its continued success he attributes to its gentle rearing: "With a festival, the one thing that gives longevity is its relevance to its audience and its ability to allow that relevance to move and change. It's love and care and nurturing, the way you would love and care and nurture a child. It sounds daft, but I think it's true – you have to love every aspect of it, really. And Michael has done that with Glastonbury. In fairness, I've always been Michael's apprentice."

Power is in agreement. He was the founder of the Mean Fiddler empire, and at one time owned 40% of the Glastonbury festival, though for the last two years he has been ploughing his expertise into the Hop Farm festival in Kent – a small-scale festival that arguably shares much in common with the early days of Glastonbury. "Everyone's chasing the same acts, and it's hard to get a line-up that's different," he says. "But Glastonbury is an institution. I don't think there's any festival in the world that can compare."

Still, each year, it seems to the Eavis family that the festival is just as precariously balanced as the last. "I really saw the festival as something finite. Each year seemed like the last one," says Emily. "There were so many obstacles, the travellers, the council . . ." For her father, the low points were two-fold: "The terrible weather forecasts," he says. "And in the 80s, when there were a lot of convoy settlements, the police calling me up and saying, 'There's 250 vehicles leaving Norwich, they'll be with you by 9.30 tonight.'"

Billy Bragg, a Glastonbury veteran now, remembers that decade in the festival's history more fondly. "My first Glastonbury was in 1984," he says. "I slept in a car behind the Pyramid Stage, which in those days was a converted cowshed. The highlights were playing football with the Smiths and the Housemartins, and seeing Elvis Costello deliver a blistering two-hour set." But for Bragg it was then – and remains today – the politics of Glastonbury that makes it special. "In the 1980s, it was a gathering place for those of us who opposed Margaret Thatcher," he says. "We aim to keep that red edge alive in the LeftField."

Continued success

Indeed, this is perhaps the key to Glastonbury's continued success – it is big enough and broadminded enough to be able to provide a home for everyone; those seeking the laid-back hippy culture of the early days can head to the Green Fields; those in pursuit of new bands can linger around the John Peel Tent; while anyone in search of late-night debauchery will be welcomed at Shangri-la, Arcadia and, this year, new areas such as Black 9 and The Common.

These days, the festival's future increasingly lies in the hands of Emily Eavis, who abandoned her studies at Goldsmiths College to return to the family farm. She grew up with the festival, of course, and memorably, aged five, was ushered on to the Pyramid stage to play the violin. "That," she says, "was when I realised how big it was." Most of her memories of the festival remain less tangible. "It's probably more about the senses," she says, "the smells and the feeling of excitement – I really remember the annual build-up, and then the festival campfire dusk smell; it's a smell I don't think you get at any other festival."

Quite whether Glastonbury will weather another 40 years remains to be seen. As the plans for this June begin in earnest, with stages to be built and hundreds of portable toilets installed, Emily will only nod vaguely to ideas for the future and the "couple of people" who have yet to headline the Pyramid Stage. Her father, meanwhile, returns to talking of the dairy farm. "The cows come in before the festival," he says. "We do all the silaging by 4 June and that goes into the fodder store before all the people come in. And then, after the festival, the cows come out again." He stops and seems to consider this yearly routine, the delicate balance between farmer and festival organiser. "The cows belong here, it's prime grazing land," he says eventually. "The farm comes first, always."