Thursday 21 April 2011 17.00 EDT
First published on Thursday 21 April 2011 17.00 EDT

Making a film – any film – is a journey of exploration. If you knew at the beginning what you know at the end, why bother to make the film? This is one of the reasons I am often spurned by commissioning editors: I begin with no script, certainly no "agenda" (their favourite word), and no schedule. Not much of a budget either, come to think of it.

It was 40 years ago, while filming Benjamin Britten, that I first thought of making a film about Gustav Holst. I had noticed a photograph of the young Holst in Britten's music room, and asked him why. He told me, "I owe him more than I can tell you." Which was odd, because you almost never heard the name of Holst mentioned among of the pantheon of the 20th century's great composers.

Holst's feisty daughter Imogen lived in Aldeburgh and worked as Britten's copyist and occasional editor. She was also one of the co-founders of the Aldeburgh festival. She pressed me to make a film about her father – but somehow everything she told me about him didn't quite add up. Was he really the reclusive lover of English folk songs who taught at St Paul's Girls' School and just happened to have written The Planets? No matter that this is the most recorded piece of British music, that the famous melody from Jupiter is played at every Remembrance Day service, as it was at Princess Diana's funeral, that in every lazy piece of television journalism Mars props up images of war and carnage. I am always being asked: "Did Holst write anything else?"

This ignorance extends to our music colleges. For instance, at a meeting with the bosses of the Royal College of Music (where Holst had studied and taught), I suggested we should consider a Holst piece called Beni Mora. I was told it was rather dreary and not worthy of inclusion.

The BBC had made two previous films about Holst, the first in 1966, in black and white, which stated: "Gustav Holst wrote The Planets, was much influenced by folk song, and, with his friend Ralph Vaughan Williams, came from the Cotswolds." Much of this material was recycled in a 100th birthday film in 1974. I had made several attempts during the last 40 years to interest the remnants of the BBC's music and arts department in a film about Holst, only to be told, "We've done that," or such a film was "certainly not on our agenda".

It was therefore to my surprise that 18 months ago I was offered the chance to make my film. The money I was allocated was pitiful – less than I had been given for my film about Britten almost 30 years earlier. But even if it meant remortgaging my house (it has), I felt I must set out on the road of exploration.

Holst had lived in Thaxted. According to Imogen, he and Vaughan Williams had made many walking expeditions into the Essex countryside "looking for folk songs, and had come across this beautiful village and its imposing church". Two things jumped out when I visited. First, I learned that, contrary to myth, neither Vaughan Williams nor Holst much cared for these walks – Vaughan Williams because of his bulk, Holst because of his poor health (neuritis, asthma and terrible eyesight). Second, when Holst first stepped inside Thaxted's enormous parish church, he saw two flags hanging there: the red one of the Socialist party, and that of Sinn Féin. The union flag, symbol of imperialism, had been forbidden by the vicar, a remarkable man called Conrad Noel. Holst knew he had found a soulmate.

Suddenly, the facts that Holst had trained the Hammersmith Socialist Choir, that he had taught at the newly formed Morley College (whose purpose was to bring education to the impoverished working classes), and that he had exhausted himself taking his Can't Sing Choir to perform in the slums of the East End, all fell into place. In Thaxted, Noel, aided and abetted by Holst, set up a printing press publishing such tracts as Freedom from Social Injustice, and Jesus the Revolutionary Leader. When the first world war began, he pinned notices on the church door saying such things as, "The Bell will toll at Noon for those Slain by Imperial Aggression."

Holst was born in Cheltenham in 1874, and that was my next port of call. The Regency terraced house is now run as a "Victorian experience". It has his piano, some portraits, and a cot. Compared with the throbbing industry that is Britten's old house in Aldeburgh, it is pathetic. It peddles the familiar twaddle about folk songs and Gloucestershire and, of course, The Planets. There had to be more to it than this.

As I read through the local newspapers, another strange fact stared out at me. Cheltenham in the 1890s was famous for its curries. Curries? The town was full of retired civil servants and army officers who had served in India, many of whom had brought Indian servants back with them. And the shock of listening to one of Holst's earliest orchestral works (rarely, if ever, performed), called The Cotswold Symphony, with its savagery and oriental sounds, made me realise this music owed little or nothing to English folk song. Its depiction of the countryside was, if anything, more akin to Hardy – bleak, harsh, unforgiving. And oriental? So maybe this was why Holst had taught himself Sanskrit and written mesmeric music based on Hindu philosophy and Satyagraha, long before it was all "discovered" by minimalists such as Philip Glass.

But the key to Holst's sound world still eluded me. I was aware that the very familiarity of The Planets has dulled our appreciation that the piece, in terms of its orchestration alone, is totally revolutionary. The famous rhythm, five beats to the bar (for a march!), is hammered out initially by the violins hitting the strings with the wood of the bow "to make the sound of rats scurrying across a stone floor". I listened over and over again to every piece of Holst's music I could find – to the astonishingly lucid choral writing and its obsession with death, to the string music often in several keys at the same time, and finally to that piece the Royal College thought unworthy, Beni Mora, especially the third, mysteriously titled movement, In the Street of the Ouled Naïls. What did that mean? Dancing girls? Brothels?

In 1908, Holst had travelled to Algeria, mostly for health reasons. There, over a period of several months, he cycled to the Sahara. As it did to TE Lawrence some 10 years later, something happened to Holst when dwarfed by those mountainous dunes. In her biography of her father, Imogen never mentions it; even in the more thorough biography by Michael Short, it only merits half a line. But I knew I had found the key.

Add this to Holst's passionately felt socialism and his profound understanding of Hinduism, and The Planets begins to make sense: not as an astrological chart à la Mystic Meg, but as a pilgrim's progress from the ferocity of industrialised capitalism (Mars) towards a karma of enlightenment (Neptune). Holst called the work Seven Pieces for Large Orchestra; the names of the planets were added later. But its colossal success has blinded us to the mass of other music he wrote – much of it forgotten, some still unpublished 75 years after his death.

Holst was convinced he was a failure. Many works were not performed professionally in his lifetime, others more or less instantly forgotten. I believe his time has come, and I just hope I have done him some justice.

• Tony Palmer's film Holst: In the Bleak Midwinter is on BBC4 on 24 April.