When I saw two poems by Ted Hughes on the front of the Times in January 1998, I was as surprised as any other reader. These were trailers from Birthday Letters, a cycle of poems addressed to his long dead first wife, Sylvia Plath. Irrespective of the poems' literary merit (and they were very fine), their publication was a sensation, an unexpected finale to an infamous romantic tragedy.

There was another reason for my surprise. One of the poems, 'St Botolph's', an account of the first, intense encounter between Hughes and Plath, was dense with astrological imagery, impenetrable to anyone unversed in deciphering a horoscope. 'The conjunction combust my natal Sun/ Venus pinned exact on my midheaven,' it chattered, commenting on the planetary positions that attended that fateful meeting, before concluding, remarkably, that the two poets had that night been 'married by the solar system'. Other poems in Birthday Letters - 'Horoscope', '18 Rugby Street' - likewise suggested that a writer celebrated principally for his nature poetry had an unsuspected life as a star-gazer. The poet laureate had outed himself as an astrologer!

As much was confirmed by Hughes's publishers, Faber and Faber, who informed me that Hughes always stipulated the exact publication dates of his books. Birthday Letters, it turned out, was published on the day of a new moon, 29 January 1998. Its US edition followed on the next new moon.

What would the literary establishment make of Ted the planeteer? Disappointingly little, it transpired. Out of embarrassment or indifference, the critics avoided comment on the astrological references. There was the odd remark about the poet's 'dottier beliefs', and Seamus Heaney described Hughes and Plath as 'mythopoetic' writers, but otherwise nothing. Ariel's Gift, a 'comprehensive guide' to Birthday Letters by Times critic Erica Wagner, managed to avoid addressing astrology anywhere in its 200 pages.

I was not altogether surprised. Astrology, for all the popularity of newspaper sun sign horoscopes, remains beyond respectability, the province of the potty. Look at the derision heaped on Ronald Reagan and Princess Diana when their consultations with astrologers became public knowledge. The case of Hughes and Birthday Letters is a paradigm of the way astrology and kindred 'irrationalist' systems help shape culture while going unacknowledged or belittled.

Over the last century, astrology has seeped into poetry, music, film, psychology and even science, and enchanted titans such as Yeats and Jung, yet it remains a renegade and ridiculed subject. Remarking on Yeats's passion for esoteric ideas, W.H. Auden wondered: 'How on earth could a man of Yeats's gifts take such nonsense seriously?' His question can as justly be inverted; if a man of Yeats's gifts took this nonsense seriously, maybe it has something to tell us. It is a delicious irony that Yeats is the favourite poet of astrology's fiercest and most eminent critic, Richard Dawkins, whose last book, Unweaving the Rainbow, included a thunderous attack on Keats, Blake, Yeats and other poets of genius for their lack of scientific rigour and 'prejudice against reason'. Dawkins certainly knows about prejudice. For him, astrology is a wicked fraud, 'an aesthetic affront' whose 'pre-Copernican dabblings demean and cheapen astronomy'. Astrologers, whom he 'detests', should be prosecuted. 'We should learn to see the debauching of science for profit as a crime,' rasps Dawkins, transformed from Beacon of Reason to Chief of Thought Police.

The scientific model of the universe upheld by Dawkins does, indeed, present astrologers with a problem. There is simply no known mechanism which might link human character and behaviour to the movement of planets and the position of stars, though the 'holographic universe' posited by quantum physicist David Bohm, an associate of Einstein, offers one explanation of how such a link might function.

Such objective proof as exists to support the astrologers' age-old adage 'as above, so below' is patchy and contested. The exhaustive research of French statistician Michael Gauquelin, who found, for example, a powerful correlation between sports champions and the position of Mars in their birth charts, remains the subject of bitter disputes. Here, the prevailing scientific attitude to Gauquelin's findings was summed up by psychologist Hans Eysenck as: 'Don't confuse me with facts when my mind is made up.'

Proof, however, comes in many shapes, not least in the guise of personal experience. Symbols are, in their way, just as 'real' (a favourite Dawkins adjective) as the constructs of hardhat science. Symbolism challenges consciousness and nourishes imagination, inviting us to uncover deeper levels of meaning. Astrology's richly symbolic language, resonating with the myths of the ancients, is a major part of its appeal, not least for artists and others seeking to understand what Jung termed the 'archetypes' of the human psyche.

Among poets, it occurs in the work of Ted Hughes, Louis MacNeice (who wrote a book on astrology), and Rudyard Kipling (who perhaps learnt something of it through his freemasonry). Then there is Yeats himself, whose most famous poem, 'The Second Coming', has part of its genesis in the astrological concept of the 'great ages'. I suspect Yeats's 'rough beast, slouching towards Bethlehem to be born' was in part his 'new age', a version of 'the Age of Aquarius' which had become a fashionable catchphrase in astrological circles at the time he wrote the poem.

While Yeats was writing 'The Second Coming', Gustav Holst was composing The Planets . Holst based his masterpiece on the astrological meanings of the heavenly bodies, though he kept this concealed at the time, no doubt mindful that the astrologer on whose work his suite was modelled, Alan Leo, had recently been prosecuted for 'fortune-telling'.

Later musicians felt able to express their interests more openly. Jazz colossus John Coltrane wrote several works with astrological themes late in his life, among them 'Saturn', 'Jupiter', 'Venus' and 'Leo', and according to his widow, Alice, used a scheme 'with 12 tones correlating to the 12 zodiac signs'. Pop music is littered with cosmic references, whether you want to 'Breakfast on Pluto' with Sinatra or were 'Born Under a Bad Sign' with Albert King.

More surprising was the realisation that the century's most celebrated love story, Gone With the Wind, is a thinly disguised astrological allegory. Margaret Mitchell based the characters of her torrid epic on the zodiac, leaving a blatant trail of clues which were only picked up in 1978 when US astrologer Darrell Martinie was shown photocopies of notes from Mitchell's library.

Scarlett O'Hara proves to be a dashing Aries, named after the colour of her ruling planet Mars. Her father is a Taurus who owns a farm in earthy Taurean fashion, the Tara plantation. Scarlett's twittering aunts, Prissy and Pitty Pat, are a pair of Geminis whose names comply with the sign's love of wordplay. Rhett Butler is a passionate Leo. Bonnie is the third fire sign, Sagittarius. Being born under the sign of the centaur, she meets her end in a horse-riding accident. The list goes on ...

In the Harry Potter books, our hero is reluctantly learning star lore from cranky professor of divination, Sybill Trelawney, and this turns out to be a refreshingly accurate reflection of real-life astrology. Is Joanna Rowling another secret stargazer?

Such outcrops of influence in popular and high culture alike testify to astrology's vigorous, unruly character. Having survived the edicts of emperors and popes, it is not surprising that it continues to defy scientific rationalism today. Accordingly, it comes in many guises; as newspaper horoscopes, as psychology, as agony aunt, as adviser to the rich and powerful. Charles de Gaulle, seemingly a model of the man of will, is the latest president to be named as taking astro-advice ,though contrary to popular legend, Adolf Hitler was uninterested in the subject.

If science and astronomy really do want to win popular respect, as their champions claim, they might begin by admitting astrology's role in their own history. Instead, science airbrushes out the interests of astronomical pioneers like Johannes Kepler, who wrote exten sively on the links between numbers, music and astrology, and whose ideas on reforming astrology reveal an approach strikingly akin to that of modern astrologers, disliking predictions and favouring character analysis.

Galileo, hailed as the first man of modern astronomy, had a lifelong interest in astrology, and even science's most potent figurehead, Sir Isaac Newton, while no astrologer, was a closet alchemist. Robert Boyle, one of the founding fathers of modern chemistry, was also fascinated by alchemy.

Astrology has flourished in recent decades partly because it offers an escape from a cold, purposeless cosmos into a universe charged with symbolic meaning and individual significance. For most of its followers, the approval of physics doesn't matter. Despite the tremendous successes and insights of physics, biology and other branches of science, these remain unable to supply the answer to the questions that hang over numerous heads: 'What am I doing here? What is my life for?'

Astrology is not a religion, but large parts of its practice assume there is a spiritual dimension to human life, a mystical connection to the universe, which it can help clarify. Richard Tarnas, a bestselling US intellectual, has described astrology as 'the most feminine science', which is why male-dominated hard science is so scornful.

Astrology promises self-direction through self-knowledge and co-operation with the wider tides of the cosmos, an escape from nihilism and a return to psyche, to soul. Of course, for self-respecting physicists, there is no such thing as spirit, let alone 'cosmic tides'. Most of the world's people, however, continue to disagree with them.

 Neil Spencer's True As the Stars Above is published by Gollancz. To order it for £7.99, call Observer CultureShop on 0800 3168 171. His article, 'Harry Potter and the Astrologer's Chart', is published in the December issue of the Astrological Journal.