This is a sample of the writing Benjamin Britten set to music in his first opera, Paul Bunyan: "Let the dog who's the most sentimental of all / Throw a languishing glance at the hat in the hall / Struggle wildly to speak all the tongues that he hears / And to rise to the realm of Platonic ideas."

And here is a fair sample of the writing he commissioned, set and seems to have thought adequate in his last opera, Death in Venice, 34 years later: "Mysterious gondola / a different world surrounds you / a timeless, legendary world / of dark lawless errands / in the watery night. / How black a gondola is – / black, coffin-black, / a vision of death itself / and the last silent voyage."

Britten is always said to have been a sophisticated admirer of poetry, and to have exercised a connoisseur's pleasure in setting it. The claim seems plausible, apart from one thing. His first opera's libretto was written by WH Auden, who now clearly seems the greatest poet in English since Tennyson. After Paul Bunyan, Britten chose as his librettists Montagu Slater, Eric Crozier and, no fewer than three times, Myfanwy Piper, the author of the hopeless fourth-form effusions of the Death in Venice libretto. When a more distinguished writer such as William Plomer was engaged – for Gloriana and the three church parables – his work was not permitted to display its usual mordant originality.

Auden's relationship with Britten – the subject of a new play by Alan Bennett – was at its most creative and fervent for the five years after 1936 or so, resulting in half a dozen major works and a substantial body of songs. It occurred, however, at the very beginning of Britten's career, and at a brilliant-prodigy stage of Auden's. After 1947, they hardly even spoke, and Auden was accustomed to say that Britten was the only friend he had ever had with whom he had subsequently irrevocably quarrelled. (Bennett's new play, The Habit of Art, imagines a meeting between the two some 25 years later.) For Britten, on the other hand, it was a different matter; he made quite a career out of casting those who had committed some blunder into the outer darkness. Their collaboration in the late 1930s was fiery and produced some thrilling objects. Yet Britten used different, and lesser, writers to create his best operas; Auden rose to the challenge and worked with a much more important composer than Britten – Stravinsky – to write one of the two or three greatest operas of the century, The Rake's Progress.

Britten and Auden were brought together in 1935 by a very 1930s organisation, the General Post Office Film Unit, which was devoted to producing documentary films about modern-day life. Both at the time had a mild sort of devotion to communist causes. Britten wrote in his diary that summer about a performance of Elgar's first symphony: "I swear that only in imperialistic England could such a work be tolerated." Auden was coming to the end of what might be termed his Pylon Period, the style which would see him praise industrial landscapes in "Letter to Lord Byron".

Auden's Oxford tutor, Nevill Coghill, had observed that "Auden is in the imperative", meaning the human being rather than the poet. In 1935 he was a commanding presence across the English-speaking world. Britten was a mere boy, though one of evident enormous gifts. Auden was to observe that he had never seen such "extraordinary musical sensitivity in relation to the English language" as in Britten. The GPO unit set them to work together, Britten setting the beautiful Auden lyric "O lurcher-loving collier, black as night" for a documentary, Coal Face, writing music for other GPO Auden-scripted films, such as Negroes ("Chorus: Beside the long Niger they lost their freedom . . ."), The Way to the Sea and the great Night Mail, still unsurpassed as a marriage of film, music and poetry ("This is the Night Mail crossing the border / Bringing the cheque and the postal order . . ."). Britten was so inexperienced with that last one that he forgot to leave a pause for the rhythmic speaker, Stuart Legg, to breathe, and the recording had to be manipulated as far as the rudimentary technology allowed.

Clearly, from Auden's point of view, Britten's fascination was not just that of a marvellous musical prodigy. As Peter Parker has demonstrated in his life of Christopher Isherwood, Auden occupied the place of plain best friend in that relationship, always having to settle for the boys Isherwood wasn't interested in. Without conventional good looks, he had always relied on his amazing conversation to get his way. Whether startling the mothers of his college friends when staying with them ("Mrs Carritt, this tea tastes of tepid piss") or, no doubt, explaining to new chums why homosexuality was the only rational choice to take, he had always won others over through his powers of speech.

For a while, Britten formed a kind of project for Auden and his entire group. Isherwood took Britten in 1937 to the notorious Jermyn Street Turkish Baths. "Well," the film director Basil Wright asked Isherwood afterwards, "have we convinced Ben he's queer, or haven't we?" A glance at Britten's diary, had it been available, would have demonstrated what the problem was. "Very pleasant sensation," Britten wrote of the visit. "Completely sensuous, but very healthy. It is extraordinary to find one's resistance to anything gradually weakening."

Britten's unswerving attachment to the "healthy" comes out in his private reflections. He wrote of an old schoolboy acquaintance, David Layton, that "he is a very good sort – clean, healthy living and balanced". More experienced boys knew exactly how to write to Britten to get a result. Wulff Scherchen, whom Britten met at 14 and pursued more seriously at 18, was to inspire Young Apollo and the most frankly erotic of the Les Illuminations song cycle of 1939. Replying to Britten's speculative letter, he gets straight to the point. Yes, he remembers Britten from four years ago, he writes: "I was in shorts and sandals (as I am now) and it started to rain. I got thoroughly wet . . ."

Whether by luck or calculation, or just by calling up in the composer's mind the image of a wet 14 year old, Scherchen could effortlessly hit the note to get a response from Britten. Auden's approaches, on the other hand, reflected his highly didactic personality. They were almost comically unlikely to get results, and not just because Auden was seven years older than Britten, then in his early 20s. But his obsession with leading Britten into bed did result in a series of poetic masterpieces. The lyric "Underneath the abject willow", from March 1936, is addressed to Britten: "Walk then, come / No longer numb / Into your satisfaction." Britten wrote in his diary only of a "bad inferiority complex in company of brains like Basil Wright, Wystan Auden and William Coldstream". In May, another poem seems to relate to a rejection by Britten of Auden – "You love your life and I love you / So I must lie alone."

At this period, it is sometimes hard to distinguish, in Auden's writing on music, whether the subject is the art of music or specifically Britten. "There is no creature / Whom I belong to, / Whom I could wrong . . . I shall never be / Different. Love me," Music says in Auden's Hymn to St Cecilia, wonderfully set by Britten in 1942 as their friendship was coming to its end. His sonnet "The Composer", one of a series of speculations on particular or generic artistic figures, suggests he had recently spent a certain amount of time mooning over one composer; the lines "Only your notes are pure contraption / Only your song is an absolute gift" are ironic, considering how very literary a composer Britten turned out to be. There may even be a small dig, in one of the charming cabaret songs, at Britten's taste for what Auden called, in a fateful letter, "thin-as-a-board juveniles"; Britten set it to music, and it was performed at a riotous party to bid farewell to Auden and Isherwood, on their way to the Sino-Japanese war in 1938. Britten may not have noticed that the comic song began with the line "Some say that love's a little boy . . ."

In 1939, Auden and Isherwood performed their famous bunk to America, and shortly afterwards Britten and his new friend, soon to be his lover and lifetime partner, Peter Pears, followed them. It was not the same. By the time of Britten and Pears's arrival, Auden had met his lifetime partner, Chester Kallman. They all lived together for a time in a celebratedly bohemian household at 7 Middagh Street in Brooklyn, along with Paul and Jane Bowles, two or three of the Mann children (Auden dashingly married Erika at one point), Carson McCullers and Gypsy Rose Lee. Sheryl Tippins wrote an enjoyable book about the bizarre ménage, capturing the highly tiresome tone of the public exchanges between Auden and Kallman: "'I am not your father, I'm your mother!' 'You're not my mother! I'm your mother! . . . You're my father!'"

Auden, too, was a notoriously slapdash housekeeper. Years later, Vera Stravinsky found a bowl of brown water abandoned on the floor of the bathroom during an Auden-Kallman dinner party, and flushed it away; she later discovered she had thrown away the pudding for the evening. Could it possibly have been the state of the house in Middagh Street that led to unmeltable frostiness between first Pears and Auden, and subsequently Britten and Auden too? Certainly, in later life, a question about Middagh Street to Pears could always set off a fit of eye-rolling.

Paul Bunyan, Britten and Auden's largest collaboration, is one of those works that one wants to be a masterpiece, and has a lot to be said for it; the poetry represents Auden at his two extremes, the brilliantly clever merchant of paradoxes and rhyming games, and the author of exquisitely framed conversational simplicities. The music is deft and often memorable; the idea of the little opera, of an unseen giant Paul Bunyan and the founding of a community at America's birth, ought to work perfectly well. But the American critics poured scorn on it at its premiere in May 1941, perhaps irritated by two chic English draft-dodgers taking on a heroic American national myth. Britten never sought to have it performed again in his lifetime.

Shortly afterwards, Auden dealt the relationship a fatal blow by yielding to his didactic urge, and writing the sort of letter which no one should write to a friend, putting him straight about a number of defects in his character: "I am certain too that it is your denial and evasion of the demands of disorder that is responsible for your attacks of ill-health . . . you are and probably always will be surrounded by people who adore you, nurse you and praise everything you do . . . you are always tempted to make things too easy for yourself in this way, ie to build yourself a warm nest of love . . . by playing the loveable talented little boy."

After that letter of January 1942, the relationship was more or less over. Auden tried to persuade Britten to set one last thing, his great "For the Time Being: A Christmas Oratorio". But as any reader could have told him, this long poem does all the music itself. It was in no need of an orchestra and chorus to add to the splendid effects of the verse.

There are a surprising number of scores in the Auden-Britten catalogue – Clive James once said the results of the encounter were meagre, but Donald Mitchell, in a book on the subject, thought it might, in the end, amount to more than the Brecht-Weill collaborations. Many of them are brilliantly clever – the cabaret songs are irresistibly good; the allegorical treatment of prewar international politics, Our Hunting Fathers, still startles with its brief flash of terror as the medieval catalogue of hawks' names comes down at the end to just two – "German. Jew." Paul Bunyan will always be revived as an occasional curiosity. Edward Mendelson observed that in the 1930s King Arthur – the 1691 opera by Purcell and Dryden – "was the first and still the only libretto written by a major English poet for a major English composer. Paul Bunyan would be the second."

In the end, Britten's subsequent career showed that he worked best with people not quite up to his level. Auden's career as a librettist displayed, in the magnificent Rake's Progress, that he needed an artist on the scale of a Stravinsky to deal with his invention. For a few years the two came together; they were never truly compatible, artistically or as people, and their joint products are tantalising rather than fulfilled. But they were exceptional creative figures, and if they went wrong, they did so in a lastingly interesting way.

The Habit of Art is at the Lyttelton Theatre, London, until March. Box office: 020 7452 3000.