I saw and heard Captain Beefheart and the Magic Band play the Rainbow Theatre, formerly the Finsbury Park Astoria, London, in 1973. I remember the performance being wild and other-worldly; but then, an hour beforehand I had partaken of a slice of cake baked by my flatmate which contained a special ingredient that may have messed with my mind somewhat. That was de rigueur then.

Little evidence remains of any of this, though there's a rather iffy sound recording on YouTube. Soon after, I wrote a poem about my experience, which I sent to John Peel at BBC Radio One, the Captain's main champion in the UK. To my delight, I got a handwritten postcard back, thanking me for my endeavour and expressing the wish that he'd partaken of that cake, to which the poem made reference.

I know I treasured that postcard, though I can't locate it now; it's probably in a mouldering manila file in the shed. As for the poem, I retrieved it, thinking maybe to reproduce a line or two as a memorial to Beefheart. Alas, no; it's so mindblowingly godawful that I have had to lay it to rest again.

Strangely, I'd been thinking of writing a post about Beefheart before his death was announced this week. The spur was reading the huge memoir (864pp!) written by John French (aka "Drumbo"), the Magic Band's drummer on and off over a period of years (although he wasn't at the Rainbow - the drums were played on that occasion by Ed Marimba). I'd been reading it side by side with Robert DG Kelley's almost equally voluminous biography of Thelonious Monk, and the comparisons between two musicians not commonly thought of together were enlightening. I'll return to this later.

Beefheart: Through the Eyes of Magic (Proper Music Publishing, 2010) is a blow-by-blow (sometimes literally) account of life with Don Van Vliet and the ever changing Magic Band from the Safe As Milk era to Doc at the Radar Station, with some significant gaps where French resigned from or was fired from the band, as happened from time to time. Poorly edited and repetitiously written, it's nevertheless in places a riveting insight into the genesis and development of the prototype avant-rock band, with all the attendant monomania, paranoia, broken promises and shattered hopes. Much of it is in the words of the musicians themselves (sans the by now reclusive Beefheart) whom French interviewed copiously over several years.

French has his own agenda, and frequently this is too intrusive, but his debunking of the myths surrounding Beefheart carries some weight. Nowhere more so than in the fascinating middle section of the book that recounts the story of the band's many months' incarceration - for that is what it amounted to - in the "Trout Mask Replica" house to rehearse the notorious album of that name.

The legend is that Beefheart, whom French, despite bitter resentment at times, acknowledges as a genius, composed every last detail of that album and taught the music, note by note, to an untutored young band, and after over a year the album was then recorded in five hours. The reality French reveals is that Beefheart was the one who was untutored; he could play no instrument other than the mouth harp, and, while brimming with musical and lyrical ideas, found great difficulty communicating these or even preserving them so that they could be recalled.

It fell to French to transcribe and notate the musical fragments the Captain came up with and, together with other members of the band, to stitch them into something approaching playable pieces. Beefheart was contemptuous towards the idea of practising ("scales are for fish") and cavalier with rehearsals - in contrast to his friend/enemy Frank Zappa, a musical sophisticate who ran a tight ship where musicians were actually paid for providing a professional service.

The Magic Band at this time were literally starving, and the conditions under which they lived are described by French as like being in a cult. Musicians regularly fell into and out of favour with Van Vliet, and once out of favour other members of the band were turned against them. This particular episode ends with French being pushed down the stairs, and expelled from the band and the house, with no money and deprived even of his own drumkit, reduced to begging a neighbour to use the phone to call his parents to come and pick him up.

And yet he came back for more. Lured by the thought that this time, maybe, fame and fortune would accrue, he returned to the band a couple of years later, and thereafter on several occasions, making his peace with Beefheart and falling out with him again. Fame of a kind happened but fortune didn't. Beefheart made more money than any band member, but even that was precious little. By 1982, he had retired to his childhood haunts near the Mojave desert to paint. A New York dealer is reputed to have told him that if he wanted to make serious money out of painting, as "Don Van Vliet", he would have to give up music. And so he did.

The patchiness of French's account is at times infuriating - intense, hallucinatory detail followed by bewildering lacunae. But there are some great insights here. One of these is the comparison with Zappa, a boyhood friend of Van Vliet's, sometimes a mentor and a benefactor, whose overtures the Captain later spurned. French characterises Zappa as a musical giant, a canny businessman who organised every aspect of his operation, never did drugs, made sure his musicians were treated fairly and successfully resisted commercial pressures. By contrast, according to French, Beefheart was a musical illiterate, a bully and a liar, claiming credit for the achievements of others, his mind partially unhinged by psychedelic drugs. Yet French thinks there was more humanity in Beefheart's sometimes bizarre music and lyrics than in Zappa's brilliant but often cold and cynical oeuvre; and as a big fan of both, I think he has a point.

Let's take the lyrics. The announcement of Beefheart's death provoked eulogies on at least one of the poetry forums I dip into, and it's a fair bet that his words as much as his music inspire affection here. There are actually not too many rock performers, to use the term loosely, whose lyrics would bear much scrutiny from those for whom words are the stock in trade. I have discussed some of them in this space before: Leonard Cohen, a legitimate poet-on-the-page whose crafted lines, with their wit and political savvy, made a successful transition to stage and album; Bob Dylan, who said he was a poet and convinced us as much, at least when he was playing a blinder in his peak years (and flickering again latterly); David Byrne, who managed delirium and satirical abstraction with Talking Heads; Elvis Costello, nowadays strangely unfashionable, but in the day quoted in John James poems and the inspiration for Tom Raworth's book title Visible Shivers ("visible shivers running down my spine"). Add your own favourites.Beefheart's lyrics, however, are just demented. On Trout Mask Replica he sings "Dachau Blues, those poor Jews" and you think this is a ghastly, ill-advised piss-take - but he was being sincere! There's no Leonard Cohen style political sophistication here. But the endless stream of warped imagery and skew-whiff insights, drenched in the idioms of the blues, surpasses the fine excess of the Surrealists.

The blues also infuses his music, but also the jazz avant-garde. Even on the relatively conventional The Spotlight Kid, a song like "Click Clack", ostensibly a classic train blues, inhabits two time signatures simultaneously; and on the further-out albums things get much, much more mental. French's book makes it clear that Beefheart had a unique and amazing musical vision, but limited means of communicating and preserving his ideas without help from more conventional musicians - whom he subsequently abused. His voice could emulate Howlin' Wolf and Otis Redding at will, in several different registers. His soprano sax playing was, to say the least, primitive. All he could do was blow free, and it was a hell of a sound, but he could never replicate what he had done before (yeah, "scales are for fishes"). Among the many jazz players of the "new thing" of the Sixties that he admired was Ornette Coleman, but it was Ornette's wildly untutored violin playing rather than his sax that his own blowing resembled. The book records a meeting between the two musicians; what Ornette thought of the Captain is not clear.

Thelonious Monk was surprisingly sniffy about fellow iconoclast Ornette, according to Robin DG Kelley's book (Thelonious Monk: The Life & Times of an American Original, JR Books, 2009). Unlike many, he didn't dispute that the younger man could play - at least when he confined himself to the alto sax and wasn't deliberately leaving his comfort zone to babble on violin or trumpet - but there was a hint of jealousy that his own "new thing" was being superseded by a newer thing. Kelley reports that he accompanied Charles Mingus in 1959 to the Five Spot, the scene of Monk's own emergence a few years earlier, to hear this new phenomenon who had become the talk of the New York scene. "Hell, I did that twenty-five years ago, but I didn't do it on every tune," was his verdict, before walking out.

Reading these two books together, I'm fascinated by the similarities and sharp differences between two of my musical heroes (I'm still waiting for a definitive book on Ornette Coleman). Unlike Don Van Vliet, Thelonious Monk was musically thoroughly literate, not the naive musician he has been mistakenly caricatured as, but with a grounding in classical piano and music theory as well as the blues. That his style of playing and composing was odd, angular, surprising, was certainly nothing to do with a deficiency of technique but a matter of musical vision.

Where he resembles Beefheart is in a dogged refusal to compromise that vision, no matter how at odds it might have been with, respectively, the jazz and rock mainstreams of the day. And also in an uncompromising authoritarianism in dealing with fellow musicians in their bands. Beefheart single-mindedly pushed through his own musical ideas, concealing from the outside world how much he relied on his colleagues to translate those ideas into practice. Monk had no need of such support, but, although capable of great generosity at times, gave no quarter when hiring musicians to play in his band. "No rehearsals, no charts." As he became famous, band members were expected to pick up his tunes by ear (he often refused to give them access to the written music even when it existed), and the gig was the rehearsal. You had to shape up, or get out.

Some of that uncompromising vision is evident in Monk's instructions to Steve Lacy, which I reproduce below. I'm indebted to Barry Schwabsky for bringing this to my attention.

"It must be always night, otherwise they wouldn't need the lights." "A genius is the one most like himself." You can imagine Beefheart saying some of these things. But hey, here's Beefheart on the 10 Commandments of Guitar Playing. "1. Listen to the birds.
That's where all the music comes from. Birds know everything about
how it should sound and where that sound should come from. And watch
hummingbirds. They fly really fast, but a lot of times they aren't
going anywhere."

Etc.

All this talk of genius inevitably brings up the topic of mental illness. Someone who thinks differently, as Monk and Beefheart clearly did, isn't, of course, necessarily "ill". Beefheart was in some respects an acid casualty - the occasion when the young Ry Cooder left the incipient Magic Band in disgust after he ruined an important gig that would have got them a ticket to the Monterey Pop Festival by falling off the stage after experiencing an acid flashback is memorably described in French's book. Sometimes Beefheart used the word "schizophrenic" in self-description. But it's more likely that, if anything, he suffered more from some kind of personality disorder that affected his dealings with others.

With Monk, the diagnosis is on firmer ground. He increasingly suffered from what is now called bipolar disorder. Whether the madness = genius cliché has any traction, it is a fact that this condition adversely affected his musical performance as it got worse – and it's true to say that being black in the USA in the 50s and 60s when you're apt to act strangely from time to time is likely to lead you into harm. Monk was locked up more than once. It wasn't funny. And the end of the story – Monk, having effectively retired in the mid-70s, his music long superseded in the public's affection by the rock'n'roll that the Magic Band was already subverting, spent his last years in his room in Baroness Nica de Koenigswarter's house, semi-estranged from his dear wife Nellie, refusing to come out and play or even speak to his erstwhile musician friends – is ineffably sad.

The same year Thelonious Monk passed from this earth – 1982 – "Captain Beefheart" also ceased to exist. We have no record of Don Van Vliet's last 28 years, painting in the desert and living with multiple sclerosis, but he was, we are told, attended until his death this week (17 December 2010) by his wife of 40 years, Jan.

Kelley's book, unlike French's on Beefheart, is a meticulous piece of scholarship, if rather dull and worthy at times in its effort to document a life almost day by day. Which can't be said for the cheerful rumbustuousness of Through the Eyes of Magic. Both books, in their different ways, and for all their faults, offer wonderful insights into the respective makers of left-field music that continues to astonish me.

(Apologies if you previously accessed an incomplete version of this post - I pressed "publish" prematurely.)