Iain Sinclair is notorious in Austin, Texas. Seven years ago he sold his literary archive to the Harry Ransom Humanities Research Center, and since then he's been shifting his papers from a scruffy lock-up in east London to the HRHRC's deodorised storage facility in downtown Austin. He's notorious because his archive arrives pretty much as it was shovelled out of the lock-up: a jumble of documents, dust, mud, mould and dead insects, "40 years of scribble and grunt" heaped into hundreds of binbags and boxes.

Pity the poor archivists, you're thinking. But pity the poor PhD students more. Already they're descending on the Sinclair holdings: fossicking through his notebooks and shopping lists, dowsing for the connections that might exist between a detail here and a fragment there, psycho-navigating the river-silt and the fly corpses. Where medieval scholars wear white cotton gloves to turn the pages of ancient manuscripts, Sinclair scholars presumably don face masks and anti-contamination suits: PhD as CSI. Hear that sound? That's the lip-smack of latex gloves being snapped on before they root around in Holding R17157/Box 9/File 4/Sinclair, I. Hear that one? That's the clickety-clack of scores of graduate dissertations currently under composition: "Textual Detritus in the London Writings of Iain Sinclair", or "Pushing the Baby Out with the Ba'ath Water: Hackney as Fallujah in Iain Sinclair's Riverine P(e)rambulations".

Don't think they aren't being written. In a little under 15 years – since the success of Lights Out for the Territory (1997) – Sinclair has gone from cult author to national treasure: quite a career path for a man who started out hawking his self-published prose-poems round the book-barrows of Farringdon and Stratford. I would guess him to be, along with his friend and mentor JG Ballard, among the most written-about of contemporary British authors. Sinclair symposia, journal special issues – they just keep coming. Neo-modernist, high-output, exceptionally gifted and prone to antic pattern-making, Sinclair is immaculately qualified for the attention of today's Eng lit academics.

How best to describe Sinclair? East London's recording angel? Hackney's Pepys? A literary mud-larker and tip-picker? A Travelodge tramp (his phrase)? A middle-class dropout with a gift for bullshit (also his phrase)? A toxicologist of the 21st-century landscape? A historian of countercultures and occulted pasts? An intemperate Wall-E, compulsively collecting and compacting the city's textual waste? A psycho-geographer (from which term Sinclair has been rowing away ever since he helped launch it into the mainstream)? He's all of these, and more.

If you know anything of his work, you'll be aware that walking is his main method and the city his chief subject. The lines of inheritance back to Guy Debord and the situationists are now well established, as are those to Baudelaire's "flâneur", Thomas De Quincey, Paul Auster, Hunter S Thompson and Patrick Keiller. In London Orbital (2002), his 600-page account of walking the M25, Sinclair describes himself as a "fugueur", preferring the word for its hint of mysticism, its dissent from the pomaded preenings of the "flâneur".

Sinclair's oeuvre now runs to seven books of fiction, 14 of poetry, and 12 of "documentary", of which Ghost Milk: Calling Time on the Grand Project is the most recent. The documentary work alone constitutes an extraordinary literary cartography of London and its edgelands. Over 15 years and getting on for 20,000 pages, Sinclair has delved into the history of a single East End building (Rodinsky's Room, co-written with Rachel Lichtenstein), investigated the city's interior (Lights Out for the Territory), stalked London's asphalt rimrock (London Orbital), and struck out into the GM wheatfields, light-industrial estates and "extruded exurbia" of its fringe counties (Edge of the Orison). At least three of these books are already, and deservedly, classics.

Ghost Milk stands in partnership with its immediate predecessor, Hackney, That Rose-Red Empire (2009). Hackney was a 600-page "deep map" of the area where Sinclair has lived for decades. The book sprawled, falling foul of the same mistake that did for Borges's imperial cartographers: the aspiration to an impossible 1:1 mapping of a territory. Sinclair couldn't resist jamming everything in: diary entries, interview transcripts, letters and "deep-memory interrogations", as well as descriptions of his walks and recollections. He reminds me at times of another Borges character, Funes the Memorious: the young man who, after a riding accident, finds that his memory is faultless in its recall of detail. Like Funes, Sinclair can seem cursed by hypermnesia. "The born-again flâneur," he wrote in 1997, is "a stubborn creature, interested in noticing everything." Such stubbornness brings problems as well as virtues – in particular a congested stylistic excess, and a troublesome equivalence of detail.

Ghost Milk has many of the same problems as Hackney. It's over-stuffed, indulgently prolix and maddeningly dispersive. What saves it, though – indeed, what makes it brilliant – is its fury. Anger drives the book forwards, and pulls its details into suggestive order: anger at corporate conjurings, civic hubris and "lachrymose orgies of nationalism". Ghost Milk is documentary writing as opposition, literature as resistance. Or, as Sinclair more calmly puts it, it is an example of memoir operating as "an element within a larger social argument".

His immediate argument is with the London Olympics; his broader target is those projects of civic enhancement that acclaim themselves as "regenerative" and find their expression most charismatically in architectural "grand projects": domes, stadiums, mega-sculptures and super-cities. "We live," Sinclair writes, "in the GP Era", and for him the "GP" will always be a function of egotism and profit: a longing on the part of men in suits to leave behind a legacy, and a longing on the part of developers to make a quick buck. "Ghost milk" is Sinclair's term for the cultural ooze that such projects exude: all those delivery documents, those primary strategic objectives, those maquettes and futuramas of the world-to-be. "Ghost milk," he writes, means "CGI smears . . . Real juice from a virtual host. Embalming fluid. A soup of photographic negatives . . . The universal element in which we sink and swim."

For Sinclair, the grandest of these folies de grandeur, the whitest of these white elephants, is the 2012 Olympics site, whose construction has been under way in east London for four years now. The early chapters of the book take us back to Sinclair's Stratford in the early 1970s, when he was doing hand-to-mouth day labour, working alongside Nigerian entrepreneurs and Irish brickies. He also details a series of postwar attempts to "regenerate" Hackney and Stratford, each accompanied by its signature grand project, such as Joan Littlewood's never-built "fun palace". These were all, in Sinclair's account, rehearsals for 2012, part of "the long march towards a theme park without a theme". There are hints of classic psychogeography here in this idea of a territorial imperative, of a place possessing and predetermining its own outcomes. Sinclair's Stratford is pre-lapsarian, but the seeds of its own fall are already present.

Elegiac notes are often struck. The book is dedicated to "the huts of the Manor Garden Allotments" (bulldozed to make way for the Olympic Park), its first section is entitled "Lostland", and it stands in one sense as a dark wunderkammer, memorialising the vanished and abolished of Stratford and Hackney. Entertainingly bitter chapters diarise the coming of the Games: the "plague of surveyors [and] hard-hat engineers", the serving of compulsory purchase orders, the futile local resistance, and the erection of the notorious blue fence that marked the perimeter of the site. Joggers and walkers find their usual footpaths blocked. Gurkhas are brought in to guard the "inner ring" of the main site; white-British manpower does the security grunt-work on the perimeter; unmanned surveillance drones fly high overhead; and walkers taking photographs of the fence have their film seized or their memory cards wiped.

I joined Sinclair four years ago to walk the boundary of the Olympics site, shortly after the blue fence had gone up. We walked it anti-clockwise, keeping the fence to our left, and I remember being surprised even then by the vehemence of Sinclair's anger at its presence. We ended that day at the last of the Stratford estates to hold out against the Olympic Development Authority. It was a remarkable scene; part Ballard and part Alamo. Flags of resistance drooped from the balconies of the doomed buildings, while all around the earth-movers were already at work, clearing the rubble and cleansing the soil.

"Circumambulation of the Olympic site," Sinclair confesses, "became an addiction"; his secondary addiction is to narrating those walks. Again and again he describes trying to "cut through the defensive ring" of the blue fence by means of trespass, disguise and chutzpah. But of course he's no match for the security guards and finds himself "held", "ejected" or apologetically escorted off the premises. This repeated act of frustrated pilgrimage becomes the charismatic motion of Ghost Milk: flâneurism radically repurposed, a kind of weaponised walking.

Sinclair's hatred of the Olympic project is unambiguous. But while we know what he loathes, it can be hard to tell what he loves. He wants to see the River Lea once more "shared by oarsmen, narrow-boat dwellers, dog-walkers, wanderers who were not filmed, not challenged by security, trusted to make their own mistakes". Fair enough. He wants, surely, to let Stratford and Hackney locally self-determine rather than have large-scale regeneration foisted on them. But hold on, no, because "if there is a less enticing blot in this country than the haemorrhaging roadcrash of the area surrounding the [Stratford] transport-hub station", where the air is so bad that it requires "gills and built-in decontamination filters" to breathe, then Sinclair has "been fortunate enough to avoid it". So why not regenerate central Stratford, then? Because that would be to succumb to "warped utopianism". Ah, right. There's a word that Sinclair seems to want to use but doesn't, presumably because of its recent contamination by ghost milk: that word being "community", as (perhaps) embodied in the Manor Garden Allotments and as (perhaps) connoting a local culture that is improvised, long-term and mutual, existing as an accumulation of small and reciprocally beneficial transactions.

Even as he denigrates civic utopianism, a different kind of dreaminess underwrites Sinclair's own position: a longing for a London that might well never have been. WH Auden once described his ideal landscape as one that contained water-wheels, grain-mills, limestone and obsolete machinery. What would Sinclair's ideal resemble? Railyards and pin-wheel poetry presses, certainly. Dockers and detectives rubbing shoulders in greasy-spoon cafes. Mystics, cranks and quiet pilgrims wandering together down towpaths. Urban planning would be handled by Andrei Tarkovsky, Allen Ginsberg would potter around handing out bennies and yodelling protest songs, the odd authenticating psychopath would occasionally commit discreet murders and, once a century, through would stride Alfred Watkins and Eric Gill, each man clutching his penis before him like a ley-liner's staff as they dowse lines of heat and force.

If you've read Sinclair before, you'll know how disorienting and exhilarating an experience it can be. If you haven't, you should – and you're in for a surprise. It can take time to acquire a reading method for Sinclair's style: time to develop the necessary stamina and the requisite orientation devices. His style is forcibly intransitive: verbs are deprived of their objects, prepositions are suppressed, conjunctions vanish, full-stops proliferate. Passage through his prose is demanding, immersion is obligatory, and cognitive dissonance is high. How do you make your way in a paragraph such as this, for instance?

I was convinced that my earlier hunch was right: buried inside the oval of the stadium was a particle accelerator. Relativity, the old Lea Valley space-time mush, was being scrambled. Outside the circuit of the blue fence, voodoo snakes, big-mouth crocodiles and eviscerated chickens were screaming on walls: Berlin '36. Mexico City '68. Munich '72.

Sinclair is often described as "digressive", but the term doesn't work, for it suggests a central path from which his divagations occur. A buzzword from the emergent critical vocabulary of "ergodic" literature may help: one doesn't read Sinclair so much as "navigate" him. It's true that opening one of his books can feel like stumbling on to a web page pre-loaded with pop-up windows. Pop! Pop! Pop! You can't close them fast enough, and eventually the screen whelps itself into incoherence. Proper nouns are especially high-density in Sinclair's prose. Placenames, cultural references and characters surface and then sink again, like chunks in a roiling pot of chowder. Sinclair veterans will know some of the characters in Ghost Milk from earlier books: Renchi Bicknell, the Blake-inspired artist and walker; the visionary Hebridean sculptor Steve Dilworth; and Sinclair's long-suffering and raspingly sceptical wife, Anna.

At one point, Sinclair deplores the Olympics site for its "refusal to connect with the street". The creation of connection is both the stylistic habit of Ghost Milk and, implicitly, its moral task. "Keeping a record of Olympic connections wherever I find them," Sinclair writes, "I snap the Bei Jing (Fish and Chips + Chinese Meals to Take Away), on a traffic island known as the Quadrant." That "+" might serve as the sign under which to read this hyperconnective book, this "museum of affinities" which has Sinclair as its madcap curator.

Connectivity can cause trouble, though. One of the book's least appealing aspects is its tendency to loose comparison: Stratford as Gaza, New Labour as Nazis, the O2 Dome as "England's Guantánamo", the Lea Valley as "our Poland, fought over by eco-romantics, entrenched Stalinists and political visionaries". The analogous is not always the same as the commensurable. Elsewhere, Sinclair's urge to connect manifests as frantic hyperlinking ("The panorama from [the] Stratford tower was echoed by the establishing shots in Bronco Bullfrog, the 1970 film by Barney Platts-Mills"; a sentence that is followed by fully five pages analysing Bronco Bullfrog) or as repetition. The department store of Bentalls in Kingston upon Thames is a "baroque reef": a nice image, until you encounter, 100 pages further on, "reef-buildings . . . in a sea of concrete" and then, 100 pages after that, "a reef of fabulous stadiums". "I feel the jolt as weary metaphors turn themselves inside out," writes Sinclair; so, at times, does the reader. But much more often you feel the jolt as new metaphors spring astonishingly into being.

Like all of Sinclair's "documentary" works, Ghost Milk proceeds by pattern rather than plot. He long ago made his "final renunciation of the burden of narrative": while the rest of us writer-walkers are pencil-chewing over the nature of our quests and how to get meaningfully from A to B, Sinclair just dodgems happily about. "I can't return to a place that is no longer there, my Olympic Park banishment is absolute. I think I'll aim at Morecambe." OK, Morecambe it is then. In this haphazard way, Stratford becomes the transport hub of the book, sending Sinclair out to the sites of other grand projects: Manchester and Hull (where Will Alsop proposed his M62 super-city), Morecambe and – by an effort of projective imagination rather than actually going there – Beijing.

Some of these side-trips feel vestigial, but Sinclair's visit to Athens – told in the penultimate chapter of the book – is superb: melancholy and acerbic. In Greece he finds Stratford's soured future; a real-life posthumous grand project. He wanders barely used stadiums and through echoey atria, and no one tries to stop him because no one cares and there's hardly anyone there. Meanwhile, off stage, the Greek economy collapses, and Sinclair understandably can't resist making a causal connection between the "nation's bankruptcy" and its Olympic folly. "The Games are just empty buildings," one Athenian tells him, "we have no use for them. But they have become monuments, so we can handle them and live with them. We are used to living among ruins. They are just ruins, they were never anything else."

The final journey in Ghost Milk is to America, where Sinclair calls in on the Harry Ransom Center to see his own archive in its final resting place: "a selective catalogue of human culture preserved against the coming nuclear winter". Don't worry – the irony isn't lost on Sinclair, who is almost always his own first and fiercest critic. He knows that in selling "the memory vault" to the Texans he has aligned himself in some sense with Seb Coe, Joan Littlewood and the other grand-projecteers: motivated, like them, by money and the thought of a legacy. For a moment, you think that the whole book has been working up to this point: that the grand project on which he's "calling time" is in fact his own writing, and that Ghost Milk is his resignation letter, published before he retires and takes up watercolours. But then Sinclair is off walking again, somewhere on the Pacific coast, and there are hints of the next book to come: "Tomorrow we would head south for that Mexico of the mind. I wanted to know just what happened when you walked thirty-five kilometres out of Guadalajara" (I'd imagine that you'd get taken hostage by a drug gang, but Sinclair seems to have survived).

"If we fail to . . . hammer out a mythology of our own, we are lost," he writes. Ghost Milk is his attempt to hammer out a mythology, or at least a language, that might contest snake-oil politics and "heritage" history. The book chronicles and contributes to a battle between cultural forms: PowerPoint presentation versus documentary non-fiction. In the end, of course, Sinclair will lose. In fact, he lost long ago. In terms of "delivery", the PPP will always beat the DN-F. Ghost milk will swill away Ghost Milk. Sinclair's acknowledgment of the futility of his task adds to the achievement of this brilliant and flawed book.