Before the publication of Iain Sinclair's last book, Hackney: That Rose-Red Empire, he was due to talk in one of the borough's libraries.

To his surprise, the library cancelled, apparently acting on orders from Hackney officials. Sinclair had criticised the development of Stratford for the 2012 Olympics, so he was persona non grata. The fact that the book – which the censors seem not to have read – was a eulogy to (old) Hackney was an irony not lost on Sinclair. So the promise that his new book would explore further his disgruntlement about the folly of this "vanity project", the 2012 Games, was enticing.

Ghost Milk mixes memoir, geotopography and an examination of ambitious public projects founded on political expedience. Unfortunately, its meandering style is frustratingly difficult to digest. Sinclair treats the book like a jolly, boozy, one-sided conversation. The pages are littered with references to friends, both famous (e.g. JG Ballard) and unknown. This is engrossing in parts, such as the commentary from the Chinese poet Yang Lian. But as soon as the reader gets to grips with one setting, Sinclair scoots off on another tangent. The chapters start with vague premises of themes, but stems have sidetracks which themselves have sidetracks. It's the journey that counts for Sinclair, but this means that destinations are abandoned without full dissection, and key themes only returned to 200 pages on. Sinclair also has an idiosyncratic style in which scenes are often described without recourse to verbs: "A warm breath, windows down, of a green morning." This technique works better in scripts.

This is all a great pity because Sinclair is an incisive writer when he stays on track. What he tells us about the Olympic development plan is shocking. It was sold as being beneficial to the lives of residents, but Sinclair says that the residents had far better amenities before the builders arrived. Allotments were bulldozed; car-parks will appear where once there was greenery.

Sinclair refers us back to the philanthropists of the 1930s who improved the lives of East Londoners by building leisure facilities. And he explains in excoriating detail the escalating costs and questionable practices associated with the Olympic development. According to him: contractors rarely give true estimates for grand public projects for fear they'd never be picked; £100m, earmarked as compensation to those ousted from their homes and businesses, has "gone missing"; and development agencies have seen such things as the payment of £13.5m to a Christian centre to vacate a site that it was occupying illegally. More disturbingly, Sinclair alleges that deeply-buried radioactive thorium, from the manufacture of fluorescent watch dials, has been exposed by the development and has contaminated the water table.

There is also a sobering litany of evidence from previous Olympic cities that the legacy can be dire. In Beijing, thousands were evicted from courtyard-centred homes and consigned to tower blocks sandwiched between motorways. In Mexico City in 1968, the President was urged by the Olympic Committee to deal with protesters: 10,000 troops were dispatched, and estimates of those killed range from 44 to more than 300.

Infuriatingly undisciplined but embedded with nuggets of brilliance and of gorgeous urban poetry, Sinclair needs a ruthless editor to render his rambling wanderings more palatable.