There seems to be no end of books about the British empire, and the fascination it holds for historians of all descriptions is inexhaustible. But while the British empire is easy to write about, it is very difficult to summarise. This is because what we call "empire" spans 400 years in time and thousands of miles in space; every continent on Earth was directly affected by it.

The empire existed as a concept in the early 17th century and lasted, as a factor in world affairs, to the second half of the 20th century. Yet it changed so much in that time that it is difficult to trace any continuities or consistent characteristics it may have possessed.

This diversity makes it easy for historians to write what they like about empire, but that doesn't mean that certain facts can, or should, be obscured. For much of the colonial empire, for most of the time, there was no idea of promoting democracy; the slave trade, in the earlier days of empire, was an unmitigated evil; the Victorian public school was a centrally important institution of empire; sport was highly valued as an arena in which manly virtues and character could be displayed; there was a wide degree of racial arrogance on the part of the administrators.

It is to the credit of Jeremy Paxman that his sane, rather detached view of empire understands these truths. Paxman has an eye for detail, and his sharp intelligence quickly detects, and ridicules, the eccentric and plain weird. Any wider conclusions are, however, avoided in this readable introductory account.

Paxman is also very conscious of the idiosyncrasies of so many of the servants of empire, describing TE Lawrence as a figure in the tradition of Livingstone and Gordon, "the courageous, half-cracked maverick unshakeably loyal to the empire". In an equally shrewd aside, Paxman observes that a surprising number of empire-builders died childless. Such comments reveal a feel for the subject, even if the book does not give many indications of long hours spent among dusty archives.

Writers on this sprawling, often nebulous subject are often anxious to define what they mean by empire, or to set out limits to their subject area. Paxman, with characteristic insouciant cockiness, doesn't bother. So, characters as diverse as Walter Raleigh, Winston Churchill, Mahatma Gandhi, Clive of India, Cecil Rhodes, Kitchener, David Livingstone and TE Lawrence, to name but a handful, crowd the narrative of this new book. Consequently, the effect is rather rambling: dry and self-assured, yet without structure or extensive elucidation of key themes.

Geographically, Paxman covers the raj in the 19th century, India under Robert Clive and the Mughals, the Middle East, Kenya, South Africa, Sudan, west Africa, Australia, New Zealand, Canada, even Ireland. Such a widespread coverage in a book of fewer than 400 pages is impressive, but it does lead to an impression of breathless chaos – much like the empire itself, one can imagine. At the end, readers will be left scratching their heads, trying to make sense of the whole thing.

The rather rambling nature is revealed by the fact that in this slim, elegantly produced volume, Henry Newbolt's famous (or notorious) imperial anthem, "Vitaï Lampada", is reproduced in full not once but twice, on pages 12 and 196–7.

Paxman's subtitle is "What Ruling the World Did to the British", and in this phrase he pays homage to the household gods of modern contemporary history. It has been fashionable, of late, to say that the British empire changed Britain as much as it did the colonies, but what fresh insight is gained from this trite observation remains obscure. Paxman nods obediently to this, but one detects a lack of engagement with the subject; he mentions that several British cities have been changed by immigration from former colonies, and that there is now more variety of cuisine to be found on the high street, but he reaches no earth-shattering conclusions on this interesting theme.

Of course, as someone who has just written a book about the British empire, I am very well aware how easy it is to pick holes in any book on this subject. The common objection, the easiest comment in the world to make, is the "What about ...?" question. Any book on empire will omit, by necessity, vast tracts of the imperial experience, and so critics can easily find facts and details to contradict an author's bold generalisations.

This hole-picking is a rather futile game; any first-year undergraduate can play it. It is, of course, much harder to try to generalise, to make some coherent story out of a phenomenon as extensive and durable as the British empire. Naturally, Paxman is too cool, sophisticated and careful a journalist to attempt anything resembling a grand theory or central argument about empire. His book is episodic, fragmentary and languid in its approach. Reading it felt rather like being led through a tropical garden by a weary retired army officer. Nothing really excites the author: though he expresses admirable outrage at the barbarities of the slave trade, the tone is one of detached, knowing indifference.

The strength and weakness of Paxman's book are therefore in its range and diversity. As a taster, a teasing amuse bouche, Empire: What Ruling the World Did to the British is effective and entertaining, yet it seems unlikely that a devotee of imperial history will learn much from this performance.

Kwasi Kwarteng is Conservative MP for Spelthorne and author of Ghosts of Empire (Bloomsbury)