Early on in his book Eugene Rogan, who teaches the modern history of the Middle East, confesses that in "any free and fair election in the Arab world today, I believe the Islamists would win hands down". Again, towards the end of this engrossing and capacious book, he reiterates that the "inconvenient truth about the Arab world today is that, in any free and fair election, those parties most hostile to the United States are most likely to win".

Today, Arab fear of the west and resentment at the humiliating and socially damaging effects of westernisation fuels Islamism and the spread of terrorism. How have we come to this pass? Rogan answers this question by tracing the history of Arab hopes and ultimate disappointments from the early 16th century, when the Ottomans conquered most of the Arab world, to the present day. This is primarily a modern history, and the 16th, 17th and 18th centuries are briskly treated.

That was an age when the Arab provinces of the Ottoman empire were ruled by despotic local kleptocrats. In The Arab Awakening (1938), the Palestinian historian George Antonius wrote of the period: "Sensational figures stalk across the stage of these three centuries, now martial and heroic like Fakhruddin and Daher al-Umar, now merely brutal and sanguinary like Ahmad al-Jazzar and the Mamlukes of Cairo; but always solitary and self-seeking. They appear and disappear in tedious succession . . . never overthrowing or seriously threatening the hold which Soliman the Magnificent had fastened on the Arab world."

Rogan's narrative slows as he details the growing familiarity of the Arabs and their Turkish, Circassian or Albanian masters with European manners and technology, and the expansion of commercial links with Europe. He proceeds to the opening decades of the 20th century, when the Ottoman grip over the Arab provinces actually strengthened, thanks in part to the spread of railways and telegraphy. But Turkey's defeat in the first world war, coupled with ambiguous promises by British politicians and starry-eyed pronouncements by Woodrow Wilson, led early pan-Arab nationalists to believe their moment had come.

Their disappointment was bitter when the victorious British and French carved up the Arab lands as spoils of war. It was only in the 50s and 60s that the British and French moment in the Middle East came to an end. However, the triumph of local Arab nationalism in Egypt, Lebanon, Tunisia and elsewhere meant the abandonment of the pan-Arab dream. There were other disappointments, of which the catastrophic defeat of the Arab armies in Palestine in 1948 was the most obvious.

Algerian independence was achieved at the cost of a remarkably hard-fought and bloody war. The attempt by Egypt and Syria to form a United Arab Republic was a disaster. So was Nasser's intervention in the Yemeni civil war. Above all, the socialist policies pursued by nationalist regimes such as those of Nasser in Egypt and the FLN in Algeria failed to deliver prosperity. That, and the increasingly brutal methods of repression employed by secular nationalist regimes, has inspired many to place their hopes in the Muslim Brotherhood and other Islamist groups.

The vivid narrative of The Arabs is based on frequent recourse to quotations from witnesses to the events they describe. Thus, Budayri, a barber diarist, recorded the barbershop gossip in 18th-century Aleppo. The historian Jabarti observed the arrival of Bonaparte's army in Cairo in 1798 with admiration tempered by a heavy admixture of cynicism. He gave an account of Bonaparte's savants conducting flashy scientific experiments, hoping to awe the city's religious scholars. They were not so easily impressed, and one of them told Bertholet: "This is all well and good, but can they make it so that I would be here and in Morocco at the same time?" When Bertholet shrugged his shoulders, the Egyptian scholars concluded that French sorcery wasn't up to much.

Rifa'a al-Tahtawi's account (translated into English as An Imam in Paris (1826-1831)) detailed an Egyptian's impression of the manners and customs of the French in the early 19th century. His observations mingled admiration with revulsion. For example, he was shocked to find that in France the "men are slaves to the women here and under their command irrespective of whether they are pretty or not".

Faisal, who was imposed by the British as king of an artificial nation – Iraq, in 1921 – wrote of his unloved and unloving subjects: "There is still – and I say this with a heart full of sorrow – no Iraqi people, but unimaginable masses of human beings devoid of any patriotic idea, imbued with religious traditions and absurdities, connected by no common tie, giving ear to evil, prone to anarchy, and perpetually ready to rise against any government whatsoever." It is as if he was writing to warn the American and British troops occupying Iraq in the 21st century.

Then there is Sayyid Qutb, Egyptian litterateur and boulevardier turned Islamic fundamentalist, inveighing against the immorality of the lyrics of "Baby It's Cold Outside". Qutb's Qur'an-based opposition to westernisation and the brutal regime of Nasser took him to the gallows in 1966.

Britain and France do not come out well. For example, in 1906 a British hunting party shot the pigeons belonging to the peasants of the village of Dinshawy on the Nile Delta. In the riot that followed, a British officer was injured and later died. Four Egyptians were hanged and others were sentenced to long terms of hard labour or floggings. Such high-handed justice provoked nationalist resistance in the decades that followed, but Bimbashi McPherson, the British security chief who faced down strikes and demonstrations in 1919, was contemptuous: "Howling lunatics in the streets, women emancipated for the occasion making stump orations, children and rapscallions of all sorts shouting ribald doggerels in contempt of fallen tyrants."

In 1918, the British who took over from the defeated Ottomans in Iraq promised a national government and self-determination, before imposing by force of arms a British mandate on the hostile Arabs and Kurds. The French behaved with even greater colonial arrogance and brutality in Morocco, Algeria and Syria.

One of the depressing if instructive messages of this book is that terrorism can work. In the early 1940s, while Britain was at war with Nazi Germany, Menachem Begin, the leader of Irgun, and Yitzhak Shamir, the leader of Lehi, waged terrorist campaigns against the British in Palestine. In 1945, they were joined by the larger Haganah. Ninety-one people were killed when Irgun operatives blew up the King David Hotel in Jerusalem. Such atrocities played a leading role in persuading the British they could not continue to exercise a mandate in Palestine. They withdrew in 1948, leaving the Palestinian Arabs to face the much better-armed and organised Jews.

In the war that followed, Rogan says the "image of a Jewish David surrounded by a hostile Arab Goliath is not reflected in the relative size of Arab and Jewish forces". In general, he has a taste for revisionist and counter-intuitive history. For example, Israel's occupation of the West Bank made it easier for Yasser Arafat's Fatah to organise resistance. And Sadat's expulsion of Soviet military advisers in 1972 led to Russia increasing its military supplies to Egypt.

Rogan was a student of Albert Hourani, the author of A History of the Arab Peoples, an eloquent and predominantly upbeat account of Arab achievements over the centuries. Rogan's version, hard-nosed and sadder, is no less eloquent, and compulsively readable.

Robert Irwin's books include For Lust of Knowing: The Orientalists and Their Enemies (Penguin).