The mosque at Kew Gardens stood on a little hill, close to the Chinese pagoda and the Alhambra. Of these three imitations of global architecture created between 1758 and 1763 in Princess Augusta's landscaped gardens, only the pagoda, built in brick and wood at full size, survives.

But what about the vanished mosque? To judge from an 18th-century engraving, it was closely based on the Süleymaniye in Istanbul. It is hard to imagine how such a pastiche would be received today. In the 18th century it was a sincere homage to a religion considered by intellectuals like Edward Gibbon to be far more philosophical than Christianity.

The 18th-century Enlightenment was one of the great revolutionary moments in human history. It was a time when a culture - Europe's - subjected its deepest beliefs to irony and critique. The results of such radical thinking were mind-bogglingly disruptive.

By the end of the 18th century, a European state would not just execute its monarch but abolish Christianity. In 1793, in the most extreme phase of the French revolution, the Bishop of Paris was forced to confess himself "a charlatan" and to declare that from now on "there should be no other public cult than liberty and holy equality".

This movement now has a permanent display dedicated to it at the heart of a museum that can honestly claim the Enlightenment as parent. The British Museum was founded in 1753 as a public institution where you could see all manner of natural and artificial curiosities, as well as read books.

When the museum was separated from the British Library in 1998, the King's Library - the gilded gallery built to house George III's book collection - became vacant. Now it is the home of a new permanent display that resurrects the Enlightenment museum, in which Neil MacGregor, the director of the British Museum, hopes to create a centre of global citizenship nurturing "dialogues between civilisations".

Before visiting this exhibition we might consider how unsettling and liberating "dialogues between civilisations" 250 years ago could be. People said things in the 18th century that are still risky today. Especially about religion.

The de-Christianisation of France was a direct consequence of a century of scepticism and relativism. Voltaire's Philosophical Dictionary, published in 1764, was attacked, banned and read avidly across Europe. It's not hard to see why it remained on the Vatican's index of forbidden books until the index was abolished.

The book's title is misleading: its alphabetically organised subjects (Atheism, Chinese Catechism, Fanaticism, Superstition, Toleration) provide neither a comprehensive digest of philosophical thought nor an explication of philosophical terms. Voltaire's theme is religion and his project is its philosophical critique. He ruthlessly exposes the historical implausibility of the gospels. Why, he wonders, does the ancient Jewish historian Josephus fail to mention so important an event as the life of Christ? Why, in his comprehensive account of the cruelties of Herod, does Josephus omit the Massacre of the Innocents?

Gibbon subjects Christianity to a similarly withering historical critique in his Decline and Fall of the Roman Empire (1776-88). He ironically praises the early Christian ascetics. "The acquisition of knowledge, the exercise of our reason or fancy, and the cheerful flow of unguarded conversation may employ the leisure of a liberal mind. Such amusements, however, were rejected with abhorrence or admitted with the utmost caution by the severity of the fathers..."

Gibbon is the Jane Austen of doubt, wittily chronicling Christian absurdities. One contemporary reader, James Boswell, called him "an infidel puppet". There was more to this denunciation than meets the eye. Not only did Gibbon criticise Christianity, he admired Islam.

"The base and plebeian origin of Mohammed is an unskilful calumny of the Christians," begins Gibbon's sympathetic account of the rise of Islam. He praises Mohammed as an orator and a leader, and respects the Koran. For Gibbon, Islam is attractively free from the hysterical superstitions of Christianity, tolerant in its acknowledgement of earlier prophets, including Christ. "The Koran," he writes, "is a glorious testimony to the unity of God. The prophet of Mecca rejected the worship of idols and men, of stars and planets..."

Gibbon's praise of Islam is strictly on his own terms, though. Just as Voltaire reinvented Confucianism in his own image as rational philosophy, so Gibbon imagines Islam as a religion for sceptics. "A philosophical atheist might subscribe the popular creed of the Mohammedans," he says provocatively, "a creed too sublime perhaps for our present faculties."

There was a cult of all things "Mohammedan" in 18th-century Europe, just as there were cults of China and, at the end of the century, ancient Egypt. But this went beyond building pagodas, mosques and mini-Alhambras. The most radical thinkers systematically juxtaposed the world's belief systems in order to show that insights, as well as fallacies, were common to all. They were, to use the modern word, relativists. "We are all steeped in weaknesses and errors," says Voltaire in his Philosophical Dictionary. "Let us forgive one another's follies - it is the first law of nature. The Parsee, the Hindu, the Jew, the Mohammedan, the Chinese deist, the Brahman, the Greek Christian, the Roman Christian, the Protestant Christian, the Quaker Christian trade with each other in the stock exchanges of Amsterdam, London, Surat or Basra: they do not raise their daggers..."

It is significant that Voltaire uses the image of the marketplace. Trade brought Europeans face to face with the inhabitants of remote elsewheres. The British believed it was universal human nature.

Commerce with the Ottoman empire was one of the prime ways in which Europeans came into contact with Islam. Gibbon poured cold water on western images of the Turkish conquest of Constantinople in 1453 as a one-sided tragedy, and argued that the transformation of Haghia Sophia from a church to a mosque freed the late-Roman architectural masterpiece of Christian clutter.

When Lord Elgin was sent as ambassador extraordinary to Constantinople in the early 19th century, Lady Elgin wrote home about the splendours of the Topkapi Serai: "In a window there were two turbans covered with diamonds. You can conceive nothing in the Arabian Nights equal to that room."

The objects that came from other places unsettled prejudices, although not perhaps those of Samuel Johnson. When the great lexicographer called East Indians "barbarians", his biographer Boswell objected. "'You will except the Chinese, sir?' Johnson: 'No, sir.' Boswell: 'Have they not arts?' Johnson: 'They have pottery.'" But it is Boswell who speaks for mainstream 18th-century opinion: he tolerates Johnson's philistinism as he tolerates (as a Scot) Johnson's asinine remarks about the Scottish.

Boswell's Life of Samuel Johnson epitomises the Enlightenment belief in sociability, in give and take. This ability to take pleasure in difference is a long way from the wars, tortures and martyrdoms that had previously settled theological disputes. Voltaire was psychosomatically ill every year on August 24, the anniversary of the St Bartholomew's Day Massacre of Protestants by Catholics in Paris in 1572.

Gibbon wrote scathingly about the intolerance of early Christianity: "The primitive church ... delivered over, without hesitation, to eternal torture the far greater part of the human species." This was in a country just starting to overcome the bigotry that had turned its (and Ireland's) fields red in the 17th century.

How can you contain the wild and dangerous ideas and imaginings of the Enlightenment in an exhibition - or, for that matter, a museum? Rather well, it turns out. Brilliantly, even. The British Museum has done something almost unprecedented. For years, artists, curators and historians have been imagining the museum as a work of art. But no museum of this stature has thought to make such an exhibition of itself on this permanent and serious scale - to make its own history its inquiry.

You enter the King's Library from the white contemporary space of the Great Court and are immediately in another time. Ahead of you is a giant ancient Roman vase - or rather, a fake, put together from fragments by the architect and fantastic illustrator Giovanni Battista Piranesi.

The extravagant invention of Piranesi is a reminder that not even the cult of ancient Greece and Rome in the 18th century can easily be understood as European triumphalism. Piranesi portrayed ancient Rome, in his prints of colossal ruins and prisons, as a place of diabolical sublime power very similar to the realm of William Beckford's fictional Caliph, Vathek.

It turns out that beneath its modern collections, the British Museum has long hidden a ghost collection, an embarrassing trove of oddities dating back to its origins. Now these have been brought out of basement cupboards to once more delight and instruct.

There is a "mermaid" made from a monkey and a fish; there are numerous classical fakes; and, contrary to every expectation you might have of the British Museum, there are animals. The original 18th-century museum encompassed natural history as well as every other knowledge, and so the King's Library has items from the foundation collections of today's Natural History Museum, including species novel and confounding in the 18th century, such as the platypus.

Fossils, minerals, mummies (including the "mummy's finger" used in the early 18th century by Hans Sloane, the doctor whose collections were the foundation of the museum) and objects brought back from Cook's Pacific voyages: curiosities without end, representing an age whose curiosity was insatiable. The 18th century was interested in modelling knowledge, in encyclopedias and dictionaries. Collecting was another encyclopedic enterprise. At this stage it did not yet sustain the ordered, and ordering, classifications of Victorian science. The world for a moment was held in suspension, in a vacuum flask, and everything - from the age of the earth to the meaning of hieroglyphs - was still to be asked.

Islam, too, was to be considered, to be studied. Sloane's cabinet of curiosities contained amulets inscribed with texts from the Koran that were carefully translated for him into Latin; the entire Koran was translated into English in 1734.

Among all the tales of exploration and interpretation, a cynic might say, this wonderful display conveniently distracts attention from the most notorious skeleton in the cupboard of the museum's early history: Lord Elgin's removal of the frieze of the Parthenon from Ottoman-ruled Athens in 1799. But who was the man of the Enlightenment - Elgin, or his critic Lord Byron?

Elgin and his party were magnificently entertained in Istanbul, and their relationship with the rulers of the Ottoman Empire was good enough to procure a licence from the Sultan to remove "any pieces of marble with inscriptions or figures thereon" from the Acropolis. Ever since, critics have attacked the value of this licence and the propriety of Elgin making a deal with the despotic rulers of Greece. But Elgin was making the assumptions of the 18th century. Christian demonisation of Turkey was specious, argued Gibbon. The Marbles were a legitimate trade in the emporium of the Enlightenment, thought Elgin.

In the Romantic age that followed the Enlightenment, even as Napoleon read and admired the Koran, modern, ethnocentric nationalism was born. The Greek war of independence became a rallying call for Christian Europe against Islamic Turkey. And among all the other thoughts provoked by this new display, one is that maybe the campaign to return the Elgin Marbles to Athens can never be entirely free of the new Romantic faiths that ended the Enlightenment and initiated a far less attractive version of modernity.

· Enlightenment: Discovering the World in the 18th Century is at the British Museum, London WC1. Details: 020-7323 8000.