"How a man's heart swells within him, when after sea and sky and sky and sea for nearly a month, he first sees the kindly land beckon to him over the salt waves! And that land tropical!" This was how Victorian writer Henry Nelson Coleridge responded to reaching Bridgetown in 1841. He particularly liked the fact that "Barbados is the most ancient colony in the British empire."

Today when we read about Barbados it usually has something to do with Michael Winner, Rihanna, or Simon Cowell's swimwear. This is the Barbados of the super-deluxe Platinum Coast – the western side of the island, stretching up from the capital Bridgetown towards Holetown and Speightstown, of fantastic beaches and crystal Caribbean sea.

And the real estate is something to behold: the Four Seasons hotel complex followed by the Sandy Lane beach resort, the swanky Cliff restaurant, and then up to the exclusive Port Ferdinand marina. No recession here, as engineers work around the clock to dredge another huge dock and carve out a gated community for the yachting classes.

But there is another Barbados. This island, the most easterly of the Caribbean, has a remarkable history stretching back, on the one hand, to the Amero-Indians of 4,000BC and on the other to the apex of British imperialism. And, unlike some former British colonies, it has enough confidence and sense of its own identity to explore this history in a creative and compelling manner. In Barbados you can combine the indulgence of the Platinum Coast with a rewarding account of the past.

On the route from Grantley Adams International Airport towards Bridgetown, you pass Bussa Statue – a heroic depiction of a slave breaking his chains, inspired by the so-called Bussa slave rebellion of 1816. It is a healthy reminder that the landscape of modern Barbados was laid out on the back of African slave labour. "The whole is a sweet Spot of Earth, not a Span hardly uncultivated with Sugar-Canes; all sides bend with an easy declivity to the Sea, and is ever green," was how one visitor described the island in the 1730s.

For Eric Williams, the Marxist historian and future prime minister of Trinidad, the Caribbean was the "hub of Empire" – the richest place on the globe thanks to its exponential export of sugar and molasses (for rum). Beginning in the 1640s, English planters transformed this island wilderness into an agricultural engine dotted by boiling rooms, windmills and furnaces. It delivered record profits for them and a miserable existence for the tens of thousands of slaves who worked and died in the cane fields.

All of which is expertly recounted at St Nicholas Abbey. Located at the end of an ancient mahogany tree avenue, the "Abbey" (in reality a private home) is one of the very few Jacobean plantation houses still in existence. Its whitewashed 1650s gabled frontage would instinctively put you in Dorset – but the surrounding cane fields, stifling heat and boiling-house chimney brings you back to sugar-and-slave country. Now owned by a wealthy Bajan architect, St Nicholas Abbey is a well-preserved museum successfully recreating 18th-century plantation life, complete with Wedgwood pottery and Chippendale furniture. In the tropics, the accoutrements of Englishness were more important than ever.

The abbey provides a riveting guide to the sugar economy underpinning the house's wealth, complete with an inventory of the slaves drawn up for compensation purposes following abolition. The planters expected $190 for the strongest workers as well as women of child-bearing age. But this isn't just an exercise in nostalgia. The abbey now has its own distillery, and in the cane-harvest months (January to March) you are able to watch that hot, intense process of crushing, boiling and fermenting which produces the glorious nectar of Bajan rum.

Placed in hogshead barrels, the sugar was exported via Bridgetown – the port city which became, in the 18th century, "the London of the West Indies". This was the cultural capital of the Caribbean, with theatres, newspapers, clubs and a statue of Admiral Lord Nelson. Today it's more about tax-free shopping, diamond stores and catching a deep-sea diving rig, but the history here is equally worth exploring. The synagogue, dating back to the 1650s, highlights the early impact of the Jews in Barbados, fleeing the Inquisition in Brazil and bringing with them their knowledge of sugar-cane cultivation and slave financing. The nearby Parliament building (established in 1639) houses an excellent account of the history of democracy in Barbados, while the National Heroes Gallery shows footage of the Queen knighting cricketer Garry Sobers at the Savannah racecourse.

Among those drawn to Bridgetown in the 1700s was an impressionable young land surveyor from Virginia. Hoping the sea breezes would help cure his half-brother Lawrence of tuberculosis, George Washington spent a bucolic few months riding across the plantations, visiting the theatre and taking in colonial society. He also took note of the British defences at James Fort and Charles Fort, guarding the entrance into Carlisle Bay.

The George Washington House Museum overplays its hand when it suggests you cannot understand the American revolution without the Barbados connection. But what this museum does emphasise – in another fine 18th-century domestic setting – is how Barbados and the Thirteen Colonies of America formed part of a broader Atlantic Empire. There were strong commercial and cultural connections between Virginia, Massachusetts and the Caribbean right up to 1776. And then, after a suitable pause, trading resumed again with an independent America.

On the other side of the Garrison district, where the 15,000 British army forces used to billet, the Barbados Museum provides the official history of the island. Located in a former prison block, this is a museum in need of refreshing. However, it does provide a proper account of Barbados's African heritage – the thousands of slaves taken from Angola and the Congo across the murderous "middle passage" and into the cane fields. With skill and scholarship, these galleries trace a cultural inheritance systematically eliminated.

A mural for Carib beer. Photograph: Richard Cummins/CORBIS

Beyond the museums and galleries the wonder of Barbados is the historic fabric still in existence. You need to see it soon, because the old cane fields are being turned into golf courses and the chattel houses knocked down for condominiums. But the plantation houses, urban villas, Bridgetown warehouses and, above all, a proud array of Anglican churches remain a remarkable testimony to an imperial past. And there is a great sense of ownership about them. The Barbados Museum, the island's National Trust and the tourism and planning departments all understand the value of this heritage.

Of course a trip to Barbados might ultimately be about the sea, the sun and the rum. But it can also be about a series of overlapping Amero-Indian, African, American and British pasts. The island was made rich by its industrial exploitation of sugar cane. Its economy is now dependent almost entirely upon tourism and it has found in its colonial heritage an equally valuable asset which it is exploiting with similar effectiveness.

Tristram Hunt is currently working on a book about the cities of the British Empire