It is on the world's largest lake, has hillsides that sprout fire, and the national game is nerd. This is Azerbaijan. Proud winner of Eurovision and proudly independent since 1991, Azerbaijan is probably where the Vikings came from, and definitely has the most mud volcanoes in the world. But a herdsman living 2km from the volcanoes had never heard of them.

Tourism is in its infancy here. You share your flight to the capital, Baku, with a load of oilmen. Sundry Nobels and Rothschilds helped found the world's first oil and gas industry here, and today you fly in over thousands of smudgy, nodding-donkey wells, and a city of rigs sitting in the lapis blue Caspian, some linked by causeways that look like Scalextric sets.

Baku has a pretty medieval core surrounded by an old city and a new one. Old Baku is elegant, with sturdy turreted and cupola'd stone buildings. New Baku is brash, boom-built and crowned by three almost-finished skyscrapers shaped like flames, for Azer means "fire". In between, there are rather a lot of concrete apartment blocks, gifts from the eastern bloc.

Tom Ford has three shops here; a babushka sells tomatoes from a doorway next to the Dior store; boys too young to shave drive around in Lamborghinis. Baku intends to become the Dubai of the Caspian, and its permissive atmosphere makes it more swingy than its neighbours. But oil money means Baku is pricey, so I head south, towards the border with Iran.

Azerbaijanis have two driving styles: dawdle or hurtle. Samir, our driver, does both on the way to the seaside city of Lenkoran. We detour to Qobustan national park, where ancient man left cave graffiti clearly showing the longships that led Thor Heyerdahl to trace his Viking roots here.

More of Thor later. There are no road signs to the mud volcanoes, nor are they marked on the map. We eventually find them up the roughest of tracks, where they squelch, grepse and fart in a lunar, Clanger world.

Mud volcanoes in Qobustan.

Back on the coast road, recumbent, rusting oil rigs and glamorously ruined Soviet-era factories give way to wetlands where men on horseback herd flocks of buffalo. The Soviet legacy of overworked Ladas and churches with Iced Gem roofs is evident, and the sense is of a peaceful, fertile land, where you show your wealth in the artistry and gleam of your zinc roof and the gold in your teeth. Flashy smiles are an Azerbaijani speciality, as is an irresistibly gallant form of hospitality.

By the end of the day, we've seen miles of vineyards, and tremendous birdlife, and beautiful women in fuschia headscarves tending fields of greens. There are chaikhana (tea houses) every 500m, where the traditional offering of a pot of good tea and a bowl of jam is yours for a manat – less than a quid. Outside Baku, food is consistently delicious, and good value: Azerbaijani cuisine is a lovely mixture of Turkish, Iranian, Caucasian and Mongolian, characterised by impeccably fresh ingredients served in Brobdingnagian proportions.

Lenkoran is low-built and pretty, with Lada taxis a go-go, and a shoreline of charcoal sand. At Astara, close to the Iranian border, there are cloud forests, flower meadows, open-air roadside bread ovens, tea plantations and sheep sheltering in petrol stations. We wave across desultory barbed wire at bored border guards sheltering from the sub-tropical rain.

Azerbaijan was, in 1918, the world's first Islamic democracy. Though nominally Shia, the people have developed remarkable tolerance through millennia of invasions (Greeks, Romans, Arabs, Seljuks, Mongols, Arabs …) leavened with 120 years of Soviet rule. Muslims, Christians, Jews and Zoroastrians live in peace here, and all communities seem very partial to a drink. A blight on this tolerance is the Armenian occupation of the Nagorno-Karabakh region in the west. Everyone I speak to is hurt and angered by this, and confused at the UN's refusal to intervene, despite four resolutions demanding an immediate, unconditional Armenian withdrawal.

After Lenkoran we have to retrace our route to Baku, as Azerbaijan is cleaved by huge, unspanned gorges and mountain ranges. Another day, and we're on our way north to Gabala. One of many ancient khanates (an area traditionally ruled by a Khan), it is now home to Gabala FC, managed by Tony Adams, ex of Arsenal and Engerland.

As we climb out of dusty Baku, the landscape gets greener and more wooded. In this Xanadu, wild horses nibble and gallop free, and we graze on tea and spoonfuls of carnelian cherry jam. There's a river called A Gazelle Cannot Cross It, and isolated villages whose zinc roofs are a mix of Mongol and Moorish styles. A mountainside of butcher's shops features adjoining penned flocks of sheep, in the local before-and-after style. High above, an eagle eyes us. At Vandam (cue Jean-Claude jokes) we drive through a magical pistachio forest, and here's the ancient mountain city of Gabala, which is roughly the size of Stroud. Swish hotel complexes cater for Bakuvians who come for the alpine air and views, and in one of these live Tony and his assistant, Gary Stevens.

Gabala FC attracts crowds of up to 300, and lies seventh in a league of 12 teams. But Tony and Gary are upbeat optimism personified, and are enjoying the adventure and the anonymity. "They don't know me from Adam," says Tony, toasting Azerbaijan in sweet tea.

On to Shaki, ("Shaky" in football circles), while Samir talks music: Nazareth, Suzi Quatro and Shakatak are all Big in Baku.

In the charming hill village of Kish, the Albanian temple has been restored by the Norwegian government. Outside it, men are playing nerd, the local backgammon, and keeping score on an abacus, under a bust of Heyerdahl with the quotation: "Scandinavian mythology describes a god called Odin that came to northern Europe from a place called Azer. I have studied the writings and concluded that it is not mythology. It is real history and geography." By local legend a skeleton found underneath the temple showed these early Vikings to have been two metres tall, and blond with blue eyes.

One man and his wolf in Shaki.

The inhabitants of Shaki are short and dark, but wear very tall hats made of lambskin. Shaki is a glory: wooden and stone houses strung either side of a steep mountain river, a grand caravanserai-turned-scruffy hotel and a richly decorated khan's palace, outside of which a man with a stuffed wolf charges one manat to make the bulbs in its eyes flash weakly. Shaki is famous for its walnut and honey halva. A halva kitchen is called a sexi.

We breakfast the following day in Shaki's market on chewy bread, rich butter, runny honey, salty cheese and thick cream. Next stop will be Ganja, home to Nizami Ganjavi, a 12th-century poet who wrote the story Majnun and Layla, a precursor to Romeo and Juliet.

Outside Ganja is the village of Hash. I can report that Ganja is smoking, as it is wedding season. Shadliq sarayi (palaces of joy) are banqueting suites where people stuff themselves silly and dance artfully. The bride and groom, tightly primped, are enthroned upon a flower-decked plinth. Azerbaijani hospitality being generous as it is, I am invited that night to Two Weddings and a Circumcision.

Our last excursion is to Quba, up near the Dagestan border. It lies on the banks of the Qudryalçay river and is everything you'd imagine a beautiful, remote Caucasus town to be, and more. Its left bank is a grid of low-built villas hung with cherry blossom and wisteria. There's a fascinating carpet workshop where you can have your photo woven into a rug, and a forest restaurant serving the best lula (minced lamb) kebabs you will ever taste. Guba's right bank is even more remarkable.

This is Krasnaya Sloboda, or "red village", home to 3,500 Mountain Jews, whose forefathers came from Iran and Iraq a few thousand years ago, and who live today in perfect tranquillity. Rav Adam runs the yeshiva religious school and preaches tolerance and love to his skullcapped charges. In Soviet times one of the synagogues became a sock factory; now restored it commands a view of the wide river and the shiny mosque on the opposite bank.

A wedding party at Ganja.

Oil wealth means that Azerbaijan is developing fast, but outside Baku time is still a commodity to be spent freely on many small pleasures. Over many pots of tea, I muse that Azerbaijan isn't everyone's cuppa, but if you're an aficionado of post-industrial ruins, of amazing nature, of exotic headgear, of world history, of kind gentleness and wild strangeness, Azerbaijan will surely light your fire.

• This article was amended on 23 May 2011. The original contained a misspelling of Suzi Quatro. This has been corrected.