Tahir Shah at the entrance of the library in his house, Dar Khalifa. Photograph: Zacarias Garcia

Close your eyes and think of Casablanca, and your mind most probably comes alive with images of Bogart, Bergman, and Sam tickling the ivories in the smoke-filled Rick's Café. And there's nothing wrong with that. Except that Casablanca the movie has almost nothing to do with Casablanca the city. Shot almost exclusively in Hollywood, the wartime film portrays Casa (pronounced "Caza" by locals) as a cosmopolitan colonial crossroads of the exotic east. It features just one (uncredited) local character, Abdul the doorman. It's a fabulous example of Hollywood not letting the facts get in the way of a good story line.

Six years ago I dragged my wife and two small children from our cramped flat in the East End of London to live in Casablanca. We bought a rambling mansion with five courtyards, gardens and a pool in the middle of the sprawling Sidi Ghanem shantytown. It is quite the most magical spot but the learning curve has been a steep one, especially when we learned that the house was said to be infested from the floor to the rafters with wicked spirits, known as jinn. After lengthy exorcisms and endless renovations, we set about getting to know the city that had become our new home.

Casablanca seems to bear the brunt of every Moroccan joke, while being given a wide berth by most tourists. But spend a little time here and you begin to see that those who scorn it are missing something very magical.

In recent years the international jet-set have discovered Marrakech and a handful of other Moroccan cities. They wax lyrical about the "real" Morocco they've found in the narrow streets of labyrinthine medinas. Yet none of them ever mentions Casablanca, except to relate how they escaped it as quickly as they could. And that's the first thing that appeals to me about Casa – the absolute lack of tourists.

Eat in one of its many restaurants and the food is consistently good – because they rely on repeat business, rather than on tourists they will never see again. The restaurants, cafes and nightspots are full of Moroccans, not tourists, and thus far more atmospheric.

When the French took control of the small Portuguese-built port of Casablanca a century ago, they set about transforming it into a showcase of their colonial might. From the outset, they conceived it as a pleasure dome of art-deco architecture and European culture, the kind of city in which local Moroccans would be reminded of the imperial motherland. Luring the greatest architects from France with the prospect of building a city from scratch, the French administrators had Casablanca designed from the air, the first city in history to be laid out by aeroplane.

Gradually, art-deco Casa arose, its shaded arcades and lavish palm-lined avenues abounding with shops purveying the latest in Paris chic. These days, downtown Casablanca is still a jewel, albeit one hidden in a layer of disorder, peeling paint and grime. But, for me, that's part of the attraction. Traipse the main drag, Boulevard Mohamed V, and look at the details. They're everywhere: in the wrought ironwork, the old-fashioned signage, and in the arabesque twirls adorning the buildings. From the moment you arrive you know that this is the work of master craftsmen.

Halfway down the first stretch of the main avenue, set off to the right, is the fabulous Cinema Rialto, a newly restored art-deco gem that would have once screened Hollywood's Casablanca to a bemused audience.

Around the corner is Le Petit Poucet, one of the city's most historic dining haunts, where the likes of Albert Camus, Edith Piaf, and Antoine de Saint-Exupéry would while away the afternoons. Just across the street is another retreat with the same lived-in feel, the Marché Central, built in the Franco-Moorish style for which Casa is now famous. At lunchtime its cluster of cafes serve grilled fish fresh from Casablanca's fishing port, a stone's throw away.

At first light the fleet of little trawlers come in, laden with the Atlantic catch. The place buzzes with activity, and with cats, damp nets dragged ashore for mending and crates of seafood packed in gleaming ice. A few metres from the quayside is the city's battered old sea wall. Behind it lies the medina, the part of Casa that predates the French protectorate.

Strolling through it is something that few Moroccans, let alone tourists, ever do. In many ways it's a closed quarter, a place you venture to only if you have reason to do so. Inside the walls is a hotchpotch of architecture – some of it Portuguese, some Moorish – most of it untouched in the past century and a half. Played out within the quartier is a wonderful routine, a melody of life.

There are carts laden with seasonal vegetables and fruit, children playing with marbles and homemade hoops, and old women sitting in doorways fanning themselves in the afternoon heat. There are door-to-door knife sharpeners, tailors, water-sellers, goats grazing on scraps of old newspaper, and stalls brewing tea infused with fragrant mint. When I was in the medina last week I met an ancient Saharan man who was selling magical potions from a battered ebony box. Before I had a chance to open my mouth, he opened the box and pulled out a fibrous strand of root. "Viagra from the desert," he said, grinning. "Chew it before you go to bed and in the morning your wife will be smiling very hard."

To really understand Casablanca, you have to know the medina, the spot where the seed of the city fell long ago. The best entry point is the Marrakech gate, with its imposing clock tower. There are no reliable maps but, with a little twisting and turning, you may find your way to the old Portuguese fortress that backs onto the old city.

Marked by great iron cannons pointing out to sea, the bastion is known as La Sqala. It's a restaurant now, offering exceptional cuisine from all corners of Morocco. Famous for its salads and fish tagines, La Sqala is the exception to the rule that says Moroccan food is best eaten in the home. Beyond it is the huge and almost finished Casablanca Marina. And, less than a mile up the coast, is the equally enormous great Mosque of Hassan II. It's the third-largest mosque in the world, built on a jaw-dropping scale, and open to non-Muslim visitors (on guided tours).

A little further up the coast road is a French-style brasserie, Le Relais de Paris, a favourite with well-heeled locals and celebrated for its seafood.

Beyond Le Relais is the Corniche, a stretch of nightclubs, restaurants and cafes that back onto the endless Casablanca sands. In the summer it's awash with local people basking in the beach clubs or taking it easy in the shade, sipping orange juice and the ubiquitous café noir. And, once the day's heat has waned, it's packed with families taking a stroll before dinner.

All regions of Morocco are represented in Casa. You can find people from the farthest reaches of the Sahara, from the Mediterranean coast, and from the High Atlas mountains. And where you find them, you find their cuisine and their customs.

At the same time, there's an intoxicating oddness about the city, fragments of life that many Casablancans hardly know exist. For sheer incongruity my favourite haunt is the nightly greyhound races, held at the old French velodrome. It's a gambling den par excellence, patronised by an underworld of Moroccan men dressed in dark woollen djellabas, all of them desperate to strike it big.

Then there is the idyllic quarter of Habbous, with its shady Moorish arches and cool stone lanes. It looks centuries old, but was built during the French occupation as a showcase of Moroccan harmony – as seen through European eyes.

Another of my off-beat preoccupations is Casablanca's junk yards, flea markets and antique shops, well-stocked with loot left by the French. I've lost count of the number of grand pianos I have seen for sale cheap, or hulk-like mahogany sideboards, cast-iron safes, and immense crystal chandeliers. And it's not just unwieldy antiques that are on offer; there are cut-crystal champagne glasses, ice buckets and silver cocktail shakers from the 20s, old valve radio sets and two-piece telephones, tinplate toys and accordions, samovars and espresso machines.

One of the least-celebrated flea markets, known as "Soco de Moina", is in the middle-class neighbourhood of Hay Hassani, inland from the coast. I have furnished my home from treasures discovered there. Best of all are the 30s roll-top baths and washbasins, as wide as cattle troughs, ripped from old villas. In Casablanca anyone with real money shuns architectural salvage.

On weekends, a popular place to hang out is on the beach near the shrine of Sidi Abdel Rahman. Families sit on the sand slurping snails, which they cook in great copper pots, or searching for crabs in the rock pools at low tide. And those troubled by personal difficulties venture up onto the shrine itself.

At low tide you clamber over the rocks, while at high tide you either wade or get pulled across in a tractor's inner tube. Clustered around the shrine are a number of little rooms in which healers and mendicants live. For a few coins, some will tell your fortune or, more importantly, help to rid you of jinn. To determine your situation, they take a scrap of lead foil, melt it in a crucible, and then pour it into a pot of cold water. Then, rather like someone reading tea leaves, they remove the contorted form of metal and inspect it for a sign.

In the six years I have lived here, I've found that Casablanca is like a great knotted ball of knitting yarn. Given time, the city unravels itself, revealing the most amazing corners, a vibrant kaleidoscope of Moroccan life found nowhere else.

Living in our shantytown, the thing that struck me from the start was that no one does any recycling at all. The place is awash with plastic bags, bottles, glass jars, tin cans, and all sorts of other very useful waste, none of which is ever reused. Desperate to encourage recycling in Morocco, and to show ordinary people how to create livelihoods from garbage, we started a project aimed at transforming the waste into products that can be sold and used in the community. For example, with a simple domestic iron, plastic bags are being fused together to make plastic sheeting, and glass bottles are being cut down to make drinking glasses.

As a way of funding this fledgling project, we have recently started inviting paying guests to our home, Dar Khalifa, the Caliph's House. It is palatial compared with many of the dwellings around us, but a stay at the house still gives an unusual view of a Casablanca shantytown, from the inside out.

For the city's detractors, who claim that Casablanca is buried and lost, there is big change afoot. Under the King of Morocco's supervision, the city is about to undergo a major cultural and architectural renaissance.

For a few scorching nights each summer, the city erupts in sound, making it a musical crossroads between east and west. On the playbill there's gnawa percussion from the depths of Sahara, Brazilian samba, Berber folk, rock, rap, and even Moroccan punk. The Casablanca music festival is one of several new projects aimed at waking the city from its slumber. And a series of grand constructions are expected not only to transform the urban skyline, but to reinvigorate the entire city.

These projects include Anfa Place, an imposing residential and commercial complex being built by Sir Norman Foster; the Morocco Mall, Africa's biggest shopping centre; and the Casablanca Marina, billed as a celebration of the city's sea-faring heritage. Yet another project is the transformation of the old airport – not far from the city centre – into a communal park.

As someone living a love affair with Casablanca, I sometimes find myself shuddering at the thought of all the visitors the change will bring. My anxiety is inspired by selfishness of course, because I'd rather keep things how they are now, and have it all to myself.

•Royal Air Maroc (020-7307 5800) flies to Casablanca from Heathrow and Gatwick from £235 rtn inc taxes. There are two suites at the Caliph's House costing €200 a night for two, including breakfast and dinner (children under-12s stay free). Alternatively, the Barcelo Hotel has doubles from €76.

Tahir Shah is author of The Caliph's House: A Year in Casablanca (Bantam Books, £8.99)