Before the church bells, before the buskers, and shortly before the smell of kippers, there is the sound of Keith Gilpin’s dust cart trundling over the Church Street cobbles, St George’s flags flying proudly. This is how a Whitby morning begins.

“Look at that,” says the 62-year-old street cleaner, sweeping a high-vis arm towards the sea view. “Tell me what’s better than this, lad? I like a beer and I like a bet, but you can’t beat a sunrise. Of all life’s pleasures, often the free ones are the best.”

Keith has a point. The old fishing port of Whitby, on the North Yorkshire coast, has some claim to being the most picturesque town in England. The eye is caught first by the ruined abbey high on the east cliff, then zigzags down to the waterside through a maze of higgledy-piggledy cottages with red roofs. The curved arms of the twin piers draw the gaze out to sea, their shape mirrored by the whale’s jawbone arch that crowns the cliff to the west.

That view alone is reason enough to come, but now there is another: Whitby has planted a flag on Britain’s foodie map. Andrew Pern, well known for his Michelin-starred gastropub in Harome, claims that the town is on its way to becoming “the Padstow of the north”, to which end he has opened The Star Inn The Harbour. “Whitby is famous for the sticks of rock, the kiss-me-quick, the fish and chips, but there’s a lot more that the North Sea has to offer,” the chef says.

Battered ‘woof’ and chips from The Star Inn The Harbour. Photograph: Gary Calton for the Observer

His restaurant, in the former tourist information building on the harbour, has a nautical look – ropes, creels, framed postcards – but it’s tastefully done, not the full Captain Haddock. Starters include woof soup: “Scarborough woof” is a local name for the Atlantic wolffish. Mains include Whitby crab salad with seashore vegetables and avocado ice-cream. Those two dishes are £7 and £16 respectively, which is dear for Whitby; indeed, a glance at the menu prompts my father-in-law, whose Yorkshireness begins at his wallet, to ask his favourite rhetorical question: “’Ow much?”

If you want fish and chips, the Star Inn offers that, too, although devotees of the Magpie, the town’s most celebrated chippy, will be pleased to learn that it has opened again for takeaways following a fire that destroyed its roof.

On the other side of the harbour, across the famous swing bridge, is Whitby’s other cult food stop. Fortune’s kipper shop is on Henrietta Street, as it has been since 1872, a child’s drawing of a building with barely a straight line in it. Old photos on the walls depict the six generations of one family who have worked here. The tiny smokehouse next door, where the herring hang for 18 hours, is one of the town’s sacred marvels, the walls glossy black and dripping with tar built up over decades. It is like being inside a pirate’s lung or a Mark Rothko painting. They scrape the tar out, it is reckoned, about once a century. No need to rush these things.

Like other seasonal economies, Whitby relies on eastern European immigration – most notably, the arrival from Romania in 1897 of one Count Dracula. Bram Stoker, inspired by a visit, set part of his novel here, and this association with the Transylvanian count is now a significant tourist draw, in particular among the masses who flock here for Whitby Goth Weekend in April and October.

How important is the so-called “velvet pound” to local businesses? “Massive,” says Kev Riley, who runs the Bats & Broomsticks guesthouse (doubles from £75), where guests breakfast by candlelight and enjoy the music of the Damned with their cornflakes. “We cater,” says Kev, “for people who want something different.”

Whitby has a deep geological strangeness – as if some fault occurred around the time ammonites became fossilised in the seashore, and the town has been weird ever since. By all means take your ferret for a walk along the beach; feel free to dress as a vampire while shopping in the Co-op. No one will bat an eye; no one will eye your bat.

Back on the west side, near the Magpie, is a small cabin covered in cheery writing inviting passers-by to learn their future from the “Spiritualist & Clairvoyant” within. The cabin was inhabited for almost half a century by Lee Ester Alita Lee, mother of the present occupant, Elizabeth Connie Smith. “Last year was me mam’s last season. She has dementia and started to drift, so this tradition was handed down to me. I’ve had the gift ever since I was born.”

Smith is 54. She offers palm and crystal ball readings in a cosy consulting room decorated with signed photos of clients past: Chas and Dave, Big Daddy, Cher, the cast of Heartbeat. It is all part of the seaside experience.

What, then, does she foresee for the town itself? “For Whitby?” she considers with a smile. “I think the future’s looking very good indeed.”