For many theatregoers a big night out on the town means some expensive seats for a hit feelgood musical. And London box office figures, just out, reveal that well over eight million tickets were sold last year to those in search of such all-singing, all-dancing live entertainment.

But how might the evening go if, rather than some high-kicking razzmatazz, the show on stage told instead of the political negotiations surrounding the demise of the charity Kids Company? For this is the unlikely subject of a musical already in production at the prestigious Donmar Warehouse in the West End. Or how might an audience feel after a performance of Holy Crap, now running across town at the King’s Head theatre pub in Islington, in which an ailing religious television channel decides to start making tawdry porn? And what would be the prevailing mood after a trip to the Print Room at the Coronet in Notting Hill to see Remnants, a musical version of a memoir about the aftermath of genocide in Bosnia? Perhaps there will be some light relief later in the year, with the opening of a major new musical, Heaven on Earth the Musical; the plot is the descent of mankind after Adam and Eve’s doomed adventures in paradise, billed as a story of “love, loss, and redemption, and an exploration of the universal search for the meaning of life”.

Inspiration for musical theatre has frequently come from left field but the number of productions drawing on increasingly bizarre real-life (and sometimes imagined) scenarios is burgeoning – and it is easy to see why.

On Broadway roughly 80% of theatrical revenue already comes from musicals, and the Society of London Theatres’ new box-office figures confirm that last year 57% of tickets sold were for musicals, bringing in £400,812,904. When such shows work, they are much more lucrative than a play, despite a heavy initial outlay.

This summer, clear signs of the strong public demand for new musicals has led to an extraordinary variety of source materials. In the last six months, plans have been announced for original productions based on Nick Hornby’s football book Fever Pitch, on the Nativity! film franchise and on the 1970s television sitcom The Liver Birds. The acclaimed American director Jay Scheib admires the way the fans of musical theatre are always seeking something fresh. “It is very different to the limited repertoire in the opera world, where I spend a good deal of my time,” said Scheib, who is at the helm of another big new show, the London Coliseum’s extravaganza Bat out of Hell, a fantasy romance created by Jim Steinman around Meat Loaf’s chart hits.

Perhaps the hardest sell of recent times was Urinetown: the Musical, a critically well-received 2001 satire set in a dystopia where the public right to go to the loo has become a commercial commodity. Recent hits such as Hamilton, which tells the story of an American founding father, or London Road, based on the community experience of serial killings in Ipswich, also prove that unpromising subject matter does not mean the audience is left feeling cheated of a good time.

Even when a musical is new, its plot or songs are often pre-loved. Twenty-five years after Fever Pitch was published, after two films and a play, the book is to get its first-ever musical staging this September courtesy of Highbury Opera theatre, at a venue close to the home of Arsenal football club, which is central to its plot.

A month later, the popular festive film series Nativity! will also open as a show at Birmingham Repertory theatre, before setting off on a short tour ending in London and Leeds. “I’ve always believed that Nativity! would make a fantastic stage musical,” Debbie Isitt, the creator and director of the films said. “It is so full of joy, the children are so sweet and funny and the songs so catchy that it lends itself to being the perfect Christmas musical.”

A scene from London Road, a musical about the community affected by the Ipswich serial killings. Photograph: Tristram Kenton for the Observer

Fans who remember the mordant wit of The Liver Birds, Carla Lane’s hit TV show, have already been treated to an early concert performance at Liverpool’s Epstein theatre last month to test out songs written for Liver Birds Flying Home.

“Audiences respond really well to a surprise, and so that can be a commonly known story told in a way that is unexpected. They may see something well known in a new light,” said Scheib, whose own show, has already extended its run due to ticket demand.

“If you are working with old music or songs, you still have to come to it as if it was written this morning and you have to defy expectations,” he said. Another production shortly to give some old songs a fresh workout is playwright Conor McPherson’s Girl from the North Country, which opens at London’s Old Vic and features a selection of Bob Dylan classics. Set in Minnesota in 1934, it chronicles life for an impoverished family running a guesthouse.

“Opera is conventionally more serious, but a musical can go as dark as opera too if it needs to,” said Scheib. “It is certainly possible to tackle difficult subjects, as long as you approach them honestly.”

Proof of Scheib’s argument is found in cult hits such as the 1996 American show Rent, which dealt with the impact of the Aids epidemic, or in Spring Awakening, based on Wedekind’s gloomy 1906 play. All the same, the musical genre is still widely associated with toe-tapping, cheery fare. Shows recycling warmhearted popular films, such as The Full Monty, Bend it Like Beckham or Made in Dagenham, are regarded as truer to type. Although sometimes, as in the case of Kinky Boots or the new Calendar Girls showThe Girls, a later musical version wins more plaudits than the original film.

But after the triumphs of Spamalot and Jerry Springer: The Musical, self-mockery is often a key ingredient. The Book of Mormon has satire at its core, while Tim Minchin’s Groundhog Day also involves parody of its own form.

“I hope there will always be room for satire, and especially these days,” said Scheib. “In any case, I don’t think dark, emotional subjects and satire are mutually exclusive. I am working on a romance now, but we also have some moments of humour in there.”

The time-honoured theatrical truth is that even a show which looks like a surefire success can fall on its face. From Lionel Bart’s notorious 1965 Robin Hood flop Twang!, to the Spice Girls show Viva Forever!, a jolly exclamation mark will not stop a bad show from failing.

Punters in search of a lighthearted evening of musical entertainment are probably safest heading for The Wind in the Willows, at the London Palladium. Or, if a really reassuring night is essential, wait for Big Hugs, the first stage show to star the Teletubbies.

FIVE UNLIKELY MUSICALS

The Public Administration and Constitutional Affairs Committee Takes Oral Evidence on Whitehall’s Relationship with Kids Company A new Donmar Warehouse show, based on a House of Commons hearing, about “how civil society holds itself to account”.

Heaven on Earth The story of Adam and Eve and the descent of man. A live arena spectacular written by Sarah Jeffs, touring the UK from December.

London Road The National Theatre’s 2011 hit show by Alecky Blythe and Adam Cork was based on the verbatim testimony of residents living around the site of a series of killings in Ipswich in 2006.

Urinetown A satirical 2001 Tony Award-winning show that began off-Broadway, set in a world where urinating is an expensive business. A London production directed by Jamie Lloyd opened in 2014.

The Fields of Ambrosia A black comedy based on the 1970 film The Travelling Executioner, set deep in the rural American south. It opened in 1993 in America and came to London three years later. Praised for its tunes, its subject matter was hard to market.