Colin Firth makes his way down an opulent and very long room, its ceiling elaborately studded with gold mouldings, its panelled walls oppressively gilt-sprigged, garlanded, swagged and punctuated by Louis XIV-style cherubs. Firth is playing George VI on his way to make his first wartime broadcast and the room is in Buckingham Palace – except it isn't, of course; it's actually in Lancaster House, tucked away between the Mall and St James's.

Now used by the Foreign Office for diplomatic conferences and assemblies, Lancaster House may actually be the grander of the two: when Queen Victoria once visited, she told its owner: "I have come from my house to your palace." The overblown state drawing room, with its exuberant gilt trellises and traceries, suits the logic of a film that deals with the pressures facing a man who is intensely shy and hampered by a terrible stammer yet reluctantly becomes a public figure.

The King's Speech shifts between grand royal residences and depression-scuffed 1930s London, with much of the action taking place in the consulting room of Australian speech therapist Lionel Logue (Geoffrey Rush), a maverick who works in a plain, scruffy space with dark wood and not much furniture. The big question of the film is how, by visiting this room, Firth's character will overcome his stiffness, uncover the private man beneath the protocol and prove that, unlike his elder brother, off gallivanting with Mrs Simpson, he is in touch with grimy 1930s Britain.

The production team were determined not to make a prettified costume drama. "When we were financing the film, there was a belief that this kind of movie had become unpopular, that audiences wouldn't connect with it," says Iain Canning, the producer. "Everybody felt it was important to bring back those aspects of cinema that Britain does very well – historical drama with rich production design – so we were keen to avoid the cliches of period films; we wanted London to be authentic, not look like some strange postcard from the past – clean, with blue skies."

They looked at filming in Glasgow, Liverpool and Dublin, but eventually decided to shoot in London in the low light of last December and January. Director Tom Hooper has recreated London in Lithuania, South Africa, the US and Hungary, but never previously here in his home town, let alone in Regent's Park, which he walks through on his way to the cutting room. The team anticipated that there would be enough peasoupers to hide inconveniently modern lampposts, and that by using the real Harley Street and a tenement building in Kennington for Logue's home, the shots would have more depth and resonance than is possible with CGI.

Hooper wanted a "smoggy, grungy look", according to Amy Merry, who worked on the production design. "When we were shooting exteriors we threw dirty water over everything. We filmed in Harley Street on a Sunday so we closed the road in the early hours and a gritting van came along at 5am and covered the ground with dirt. Then we pumped out so much smog that we set off the fire alarms in John Lewis."

The actual rooms Logue practised in were too small to film so the team found a building a block away, 33 Portland Place, which has an unusual vaulted room with large leaded windows at one end, reminiscent of a Venetian palace, and roof lights that make it look a bit like an artist's studio and allow some light in on the pervasive gloom.

This room also has extraordinary walls, decorated in distressed browns and oranges like an exotic damaged fresco, if in autumnal colours. When the team arrived, one area was covered in wallpaper infused with oil, which had then been half-scraped, half-burned off. Eve Stewart, the production designer, loved the mottled, peeling effect so much that she decided to reproduce it across the whole wall. The room became so striking that very little furniture was required, and the almost empty set gives the actors space, in Rush's case to make Logue theatrical and expansive; in Firth's, to allow his character, who in the manner of royals in those days has far too many names – Bertie, the Duke of York, King George VI – to emerge from behind his carapace.

Stewart, who previously recreated 1950s London in Vera Drake, researched for a couple of months at the Victoria and Albert Museum and the Geffrye Museum of interior design, as well as in photographic libraries and by reading Logue's diaries. She describes the consulting room as a method set, the few props all there for a reason: model aeroplanes to signal the warmth of his life with his sons; a photograph of himself playing Othello in an amateur production (he was a failed actor); some games he might have used to teach children with speech impediments, which Amy Merry researched at the Institute for the Deaf. "You'd bring a prop to Tom," Merry says, "and he'd always want to know if it was true to life."

Before Edward VIII's abdication, the Duke and Duchess of York lived at 145 Piccadilly, the location for which was (surprisingly, when you see the film) also 33 Portland Place. In addition to the wood-panelled extension, the house has a number of elegant eau-de-nil rooms designed by Robert Adam. Firth and Helena Bonham Carter didn't even have to go outside to cross the fictional barrier between stuffy royal residences and ordinary London.

The website of 33 Portland Place describes the house as "shabby chic". In the film, it looks elegant although not particularly luxurious and certainly not very comfortable. (The family seem to spend a lot of time on the landing.) "It would have been tempting to make everything look lush, but it was the Depression and no one did things up," Stewart says. "The royals also had a kind of disregard for decor. We got hold of the diaries of the princesses' nanny and she said 145 Piccadilly was a horrible dusty building, draughty and not very well heated."

The King's Speech opens, dramatically enough, at Wembley Stadium, a sequence that was shot at Leeds United's Elland Road ground and the Grattan Stadium in Bradford. Hooper had directed The Damned United at Elland Road, so knew it was one of the few places that could stand in for the old Wembley, where, in 1925, the Duke's stammer first came to public attention when he gave a halting, hopeless, humiliating Empire Day speech.

The crew had to wait until 10pm to get into the stadium, following a match; overnight, the stands were filled with an inflatable crowd. According to Amy Merry, these blow-up people – actually only blow-up upper bodies – are much more convincing than CGI. "They look very funny when they arrive. They came in period costume because they'd already been used for The Changeling. But once they're put in the seats and inflated, they look great."

The biggest challenge for the production team, Stewart says, was "no money and lots of sets". At less than £10m, the budget was tight; Lancaster House alone cost £20,000 for a day and supplied only two scenes – the walk to make the broadcast and the official photograph afterwards. The speech itself was shot off-site, "pretty much in a cupboard".

The production also used Battersea power station, which did duty as the BBC wireless control room, using machinery that was already there, supplemented with dials moulded from flan cases. Logue's home was an atmospherically beautiful tenement in Kennington, south London, where, again, there was a lot of dirtying-down and gritting over road markings. "We destroyed the window frames – only with paint, of course, but a lot of the residents were alarmed," Merry says. Locals wanted to keep the Bovril ad the team painted on a wall, but it had to be removed with a water cannon. It would have peeled off in the rain eventually anyway.

Despite the royal settings, The King's Speech is a remarkably brown film. The palaces are intimidatingly, rather than comfortably, luxurious. The production design plays into the sense of Firth's character struggling to be a king, which, for him, means struggling to be himself.

"There was a real commitment on the part of the whole team to put the money on screen," Canning says. "We knew if we could make the setting authentic, paradoxically that would make the story timeless. We wanted nothing out of place to jolt you away, nothing to get in the way of the emotions."