The only Conservative MP in Scotland is relieved the most-repeated joke in Scottish politics – "There are more pandas in Scotland than Tory MPs" – no longer just applies to his party.

Following the SNP landslide in Scotland, there is only one Conservative, one Labour, and one Liberal Democrat MP in Scotland, whereas there are two pandas in Edinburgh Zoo.

Scottish secretary David Mundell – MP for Dumfriesshire, Clydesdale and Tweeddale – now hopes the responsibility of being the butt of the panda joke, which was coined in 2010 about him and has been repeated ever since, will now be shared around.

"What never ceased to amaze me were the people who thought I'd never heard that joke before," the Scottish secretary told BuzzFeed News. "People always came up to me thinking they had some fresh or novel insight to give me, and I was able to tell them that I'd heard it many, many times before."

However, Mundell isn't convinced the joke is about to go away: "Had there been more than one Scottish Conservative MPs after the election, I was quite sure the SNP would have gone out and rented more pandas."

Mundell said he has adjusted well to the new SNP-dominated group of Scottish MPs, as nothing much has changed for him. "For 10 years now, it's been me and 58 other Scottish MPs, that's what it is still – I'm quite used to it," he said.

Dominic Lipinski / PA Wire/Press Association Images

But it will be a new experience for the Lib Dems' Alistair Carmichael and Labour's Ian Murray. Mundell assured them that that they will not find being a sole Scottish MP of their parties too lonely an experience, although they might find it quite tiring.

"People used to ask if I felt lonely, but I never felt lonely," said Mundell. "All of us have colleagues [in parliament], that's very important – Alistair only has seven, Ian has 200 odd, and if they're like my party then they'll be supportive.

"To be honest, the one thing I would most like a colleague for is late night television," he continued. "They will now have what I had, they're the only MPs who can appear for their parties and that can be difficult... You can't do all the things that people will be demanding of you."

Mundell was not surprised by what he called the "SNP tsunami" earlier this month, and realised before the end of last year that he had a fight on his hands to win his seat. In the end, he won by less than 800 votes.

He believes he managed to hold on due a combination of energetic campaigning and tactical voting to keep the SNP out, and he will use his experience to help his Scottish Conservative colleagues attempt to fend off the SNP in next year's Holyrood election.

Mundell also said the rise of the SNP might help the Conservatives recover north of the border – where they have been less popular than Labour for decades – as they could emerge as the main unionist party.

"Certainly, I think the fact that the area I represent is one of the strongly unionist part of Scotland helped – a lot of people did not want to wake up to having an SNP MP who didn't reflect their views on the future of Scotland," he said.

With only three unionists out of Scotland's 59 MPs, there is a risk of people who believe in the union being underrepresented. But Mundell points out that "the vast majority" of MPs in Westminster are unionists, and will argue the cause for the 55% of Scots who voted No in last year's referendum.

"The case for the United Kingdom is not simply to be made by three MPs from Scotland – it's to be made by a whole range of MPs from across the United Kingdom from a range of parties," he said.

"I acknowledge that the SNP were underrepresented compared to their support in Scotland in the previous parliament. This time I would argue they're overrepresented, but that's the nature of politics."

After the excitement of the general election, and now the induction process over, the real work of being an MP will begin and Mundell suspects the new SNP MPs, and new MPs throughout the house, might be in for a shock.

"All these new people will get the really difficult constituents now, who have long-running cases which have been going on for 25 years," he said.

"The first wave of constituents you get are always the most difficult and challenging."