Iain Duncan Smith, the work and pensions secretary, was called a "ratbag" as he was heckled by anti-cuts protesters who gatecrashed a speech in Edinburgh on Wednesday.

Smith was about to deliver a speech on pensions reform when campaigner Willie Black began haranguing the MP, calling him a "parasite" and a "ratbag" for pursuing social security cuts that would leave "millions" of people homeless.

After Black, who had booked in for an overnight stay at the George hotel to get into the event hosted by Capita, was escorted from the room, two disability rights campaigners also barracked Smith. Protesters also gathered outside the hotel.

Referring to the sole Tory MP in Scotland, Black, from the North Edinburgh Fights Back campaign, told Smith there were more pandas in the country than Tories (there are 15 Tory MSPs in the Scottish parliament).

As Smith prepared to address his audience, Black shouted: "Why are you here in Scotland? We have a different philosophy. We've not elected you and your Tory cohorts. We don't want you; we don't need you. We've more pandas than you. You've only got one MP and we want a different Scotland that cares for people who are distressed and in poverty.

"You're going to make millions of people homeless – people that definitely need help. You're making the richer, richer and the poor poorer," Black said.

Claiming the bedroom tax would be the new poll tax in Scotland, he added: "And we're going to see the end of you. Back to England where you belong, you ratbag."

The work and pensions secretary, whose policies on welfare have fuelled centre left and Scottish National party campaigning for independence, also held private meetings with the Scottish Lib Dem leader Willie Rennie and Nicola Sturgeon, the deputy first minister. He then held a private session with a welfare reform committee at the Scottish parliament.

Before being escorted from the room, the two disability campaigners, Jonathan Smith and Charli Saben Fox, sitting at another table with a guide dog, shouted: "We want social justice and equality … We're coming for you."

Jonathan Smith, who receives income support, disability living allowance and housing benefit, told the BBC: "If we don't do it then we may as well just go out and shoot ourselves. A bullet in the head is probably easier, kinder to most of us than just letting us rot away."

Charli Saben Fox, who is disabled and has a son with learning difficulties, said: "If they take away everything from us we have nothing to lose … Iain Duncan Smith has got a fight on his hands if he thinks we are just going to lay down and die."

Meanwhile, the Scottish government has rejected calls from the Scottish Labour for emergency legislation to stop councils and housing associations from evicting tenants who run up arrears due to the bedroom tax, which comes into force across the UK next week.

Margaret Burgess, the Scottish housing minister, said that would allow tenants to run up debts and put already overstretched housing providers under greater financial pressure.

Her government could not afford the £65m needed to meet the cost of the cuts. SNP councils, however, are expected by ministers to observe a party policy not to evict council tenants who run up arrears.