The chief executive of Oxford city council, Peter Sloman, has hit out at the government's planned changes to the welfare system claiming the reforms – including the bedroom tax and benefit caps – have taken £34m out of Oxford's local economy.

Speaking at the Guardian's fifth Local Government Leaders Quarterly event in London last week, Sloman said: "As an officer you are not supposed to be political at all so this isn't a party political point but it seems to me taking £34m from the poorest people in our community is a bit of a sick exercise."

He also claimed that "the amount of vindictiveness towards these people [on benefits] you see in the media from government ministers in terms of painting them as those who don't want to get out of bed in the morning or leaving the curtains closed, is not helpful in terms of dealing with the problems these families face."

Sloman warned that in Oxford there were 166 families affected by the cap and 956 living in under occupied dwellings that will be hit by the bedroom tax. Between those families there are 1,200 children who will be seeing their family income drop.

However, Sloman said welfare reform also provided an opportunity to take local action to help move residents into work.

"We must ready ourselves – there is a lot we can do. We are launching a scheme to help people deal with putting lodgers into their house. I think it is awful really, but we are doing it because that is the solution for lots of people. We've also moved about 300 people extra on our mutual exchange register."

Other speakers at the event, which discussed how councils should manage the impact of welfare reform, included Katie Shaw, head of welfare policy at Citizens Advice; Dave Simmonds, chief executive of Inclusion, an organisation that promotes social inclusion in the labour market; and, Liz Goodall, chief executive of North Dorset district council.

Goodall echoed Sloman's advice, arguing that councils must step in and act on welfare reforms. She said practical work must be done or councils would be "swamped in crisis".

Simmonds shared his fears with the room, stating: "We still cannot double guess these consequences but we have to do our best. We have to be prepared for those who fall through the net."

But Shaw, who leads on welfare issues for Citizens Advice, was concerned about the lack of government analysis. "There has been lots of impact assessment on each individual change but not on the cumulative impact of the reforms taken together. We know it will be the same people hit by each of these cuts many times in a lot of these cases," she said.