Charities face a "barrage of attacks on their freedom to defend the most vulnerable", which have grown more serious in each of the last four years, the Panel on the Independence of the Voluntary Sector has said in its final report.