It is wrong to define the argument over the bill as one between security and liberty.

Home Secretary Theresa May warned MPs who oppose the bill: "Do not put politics before people’s lives." Photograph: Getty Images.

For almost a decade, the intelligence and policing community has been worried that changes in modern communications – mainly the shift from telephones to more varied devices and networks – would make their work more difficult. And each government of the day has tried (and failed) to push through legislation to update capabilities in the face of staunch opposition.

The latest attempt is the draft Communications Data Bill. To recap, the Bill aims to improve government access to internet "communications data" (who, when, and where you communication with someone, but not what you say to them). Communications data are vital for law enforcement, intelligence work and the Crown Prosecution Service. When these agencies needed this information, they used to go to the small number of telecommunications companies who collected it. Now of course we communicate via e-mail, social media messaging, phone apps, game platforms and more - so now communications data are being generated by all sorts of companies who don’t routinely collect or save it, and so it's not always there when needed. The Bill is asking/demanding/paying relevant companies to collect and retain it, so that such data is available on request.

Such was the furore when it was announced back in April that a pre-legislative joint committee was scrambled to calm the maelstrom and review the proposals. Today it published its report[3]. Anyone familiar with the Bill would not have been surprised by its findings. To sum it up as briefly as possible: yes, the government needs to improve access to communications data, but the Home Office did not make the case well enough, nor consult widely enough. The powers contained in the Bill are too broad in scope - and greater oversight is needed. Go back and re-draft it. The committee has been extremely diligent (and I hope David Davis MP will apologise to the chair, Lord Blencathra, whom he had little faith in). But I think there are three big lessons from this whole affair.

First, the bitter rancour surrounding this Bill is caused by a fundamental problem. State surveillance needs to be proportionate and necessary, but these words are losing all meaning, because no one really understands the technology and the possible risks and benefits that come with it as internet-enabled devices become ever more ubiquitous, least of all our law makers. When critics protest the measures will not work, or 'we're the only country that does this', the Home Office are unable to respond with technical details for obvious reasons. And the government wants to make the legislation broad – ‘future proofed’ - because the technology will have moved on again soon, and they don’t want to go through this again in two years. This Civil liberty groups, understandably, weren’t too pleased about that. This is going to get worse in future.

Second: terrorism and paedophiles do not automatically trump digital rights. The Home Secretary, Theresa May, has consistently argued the new Bill is essential to tackle terrorism, serious crime and paedophilia. She may have believed that any law strengthening powers to do that would be more or less accepted. In that, I believe she may have underestimated how important digital freedoms and data sovereignty are to people today. Surveys consistently show that data and privacy are among the top concerns citizens have. The online and privacy community – often tech savvy, networked, and highly defensive of internet freedom – are a powerful lobby group.

Third, any legislation about security powers tends to degenerate quickly into a debate about terrorists versus an Orwellian dystopia. Last Monday, in an interview with the Sun[4], the Home Secretary said that anyone against the draft Bill is putting politics ahead of people’s lives; and came close to saying anyone opposed to the Bill is taking the side of those criminals, terrorists and paedophiles. It was an unfortunate and inaccurate charge levelled against the Bill’s many thoughtful and principled opponents. But the Bill’s opponents have also been alarmist by (inaccurately in my view) calling it a "snoopers' charter" and claiming it represents mass internet surveillance. Suggestions that this Bill would put us alongside China, Iran and Khazakstan are wholly inaccurate. When giving evidence to the committee, the Observer journalist Henry Porter (who also called the Bill the "megalomaniac dream" of a senior civil servant) said that these measures could become "the structure for a police state" – something the Crown Prosecution Service and others that have actually used communications data dismissed.

Ignore the papers this morning: the committee’s response is a sober and careful one. At core, it recognises that the philosophical pros and cons of the Bill are not really about security versus liberty, but the more nuanced debate of pro-active data collection (collecting and retaining data so it’s there if you need it) against a more limited one (using what you have), and whether we need to ‘future proof’ law of this kind (it thinks not). It realises that modern communication has changed dramatically, and law enforcement must keep up, including in designing a regulatory system that reflects the changing concerns people have about privacy. It recognises this is not easy and will ultimately fall to Parliament.

This means going back and re-drafting a Bill that trades a bit of operational secrecy for clarity about when and where the measure will be used. Above all, the Home Office should consult more widely, and then set about some improved drafting to eliminate worrying ambiguities, give a tighter clarity of purpose, and include tougher scrutiny and oversight. A revised Bill could keep both sides content. This is all less interesting than terrorists and Orwell, of course, but then making law usually is.

Jamie Bartlett is the head of the Violence and Extremism Programme and the Centre for the Analysis of Social Media at Demos.