Ministers are keen to see much more of the government's data being used to drive innovation and improve people's lives. To achieve that, Bill McCluggage tells Matthew D'Arcy, government must adopt a strategic approach to open data instead of just pushing it out – and hoping for the best

Opening up government gives millions of people the "opportunity to change their lives for the better", says David Cameron.

Government talks big. It wants open data to bring about changes in how public services are delivered. And it wants businesses to use data to generate billions of pounds. But has government done enough of the right things to make sure these opportunities can be realised?

Open data is raw material, ministers like Francis Maude keep repeating. It is a substance that can be used to drive forward transparency and to innovate.

Traditional raw materials like coal and oil have all fuelled the beginnings of

innovation and even industrial revolution. But they first had to become accessible. Then they were harnessed. Someone had to burn the coal to make it useful. Pharmaceuticals and much more came from oil – but only after innovators acted.

Signs of the harnessing of open data have started to appear. We see applications for transport systems, for flood warnings, for finding out about your GP online and even for checking crime levels.

But despite the potential, fears are emerging that vast amounts of government data isn't being used.

Bill McCluggage, who worked at the top of Whitehall as the government's deputy chief information officer for years under both Labour and the coalition, tells Public Servant that while open data is "fantastic", more must be done by government to help companies use it.

"They have bunged it across the other side of the fence and hoped somebody would make use of it to benefit society," he says. "We don't seem to have had any strategic planning about how government will consume. It is a leap of faith – push it over and hopefully somebody will do something novel with it."

A market failure has happened in the open data environment, he says, but the Open Data Institute will go some way to address this. Until then companies are being left ignorant of what demand exists – knowledge they need before they can afford to commit resources to using the government's data.

"At the moment I don't see the demand side stimulating and asking for specific products to help them from the open data that's been pushed out," he says.

Prominent transparency figures, including Tim Kelsey who is now driving a data revolution in the NHS, warn that there aren't enough stories about how open data is being applied.

McCluggage, currently chief technologist for the public sector at EMC, agrees. "There is a real risk that we can all see the principles but the stories don't follow quickly enough.

"Those stories aren't just to persuade government to do stuff, but also to explain what the business model is." Without this understanding, he adds, businesses are effectively being asked to use the data "philanthropically".

Government is even being accused of competing with the very same businesses it wants to innovate. Websites like Police.uk currently rival private sector alternatives.

"If government comes along and develops an app, it's not really competition, it's not even a cartel, it's a sole consumer marketplace," says McCluggage. "You have got to give the market a breathing space. Two years, three years – a non-compete clause."

Chris Yiu, head of Policy Exchange's digital government unit, agrees. "How can you compete with a product that comes with a departmental badge stuck on it, and is delivered for free?" he asks. Rules of engagement are needed to stop departments "building apps on a whim and crushing innovation". These rules must also prevent gaps where the private sector fails to build required apps.

"Part of the answer will be if you build something in-house, open source it and make it available for other people to improve," he tells Public Servant. "We have got a long history of building proprietary systems in government. That is not the way people are thinking anymore."

Ministers are trying to make open data work – and not just Francis Maude in the Cabinet Office. In November, International Development Secretary Justine Greening set out plans to help millions in the developing world by using mobile technology to open governments and to give people a say on how international aid is spent.

And Education Secretary Michael Gove has said he wants to open access to the National Pupil Database after being forced to deny data to genuine researchers who want to explore "sexual exploitation, the impact on the environment of school transport, and demographic modelling".

Yiu says the government is now getting better at releasing the right data to the right people. But it needs an aggressive prioritisation of what data to publish.

"There is always this tension that policymakers want to do a very structured release. Often you can't tell what the cool exciting thing will be that someone does. Sometimes you have to take a deep breath to get all the information out. But, frankly, lots of datasets are still six months out of date, nine months out of date. We are still in this era of publishing snapshots."

Yiu and McCluggage both see that real time information will be important. And this will not just mean open data – but big data.

"Big data analytics is about the predictive nature that can be gleaned rather than things which are statistical and historic," says McCluggage.

"It allows us to focus on new services in a different way – it focuses on wellness rather than dealing with chronic illnesses. Business intelligence looks backwards. Intelligence data can be a year old. Big data is looking at information in real time."

Ministers are keen to see much more of the government's data being used to drive innovation and improve people's lives To achieve that Bill McCluggage tells Matthew D'Arcy government must adopt a strategic approach to open data instead of just pushing it out – and hoping for the best