That program is bringing together more personalized data about its 860,000 students, 2,000 schools and 178 districts. It was recently featured at an MIT Data Quality symposium as a case study by Informatica, which Colorado is using for functions like the extract, transfer and load (ETL) portion of a significant data project that is built on an Oracle database and uses Cognos reporting tools.

"RISE is our information backbone for all of our education reform initiatives," said Domagala. Developing it was spurred by a federal grant in 2009.

Since then, Domagala's group has developed a model looking at percentile growth of performance, drawing from historic state assessment and other data. It's also launched Schoolview, a Web portal that shows state, district and school performance, and through password-protected access, individual performance. It wants to see if it can compare specific classes to one another, assess individual schools on their performance growth, and possibly also districts. Colorado public school districts range from between 30 and nearly 80,000 students, and "you can imagine the technology differences between those types of districts," Domagala said. The median district in the state has 800 kids.

Daniel Domagala, CIO, Colorado Department of Education

Ideally, policy makers would be able to tell if specific programs, say, for reading intervention, have an impact on student performance.

"Information doesn't tell the whole story, but we've been able to get a glimpse on what's working or not working," Domagala said. "We probably have a ways to go to where we can definitively say here are things that are proven to work and here are things the data is not supporting."

The state wants to help show students and educators the difference between their performance and where they need to be if they want to be college or career ready. It also would like to use performance data to develop an "early warning system." For instance, if it finds that when middle school students struggle in math, it ultimately affects their ability to get into college, it could develop ways to intervene and boost those students' long-term performance.

Such intervention is still a concept, Domagala said. He thinks the state has the data it needs to do such a project, but it needs to develop the analytical framework to do so.

He said the most progress has been made with remediation rates, using performance data to help schools understand when students are ready to move on from a subject and when they need help.

Its data analysis has also helped reframe how Colorado looks at overall school performance. Instead of just being measured on absolute performance, schools now are also measured by performance growth rates. "That's changed the conversation, leveled the playing field a little bit for schools and districts that have struggled," said Domagala.

Colorado has expanding data gathering to include pre-kindergarten and post-secondary students. The "PK-20" program is part of a longitudinal database that will include data from five agencies, among them including human services, labor and corrections. Ultimately, Colorado hopes to understand how specific education initiatives contribute to life success.

That's going to take time, Domagala said. "There's so much we'd like to do, but within a government environment it's incredibly hard to find the talent we need to make the quantum leaps in information."

The big data market is not just about technologies and platforms -- it's about creating new opportunities and solving problems. The Big Data Conference provides three days of comprehensive content for business and technology professionals seeking to capitalize on the boom in data volume, variety and velocity. The Big Data Conference happens in Chicago, Oct. 21-23.

Hi Michael, thanks for highlighting our efforts in Colorado. Just one clarification - our largest school district has over 80,000 students (not 8,000), representing Jefferson county and the western suburbs of Denver. Denver Public Schools is a close second at over 70,000 students.

Hi There, in Colorado, adaptive or personalized learning is happening sporadically at the local school and district level. While the State is supporting these efforts, most of the instructional data remains only at the local level.

As far as backlash against personalized learning and student privacy, Colorado is actively experiencing this - especially through an inBloom pilot project (which David recently wrote an article on). Some of the backlash has been constructive criticism, to which we are bolstering policies and communication strategies.

Domagala told me what he's doing technically isn't big data -- only hundreds of thousands of records, for instance. But it's big enough data to create analytical challenges. He also said Colorado is far from the only state trying to do what he's doing. I'd love to hear from other CIOs about their efforts.

Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.

Why should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.