How generating conversations can become one of the most important data assets for any organization.

At O’Reilly Research, we focus our attention on trends in technology adoption — which tools are adopted and in which industries. In doing so, we uncover interesting cross-disciplinary opportunities and discover what we can learn from innovations in other fields.

We’ve recently learned about the increasing role of data in the fashion industry, so we set out to uncover some of the players who are making disruptive changes using technology and analytics.

If you want to engage with the data that's surrounding you, Mining the Social Web is the best place to start.

When we first published Mining the Social Web, I thought it was one of the most important books I worked on that year. Now that we’re publishing a second edition (which I didn’t work on), I find that I agree with myself. With this new edition, Mining the Social Web is more important than ever.

While we’re seeing more and more cynicism about the value of data, and particularly “big data,” that cynicism isn’t shared by most people who actually work with data. Data has undoubtedly been overhyped and oversold, but the best way to arm yourself against the hype machine is to start working with data yourself, to find out what you can and can’t learn. And there’s no shortage of data around. Everything we do leaves a cloud of data behind it: Twitter, Facebook, Google+ — to say nothing of the thousands of other social sites out there, such as Pinterest, Yelp, Foursquare, you name it. Google is doing a great job of mining your data for value. Why shouldn’t you?

There are few better ways to learn about mining social data than by starting with Twitter; Twitter is really a ready-made laboratory for the new data scientist. And this book is without a doubt the best and most thorough approach to mining Twitter data out there. Read more…

Intel’s taking the lead in the new “data economy”

Intel is looking to take the lead in what it has dubbed the “data economy,” helping consumers and individuals realize and retain more value from their personal data. Antonio Regalado and Jessica Leber report at MIT Technology Review that the the world’s largest computer chip maker has launched a “Data Economy Initiative.” Ken Anderson, a cultural anthropologist who is in charge of the project, described the initiative to them as “a multiyear study whose goal is to explore new uses of technology that might let people benefit more directly, and in new ways, from their own data.”

As part of the initiative, Intel is funding hackathons to encourage developers to experiment with personal data in new ways, Regalado and Leber note. “[Intel] has also paid for a rebellious-sounding website called We the Data,” they report, “featuring raised fists and stories comparing Facebook to Exxon Mobil.” Read more…

Nigel Shadbolt on AI, ODI, and how personal, open data could empower consumers in the 21st century.

After years of steady growth, open data is now entering into public discourse, particularly in the public sector. If President Barack Obama decides to put the White House’s long-awaited new open data mandate before the nation this spring, it will finally enter the mainstream.

As more governments, businesses, media organizations and institutions adopt open data initiatives, interest in the evidence behind release and the outcomes from it is similarly increasing. High hopes abound in many sectors, from development to energy to health to safety to transportation.

“Today, the digital revolution fueled by open data is starting to do for the modern world of agriculture what the industrial revolution did for agricultural productivity over the past century,” said Secretary of Agriculture Tom Vilsack, speaking at the G-8 Open Data for Agriculture Conference.

As other countries consider releasing their public sector information as data and machine-readable formats onto the Internet, they’ll need to consider and learn from years of effort at data.gov.uk, data.gov in the United States, and Kenya in Africa.

One of the crucial sources of analysis for the success or failure of open data efforts will necessarily be research institutions and academics. That’s precisely why research from the Open Data Institute and Professor Nigel Shadbolt (@Nigel_Shadbolt) will matter in the months and years ahead.

In the following interview, Professor Shadbolt and I discuss what lies ahead. His responses were lightly edited for content and clarity.Read more…

Rep. Issa expressed support for reforming FOIA to include personal data held by companies.

The Freedom of Information Act (FOIA), which gives the people and press the right to access information from government, is one of the pillars of open government in the modern age. In the United States, FOIA is relatively new — it was originally enacted on July 4, 1966. As other countries around the world enshrine the principle into their legal systems, new questions about FOIA are arising, particularly when private industry takes on services that previously were delivered by government.

In that context, one of the federal open government initiatives worth watching in 2012 is ‘smart disclosure,’ the targeted release of information about citizens or about services they consume by government and by private industry. Smart disclosure is notable because there’s some “there there.” It’s not just a matter of it being one of the “flagship open government initiatives” under the U.S. National Plan for open government or that a White House Smart Disclosure Summit in March featured a standing room only audience at the National Archives. When compared to other initiatives, there has been relatively strong uptake of data from government and the private sector and its use in the consumer finance sector. Citizens can download their bank records and use them to make different decisions.

Earlier this summer, I interviewed Representative Darrell Issa (R-CA) about a number of issues related to open government, including what he thought of “smart disclosure” initiatives.

Dr. Nadav Aharony used phone sensors to explore personal behaviors and community trends.

It’s clear at this point that the smartphone revolution has very little to do with the phone function in these devices. Rather, it’s the unique mix of sensors, always-on connectivity and mass consumer adoption that’s shaping business and culture.

Dr. Nadav Aharony (@nadavaha) tapped into this mix when he was working on a “social MRI” study in MIT’s Media Lab. Aharony, who recently joined us as part of our ongoing foo interview series, described his vision of the social MRI:

“If you think about it, the three things you take with you when you go out of your home are your keys, your wallet and your phone, so our phones are always with us. In aggregate, we can use the phones in many people’s pockets as a virtual imaging chamber. So, one aspect of the social MRI is this virtual imaging chamber that is collecting tens or hundreds of signals at the same time from members of the community.” [Discussed at 1:16]

Aharony’s work focused on 150 participants (about 75 families) that were given phones for 15 months. During that time, more than one million hours of “continuous sensing data” was gathered with the participants’ consent. The data was acquired and scrubbed under MIT’s ethics guidelines, and for extra measure, Aharony included his own data in the dataset.

Collecting the data was just the beginning. Parsing that information and creating experiments based on emerging signals is where the applications of a social MRI became significant.Read more…

"Choice engines" are helping consumers make smarter decisions through personal and government data.

Smart disclosure is when a company or government agency provides consumers with periodic access to personal data in an open format. Citizens can put their own data assets to work in making better choices about finance, healthcare, travel, energy, education, real estate and more.