In the first edition of our free Big Data Now anthology, the O’Reilly team tracked the birth and early development of data tools and data science. Now, with the second edition, we’re seeing what happens when big data grows up: how it’s being applied, where it’s playing a role, and the consequences — good and bad alike — of data’s ascendance.

We’ve organized the 2012 edition of Big Data Now into five areas:

Getting Up to Speed With Big Data — Essential information on the structures and definitions of big data.

Catch live keynotes from this week's Strata Conference in London.

Experts from across the data world are coming together at the O’Reilly Strata Conference in London this week. You can watch live keynotes from the event below (full broadcast schedule is available here).

The convergence of data, privacy and cost have created a unique opportunity to reshape health care.

Health care appears immune to disruption. It’s a space where the stakes are high, the incumbents are entrenched, and lessons from other industries don’t always apply.

Yet, in a recent conversation between Tim O’Reilly and Roger Magoulas it became evident that we’re approaching an unparalleled opportunity for health care change. O’Reilly and Magoulas explained how the convergence of data access, changing perspectives on privacy, and the enormous expense of care are pushing the health space toward disruption.

As always, the primary catalyst is money. The United States is facing what Magoulas called an “existential crisis in health care costs” [discussed at the 3:43 mark]. Everyone can see that the current model is unsustainable. It simply doesn’t scale. And that means we’ve arrived at a place where party lines are irrelevant and tough solutions are the only options.

“Who is it that said change happens when the pain of not changing is greater than the pain of changing?” O’Reilly asked. “We’re now reaching that point.” [3:55]

(Note: The source of that quote is hard to pin down, but the sentiment certainly applies.)

This willingness to change is shifting perspectives on health data. Some patients are making their personal data available so they and others can benefit. Magoulas noted that even health companies, which have long guarded their data, are warming to collaboration.

At the same time there’s a growing understanding that health data must be contextualized. Simply having genomic information and patient histories isn’t good enough. True insight — the kind that can improve quality of life — is only possible when datasets are combined.

Julien Smith on the realities of modern safety and our misfiring flinch reflexes.

The subject came up during our interview at Foo Camp 2012 — part of our ongoing foo interview series — in which Smith argued that our brains and innate responses don’t always map to the safety of our modern world:

“We’re in a place where it’s fundamentally almost impossible to die. I could literally — there’s a table in front of me made of glass — I could throw myself onto the table. I could attempt to even cut myself in the face or the throat, and before I did that, all these things would stop me. You would find a way to stop me. It’s impossible for me to die.”

Smith didn’t test his theory, but he makes a good point. The way we respond to the world often doesn’t correspond with the world’s true state. And he’s right about that not-letting-him-die thing; myself and the other people in the room would have jumped in had he crashed through a pane of glass. He would have then gone to an emergency room where the doctors and nurses would usher him through a life-saving process. The whole thing is set up to keep him among the living.

Acknowledging the safety of an environment isn’t something most people do by default. Perhaps we don’t want to tempt fate. Or maybe we’re wired to identify threats even when they’re not present. This disconnect between our ancient physical responses and our modern environments is one of the things Smith explores in his book The Flinch.

Heavy data, open source strategies for businesses, and collaborating on code.

This week on O’Reilly: Jim Stogdill said data is getting heavier relative to the networks that carry it around the data center; Simon Phipps revealed open source community strategies relevant to the enterprise; and Team Geek authors Brian Fitzpatrick and Ben Collins-Sussman discussed the importance of developer collaboration.

Dr. Nadav Aharony used phone sensors to explore personal behaviors and community trends.

It’s clear at this point that the smartphone revolution has very little to do with the phone function in these devices. Rather, it’s the unique mix of sensors, always-on connectivity and mass consumer adoption that’s shaping business and culture.

Dr. Nadav Aharony (@nadavaha) tapped into this mix when he was working on a “social MRI” study in MIT’s Media Lab. Aharony, who recently joined us as part of our ongoing foo interview series, described his vision of the social MRI:

“If you think about it, the three things you take with you when you go out of your home are your keys, your wallet and your phone, so our phones are always with us. In aggregate, we can use the phones in many people’s pockets as a virtual imaging chamber. So, one aspect of the social MRI is this virtual imaging chamber that is collecting tens or hundreds of signals at the same time from members of the community.” [Discussed at 1:16]

Aharony’s work focused on 150 participants (about 75 families) that were given phones for 15 months. During that time, more than one million hours of “continuous sensing data” was gathered with the participants’ consent. The data was acquired and scrubbed under MIT’s ethics guidelines, and for extra measure, Aharony included his own data in the dataset.

Collecting the data was just the beginning. Parsing that information and creating experiments based on emerging signals is where the applications of a social MRI became significant.Read more…

Team Geek authors Brian Fitzpatrick and Ben Collins-Sussman on coding myths and collaboration.

If we accept that software development is a team activity (it is), the importance of collaboration and communication becomes clear. Team Geek authors Brian Fitzpatrick and Ben Collins-Sussman discuss the nuances of modern programming in this interview.

Reevaluating criticism of visualizations, why websites still matter, Amazon as friend and foe.

This week on O'Reilly: Andy Kirk made the case for open-minded criticism of visualizations, Brett Slatkin explained why you still need to own a website, and Greenleaf Book Group CEO Clint Greenleaf discussed the complicated relationship between publishers and Amazon.

Brett Slatkin on the federated social web and why a website still matters.

Brett Slatkin's hope for a federated social web hasn't worked out as expected, so he's shifting perspective from infrastructure to user behavior. Here he explains why you shouldn't abandon your website for third-party platforms.