Information and the Web Worker

You probably heard about the recent release of the IDC study “The Expanding Digital Universe.” It was pretty hard to miss. The big headline-grabber news was that the amount of worldwide information is projected to grow from 161 exabytes in 2006 to 988 exabytes in 2010. Sadly, most of the news services covered this with a “wow, what a lot of bytes!” attitude, perhaps with a sidebar explaining what an exabyte is (a million terabytes, in case your math is a bit rusty). That’s a shame, because there’s much else to be found if you take the time to read the full study.

For example, no matter how much spam you get, only about 3% of the world’s digital information is in the form of e-mail. Most of the bits come in the form of moving and still images: TV signals, movies, camera phone images, digital pictures. It’s also worth noting that IDC is counting replicated data as well as created and captured data in figuring out the size of the digital universe, so one movie on a million DVDs counts a million times.

But most interesting from the web worker point of view is one bleak little fact: 2007 is the year that our ability to stuff bits into the digital universe will outstrip our ability to store them. By 2010, the total amount of data will overwhelm the total amount of digital storage by a factor of nearly two to one. Whether it’s that e-mail offering to sell you a timeshare condo, the picture of your niece that you sent wirelessly to your mom, or a show that you recorded to watch later, something is going to be lost forever – and looking at the trends, the proportion of things that get lost forever will keep increasing.

Some people will look at this and see doom. I see opportunity. James Governor recently posted a piece on knowledge workers as switchboard operators in which he emphasized the role of many web workers in making connections. We’re the ones who put people and knowledge together. We’re the magpies who pick through the huge pile of shiny things out there, pull out the bits that attract us, and arrange them in interesting new ways. This is a skill (and dare I say it, a career) that’s getting more and more important as that tidal wave of data continuously threatens to overwhelm us.

This is where, for the foreseeable future, people will continue make a difference. The stories claiming there will be 988 exabytes of information in 2010 are simply wrong. There will be 988 exabytes of raw data, but information is something else again: data that is interesting and useful and above all meaningful. Web workers, as explorers in the vast supply of the world’s data, will continue to bring home the information to enrich us all.