I recently spent the weekend up in the lakes region of New Hampshire, and made the rounds of all the various country stores and craft shops that are a staple of the area. In one shop, I noticed something I hadn’t seen in a long time: a large set of Russian nesting dolls. The quality, craftsmanship, and level of detail was impressive. As I removed the cover on the first intricately painted doll, nestled inside was the next, slightly smaller one, a duplicate of the outer doll, with all of the same detail on a slightly smaller scale. As I continued opening each successive doll, revealing the next smaller duplicate, I started looking for discrepancies, loss of detail in the duplication as I dove recursively into smaller and smaller dolls. The quality and attention to detail, even at the smallest level, remained for each of the dolls. Each doll was a representation of information, just on different levels and scales.

Internet of Things, Small Data Inside the Big Data

We can’t seem to have a technology discussion without the topic of the Internet of Things (IoT) entering the conversation. These ‘Things’ can be anything, from a small wearable wristband to large items such as your car or house. The data transmitted by these devices, small intricate details, are sent to the cloud somewhere. Imagine for a moment, starting to combine these pieces of data, all relating to an aspect of you. The combining data forms larger perspectives- a bigger ‘doll’, still containing the small ‘dolls’ of data within. Then your data ‘doll’ gets combined with your neighbors, forming another larger one. We can continue growing and expanding upwards, creating representations of information about you, your home, your neighborhood, your town, at different levels and different scales. Imagine being able to move up and down those perspectives as easily as opening up Russian nesting dolls.

Are we there yet? To gain some perspective, Gartner predicts that there will be 4.9 Billion (with a B) connected ‘Things’ by the end of this year, up 30% from last year. They further predict that the number of ‘Things’ will exceed 25 Billion in another 5 years. No matter how you view it, that’s a lot of devices and data going off into the cloud. The Internet of Things is, and will be, a huge contributor to the whole concept of big data. The real question is: Can this data, and the combining together of data to form bigger views and perspective, work to provide real value to the business? Or, is it going to result it just huge piles of information for information sake?

It Still All Comes Down to Understanding the Data

While the Internet of Things is and will be a huge contributor to the world of big data, it doesn’t fundamentally change the risks and challenges surrounding big data. Data without an understanding of the meaning is just taking up storage space. A corollary to that is combining different aspects from various Internet of Things devices, without understanding if there is a meaningful and useful relationship which could result in false correlations or misleading perspectives being displayed by your ‘data dolls’. (As I have mentioned in other blogs, www.tylervigen.com contains many entertaining examples of false correlations). The proper understanding requires a combination of business and technical skillsets. An understanding of the business is always key, but not enough with these potential huge volumes of data. Proper analysis and correlation of information requires deep technical skills in computer science, communication, and statistics.

No technology negates the need for good planning and design. The Internet of Things provides us all types of structured and unstructured data, and there are lots of powerful analytic tools available to work with that data. In order to obtain true business value, one must still plan and design how to leverage and balance all the information and its analysis. They must provide the appropriate resources working with the business to ensure there is the correct understanding of the data that is available, combining the information to form larger perspectives. Those that succeed will be able to build useful sets of nested data, providing business value as you move up and down those perspectives, and providing valuable business information at all levels by leveraging those perspectives.