Tag Archives: Data Integration Tools

The concept of the “Internet of Things” (IOT) is about getting devices we leverage in our daily lives, or devices used in industrial applications, to communicate with other devices or systems. This is not a new notion, but the bandwidth and connectivity mechanisms to make the IOT practical is a recent development.

My first job out of college was to figure out how to get devices that monitored and controlled an advanced cooling and heating system to communicate with a centralized and automated control center. We ended up building custom PCs for the application, running a version of Unix (DOS would not cut it), and the PCs mounted in industrial cases would communicate with the temperature and humidity sensors, as well as turn on and turn off fans and dampers.

At then end of the day, this was a data integration, not an engineering problem, that we were attempting to solve. The devices had to talk to the PCs, and the PC had to talk to a centralized system (Mainframe) that was able to receive the data, as well as use that data to determine what actions to take. For instance, the ability determine that 78 degrees was too warm for a clean room, and that a damper had to be open and a fan turned on to reduce the temperature, and then turn off when the temperature returned to normal.

Back in the day, we had to create and deploy custom drivers and software. These days, most devices have well-defined interfaces, or APIs, that developers and data integration tools can access to gather information from that device. We also have high performing networks. Much like any source or target system, these devices produce data which is typically bound to a structure, and that data can be consumed and restructured to meet the needs of the target system.

For instance, data coming off a smart thermostat in your home may be in the following structure:

Device (char 10)
Date (char 8)
Temp (num 3)

You’re able to access this device using an API (typically a REST-based Web Service), which returns a single chunk of data which is bound to the structure, such as:

Device (“9999999999”)
Date (“09162014”)
Temp (076)

Then you can transform the structure into something that’s native to the target system that receives this data, as well as translate the data (e.g., converting the Data form characters to numbers). This is where data integration technology makes money for you, given its ability to deal with the complexity of translating and transforming the information that comes off the device, so it can be placed in a system or data store that’s able to monitor, analyze, and react to this data.

This is really what the IOT is all about; the ability to have devices spin out data that is leveraged to make better use of the devices. The possibilities are endless, as to what can be done with that data, and how we can better manage these devices. Data integration is key. Trust me, it’s much easier to integrate with devices these days than it was back in the day.

Thank you for reading about Data Integration with Devices! Editor’s note: For more information on Data Integration, consider downloading “Data Integration for Dummies“

In my last posting, I listed the functional criteria that data integration platforms must support in order to address the comprehensive needs of most organizations. But it’s not only important to consider what the platform can do—it’s important to think about how things get done. One key aspect is how unified the platform is, as unification can radically simplify deployment and management of the platform.

What does “unified” mean when it comes to a data integration platform? There are a lot of ways to define this, but the most important is from the viewpoint of the users of the platform. You can have an extremely elegant, unified technical architecture underneath a platform, but if users still have a disjointed experience, that architectural elegance doesn’t really matter. A unified user experience is a product of both the design of the user tools, as well as how metadata is shared.

From a tools perspective, a common look and feel, and cross-tool integration, is key to making it easy for users to ramp up on the tools, and to reuse assets across them. On the other hand, a unified experience does not mean trying to cram every single function or capability into a single tool.

That is not practical because there are many different roles involved in data integration—from stewards and analysts to architects and developers—and they each have different tasks to do, and bring different skills to the table. So the tools have to be tailored for each role, but still foster collaboration across the different users and roles. (more…)