Integrating APIs to Make Data Integration More Valuable

Those who leverage data integration tools already know how valuable they can be. Primarily, they can meditate the differences in schema and content as data flows from the source system to the target system. These tools can also split data, combine data, aggregate data, and, in some cases, abstract or virtualize the data.

However, the integration of third party APIs is often overlooked as a means to enhance the data integration process. These capabilities these APIs deliver to the data integration flows can include:

Language translation

Currency conversion

Data validation

Access to social media data

Current temperature based upon a location

Zip code validation

Etc.

The mechanisms are well tested and easy to follow. Within your data flow, you typically take the content of a message, or other unit of data, and submit it to an API. The API returns the result, and then the message, or other unit of data, is updated. For instance, convert currency from US to Euros on the fly, or make sure that a Zip code is valid. The uses are typically very fine-grained, and they usually add value to the data.

There are two emerging types of APIs: Public and private. Much like public clouds, public APIs are built, deployed, and managed by cloud services that expose APIs for public use. Private APIs, like private clouds, are built for private consumption, typically for a single application or data integration tool that exists on a private platform. Both types can be leveraged by data integration tools, as long as access to the APIs are provided.

Figure 1 depicts the growth of public APIs since 2005, as tracked by programableweb.com. The use of services in emerging enterprise architecture drove most of this growth, such as SOA, as well as the rise of cloud computing. The use of APIs continues to accelerate as more public cloud providers upgrade and increase the breadth of their services, and enterprises continue to learn the value of APIs.

Growth of API usage from 2005 to 2013, as tracked by ProgrammableWeb.com, is reflective of API usage within IT in general. Note that growth is accelerating.

The idea is to leverage APIs that others build and maintain, versus leveraging DIY APIs. This makes the use of APIs much more cost effective, and speeds your time to market. However, there are certain APIs that you may have to build, to exist either inside or outside of your data integration engine, depending upon your requirements.

As data integration applications become more complex, and the use of data becomes more sophisticated, the use of external APIs to deal with some data operations will become more commonplace. These were typically home spun APIs in the past. As time goes on, more and more APIs will be delivered by public cloud/API providers. Now is a great time to test this concept, if you have not done so already.