One frustration for companies eyeing Hadoop has been a lack of operations visibility. Attunity is looking to eliminate this roadblock, thanks to the rollout of Attunity Visiblity for Hadoop’s new capabilities for intelligent data management designed for data lakes.

One nagging frustration for companies eyeing Hadoop has been a lack of operations visibility. Attunity is looking to eliminate this roadblock, thanks to the rollout of Attunity Visiblity for Hadoop’s new capabilities for intelligent data management designed for data lakes.

Attunity Visibility for Hadoop adds a host of data lake analytics and management features many organizations have been looking for, according to Attunity execs.

“Managing and monitoring live Hadoop data lakes has become more challenging as systems continue to increase in scale. . . . Capacity planning was a major piece missing in Hadoop. There has just not been enough of a granular, detailed sense of what files were driving,” Attunity’s senior director Kevin Petrie told IDN. “With Attunity Visibility for Hadoop, we’re deepening the analytics capability and visibility at the file and storage level.”

Attunity Visibility for Hadoop comes as many enterprises looking to pursue big data strategies – especially long-time data warehouses – have felt a lack of visibility into Hadoop and data lake environments, had forced them to make an uncomfortable trade-off. That trade-off being: to gain Hadoop’s benefits, they’ll have to give us all those valuable EDW analytics that support capacity planning and other informed data management practices.

Attunity execs expect to resolve this trade-off, by bringing Hadoop analytics to its Attunity Visibility intelligent data management platform. “Building on the extensive experience we built into Attunity Visibility in the enterprise data warehouse market, we now offer similar comprehensive capabilities for the Hadoop market, all within a single innovative and unified platform,” Attunity’s Itamar Ankorion, executive vice president for corporate strategy said in a statement.

Attunity Visibility for Hadoop offers top-line summary views of the data, as well as finer-grained, drill-down views into user actions for file groups and directories. Attunity Visibility for Hadoop’s expanded capabilities include analytics across storage levels in the Hadoop File System (HDFS), as well as Hadoop data processing engines such as MapReduce, Tez, Hive and Cloudera Impala.

“These features will let users quickly see more about how various components are performing, Petrie said. “Users can see and understand CPU usage, for example, and use that information to optimize data placement. This is also helpful in troubleshooting to quickly identify what users or apps may be creating a bottleneck,” he told IDN.

Attunity Visibility for Hadoop also offers automatic alerts and reporting in the case any pre-set thresholds are passed. “Now, IT will have the visibility they need to make smarter investments,” Petrie said.

To build Attunity Visiblity for Hadoop, Petrie told us his company leveraged a host of valuable and long-held learnings from its support for data warehousing resource management.

“Data warehouse users have been able to use Attunity Visibility to understand data and resource consumption for structured data. They can now capture and apply those metrics to unstructured data that may be in their Hadoop data lake,” Petrie said. “IT teams will have the information and insights to track how specific users groups are using their data lake resources, and even track growth trends over time. In the end, all this will let IT make much more informed capacity planning decisions.”

Ability to analyze data usage from a variety of perspectives: This lets admins identify unused data that can be retired or archived

Integrated dashboard with easy to consume BI and analytics to enable IT teams to diagnose and assess performance bottlenecks

Petrie also went under-the-covers, to explain how Attunity was able to leverage its data warehouse management technologies and expertise for the new needs of a Hadoop infrastructure.

“We’ve been providing data warehouse users with capabilities for understanding data usage and resource consumption. This provided very detailed metrics for resource planning and future investment. The technologies that would support rigorous capacity planning have been a missing piece in Hadoop. While Hadoop adoption and use has certainly been growing fast, organizations have not had enough of a granular, detailed sense of what files were driving that growth. Now, IT will have the visibility they need to make smarter investments,” Petrie said.

The need for better visibility, management and control of Hadoop and data lakes comes as the industry continues to push Hadoop out of the lab (doing experimental projects or PoCs (proof of concepts) to on-going, enterprise-scale actual production, Petrie added. He pointed to research from research from Wikibon that predicts the Big Data market will grow more than 400% over the next 10 years – from $18.3B in 2014 up to an astounding $92.2B in 2026.