Duplicative anything – data, processes, business functions, etc, if not specifically designed, means cost and risk to the organization. For data, and the growing tidal wave of databases, content repositories, files shares, backup files, etc., the requirement to eliminate duplicate data across all significant applications must be sewn into the fabric of information management policy. Storage administrators, data architects, information security managers, and application heads collectively own the responsibility for ensuring efficient information management policy throughout an organization, and the elimination of duplicate data in online storage, backup, and archive is a primary goal of this policy. The requirements for data security and privacy protection are owned in similar ways. Here are a few key points to consider:

The use of data de-duplication features in storage arrays is one way of putting good information management policy into practice.

The same can be said about the use data encryption and key management. We've seen important moves in this direction in recent years from most storage and switch providers, including IBM, EMC, 3Par, Emulex, HP, and others.

The placement of encryption and de-duplication features in the information lifecycle become an important organizational and strategic consideration when technologies like Alberio become embedded into arrays, and solutions from firms like Ocarina and others hit the market.

De-duplication features are now becoming embedded into the IT infrastructure / storage stack. At the same time, the use of data encryption features at the desktop, application, file, and database level is increasing as the threats to information security and privacy continue to plague organizations. It is critical for most use cases, however, that encryption occur after data is de-duplicated to ensure overall integrity of the data. This means that practitioners must ensure that they consider the impacts on their data security and protection strategies as vendors leverage their R&D for de-duplication and make these capabilities part of both the infrastructure and application stacks.

Action Item: Organizations should create an information management strategy that meets both the requirements to protect the security and privacy of data end-to-end, and to eliminate duplicate data. CISOs who own encryption and CIMOs who own information management should collaborate with storage teams, application heads, and data architects to avoid type A errors that preclude the interoperability of encryption and de-duplication practices of an information management policy.