Genpact solution:Developed a multi-phase approach to port the current application to a Hadoop-based platform, which resolved capacity problems, improved performance, and saved on licensing costs

Business impact:With a newer, more scalable solution, the client can handle larger volumes while complying with stringent timelines to maintain business as usual

Business challenge

This client relies on its audit trail application as a source of timed, sequenced-order events that contain market quotation and trading information submitted by member firms. This key application is used to verify and track member firms’ submission, reporting, and processing statistics for compliance, repair, and resubmission.

The existing technology—a platform built on a Netezza DW/BI appliance and the DataStage ETL tool—was expensive, with limited scalability options, and adversely impacted trade validation with ever-increasing volumes (consistently growing by about 45% year on year). There were frequent misses in SLA due to this growing volume—and application volume was expected to grow five-to six-fold over the next two years as additional asset classes were included. The client also needed to create a central repository to receive and store consolidated audit trail data in order for regulators to view cross-market data.

Take a copy for yourself

Download PDF

Genpact approach

At the beginning of the engagement, Genpact performed a thorough root cause analysis, using Lean and Six Sigma principles. This confirmed the need for a scalable, cost-effective platform to address rising data volumes and licensing costs.

In collaboration with the client’s developers, Genpact engaged in a proof-of-concept and evaluated new designs based on Hive, NoSQL (HBase), and in-memory-based custom Map/Reduce jobs. Based on the findings, Genpact and the developers created a multi-phased solution to migrate the trade validation process to Hadoop.

Genpact solution

To address the client’s business needs, Genpact developed a multi-phased approach to port the client’s existing Netezza/DataStage application to a Hadoop-based platform:

Phase 1: Proof-of-concept to evaluate designs based on Hadoop, HDFS, Hive, and HBase

Phase 3: Migrate from Netezza to an in-memory-based custom Map/Reduce Hadoop solution, which includes Hive and HBase

This solution enabled the client to move member firms’ trade data from the proprietary NAS system to Hadoop HDFS, along with the required reference data. The data validation engine is fully customizable, and rules are created using an XML-based template that runs on a custom-built Map/ Reduce framework, as shown in figure 1.