Big Data Use Cases Across Financial Services

Examining Big Data and its specific applications in the world of finance.

According to a new Gartner report around 64% of firms have either deployed or launched a Big Data initiative in 2013. Industry watchers anticipate this number will only increase in 2014 and beyond. Financial services will be at the forefront of this initial look at Big Data taking the first baby steps in a multiyear voyage.

Few industries are as data centric or as awash in data as finance. Every interaction a client has with a banking institution can potentially produce actionable data that has potential business value associated with it. Capital Markets has always had multiple sources and silos of data in all areas, the front office, back office and the mid office. From a business perspective as well as a technology impact perspective, the biggest technical trend facing executives in financial services is undoubtedly Big Data. The former is perhaps the most important factor determining the pace of adoption.

Lines of business spanning areas like consumer banking, capital markets, wealth management are now beginning to ask questions of themselves as to how they need to not just on board all this data but also how to then draw actionable insights from it within a reasonable SLA. That puts a lot of pressure on IT to deliver on this vision as it moves from bestseller to the datacenter.

The Unrelenting Pace of Technology

We are witnessing the fastest ever rate of innovation in the data space primarily as a result of explosive developments in areas like Hadoop/ MapReduce (especially with Gen 2 & YARN), the Cloud and the NoSQL ecosystems. Couple these developments with ever favorable economics on the hardware end of the spectrum which make possible very large deployments of commodity hardware - be they pools of in-memory data grids (RAM) or scale out NAS based storage, all typically running on cheaper hardware (potentially using ever cheaper Flash/SSD technologies) deploying Open Source solutions. Software defined storage products that let IT abstract and pool storage across on-premise and cloud environments are now starting to be available at a very reasonable price point as well. All these components will provide a foundation to enable the highly scalable and self healing architectures that Big Data implementations will require.

We would like to clarify at the outset that there is no single and universal definition of Big Data (from a size perspective). It varies from an organization, line of business or even application perspective. Industry analysts widely describe the 3 V's (Volume, velocity and variety) as the trifecta from a definition perspective. Let's add a fourth to it, Veracity – which pertains to the signal to noise ratio and the concomitant problem of unclean data.

Social, Cloud, Mobility and Analytics are converging trends and one could contend that these three are all part of the Big Data lifecycle.

Social Media enables direct communication between financial services and their retail clients. These interactions can provide a valuable way to garner feedback around specific product lines as well as a way of landing a younger clientele.

Cloud plays a role in the life cycle of development, deployment and optimization of Big Data applications. Innovation at the largest banks is often shackled by the absence of a responsive and agile infrastructure. It can take days to procure servers to host bursts of workloads that may not be feasible for existing IT departments to rapidly turn around. This problem is by no means unique to Big Data but what makes it even more relevant here is that processing needs in this specialized area are typically larger than your average IT application. Cloud Management Platforms (CMP) are beginning to provide orchestration capabilities by means of workload portability around public and private clouds. Developments around OpenStack and around DevOps are providing complementary capabilities in this evolving space.

At that point, options like Amazon EMR (Elastic Map Reduce) and other hosted Big Data solutions become very attractive not just from -

1) a time to market perspective (an expendable pool of computing capacity as well as the choice of pre-canned & pre-configured Big Data stacks are all available for one to deploy)

but also

2) from a cost perspective (you only pay for what you use)

Mobile technologies results in the production of more data and that too at a very high velocity. This velocity is both ingress velocity ("the speed at which these feeds can be sent into your architecture") as well as egress velocity ("the speed at which actionable results need to be gleaned from this data"). In the era of BYOD, data is king.

Finally, Analytics is the first killer app for Big Data. Be it the low hanging fruit of reporting & dashboards to forecasting and predictive modeling and even Data Science. One of the biggest trends for 2014, is the enhancement of analytics capabilities to incorporate real streams of data at a humongous scale. Existing applications can now incorporate such functionality to derive real time meaning from this data.

Financial Services Use Cases

Big Data use-cases can be classified into two broad areas from a business value perspective. The first in how they help you build a competitive advantage in the Red Ocean part of your business and the second in now they help you develop blue oceans. The CIO's primary concern should be business value and competitiveness and not just keeping with the trends for the sake of it.
Over the coming weeks we will explore four of these distinct use cases where firms are beginning to use Big Data related technologies. There are more but we believe these will be foundational as they touch areas where the financial industry is experiencing fundamental business shifts.

The fourth bucket is the ubiquitous Investment management space where newer (and increasingly unstructured) sources of data are being pulled in and combined with existing sources as part of investment decisions without negatively impacting latency around the applications themselves.
As Chief Architect of Red Hat's Financial Services Vertical, Vamsi Chemitiganti is responsible for driving Red Hat's technology vision from a client standpoint. The breadth of these areas range from Platform, Middleware, Storage to Big Data and Cloud (IaaS and PaaS). The ... View Full Bio

Siddharth - You raise a very pertinent point. Legacy systems and their place in the cloud has been a bit of challenge for myriad reasons - older stacks,non homogeneous deployment requirements,sunset development etc etc. It has been interesting to watch all the different approaches to solving this challenge. While there is no "one size fits all" approach , most firms are developing a model where newer apps are making the move to the cloud based on a set of criteria; the others that cant make it are either being forced to develop open interfaces etc or being ported to the newer cloud standard.It is safe to say that OpenStack and other cloud OS's are all intended for non legacy apps. This is true even from a hardware and network (read SDN) perspective. Perhaps a great topic for a post one of these days :)Thank you for your comment,Vamsi

The existing legacy systems -- built up over decades -- are definitely one of the largest obstacles to adopting cloud. Not only do firms need to make the decision to use cloud, but they also need to find ways to untangle existing applications from legacy architecture. Untangling existing applications is no easy task.

hi, really looking forward to see more on the use cases. I strongly feel Cloud will be the integral driver for rest of S, M, A stack as it will drastically cut down go to market times. Will be interesting to see how quickly banks adopt it, as I feel over dependence on legacy systems can be the biggest hurdle for moving non sensitive data onto cloud.