The use of big data, in general, is still in its early stages for many industries, but the financial services industry has been dealing with big data for years. In fact, it’s already been managed and embedded into core financial processes. What used to be done in hours can now be done in minutes thanks to advanced data processing capabilities being applied to everything from capital market portfolio management applications to financial risk management. Prior to such advancements, data from previous days or weeks was analyzed to help re-strategize market approaches for the next day’s trading. But now, with more complex data analytics capabilities, financial firms are able to shorten that window for data processing and create more up-to-date strategies and trading adjustments in real time.

However, it’s not just the increasing volume of data sets that is of concern to financial firms. There’s also the velocity and variety of the data to consider. When pulling clusters of diverse databases together for both structured and unstructured data analysis, financial firms rely on having powerful processing speeds, especially as real-time insight is increasingly a key strategic factor in market analysis and trading strategies. But are financial institutions equipped with the proper infrastructure to effectively handle the three V’s of Big Data – volume, velocity and variety – and benefit from real-time data analysis?

Increasing the Value of Real-Time Operations

With real-time data analysis, financial institutions are better able to manage risk and alert customers to real-time issues. If a firm is able to manage risk in real time, that not only translates into better trading performance, but also ensures regulatory compliance. Such improvements can be seen in consumer instances with enhanced credit card transaction monitoring and fraud protection and prevention measures. But, on a larger scale, the most recognizable incident that would have benefited from better data analysis may have been the collapse of Lehman Brothers.

When Lehman Brothers went down, it was called the Pearl Harbor moment of the U.S. financial crisis. Yet, it took the industry days to fully understand how they were exposed to that kind of devastating risk. For every transaction made, it’s imperative that the financial firms understand the impact, or, as an extreme scenario, risk another “Lehman-esque” collapse. Today, with advancements in big data analysis and data processing, whenever any trader makes a trade, financial firms know what’s going to happen in real time through the risk management department–that is, if they have the right infrastructure.

Optimizing Current Infrastructure

The crux of handling the volume, velocity and variety of big data in the financial sector lies within the underlying infrastructure. Many financial institutions’ critical systems are still dependent on legacy infrastructure. Yet to handle increasingly real-time operations, firms need to find a way to wean off of legacy systems and be more competitive and receptive to their own big data needs.

To address this issue, many financial institutions have implemented software-as-a-service (SaaS) applications that are accessible via the Internet. With such solutions, firms can collect data through a remote service and without the need to worry about overloading their existing infrastructure. Beyond SaaS apps, other financial companies have addressed their infrastructure concerns by using open source software that allows them to simply plug their algorithms and trading policies into the system, leaving it to handle their increasingly demanding processing and data analysis tasks.

In reality, migrating off legacy infrastructure is a painful process. The time and expense required to handle such a process means the value of the switch must far outweigh the risks. Having a worthwhile business case is, therefore, key to instigating any massive infrastructure migration. Today, however, more and more financial firms are finding that big data analysis is impetus enough to make a strong business case and are using solutions like SaaS applications and open source software as stepping stones for complete migrations to ultimately leave their legacy infrastructure behind.

Integrating Social Data

While the velocity and variety of big data volumes from everyday trading transactions and market fluctuations may be enough of a catalyst for infrastructure migrations and optimization, now that social data is creeping into the mix, the business case becomes even more compelling.

Related Stories

When we think of Big Data, the three Vs come to mind – volume, velocity and variety. Just as the amount of data is increasing, the speed at which it transits enterprises and entire industries is faster than ever, writes Steve Baunach of StarView. Read More

The third annual O’Reilly Velocity conference brought together more than 1,000 attendees to explore web site performance. In "Scenes from Velocity 2010" we bring you the conference in pictures. Read More

Google Senior VP of Operations Urs Hölzle will be the keynote speaker at Velocity 2010, the O’Reilly Web Performance and Operations Conference June 22-24 at the Hyatt Santa Clara in Santa Clara, Calif. Read More

Add Your Comments

6 Comments

Excellent piece Patrick. You really nail how Gartner's "3Vs" of big data apply to the financial sector. It's good also to see the marketplace at large finally embracing our "3Vs" albeit 12 years after we first posited them in a piece I wrote on the "Three Dimensional Data Challenge" (ref: http://goo.gl/wH3qG).
Cheers,
Doug Laney, VP Research, Gartner, @doug_laney

Excellent article, I would like to include domain expertise as part of the existing or needed infrastructure.
The domain knowledge is very important for proper usage of all that big data has to offer, otherwise correlation can be mistaken for causation, time and effort spent on acquiring and utilizing incremental data with diminishing marginal returns.
This will also address my contribution to the Vs of big data, an understanding of the vacuum in whatever the big data is showing, data limitations have to be understood and factored in otherwise expensive mistakes can be made. I covered that at International Big Data Week events this year.
Cheers
Kamal

Sarah TrellJuly 31, 2013 at 12:08 pm

Great post Patrick. May I also propose 'V' for viability - http://www.pros.com/big-vs-big-data/

Information is being created at a faster pace than ever before with these varied channels of social data that are increasing their daily output of content. Thanks to social data, businesses are now able to uncover the latent, hidden relationships among these variables.

Get Daily Email News from DCK!

Subscribe now and get our special report, "The World's Most Unique Data Centers."