Capital Markets are the face of the financial industry to the general public and generate a large percent of the GDP for the world economy. Despite all the negative press they have garnered since the financial crisis of 2008, capital markets perform an important social function in that they contribute heavily to economic growth and are the primary vehicle for household savings. Firms in this space allow corporations to raise capital using the underwriting process.

However, it is not just corporations that benefit from such money-raising activity – municipal, local and national governments do the same as well. Just that the overall mechanism differs – while business enterprises issue both equity and bonds, governments typically issue bonds. According to the Boston Consulting Group (BCG), the industry will grow to annual revenues of $661 billion in 2016 from $593 billion in 2015 – a healthy 12% increase. On the buy side, the asset base (AuM – Assets under Management) is expected to reach around $100 trillion by 2020 up from $74 trillion in 2014.[1]

Banking and within it, capital markets continues to generate insane amounts of data. These producers range from news providers to electronic trading participants to stock exchanges which are increasingly looking to monetize data. And it is not just the banks, regulatory authorities like the FINRA in the US are processing peak volumes of 40-75 billion market events a day [2].

In addition to data volumes, Capital Markets has always possessed a variety challenge as well. They have tons of structured data around traditional banking data, market data, reference data & other economic data. You can then factor in semi-structured data around corporate filings, news, retailer data & other gauges of economic activity. An additional challenge now is the creation of data from social media, multimedia, etc. – firms are presented with significant technology challenges and business opportunities.

I have cataloged the major ones below based on my work with the majors in the spectrum over the last year:

Client Profitability Analysis or Customer 360 view

With the passing of the Volcker Rule, the large firms are now moving over to a model based on flow-based trading rather than relying on prop trading. Thus it is critical for capital market firms to better understand their clients (be they institutional or otherwise) from a 360-degree perspective so they can be marketed to as a single entity across different channels—a key to optimizing profits with cross selling in an increasingly competitive landscape.

Regulatory Reporting – Dodd Frank/Volcker Rule Reporting

Banks have begun to leverage data lakes to capture every trade intraday and end-of-day across its lifecycle. They are then validating that no proprietary trading is occurring on on the banks behalf.

CCAR & DFast Reporting

Big Data can substantially improve the quality of raw data collected across multiple silos. This improves the understanding of a Bank’s stress test numbers.

Timely and Accurate Risk Management

Running Historical, stat VaR (Value at Risk) or both to run the business and to compare with the enterprise risk VaR numbers.

Timely and Accurate Liquidity Management

Measuring a variety of credit and market stress scenarios and then to be able to look at the liquidity impact of those scenarios.

Timely and Accurate Intraday Credit Risk Management

Understanding when & if deal breaches a tenor bucketed limit before they book it. For FX trading this means that you have about 9 milliseconds to determine if you can do the trade. This is a great place to use in memory technology like Spark/Storm and a Hadoop-based platform.

Timely and Accurate Intraday Market Risk Management

Leveraging Big Data to market risk computations ensures that Banks have a real time idea of any market limit breaches for any of the tenor bucketed market limits.

Reducing Market Data Costs

Market Data providers like Bloomberg, Thomson Reuters, and other smaller agencies typically charge a fee each time data is accessed. With a large firm, both the front office and Risk access this data on an ad-hoc fairly uncontrolled basis. A popular way to save on cost is to negotiate the rights to access the data once and read it many times. The key is that you need a place to put it and that is the Data Lake.

Trade Strategy Development & Backtesting

Big Data is being leveraged to constantly backtest trading strategies and algorithms on large volumes of historical and real-time data. The ability to scale up computations as well as to incorporate real-time streams is key too.

Sentiment-based Trading

Today, large-scale trading groups and desks within them have begun monitoring economic, political news and social media data to identify arbitrage opportunities. For instance, looking for correlations between news in the middle east and using that to gauge the price of crude oil in the futures space. Another example is using weather patterns to gauge demand for electricity in specific regional and local markets with a view to commodities trading. The real-time nature of these sources is information gold. Big Data provides the ability to bring all these sources into one central location and use the gleaned intelligence to drive various downstream activities in trading and private banking.

Market & Trade Surveillance

Regulatory Organizations (SRO) like the FINRA in the US – all of which have dedicated surveillance departments set up to detect any activity that compromises the markets. However, capital markets players on the buy and sell side also need to conduct extensive trade surveillance to report up internally. Pursuant to this goal, the exchanges & the SRO’s monitor transaction data including orders and executed trades & perform deep analysis to look for any kind of abuse and fraud. Big Data shines at this use case as discussed here.

Buy Side (e.g., Wealth Management)

AML Compliance

The Final Word

It is no longer enough for CIOs in this space to think of tactical Big Data projects, they must be thinking around creating platforms and ecosystems around those platforms to be able to activities that generate a much higher rate of return.