Transcript of "Big data cloud architecture"

1.
Big Data in the Cloud

2.
S&P Capital IQS&P Capital IQ combines two of ourstrongest brands - S&P, with its longhistory and experience in the financialmarkets and Capital IQ, which is knownamong professionals globally for itscomprehensive company and financialinformation and powerful analyticaltools.

3.
Agenda• Creation of Excel Plug-in with Global Data, Global Sales and US based servers• High Performance data gets for Big Historical Time Series Data.• QA

4.
S&P Capital IQ Excel Plug-in• Excel Plug-in provides thousands of data points on Demand• Allows customers anywhere in the world to use our data assets on their desktops on demand• It needs to be a fast user experience everywhere in the world

5.
Global Customers US Data Center Average Response Time MillisecondsFrom: London To: New Jersey 400From: New York To: New Jersey 30From: Melbourne To: New Jersey 800 Response times rounded

6.
Global Customers Global Data Center Average Response Time MillisecondsFrom: London To: Ireland 400 to 40From: New York To: New Jersey 30 to 30From: Melbourne To: Singapore 800 to 60 Response times rounded

8.
How do we make it even faster? Smart Cache Pre-send data Router- Move data the customers uses the most to their desktop.- Automatically get the data for the customer.- Learn to send the right data to the customer.

11.
Smart Caching Data Lessons Learned• The algorithm works similar to a website matching engine for shopping.• Different in that the customer does not see the recommendations they just have a faster experience• All data sets are used to learn but only large data sets are custom packaged for delivery• Sometimes it is easier to just send the entire package when the data set is small enough and used by the customer.• Don’t expect success day 1 or day 30 the longer you learn the more accurate it should become• Not a replacement for simple logic• Algorithm requires constant feeding and attention.• There are cases where you can’t learn about your user such as when they share ID’s.

13.
High Performance Data Gets• Some data assets due to size are still routed back to the US• Big Data sets ~10T of time series data• As those data assets became more popular we needed to move the right data to the cloud• Cannot synchronize the data so fast loads are required• Single Milliseconds get times

14.
High Performance Data Gets• Using Hadoop learn what are the most used large data assets.• Move the subset of data identified as the most used data to the cloud.• Fast loading of millions of records• Allow for Single Milliseconds data retrieval times

15.
High Performance Data GetsCassandra Hbasehttp://cassandra.apache.org/ http://hbase.apache.org/Apache Cassandra is a highly scalable, eventually consistent, HBase is an open-source, distributed, versioned, column-orienteddistributed, structured key-value store. Cassandra brings together store modeled after Googles Bigtable: A Distributed Storagethe distributed systems technologies from Dynamo and the data System for Structured Data by Chang et al. Just as Bigtablemodel from Googles BigTable. Like Dynamo, Cassandra leverages the distributed data storage provided by the Google Fileis eventually consistent. Like BigTable, Cassandra provides a System, HBase provides Bigtable-like capabilities on top of HadoopColumnFamily-based data model richer than typical key/value and HDFS.systems.Cassandra was open sourced by Facebook in 2008, where it was Hbase is similar to an RDBMS in that it has the concept of tables;designed by Avinash Lakshman (one of the authors of Amazons however, columns in Hbase tables are not fixed in number or dataDynamo) and Prashant Malik ( Facebook Engineer ). In a lot of type and can have any data type which varies from one row to theways you can think of Cassandra as Dynamo 2.0 or a marriage of other.Dynamo and BigTable. Cassandra is in production use at Facebookbut is still under heavy development.We tried to do a similar POC using Cassandra with a smaller subsetof data because of above mentioned hardware restrictions.Unlike Hbase and RDBMS, there is no concept of a table. Insteadwe have columns, column families and Keypsaces.

17.
High Performance Data Gets• Virtual Oracle instances did not meet our performance needs.• EMR needed for Hbase was not cost effective for data gets.• Hbase is difficult to implement in AWS due to the hardware requirements of Hadoop• Cassandra can be segmented logically for Big Data Assets with minimal to no performance degradation in AWS