Small Business Technology News

Big Data Still Is Not Fast Enough

Big Data needs to be faster. Fortunately, new hardware and a new memory-centric distributed storage system may kick things up a notch.

Big Data isn't big news anymore.

It's now standard for smart companies to use Big Data to mine data and develop actionable business insights.

The problem is that Big Data is slower than it needs to be. If a Big Data query takes fifteen minutes, or even hours, to complete, that slow throughput can really limit a company's ability to extract meaningful information from massive datasets.

By unifying storage and memory, 3D Xpoint eliminates data access bottlenecks that have slowed Big Data down to date. The technology was jointly developed by Intel and Micron and is just now getting to market.

But while faster hardware can certainly improve things, the real trick is figuring out how to keep the right data in memory and to get it into memory quickly. This is especially challenging when data is distributed across multiple data centers in different geographies.

Even with better hardware like 3D Xpoint, the ultimate solution to speed up Big Data must include some really great software.

Enter Alluxio

Alluxio (formerly name Tachyon) is an open source project that provides the needed interface between between Big Data applications and the storage systems.

It's being used by big companies like Baidu, Barclays, Alibaba and Rackspace with promising results.

Indeed, Senior Architect Shaoshan Liu of Baidu recently ran some Alluxio tests and told ReadWrite that the "performance was amazing. With Spark SQL alone, it took 100-150 seconds to finish a query; using Alluxio, where data may hit local or remote Alluxio nodes, it took 10-15 seconds."

What It Means for All of Us

Big Data can solve myriad business and societal challenges.

There are thousands of examples of Big Data in action.

One that I just learned about this week is technology that is combing all the data on the Dark Web to find zero day exploits that cybercriminals are using to hack people and organizations.