MIT researchers resort to flash for Big Data storage

An MIT research team next month will show off a networked system of flash storage devices they say beats relying on DRAM and networked hard disks to handle Big Data demands.

An MIT research team next month will show off a networked system of flash storage devices they say beats relying on DRAM and networked hard disks to handle Big Data demands.

The copious amounts of data now collected for analysation by organisations overtaxes computers' main memory, but linking hard disks across an Ethernet network to solve the problem proves too slow, according to the researchers.

Their Blue Database Machine, or BlueDBM (sounds like an IBM product!), consists of flash devices controlled by serially networked field-programmable gate array chips that can also process data. The researchers say flash systems can find random pieces of information from within large data sets in microseconds, whereas the seek time for hard disks can be more than double that.

The industry is increasingly seeing flash-based technology support products, such as Cisco's latest servers, designed to handle heavy loads.

The current BlueDBM prototype is based on a four-node network, though a 16-node net is in the works in which each node will run at 3GB/sec on a 16TB/sec to 32TB/sec network.

BlueDBM, which will be presented in February at the International Symposium on Field-Programmable Gate Arrays in Monterey, Calif., comes from a team led by Sang-Woo Jun, a graduate student in the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT, fellow CSAIL graduate student Ming Liu, and Arvind (that's his full name), Professor of Electrical Engineering and Computer Science.

"If we're fast enough, if we add the right number of nodes to give us enough bandwidth, we can analyze high-volume scientific data at around 30 frames per second, allowing us to answer user queries at very low latencies, making the system seem real-time," says Lui, in a statement. "That would give us an interactive database."