Interesting. Seems like use of specialized hardware for processing of large volumes of data becomes a reality. I think it is a smarter solution than putting together a lot of cheap commodity PCs together like in MapReduce.

It was Netezza’s marketing group that opted for euphony over precision in labeling the block of logic performing decompression the “Compress Engine”. Notice how they carefully avoided calling it the “Compression Engine” or the “Decompression Engine”. The ensuing confusion was entirely predictable.

Netezza has always focused on scan-mostly activities. Thus it makes sense to invest FPGA engineers and gates in decompression. All the more so since decompression is deterministic — the bit stream expresses exactly what to do.

By contrast, compression is a much more challenging activity. Nearly any compression scheme presents redundant encoding opportunities. Sometimes making good choices requires look-ahead or at least back-patching. That sort of stuff is _really_ hard to implement in an FPGA. Netezza’s experience is that with careful implementation software compression can be implemented with minimal impact on CPU utilization.

Finally, let me close with an historical analogy from the world of computer graphics. The earliest graphical workstations had only a bit mapped frame buffer manipulated in a high level language (e.g. Xerox Alto + Smalltalk). The drive for increasing graphics performance lead successively to carefully crafted assembly language (e.g. Mac toolkit), pixel interpolation hardware, and ultimately full graphics pipelines of ever increasing complexity. Yet today does anyone argue against “proprietary” graphics hardware? Could it be that hardware support for processing database fields is as inevitable as hardware support for generating pixels?

Fantastic beat ! I would like to apprentice at the same time as you amend your website, how could i subscribe for
a blog site? The account aided me a applicable deal. I have been tiny bit familiar of this your broadcast provided vibrant clear idea