Blazing Fast Index Performance with Plasma

In the earlier post, we got an opportunity to introduce Plasma as the new performant storage engine that powers Global Secondary Indexes in Couchbase 5.0(beta out!). We saw some of the features of Plasma and how it is designed with the latest hardware trends in mind so that Index Performance is vastly improved.

We truly believe that Software has to leverage many of the capabilities of the underlying Hardware that it runs on for it to be successful. By re-looking at the problem from scratch, our engineers leveraged lock free data structures, along with numerous other optimisations, and came up with a memory-first design in Plasma. This enables us in offering the best experience to our customers who demand the lowest latencies and the best throughput possible.

Comparing ForestDB with Plasma is not essentially an apples-to-apples comparison as we have completely rewritten the storage engine from scratch and re-architected the entire component. While ForestDB could see only upto 10-20k mutations per second, Plasma sails through with ease even with close to 100k mutations per second; and this is on markedly much larger datasets as well. Having said that, on some of our in-house performance tests, we observe for secondary index scan:

Latencies are lower by more than 3x

Throughput is 8x better

With stale=false (i.e, waiting for all mutations to be processed) : Latencies are 50x lower and Throughput is atleast 120x better

Memory usage and disk usage have been substantially reduced. In-memory workloads(i.e., 100% memory resident) occupy upto 50-60% less memory compared with ForestDB; and disk usage has reduced upto 90% in certain cases. Due to low-write amplification(i.e., number of writes required to persist an update), the lifetime of the SSD device also improves.With such performance numbers, we have opened the floodgates for our customers to take Couchbase Server 5.0 on a spin and validate how it stands in real world battles. We are eager-eyed and waiting to hear your stories.

Performance is not an afterthought in this land!

Note: For our EE customers: If you are using ForestDB currently, during the upgrade to Couchbase Server 5.0, we automatically recreate your indexes in Plasma.

Posted by Venkat Subramanian, Product Manager

Venkat dabbles in product development and product management and has been developing data/analytics platforms & products. Significant chunk of his experience has been with Oracle, where he transitioned from being an Engineer in Oracle’s Enterprise Manager team to Product Manager for Oracle's BI/Analytics suite of products. He has worked in startups in the past helping develop machine-learning/NLP products and distributed decisioning systems. He lurks around at @venkasub.