Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

10 SRE’s, with a tech-lead Sponsored by a SRE Director Input from Software Engineers on development

Founded in 2002, LinkedIn has grown into the world’s largest professional social media network 30 offices in 24 countries, Available in 24 languages More than 450+ million members worldwide

LinkedIn started as an Oracle shop To-date, we still run a significant number of Oracle databases Oracle is fine for writes, scaling reads becomes challenging HyperGrowth == Scaling challenges Scaling writes isn’t a common problem in most cases Scaling reads to 100k+ QPS, is challenging Failures in read-scaling infra can take down back-end systems Applicable use-cases Simple cache store Pre-warmed Read-through SoT Store

Until 2012, we were only using Memcache as a non SoT In-Memory store Drawbacks of memcache: Difficult to pre-warm, not easy to copy-data No native sharding for clusters, had to write our own Restarting memcache servers caused problems Couldn’t copy data across for new DC’s, expanding clusters etc Mid-2012, started testing Couchbase

Evaluated replacement systems for Memcached: Mongo, Redis, and others Couchbase had distinct advantages Simple replacement for memcache  JAVA Spring made this simpler Built-in replication and cluster expansion, significantly reduces ops-workload Automatic partitioning, doesn’t become a concern anymore Low-latency, reads from disk are still very fast Async write to disk, can write a low of data at once without it being a problem Lots of API’s that make tooling relatively simple

Insert fuse architecture

We have a deduplication filter in stork that you can take advantage of to make sure we don't send duplicates of your email. This is highly recommended for any email using kafka (kafka can potentially deliver your email to our system twice)

Don’t use as SoT store as Espresso is our primary key-value store

Couchbase Connect 2016

1.
Michael Kehoe
Staff Site Reliability Engineer
LinkedIn
Going all in:
From single use-case to many

5.
$ whatis CBVT
5
Michael Kehoe
• Couchbase Virtual Team
• ~10 SRE’s
• 2 Software Engineers
• Sponsored by SRE Director
• 5-90% of their time to support Couchbase
• Encourage as many people to contribute as possible
• What do we do?
• Operational work on Couchbase clusters
• Evangelize the use of Couchbase within LinkedIn
• Develop tools for the Couchbase Ecosystem

6.
6
The LinkedIn Story
• Founded in 2002, LinkedIn has grown into the world’s largest professional social
media network
• 30 offices in 24 countries, Available in 24 languages
• More than 450+ million members worldwide

24.
Management
24
Operational Tooling
• We want to see a world-view of all the clusters we run
• Having bucket cluster/server level statistics is useful
• Having a global view of who owns and operates each cluster/ bucket is useful

26.
26
Conclusions
• Couchbase was a natural fit into our existing infrastructure
• Building an ecosystem around Couchbase was important to us and has helped
Couchbase be successful at LinkedIn
• Expanding use of Couchbase
• In the past year we’ve grown the number of buckets over 50%
• Starting to use Views in production
• Moving Couchbase into LinkedIn standard deployment infrastructure