When In Doubt, Build It Yourself (And Open Source The Code)

This story contains an interview with Peter Bourgon, a developer at SoundCloud.

It's easy to take the basic functionality of a social network for granted, but even something as simple as an "activity feed" (like so many apps have) has complex moving parts. And when a site scales, those parts begin to break down. For engineers at SoundCloud, rebuilding their feed technology was so labor intensive that they decided to spare the rest of humanity from the task. And thus, they opened sourced it.

Here's their solution: It's called Roshi, a new open source distributed storage system for "time series events" in feeds—in laymans terms, news feeds. Roshi was developed by Peter Bourgon and other engineers at SoundCloud in order to scale up without slowing down the performance of the very social streams that keep users engaged.

Let's say you follow Snoop Dogg on SoundCloud. If he reposts (SoundCloud's equivalent of retweeting) a track by an up-and-coming hip hop artist, you'd naturally expect to see the song in your stream. The traditional way of doing this, known as "fan out on write" essentially treats each user's stream (equivalent to the News Feed on Facebook) as an inbox, pushing updates out to each of them individually. It works, but it's inefficient, the storage costs add up, and changes to the social graph can become a pain to implement.

"At some point, those caveats and restrictions started affecting our ability to iterate on the stream," explains Bourgon. "To keep up with product ideas, we needed to address the infrastructure. And rather than tackling each problem in isolation, we thought about changing the model."

SoundCloud's new approach relies on a methodology called "fan-in-on-read." When you view the stream of SoundCloud users you follow, the system will grab the most recent events (favorites, reposts, and the like) of those people and then dynamically merge that information on the fly. It speeds up writes and minimizes storage, but presents new challenges.

For Bourgon and the other developers working on Roshi, the solution came in the form of a specific CRDT (Convergent Replicated Data Type). These conflict-free replicated data types "manage to sidestep a lot of the common problems and pitfalls associated with distributed systems," Bourgon explains.

"One thing that's much easier to do on the new system is to handle social graph updates," says Bourgon. "If you follow or unfollow somebody, in the old system, that could take quite a while to propagate and to become visible on your stream. In the new system, that's more or less immediately apparent."

In the short-term, this streamlines certain backend processes, but it also primes SoundCloud's infrastructure for future product updates. "You could imagine that from there, any new feature that involves dynamically adding or removing content to your stream is now a lot easier to do," says Bourgon.

That will come in handy as the platform continues to grow. Since launching in 2007, the Berlin-based service has become known as a sort of "YouTube for audio" and today sees 12 hours of audio uploaded every minute.

Of course, the SoundCloud team could have crammed any number of off-the-shelf products into this hole and done the trick. The old system, for example, was based on Apache Cassandra, a distributed database system that probably could have been bent to meet SoundCloud's needs in this case as well.

"The problem that we're solving with this thing is so simple to articulate and so simple to get your head around that it felt like the cognitive burden of learning a massive system like Cassandra and operating such a big black box felt like it wasn't worth our time," says Bourgon. "Not when we could relatively easily—Roshi is around 2,000 lines of code—craft something that solves specifically the problem we're trying to solve."

In this instance, crafting a homegrown, hyper-specific solution turned out to be more cost-effective and operationally useful than shoving some generic product into place. Of course, that's not always the case, but Bourgon hopes that projects like Roshi will help bolster the case for building, as opposed to buying, whenever appropriate.

"For me, the biggest thing is the shift in philosophy," Bourgon says. "I think a lot of startups are eager to buy software off the shelf that seems like it might fit and then sort of hammer it into place and solve whatever problem they might be solving. And it's not an invalid strategy. But I think we lose sight of the fact that that's just one option."