Designing Features for Mature Systems: Lessons Learned from Manta

Slides: Video:

This video is also available in the GOTO Play video app! Download it to enjoy offline access to our conference videos while on the move.

or

Designing a distributed system (or any system) effectively requires careful design decisions based on tradeoffs within a set of constraints. Making changes to a mature system, whether by adding a new feature or just improving it, constrains the problem further: Changes should not violate existing invariant of the system, and they should work harmoniously in the context of tradeoffs made for the system long ago.

This talk will explore a case study of expanding a distributed system: designing a multipart upload API for Manta, Joyent’s highly-scalable distributed object store. We will talk about the design goals of Manta, then discuss the goals of the new API and why users wanted it. From there, we will design the feature from the ground up, discussing how to achieve the goals of the API -- including idempotent distributed commit and abort operations and performance goals for other operations -- within Manta’s established invariants and design goals.

Distributed systems can be especially difficult to reason about -- especially when making changes to them -- and the lessons learned from adding a multipart upload API to Manta resonate with me for all systems I work on.

GOTO is the enterprise software development conference designed for team leads,
architects, and project management and is organized for developers, by developers.
As software developers and architects ourselves, we wanted to craft the ultimate
conference.

The result is a high quality conference experience where a tremendous amount of
attention and investment has gone into having the best content on the most important
topics presented by the leaders in our community, staged in an intimate environment
needed to support as much learning and networking as possible.