Kinesis is a novel data placement model for distributed storage systems. It
exemplifies three design principles: structure (division of servers into a few
failure-isolated segments), freedom of choice (freedom to allocate the best
servers to store and retrieve data based on current resource availability), and
scattered distribution (independent, pseudo-random spread of replicas in the
system). These design principles enable storage systems to achieve balanced
utilization of storage and network resources in the presence of incremental
system expansions, failures of single and shared components, and skewed
distributions of data size and popularity. In turn, this ability leads to
significantly reduced resource provisioning costs, good user-perceived response
times, and fast, parallelized recovery from independent and correlated failures.
This paper validates Kinesis through theoretical analysis, simulations, and
experiments on a prototype implementation. Evaluations driven by real-world
traces show that Kinesis can significantly out-perform the widely-used Chain
strategy for placing replicas; Kinesis saves resource requirements by up to 25{\%},
improves end-to-end delay by up to 40{\%}, and recovers from failures orders of
magnitude times faster than Chain in our evaluations.