Understanding the behavior of modern Internet systems is becoming increasingly difficult. Networked applications take advantage of distribution, composition, and layering. Distribution refers to applications that reside in multiple places in data centers. Composition is a technique where applications can be built up from smaller, special purpose software components. Lastly, layering is where applications make use of multiple protocols to obtain benefits from each of them.

These factors lead to the following challenges. First, the use of layering makes tracing application behavior hard. While each layer typically has some type of trace and measurement tool, the tools operating on different layers do not communicate with each other. Secondly, the use of composition means that tracing individual requests through large numbers of components is nontrivial. Third, failures at one part of the system can manifest themselves at other parts. Lastly, since network operators are concerned about the privacy of their network information, any type of measurement tool must give operators the ability to control the release of this sensitive information.

To address these challenges, we propose the X-Trace tool, which is a multi-layer, multi-domain, and multi-application trace framework. In our system, the client adds special metadata to application requests. X-Trace enabled network devices propagate this metadata along each layer, as well as down to lower layers. As devices see this metadata, they issue reports, which describe the traffic passing through them. We have built a report collection and aggregation infrastructure that stores the reports generated by requests flowing through the system. Clients can collect these reports and reconstruct the request path from them.

We have retrofitted X-Trace into several protocols, and have deployed it in an open-source photo sharing site. X-Trace is freely available and open-source.