At QCon San Francisco Idit Levine presented "Debugging Containerized Microservices", and outlined the issues of debugging a distributed microservice-based system, and provided three potential approaches to overcoming the inherent challenges. The talk also introduced a new open source microservices debugger that Levine is working on, Squash, which integrates with the VS Code IDE and allows developers to debug microservices applications deployed onto Kubernetes.

Levine, Founder and CEO of solo.io, began the talk by stating that debugging microservices is currently very difficult. A monolithic application typically consists of a single process, and attaching a debugger to this process allows the viewing of the complete applications state and the flow of execution. A microservice-based application is inherently a distributed system consisting of multiple processes communicating over a network, and this adds significant complexity to the challenges of debugging effectively.

The remainder of the talk presented three potential approaches to debugging microservices: distributed tracing, using the open source Squash microservices debugger that Levine has created, and using a service mesh.

Distributed tracing tooling, such as Open Zipkin -- which implements the Open TracingCloud Native Computing Foundation (CNCF) hosted API specification -- can be used to monitor and understand the flow of execution through a microservices-based application. The advantages of this approach are that it is easy to output to any logging tool, even from OSS components; it enables critical path analysis, and developers can drill down into request latency and other associated trace context metadata in very high fidelity; and operators can conduct system topology analysis and identify bottlenecks due to shared or contended resources. Disadvantages to this approach include that tracing does not allow run-time debugging or modification of application state; the approach often requires wrapping/decorating and changing the code, which can also incur a performance penalty at the runtime; and there is no holistic view of the application state - developers can only see what was printed.

The second approach presented was demonstrated via the open source Squash microservices debugger that Levine and here team at solo.io are working on. Squash currently only supports debugging of microservice applications within CS Code that are written in a language that can be debugged by Delve (Golang) or GDB (C++, Objective C, Java etc) and that are deployed to the Kubernetes container orchestration platform -- although more IDE, language and runtime platform support is planned in the future (and community contributions are encouraged).

The Squash architecture consists of a Squash server that is deployed and runs on Kubernetes, which holds the information about the breakpoints for each application, orchestrates and controls the squash clients. The Squash clients deploy as daemon set on Kubernetes node, and the client wraps as docker container, and also contains the binary of the debuggers. Squash uses an IDE as its user interface -- currently only VS Code --- and after installing the Squash extension, Squash commands are available in the IDE command palette.

The third approach presented for debugging microservices was to utilise the capabilities of a service mesh, such as Istio / Envoy. A service mesh data plane, such as Envoy, touches every packet/request in the system, and is responsible for service discovery, health checking, routing, load balancing, authentication/authorization, and observability. The service mesh control plane, such as Istio, provides policy and configuration for all the running data planes in the mesh. These properties provide ideal points of introspection and execution flow control, although outside of Istio's integration with the Open Zipkin and Jaeger distributed tracing systems and centralised logging, currently not many debugging tools exists within this space.

Levine concluded the talk by suggesting that the ultimate solution would be to integrate all three of the debugging approaches mentioned, and encouraged the audience to get involved via the solo.io Slack channel, and contribute to the work she is doing with Squash.

The slides for Idit Levine's talk "Debugging Containerized Microservices" (58Mb PPTX) can be downloaded from the QCon SF website, and the Squash microservice debugger code can be found within the project's GitHub repository. The video for the talk will be made available via InfoQ over the coming months.

InfoQ Weekly Newsletter

Join a community of over 250 K senior developers by signing up for our newsletter. If you are based in the EEA, please contact us so we can provide you with the protections afforded to you under EEA protection laws.

Is your profile up-to-date? Please take a moment to review and update.

Email Address

Note: If updating/changing your email, a validation request will be sent

Company name:

Keep current company name

Update Company name to:

Company role:

Keep current company role

Update company role to:

Company size:

Keep current company Size

Update company size to:

Country/Zone:

Keep current country/zone

Update country/zone to:

State/Province/Region:

Keep current state/province/region

Update state/province/region to:

Subscribe to our newsletter?

Subscribe to our architect newsletter?

Subscribe to our industry email notices?

By subscribing to this email, we may send you content based on your previous topic interests. See our privacy notice for details.

You will be sent an email to validate the new email address. This pop-up will close itself in a few moments.

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.