Welcome!

OpenTracing Tutorial - Lesson 3

Difficulty:Beginner

Estimated Time:10 minutes

In lessons 1 and 2, we've seen what are the basic concepts used in OpenTracing, what is the Trace Context and how to link spans together, to form one single trace. However, distributed systems are very common nowadays, specially in the shape of microservices, so, having a trace for a single process wouldn't help us much in understanding the big picture on a complex distributed system.

In this lesson, we'll learn how to propagate the context across process boundaries, so that we have one single trace with spans from different microservices.

Steps

OpenTracing Tutorial - Lesson 3

Step1 of 5

Hello-World Microservice App

We'll start this lesson based on where we left the last lesson, plus a small refactoring to avoid duplicating code. We'll also change the formatString and printHello methods to make RPC calls to two downstream services, formatter and publisher.

With all that in place, let's switch to the Java version of the tutorial: cd opentracing-tutorial/java.

To test it out, run the formatter and publisher services in separate terminals. On the first, run ./run.sh lesson03.exercise.Formatter server. Then, click on the "+" sign close to the terminal tab title and open a new terminal. On this new terminal, change to the Java version of the tutorial as well cd opentracing-tutorial/java and run ./run.sh lesson03.exercise.Publisher server.

NOTE: for each new terminal, don't forget to change to the Java version of the tutorial

On a third terminal, execute an HTTP request against the formatter: curl 'http://localhost:8081/format?helloTo=Bryan'

And then, execute and HTTP request against the publisher: curl 'http://localhost:8082/publish?helloStr=hi%20there'

The publisher stdout will show "hi there".

As our client is already instrumented, we need to set the Jaeger's endpoint: export JAEGER_ENDPOINT=http://host01:14268/api/traces

NOTE: for each new terminal, we need to set the env var JAEGER_ENDPOINT if we are running an instrumented server/client

Finally, let's run the client app as we did in the previous lessons: ./run.sh lesson03.exercise.Hello Bryan

We will see the publisher printing the line "Hello, Bryan!".

If we open this trace in the UI, we should see three spans, just like the one we had at the end of lesson 2.

Inter-Process Context Propagation

Since the only change we made in the Hello.java app was to replace two operations with HTTP calls, the tracing story remains the same - we get a trace with three spans, all from hello-world service. But now we have two more microservices participating in the transaction and we want to see them in the trace as well. In order to continue the trace over the process boundaries and RPC calls, we need a way to propagate the span context over the wire. The OpenTracing API provides two functions in the Tracer interface to do that, inject(spanContext, format, carrier) and extract(format, carrier).

The format parameter refers to one of the three standard encodings the OpenTracing API defines:

TEXT_MAP where span context is encoded as a collection of string key-value pairs,

BINARY where span context is encoded as an opaque byte array,

HTTP_HEADERS, which is similar to TEXT_MAP except that the keys must be safe to be used as HTTP headers.

The carrier is an abstraction over the underlying RPC framework. For example, a carrier for TEXT_MAP format is an interface that allows the tracer to write key-value pairs via put(key, value) method, while a carrier for Binary format is simply a ByteBuffer.

The tracing instrumentation uses inject and extract to pass the span context through the RPC calls.

Instrumenting the Client

In the formatString function we already create a child span. In order to pass its context over the HTTP request we need to call tracer.inject before building the HTTP request:

Notice that we also add a couple additional tags to the span with some metadata about the HTTP request, and we mark the span with a span.kind=client tag, as recommended by the OpenTracing Semantic Conventions. There are other tags we could add.

Instrumenting the Servers

Our servers are currently not instrumented for tracing. We need to do the following:

Create an instance of a Tracer, similar to how we did it in Hello.java

The logic here is similar to the client side instrumentation, except that we are using tracer.extract and tagging the span as span.kind=server. Instead of using a dedicated adapter class to convert JAXRS HttpHeaders type into io.opentracing.propagation.TextMap, we are copying the headers to a plain HashMap<String, String> and using a standard adapter TextMapExtractAdapter.

Now change the FormatterResource handler method to use startServerSpan:

Note how all recorded spans show the same trace ID. This is a sign of correct instrumentation. It is also a very useful debugging approach when something is wrong with tracing. A typical error is to miss the context propagation somwehere, either in-process or inter-process, which results in different trace IDs and broken traces.

Debugging Scenarios

Help

Katacoda offerings an Interactive Learning Environment for Developers. This course uses a command line and a pre-configured sandboxed environment for you to use. Below are useful commands when working with the environment.

cd <directory>

Change directory

ls

List directory

echo 'contents' > <file>

Write contents to a file

cat <file>

Output contents of file

Vim

In the case of certain exercises you will be required to edit files or text. The best approach is with Vim. Vim has two different modes, one for entering commands (Command Mode) and the other for entering text (Insert Mode). You need to switch between these two modes based on what you want to do. The basic commands are: