Learning Transport Agent Pipeline Tracing Hands-on (Part 1)

Alexander Zammit has been developing server applications for over 15 years. Most of his works involve Exchange integrated applications, including a FAX server, a mail security product and anti-spam products.

Pipeline tracing opens a window into the email transport. It uncovers the email modifications carried out by transport agents. Non-Delivery Reports (NDRs) and other Exchange generated emails can also be exposed in this manner. Here we learn how to use pipeline tracing by example.

Included in Exchange 2007 and 2010, pipeline tracing is one of the features available for troubleshooting email transport issues. This gives us multiple dumps of emails as they flow through the various transport agents running on Edge/Hub Transport servers. In this manner we get a snapshot of the email just before and after each of the agents completes processing.

Although not specific to anti-spam, pipeline tracing is especially useful when analyzing how anti-spam and other email hygiene agents are handling and modifying emails.

Something about Agents

Before we start with the tracing examples let's have a quick look at the configuration side. Since pipeline tracing is intended to troubleshoot transport agents it's a good idea to start from the list of installed agents. From the shell run: Get-TransportAgent

This is what I had on my Exchange 2010 test Hub Transport server (Note: This article also applies to Exchange 2007):

To follow this article it is not necessary for you to have exactly the same set of agents. However it would help if you follow the examples on a transport server that has the Exchange anti-spam agents installed. For details on the anti-spam agents check The Exchange 2007 Content Filter Agent. This article is applicable to both Exchange 2007 and 2010.

Configuring Pipeline Tracing

Pipeline tracing is configured and enabled on transport servers using the cmdlet: Set-TransportServer

There are three settings that can be configured. Firstly we specify a sender address, whose emails are to be traced. Effectively this is a filter that is matched against the SMTP envelope sender address (MAIL FROM). If the filter is matched, the email snapshots are dumped to disk. This filter is critically important since tracing all emails could quickly exhaust the available disk resources.

Test 1 and 2: Tracing Non-Delivery Reports NDRs

In this article I will use telnet to submit test emails. This allows us to directly control the envelope sender address. If you need a refresher on how to submit emails using telnet check Telnet Port 25.

Tracing NDRs and other server generated emails is done by setting the PipelineTracingSenderAddress to an empty sender. Here is how I setup pipeline tracing for the HAMRUN transport server: Set-TransportServer HAMRUN -PipelineTracingSenderAddress "<>"Set-TransportServer HAMRUN -PipelineTracingEnabled $True

Configured in this manner, any email with an empty sender will be traced (not just those generated by Exchange). To confirm this submit an email using the command sequence that follows. From the command prompt:

Note: 192.168.0.60 is the IP of the test server HAMRUN. The Exchange email domain is wtest-dom1.local. You will need to replace these with your own values.

Under the pipeline tracing folder you should find a new directory with the snapshots of this email. But before looking at this directory let's try a more interesting email. I will now send an email to an invalid recipient address so as to generate an NDR.

Note: For this to work make sure you don't have Recipient Filtering | 'Block messages sent to recipients that do not exist in the directory' selected. Otherwise the email would be rejected immediately and no NDR is generated.

Note how the MAIL FROM is no longer an empty sender. On my test server there is no mailbox with the address WrongAddress@wtest-dom1.local. So Exchange will respond to this with an NDR that will be caught by pipeline tracing.

This is what my tracing destination directory looked like:

If you open any of the eml files you will see the NDR itself:

The multiple copies of the same email are what this tracing functionality is all about. It shows how the email change as it starts and finishes processing for each of the transport agents. Very often the differences are minimal. In other cases the changes are more substantial. It all depends on what changes the transport agents perform for that particular email. We will take a closer look at these EML files in the second part of this article.

Final Tips

Today we introduced transport agent pipeline tracing. We saw how this is configured and produced our first traces. In today's examples we captured an NDR showing how this functionality can be used to sniff emails generated by Exchange itself. In the second part we will immediately start with more examples, applying pipeline tracing to agent troubleshooting. We will also look at the snapshots and gain a better understanding of the information contained in the trace output.