A while ago I was involved in a project at Clyral where we developed a web based support ticketing system (similar to Zendesk). One of the things we focused on was dealing with the detection of new content in an email conversation. Typically, emails would bounce back and forth between a customer and an agent and we wanted to ensure that our web interface wasn’t cluttered with the full email thread for every reply the customer sent in (the original conversation was always included in outbound emails for convenience).

When we looked at implementing the mechanism which would only extract new content from an inbound email, we found it surprising that there was no real standard for demarcating the beginning/end of an email in an email conversation. It seemed obvious that it would be useful for email clients to stick to a common standard in terms of how they would format existing content when replying to an email, if only to be able to separate the conversation view when viewing an email after a number of exchanges. Off the top of my head, both Outlook and Gmail support this functionality. If you use one of the newer versions of Microsoft Outlook, you’ll notice that the client is capable of detecting the boundaries between emails (take a peak at the screenshot below). The Gmail web client also provides a mechanism to collapse an email conversation into a logical group of messages by detecting new content in an email conversation.

While there didn’t seem to be a standard way of achieving this, we managed to get a decent solution in place for our ticketing system. Since we had control over the format of the outbound email, we could standardise the email format such that we could easily detect new content when a customer responded. This was not a full proof solution, so in the end, we implemented a relatively simple heuristic method that could deal with most of the common mail clients out there. Implementing this feature made it far easier to manage conversations with customers. Obviously, we always kept the original email as customers would sometimes reply with changes to the original email content (such as answering questions inline).

Last year, we started using FogBugz as our general case and project management system. What we discovered was that FogBugz isn’t that smart at managing email conversations. Outbound emails do not include the full email conversation and if you don’t use the web interface to respond, the boundary between the new content and previous communication isn’t detected at all. This usually leads to a very cluttered case view where you need to scroll over copious amounts of duplicate text.

Thankfully, FogBugz has a nifty feature which allows you to customise the front end with JavaScript and CSS. To deal with this problem, I implemented a very simple JavaScript customisation which scans over the content in a case and hides any email text which is superfluous. You always have the capability to toggle the content if you need to inspect it. I’ve included the code for the customisation below. We use Microsoft Outlook (and most of our clients do as well), so the solution works well for us. Replies from Gmail should be supported as well. The code simply scans email for a new line starting with ‘From: ..’ and splits the email there. It’s not rocket science.