Answered by:

Outlook 2013 behavior as an IMAP client

Question

Be aware of this thread over in Office IT Pro:
http://social.technet.microsoft.com/Forums/en-US/officeitpro/thread/e8122c0c-7abd-4d1d-957b-cec5f5a7167f/

In short, when Outlook 2013 is used as an IMAP client, it *removes* X- mail headers while moving messages between IMAP folders. It also rewrites a header to identify Outlook as the
X-Mailer. Both of these are extremely undesirable behaviors, may well be against RFCs, and should be corrected post haste.

I would like to summarize what we have discussed on this thread. Before doing so, it’s important for me to state more clearly the purpose of this forum. We discuss both documentation (specification and standards) and implementation
here. However, our purpose in doing so, is always to accurately document the current protocols and formats used as well as the current product behavior. It is not the purpose of this forum to change the product behavior. We do report product behavior
deviations and problems but, unless clearly or explicitly documented, usually leave it to the product support channels to follow up with the product development teams for decisions on design and implementation changes.

Having said that, here are the current states of the topics we’ve discussed:

When Outlook shuffles the trace header fields, it is a violation of the RFC, which states that these header fields MUST NOT be reordered. This anomaly is not documented. There is a report on this, the current plan is that this
behavior will be documented.

The RFC states “header fields SHOULD NOT be reordered when a message is transported or transformed”, the same paragraph explicitly explains “It is important to note that the header fields are not guaranteed to be in a particular
order. They may appear in any order”. Outlook does not keep the original order; there is a report on this, the current plan is that this behavior will be documented.

The non-standard header fields, e.g. “Delivered-To”, are not preserved. There is no documentation that prescribes what to do with the non-standard header fields, specifically, not starting with “X-“, the assumption on our part
(which we understand you do not share) is that the behavior to keep them or delete them is equally valid. Outlook deletes them, the current plan is that this behavior will be documented.

If you want to change any of the above behavior, please send an email to dochelp and indicate it is for me and I will connect you with the product support team for Outlook to continue the conversation.

I reproduced your scenario using Exchange and Outlook 2013, the
X-Mailer: Microsoft Outlook 15.0
remained unchanged. In order to investigate the case you described, I would like to see an unencrypted network capture starting from the session establishment and see how the header, more exactly the ‘X-Mailer’ has been changed. Be sure that the capture doesn’t
contain any confidential information; the capture will probably be small enough so you can send it as attachment to ‘dochelp (at) microsoft (dot) com’ and in the e-mail indicate that it is for me.

I have verified the source issue reported in the Office IT Pro. The repro path requires a non-Outlook client, an IMAP server (can be Exchange), and Outlook 2013:

1] Using the non-Outlook client, send a message to an address on the IMAP server.
2] Setup Outlook 2013 to use the IMAP server, *as* an IMAP client. (The bug is in Outlook 2013's IMAP code.)
3] Using Outlook 2013, view the message's Internet Headers.
4] Copy the Internet Headers to a text file for comparison.
5] Using Outlook 2013, MOVE the message to another folder on the IMAP server.
6] Once again, view the Internet Headers. Save this set to another text file.

The headers captured in [4] and [6] are dramatically different. The move operation in [5] *removes* all the X- and Received headers, and rewrites the X-Mailer header to read "Outlook 15.0". This is the erroneous behavior.

In my test just now, the original message was sent with Apple Mail on an OS X computer. The header as viewed with Outlook 2013 is:
X-Mailer: Apple Mail (2.1508)

With Outlook 2013 (connected via IMAP to the mailbox), I moved the message to a folder on the same account. On viewing the message in the destination folder, the Received and X-headers are *gone*. The X-Mailer header now reads:
X-Mailer: Microsoft Outlook 15.0

I will email these header results into dochelp. In the meantime, please repro this case. I think you'll find the same results.

Very good. I can confirm this is a change in behavior from Outlook 2010 to Outlook 2013. I depend on this to help fight spam. When I move messages to the junk folder, I now lose all history ("Received" headers). Really a problem!

Has there been any progress made on this issue? Users of Outlook 2013 have this problem with our IMAP server (emailtopia Response Manager) and we are anxious for a solution. Hoping a fix has made it into the imminent Office 2013 SP1.

RFC3501 doesn’t specify whether the experimental, user-defined headers should or shouldn’t be preserved, changed, or deleted when a message is processed by the client, e.g. moved to another folder. Outlook 2013 changed the behavior from its previous version
regarding the experimental headers. In a future release of Outlook the experimental X- mail headers will be preserved; it is not decided, yet, that the changed behavior will be back-ported or not.

Why are you talking about X- mail headers? OL2K13 kills all headers, not only X- headers! Maybe, RFC3501 does not talk about X- headers, but show me one client who is replacing all headers during a move to another folder. I don't think that the current behavior
is wanted by design, it's just a bug!

Unfortunately the newly released SP1 does not fix the mail header issue, so I am sure for now that this won't be fixed anymore...

RFC3501 doesn’t specify whether the experimental, user-defined headers should or shouldn’t be preserved, changed, or deleted when a message is processed by the client, e.g. moved to another folder. Outlook 2013 changed the behavior from its previous version
regarding the experimental headers. In a future release of Outlook the experimental X- mail headers will be preserved; it is not decided, yet, that the changed behavior will be back-ported or not.

Thanks, Vilmos

Hi Vilmos!

Does that change in client behaviour also include no longer rewriting the X-Mailer from its original value to Outlook 15.0?

Beyond the application of third party tools using X-* mailheaders, changing and/or removing "Received" as well as "X-Mailer" etc headers does make tracing back client/server issues significantly harder. (When using an MTA somewhere other
than Exchange; yes, it happens; yes, it may not be what you wish for. We live in an imperfect world.)

I am aware, that
COPY in RFC 3501 does not specify how headers are specifically handled.
For some reason, to date all common email clients left this information untouched and preserved it.

If you contend that "should" and "preserve flags and internal date" only require you to preserve exactly that, I wonder when a new feature will arise to automatically rewrite "Subject" and "To"/"From" as
well, given that they are not part of the message body either.

Please do help us out here: what's your rationale behind throwing away (and rewriting, as with X-Mailer) all data in the header except for a few, that until Outlook 2013 everyone else writing IMAP clients thought worthy enough to preserve?

RFC3501 doesn’t specify whether the experimental, user-defined headers should or shouldn’t be preserved, changed, or deleted when a message is processed by the client, e.g. moved to another folder. Outlook 2013 changed the behavior from its previous version
regarding the experimental headers. In a future release of Outlook the experimental X- mail headers will be preserved; it is not decided, yet, that the changed behavior will be back-ported or not.

Thanks, Vilmos

Hello Vilmos,

The behavior suggests that Outlook 2013 accomplishes the move task with the APPEND command, not COPY. With COPY the task is executed at the server, optimizing bandwidth; APPEND is a client-task, effectively uploading a *new* copy of the message to
the server.

If this is the case, the issue is two-fold:
[1] APPEND is a wasteful use of client resources and network bandwidth.
[2] OL2013's APPEND omits/rewrites prior headers, destroying valuable history.

Like others who have replied, we strongly urge you to reconsider the switch from using COPY to using APPEND, but as long as you are doing it, please keep the message being copied intact. As has been pointed out, it's not just "X-" headers that are being
lost. I can't see this as anything other than a fairly serious bug and hope to see the fix for it "back-ported" to 2013 in a upcoming service pack.

Please give a list of the headers, which were removed; I’ll check whether the behavior of Outlook 2013 is consistent with the normative provisions defined by RFC3501 and [MS-STANOIMAP]. If you could point to the specific clauses in those documents that specify
retaining user-defined (“X-“) headers when moving a message from one folder to another, it would be helpful. I wrote in my previous posting that the current behavior will be changed in a future release of Outlook.

Hi Ben,

The details of the headers preservation change are not finalized, yet, the X-Mailer will be preserved as it is. I would not comment on the implementation of the protocol, the purpose of this forum is to discuss the protocol itself, see my closing paragraph.

Hi Andrew,

As I wrote before on this forum we discuss the protocol, not its effectiveness. The header handling was discussed before. Please see my closing paragraph below.

Hi Mike,

My answer to you is the same as to Robert, see above.

---------------

Summary of this thread: Outlook 2013 and its previous versions comply to RFC3501 and its companion [MS-STANOIMAPI] even they behave differently regarding header handling; it is possible because of the too concise wording of RFC3501. Because multiple posters
asked for the preservation of the information in the headers I filed a bug against Outlook. The behavior change suggestion in the bug was honored and a future release of Outlook will contain it. It is not decided, yet, that the changed behavior will be back-ported
or not, I'll post it when the information is available. For the other, not answered topics thehttp://social.msdn.microsoft.com/Forums/exchange/en-US/home?forum=outlookdev
forum looks more appropriate.

We currently have no plans to back port the mentioned change in behavior to Outlook 2013. If you would like to pursue this further, please contact Outlook product support as this
is a product request and not specific to the Open Specifications document set. The product support channel for this would be a direct engagement with Microsoft Office Support through either
Premier Support or
Standard Support .

I guess there is a good thing Thunderbird is out there. It's sad that something which has "worked" in all previous versions of Outlook and now doesn't work is not considered a bug. I feel like I've spent bad money.

BTW, removing headers makes it hard to track SPAM origins and analyze why a message might have been properly labeled or mislabeled as Junk Email. This means that Outlook 2013 is actually participating the global proliferation of SPAM by covering
up attempts to stop it.

Never thought that this would happen after reporting this BUG on 18 Dec 2012(!!) for the very first time: http://social.technet.microsoft.com/Forums/en-US/officeitpro/thread/e8122c0c-7abd-4d1d-957b-cec5f5a7167f/

I appreciate your efforts and would like to think I understand your motives,
I'd like to follow up on this issue as well.

First off, I understand that re-serialization of object models/data (which appears to happen in Outlook 2013 as the full stack of headers is incorporated in some kind of data model) causes the result to be in the order the internal model saves data... which
certainly doesn't have to be in the same sequence as the original information.

This may not be an issue easily worked around without too great an effort, especially given the performance and maintenance considerations the Outlook development team surely has to adhere to.

All in all, however, I'd like to thank you for not only considering but implementing the original request!
It definitely made a difference for me: custom solutions that use header information are possible again, even if my clients use Outlook 2013.

Now my two questions, for each of you:

@Robert: would you mind providing a detailed example of a full email header before and after Outlook 2013 touched it? I'd greatly appreciate it (especially if it includes more than two "Received: " lines.)

@Vilmos: if your time permits it, would you mind pointing me to the .msp or KB entry that includes this updated functionality?
Thanks!

Beyond that:

Any more precise header preservation feature request might be better suited for a new discussion, pointing out the exact details that we (Robert?) need. Without such details, guessing what the customer wants is always a very hard - if not impossible - task
to achieve. After all, we've come very far and I think this has been one of my best experiences in terms of "they finally fixed something I wouldn't have believed they ever would" at Microsoft.

for the last months I tested after each Patch-Day (2nd Tuesday every month) whether the header-issues are resolved, moving one mail from my inbox to another imap-folder. Every time all headers were lost, only some Outlook internal headers were present. However,
after the last Patch-Day, 12 August 2014, I noticed that the headers are preserved. Unfortunaley, not in the correct order as they were in my inbox, i.e. the Return-Path isn't any longer at the top, but somewhere in the midst of the headers, the Received-Headers
are not in the correct order, but upside down, etc.

So, it isn't very useful for me, as my Outlook Add-In expects the headers to be 'untouched' like before in Outlook 2007 and Outlook 2010.

I also searched the KB entries of the last Patch-Day, but didn't find any notice about the header 'fix'.

to Vilmos: PLEASE inform MS that the recent 'fix' needs another fix regarding the correct order of the headers.

Robert, Benjamin basically answered your posting. The subject of this thread is the preservation of the information in the header, this is done. As Benjamin suggested, please provide
an example and describe what order changes have occurred and I’ll check whether is any violation of the documented behavior; e.g. if “Received:” is before “Return-Path:”, it is a problem, but the order change of the “Received:”s is not.

This is great news. I hope the issues around this fix are worked out. As far as preserving the order of headers, that should not be difficult. In Java the LinkedHashMap gives you the functionality of speed and preserving iteration order. Without this class
a HashMap (dictionary in .net terms?) and a LinkedList working together should work. As this would just require allocating a couple of pointers per header line, there should be very little overhead.

Please consider this. Header order is also important. Spam reports should stay in the order they were added. This way if two SPAM engines process a message, its possible to see how each one functioned. It is common for two or even
three Spam engines to see a message. There would be a Spam engine at the sender. Another at the relay host. And a final one at the receiving server. If you also want to count Outlook's own Spam engine there would be a fourth.

As you can see, Outlook is still screwing up the headers once a piece of email is marked as Junk. All of the Spam Assassin headers are gone. Can't see why this passed (or failed for that
matter.) Tracing it back to the source server is pointless, because Microsoft has taken the liberty of
erasing that information from the email. I really wish I didn't waste my money on Outlook 2013. It's basically a downgrade from all previous versions.
The really sad thing is that there are claims on this board that this *very* unwanted behavior has been addressed. We'll be using something else. Don't expect me to every update my Office version again.

As soon as you mark email as junk, there goes all of the historical information, and the ability to trace it back to its source.

As someone wanted a proof of scrambled headers with the so-called 'fix' installed, I have taken one mail in my inbox (headers intact) and moved it to another IMAP-Folder (headers scrambled after move).

Robert, Benjamin basically answered your posting. The subject of this thread is the preservation of the information in the header, this is done. As Benjamin suggested, please provide
an example and describe what order changes have occurred and I’ll check whether is any violation of the documented behavior; e.g. if “Received:” is before “Return-Path:”, it is a problem, but the order change of the “Received:”s is not.

Thanks, Vilmos

Hi Vilmos,

please look at my posting above to see my provided information about header-corruption.

May I ask, WHY the change of the 'Received:' order is OK for you? It's definitely not OK for me and I suppose not OK for 99.99% of the rest of us!

The Return-Path is now somewhere in the midst of the headers!

Please remember that the Headers should be left UNTOUCHED - Outlook 2007 and 2010 are working 100% correctly! Why is it so difficult to get OL 2013 working in the same manner? You have the source-code of OL2007/2010, so please look at it and fix OL2013 once
and for all -> PLEASE!!

This is happening on any move out of the inbox.WHOOPS!!! Just noticed I'm not on the right version of Outlook 2013, but might as well document the Version: 15.0.4631.1004
behavior. Will retest after the download.

Outlook 2013 should never alter an email. If I copy it then it should make an exact copy. If I move it, them it should make an exact move. The move operation on a Linux/Postfix setup is you cut the text data from one file and you insert it
into another file. There is no magic to it; therefore, no magic should be performed.

A carefully crafted bash script can do the job and do a better job than Outlook 2013. Outlook 2013 should never alter the content of the message. If it really needs to add an header that would be okay, but anything beyond that is just inconsiderate.
The email does not belong to Outlook or Microsoft. So Outlook nor Microsoft shouldn't just go around changing things.

It is very possible, to almost a 100% certainty with smart phones and tablets, that multiple email clients will access an email message. Outlook 2013 needs to be considerate of that. Right now I can use Outlook 2007, Outlook 2010, Windows Mobile
5, Windows Mobile 6, Thunderbird, the email client for PalmOS, the email client for Android, Squirrel Mail, Round Cube, Ximian Evolution, versions of Outlook Express, or even the Java Mail API, and not a single one of them would take it upon itself to
start deleting headers. I can even use them all at the same time without causing much in the way of problems. I wouldn't attempt to do concurrent moves, but one use at a time would be safe.

None of them basically attempt to take control of the IMAP account. Why does Outlook 2013 basically think the IMAP account belongs to it? I can tell you right now, it doesn't. It makes Outlook 2013 a bad guest. It's the type of guest who
would come over and throw out your dishes because it doesn't like the color. I understand if you need to leave behind a little "trash" like your own email header. Just do it like a good guest would: Put it in line with everything else, and don't
mess with anything that is not yours without permission.

I’m sorry that the update to Outlook 2013 to retain “X-“ headers has not resolved all of this community’s concerns. Based on the comments in this thread we have broken the remaining
concerns into two groups:

The ordering of header fields is not retained when a message is moved from one IMAP folder to another IMAP folder in Outlook. RFC 2822, section 3.6 states that “It is important
to note that the header fields are not guaranteed to be in a particular order.” However, that paragraph goes on to state that “the trace header fields and resent header fields MUST NOT be reordered”. Microsoft’s implementation of moving messages between IMAP
folders in Outlook 2013 does not follow this requirement, and we will update the implementation notes in [MS-STANOIMAP] to describe Outlook 2013’s behavior of returning the headers in an arbitrary order.

The non-standard headers, e.g. “Delivered-To”, are not preserved when moving a message
between IMAP folders. The recent fix was written to preserve all non-standard header fields that use the method described in RFC 822 for naming user-defined fields, specifically, by using the string “X-“ at the beginning of the header name. As the "Delivered-To"
header is not a standard header, interpretation and retention of that field is left up to each implementer. Microsoft’s implementation in Outlook 2013 does not retain non-standard fields that do not begin with “X-“.

If you wish to discuss changing the behavior of Outlook with regards to either of these two items, please send an email to "dochelp (at) microsoft (dot) com" and indicate it
is for me and I will connect you with the product support team for Outlook to continue the conversation.

I have asked several times WHY Outlook 2013 alters the Mail-Headers, but did not get an answer yet. There must be a reason behind this! So, WHY? OL2K7 and OL2K10 work perfectly, so why not OL2K13?

I am the Chief Software Architect of 'f Mail Server', a SMTP Anti-Spam Server, written completely in C#. I have worked on E-Mail related stuff for nearly two decades and I 've never encountered such a strange behavior like your client acts on mail moves
between IMAP-Folders.

I found this BUG, yes it is a bug (no need for discussion), on 18 Dec 2012 (!!) and reported this in: http://social.technet.microsoft.com/Forums/en-US/officeitpro/thread/e8122c0c-7abd-4d1d-957b-cec5f5a7167f/

In 3 months we are talking about this for 2 years! I've neither got an answer WHY this has been implemented as MS did, nor why there is need for discussion to NOT get it fixed. Please explain WHY such a trivial thing gets delayed in endless discussions and
does not get fixed.

Regarding the 'Delivered-To:' header: I agree 100% with
Anonymous6314613 that Mail-Headers must stay UNTOUCHED! It's bad practice to say: it's not described in the RFC, so we can do what we want. As
Anonymous6314613 said: EVERY other Mail-Client (even OL2K7 and OL2K10) work 'correctly', but not OL2K13.

So please enlighten us, WHY you (or the Outlook Dev.-Team) 'refuse' to correct OL2K13 to work like OL2K7/10?

Thank you Vilmos. I hope your manager is aware of the hard work you've put in here for us.

Will be emailing you.

Is there a reason the "move" operation isn't done with server side commands? What if I move a 10 MB email over a slow link? Why not let the email server do it's job? Why should
I have to risk paying a data tariff? Consider I might be on an airplane.

I strongly urge your team to use this:

http://tools.ietf.org/html/rfc3501#section-6.4.7

BTW, RFC 2822 is obsoleted by http://tools.ietf.org/html/rfc5322. However there are other RFCs which need to be considered: http://tools.ietf.org/html/rfc6376#section-3.5, http://tools.ietf.org/html/rfc4408#section-7, http://tools.ietf.org/html/rfc4021

I'm sure there more. There will be more. One day there might even be a header tagged in there to track whether or not "postage" was paid. You can never guess. So please, please, just COPY the message.

I would like to summarize what we have discussed on this thread. Before doing so, it’s important for me to state more clearly the purpose of this forum. We discuss both documentation (specification and standards) and implementation
here. However, our purpose in doing so, is always to accurately document the current protocols and formats used as well as the current product behavior. It is not the purpose of this forum to change the product behavior. We do report product behavior
deviations and problems but, unless clearly or explicitly documented, usually leave it to the product support channels to follow up with the product development teams for decisions on design and implementation changes.

Having said that, here are the current states of the topics we’ve discussed:

When Outlook shuffles the trace header fields, it is a violation of the RFC, which states that these header fields MUST NOT be reordered. This anomaly is not documented. There is a report on this, the current plan is that this
behavior will be documented.

The RFC states “header fields SHOULD NOT be reordered when a message is transported or transformed”, the same paragraph explicitly explains “It is important to note that the header fields are not guaranteed to be in a particular
order. They may appear in any order”. Outlook does not keep the original order; there is a report on this, the current plan is that this behavior will be documented.

The non-standard header fields, e.g. “Delivered-To”, are not preserved. There is no documentation that prescribes what to do with the non-standard header fields, specifically, not starting with “X-“, the assumption on our part
(which we understand you do not share) is that the behavior to keep them or delete them is equally valid. Outlook deletes them, the current plan is that this behavior will be documented.

If you want to change any of the above behavior, please send an email to dochelp and indicate it is for me and I will connect you with the product support team for Outlook to continue the conversation.

I understand your frustration.As I mentioned previously, I would be more than happy to connect you to one of our Outlook specialists to continue the conversation around these issues that you would like to see addressed.Others have taken advantage of this offer. Unfortunately, this forum is not the best channel to pursue these behavior changes.

I know this is an old thread and it doesn't seem to have arrived at any kind of satisfactory conclusion, but I'm just wondering if you ever got an answer or figured out a resolution to the issue.

I am currently seeing the same behavior in Outlook 2016 and this is the only thread I've come across where someone is actually clearly expressing the root of the problem. I'd appreciate any additional insight if you've made any headway on this problem over
the past two years. Unfortunately, it doesn't seem like a priority to MS.

I've only just (as far as I remember ... I can forget things / rediscover over the years) come across this issue.

It's wasted several hours ... as I run my own hMailServer installation and when trying to trace an incoming DKIM issue where the sender used two DKIM signatures - the first not working because of no public records, and the second passing ... but effectively
tripping my anti-spam, I was bewildered by not being able to find the DKIM signatures in the email ... not even on the eml file in the hMailServer data directory.

I'd also tried registering to a service (the sender) using my Gmail address ... from that I could see there were two DKIM signatures.

I thought it must be a problem with hMailServer!!! With DKIM Spam detection on, both signatures disappeared. With DKIM Spam detection off, one signature disappeared. Hours and hours I wasted trying to figure out this puzzle - including
emailing the sender with copies of the header. I was slow on the uptake even with the x-mailer = there ... simply because it was such an impossible thing to grasp that Outlook would behave in this way. I mean WHY? WHY MicroSoft? It
doesn't make sense. Is there an actual reason? Or is it a screw-up?