iOS 6.1 hammering Exchange, dragging down server performance.

iOS 6.1 devices are hammering Exchange servers with excessive traffic, causing performance slowdowns that led Microsoft to suggest a drastic fix for the most severe cases: throttle traffic from iOS 6.1 users or block them completely.

"When a user syncs a mailbox by using an iOS 6.1-based device, Microsoft Exchange Server 2010 Client Access server (CAS) and Mailbox (MBX) server resources are consumed, log growth becomes excessive, memory and CPU use may increase significantly, and server performance is affected," Microsoft wrote on Tuesday in a support document.

The problem also affects Exchange Online in Microsoft's Office 365 cloud service. Office 365 customers may get an error message on iOS 6.1 devices stating "Cannot Get Mail: The connection to the server failed." The Microsoft support article says both Apple and Microsoft are investigating the problem.

Microsoft suggests several fixes, starting out gently, then escalating to the complete blockage of iOS 6.1 devices. Based on the fixes suggested, the problems may be caused when iOS devices connect to Exchange calendars.

The first workaround is "do not process Calendar items such as meeting requests on iOS 6.1 devices. Also, immediately restart the iOS 6.1 device."

If that doesn't work, users are instructed to remove their Exchange accounts from their phones or tablets while the Exchange Server administrator runs a "remove device" command on the server side. After 30 minutes, users can add the Exchange accounts back onto their devices but should be advised "not to process Calendar items on the device."

If that doesn't work, the fixes get more serious. The next method is for the server administrator to create a custom throttling policy limiting the number of transactions iOS 6.1 users can make with the server. "The throttling policy will reduce the effect of the issue on server resources," Microsoft notes. "However, users who receive the error should immediately restart their devices and stop additional processing of Calendar items."

One Exchange administrator who created a throttling policy through PowerShell to solve the problem provides a guide here, but Microsoft also has a page providing instructions.

Finally, the last method Microsoft recommends is to block iOS 6.1 users. "You can block iOS 6.1 users by using the Exchange Server 2010 Allow/Block/Quarantine feature," Microsoft notes. (See this post for more detailed instructions.)

Businesses of all sizes limiting or blocking iOS devices

We don't know exactly how widespread this problem is. It's clearly not affecting everyone, but the impact seems to run the gamut from small businesses to large.

It turned out that the 300GB VMware virtual machine hosting the Exchange server was full. "You can imagine our surprise when that VM filled up overnight," Ray said. "If we were running Exchange in a typical hardware-based server with a 1TB drive, it would have taken us a week to realize the problem."

How did it happen, and how did the company get things working "normally" again? "The transaction log had 200,000 records and was the indication of a problem," Ray said. "Our temporary solution has been to ask iOS users to switch to manual pull rather than ActiveSync push. For heavy e-mail users, we are recommending an automatic pull every 30 minutes. So far, that seems to have kept Exchange happy with no other issues since last week. Let's hope that Apple and Microsoft put their heads to together and fix this soon."

We heard from several other people on Twitter that they have been bit by the iOS 6.1/Exchange problem. One said, "My 22,000+ employee enterprise has blocked iOS 6.1, execs all have iOS."

A support thread on Microsoft's Exchange Server site was opened January 31 to discuss the excessive logging caused by iOS 6.1. The server administrator who began the thread identified an iPad that "caused over 50GB worth of logs" on a single database.

The thread got more than a dozen replies. One Exchange administrator explained that "malformed meetings on a device cause the device to get into a sync loop which causes excessive transaction log growth on the Exchange mailbox servers." This in turn "will cause Exchange performance issues and potentially transaction log drives to run out of disk space which would then bring down Exchange."

To solve the problem, this admin simply "disabled all iOS 6.1 on our Exchange system."

iOS 6.1 was released on January 28. iOS 6.1.1 came out a couple of days ago, but for now it can only be installed on the iPhone 4S and is designed to fix cellular performance and reliability. Apple didn't mention anything about Exchange fixes when releasing this latest version. Last year, iOS 6.0.1 fixed an Exchange problem that could lead to entire meetings being canceled when even a single iOS user declined a meeting invitation.

The iOS 6.1 problem isn't the first time iOS has caused Exchange servers to perform poorly. An Apple support article from 2010 describes sync problems in iOS 4 and says, "Exchange Server administrators may notice their servers running slowly." At the time, Microsoft noted iOS 4 led to "Exchange administrators... seeing heavier than normal loads on their servers from users with iOS devices." Microsoft got in touch with Apple to fix that problem.

We've asked both Apple and Microsoft how many users are impacted by the latest problem, and when a more permanent fix is coming. We also asked Apple if it agrees with the workarounds suggested by Microsoft. Microsoft told us it has nothing else to say, as the "support article contains the latest." Apple has not responded to our request for comment as of yet.

UPDATE: Apple posted a support document of its own today, describing the problem thusly:

When you respond to an exception to a recurring calendar event with a Microsoft Exchange account on a device running iOS 6.1, the device may begin to generate excessive communication with Microsoft Exchange Server. You may notice increased network activity or reduced battery life on the iOS device. This extra network activity will be shown in the logs on Exchange Server and it may lead to the server blocking the iOS device. This can occur with iOS 6.1 and Microsoft Exchange 2010 SP1 or later, or Microsoft Exchange Online (Office365).

Apple's suggested fix is to turn the Exchange calendar off and back on again within the iPhone's settings. An operating system update to fix the problem is on the way. "Apple has identified a fix and will make it available in an upcoming software update," Apple said.

Promoted Comments

Full disclosure, I work for MS in the Exchange product group. I cannot go into details about this iOS condition beside what has been posted publically by each company at this time, but that Apple has posted their support article stating they will be issuing an iOS fix. http://support.apple.com/kb/TS4532

Exchange transaction logs contain the transactions made within the Exchagne databases themselves. You can't open a transaction log and read the data in text form as they basically contain instructions to the Jet database similar to "Change page <10000> to contain <XYZ>" and nothing more. If you get a client with a runaway process it can end up with endless T-Logs being created. Thankfully there are mechanisms for customers to detect these conditions taking place as well as ways to block access to such devices when this kind of thing happens.

At a high level a transaction would go...

1. A client makes some kind of change to an object (e.g. marks a message as read).

2. Exchange makes that change in memory first.

3. Exchange logs that change to a transaction log file second.

4. Exchange commits the change to the database third at some point later (it isn't very long, but it isn't instant either).

5. <Optional> These transactions may be shipped either in memory or via log file to another copy of the database.

The change (or transaction) is always written to the t-log BEFORE the database so there is a record of the transaction in case the database file is ever lost in the future before it is backed up. This helps maintain the four pillars of being an ACID database. This old Exchange article talks about it if you search for ACID. http://support.microsoft.com/kb/271987

The only time transaction logs can be safely flushed is automatically if Circular Logging is enabled, or after a Full or Incremenal backup that integrates with the Exchange backup API notifying the database engine the files have been protected.

I am in charge of a 10,000 seat Exchange 2010 SP2 environment and have been doing tons of analysis on this problem the last few days. Hopefully I can shed some light on exactly how this is hurting Exchange, since I see a lot of speculation and incorrect info being spread around.

The problem is twofold.

First, affected iOS 6.1 devices are hammering CAS with rapid, repeated attempts to synchronize the same data, thereby increasing CPU and memory utilization on the CAS. How rapid and repeated? In my environment the average EAS device syncs roughly 600-800 times per day. Affected iOS 6.1 devices are synchronizing anywhere from 50,000 to 80,000 times per day... per device, if the user has multiple devices. Yeah, ouch.

Second, it's not just a DoS on the CAS... the transactions are actually making it into the store, hence transaction log and database growth! The data is not being stored in the user's normal folders, though, so the user's quota doesn't stop the data growth as you might expect. The data growth is occurring in the Recoverable Items folder (formerly known as the dumpster) which does NOT adhere to the users normal quota. In my worst case mailbox, the user has 63GB of iOS-created garbage sitting in his Recoverable Items, yet has a 1GB mailbox quota. This account alone is responsible for almost doubling the size of the database it's on.

If you look in an affected user's recoverable items in Outlook, you will see the same 1 or more messages, actually meeting request responses, showing up several times per minute for however long the problem has been occurring for that user. My worst case user's device had been trying to sync 4 meeting responses, several times per minute, since he updated his device on 1/28/13. 4 messages times 6-10 times a minute times 2 weeks... voila! 63GB of garbage in his mailbox.

Since iOS 6.1 was released my Exchange databases have increased in size by roughly 30%, whereas before January 28th they were pretty much at a steady state due to our rather draconian email purge policy. The daily IIS log size on the CAS which handle EAS traffic has almost doubled. But my Exchange servers aren't melting down... what's really killing me is the space this is chewing up in our backup solution. We are consuming an additional 150GB a day, every day, on our backup servers. Ouch again. This is actually how I noticed the problem in the first place.

In answer to why everyone isn't seeing it, all I can say is that out of 700 or so iOS 6.1 users in my environment it is only really noticeably affecting about 2 dozen mailboxes. That's a pretty small percentage. Most of our users have been trained by Apple's previous ActiveSync SNAFUs to NOT manage their Exchange calendar on their iDevice. That's been the only saving grace.

Is iOS capable of being rolled back to a previous version? It would seem to be the best solution.

I think you might be able to do it if you've saved the previous version on your computer. I don't remember the specifics, but it's certainly beyond what any typical user would want to do. Apple doesn't make it easy.

Is iOS capable of being rolled back to a previous version? It would seem to be the best solution.

If you've jailbroken Cydia keeps a copy of whatever keys are necessary to do that up to the each version you've jailbroken with, but that's the only method I know of.

Edit for better explanation:the ISPW files you restore from are signed by Apple, and each device needs a key to have them installed. When you try to restore an image, iTunes asks some Apple server for the keys which will say whether it's ok to install or not. Old versions are generally declined once a new version comes out. Cydia gets around this by requesting Apple's servers for your devices keys and keeps a copy on their own server. If you edit your hosts file, you can have iTunes' request get rerouted to Cydia's servers pretending to be Apple to let you roll back to any keys you have cached with Cydia from jailbreaking.

Is iOS capable of being rolled back to a previous version? It would seem to be the best solution.

I think you might be able to do it if you've saved the previous version on your computer. I don't remember the specifics, but it's certainly beyond what any typical user would want to do. Apple doesn't make it easy.

There is no official way to roll iOS back. You can if you saved a previous version and don't mind hacking some, but not officially.

Is iOS capable of being rolled back to a previous version? It would seem to be the best solution.

You can do it, but it's not officially supported. Basically, you restore a system image in iTunes. You have to find a proper IPSW image file to restore on the web. I'm assuming you can keep your user data if you do a data backup before re-imaging and you restore the backup afterwards.

Interesting....no issues here. Most of our construction managers are on iOS as all the IT department and many managers and above. Overall, we have about 100 people on iOS devices. However we have a massive amount of storage so who knows. It may take awhile for this to happen to us....

Perhaps exchange is the problem? Maybe move to a competing system. Particularly if said enterprises are using 365, switch over to Google. It costs less too, and is cheaper. I'm sure both Apple and Microsoft will work to patch it, there is too much to loose for both companies.

If Exchange was the problem, then it would be affecting the other mobile (and desktop) OS's that connect to it. Since it's isolated to iOS 6.1, it would point to an iOS 6.1 problem, simple deduction.

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

I've heard of complaints with IOS 6 since it came out. You can just blame Microsoft, but Microsoft watches... If IOS is hammering exchange servers, it would behoove apple to fix this thing. It has to be burning thru peoples data plans too.

Perhaps exchange is the problem? Maybe move to a competing system. Particularly if said enterprises are using 365, switch over to Google. It costs less too, and is cheaper. I'm sure both Apple and Microsoft will work to patch it, there is too much to loose for both companies.

Because scrapping your entire email and calendar infrastructure at great expense, labor and downtime is an entirely appropriate response to a bug in the software of one new device OS version.

This wouldn't fix any problems with traffic, but could an admin just set up a script to clear consecutive logs from the same user? (Just as a stop-gap solution until Apple releases a fix) This fix would seem to remove the problem of server logs filling up too quickly.

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

I understand that it was only triggered from iOS, but that must include some kind of interaction with a bug in Exchange. The server just fundamentally shouldn't allow anyone but the meeting "owner" to cancel the meeting.

It sounds like apple and Microsoft should be talking to each other more. After all, it's not like there's an alternative for exchange, is there? Especially if your organization already invested in it, and apple supports it.

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

I understand that it was only triggered from iOS, but that must include some kind of interaction with a bug in Exchange. The server just fundamentally shouldn't allow anyone but the meeting "owner" to cancel the meeting..

Most certainly true for the meeting problem (would be an awful protocol otherwise), but in this case the problem seems squarely on the devices site: You can't really do much against DDOS attacks apart from blocking or slowing down the offending devices.

Is iOS capable of being rolled back to a previous version? It would seem to be the best solution.

I've seen log issues on Exchange servers from prior iOS devices. In the logs you pretty much have all the iOS devices in an organization constantly hammering away to refresh the same data. Android and BlackBerry devices can be found in the logs at less than 1/10th the connection frequency per device.

Whatever they did with iOS 6.1, they took it up a notch with the polling and full datastore refreshes.

One iOS device can account for about 200+ Mb worth of log data in the Exchange logs in 24 hours.

The problem seems limited to a couple specific Exchange server versions. iOS 6.1 pre-release versions were available for 2-3 months before it was released, it's curious why the issues weren't identified then.

It's a pretty narrow edge case only affecting a couple versions of Exchange. It's only triggered by responding to a request to change a single instance of a recurring meeting. And it probably takes multiple iOS users triggering the bug before it really gets to the point where its noticeable in the logs.

If you've got a few iOS users in a company beta-testing 6.1, and only fraction of them trigger the bug, then IT probably doesn't notice the extra load.

If you've got a lot of iOS users who have just upgraded to the release version of 6.1, and someone reschedules a recurring company-wide meeting, then a bunch of iOS clients are now hammering the Exchange server.

EDIT: Looking back on this comment, I think "narrow edge case" understates the problem. But I could still see how this could be overlooked in a beta test. Only some users in an enterprise would be testers, and only some of those would trigger the problem. Since it only hits the logs, the user would never notice. And if there aren't a ton of users with the problem, then IT probably doesn't notice the potential scale of the problem.

If Exchange was the problem, then it would be affecting the other mobile (and desktop) OS's that connect to it. Since it's isolated to iOS 6.1, it would point to an iOS 6.1 problem, simple deduction.

I think there is enough blame to go around All these iOS bugs have made it quite clear that Exchange was written to be a bit too trusting of the client. In this case just a few misbehaving clients can DoS the entire server. In previous cases clients were allowed to modify things that they shouldn't have had permission to do. Hopefully opening Exchange up to other clients will result in it becoming more robust and secure over time.

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

I understand that it was only triggered from iOS, but that must include some kind of interaction with a bug in Exchange. The server just fundamentally shouldn't allow anyone but the meeting "owner" to cancel the meeting..

Most certainly true for the meeting problem (would be an awful protocol otherwise), but in this case the problem seems squarely on the devices site: You can't really do much against DDOS attacks apart from blocking or slowing down the offending devices.

Agreed. My comment was really focused on that previous issue.

In fact, Apple claims to have already found the problem and is working on the fix.

Apple continue to be (and always have been) utterly useless in the enterprise space..

"utterly useless"? That statement literally boggles my mind. It seems like a fairly tremendous amount of people find their products useful or this wouldn't be that big of a deal. Are they end-user focused rather than administrator focused? Absolutely. But that hyperbole is more than a little disproportionate compared to their legitimate shortcomings.

I think there is enough blame to go around All these iOS bugs have made it quite clear that Exchange was written to be a bit too trusting of the client. In this case just a few misbehaving clients can DoS the entire server.

I agree! I mean there are all those great solutions out there that make DDOS attacks completely ineffectual, like.. umn - oh right the stuff proposed by MS with the obvious downsides which is why you generally don't want them.