Share this story

iOS 6.1 devices are hammering Exchange servers with excessive traffic, causing performance slowdowns that led Microsoft to suggest a drastic fix for the most severe cases: throttle traffic from iOS 6.1 users or block them completely.

"When a user syncs a mailbox by using an iOS 6.1-based device, Microsoft Exchange Server 2010 Client Access server (CAS) and Mailbox (MBX) server resources are consumed, log growth becomes excessive, memory and CPU use may increase significantly, and server performance is affected," Microsoft wrote on Tuesday in a support document.

The problem also affects Exchange Online in Microsoft's Office 365 cloud service. Office 365 customers may get an error message on iOS 6.1 devices stating "Cannot Get Mail: The connection to the server failed." The Microsoft support article says both Apple and Microsoft are investigating the problem.

Microsoft suggests several fixes, starting out gently, then escalating to the complete blockage of iOS 6.1 devices. Based on the fixes suggested, the problems may be caused when iOS devices connect to Exchange calendars.

The first workaround is "do not process Calendar items such as meeting requests on iOS 6.1 devices. Also, immediately restart the iOS 6.1 device."

If that doesn't work, users are instructed to remove their Exchange accounts from their phones or tablets while the Exchange Server administrator runs a "remove device" command on the server side. After 30 minutes, users can add the Exchange accounts back onto their devices but should be advised "not to process Calendar items on the device."

If that doesn't work, the fixes get more serious. The next method is for the server administrator to create a custom throttling policy limiting the number of transactions iOS 6.1 users can make with the server. "The throttling policy will reduce the effect of the issue on server resources," Microsoft notes. "However, users who receive the error should immediately restart their devices and stop additional processing of Calendar items."

One Exchange administrator who created a throttling policy through PowerShell to solve the problem provides a guide here, but Microsoft also has a page providing instructions.

Finally, the last method Microsoft recommends is to block iOS 6.1 users. "You can block iOS 6.1 users by using the Exchange Server 2010 Allow/Block/Quarantine feature," Microsoft notes. (See this post for more detailed instructions.)

Businesses of all sizes limiting or blocking iOS devices

We don't know exactly how widespread this problem is. It's clearly not affecting everyone, but the impact seems to run the gamut from small businesses to large.

It turned out that the 300GB VMware virtual machine hosting the Exchange server was full. "You can imagine our surprise when that VM filled up overnight," Ray said. "If we were running Exchange in a typical hardware-based server with a 1TB drive, it would have taken us a week to realize the problem."

How did it happen, and how did the company get things working "normally" again? "The transaction log had 200,000 records and was the indication of a problem," Ray said. "Our temporary solution has been to ask iOS users to switch to manual pull rather than ActiveSync push. For heavy e-mail users, we are recommending an automatic pull every 30 minutes. So far, that seems to have kept Exchange happy with no other issues since last week. Let's hope that Apple and Microsoft put their heads to together and fix this soon."

We heard from several other people on Twitter that they have been bit by the iOS 6.1/Exchange problem. One said, "My 22,000+ employee enterprise has blocked iOS 6.1, execs all have iOS."

A support thread on Microsoft's Exchange Server site was opened January 31 to discuss the excessive logging caused by iOS 6.1. The server administrator who began the thread identified an iPad that "caused over 50GB worth of logs" on a single database.

The thread got more than a dozen replies. One Exchange administrator explained that "malformed meetings on a device cause the device to get into a sync loop which causes excessive transaction log growth on the Exchange mailbox servers." This in turn "will cause Exchange performance issues and potentially transaction log drives to run out of disk space which would then bring down Exchange."

To solve the problem, this admin simply "disabled all iOS 6.1 on our Exchange system."

iOS 6.1 was released on January 28. iOS 6.1.1 came out a couple of days ago, but for now it can only be installed on the iPhone 4S and is designed to fix cellular performance and reliability. Apple didn't mention anything about Exchange fixes when releasing this latest version. Last year, iOS 6.0.1 fixed an Exchange problem that could lead to entire meetings being canceled when even a single iOS user declined a meeting invitation.

The iOS 6.1 problem isn't the first time iOS has caused Exchange servers to perform poorly. An Apple support article from 2010 describes sync problems in iOS 4 and says, "Exchange Server administrators may notice their servers running slowly." At the time, Microsoft noted iOS 4 led to "Exchange administrators... seeing heavier than normal loads on their servers from users with iOS devices." Microsoft got in touch with Apple to fix that problem.

We've asked both Apple and Microsoft how many users are impacted by the latest problem, and when a more permanent fix is coming. We also asked Apple if it agrees with the workarounds suggested by Microsoft. Microsoft told us it has nothing else to say, as the "support article contains the latest." Apple has not responded to our request for comment as of yet.

UPDATE: Apple posted a support document of its own today, describing the problem thusly:

When you respond to an exception to a recurring calendar event with a Microsoft Exchange account on a device running iOS 6.1, the device may begin to generate excessive communication with Microsoft Exchange Server. You may notice increased network activity or reduced battery life on the iOS device. This extra network activity will be shown in the logs on Exchange Server and it may lead to the server blocking the iOS device. This can occur with iOS 6.1 and Microsoft Exchange 2010 SP1 or later, or Microsoft Exchange Online (Office365).

Apple's suggested fix is to turn the Exchange calendar off and back on again within the iPhone's settings. An operating system update to fix the problem is on the way. "Apple has identified a fix and will make it available in an upcoming software update," Apple said.

Promoted Comments

Full disclosure, I work for MS in the Exchange product group. I cannot go into details about this iOS condition beside what has been posted publically by each company at this time, but that Apple has posted their support article stating they will be issuing an iOS fix. http://support.apple.com/kb/TS4532

Exchange transaction logs contain the transactions made within the Exchagne databases themselves. You can't open a transaction log and read the data in text form as they basically contain instructions to the Jet database similar to "Change page <10000> to contain <XYZ>" and nothing more. If you get a client with a runaway process it can end up with endless T-Logs being created. Thankfully there are mechanisms for customers to detect these conditions taking place as well as ways to block access to such devices when this kind of thing happens.

At a high level a transaction would go...

1. A client makes some kind of change to an object (e.g. marks a message as read).

2. Exchange makes that change in memory first.

3. Exchange logs that change to a transaction log file second.

4. Exchange commits the change to the database third at some point later (it isn't very long, but it isn't instant either).

5. <Optional> These transactions may be shipped either in memory or via log file to another copy of the database.

The change (or transaction) is always written to the t-log BEFORE the database so there is a record of the transaction in case the database file is ever lost in the future before it is backed up. This helps maintain the four pillars of being an ACID database. This old Exchange article talks about it if you search for ACID. http://support.microsoft.com/kb/271987

The only time transaction logs can be safely flushed is automatically if Circular Logging is enabled, or after a Full or Incremenal backup that integrates with the Exchange backup API notifying the database engine the files have been protected.

I am in charge of a 10,000 seat Exchange 2010 SP2 environment and have been doing tons of analysis on this problem the last few days. Hopefully I can shed some light on exactly how this is hurting Exchange, since I see a lot of speculation and incorrect info being spread around.

The problem is twofold.

First, affected iOS 6.1 devices are hammering CAS with rapid, repeated attempts to synchronize the same data, thereby increasing CPU and memory utilization on the CAS. How rapid and repeated? In my environment the average EAS device syncs roughly 600-800 times per day. Affected iOS 6.1 devices are synchronizing anywhere from 50,000 to 80,000 times per day... per device, if the user has multiple devices. Yeah, ouch.

Second, it's not just a DoS on the CAS... the transactions are actually making it into the store, hence transaction log and database growth! The data is not being stored in the user's normal folders, though, so the user's quota doesn't stop the data growth as you might expect. The data growth is occurring in the Recoverable Items folder (formerly known as the dumpster) which does NOT adhere to the users normal quota. In my worst case mailbox, the user has 63GB of iOS-created garbage sitting in his Recoverable Items, yet has a 1GB mailbox quota. This account alone is responsible for almost doubling the size of the database it's on.

If you look in an affected user's recoverable items in Outlook, you will see the same 1 or more messages, actually meeting request responses, showing up several times per minute for however long the problem has been occurring for that user. My worst case user's device had been trying to sync 4 meeting responses, several times per minute, since he updated his device on 1/28/13. 4 messages times 6-10 times a minute times 2 weeks... voila! 63GB of garbage in his mailbox.

Since iOS 6.1 was released my Exchange databases have increased in size by roughly 30%, whereas before January 28th they were pretty much at a steady state due to our rather draconian email purge policy. The daily IIS log size on the CAS which handle EAS traffic has almost doubled. But my Exchange servers aren't melting down... what's really killing me is the space this is chewing up in our backup solution. We are consuming an additional 150GB a day, every day, on our backup servers. Ouch again. This is actually how I noticed the problem in the first place.

In answer to why everyone isn't seeing it, all I can say is that out of 700 or so iOS 6.1 users in my environment it is only really noticeably affecting about 2 dozen mailboxes. That's a pretty small percentage. Most of our users have been trained by Apple's previous ActiveSync SNAFUs to NOT manage their Exchange calendar on their iDevice. That's been the only saving grace.

Although now almost irrelevant in market share, this does not occur with GroupWise servers of any version (that I know of) using the same protocols to interface with iOS, so perhaps MS isn't entirely in the clear. And only with Exchange 2010 SP1, what did they change with SP1 ?

It seems weird to me that Exchange is even able to be brought down by excess log file generation. That seems like a thing that they should not put in the hands of external sources. I would think that the exchange server would be able to sense how much space is remaining and either stop saving repetitive logs or dropping old logs for new ones... or something. It's not like Exchange HAS to be stupid.

Obviously, Apple should get their OS worked out as well... but my feeling is that this should be a thing that is capable of maybe a little slow down... not a complete takedown of Exchange. Sounds like work to be done on both sides to me.

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

I understand that it was only triggered from iOS, but that must include some kind of interaction with a bug in Exchange. The server just fundamentally shouldn't allow anyone but the meeting "owner" to cancel the meeting.

EDIT: trimmed the quote to the relevant part.

To be more specific (link below), when an iOS 6 user declined a meeting, they would then send meeting cancellation notices to the entire e-mail distribution. So other users would receive a meeting cancellation, and most likely accept it. For small meetings, the odds of someone declining with an iOS 6 device was low (and the fact that you knew the meeting organizers). For larger division wide meetings, it's much less obvious, as the execs have multiple TA's that handle their agenda.

Anyways, shortly afterwards, there was a corporate wide e-mail (this is a fortune 500 tech company) telling everyone to NOT decline a meeting from an iOS 6 device and recommending anyone using iOS devices (for corporate use) to refrain from upgrading to iOS 6.

It seems weird to me that Exchange is even able to be brought down by excess log file generation. That seems like a thing that they should not put in the hands of external sources. I would think that the exchange server would be able to sense how much space is remaining and either stop saving repetitive logs or dropping old logs for new ones... or something. It's not like Exchange HAS to be stupid.

Obviously, Apple should get their OS worked out as well... but my feeling is that this should be a thing that is capable of maybe a little slow down... not a complete takedown of Exchange. Sounds like work to be done on both sides to me.

There are probably lots of regulations that require logs to be kept for a specific amount of time. Apart from the used up space there's also the problem of using up lots of CPU resources.

But yes I'm also a bit surprised that there isn't an option for a rolling log - sensible logging frameworks offer that one out of the box and for people who aren't required to keep logs for a longer period of time that'd be the least problematic solution.

It seems weird to me that Exchange is even able to be brought down by excess log file generation. That seems like a thing that they should not put in the hands of external sources. I would think that the exchange server would be able to sense how much space is remaining and either stop saving repetitive logs or dropping old logs for new ones... or something. It's not like Exchange HAS to be stupid.

Obviously, Apple should get their OS worked out as well... but my feeling is that this should be a thing that is capable of maybe a little slow down... not a complete takedown of Exchange. Sounds like work to be done on both sides to me.

These are not just logs. They are transaction logs that are used to recover lost data or repair corruption in the storage groups. They give the ability to restore data that has come in since the last backup. They need to store every transaction.

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

Seems like a DoS bug in Exchange Server 2010. A typical denial of service by creating huge log files should not be possible in this day and age. Hell, simple throttling would help. Removing excessive duplicate logs would also help. Rolling over logs with compression would help too.

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

Lulz, ANY database system which cares about data integrity can be DOS'd by an authenticated user unless you put draconian per-user resource limits in place, and keeping track of those resource limits uses up system resources. For example I could put that kind of limits on users in our Oracle environment, but just turning on the tracking for per-user resource limitations uses up more resources on the server than our baseline operations so it's simply not worth it, monitor your systems and kick/block abusive users is far easier and cheaper. This is exactly why MS recommends the actions they do, first try to fix the client, then slow them down, and if they're still abusing system resources ban them.

Seems like a DoS bug in Exchange Server 2010. A typical denial of service by creating huge log files should not be possible in this day and age. Hell, simple throttling would help. Removing excessive duplicate logs would also help. Rolling over logs with compression would help too.

Exchange can already set throttling policies. I'm willing to bet most on-premise Exchange installations never had a throttling policy set because excessive transaction logs were not an issue. At least, not until a very popular client with a big flaw came along.

And these aren't logs in the typical sense. They're transaction logs, things to write to a database.

To what extent should one expect a client to behave well with this server? Is the exchange server designed to enable clients (other than those written by MS) to be interoperable with it? If one wrote a client, how does one test for interface compliance?

Perhaps exchange is the problem? Maybe move to a competing system. Particularly if said enterprises are using 365, switch over to Google. It costs less too, and is cheaper. I'm sure both Apple and Microsoft will work to patch it, there is too much to loose for both companies.

If Exchange was the problem, then it would be affecting the other mobile (and desktop) OS's that connect to it. Since it's isolated to iOS 6.1, it would point to an iOS 6.1 problem, simple deduction.

No, your deduction is a little too simple. Even if the problem lies solely with iOS 6.1, which I doubt, Exchange *also* has a problem, in that it will allow itself to be killed by a single or a few clients.

Quote:

Although I have to say the funniest issue I've seen with iOS 6 and Exchange was when an iOS 6 user decides to decline a meeting, it would cancel the meeting for everyone. Since it was a division wide quarterly (attended by 10K+ people), the execs had to explain how it was not actually canceled.

That can't possibly be a client's fault, that the server allows an invited user to seize control of an item without credentials. Or even with credentials, how would the server allow that without an explicit superuser thing going on.

Apple writes some pretty tight software in it's own house. They let this fact hurt them when they interface with others; they seem to approach it with a kind of arrogance. A: "this is the way it should be" mindset, with little obligation to test unusual scenarios.

Apple is that problematic and slightly too smart user in the office that always finds whatever flaw you've left in your deployment or software. All IT support people have one of these people in their group (God help you if you have a lot, like in a web design group). Their existence is a pain in the ass, but they make us better in the end.

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

Standard procedure. You can DDOS anything that keeps permanent logs ... just keep the retries with faulty requests going ... the more you send, the more get logged.

Quick fix -- disable diagnostic and transaction logging. After all who needs to know what went wrong?Quick fix -- Enable deletion of old entries. After all historical transaction logs are not required (except when mandated by law or corporate policy)Quick fix -- Install a disk farm to store the logs on and copy them off to archive storage daily. Something every corner store should be doing....

Apple is sending Exchange a request that is malformed in some manner and when informed of the error, it immediately repeats the identical request until the user resets the application or the log runs out of storage. This is not a server error unless the server is rejecting a valid request. It is simply an accidental demonstration of how to DDOS a server that keeps permanent logs--using authorized devices in the hands of authorized users running an authorized application. All the blackhat has to do is introduce a malicious update to the corporate client software.

No, your deduction is a little too simple. Even if the problem lies solely with iOS 6.1, which I doubt, Exchange *also* has a problem, in that it will allow itself to be killed by a single or a few clients.

Exchange doesn't have the problem. Well, it has a human one in that few admins have actually set throttling policies.

It seems weird to me that Exchange is even able to be brought down by excess log file generation. That seems like a thing that they should not put in the hands of external sources. I would think that the exchange server would be able to sense how much space is remaining and either stop saving repetitive logs or dropping old logs for new ones... or something. It's not like Exchange HAS to be stupid.

Obviously, Apple should get their OS worked out as well... but my feeling is that this should be a thing that is capable of maybe a little slow down... not a complete takedown of Exchange. Sounds like work to be done on both sides to me.

You have to understand what the "log" is. Exchange logs are only purgable after a successful DR process has occurred. The streaming logs actually contain the life mail data that has yet to be moved to the permanant database mailbox store. In addition, there's a secondary log that acts as a fallback point allowing the streaming database to be rebuilt in an error condition. Those logs are only purged by Exchane's internal DR engine after a successful backup. They are not audit logs.

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

Lulz, ANY database system which cares about data integrity can be DOS'd by an authenticated user unless you put draconian per-user resource limits in place, and keeping track of those resource limits uses up system resources. For example I could put that kind of limits on users in our Oracle environment, but just turning on the tracking for per-user resource limitations uses up more resources on the server than our baseline operations so it's simply not worth it, monitor your systems and kick/block abusive users is far easier and cheaper. This is exactly why MS recommends the actions they do, first try to fix the client, then slow them down, and if they're still abusing system resources ban them.

Agreed. The very fact they did not detect runaway transaction log growth BEFORE it hit the end of the disk means they clearly were not following Exchange best practices for operations management in the first place. Granted, 300GB of logs in 24 hours, that's extreme, and even if they saw the activity they might not have been able to work out a resolution in time and implement it, but not getting alerts when your log volume was more than 50% full, that's an issue by itself. Also, sounds like they had a LOT of users on a single server that should have been a CCR cluster. Also, it was a VM, but they hard set the volume size on a log volume? Seems like a whole series of poorly thought out and monitored architectural decisions that led to the outage. Yes, Apple may be a leading factor, but runaway logs are not exactly uncommon in exchange. Hell, just a "me too" reply-all chain can kill even reasonably sized mail servers....

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

bug, not so much. Poorly implemented architecture that did not follow common best practices, nor implement the already available filtering and limiting tools was part of the issue. Not using a realtime operations monitoring tool (like SCOM) which would have detected this runaway log BEFORE the server crashed is also an issue, as was deploying exchange log volumes to a VM without enabling dynamic sizing.

Also, even on BYOD devices, any such piece of code should be easy to keep away from your exchange server by creating and distributing through exchange profiles that limit what users can do with devices that interact with mission critical corporate systems, by either limiting what apps can be installed on a users device, or directly managing their config, or forcing that exchange traffic through a dynamically created VPN other apps could not access.

Isn't this the problem that was fixed in 6.1.1? Seems odd to read this article with no mention of that already-released update.

Article:

iOS 6.1.1 came out a couple of days ago, but for now it can only be installed on the iPhone 4S and is designed to fix cellular performance and reliability. Apple didn't mention anything about Exchange fixes when releasing this latest version.

...

Apple's suggested fix is to turn the Exchange calendar off and back on again within the iPhone's settings. An operating system update to fix the problem is on the way. "Apple has identified a fix and will make it available in an upcoming software update," Apple said.

Full disclosure, I work for MS in the Exchange product group. I cannot go into details about this iOS condition beside what has been posted publically by each company at this time, but that Apple has posted their support article stating they will be issuing an iOS fix. http://support.apple.com/kb/TS4532

Exchange transaction logs contain the transactions made within the Exchagne databases themselves. You can't open a transaction log and read the data in text form as they basically contain instructions to the Jet database similar to "Change page <10000> to contain <XYZ>" and nothing more. If you get a client with a runaway process it can end up with endless T-Logs being created. Thankfully there are mechanisms for customers to detect these conditions taking place as well as ways to block access to such devices when this kind of thing happens.

At a high level a transaction would go...

1. A client makes some kind of change to an object (e.g. marks a message as read).

2. Exchange makes that change in memory first.

3. Exchange logs that change to a transaction log file second.

4. Exchange commits the change to the database third at some point later (it isn't very long, but it isn't instant either).

5. <Optional> These transactions may be shipped either in memory or via log file to another copy of the database.

The change (or transaction) is always written to the t-log BEFORE the database so there is a record of the transaction in case the database file is ever lost in the future before it is backed up. This helps maintain the four pillars of being an ACID database. This old Exchange article talks about it if you search for ACID. http://support.microsoft.com/kb/271987

The only time transaction logs can be safely flushed is automatically if Circular Logging is enabled, or after a Full or Incremenal backup that integrates with the Exchange backup API notifying the database engine the files have been protected.

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

It's an authenticated user. When you access any e-mail service as an authenticated user your session is making thousands of database calls to collect the relevant data for your session. Any monolithic e-mail infrastructure (i.e. one server) is subject to an excessive access and request DoS by an authenticated user trying to get data by definition they have permission to access.

I am sure in a monolithic, non-clustered implementation that every vendor's e-mail server software can be crashed (or at least DoS'd into non-service) by a DoS style request pattern by an authenticated user through a relatively direct API.

Also to clarify my earlier post regarding iOS going back a few versions clogging the Exchange logs with excess traffic. I should have mentioned I have only noticed this from iPhones, particularly 4/4S/5 models. The iPad doesn't appear to have such issues, or to be closer to the access pattern of any other Android or Blackberry device.

As for Apple commenting that this is only an issue with certain patch levels of Exchange, the problem is still with the access pattern of their Exchange client. You can watch the logs in real time and your jaw just drops trying to figure out why a single iPhone needs to make 20-30 connection and data requests in a few seconds.

Network access hogging is not just an issue on iPhones though. I have noticed certain Cisco and older Linksys APs resync everyone's connections after a MBA / MBP connects and starts making a high rate of individual requests (I am sure the issue is due to a NAT or session table which is too small in most cases). Not every version of OSX does it, but I really haven't had it become a noticeable enough issue to investigate further. I rather prefer the fact that Windows takes a small hit to overall usable network bandwidth to measure latency, effective bandwidth, and consistency in order to operate cooperatively on a network.

Perhaps exchange is the problem? Maybe move to a competing system. Particularly if said enterprises are using 365, switch over to Google. It costs less too, and is cheaper. I'm sure both Apple and Microsoft will work to patch it, there is too much to loose for both companies.

Because scrapping your entire email and calendar infrastructure at great expense, labor and downtime is an entirely appropriate response to a bug in the software of one new device OS version.

"Bill from accounting's iPhone is hammering our Exchange server. Should we, you know, give him a new phone?"

If you've got a few iOS users in a company beta-testing 6.1, and only fraction of them trigger the bug, then IT probably doesn't notice the extra load.

If you've got a lot of iOS users who have just upgraded to the release version of 6.1, and someone reschedules a recurring company-wide meeting, then a bunch of iOS clients are now hammering the Exchange server.

The issue is not a company beta testing iOS - but that apple should be testing it before they release it. Or do you think it is ok for apple to tell enterprise users that they don't test their OS?

While I am not impressed by my experience with Apple in 'enterprise', I have to wonder how this doesn't qualify as a DoS vulnerability in Exchange.

If an iOS device can hammer an Exchange server to death purely by accident with nothing more than a peon-level user's account credentials, I shudder to think what a deliberately and maliciously malformed client could do with the same(easily available) credentials.

Obviously, in practice, 'don't fuck up common servers' is a good QA guideline for clients; but this is the brutal, relentlessly hostile, contemporary internet, if your server can be fucked up by a client, somebody is probably building a client designed to do exactly that as we speak. This. Should. Not. Be. Possible. If it is, you have a bug that needs fixing.

bug, not so much. Poorly implemented architecture that did not follow common best practices, nor implement the already available filtering and limiting tools was part of the issue. Not using a realtime operations monitoring tool (like SCOM) which would have detected this runaway log BEFORE the server crashed is also an issue, as was deploying exchange log volumes to a VM without enabling dynamic sizing.

Also, even on BYOD devices, any such piece of code should be easy to keep away from your exchange server by creating and distributing through exchange profiles that limit what users can do with devices that interact with mission critical corporate systems, by either limiting what apps can be installed on a users device, or directly managing their config, or forcing that exchange traffic through a dynamically created VPN other apps could not access.

Say they WERE following best practises and detect a runaway log - what would you suggest they do? Other than boot that authenticated user off exchange as they are still sending malformed requests?

All you Mac guys should just give up trying to pin this on Microsoft. As many of you frequently remind the rest of us, these Windows Admins have much more experience troubleshooting computer problems than you do and they all say it is Apple's bug.

I'm getting annoyed with the sensationalized headlines here at ars. This isn't cnet.

Sensationalized? My company, with 10s of thousands of employees, had access shut down for ANY device using iOS 6.1 yesterday. Everyone who chose iPhone for their BYOD can not access calendar, etc. It's stunning to me that the actual press doesn't report on this, but the Home Depot 'drop in the bucket' has been on an endless loop of reporting.

I am in charge of a 10,000 seat Exchange 2010 SP2 environment and have been doing tons of analysis on this problem the last few days. Hopefully I can shed some light on exactly how this is hurting Exchange, since I see a lot of speculation and incorrect info being spread around.

The problem is twofold.

First, affected iOS 6.1 devices are hammering CAS with rapid, repeated attempts to synchronize the same data, thereby increasing CPU and memory utilization on the CAS. How rapid and repeated? In my environment the average EAS device syncs roughly 600-800 times per day. Affected iOS 6.1 devices are synchronizing anywhere from 50,000 to 80,000 times per day... per device, if the user has multiple devices. Yeah, ouch.

Second, it's not just a DoS on the CAS... the transactions are actually making it into the store, hence transaction log and database growth! The data is not being stored in the user's normal folders, though, so the user's quota doesn't stop the data growth as you might expect. The data growth is occurring in the Recoverable Items folder (formerly known as the dumpster) which does NOT adhere to the users normal quota. In my worst case mailbox, the user has 63GB of iOS-created garbage sitting in his Recoverable Items, yet has a 1GB mailbox quota. This account alone is responsible for almost doubling the size of the database it's on.

If you look in an affected user's recoverable items in Outlook, you will see the same 1 or more messages, actually meeting request responses, showing up several times per minute for however long the problem has been occurring for that user. My worst case user's device had been trying to sync 4 meeting responses, several times per minute, since he updated his device on 1/28/13. 4 messages times 6-10 times a minute times 2 weeks... voila! 63GB of garbage in his mailbox.

Since iOS 6.1 was released my Exchange databases have increased in size by roughly 30%, whereas before January 28th they were pretty much at a steady state due to our rather draconian email purge policy. The daily IIS log size on the CAS which handle EAS traffic has almost doubled. But my Exchange servers aren't melting down... what's really killing me is the space this is chewing up in our backup solution. We are consuming an additional 150GB a day, every day, on our backup servers. Ouch again. This is actually how I noticed the problem in the first place.

In answer to why everyone isn't seeing it, all I can say is that out of 700 or so iOS 6.1 users in my environment it is only really noticeably affecting about 2 dozen mailboxes. That's a pretty small percentage. Most of our users have been trained by Apple's previous ActiveSync SNAFUs to NOT manage their Exchange calendar on their iDevice. That's been the only saving grace.