This is the 38th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization or MSPs for the series PM Lee S to get started.

“Have we been hacked?” Those were the words I heard during a conversation with my CEO, his fellow executives, and my boss.

The day started something like this: A ticket comes across for deactivation of an employee. Let’s call him Ezekiel. Nothing new. I do all my checks to make sure Ezekiel was disabled in AD, blah, blah, blah — you know the routine. An hour later, my boss calls me down to the CEO’s office. He heads me off at the door.

Boss: “You disabled Ezekiel’s account, RIGHT?”

Me: “Of course. Why? What’s going on?”

He moves aside so I can enter the office where I see a room full of top executives, all displaying angry red faces, clutching several printed out documents.

Exec 1: “THIS is what’s up,” he says, throwing a small stack of papers into my chest.

A quick glance and I can tell the papers are printouts of emails directed to several inside and outside partners describing, in detail, certain actions of the CEO’s wife, along with proprietary information about the company. I check the sender. To my surprise, its Ezekiel, the very same user I disabled not more than two hours ago.

Boss: “Check the timestamp! Have we been hacked?” he asks, pointing to the top of the paper.

The timestamp showed it was sent out 15 minutes ago, which by everything I was ever taught or knew about being a Systems/Network Administrator was impossible… Or was it? From here, it’s just a bunch of raised voices yelling and talking over one another for the next 10 minutes with all eyes on me while I slowly back myself into a corner cowering like a beaten dog. I don’t have any answers for them but assure them I will get to the bottom of this as I walk out with my tail between my legs.

So, how was it that Ezekiel could still send emails after I disabled his account? Did he hack into the company and enable his account? If he did, why would he only target his email account?

The experiment

Combing through logs on firewalls, servers, and wireless access points, I couldn’t find any sign of an intrusion or anomalies. So I decided to recreate the situation from scratch over the next five days and was alarmed at my findings.

I created five users with the same access as Ezekiel. Two accounts were disabled through Active Directory, just as Ezekiel was. Two had their passwords changed but the accounts remained active — a typical scenario when a supervisor needs time to obtain files, emails and other data from a terminated employee’s account before it’s disabled completely. And for the last one, I disabled just the email account. I documented every step along the way and sent the instructions off to five other Exchange/domain admins at other companies to test along with me.

Each time, the results were the same. Internally, they were immediately disabled and full access was denied to everything. Externally they weren’t able to log into anything either… except Outlook Web Access and ActiveSync.

Microsoft has documented and known about this since Exchange 5.5, introduced in 1997. In fact, they did this on purpose! (There’s more info in KB articles here and here.)

Disabling a users account in Active Directory or changing the user’s password will disable that account immediately on the LAN as expected. However, per Microsoft, the user will still have access to email from any external Internet connection through Outlook Web Access for a default minimum of 15 minutes in ideal conditions.
In our testing, this turned out to be much longer than 15 minutes under non-ideal conditions, such as using Firefox (as opposed to the latest updated version of IE) or an iOS or Android phone (instead of a Windows mobile device). In fact, in my testing, I found I was still able to connect to a disabled account using Firefox for around three hours, and using an Android phone with ActiveSync, up to six hours. The time can also be extended should the user already have his/her OWA account opened outside the organization at the time of disabling the account. What’s more, changing the user’s password does nothing while this token is still cached. You can change the user’s password 100 times and not only will they still be able to use their original password during the 15-minute interval, but they can now successfully use ANY of the 100 passwords you just changed it to, increasing the odds of a successful brute force attack with each password change.

The findings

At this point, anyone even remotely intelligent should be freaking out, and for good reason. So what’s going on here? Why is this even allowed?

In short, it’s to improve performance in IIS. When a user logs into Outlook Web Access, the credentials for the request are used to create a token on the server. This token is then used during the duration of the session to access files (Exchange 2007 allowed browsing of shared server resources, which is no longer allowed in Exchange 2010 — and now we know why) or other system resources such as email. The token is kept in cache by default for 15 minutes. There’s obviously more going on than just this, but that’s the short and sweet. This means that at any given time, changing a user’s password or disabling their account in Active Directory will not take effect on the Exchange server regarding WAN access for up to 15 minutes (under ideal conditions) depending on when IIS last cleared the token cache.

So, why 15 minutes — why don’t we just lower it? There’s a reason to the madness. Each time a user logs into OWA, it creates a storm of ASP packets to the server for authentication along with other things. This isn’t so bad if it’s only happens every 15 minutes, but lower this setting in a larger organization, and your phone starts ringing from users complaining about performance on Exchange or frequent disconnect notices. Microsoft chose 15 minutes as the happy medium between performance and security.

I hope this article has provided the info necessary to approach HR and upper management on the risks associated with using Outlook Web Access in an organization and to develop a policy going forward for terminated employees that may be considered “high risk” as we learned Ezekiel was.

One thing I noticed immediately: A user's account doesn't need to be enabled to have access to files or e-mail.

I'm paranoid so I use Eventlog Analyzer to check login events and alert me to anomalies. We also use it to send alerts to IT and HR everytime an account is disabled. That way we never get the question: "Was so and so's account disabled?" We can say, "Yes" and have HR back us up.

Well this is disturbing. But as I was reading this I got to the part about first seeing all these emails from Ezekiel and the first thing that came to mind was that the senders address had been spoofed. Easy enough to make an email look like it came from anyone so if it was me I could send in emails from my Comcast account and change the senders address to my work one. Am I wrong?

I have noticed this myself, not in this situation thankfully. I had noticed in particular with iphones using active sync that if a user changes their password the phone will continue to send and receive email for up to a day before the token it has expires and the user is prompted to enter the new password.

I guess we should all add delete the users mailbox to our list when disabling an account. this way it is not accessible.

we can always re-attach it to a different user if we need to access it later

Please inform us how the C-Suite received your final explanation? Did you get a sense that they thought you were bullshitting them or that you were being straight with them?

I absolutely hate it when vendors hang IT employees out to dry like this. Thank you for this info though. We may have to make some readjustments to how we handle letting people go. But then again, there may be people who leave the company of their own will and are disgruntled.

Perhaps we should disable all accounts prior to the exit interview for mutual separations? Perhaps management should give IT a heads up at least 15-minutes prior to terminating an employee? Perhaps this is a situation where a slight HR policy modification could help mitigate this security loophole.

Perhaps management should give IT a heads up at least 15-minutes prior to terminating an employee? Perhaps this is a situation where a slight HR policy modification could help mitigate this security loophole.

This is always the best answer ITSlave. Our policy here (leaving ex-employee e-mails around so people can pick through stuff looking for important client correspondence) leaves us exposed to this exact situation.

We have a barracuda message archiver and could delete the accounts entirely and use that instead but management wants the convenience of just attaching the ex-employees e-mail to their outlook profiles so we leave the accounts online.

I got the heebeejeebies when I read this and am using 365. I guessed as the article mentioned that the problem has been fixed on Exchange 2010 that it would not be a problem on 365. With a test user, I changed the login setting to blocked and then attempted to access through OWA and I was not allowed in.