and other brilliant error messages

Exchange 2010/2007 mixed mode – broken UM

This one was a very obscure issue. I recently installed the first Exchange 2010 server in our infrastructure and haven’t really configured anything on it yet. It turns out that at around the same time our Exchange 2007 Unified Messaging stopped working but it took a while for anyone to notice. Long enough in fact that I strongly suspected our PABX system was at fault, especially since it has recently had some hardware failures.

The symptoms were a total inability to leave messages on the UM server, but strangely it could still be contacted by dialing the voicemail number. The audio message was something like ‘we cannot connect you at this time’. The fact that an Exchange voice prompt can be heard strongly suggests that the telecoms system is ok.

The event log errors on the Exchange server are 1082:

Event Type: Error
Event Source: MSExchange Unified Messaging
Event Category: UMService
Event ID: 1082
Date: 04/10/2011
Time: 16:05:38
User: N/A
Computer: EXCH-01
Description:
The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file “C:\Program Files\Microsoft\Exchange Server\UnifiedMessaging\voicemail429a59b-e196-4b25-940d-796c736fb93d.txt”. Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.

Firstly I tried increasing the Exchange UM log level but that wasn’t helpful at all. Error 500 implies some sort of auth issue so I double-checked all the certificates were valid (Get-ExchangeCertificate | fl). Then I started reading. There isn’t a huge amount on this subject (most of what’s written relates specifically to 2010) but I found this excellent write-up by Terence Luk. His were the same symptoms I was seeing, but even creating a new Hub Transport Receive Connector didn’t fix it.

What was different about my infrastructure? Well in my environment all the roles (including UM) are on the same server. So I had to telnet from the Exchange server itself to test properly. And then I noticed something strange. My Exchange server was referring to itself by the IP of one of its iSCSI NICs, which are on a private IP range. Why? No duplicate A records in DNS from spurious interface registrations, no Client or Microsoft Networks assigned on the iSCSI NICs. A simple ipconfig seemed to list them in an unusual order, with the actual Local Area Connection in the middle. This was probably the cause. Next stop Eric Reitz’s How to set/view the NIC bind order in Windows. A new menu I never new existed until this point!

And sure enough, this fixed it. The Exchange 2007 server’s NIC binding order had been like this for well over a year and never caused any issues, but I think that the addition of the first Exchange 2010 server must have raised the security, probably implementing TLS for inter-role UM traffic where it wasn’t used before.