Next I used LDP.exe to translate the SID from the error message into something readable.

After investigating the problem I found out that "SeSecurityPrivilege privilege" translates to "Manage audit and security log" under user rights assignment in group policy. Exchange setup automatically adds "DOMAIN\Exchange Enterprise Servers" and "DOMAIN\Exchange Servers" to the "Manage audit and security log" user rights assignment on the Default Domain Controllers Policy.

My client had unlinked the Default Domain Controllers Policy from the Domain Controllers OU and created their own custom policy - NOT RECOMMENDED. Restoring this policy resolved the problem.

- We can see connections trying to be established from 10.4.8.1 (in a remote AD site) and 10.4.2.8 (in the same AD site)- We can see the connections are hitting the receive connector SVR01\External- The receive connector is terminating the connection with "Service closing transmission channel"

If we look at the receive connector "External" on SVR01 we notice that Exchange Server authentication is not enabled meaning SVR01 is not able to receive mail relay from other hub transport servers in the Exchange organisation.

Thursday, December 22, 2011

After working with a Hybrid Office 365 deployment with Threat Management Gateway performing SSL offloading to an Exchange 2010 SP2 hybrid server for one of my customers I experienced a number of gotcha's which are not documented.

MRSProxy with SSL Offloading

The first issue was with MRSProxy. I found out the hard way that MRSProxy "/EWS/mrsproxy.svc" does not support SSL offloading. The MRSProxy connection must hit the Exchange 2010 Client Access Server on TCP443 using a secure SSL connection. When using SSL offloading with Forefront Threat Management Gateway as soon as unsecure HTTP port 80 connections were passed to MRSProxy on the Exchange 2010 Hybrid server we were receiving was (404) Not Found in the IIS logs.

We created a separate TMG firewall rule with a path rule of "/EWS/mrsproxy.svc" which is processed before our Hybrid firewall rule. This path rule re-encrypts and forwards to the Hybird server on TCP443. All other connections come through on TCP80.

Free/Busy with Office 365

The second issue was with free/busy. Users on-premises were able to view Free/Busy for users in Office 365, however users in Office 365 were unable to view Free/Busy for users on-premises. This was caused by the Web Listener being configured for Basic or Windows Integrated authentication. Free/Busy over an Organization Relationship uses Microsoft.Web.Services.SoapContext to authenticate free/busy... in other words the authentication is handled by the .NET framework. Threat Management Gateway must pass through authentication requests to /EWS/*. Setting the TMG listener to No Authentication achieves this.

From Exchange Server 2003 to Exchange 2010 SP2 (as of this writing) supports the ability to send email from different email addresses as long as the "send as" address has the same email address suffix of the users primary email address. For example I have an email address of clint.boessen@4logic.com.au. Provided I have been granted permissions I can send as from different addresses such as accounts@4logic.com.au or orders@4logic.com.au. However if my Exchange server has different Accepted Domains such as @kbomb.com.au I am not able to send as accounts@kbomb.com.au even if I have been granted permissions. Doing so will result in the following error:

You can't send a message on behalf of this user unless you have permission to do so. Please make sure you're sending on behalf of the correct sender, or request the necessary permission. If the problem continues, please contact your helpdesk.

Microsoft's claims this functionality is "by design" and there is no intention to create a resolution at this stage.

There is a 3rd party tool called ChooseFrom which you install on your Exchange 2007/2010 server. This tool enables Send As functionality from different domains however its rather pricey. As of this writing the asking price was 286.46 USD for a 1-2 user license, $214.84 USD if you want over 3 licenses, $7161.50 USD for a site license or $30078.30 USD for an enterprise license in which they will also provide you with the source code. If you are interested in this tool please see the following website:

As an alternative you can create a new Outlook Profile and create a POP account. Outlook 2010 supports multiple profiles at the same time. In the address enter your Exchange 2010 hub transport server and configure SMTP authentication. In the POP3 server address enter something bogus like mail.local. Next you need to configure the send and receive group for your POP3 profile. To do this perform the following steps:

1. Under Send/Receive Groups select Define Send/Receive Groups.2. Select your group name or define a new group and click Edit.3. In the Send/Receive Settings disable Receive mail items.

Now when you send/receive you wont get an error about not being able to receive from the bogus pop server.

Friday, December 16, 2011

When running the Hybrid Configuration Wizard I noticed the wizard crashes when attempting to create a Send Connector if a wild card certificate is attached to the default web site in IIS. What the Hybrid Configuration Wizard does is look at the common name of the certificate associated with the IIS service (verify this by using Get-ExchangeCertificate) and attempts to associate the common name as the FQDN on the Send Connector for the purpose of SMTP encryption. In a wildcard certificate *.example.com is the common name of the certificate. *.example.com is not a valid FQDN for a send connector in Exchange as thee FQDN must be associated to a domain name, not a wild card name. As a result the wizard crashes when trying to set *.example.com as the FQDN on the send connector. The following error is generated...

Additional troubleshooting information is available in the Update-HybridConfiguration log file located at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Update-HybridConfiguration\HybridConfiguration_12_16_2011_5_58_59_634596119396658235.log.

I have a work around which will allow you to complete the Hybrid Configuration Wizard. What you need to do is use another certificate with a valid common name such as mail.example.com so the wizard is able to create its Send Connector. If you use a self signed certificate or a certificate issued by an internal certificate authority the wizard will fail. You must use a public certificate trusted from a public root certificate authority.

Does this mean I need to purchase another digital certificate, I already own a wild card certificate?

No, you can use a digital certificate from http://www.freessl.com/ (provided by RapidSSL) which is free for 30 days. This certificate is not a SAN (subject alternative name) certificate which is trusted under multiple names. As a result you must issue the certificate to autodiscover.yourdomain.com as the autodiscover service is required when running the Update-HybridConfiguration command. The autodiscover address must be pointed at the same address for the hybrid server in which your trying to configure on your firewall. When you finish creating your new certificate and associating it with the IIS service and SMTP service, re-run the Update-HybridConfiguration tool. Your send connector will be created with autodiscover.yourdomain.com as the FQDN. Now you can re-assign your wild card certificate to the IIS service and SMTP service and update the FQDN on your send connector to a domain name of your choice in alignment with the wild card certificate such as mail.example.com.

SSL Offloading...

If your running SSL offloading, i.e. your Exchange web services are all running on port 80 you still need to link a digital certificate to your IIS service to complete the wizard. Simply untick Require SSL on the various Exchange web services in IIS manager. The digital certificate you use needs to be valid for the FQDN on the SMTP service and must be linked to IIS and SMTP (Get-ExchangeCertificate).

Also if your using SSL offloading you do not have to worry about using autodiscover.yourdomain.com on the certificate associated with IIS as the autodiscover requests will be hitting your Exchange server on port 80... another device such as Threat Management Gateway or an F5 BIG-IP will be decrypting the traffic and forwarding it to Exchange 2010.

Thursday, December 15, 2011

I experienced an issue with Microsoft Network Load Balancing (NLB) configured in a multicast mode cluster. I was able to contact my cluster virtual IP address from my other computers on the same subnet (Layer 2), However I was unable to contact the Virtual IP address from a different subnet by routing through my Cisco router (Layer 3).

After a little investigation I stumbled across the following Cisco website:

To resolve this issue I was required to add the MAC address for the Virtual IP a of my NLB cluster to the Cisco router as a static MAC entry.

To grab the MAC address of the Virtual IP address first you need to make connectivity with the IP from another PC on the same subnet. Ping will do the trick. Then display the ARP cache on the PC with arp -a.

To add the static MAC address entry to the Cisco Router use the following commands.

conf tarp 10.4.2.15 03bf.0a04.020f arpa

This will ensure you can route layer 3 to the Virtual IP of your NLB cluster when in Multicast mode.

For a detailed description as to why this happens refer to the above Cisco link.

When using the Exchange Remote Connectivity Analyzer (ExRCA) using the Office 365 Microsoft Single Sign-on (BETA) tool you may experience the following error message.

ExRCA is attempting to retrieve and analyze a security token for user mr.cloud@4logic.com.au.An error occurred while attempting to retrieve and analyze the security token.Test StepsExRCA is attempting to authenticate to the security token service at https://fs.4logic.com.au/adfs/services/trust/2005/usernamemixed.An error occurred while attempting to retrieve the security token. Tell me more about this issue and how to resolve itAdditional DetailsThe Security Token service indicated that the authentication failed. Check the user name and password and try again.

Resolution:

I have seen this problem occur in two circumstances:

a.) the username and password you have entered is incorrect.

b.) the UPN suffix for the user account has not been updated yet to a public UPN suffix. Do this in AD Users and Computers. I set it to @4logic.com.au.

You may have to create a public UPN suffix under Active Directory Domains and Trusts first.

Today I went to connect to Office 365 with single sign-on only to notice that it is no longer working. When using the Exchange Remote Connectivity Analyzer (ExRCA) using the Office 365 Microsoft Single Sign-on (BETA) tool I received the following error:

Validating ADFS metadata for the on-premises ADFS server.There was a problem validating the ADFS metadata.Test StepsRetrieving ADFS metadata information from metadata exchange URL https://fs.4logic.com.au/adfs/services/trust/mex.ExRCA failed to retrieve ADFS metadata.Tell me more about this issue and how to resolve itAdditional DetailsAn HTTP 503 Service Unavailable response was received while trying to validate ADFS metadata.

On my internal network when I tested https://fs.4logic.com.au/adfs/fs/federationserverservice.asmx from Internet Explorer I received

Service UnavailableHTTP Error 503. The service is unavailable.

After further investigation we noticed the AD FS 2.0 Windows Service was not running on my AD FS 2.0 servers.

After starting this service the issue was resolved. If we navigate to https://fs.4logic.com.au/adfs/fs/federationserverservice.asmx to test we can verify that our AD FS servers are giving us XML.

Please note that you cannot use https://fs.4logic.com.au/adfs/fs/federationserverservice.asmx to test your Federation Proxy servers. This test can only be used against AD FS 2.0 servers, not the AD FS 2.0 Proxy servers. When passing this address through your federation proxy you will ALWAYS receive:

Service UnavailableHTTP Error 503. The service is unavailable.

Why was the service not running?

I looked into why the service was not running and in the event log I noticed the service crashed during startup. The following error was generated in the SYSTEM event log:

Looking at the service we can already see it is configured with a delay start.

Interestingly I noticed on the recovery tab of the service by default it is configured to Restart the Service on the First and Second failure. This hints to me that the product team is already aware of this issue with the service crashing on startup. This is most likely due a dependency not starting in time. My AD FS 2.0 servers are also domain controllers which is recommended for small organisations. Being domain controllers means the startup time for these servers is even longer then normal. What I did was in the "Restart service after" box I entered in 2 minutes to ensure the delay was longer. This resolved the problem.

Execution of the Get-FederationInformation cmdlet had thrown an exception. This may indicate invalid parameters in your Hybrid Configuration settings.

Federation information could not be received from the external organization. at Microsoft.Exchange.Management.Hybrid.RemotePowershellSession.RunCommand(String cmdlet, Dictionary`2 parameters, Boolean ignoreNotFoundErrors) '.

Additional troubleshooting information is available in the Update-HybridConfiguration log file located at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Update-HybridConfiguration\HybridConfiguration_12_8_2011_18_2_52_634589641723224440.log.

Externally my autodiscover was working correctly which I confirmed by using the Exchange Remote Connectivity Analyzer (ExRCA).

However what we found was internally the Get-FederationInformation does not use the Service Connection Point (SCP) objects in Active Directory for performing Autodiscover. As a result it must be able to resolve autodiscover.4logic.com.au to an internal IP address.

We have two ways we can do this... add your public DNS zone to your internal network and populate it with internal IP addresses or create an entry in your hosts file on your Hybrid Server. This is what I did.

The hosts file is located under C:\Windows\System32\drivers\etc

Also your certificate attached to Exchange must have a valid SAN name for your autodiscover record.

Generally the Network Load Balancing Virtual IP Addresses do not get registered in DNS automatically. However I found out that if Network Load Balancing is installed and configured on a DNS server, both the virtual network adaptor address and dedicated network adaptor address get registered in DNS.

For my two domain controllers QV1-DC1 and QV1-DC2 there were two A records for each... the servers IP address and my virtual NLB address. DNS round robin, which is enabled by default on all Windows DNS servers was distributing at random either the servers IP address or the virtual address.

The problem here is the virtual address was only listening on port 443 meaning no Active Directory queries could reach the domain controller for any hosts who resolved the virtual IP address.

I found a fix which allowed me register the IP addresses I want on each of my servers to DNS, instead of turning of dynamic DNS updates all together. This registry key is:

In Exchange when you run the test commands such as Test-OutlookWebServices or Test-FederationTrust for example it requires a Exchange Test account. This is usually created by running the New-TestCasConnectivityUser.ps1 script from the Exchange 2010 scripts Directory C:\Program Files\Microsoft\Exchange Server\V14\Scripts.

If you don't create a test user account you will receive such errors in PowerShell:

However today when I ran the New-TestCasConnectivityUser.ps1 it complained that it could not find the Users OU, a problem which I had not seen before. I know that the password I supplied met the complexity requirements. This is the error I was receiving.

Wednesday, November 23, 2011

By default Exchange 2010 Management Shell (EMS 2010) cannot manage/access information stored on the Exchange 2007 servers such as IIS configuration in the example below. When attempting to get information from an Exchange 2007 server in an Exchange 2010 management shell you will receive the following error:

Tuesday, November 22, 2011

Prior to Windows Vista SP1, the Windows RPC/HTTP client-side component required that the Subject Name (aka Common Name) on the certificate match the "Certificate Principal Name" configured for the Outlook Anywhere connection in the Outlook profile. Therefore, as a best practice, you should ensure that your certificate common name is configured as the name Outlook Anywhere references which can be achieved by using the Set-OutlookProvider cmdlet with the -EXPR parameter.

To understand how to configure the EXPR parameter please view this blog post:

Monday, November 21, 2011

I was asked by a client where Exchange 2003 Relay Restrictions are stored? The SMTP Virtual Server in Exchange 2003 is actually apart of IIS and as a result its configuration is not stored in Active Directory (for the most part). This means if you recover an Exchange 2003 server using the Disaster Recovery switch you will need to re-enter this configuration.

Thursday, November 17, 2011

The F5 BIG-IP has a template for Exchange 2010 which assists administrators with configuring load balancing for Outlook Anywhere, Active Sync and Outlook Web App. This template does not configure SMTP load balancing. There are many circumstances where you may want an SMTP endpoint IP address to be highly available and load balanced between multiple hub transport servers.

In this post I will go through and show you how to configure the BIG-IP LTM for load balancing the SMTP protocol and the challenges associated with this. This article was written using the F5 BIG-IP LTM VE version 10.2.3.

Create a Health Monitor

Create a health monitor which monitors the Exchange 2010 SMTP service on our Exchange 2010 servers. The heath monitor will send SMTP HELO requests on a regular basis to ensure the SMTP servers are healthy.

Expand Local Traffic and click Add next to Monitors.

I called the monitor SMTP_Monitor. Set the Type to SMTP. Provided an interval of 120 seconds meaning the monitor will send an SMTP HELO every 2 minutes to the Ex2010 servers to see if they are still online. Configured the Alias service port to 25.

Create a Pool for the SMTP Servers

A load balancing pool is a logical set of devices, in our case SMTP servers, that you group together to receive and process traffic. To create a new pool under Pool List click Add.

I called the pool smtp_pool. Select the SMTP_Monitor we created earlier. Select the load balancing method as Round Robin. Add the SMTP servers to our pool in which we wish to distribute inbound SMTP connections to.

Create an SMTP Virtual Server

Create an SMTP Virtual Server on the F5 BIG-IP which will allow the BIG-IP system to listen on TCP25 to load balance incoming SMTP sessions. To do this under Virtual Servers --> Virtual Server List click add.

I called the SMTP Virtual Server SMTP_VS. Under Destination I specified 172.16.51.174. This is the Virtual IP the BIG-IP will listen on for incoming SMTP traffic. Select a service port of 25 and place the device in an enabled state.

Under configuration set SNAT Pool to Auto Map (I have explained what Auto Map is below).

Scroll down further and under resources set the Default Pool to smtp_pool along with the default persistence profile to source_addr.

Test our BIG-IP SMTP Virtual Server

The F5 BIG-IP device should now be configured to load balance SMTP requests between the two Exchange 2010 servers. In your Virtual Server List the SMTP_VS should come up green.

From a command prompt verify you can telnet our SMTP virtual server on 172.16.51.174 on port 25.

We can see that it successfully connected to one of the SMTP servers in our load balancing pool "smtp_pool"

At this point your F5 BIG-IP is successfully load balancing SMTP.

The Problem...

When building an email solution it is absolutely critical to avoid becoming an open SMTP relay. Most organisations implement relay restrictions by locking anonymous relay down to certain source IP addresses on their internal network such as applications and printers. Source IP is generally the preferred method as Administrators do not have to deal with SMTP authentication methods. The list of IP addresses who are allowed relay anonymously are usually configured on the Exchange SMTP receive connectors. However when dealing with load balancers such as a F5 BIG-IP Local Traffic Manager this becomes a difficult task.

Why?

Whilst load balancing connections the F5 BIGIP uses SNAT to re-write the source IP address on the SMTP packets to one of its "Self IP" addresses or "Virtual IP" addresses. This means the Exchange servers will see all requests coming from the same IP address making it impossible to determine which request belongs to what client. This is illustrated in the following diagram:

However I have a workaround for you. If we setup two SNAT addresses on the F5 BIG-IP for example 172.16.51.174 and 172.16.51.175 we can configure our BIG-IP to say any source IP addresses that need to be an anonymous open relay hit our Exchange 2010 servers from 172.16.51.174 ELSE hit our Exchange 2010 servers from 172.16.51.175. This solution means we need to configure our list of allowed IP addresses for SMTP relay on our F5 BIG-IP instead of our Exchange SMTP Receive Connectors.

Create a Data Group List

First we must create a list of IP addresses we want to allow anonymous relay for on our F5 BIG-IP. These are the IP addresses we would normally configure on our Exchange receive connectors. To do this we need to create a new data group list.

Add a new data group list by expanding iRules --> Data Group List and clicking the add button.

I called my Data Group List smtp_relay_allowed and specified the IP address 172.16.51.21. You can add as many IP addresses as you want for anonymous relay.

Create a new iRule

An iRule is a powerful and flexible feature of BIG-IP devices which provide you with unprecedented control to directly manipulate and manage any IP application traffic. By creating an iRule we can instruct the BIG-IP to return a different SNAT address based on on the condition. We want to instruct our BIG-IP to perform the following:

IF a clients source IP is on our Data Group List THEN use an SNAT address of 172.16.51.174 ELSE use the SNAT address of 172.16.51.175.

To create the iRule under Local Traffic Select iRule --> iRule List and click the add button.

I called my iRule smtp_irule and created the following code to perform my required conditions as mentioned above.

Automap is a feature of the BIGIP where it automatically selects a Self IP at random to use for the SNAT translation. A Self IP is an IP you have assigned to the BIGIP manually under your network configuration. This is different to a Virtual IP address which is created when you setup a virtual server. I only have one Self IP on my BIG IP set to 172.16.51.175 and one virtual IP set to 172.16.51.174 used by all my F5 virtual servers on different TCP ports. As a result automap will ONLY select 172.16.51.175.

Why would you want multiple source SNAT IP addresses?

For each connection made from the BIG-IP to your load balanced servers a TCP source port needs to be opened for the communication. TCP only has 65535 ports for source and destination traffic so if the number of connections exceeded the number of available ports, the BIG-IP would not be able to take new connections. In this event you could add an additional Self IP and rely on the Automap feature or create an SNAT pool which is a predefined list of IP addresses the BIG-IP is allowed to use for SNAT.

I recommend reading the chapter on SNAT configuration from the F5 website:

On your Exchange 2010 servers create your two receive connectors as normal. Create one receive connector configured for your anonymous relay and setup yoru default receive connector for all other SMTP traffic. Both receive connectors must listen on port 25. Do this on all Exchange servers in your pool.

For information on how to configure your application receive connector for anonymous relay follow this blog post: