Wednesday, December 24, 2014

I had a customer where Exchange 2013 FAST Search Technology failed and rebuilding the search indexes for the databases did not resolve the problem on a single Exchange 2013 SP1 server.

In this blog post I'm going to share with you the symptoms of my issue along with the resolution.

Symptoms of Issues

Users on the network were unable to search their mailbox from Outlook Web App or using Microsoft Outlook in online mode.

In addition to not being able to search in Outlook Web App or Microsoft Outlook, the Test-ExchangeSearch cmdlet failed with "Time out for test thread".

Despite this issue, the ContentIndexState on the databases remained in a Healthy report status with the Get-MailboxDatabaseCopyStatus cmdlet as shown below:

In the Server Application logs the following error was logged numerous times:

Log Name: ApplicationSource: MSExchangeISDate: 4/12/2014 1:54:06 PMEvent ID: 1012Task Category: GeneralLevel: ErrorKeywords: ClassicUser: N/AComputer: ExchangeServer.domain.localDescription:Exchange Server Information Store has encountered an error while executing a full-text index query ("eDiscovery search query execution on database a0976351-948a-4625-8840-f649f8b98e0e failed."). Error information: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0821592. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847
Checking the size of the catalog folder was only around 16MB when the database size was 150GB. From my experience a FAST index for a 150GB database is generally between 5GB and 10GB in size depending on how small the items are inside the database.

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):

7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2013 server with repadmin /syncall /APeD.8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatusand review the ContentIndexState.

When the Content Index process has finished crawling we can see the index size of the folder is now 8GB and searching is now working as normal.

Tuesday, December 23, 2014

In todays blog I am going to show you how to filter Message Tracking Log output from the Get-MessageTrackingLog command. I had a customer from an organisation example.com who wanted to see all external recipients who received an email with the message subject "Staffing Update".

Without using some PowerShell filtering, you can view all tracking log related entries for this against all Transport Servers in your organisation by running:

This command will return all messages relayed in the last 30 days by default which is the amount of time the message tracking logs hang around for by default.

However if we want to meet my customers requirement, we want the output to not contain any recipients for "*@example.com" so that we can focus on emails leaving the company. When I say external users or "leaving the company", I mean users who do not have an email address for "*@example.com", you may have an Exchange Organisation where you have multiple accepted domains setup for multi-tenancy!

To do this we need to do some filtering with PowerShell. This can be achieved using the where{ } command as follows:

I have used this tool successfully with previous versions of Exchange Server however with Exchange Server 2013 this tool does not work. I have tested PST Capture 2.0 with Exchange Server 2013 SP1 and Update Rollup 6, both unsuccessful. Testing has been performed both in my lab environment and at a customer site.

I logged this issue with Microsoft under a professional services support case in which Microsoft were able to reproduce the issue in their lab environment. This issue was then escalated within Microsoft and an issue identified with the tool with no fix provided. Microsoft explained they are under no obligation to develop a hotfix for this tool as the End User Licensing Agreement (EULA) states that the software is provided "as is" and they are not under obligation to support it should issues or bugs be flagged with the software. This is shown in my screenshot below.

When attempting to import a mailbox with PST Capture 2.0, the following error message is flagged in the tool:

Import error: Error opening mailbox

This issue is generally caused by one of the following issues as documented on the following website:

The PST Capture tool has a more advanced logging available for errors which can be found under the following location:

"C:\ProgramData\Microsoft\Exchange\PST Capture\Logs\MapiComServer 15"

The technical error which is generated when attempting to import to Exchange 2013 is as follows:

Failed to open default store - HR 8004011d

Hopefully Microsoft does get around to developing a fix/work around for this product as it will be a shame to purchase this tool from Red Gate Software and then not maintain it hence making it redundant.

Wednesday, December 3, 2014

In a previous post I wrote about how to change the network location or "firewall profile" in Windows Server 2012 using PowerShell as you could no longer do this from the GUI like you could in previous versions of Windows Server. This article can be found under the following URL:

However yesterday I had a problem at a customer running Windows Server 2008 where I was unable to change the network location from the GUI using Network and Sharing Centre and as a result I had to revert to the registry. Tracking down the registry key responsible for the network profile was a difficult task so I decided to blog this one.

Tuesday, December 2, 2014

In Windows Server 2012 and Windows Server 2012 R2 by default all users have a Power Shell icon and a Server Manager icon pinned to their start menu. When setting up a Remote Desktop Session Host with Windows Server 2012 Remote Desktop Services you may not want users to have the Server Manager icon or Power Shell icon especially when desktop access is enabled.

In previous versions of Windows Server there was a Group Policy setting called "Remove pinned programs list from the Start Menu" which administrators could use to remove these pinned applications from the start menu from Remote Desktop Session Hosts.

This policy no longer works on Windows Server 2012 and there is no well documented method available for doing this on the Internet.

In Windows Server 2012, the three default pinned applications on the start menu are located under the following location:

The User Pinned\Taskbar folder in the Quick Launch directory does not exist in the Default profile in Windows Server 2012 as shown below.

So how do these icons get created in the users profile and where do they come from?

This process appears to be hardcoded into Windows Server 2012 and executed upon creating a new user profile. The user profile creation process creates Quick Launch\User Pinned\TaskBar folder as part of the login then copies a bunch of "lnk" shortcut files from the "All Users Profile".

"C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\TaskBar\Windows PowerShell.lnk"How to Prevent the icons from getting Populated to User Profiles
To stop the "Windows PowerShell.lnk" and "Server Manager.lnk" shortcuts from getting pinned to the start menu on all users, simply delete these icons from the All Users profile under "C:\ProgramData" from the following locations:

Now when new users login for the first time, the User Profile creation process is unable to copy the "Server Manager.lnk" and "Windows PowerShell.lnk" shortcuts as they no longer exist as shown in the following screenshot.

Sunday, November 30, 2014

There is a bug in Windows Server 2012 R2 which causes Print Management Console to display no printers as shown in the screenshot below. This makes it impossible to deploy printers with Group Policy using the local server.

Print Management Console does however show drivers for any printers installed.

If you connect to the print server remotely from another print server such as a 2008 R2 server, the printers are displayed in the Print Management console as shown below.

This bug is triggered when Microsoft Hotfix KB2995388 is installed on a 2012 R2 print server which is the case in our scenario as shown in the "systeminfo" extract:

To resolve the issue, uninstall KB2995388 on the 2012 R2 Print Server and reboot.

Tuesday, November 18, 2014

I came across a bug with Windows Server 2012 R2 where a spoke server randomly failed to replicate to the hub server. In total there were three DFSR servers partaking in replication within a single replication group.

The following diagram shows an overview of the topology and the server experiencing issues:

When restarting the DFS-R service on SPOKE2, SPOKE2 gets reported to be in an Indeterminate State for approximately 2-3 hours.

After being in an Indeterminate State for 2-3 hours, the status change to “Auto Recovery” for approximately 6 more hours. During this time the DFS-R service generates a large amount of disk activity as it goes through and checks all the files. This can be observed using Windows Resource Monitor.

The Auto Recovery process never completes successfully, nor are there any errors in the event log on SPOKE2. Rebooting SPOKE2 or restarting the DFS-R service results in the server going back to an indeterminate state for another 2-3 hours then starting the Auto Recovery process again.

Resolution

This issue is caused by a bug with Windows Server 2012 R2 documented on Microsoft KB 2951262.

You must manually request the Hotfix which Microsoft will email to you and install it on the effected servers. After installing the hotfix the server will revert to an Indeterminate state then start Auto Recovery again however this time after the Auto Recovery process, it will resume replication as normal.

I was in the process of migrating to Exchange 2013 "Modern Public Folders" for a customer of mine here in Western Australia from an Exchange 2010 SP3 server to Exchange 2013 SP1. After commencing the migration with the New-PublicFolderMigrationRequest cmdlet the migration request shortly after failed.

Looking at the Migration Request Statistics with the following cmdlet it came up with StatusDetail as FailedOther:

FailureCode: -2146233088FailureType: DataValidationExceptionMessage: Error: Property expression "Outlook Security Settings" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase and lowercase), digits from 0 to 9, !, #, $, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in the alias, but each period should be preceded and followed by at least one of the other characters.

In the public folder structure at my customer, there was folder named "Outlook Security Settings" as per the error message.

This folder name or any of the sub items did not appear to breach any of the invalid characters as per the stated error message. Luckily this folder was not required by the business and I simply remove the problematic public folder from the Exchange 2010 server and repeated the move request. After preforming this I was able to successfully complete the public folder migration request.

Monday, November 10, 2014

A customer of mine upgraded their Domain Functional Level (DFL) and Forest Functional Level (FFL) to Windows Server 2008 R2 yesterday. Today when employees started work, they experienced lengthy login times, did not receive their network drive mapping to the file server and were unable to connect to Exchange Server 2010 with Microsoft Outlook 2010.

The first thing I did was have a look at the Active Directory replication after the functional level upgrade using the following command "repadmin /showrepl" on one of the Active Directory domain controllers. This showed the following error:

Last error: -2146892990 (0x80090342)The encryption type requested is not supported by the KDC

Generally this transition is smooth and does not cause problems however in this instance the KDC did not detect the functional level change and continued to operate using the legacy 2003 functional level encryption technology. As a result the error "The encryption type required is not supported by the KDC".

To resolve this problem was very simple, we simply restarted the Kerberos Key Distribution Centre on all of the Active Directory domain controllers in the domain.

Tuesday, November 4, 2014

When migrating to Exchange 2013 modern public folders, you may want to repeat the migration especially if you need to tweak the Public Folder-to-Mailbox Mapping File and perform the migration again.

To repeat the migration progress you need to perform the following steps:

Removing all public folders from the newly created public folder mailbox with Get-PublicFolder –Recurse | Remove-PublicFolder (make sure you run this on Exchange 2013 Management Shell so you don't delete the public folders on your 2007/2010 environment).

At this point you would think you should be ready to migrate the public folders to Exchange 2013 again however there is a final step which needs to be performed. This final step has caught out many Exchange Admins and is not documented at all online. Without the final step, the following error is experienced:

"Failed to submit the request because public folder migration has already been successfully completed previously"

This error occurs if the PublicFolderMigrationComplete attribute is set to $true on the Exchange Organisation Configuration as shown in the following screenshot.

Simply use the following PowerShell command to set the PublicFolderMigrationComplete attribute to $false.

Set-OrganizationConfig -PublicFolderMigrationComplete $false

After this final step you can go ahead and remigrate your public folders to Exchange 2013 from scratch.

Tuesday, October 14, 2014

One of my customers had a mailbox user which generated two Out of Office messages whenever her Out of Office automated replies were enabled. The first Out of Office message is the message the user set on her mailbox as the automated reply for both internal and external Out of Office. The second Out of Office message is a different message for a user which no longer exists on the network and the contents of the Out of Office message does not display in Outlook Web App or Outlook.

The legitimate Out of Office is provided with a message subject of:

Automatic reply:

The second unwanted Out of Office is provided with a message subject of:

Out of Office

I created a test user account and send the problematic user a test email when her Out of Office was enable. The following screenshot shows the extra Out of Office message sent:

First thing I checked was the MailboxAutoReplyConfiguration set on the mailbox which shows the Out of Office message set on the Mailbox for both Internal and External recipients. This did not reveal the second Out of Office message causing the problem for this user account.

This means the second problematic Out of Office response must be stored elsewhere in the users mailbox. Before going gown the path of downloading MFCMAPI and manually browsing through the problematic mailbox for the second Out of Office rule, I decided to run the "outlook.exe /cleanrules" Outlook 2010 command. This command cleans and removes both server side and client side rules including any Out of Office rules which may exist in the mailbox.

Simply close Outlook on the users PC and then execute the following command:

After cleaning the rules and retesting the users Out of Office from my test mailbox, I only received the single Out of Office message which is the one we configured on the users mailbox as shown in the following screenshot.

Summary

Whilst a resolution to the issue was found, I am not sure what caused this issue. The problematic mailbox is a member of a Database Availability Group cluster and was migrated cross-forest from an Exchange 2007 environment. This should not cause problems as Exchange 2007 utilises a similar architecture for Out of Office messages as Exchange 2010 supporting the concept of both Internal and External Out of Office responses. I believe this mailbox has been around for a long time and this legacy rule may have been brought across from Exchange 2003 as this particular customer uses role based mailboxes and when new users come on-board they inherit the last users mailbox keeping the same email address.

The error occurs if the account being moved does not have correct Exchange 2010 permissions set on the user account in Active Directory. On the user account in Active Directory Users and Computers on the Security Tab under Advanced, "Include inheritable permissions from this object's parent" was not selected as shown below:

Note: In order to see the Security Tab on the user account in Active Directory Users and Computers you must enable "Advanced Features" which can be found under the view menu.

Simply select Include inheritable permissions from this object's parent.

In this blog post I am going to touch on these two topics a bit further as this issue will impact many organisations moving to Exchange 2013 for those still running Windows XP (despite the fact it is no longer supported) seeming Exchange 2013 only supports RPC over HTTP or MAPI over HTTP (out of scope for this article). To recap what I wrote about in the previous articles 5 years ago, for RPC over HTTP(s) aka Outlook Anywhere to work on any version of the Outlook Client on the XP, the MSSTD value must match the "Common Name" on the certificate. What am I talking about? Well let me show you...

The MSSTD is specified under "Only connect to proxy servers that have this principal name in their certificate" which can be found within Outlook under Account Settings, Open the Account, More Settings, Connection Tab and finally Exchange Proxy Settings.

This value must match the "Common Name" of the certificate which in this example is mail.example.com as shown below next to "Issued to:"

From Windows Vista onwards the MSSTD can match any name in the Certificate which includes the Certificate Common Name as shown above and any Subject Alternative Names (SAN) which may exist on the certificate. These are often used and can be easily located on the details tab of a certificate in Microsoft Windows as shown below:

For Windows XP the "Only connect to proxy servers that have this principal name in their certificate" value MUST MATCH the common name on the Certificate. The symptom for having this not matching is the user continuously being prompted for credentials in an infinite loop as I addressed under the article Outlook Anywhere keeps prompting for Password.

Note: This also applies to Windows Server 2003 for people with legacy Terminal Server / Citrix environments.

What about for Wild Card certificates, how do I set the "Only connect to proxy servers that have this principal name in their certificate" MSSTD value for these for legacy Windows XP clients?

For Wild Card certificates the MSSTD value must be set to:

msstd:*.domain.com

You must actually put the astricts in the DNS name instead of the name the client is connecting to in order for Windows XP to be happy so it matches the exact name of the Wildcard certificate.

What about Autodiscover?

Now that we understand that the MSSTD must match the common name of the certificate for our legacy Windows XP clients, how do we automatically push this out to our clients via Autodiscover?

Exchange 2013 no longer directly uses EXPR/EXCH Outlook Providers which we configured in Exchange 2007/2010, it has two different dynamically generated EXHTTP providers. Users with mailboxes on 2013 will get one set of EXHTTP settings for internal usage and one set of EXHTTP settings for external usage. You will see this returned to the Outlook client in the Autodiscover response via what looks like a new provider, ExHTTP.

However ExHTTP isn’t an actual provider, it is a calculated set of values from the EXCH (internal Outlook Anywhere) and EXPR (External Outlook Anywhere) settings. The concept of internal Outlook Anywhere and External Outlook Anywhere is new in Exchange 2013 as we

got rid of direct RPC to the Exchange server. We want the ability to set NTLM authentication for internal and Basic authentication for External as well as different connection URLs.

To change the internal MSSTD returned value from Autodiscover use the following command:

Monday, October 6, 2014

When installing Update Rollup 7 on Exchange Server 2010 Service Pack 3 I received the following error message:

Setup Wizard for Update Rollup 7 for Exchange 2010 Service Pack 3 (KB2961522) ended prematurely.Setup Wizard for Update Rollup 7 for Exchange 2010 Service Pack 3 (KB2961522) ended prematurely because of an error. Your system has not been modified. To install this program at a later time, please run the installation again.

This error was encountered after running the Exchange2010-KB2961522-x64-en.msp file downloaded from the Microsoft Website.

In the ServiceControl.txt file located under C:\ExchangeSetupLogs the following errors were present:

[12:58:25] Service 'IIS Admin Service (IISAdmin)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeAB'.[12:58:25] Service 'Microsoft Exchange Address Book (MSExchangeAB)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeADTopology'.[12:58:25] Service 'Microsoft Exchange Active Directory Topology (MSExchangeADTopology)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeAntispamUpdate'.[12:58:25] Service 'Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeEdgeSync'.[12:58:25] Service 'Microsoft Exchange EdgeSync (MSExchangeEdgeSync)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeFBA'.[12:58:25] Service 'Microsoft Exchange Forms-Based Authentication service (MSExchangeFBA)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeFDS'.[12:58:25] Service 'Microsoft Exchange File Distribution (MSExchangeFDS)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeIMAP4'.[12:58:25] Service 'Microsoft Exchange IMAP4 (MSExchangeIMAP4)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeIS'.[12:58:25] Service 'Microsoft Exchange Information Store (MSExchangeIS)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeMailboxAssistants'.[12:58:25] Service 'Microsoft Exchange Mailbox Assistants (MSExchangeMailboxAssistants)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeMailboxReplication'.[12:58:25] Service 'Microsoft Exchange Mailbox Replication (MSExchangeMailboxReplication)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeMailSubmission'.[12:58:25] Service 'Microsoft Exchange Mail Submission (MSExchangeMailSubmission)' cannot be configured due to the following error: Access is denied[12:58:25] Enabling service 'MSExchangeMonitoring'.and so on..

Generally when you run a .msp update file it automatically provides an User Account Control (UAC) exception however on this customers network it did not. As a result UAC prevented the update from performing the required tasks.

To resolve the problem I simply needed to run the update from an elevated command prompt window as follows:

This error occurs if the original Exchange 2010 SP1 or SP2 installation did not finish "Finalizing Setup" which is the final stage in the installation process. Failure means important registry keys are not set correctly. In my case the Finalizing Setup did not complete because of the following error:

This error is generated if the MailboxRole is missing a ConfiguredVersion string value under the following registry key:

HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\MailboxRole

As you see in the following screenshot the ConfiguredVersion string value is missing:

Simply create the string value called ConfiguredVersion and give it the same value as UnpackedVersion as shown in the following screenshot:

After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup. Google for this, lots of information already available on the Internet

When attempting to install Exchange 2010 SP3 service pack on an Exchange 2010 SP1 server I received the following error:

This server role can't be installed because the following roles aren't current: AdminToolsRole

This error occurs if the original Exchange 2010 SP1 or SP2 installation did not finish "Finalizing Setup" which is the final stage in the installation process. Failure means important registry keys are not set correctly. In my case the Finalizing Setup did not complete because of the following error:

The above error is triggered if the ConfiguredVersion does not match the UnpackedVersion registry key. These registry keys can be found under the following location:

HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\AdminTools

To fix this I made sure the ConfiguredVersion key matches the UnpackedVersion key as shown in the following screenshots:

Before:

After:

After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup. Google for this, lots of information already available on the Internet.

Note: If you experienced this error you may also experience the following:

The mailbox role is the last role which is installed as part of the Exchange 2010 SP1 setup process. Despite this error occurring, the Exchange 2010 SP1 installation did complete with all Exchange 2010 services present on the server, started and in a functional state. The Exchange 2010 SP1 management tools were also installed and working.

As a result I pushed on and began installing the Exchange 2010 SP3 upgrade installation package. The Exchange 2010 SP3 upgrade installation package also did not complete the installation correctly and failed with the same error:

In the Exchange 2010 with SP1 installation package, the Management Tools were installed before the Hub Transport, Client Access and Mailbox Server roles. When running Exchange 2010 SP3 upgrade package, the Management Tools are installed after the Hub Transport, Client Access and Mailbox Server roles and as a result were not upgraded as part of the Service Pack. When the Service Pack failed on the Mailbox role it skipped updating the Management Tools - as a result this problem needed to be addressed.

The error is stating it could not resolve user or groups for Discovery Management - there is only one discovery user created by default with Exchange 2010 called:

DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}

As a result I decided to recreate this mailbox by deleting it and recreating it. This was done with the following commands:

After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup. Google for this, lots of information already available on the Internet.

After making these changes I was able to successfully patch the Exchange 2010 SP1 server with Exchange 2010 SP3.