Wednesday, December 24, 2014

Exchange 2013 FAST Search Technology Failed

I had a customer where Exchange 2013 FAST Search Technology failed and rebuilding the search indexes for the databases did not resolve the problem on a single Exchange 2013 SP1 server.

In this blog post I'm going to share with you the symptoms of my issue along with the resolution.

Symptoms of Issues

Users on the network were unable to search their mailbox from Outlook Web App or using Microsoft Outlook in online mode.

In addition to not being able to search in Outlook Web App or Microsoft Outlook, the Test-ExchangeSearch cmdlet failed with "Time out for test thread".

Despite this issue, the ContentIndexState on the databases remained in a Healthy report status with the Get-MailboxDatabaseCopyStatus cmdlet as shown below:

In the Server Application logs the following error was logged numerous times:

Log Name: ApplicationSource: MSExchangeISDate: 4/12/2014 1:54:06 PMEvent ID: 1012Task Category: GeneralLevel: ErrorKeywords: ClassicUser: N/AComputer: ExchangeServer.domain.localDescription:Exchange Server Information Store has encountered an error while executing a full-text index query ("eDiscovery search query execution on database a0976351-948a-4625-8840-f649f8b98e0e failed."). Error information: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0821592. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847
Checking the size of the catalog folder was only around 16MB when the database size was 150GB. From my experience a FAST index for a 150GB database is generally between 5GB and 10GB in size depending on how small the items are inside the database.

.\installconfig.ps1 -action I -datafolder "c:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"This script will recreate the files under "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis" that we backed up/removed previously.6. Once the script installs the data, you need to uninstall and reinstall again (don't ask, it did not work for me without doing this):

7. Force Active Directory Replication from a domain controller in the same AD Site as the Exchange 2013 server with repadmin /syncall /APeD.8. Check on the Exchange Server with Get-MailboxDatabaseCopyStatusand review the ContentIndexState.

When the Content Index process has finished crawling we can see the index size of the folder is now 8GB and searching is now working as normal.

7 comments:

Clint - thank you for this information. We are suffering through the exact same issue but it seems to be confined to one database. We checked the catalogue folder size and each one is anywhere between 13 - 15 gb so not sure if this is the issue though. When we test using Test-ExchangeSearch we get a time out on test thread for mailboxes on this database only. Thoughts?

Unfortunately it did no work. When I try to reinstall the search, it only creates AdminNode1 and then aborts after a certain time (timeout). we could not manage to get the index running again. It also tries to get a noderunner task up and running, but it seems to fail (timeout). It seems like a permission issue if we try to reinstall search (with elevated rights, as admin).