Cloud Search Service Application Crawl Performance

I’ve been working on a Hybrid Search farm using SharePoint Server 2013 with the November 2016 Cumulative Update. The Cloud Search Service Application crawl performance was extremely poor, below 1 document per second (DPS) submitted to the SharePoint Online Search index. What I was seeing was Azure was throttling my index submissions.

This particular Content Source had roughly 9 million items to index, and due to these throttling events, the indexing was taking significantly longer than I had hoped for; to give you an idea, roughly 550k items in 150 hours. Eventually, a support case with Microsoft was opened via the Office 365 tenant. That support told me because it is hybrid, I needed to have a standard support case opened via Product Support Services. This is despite the option to pick “hybrid search” as a category in the Office 365 tenant support ticket creation.

Back on-prem, support had me implement a new property, “EnableNoGetStatusFlight”, on the Cloud SSA.

Once the property is in place, you must restart the OSearch15 service on servers hosting the Crawl component. There is no other documentation on this property yet, but it prevents the crawler from requesting the status of submitted batches to SharePoint Online, thus increasing throughput.

After implementing this property, roughly 1 million items were crawled in the span of about 14 hours. That’s a significant improvement to our DPS.

This isn’t a cure-all and you still have a chance to be throttled as it depends on the rate in which you are submitting items to Azure as well as what load the SharePoint Online search farm is under.